SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Udaya Ghai†*
Amazon  
ughai@amazon.com
Arushi Gupta
Princeton University & Google DeepMind arushig@princeton.edu
Wenhan Xia
Princeton University & Google DeepMind  
wxia@princeton.edu
Carnegie Mellon University karansingh@cmu.edu
Princeton University & Google DeepMind  
ehazan@princeton.edu
We investigate robust model-free reinforcement learning algorithms designed for environments that may be dynamic or even adversarial. Traditional state-based policies often struggle to accommodate the challenges imposed by the presence of unmodeled disturbances in such settings. Moreover, optimizing linear state-based policies pose an obstacle for efficient optimization, leading to nonconvex objectives, even in benign environments like linear dynamical systems.
Drawing inspiration from recent advancements in model-based control, we introduce a novel class of policies centered on disturbance signals. We define several categories of these signals, which we term pseudo-disturbances, and develop corresponding policy classes based on them. We provide efficient and practical algorithms for optimizing these policies.
Next, we examine the task of online adaptation of reinforcement learning agents in the face of adversarial disturbances. Our methods seamlessly integrate with any black-box model-free approach, yielding provable regret guarantees when dealing with linear dynamics. These regret guarantees unconditionally improve the best-known results for bandit linear control in having no dependence on the state-space dimension. We evaluate our method over various standard RL benchmarks and demonstrate improved robustness.
Model-free reinforcement learning in time-varying responsive dynamical systems is a statistically and computationally challenging problem. In contrast, model based control of even unknown and changing linear dynamical systems has enjoyed recent successes. In particular, new techniques from online learning have been applied to these linear dynamical systems (LDS) within the framework of online nonstochastic control. A comprehensive survey can be found in Hazan and Singh [2022]. The key innovation in the aforementioned framework is the introduction of a new policy class called Disturbance-Action Control (DAC), which achieves a high degree of representational capacity without compromising computational efficiency. Moreover, efficient gradient-based algorithms can
*Work performed while at Princeton University and Google. † denotes equal contribution.
be employed to obtain provable regret bounds for this approach, even in the presence of adversarial noise. Crucially, these methods rely on the notion of disturbance, defined to capture unmodeled deviations between the observed and nominal dynamics, and its availability to the learner.
This paper explores the potential of applying these disturbance-based techniques, which have proven effective in model-based control, to model-free reinforcement learning. However, it is not immediately clear how these methods can be adapted to model-free RL, as the disturbances in model-free RL are unknown to the learner.
We therefore develop the following approach to this challenge: instead of relying on a known disturbance, we create a new family of signals, which we call "Pseudo-Disturbances", and define policies that use "Pseudo-Disturbance" features to produce actions. The advantage of this approach is that it has the potential to produce more robust policies. Again inspired by model-based methods, we aim to augment existing reinforcement learning agents with a "robustness module" that serves two purposes. Firstly, it can filter out adversarial noise from the environment and improve agent performance in noisy settings. Secondly, in cases where the environment is benign and simple, such as a linear dynamical system, the augmented module will achieve a provably optimal solution. We also empirically evaluate the performance of our method on OpenAI Gym environments.
In this work, we make the following algorithmic and methodological contributions:
- In contrast to state-based policies commonly used in RL, Section 3 defines the notion of a disturbance-based policy. These policies augment traditional RL approaches that rely strictly on state feedback.
- We develop three distinct and novel methods (Sections 3.1, 3.2, 3.3) to estimate the Pseudo-Disturbance in the model-free RL setting.
- We develop a new algorithm, MF-GPC (Algorithm 1), which adapts existing RL methods to take advantage of our Pseudo-Disturbance framework.
- We empirically evaluate our method on OpenAI Gym environments in Section 5. We find that our adaptation applied on top of a DDPG baseline performs better than the baseline, significantly so in same cases, and has better robustness characteristics.
- We prove that the proposed algorithm achieves sublinear regret for linear dynamics in Theorem 4. These regret bounds improve upon the best-known for bandit linear control in terms of their dependence on state space dimension (Appendix E). Notably, our bounds have no dependence on the state dimension, reducing the state-of-the-art regret bound by factors of  $\sqrt{d_x}$  for convex losses and  $d_x^{2/3}$  if losses are additionally smooth, signalling that our methodology is better suited to challenging high-dimensional under-actuated settings.
1.2 Pseudo-Disturbance based RL
A fundamental primitive of the non-stochastic control framework is the disturbance. In our RL setting, the system evolves according to the following equation
$$
\mathbf {x} _ {t + 1} = f \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) + \mathbf {w} _ {t},
$$
where  $\mathbf{x}_t$  is the state,  $\mathbf{u}_t$  is control signal, and  $\mathbf{w}_t$  is a bounded, potentially adversarially chosen, disturbance. Using knowledge of the dynamics,  $f$ , non-stochastic control algorithms first compute  $\mathbf{w}_t$ , and then compute actions via DAC, as follows
$$
\mathbf {u} _ {t} = \pi_ {\text {b a s e}} (\mathbf {x} _ {t}) + \sum_ {i = 1} ^ {h} M _ {i} ^ {t} \mathbf {w} _ {t - i}.
$$
Here  $\pi_{\mathrm{base}}$  is a baseline linear controller, and  $M^t$  are matrices, learned via gradient descent or similar algorithms. For linear systems, the DAC law is a convex relaxation of linear policies, which allows us to prove regret bounds against powerful policy classes using tools from online convex optimization.
To generalize this approach, without a model or knowledge of the dynamics function  $f$ , both defining and obtaining this disturbance in order to implement DAC or similar policies becomes unclear. To
address this, we introduce the concept of a Pseudo-Disturbance (PD) and provide three distinct variants, each representing a novel signal in reinforcement learning. These signals have various advantages and disadvantages depending on the available environment:
1. The first notion is based on the gradient of the temporal-difference error. It assumes the availability of a value function oracle that can be evaluated or estimated online or offline using any known methodology.
2. The second notion also assumes the availability of a black-box value function oracle/generator. We assign artificial costs over the states and generate multiple auxiliary value functions to create a "value vector." The Pseudo-Disturbance is defined as the difference between the value vector at consecutive states. This signal's advantage is that it does not require any zero-order optimization mechanism for estimating the value function's gradient.
3. The third notion assumes the availability of an environment simulator. The Pseudo-Disturbance is defined as the difference between the true state and the simulated state for a specific action.
For all these Pseudo-Disturbance variants, we demonstrate how to efficiently compute them (under the appropriate assumption of either a value function oracle or simulator). We provide a reduction from any RL algorithm to a PD-based robust counterpart that converts an RL algorithm into one that is also robust to adversarial noise. Specifically, in the special case of linear dynamical systems our algorithm has provable regret bounds. The formal description of our algorithm, as well as a theorem statement, are given in Section 4. For more general dynamical systems, the learning problem is provably intractable. Nonetheless, we demonstrate the efficacy of these methods empirically.
Model-free reinforcement learning. Reinforcement learning [Sutton and Barto, 2018] approaches are classified as model-free or model-based [Janner et al., 2019, Ha and Schmidhuber, 2018, Osband and Van Roy, 2014], dependent on if they attempt to explicitly try to learn the underlying transition dynamics an agent is subject to. While the latter is often more sample efficient [Wang et al., 2019], model-free approaches scale better in that their performance does not prematurely saturate and keeps improving with number of episodes [Duan et al., 2016]. In this paper, we focus on adaption to unknown, arbitrary disturbances for model-free reinforcement learning algorithms, which can be viewed as a tractable restriction of the challenging adversarial MDP setting [Abbasi Yadjkori et al., 2013]. Model-free approaches may further be divided into policy-based [Schulman et al., 2015, 2017], value-based approaches [Mnih et al., 2013], and actor-critic approaches [Barth-Maron et al., 2018, Lillicrap et al., 2016]; the latter use a learnt value function to reduce the variance for policy optimization.
Robust and Adaptive reinforcement learning. Motivated by minimax performance criterion in robust control [Zhang et al., 2021, Morimoto and Doya, 2005] introduced to a minimax variant of Q-learning to enhance of he robust of policies learnt from off-policy samples. This was later extended to more tractable formulations and structured uncertainty sets in Tessler et al. [2019], Mankowitz et al. [2019], Pinto et al. [2017], Zhang et al. [2021], Tamar et al. [2013], including introductions of model-based variants [Janner et al., 2019]. Another approach to enhance the robustness is Domain Randomization [Tobin et al., 2017, Akkaya et al., 2019, Chen et al., 2021a], wherein a model is trained in a variety of randomized environments in a simulator, and the resulting policy becomes robust enough to be applied in the real world. Similarly, adversarial training [Mandlekar et al., 2017, Vinitsky et al., 2020, Agarwal et al., 2021] has been shown to improve performance in out-of-distribution scenarios. In contrast to the previously mentioned approaches, our proposed approach only adapts the policy to observed disturbances at test time, and does not require a modification of the training procedure. This notably means that the computational cost and sample requirement of the approach matches that of vanilla RL in training, and has the benefit of leveraging recent advances in mean-reward RL, which is arguably better understood and more studied. Adaption of RL agents to new and changing environments has been similarly tackled through the lens of Meta Learning and similar approaches [Wang et al., 2016, Nagabandi et al., 2018, Pritzel et al., 2017, Agarwal et al., 2021].
Online nonstochastic control. The presence of arbitrary disturbances during policy execution had been for long in the fields of robust optimization and control [Zhou and Doyle, 1998]. In contrast to minimax objectives considered in robust control, online nonstochastic control algorithms (see Hazan and Singh [2022] for a survey) are designed to minimize regret against a benchmark policy class, and thus compete with the best policy from the said class determined posthoc. When the benchmark policy class is sufficiently expressive, this approach has the benefit of robustness against adversarially chosen disturbances (i.e. non-Gaussian and potentially adaptively chosen [Ghai et al., 2021]), while distinctly not sacrificing performance in the typical or average case. The first nonstochastic control algorithm with sublinear regret guarantees was proposed in Agarwal et al. [2019] for linear dynamical systems. It was subsequently extended to partially observed systems [Simchowitz et al., 2020], unknown systems [Hazan et al., 2020], multi-agent systems [Ghai et al., 2022] and the time-varying case [Minasyan et al., 2021]. The regret bound was improved to a logarithmic rate in Simchowitz [2020] for strongly convex losses. Chen et al. [2021b] extend this approach to non-linearly parameterized policy classes, like deep neural networks. Bandit versions of the nonstochastic control setting have also been studied [Gradu et al., 2020, Cassel and Koren, 2020, Sun et al., 2023] and are particularly relevant to the RL setting, which only has access to scalar rewards.
1.4 Paper Outline
After some basic definitions and preliminaries in Section 2, we describe the new Pseudo-Disturbance signals and how to create them in a model-free reinforcement learning environment in Section 3. In Section 4 we give a unified meta-algorithm that exploits these signals and applies them as an augmentation to any given RL agent. In Section 5 we evaluate our methods empirically.
An overview of notation can be found in Appendix A. Appendix B contains additional experimental details. Generalization of our algorithm to discrete spaces is provided in Appendix C. Proofs for Section 3 are provided in Appendix D, while the main theorem is proved in Appendix E.
2 Setting and Preliminaries
Consider an agent adaptively choosing actions in a dynamical system with adversarial cost functions. We use notation from the control literature:  $\mathbf{x}_t \in \mathbb{R}^{d_x}$  is a vector representation of the state at time  $t$ ,  $\mathbf{u}_t \in \mathbb{R}^{d_u}$  is the corresponding action. Formally, the evolution of the state will follow the equations
where  $\mathbf{w}_t$  is an arbitrary (even adversarial) disturbance the system is subject to at time  $t$ . Following this evolution, the agent suffers a cost of  $c_t(\mathbf{x}_t,\mathbf{u}_t)$ .
In this work, we adapt model-free reinforcement learning algorithms to this more challenging case. The (easier) typical setting for model-free methods assume, in contrast, that the disturbance  $\mathbf{w}_t$  is sampled  $iid$  from a distribution  $\mathcal{D}$ , and that the cost functions  $c(\mathbf{x},\mathbf{u})$  is fixed and known. Central to the study of model-free methods are the notions of the state and state-action value functions, defined as the discounted sum of future costs acquired by starting at any state (or state-action pair) and thereafter following the policy  $\pi$ . For any policy  $\pi$ , we denote the state and state-action value functions, which are mappings from state or state/action pair to the real numbers, as
$$
Q _ {\pi} (\mathbf {x}, \mathbf {u}) = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c (\mathbf {x} _ {t} ^ {\pi}, \mathbf {u} _ {t} ^ {\pi}) \Bigg | \mathbf {x} _ {0} ^ {\pi} = \mathbf {x}, \mathbf {u} _ {0} ^ {\pi} = \mathbf {u} \right], V _ {\pi} (\mathbf {x}) = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} c (\mathbf {x} _ {t} ^ {\pi}, \mathbf {u} _ {t} ^ {\pi}) \Bigg | \mathbf {x} _ {0} ^ {\pi} = \mathbf {x} \right],
$$
where expectations are taken over random transitions in the environment and in the policy.
A special case we consider is that of linear dynamical systems. In these special instances the state involves linearly according to a linear transformation parameterized by matrices  $A, B$ , i.e.
$$
\mathbf {x} _ {t + 1} = A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}.
$$
2Although we consider continuous state and action spaces in this section and the remainder of the main paper, we handle discrete spaces in Appendix C.
3 Pseudo-Disturbance Signals and Policies
In this section we describe the three different Pseudo-Disturbance (PD) signals we can record in a general reinforcement learning problem. As discussed, the motivation for this signal comes from the framework of online nonstochastic control. We consider dynamical systems with an additive misspecification or noise structure,
$$
\mathbf {x} _ {t + 1} = f (\mathbf {x} _ {t}, \mathbf {u} _ {t}) + \mathbf {w} _ {t},
$$
where the perturbation  $\mathbf{w}_t$  does not depend on the state. Using perturbations rather than state allows us to avoid recursive structure that makes the optimization landscape challenging and nonconvex. As discussed, we introduce Pseudo-Disturbance signals  $\hat{\mathbf{w}}_t\in \mathbb{R}^{d_w}$  in lieu of the true disturbances. We note that the PD dimensionality  $d_{w}$  need not be the same as that of the true disturbance,  $d_{x}$ .
An important class of policies that we consider henceforth is linear in the Pseudo-Disturbance, i.e.
$$
\Pi_ {\mathrm {D A C}} = \left\{\pi (\mathbf {x} _ {1: t}) = \pi_ {\mathrm {b a s e}} (\mathbf {x} _ {t}) + \sum_ {i = 1} ^ {h} M _ {i} \hat {\mathbf {w}} _ {t - i} \Bigg |   M _ {i} \in \mathbb {R} ^ {d _ {u} \times d _ {w}} \right\}.
$$
Here  $\Pi_{\mathrm{DAC}}$  denotes the policy class of Disturbance-Action-Control. The fact that  $\mathbf{w}_t$  does not depend on our actions allows for convex optimization of linear disturbance-action controllers in the setting of linear dynamical systems, see e.g. Hazan and Singh [2022].
We would like to capture the essence of this favorable phenomenon in the context of model free RL, but what would replace the perturbations  $\mathbf{w}_t$  without a dynamics model  $f$ ? That's the central question of this section, and we henceforth give three different proposals for this signal.
An important goal in constructing these signals is that in the case of linear dynamical systems, it recovers the perturbation. This will enable us to prove regret bounds in the case the environment is an LDS.
3.1 Pseudo-Disturbance Class I: Value-Function Gradients
The first signal we consider is based on the gradient of the value function. The value function maps the state onto a scalar, and this information is insufficient to recreate the perturbation even if the underlying environment is a linear dynamical system. To exact a richer signal, we thus consider the gradient of the value function with respect to the action and state. The basic goal is to implement the following equation
$$
\hat {\mathbf {w}} _ {t} = \nabla_ {\mathbf {u}} (\gamma V _ {\pi} (f (\mathbf {x} _ {t}, \mathbf {u}) + \mathbf {w} _ {t}) - (Q _ {\pi} (\mathbf {x} _ {t}, \mathbf {u}) - c (\mathbf {x} _ {t}, \mathbf {u})) | _ {\mathbf {u} = \mathbf {u} _ {t}},
$$
where  $f(\mathbf{x}_t, \mathbf{u}) + \mathbf{w}_t$  represents the counterfactual next state after playing  $\mathbf{u}$  at state  $\mathbf{x}_t$ . Note, this signal is a gradient of the temporal-difference error [Sutton and Barto, 2018], in fact being syntactically similar to expected SARSA. If  $\mathbf{w}_t$  was in fact (iid) stochastic with  $V_{\pi}, Q_{\pi}$  as corresponding value functions, this term on expectation would be zero. Therefore, this signal on average measures deviation introduced in  $\mathbf{x}_{t+1}$  due to arbitrary or adversarial  $\mathbf{w}_t$ . We can also view this expression as
$$
\hat {\mathbf {w}} _ {t} = \nabla_ {\mathbf {u}} \left(\gamma V _ {\pi} \left(f \left(\mathbf {x} _ {t}, \mathbf {u}\right) + \mathbf {w} _ {t}\right) - \gamma V _ {\pi} \left(f \left(\mathbf {x} _ {t}, \mathbf {u}\right)\right)\right) | _ {\mathbf {u} = \mathbf {u} _ {t}}.
$$
$V_{\pi}$  is quadratic in the linear quadratic regulator setting, so this becomes a linear function of  $\mathbf{w}_t$ . Computing  $\nabla_{\mathbf{u}}V_{\pi}(f(\mathbf{x}_t,\mathbf{u}) + \mathbf{w}_t)|_{\mathbf{u} = \mathbf{u}_t}$  analytically would require knowledge of the dynamics, but luckily this can be efficiently estimated online. Using a policy  $\pi$ , with noised actions  $\mathbf{u}_t = \pi (\mathbf{x}_t) + \mathbf{n}_t$  for  $\mathbf{n}_t\sim \mathcal{N}(0,\Sigma)$  we have the following PD estimates:
$$
\boxed {\hat {\mathbf {w}} _ {t} = \gamma V _ {\pi} (\mathbf {x} _ {t + 1}) \Sigma^ {- 1} \mathbf {n} _ {t} - \nabla_ {\mathbf {u}} (Q _ {\pi} (\mathbf {x} _ {t}, \mathbf {u}) - c (\mathbf {x} _ {t}, \mathbf {u})) | _ {\mathbf {u} = \mathbf {u} _ {t}},} \tag {1}
$$
$$
\boxed {\hat {\mathbf {w}} _ {t} = \left(c \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) + \gamma V _ {\pi} \left(\mathbf {x} _ {t + 1}\right) - Q _ {\pi} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right)\right) \Sigma^ {- 1} \mathbf {n} _ {t}.} \tag {2}
$$
These are zeroth-order gradient estimators (see [Liu et al., 2020] for a more detailed exposition). Intuitively, the second estimator may have lower variance as the expected SARSA error can be much smaller than the magnitude of the value function. An additional benefit is that this implementation only requires a scalar cost signal without needing access to a differentiable cost function.
The most important property of this estimator is that it, in expectation, it produces a signal that is a linearly transformation of the true disturbance if the underlying setting is a linear dynamical system. This is formalized in the following lemma.
Lemma 1. Consider a time-invariant linear dynamical systems with system matrices  $A$ ,  $B$  and quadratic costs, along with a linear baseline policy  $\pi$  defined by control law  $\mathbf{u}_t = -K_{\pi}\mathbf{x}_t$ . In expectation, the pseudo disturbances (1) and (2) are linear transformations of the actual perturbation
$$
\mathbb {E} \left[ \hat {\mathbf {w}} _ {t} \mid \mathbf {x} _ {t} \right] = T \mathbf {w} _ {t},
$$
where  $T$  is a fixed linear operator that depends on the system.
3.2 Pseudo-Disturbance Class II: Vector Value Functions
The second approach derives a signal from auxiliary value functions. Concretely, instead of scalar-valued cost function  $c: \mathbb{R}^{d_x} \to \mathbb{R}$ , consider a vector-valued cost function  $\mathbf{c}: \mathbb{R}^{d_x} \to \mathbb{R}^{d_w}$ . For such vector-valued cost, we introduce vectorized value and state-action value functions as
$$
V _ {\pi} ^ {\mathbf {c}}: \mathbb {R} ^ {d _ {x}} \rightarrow \mathbb {R} ^ {d _ {w}}, Q _ {\pi} ^ {\mathbf {c}}: \mathbb {R} ^ {d _ {x}} \times \mathbb {R} ^ {d _ {u}} \rightarrow \mathbb {R} ^ {d _ {w}}.
$$
$$
Q _ {\pi} ^ {\mathbf {c}} (\mathbf {x}, \mathbf {u}) = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbf {c} (\mathbf {x} _ {t} ^ {\pi}) \Bigg | \mathbf {x} _ {0} ^ {\pi} = \mathbf {x}, \mathbf {u} _ {0} ^ {\pi} = \mathbf {u} \right], V _ {\pi} ^ {\mathbf {c}} (\mathbf {x}) = \mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \mathbf {c} (\mathbf {x} _ {t} ^ {\pi}) \Bigg | \mathbf {x} _ {0} ^ {\pi} = \mathbf {x} \right].
$$
Our PD signal is then
$$
\boxed {\hat {\mathbf {w}} _ {t} = \mathbf {c} (\mathbf {x} _ {t}) + \gamma \mathbf {V} _ {\pi} ^ {\mathbf {c}} (\mathbf {x} _ {t + 1}) - \mathbf {Q} _ {\pi} ^ {\mathbf {c}} (\mathbf {x} _ {t}, \mathbf {u} _ {t}).} \tag {3}
$$
In contrast to the first approach, for a fixed set of cost functions, this approach provides a deterministic PD-signal. This is very beneficial, as at inference time the DAC policy can be run without injecting additional noise and without requiring a high variance stochastic signal. This does come at a cost, as this method requires simultaneous off-policy evaluation for many auxiliary value functions (each corresponding to a different scalar cost) before DAC can be run via  $Q$ -function evaluations at inference, both of which can be significantly more expensive than the first approach.
For the case of linear dynamical systems, if we use linear costs on top of a linear base policy, this approach can recover the disturbances up to a linear transformation. It can be seen that the values corresponding to a linear cost function  $c$  are linear functions of the state, and hence the vectorized versions are also linear functions of state. We formalize this as follows:
Lemma 2. Consider a time-invariant linear dynamical systems with system matrices  $A$ ,  $B$ , along with a linear baseline policy  $\pi$  defined by control law  $\mathbf{u}_t = -K_{\pi} \mathbf{x}_t$ . Let  $\mathbf{V}_{\pi}^{\mathbf{c}}$  and  $\mathbf{Q}_{\pi}^{\mathbf{c}}$  be value functions for  $\pi$  for i.i.d. zero mean noise with linear costs  $\mathbf{c}(x) := Lx$ , then the PD-signal (3) is a linear transformation
$$
\hat {\mathbf {w}} _ {t} = T \mathbf {w} _ {t},
$$