SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $T$  is a fixed linear operator that depends on the system and baseline policy  $\pi$ . In addition, if  $L$  is full rank and the closed loop dynamics are stable, then  $T$  is full rank.
3.3 Pseudo-Disturbance Class III: Simulator Based
The last Pseudo-Disturbance signal we consider requires a potentially inaccurate simulator. It is intuitive, particularly simple to implement, and yet comes with theoretical guarantees.
The Pseudo-Disturbance is taken to be the difference between the actual state reached in an environment, and the expected state, over the randomness in the environment. To compute the expected state, we require the simulator  $f_{\mathrm{sim}}$  initialized at the current state. Formally,
$$
\boxed {\hat {\mathbf {w}} _ {t} = \mathbf {x} _ {t + 1} - f _ {\mathrm {s i m}} (\mathbf {x} _ {t}, \mathbf {u} _ {t}).} \tag {4}
$$
The simplicity of this PD is accompanied by a simple lemma on its characterization of the disturbance in a dynamical system, even if that system is time varying, as follows,
Lemma 3. Suppose we have a simulator  $f_{sim}$  such that  $\forall \mathbf{x}, \mathbf{u}, \| f_{sim}(\mathbf{x}, \mathbf{u}) - f(\mathbf{x}, \mathbf{u}) \| \leq \delta$ , then Pseudo-Disturbance (4) is approximately equal to the actual perturbation  $\| \widehat{\mathbf{w}_t} - \mathbf{w}_t \| \leq \delta$ .
3.4 Merits of different Pseudo-Disturbance signals
Each of the three PD signals described in this section offers something a bit different. PD3 offers the most direct disturbance signal, but comes with the requirement of a simulator. If the simulator is very accurate, this is likely the strongest signal, though this method may not be suitable with a large sim-to-real gap. PD1 and PD2 on the other hand, do not require a simulator but also have a natural trade off. PD1 is simpler and easier to add on top of an existing policy. However, it uses zeroth-order estimation, so the guarantees only hold in expectation and it may have high variance. On the other hand, PD2 is not a stochastic estimate, but it requires auxiliary value estimation from the base policy. This may come at the cost of additional space and computational complexity. In many cases, this can be handled using the same deep Q-network except with a wider head, which may not be so onerous. We note that PD2 does not require specific domain engineered signals for the auxiliary rewards. For example, using the coordinates of the state representation was enough to demonstrate improvements over baselines in our experiments. For richer, higher dimensional (visual) state spaces, this can be generalized using neural representations of state as the auxiliary reward, achieved by modulating the PD2 disturbance dimension to account for the fact that the underlying dynamics are simpler.
4 Meta Algorithm and Main Theorem
In this section we define a meta-algorithm for general reinforcement learning. The algorithm takes as an input an existing RL method, that may or may not have theoretical guarantees. It adds an additional layer on top, which estimates the Pseudo-Disturbances according to one of the three methods in the previous section. It then uses an online gradient method to optimize a linear policy in the past Pseudo-Disturbances. This can be viewed as a zeroth-order model-free version of the Gradient Perturbation Controller (GPC) [Agarwal et al., 2019].
The algorithm is formally defined in Algorithm 1. A typical choice of the parametrization  $\pi (\cdot |\mathcal{M})$  is a linear function of a window of past disturbances (ie. Disturbance Action Control [Agarwal et al., 2019]).
$$
\pi \left(\mathbf {w} _ {t - 1: t - h} \mid M _ {1: h}\right) = \sum_ {i = 1} ^ {h} M _ {i} \mathbf {w} _ {t - i}. \tag {5}
$$
Algorithm 1 MF-GPC (Model-Free Gradient Perturbation Controller)
1: Input: Memory parameter  $h$ , learning rate  $\eta$ , exploration noise covariance  $\Sigma$ , initialization  $M_{1:h}^{1} \in \mathbb{R}^{d_u \times d_w \times h}$ , initial value and  $Q$  functions, base RL algorithm  $\mathcal{A}$ .
2: for  $t = 1 \dots T$  do
3: Use action  $\mathbf{u}_t = \pi_{\mathrm{base}}(\mathbf{x}_t) + \pi (\hat{\mathbf{w}}_{t - 1:t - h}|M^t) + \mathbf{n}_t$  where  $\mathbf{n}_t$  is iid Gaussian, i.e.
$$
\mathbf {n} _ {t} \sim \mathcal {N} (0, \Sigma)
$$
4: Observe state  $\mathbf{x}_{t + 1}$ , and cost  $c_{t} = c_{t}(\mathbf{x}_{t},\mathbf{u}_{t})$
5: Compute Pseudo-Disturbance [see (2),(3), (4)]
$$
\hat {\mathbf {w}} _ {t} = \operatorname {P D - e s t i m a t} \left(\mathbf {x} _ {t + 1}, \mathbf {x} _ {t}, \mathbf {u} _ {t}, c _ {t}, \mathbf {n} _ {t}\right).
$$
6: Update policy parameters using the stochastic gradient estimate (see Section 4.1)
$$
M ^ {t + 1} \leftarrow M ^ {t} - \eta c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) \Sigma^ {- 1} \sum_ {j = 0} ^ {h - 1} \mathbf {n} _ {t - i} \otimes J _ {i} ^ {t},
$$
where  $\otimes$  is an outer product and  $J_{i}^{t} = \hat{\mathbf{w}}_{t - i - 1:t - h - i}$  for (5), and more generally,
$$
J _ {i} ^ {t} = \left. \frac {\partial \pi (\hat {\mathbf {w}} _ {t - i - 1 : t - h - i} | M _ {i})}{\partial M} \right| _ {M = M ^ {t}}.
$$
7: end for
8: Optionally, update the policy  $\pi_{\mathrm{base}}$  and its  $Q,V$  functions using  $\mathcal{A}$  so that they are Bellman consistent, i.e. they satisfy the policy version of Bellman equation.
We prove the following theorem for the case of linear dynamics:
Theorem 4 (Informal Statement (see Theorem 8)). If the underlying dynamics are linear with the state evolution specified as
$$
\mathbf {x} _ {t + 1} = A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t},
$$
with  $d_{\min} = \min \{d_x, d_u\}$ , then then as long as the Pseudo-Disturbance signal  $\hat{\mathbf{w}}_t$  satisfies  $\hat{\mathbf{w}}_t = T\mathbf{w}_t$ , for some (possibly unknown) invertible map  $T$ , Algorithm 1 generates controls  $\mathbf{u}_t$  such that for any sequence of bounded (even adversarial)  $\mathbf{w}_t$  such that the following holds
$$
\sum_ {t} c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \leq \min  _ {\pi \in \Pi^ {D A C}} \sum_ {t} c _ {t} \left(\mathbf {x} _ {t} ^ {\pi}, \mathbf {u} _ {t} ^ {\pi}\right) + \widetilde {\mathcal {O}} \left(\sqrt {d _ {u} d _ {\operatorname* {m i n}}} T ^ {3 / 4}\right),
$$
for any any sequence of convex costs  $c_t$ , where the policy class  $DAC$  refers to all policies  $\pi$  that produce a control as a linear function of  $\mathbf{w}_t$ . Further, if the costs  $c_t$  are  $L$ -smooth, the regret for Algorithm 1 admits an improved upper bound of  $\widetilde{\mathcal{O}}((d_u d_{\min} T)^{2/3})$ .
In particular, the above theorem implies the stated regret bounds when the Pseudo-Disturbance is estimated as described in Equations 3 (Vector Value Function-based) and 4 (Simulator-based).
The regret bounds in Theorem 4 are strict improvements over state-of-the-art bounds in terms of dimension dependence; the latter operate with explicit descriptions of disturbances. This is achieved by using a better choice of gradient estimator, using exploration in action-space rather than parameter-space. As a result, our bounds have no dependence on the state dimension since  $d_{\mathrm{min}} \leq d_u$ . As an instructive case, for high-dimensional underactuated systems, where  $d_u < d_x$ , our regret bounds scale as  $\tilde{O}(d_u T^{3/4})$  in contrast to  $\tilde{O}(d_u d_x^{1/2} T^{3/4})$  for convex costs from [Gradu et al., 2020, Cassel and Koren, 2020], and as  $\tilde{O}(d_u^{4/3} T^{2/3})$  for smooth costs improving over  $\tilde{O}(d_u^{4/3} d_x^{2/3} T^{2/3})$  from [Cassel and Koren, 2020]. Note that the ratio by which we improve here can be unbounded, with larger improvements for high-dimensional ( $d_x \gg 1$ ) systems. See Appendix E.2 for further details, comparisons and proofs.
4.1 Derivation of update
In the algorithm, the key component is computing an approximate policy gradient of the cost. A complete theoretical analysis of our algorithm can be found in Appendix E, but we provide a brief sketch of the gradient calculation. Let  $J_{t}(M)$  denote the expected counterfactual cost  $c_{t}$  of following policy  $M$  with the same observed disturbances  $w_{t}$ . We first note that if the dynamics are suitably stabilized (which should be done by  $\pi_{\mathrm{base}}$ ), the state and cost can be approximated as a function  $C$  of a small window of previous controls.
$$
J _ {t} (M) = \mathbb {E} _ {\mathbf {n} _ {1: t}} [ c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) ] \approx \mathbb {E} _ {\mathbf {n} _ {t - h: t}} [ C (\mathbf {u} _ {t} (M) + \mathbf {n} _ {t}, \dots , \mathbf {u} _ {t - h} (M) + \mathbf {n} _ {t - h}) ],
$$
where we use  $u_{t - i}(M)$  as a shorthand for  $\pi (\hat{\mathbf{w}}_{t - i - 1:t - h - i}|M)$ . The expression here is that of a Gaussian smoothed function, which allows us to get the following unbiased single point gradient estimate
$$
\nabla_ {\mathbf {u} _ {i}} \mathbb {E} _ {\mathbf {n} _ {t - h: t}} [ C (\mathbf {u} _ {t} + \mathbf {n} _ {t}, \dots , \mathbf {u} _ {t - h} + \mathbf {n} _ {t - h}) ] = \mathbb {E} _ {\mathbf {n} _ {t - h: t}} [ \Sigma^ {- 1} C (\mathbf {u} _ {t} + \mathbf {n} _ {t}, \dots , \mathbf {u} _ {t - h} + \mathbf {n} _ {t - h}) \mathbf {n} _ {i} ].
$$
We use a single sample to get a stochastic gradient. Using the chain rule, which involves an outer product due to the tensor structure of  $M$ , we get stochastic gradients with respect to  $M$  as follows
$$
\widehat {\nabla_ {M}} J _ {t} (M) \approx C (\mathbf {u} _ {t} (M) + \mathbf {n} _ {t}, \dots , \mathbf {u} _ {t - h} (M) + \mathbf {n} _ {t - h}) \Sigma^ {- 1} \sum_ {i = 0} ^ {h - 1} \mathbf {n} _ {t - i} \otimes \frac {\partial \pi (\hat {\mathbf {w}} _ {t - i - 1 : t - h - i} | M)}{\partial M}.
$$
Finally, we note that  $M^t$  is slowly moving because of gradient descent, so we can approximate
$$
c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \approx C \left(\mathbf {u} _ {t} \left(M ^ {t}\right) + \mathbf {n} _ {t}, \dots , \mathbf {u} _ {t - h} \left(M ^ {t}\right) + \mathbf {n} _ {t - h}\right).
$$
Putting everything together, we have
$$
\left. \widehat {\nabla_ {M}} J _ {t} (M) \right| _ {M = M ^ {t}} \approx c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \Sigma^ {- 1} \sum_ {i = 0} ^ {h - 1} \mathbf {n} _ {t - i} \otimes \frac {\partial \pi \left(\tilde {\mathbf {w}} _ {t - i - 1 : t - h - i} | M\right)}{\partial M} \Bigg | _ {M = M ^ {t}}. \tag {6}
$$
Figure 1: Episode return for best performing MF-GPC model versus best performing baseline DDPG model for various OpenAI Gym environments and pseudo-estimation methods. Environment and pseudo-estimation method shown in title. Results averaged over 25 seeds. Shaded areas represent confidence intervals. We find that PD2 and PD3 perform well in these settings.
We apply the MF-GPC Algorithm 1 to various OpenAI Gym [Brockman et al., 2016] environments. We conduct our experiments in the research-first modular framework Acme [Hoffman et al., 2020]. We pick  $h = 5$  and use the DDPG algorithm [Lillicrap et al., 2016] as our underlying baseline. We update the  $M$  matrices every 3 episodes instead of continuously to reduce runtime. We also apply weight decay to line 6 of Algorithm 1. Our implementation of PD1 is based on Equation 2. PD2 can be implemented with any vector of rewards. We choose linear function  $L$  given in Lemma 2 to be the identity function. Hence  $\mathbf{c}$  in Equation 3 reduces to the state  $x_{t}$  itself. We pick  $\mathbf{V}$  and  $\mathbf{Q}$  network architectures to be the first  $d_{x}$  units of the last layer of the critic network architecture. We train for 1e7 steps as a default (this is also the default in the Acme code) and if performance has not converged we extend to 1.5e7 steps. Because the  $M$  matrices impact the exploration of the algorithm, we tune the exploration parameter  $\sigma$  for both DDPG and MF-GPC. For the baseline DDPG, we typically explore  $\sigma \in \{0.15, 0.2, 0.25\}$ . More experimental details may be found in Appendix Section B.
Results for Noisy Hopper, Walker 2D, and Ant We create a noisy Hopper, Walker 2D, and Ant environments by adding a Uniform random variable  $U[-0.1, 0.1]$  to the state. The noise is added at every step for both the DDPG baseline and our MF-GPC. We plot the results for PD2, and PD3 in Figure 1. We find that PD2 and PD3 perform relatively well in these settings. Graphs depicting all runs for different  $\sigma$  are available in Appendix Section B. MF-GPC is not guaranteed to improve performance in realistic RL settings. We find that generally PD1 does not perform well e.g. in Figure 2 a) and some examples where applying it yields performance similar to baseline are given in Appendix Section B. This is likely due to the high variance of the PD estimate. We find that neither our method nor the baseline is too sensitive to our hyper-parameter tuning (Figure 2 b)), possibly because we start with the default Acme parameters which are already well tuned for the noiseless environment.
Linear Dynamical Systems We evaluate our methods on both low dimensional  $(d_x = 2, d_u = 1)$  and a higher dimensional  $(d_x = 10, d_u = 5)$  linear systems with sinusoidal disturbances to demonstrate the improvements in dimension of our method (labeled RBPC) over BPC [Gradu et al., 2020]. We use the full information GPC [Agarwal et al., 2019] and LQR as baselines using implementations from Gradu et al. [2021]. While performance is comparable to BPC on the small system, on the larger system, BPC could not be tuned to learn while RBPC improves upon the LQR
Figure 2: Left: Episode return for PD1 for Noisy Hopper. We find that PD1 is not effective for RL settings. Right: Hyper-parameter search for PD3 on Noisy Walker. We find that neither Meta-GPC nor the baseline DDPG algorithm is too sensitive to tuning.
Figure 3: Comparison on low dimensional (left) vs high dimensional (rights) LDS.
baseline (see Figure 3). In both experiments,  $h = 5$  and the learning rate and exploration noise is tuned.
We have described a new approach for model-free RL based on recent exciting advancements in model based online control. Instead of using state-based policies, online nonstochastic control proposes the use of disturbance-based policies. To create a disturbance signal without a model, we define three possible signals, called Pseudo-Disturbances, each with its own merits and limitations. We give a generic (adaptable) REINFORCE-based method using the PD signals with provable guarantees: if the underlying MDP is a linear dynamical system, we recover and improve the strong guarantees of online nonstochastic control. Preliminary promising experimental results are discussed. We believe this is a first step in the exciting direction of applying tried-and-tested model-based control techniques for general reinforcement learning.
Yasin Abbasi Yadjkori, Peter L Bartlett, Varun Kanade, Yevgeny Seldin, and Csaba Szepesvári. Online learning in markov decision processes with adversarially chosen transition probability distributions. Advances in neural information processing systems, 26, 2013.
Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning, pages 111-119. PMLR, 2019.
Naman Agarwal, Elad Hazan, Anirudha Majumdar, and Karan Singh. A regret minimization approach to iterative learning control. In International Conference on Machine Learning, pages 100-109. PMLR, 2021.
Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory, pages 3320-3436. PMLR, 2020.
Y. Jennifer Sun, Stephen Newman, and Elad Hazan. Optimal rates for bandit nonstochastic control. arXiv preprint arXiv:2305.15352, 2023.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Aviv Tamar, Huan Xu, and Shie Mannor. Scaling up robust mdps by reinforcement learning. arXiv preprint arXiv:1306.6189, 2013.
Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215-6224. PMLR, 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint arXiv:2008.01825, 2020.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057, 2019.
Huan Zhang, Hongge Chen, Duane Boning, and Cho-Jui Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. arXiv preprint arXiv:2101.08452, 2021.
Kemin Zhou and John Comstock Doyle. Essentials of robust control, volume 104. Prentice hall Upper Saddle River, NJ, 1998.
1.1 Our Contributions 2
1.2 Pseudo-Disturbance based RL 2
1.3 Related Work 3
1.4 Paper Outline 4
2 Setting and Preliminaries 4
3 Pseudo-Disturbance Signals and Policies 5
3.1 Pseudo-Disturbance Class I: Value-Function Gradients 5
3.2 Pseudo-Disturbance Class II: Vector Value Functions 6
3.3 Pseudo-Disturbance Class III: Simulator Based 6
3.4 Merits of different Pseudo-Disturbance signals 7
4 Meta Algorithm and Main Theorem 7
4.1 Derivation of update 8
5 Experiments 9
A Notation 15
B Experiments 15
B.1 Hopper 15
B.2 Noisy Hopper 16
B.3 Noisy Walker 2D and Ant. 16
B.4 Experiments with adversarial noise 17
C Discrete State and Action Spaces 18
D Pseudo-Disturbance Proofs 18
D.1 Proof of Lemma 1 18
D.2 Proof of Lemma 2 19
D.3 Proof of Lemma 3 20
E Main Result and Dimension-efficient Bandit GPC 20
E.1 Main Result 20
E.2 Dimension-Efficient Bandit GPC 20
E.3 Idealized Cost and Proof of Theorem 9 22
E.4 Proof of Supporting Claims 24
We use the following notation consistently throughout the paper:
Symbol	Semantics
⊗	outer product
f	dynamics/transition function
fsim	simulator dynamics/transition function
dx	state dimension
du	control dimension
dw	pseudo-disturbance
dmin	min(dx,du)
xt∈Rdx	state at time t
ut∈Rdu	control at time t
wt∈Rdw	perturbation (disturbance) at time t
ct	instantaneous cost at time t
hatw∈Rdw	pseudo-disturbance at time t
nt∈Rdu	Gaussian exploration noise at time t
A,B,C	system matrices for linear dynamical system
h	history length (i.e., number of parameters) in a policy class
M1:h	h-length sequence of matrices used by MF-GPC at time t
M	policy class of h-length matrices
γ	discount factor
Vπ, Qπ	state and state-action value functions for π
Vrπ, Qrπ	vectorized value and Q functions for π for reward vectors r(x)
yt	idealized state
Ct	stationary idealized cost (function of single M) at time t
Ct	non-stationary idealized cost (function of memory) at time t
Ct,δ	Smoothed Ct using noised controls
Ft	idealized cost as a function of last controls at time t
Ft,δ	smoothed Ft
||·||	spectral norm
||·||F	Frobenius norm
We test the performance of our method on various OpenAI Gym environments. We conduct our experiments in the research-first modular framework Acme [Hoffman et al., 2020]. We pick  $h = 5$  and use the DDPG algorithm [Lillicrap et al., 2016] as our underlying baseline. We update the  $M$  matrices every 3 episodes instead of continuously to reduce runtime. We also apply weight decay to line 6 of Algorithm 1.
Our implementation is based on the Acme implementation of D4PG. The policy and critic networks both have the default sizes of  $256 \times 256 \times 256$ . We use the Acme default number of atoms as 51 for the network. We run in the distributed setting with 4 agents. The underlying learning rate of the D4PG implementation is left at  $3e - 04$ . The exploration parameter,  $\sigma$  is tuned.
Plotting We use a domain invariant exponential smoother with a small smoothing parameter of 0.1. The smoothing is applied before the mean is taken over the data. To construct the confidence intervals, we take the following steps 1) smooth the data 2) linearly interpolate each run of the data to produce a fine grid of values 3) calculate  $\sigma / \sqrt{N}$  on interpolated data.
B.1 Hopper
The OpenAI Gym Hopper environment is a two-dimensional one legged figure that consists of four body parts, namely a torso, thigh, leg and foot.
(g)
Figure 4: Raw data for PD Estimate 1 on Hopper. Each plot represents either the baseline or our method for a different setting of the exploration parameter,  $\sigma$ . We find that there are some outlier runs (for example the horizontal line in subfig f). We remove the two lowest return runs from each group before plotting.
B.2 Noisy Hopper
We create a Noisy Hopper environment by adding a Uniform random variable  $U[-0.1, 0.1]$  to qpos during training and evaluation. The noise is added at every step, and both DDPG and MF-GPC are evaluated on the same noisy environment.
PD1 We implement PD1 for Noisy Hopper according to Equation 2. We see our results in Figure 6 in the first column. We tune  $\sigma$  for the D4PG baseline in the set  $\{0.1, 0.15, 0.2, 0.25, 0.3\}$ . We find that the  $\sigma = 0.1$  performs the best. We tune  $\sigma$  for MF-GPC in the set  $\{0.1, 0.15, 0.2\}$ . We find that the  $\sigma = 0.1$  setting performs the best. We averaged our results over 25 seeds. Solid line represents the mean return over all runs and shaded areas represent standard error. We find that MF-GPC run with PD-1 has a small advantage compared to the DDPG baseline on Noisy Hopper. Removing outliers for PD1 for this specific experiment, we notice that some runs seem to be outliers. Therefore, when plotting, we remove the lowest 2 runs across all groups (of both the baseline and our method). Complete raw data (with outliers included) can be seen in Figure 4.
PD-2 PD2 can be implemented with any vector of rewards. We choose linear function  $L$  given in Lemma 2 to be the identity function. Hence  $\mathbf{c}$  in Equation 3 reduces to the state  $x_{t}$  itself. We pick  $\mathbf{V}$  and  $\mathbf{Q}$  to be the first  $d_{x}$  units of the last layer of the critic network. If  $d_{x}$  is larger than the number of atoms of the critic network (51) we take all 51 nodes from the critic network. We find that a default  $\sigma = 0.15$  performs well for MF-GPC so we do not tune  $\sigma$  further.
PD-3 In practice the expectation in Equation 4 requires estimation. We use an average over 4 copies of the environment for this estimation.
B.3 Noisy Walker 2D and Ant
We follow the basic procedure for Hopper but train for 15 million steps instead of 10 million. We report our results for the hyper-parameter sweeps in columns 2 and 3 of Figure 5. We find that PD2 and PD3 perform relatively well in these settings.
PD-2 PD2 can be implemented with any vector of rewards. We choose linear function  $L$  given in Lemma 2 to be the identity function. Hence  $\mathbf{c}$  in Equation 3 reduces to the state  $x_{t}$  itself. Recall that  $d_{x}$  is the dimension of the state space. We pick  $\mathbf{V}$  and  $\mathbf{Q}$  to be the first  $d_{x}$  units of the last layer of the critic network. If  $d_{x}$  is larger than the number of atoms of the critic network (51) we take all 51 nodes from the critic network.
Figure 5: Episode return for best performing MF-GPC model versus best performing baseline DDPG model for various OpenAI Gym environments and pseudo-estimation methods. Environment and pseudo-estimation method shown in title. Results averaged over 25 seeds. Shaded areas represent confidence intervals. We find that PD2 and PD3 perform well in these settings.
Figure 6: Left: Episode return for PD1 for Noisy Hopper. Right: Episode return for PD1 for Noisy Half Cheetah. We find that PD1 is not effective for RL settings.