Exploring the Robustness of Distributional Reinforcement Learning against Noisy State Observations
Authors: Ke Sun
Yi Liu
Yingnan Zhao
Hengshuai Yao
SHANGLING JUI
Linglong Kong
Venue: ICLR 2022 Submitted (2022)
Keywords: distributional reinforcement learning
robustness

Abstract: In real scenarios, state observations that an agent observes may contain measurement errors or adversarial noises, misleading the agent to take suboptimal actions or even collapse while training. In this paper, we study the training robustness of distributional Reinforcement Learning~(RL), a class of state-of-the-art methods that estimate the whole distribution, as opposed to only the expectation, of the total return. Firstly, we propose State-Noisy Markov Decision Process~(SN-MDP) in the tabular case to incorporate both random and adversarial state observation noises, in which the contraction of both expectation-based and distributional Bellman operators is derived. Beyond SN-MDP with the function approximation, we theoretically characterize the bounded gradient norm of histogram-based distributional loss, accounting for the better training robustness of distribution RL. We also provide stricter convergence conditions of the Temporal-Difference~(TD) learning under more flexible state noises, as well as the sensitivity analysis by the leverage of influence function. Finally, extensive experiments on the suite of games show that distributional RL enjoys better training robustness compared with its expectation-based counterpart across various state observation noises.

Under review as a conference paper at ICLR 2022
EXPLORING THE ROBUSTNESS OF DISTRIBUTIONAL REINFORCEMENT LEARNING AGAINST NOISY STATE OBSERVATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
In real scenarios, state observations that an agent observes may contain measure- ment errors or adversarial noises, misleading the agent take suboptimal actions or even collapse while training. In this paper, we study the training robustness of distributional Reinforcement Learning (RL), a class of state-of-the-art meth- ods that estimate the whole distribution, opposed to only the expectation, of the total return. Firstly, we propose State-Noisy Markov Decision Process (SN- MDP) in the tabular case to incorporate both random and adversarial state ob- servation noises, in which the contraction of both expectation-base tional Bellman operators is derived. Beyond SN-MDP with the function approx- imation, we firstly analyze the vulnerability of least squared loss in expectation- based RI., and by contrast theoretically characterize the bounded gradient of histogram- based distributional loss, accounting for the better training robust- ness of distribution RL. Finally, extensive experiments on the suite of games show that in SN-MDP-like setting both expectation- based and distributional RL can converge albeit corresponding different levels under various state observation noises. However, distributional RL can enjoys better training robustness in the more complicated noisy state observation settings compared with its expectation- based counterpart.
and distribu-
INTRODUCTION
Learning robust and high-performance policies for continuous state-action reinforcement learn- ing (RL) domains is crucial to enable the successful adoption of deep RL in roboties, autonomy, and control problems. However, recent works have demonstrated that deep RL algorithms are vul-
nerable either to model uncertainties or external disturbances (Huang et al. 2017 Pattanaik et al.
2017 Ilahi et al. 2020 Chen al. 2019 Zhang et al.) 2020 Shen al. [2200] Singh et al. [0200] Guan et al. 2020). Particularly, model uncertainties normally occur in a noisy reinforcement learn- ing environment where the agent often encounters systematic stochastic measurement errors state observations, such as the inexact locations and velocity obtained from the equipped sensors of a robot. On the other hand, external disturbances are normally adversarial in nature. For instance, the adversary can construct adversarial perturbations on state observations to degrade the performance of deep RL algorithms. These two factors lead noisy state observations that influence the perfor- mance of algorithms, precluding the success of reinforcement learning in real-world applications.
Existing works mainly focus on improving the robustness of algorithms in the test environment with noisy state observations. Smooth Regularized Reinforcement Learning (Shen et al. 2020) intro- duced a regularization to enforce smoothness in the learned policy, and thus improved robustness against measurement errors in the test environment. Similarly, the State-Adversarty Markov de- cision process (SA-MDP) (Zhang et al. 2020) was proposed and the resulting principled policy regularization enhances the adversarial robustness of various kinds of RL algorithms against adver- sarial noisy state observations. However, both of these works assumed that the agent can access clean state observations during training, which is normally not feasible when the environment is in- herently noisy, such as unavoidable measurement errors. Thus, the maintenance and formal analysis of policies robust to noisy state observations during training is worthwhile area of research.

---

Under review as a conference paper at ICLR 2022
On the other hand, recent distributional reinforcement learning algorithms, including C51 Belle-- Reggesssonn DQN (QRDQN) (Dabney et al. 2018b), Implicit Quantile Networks (Dabney et al., 2018a) and Moment-Matching DQN (MMD) (guyee al. 2020). con- stantly set new records in Atari games, gaining huge attention in the research community. However, existing literature mainly focuses on the performance of algorithms, other benefits, including the robustness in the noisy environment, of distributional RL algorithms are less studied. As distribu- tional RL can leverage additional information about distribution that captures the uncertainty of the environment more accurately, it is natural to expect that distributional RL with this better represen- tation capability can be less vulnerable to the noisy environment while training, which motivates our research.
mare et al. 2017).
In this paper, we investigate the robustness of distributional RI. against various kinds of state ob- servation noises encountered during training. Firstly, we propose general State-Noisy MDP in the tabular setting. in which we prove the convergence of distributional Bellman operator. We further extend SN-MDP to the function approximation case by considering more complex noisy state ob- servations. Notably, we analyze the vulnerability of classical RL and in contrast characterize the Lipschitz continuity blessing resulting from the Histogram distributional loss in distributional RL, which leads to bounded gradient norm. This better behaved gradient mitigates the impact of noisy states the objective function, accounting for the less vulnerability of distributional RL while training. Finally, extensive experiments demonstrate that both expectation-basee and distributional RL algorithms can converge in SN-MDP-like settings. More importantly, distributional RL algo- rithms tend to achieve better robust performance in the presence of more complex state observation noises compared with its expectation-basel counterpart that may even diverge some cases. These empirical results in Section echo our previous theoretical results in both Section 3] 4 Over- all, the training robustness advantage of distributional RL algorithms we revealed facilitates their deployment especially in the noisy environment.
2
BACKGROUND
2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING
In the tabular setting without noisy states, the agent's interaction with its environment can be nat- urally modeled as a standard Markov Decision Process (MDP), a S-tuple (S, A, R, P) S and A are the state and action spaces, P S A
[0, is the environment transition dynamics, R: : S x A x S + R is the reward function and 7 E (0, 1) is the discount factor.
Value Function vs Value Distribution. Firstly, we denote the return where s; = s as (7(s) 00
Ek=o r representing the cumulative rewards following a policy т, and +++ is reward scalar obtained the step t + k + In the algorithm design, traditional expectation- based RL normally focuses on value function V (s), the expectation of the random variable 2 (s)::
(*()): - 7 (8) = E
Σ y Tr+tk+ 8 k=
In contrast, in the distributional RL setting, we focus the value distribution, the full distribution of z"(s), (s), and the
value distribution Z (s, in the control problem where s = s, a, = a. Both of these distributions can better capture the uncertainty of returns in the MDP beyond just its expectation (Dabney et al. 2018a; Mavrin et al. 2019).
Distributional Bellman Operator. In expectation-based RL, update the value function via the Bellman operator T", while in distributional RL, the updating is on the value distribution via the distributional Bellman operator I". To derive I", we firstly define the transition operator p= : 2
Z= z A'), ~ p() ~ ~ (-IS").
where we use capital letters S" and A' to emphasize the random nature of both, and .2 indicates convergence in distribution. For simplicity, we denote 2 a) by Z(s Thus, the distributional Bellman operator I" is defined as:
Z², 2 R(s, >P=
(3)
2

---

Under review as a conference paper at ICLR 2022
More importantly, I" is still contraction for policy evaluation under the maximal form of the Wasserstein metric dp over the true and parametric value distributions (Bellemare et al. 2017 Dab ney et al., 2018b), where the p- Wasserstein metric d, is defined as
1/p
- Fz dp = ( -F) (w) du
(4)
which minimizes the distance between the true value distribution Z* and the parametric distribution Zo. F-1 is the inverse cumulative distribution function of random variable with the cumulative distribution function as In the control setting, the distributional analogue of the Bellman opti-
mality operator converges to the set of optimal value distributions, although it is in a weak sense and requires more involved arguments (Dabney et al. 2018b).
2.2 TWo KINDS OF NOISY STATE OBSERVATIONS
We investigate both random and adversarial training robustness, i.e., the performance of RL algo- rithms under these two types of noisy state observations, between the expectation based and distri- butional RL algorithms. We consider continuous state observations with continuous noises. In the random noisy state case, we apply Gaussian noises with mean and different standard deviations to state features simulate the measurement error stemming from various sources.
In the adversarial state perturbation setting, we construct white-box adversarial perturbations on state observations for the current policy during training, following the strategy proposed in (Huang al., 2017 Pattanaík et al. 2017) that leveraged the gradient information of an engineered loss function. In particular,
denote a as the "worst" action, with the lowest probability from the current policy Te me(als) is) in the training step t. Thus, the optimal adversarial perturbation T)s constrained in an c-ball, can be derived by minimizing the objective function J:
min [st + 71, TT) Σ pf log (aslo + n), s.t||1nll S
(5)
7
where p = 1 if corresponds the index of the least-chosen action, i.e. the w-th index in the vector a, otherwise of = In other words, we construct target one-hot action pf with assigned the index of the least-chosen action. Through this minimization in the form of the cross entropy loss, we can construct the state perturbations 7/ that can force the policy to choose the least-chosen action at in each step.
TABULAR CASE: STATE-NOISY MARKOV DECISION PROCESS
In this section, we extend State-Adversarial MDP (Zhang et al. 2020) to more general State- Noisy Markov Decision Process (SN-MDP), and particularly provide proof of the convergence and contraction of distributional Bellman operator in this setting.
3.1 DEFINITIONS
As shown in Figure L SN-MDP is a 6-tuple (8, A, R, P. N), where the noise generating mecha- nism N(-ss) maps the state from v(s) using either random adversarial noise with the Marko-
(3;)
Agent
(s:))
Noise
N (s;)
a q<T( v(s))
Tc+1
S
Environment
St+1
Figure 1: State-Noisy Markov Decision Process where u(s.) perturbed by the noise mechanism N.
3

---

Under review as a conference paper at ICLR 2022
vian and stationary probability (((())ss). It is worthwhile note that the explicit definition of the noise mechanism N here is based on discrete state transitions, but the analysis can be naturally extended to the continuous case if we let the state space go to infinity. Moreover, let (s)) be the set
that contains the allowed noise space for the noise generating mechanism N, i.e., v(8) E (s)).
Following the setting in (Zhang al. 2020), only manipulate state observations but do not change the underlying environment transition dynamics based on s or the agent's 's actions directly. As such, our SN-MDP is more suitable model the random measurement error, e.g., sensor errors and equipment inaccuracies, and adversarial state observation perturbations in safety-critical scenarios.
3.2 ANALYSIS OF SN-MDP FOR EXPECTATION-BASED RL
We define the value function VrON given π in SN-MDP. The Bellman Equations are given by:
(sON(s)= = ΣE (((s))))7(((((s)) Ep(dlo,a)> [ 5') + + + m(()
(6)
The random noise transits s into v(s) with a certain probability and the adversarial noise is the special case of N (vv())ss where N(v"
((v*(s)(s) (s)[s) 1 of (s) is the optimal adversarial noisy state given 8, and N [(v())]s) = O otherwise. We denote Bellman operators under random noise mechanism (-ss)) and adversarial noise mechanism (-Is) as T," and TH respectively. This implies that T, ToN V= and Th" TeN = V- We extend Theorem in (Zhang et al. 2020) both random and adversarial noise scenario, and immediately obtain that both 7," and 7,"" are contraction operators SN-MDP. We explain this in Theorem of Appendix AA
The pivotal conclusion from Theorem 3 is VT.VTUX miny ViON- This implies that the adversary attempts to minimize the value function, forcing the agent to select the worse-case action among the allowed transition probability space (-Iss) for each state s. The main proof idea that Bellman updates in SN-MDP result in the convergence to the value function for another "merged" policy x' where N' (a]s) =
The value function for the merged policy might far away from that for the original policy x, which tends to worsen the performance of RL algorithms.
Ev(e) (((())]))
3.3 ANALYSIS OF SN-MDP IN DISTRIBUTIONAL RL
In the SN-MDP setting for distributional RL, the new distributional Bellman equations use new transition operators in place of pp in Eq. 2 The new transition operators P and P for the random and adversarial settings, are defined as:
ZN, _ ZN- (S', AA), +(IV((S)) and P. (s _ ZN- (S', A)). ~ (S))
where V(S') ~ N²((1S" is the state random variable after the transition, and V*(S') is attained from (-(S1)") under the optimal adversary. Besides, S ~ a() Thus, the corresponding new distributional Bellman operators I. and 5% are:
T: (x, D R(s,a, Zx, and ZN _ +
In this sense, four sources of randomness define the new compound distribution in the SN-MDP: (1) randomness of reward, (2) randomness in the new environment transition dynamics PMT or P that additionally includes (3) the stochasticity of the noisy transition N, and (4) the random next-state value distribution 7(S", A'). Besides, the premise of the robustness of distributional RI. against noisy state observations lies in the convergence of the new derived distribution Bellman Operators in SN-MDP setting. We proved this convergence and contraction f'rr policy evaluation in Theorem] I Theorem 1. (Convergence and Contraction of Distributional Bellman operators in the SN-MDP)
Given a policy n, define the distributional Bellman operators ST and S in Eq. 8 and consider the Wasserstein metric dp. the following results hold.
(1) S is contraction under the maximal form of dg-
(2) I is also a contraction under the maximal form of dp. following the greedy adversarial rule, Le.. No (+(s') arg min mijee E. (s'',,, , d')] where a' (:V((s)) and V(5)) N(-1s).
4

---

Under review as a conference paper at ICLR 2022
We provide the proof in Appendix The convergence of distributional Bellman operators in the SN- MDP is one of our main contributions. This result allows us to deploy distributional reinforeement learning algorithmss comfortably even in settings with noisy state observations.
4 FUNCTION APPROXIMATION CASE: NOISY SETTINGS BEYOND SN-MDP
In real scenarios, especially safety-critical cases, perturbations on state observations can be more complicated. For instance, the adversary might perform attacks at certain intervals, yielding un- balanced state observation pairs with perturbed current state and a benign next state and vice versa. This type of unbalanced perturbations is outside the scope of State-Noisy MDP we ana- lyzed in the last section and can have different impacts on the convergence of expectation-base based and distributional RL algorithms. In this section, we firstly point out the vulnerability nature of expectation-basee RL although we consider the bounded rewards in Section 4.1 Next, we char- acterize the robustness blessing of distributional RL based on Histogram distributional loss (Imani & White 2018) with derived bounded gradient norms unlike the unrestricted one in expectation- based RL. Finally, as an extension we analyze the impact more complex state observations on TD convergence and further conduct a sensitivity analysis by the influence function.
Notation, We derive theoretical results with the linear function approximator. For the expectation- based RL, the value estimate S x Rdd - R is formed simply the inner product between state features x(5) and weights w E Rd, given by 0(s, w) def w x(s). At each step, the state feature can be rewritten as X
def x (St) E RRd The distributional RL setting is given in Section 4.2
4.1 VULNERABILITY FOR EXPECTATION -AASETTTOON-BEEDD RL
In the elassical RL with function approximation, a natural objective function is Mean Squared Value Error Sutton & Barto 2018) denoted VE, which is weighted by the state distribution μc:
VE (w) Σ m(s) (s) 0(s, w)]²
SES
where uls) is the fraction of time spent in s. To solve this kind of weighted least squared minimiza- tion, we leverage Stochastic Gradient Descent (SGD) and the weight updating rule can be formulated as
Wi+1 Wt + a [ta (st) - 0 (se, we)] V0 (s),
(10)
= + a (Uz w. xt) Xx
where a is the step size and the target Ug can be either unbiased estimate via Monte Carlo method
with Ue = Ex=o 7 r* or a biased estimate via TD learning with U, = T++1 + yw, X Let Ivr be the empirical version of VE(w). To investigate the vulnerability of expectation-based RL over state features, we focus on the gradient norm of gyg regarding x(s) (or xt). For a fair comparison with distributional RL in Section 4.2 we additionally bound the norm of w, i.e. |w|| < I, which can also be easily satisfied by imposing & or &g regularization. Therefore, the upper bound of gradient norm of AVE is
IVEll = 10/- - X well < U0: - w, x2|L.
(11)
(x(s)
However, this upper bound IU, w, xell ll can be arbitrary large as there is no restriction on IU, w, xel in either Monte Carlo or TD bootstrap estimate of U., which we elaborate as follows.
Case 1: Estimate U, via Monte Carlo. If we consider the bounded rewards, i.e., r E [Rmino Rmax) we can bound U, as Ue- L 7" E [i [- However, the bounded rewards can not guarantee the bounded gradient norm of IVE as w, X4 can be sufficiently large.
Run
Case 2: Estimate U, via bootstrap estimate, e.g., TD (0). Consider TD (0) where U, T++1 + Yw, X1+1- However, we can find that U0= w, x, = |r441 + w. (7x441 - x:) can still be sufficiently large due to the unrestricted term w, (7X44 x,).
Therefore, we conclude that the unbounded gradient norm of objective function regarding state features for the expectation-based RL results in its vulnerability against state perturbations.
5

---

Under review as a conference paper at ICLR 2022
4.2 ROBUSTNESS BLESSING FOR DISTRIBUTIONAL RL
We show that in the function approximation setting, the distributional loss in distributional RL can additionally yield Lipschitz continuity regarding state features, thus leading to more stable gradients relatively to expectation-based RI.
In particular, in distributional RL our goal is to minimize C (Zg, IZe), where I is the distribu- tional Bellman operator. Here we leverage histogram to parameterize the distribution Ze based on KL divergence as L. yielding the histogram distributional loss (Imani & White 2018). Specifi- cally, we uniformly partition the support of x(s) into k bins, and let function f X - [0, 1
provide k-dimensional vector f(x(s)) of the coefficients indicating the probability the target is in that bin given (().. Next, we use softmax based on the linear approximation x(s) 0; to express f. i... fi(())) exp (x(s) 0.) شز= exp (x(s)T0)) Therefore, the histogram distributional loss (Z,, IZ)) between Zg and Zo can be derived as
C8 C==
Σ p: log f (x(s))
(12)
where 0 = ..., 0x and the target probability is the cumulative probability increment of target distribution 52g within the i-th bin. Detailed derivation of the histogram distributional loss is given in Appendix [
Based on this histogram distributional loss in distribution RL, we obtain Theorem 2 (proof in Ap- pendix C). revealing that the distribution loss can result in additional Lipschitz continuity property that bounds the norm of gradient over state features x()):
Theorem 2. (Lipschitz Continuity of distributional RL) Consider the histogram distributional loss
La - - Pt log J (x()), where fx(())) exp (x(s) 0.) / exp (x(s) 02) parame- terized by 0 {f...... Ox Assume ||0,]| s for Vi 1,..,k, k, then Lo is kl-Lipschitz continuous w.E.t x(s). yielding a bounded norm of its gradient, i.e..
2 Ox((s
Lo
S kl
(13)
In conclusion, Theorem shows that distributional loss in distributional RL can additionally en- joy kl-Lipschitz continuity compared with the expectation-basel RL where the gradient norm of objective function unbounded. Thus, the bounded gradient norm regarding state features of distri- butional RL mitigates the impact of state noises on the objective function while training, therefore yielding better training robustness.
Extension of TD Convergence and Sensitivity Analysis. Beyond the aforementioned robustness analysis on both expectation-basee and distributional RL, we also conduct the analysis on the per- turbation impacts of different state observations in the TD learning based expectation including the current, next and both state observations. In particular, conditions for TD convergence in these three different noisy state observation settings are derived in Theorem 4 of Appendix [D] which are strongly related to the positive definiteness property of relevant matrixs. More impor- tantly, the convergence conditions for current and next noisy state observations are harder than the condition in the noise-free setting and are normally divergent. Nevertheless. which situation is milder heavily depends on the task. Interested readers can refer Appendix for more details. In addition, we also conduct sensitivity analysis by the influence function to characterize the effects that the state noise in particular observation has on an estimator conclusions we made are similar to the TD convergence analysis part. Note that although the exten- sion analysis is based on the expection- based RI. with linear function approximation, our empirical analysis results across both expecattionbasee and distributional RL. We defer detailed empirical verification in the following experimental section.
RL,
TheoremBJot Appendix (E) The
observations demonstrate
Remark. In this section, we firstly present the vulnerability of expectation-based RL against noisy state observations in Section 4.1 while we further derived the Lipschitz continuity blessing of distributional RL in Section 4.2, leading to more robustness against noisy state observations than expectation- RL. In Section 55] we show that our empirical observations coincide with our theoretical results to further demonstrate the robustness advantage of distributional RI.

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
We make a comparison between expectation-based and distributional RL algorithms against various noisy state observations. We select DQN (Mnih al. 2015) the baseline, and QR-DQN (Dabney et al. 2018b) as its distributional counterpart. The previous analysis is either for policy evaluation or linear function approximation, but there are natural-though in some cases deuristic-extensiii to the control setting and to non-linear function approximation.
Experimental Setup. We perform our algorithms on Cart Pole, Mountain Car, Breakout and Qbert games, We followed the procedure in (Ghiassian et al. 2020; Zhang & Yao 2019). All the experi- mental settings. including parameters, are identical to the distributional RI. baselines implemented by (Zhang, 2018 Dabney et al. 2018b). We perform 200 runs on both Cart Pole and Mountain Car and runs on Breakout and Qbert. Reported Results are averaged with shading indicating the standard error. The learning curve is smoothed over window of size 10 before averaging across runs. Please refer to Appendix for for more details.
Noisy State Observations. For the random noise, we use Gaussian noise with different standard deviations. In particular, for better presentation to compare the difference, we select the standard deviations as 0.05, 0.1 in Cart Pole, 0.01, 0.0125 in Mountain Car, 0.01, 0.05 in Breakout and 0.05 in Obert. For the adversarial noise, we followed (Zhang et al. 2020), where the set of noises B(s) is defined as an &o norm ball around with radius c, given by L Bs) == {8 ||s $ a1 S c). We apply Projected Gradient Descent (PGD) version in (Pattanaik et al., 2017). with 3 fixed iterations while adjusting e control the perturbation strength. Concretely, for better presentation we select the perturbation size as 0.05, 0.1 in Cart Pole, 0.01, 0.1 in Mountain Car, 0.005, 0.01 in Breakout, and 0.005 in Qbert.
we consider the function approximation setting with TD learning where both current and next state observations are perturbed, leading SN-MDP-like setting as the vanilla SN-MDP is a tabular case. In this case, we found both DQN and QRDQN con- verge different convergence points determined by the noise strength. Next, in Section(5.7jam we investigate the robustness advantage of distributional RL, i.e., QRDQN, relative to expectation- based RL, i.e., DQN, under various random and adversarial state observations, respectively.
Experiments Structure. In Section 5.1
5.1 SN-MDP-LIKE SETTING
In the classical SN-MDP tabular case, each state will be perturbed via the noise mechanism N. We keep this noise structure and transform it into the function approximation setting with TD learning,
Adversarial
Adversarial
182.8
Mountain Return
30g Sfici
Cart Pole: Return
16.22
200
F2P B
1094
E6,3
1000
Adversarial
Adversarial
700
1000
506
Breakout Return
Qtarr:: Return
8650.0
ABR 350
6d89
4576.6
891
8000
15.9
12.0
Figure 2: Average returns of DQN and QRDQN against adversarial state observation noises across four games. advX in the legend indicates adversarial state observations with the perturbation size c as X. Both QRDQN (solid lines) and DQN (dashed lines) converge and the convergence level is determined by the perturbation strength
7

---

Under review as a conference paper at ICLR 2022
and thus both current and next state observations would be noisy. We further investigate the training robustness of both DQN and QRDQN against either random or adversarial state perturbations across these four games.
We focus on the adversarial setting as suggested in Figure 2 It turns out that all DQN and QRDQN algorithms under various strength of adversarial state noises converge, although they eventually obtained different average returns. This empirical observation is consistent with our theoretical analysis in Section 3 where both classical and distributional Bellman operators are contractive and thus convergent. In addition, Figure [S] also manifests that the final performance that algorithms attained has a decreasing tendency as the perturbation strength, i.c., standard deviation, increases especially for DQN. Remarkably, the final performance of QRDQN is more robust than DQN against different perturbation strength, especially in Breakout and Qhert games, although both DQN and QRDQN are convergent in this SN-MDPPlike setting. For the random state observation setting, we provide the similar results in Figure with detailed explanation in Appendix
5.2 SETTING BEYOND SN-MDP WITH RANDOM STATE NOISES
Beyond SN-MDP, we further investigate the training robustness of DQN and QRDQN in the more complicated unbalanced state observation setting as mentioned in Section 4 where the agent ob- serve perturbations on current state observations. Similar results the setting with the perturbed next state observations given in Appendix P In this subsection, firstly focus the function approximation case with random noises on current state observations, and the learning curves when the agent encounters different strength of random state noises provided in Figure across four games.
Random
Random
200
Mountain Car:: Retur
Cart Pole: Return
172
120
-303
ERSCAL
272. 2
386 8
1:2
1nc
84.99
546.1
776.2
-0
con
-800
792.7
GROOEL
-1803
Sepni 18)
Sepock otf
Random
Random
700
(000]
Braskoutt Return
600
406
BSV andtet
Qbert Return
2003
505, unali
OSSCAL Lndood
-00
OR010 sontiit
@
0000
$488.2
4575.8
+090
1894.5
100
10.9
196.0
12.3
10.0
Figure 3: Average returns of DQN and QRDQN against random state observation noises across four games. randX in the legend indicates random state observations with the standard deviation X. QRDQN (solid lines) almost consistently outperform DQN (dashed lines) given the same perturba- tion strength, yielding more training robustness against noisy state observations.
illustrates that the final performance of QRDQN (solid lines) superior to DQN (DQN) in the same color given the same random state perturbations (the same standard de- viation) when both DQN and QRDQN achieved similar average returns eventually (red dashed and solid lines). This implies that QRDQN enjoys better training robustness than DQN under the ran- dom noisy state observation setting. It is worthy to note that in Mountain Car game although DQN outperforms QRDQN almost across all training process, QRDQN under random state noises (blue and green lines) can surpass DQN with the same noise setting. yielding more training robustness. A more significant result is in Breakout where the training of QRDQN is less vulnerable to vari- ous random state perturbations, while the performance of DQN degrades and even diverges as the perturbation strength increases.
From Figure
8

---

Under review as a conference paper at ICLR 2022
5.3 SETTING BEYOND SN-MDP WITH ADVERSARIAL STATE NOISES
Next, probe the training robust of DQN and QRDQN in the setting where the agent encounters the adversarial state observations in the current state in the function approximation case. Figure 4 presents the learning curves of algorithms over four games against noisy states under different ad- versarial perturbation sizes e.
Adversarial
Adversarial
Mountain Retum
SCB
Pole: Return
172
1:0
272.2
1.3
EPEECAL arditt
0
DGA sort1
420.9
366.8
-833
-1803
0.0
0.0
61
Sepuck
Adversarial
Adversarial
716
11093
Breakout: Return
D2t
Obert Return
2007
300
40R
0488.2
4575.8
241.2
206
1807.6
288.7
Figure 4: Average returns of DQN and QRDQN against adversarial state observation noises across four games. advX in the legend indicates random state observations with the perturbation size X. QRDQN (solid lines) almost consistently outperform DQN (dashed lines) given the same perturba- tion strength, yielding more training robustness against noisy state observations.
The results under the adversarial state observations are similar to those in the random noises case. Specifically, both DQN and QRDQN algorithms tend to degrade when getting exposed adversarial state observations, and even are more likely to diverge. For example, in Cart Pole both DQN (blue and green dotted lines) and QRDQN (blue and green solid lines) degraded to similar poor results 0.05 and 0.1. However, a key observation is that
when exposing to strong perturbation with QRDQN capable of obtaining desirable performance while DQN even diverges. For instance, in both Breakout and Qbert game, DQN (dotted blue lines) under the adversarial perturbation with 0.005 led to divergence, QRDQN (solid blue lines) can still attain relatively satisfactory performance eventually the condition that the performance of both two algorithms are on part with each other without any noisy state observations (red lines). Sensitivity Analysis of Different Perturbed States. To demonstrate the theoretical results (Theo- rem 4 and 5)) in the Appendix based on TD convergence condition and influence function we ever mentioned in Section 4.2 we provide the consistent results in Appendix and E Interested readers can refer to Appendix Jand for a more detailed discussion.
6 DISCUSSION AND CONCLUSION
The Lipsehitz continuity blessing based on the histogram distributional loss, but more ex- pected that similar conclusions can be made under Wasserstein Crammer distance these dis- tances are more approachable in real distributional RL algorithms. We leave it as future works.
In this paper, explored the training robustness of distributional RL against noisy state observa- tions. After the convergence analysis of distributional RL in the SN-MDP, we proved the Lipschitz continuity property of distributional RL, accounting for its less vulnerability. We also provided the TD convergence conditions and sensitivity analysis on more complex noisy settings. Experimental observations coincides with our theoretical results.

---

Under review as a conference paper at ICLR 2022
Ethics Statement. Our works reveals that distributional RL enjoy the training robustness against noisy state observations. The advantage is useful to defend against the poisoning attacks, thus contributing to the privacy of algorithms. Based experience, there is other ethic concerns of our work.
Reproducibility Statement. For the theoretical part, we clearly state the related assumption and detailed proof process in the appendix. In terms of the algorithm, our implementation is directly adapted from the public RL algorithms, including DQN and QR-DQN.
REFERENCES
Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. International Conference on Machine Learning (ICML., 2017. 2017.
Vivek S Borkar and Sean P Meyn. The ode method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization, 38(2):447-469, 2000.
Tong Chen, Jiqiang Liu, Yingxiao Xiang, Wenjia Niu, Endong Tong, and Zhen Han. Adversarial attack and defense in reinforcement learning-froe from ai security view. C 2(1):11, 2019.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. International Conference on Machine Learning (ICML), 2018a.
Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. Association for the Advancement of Arlificial Intelligence (AAAI), 2018b.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q- learning. Learning for Dynamics and Control, pp. 486-489, PMLR, 2020.
Sina Ghiassian, Andrew Patterson, Shivam Garg, Dhawal Gupta, Adam White, and Martha White. Gradient temporal -iffereecee learning with regularized corrections. In International Conference on Machine Learning, pp. 3524-3534, PMLR, 2020.
Ziwei Guan, Kaiyi Ji, Donald J Bucci Jr, Timothy Y Hu, Joseph Palombo, Michael Liston, and Yingbin Liang. Robust stochastic bandit algorithms under probabilistic unbounded adversarial attack. In AAAI, pp. 4036-4043, 2020.
Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. Advances in Neural Information Processing Systems, 2017.
Peter Huber. Robust Statistics, volume 523. John Wiley & Sons, 2004.
Inaam Ilahi, Muhammad Usama, Junaid Qadir, Muhammad Umar Janjua, Ala Al-Fuqaha, Dinh Thai Hoang, and Dusit Niyato. Challenges and countermeasures for adversarial attacks on deep rein- forcement learning. arXiv preprint arXiv:2001. 099644, 2020.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International Conference Machine Learning, pp. 2157-2166. PMLR, 2018.
Borislav Mavrin, Shangtong Zhang, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. International Conference on Machine Learning (ICMI.), 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature. 5 318(750):559-555 2015.
Thanh Tang Nguyen, Sunil Gupta, and Svetha Venkatesh. Distributional reinforcement learning with maximum mean discrepancy. Association for the Advancement of Artificial Intelligence (AAAI), 2020.
10

---

Under review as a conference paper at ICLR 2022
Anay Pattanaik, Zhenyi Tang, Shuijing Liu, Gautham Bommannan, and Girish Chowdhary. Robust deep reinforcement learning with adversarial attacks. Advances in Neural Information Processing Systems, 2017.
Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, pp. 8707- 8718. PMLR. 2020.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distribu- tional reinforcement learning, arXiv preprint DXXiv:0055.00855 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction. MIT press, 2018.
Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on observations. Advances in Neural Information Processing Systems, 2020.
Shangtong Zhang. Modularized implementation of deep algorithms in pytorch. https:// 2018.
github wwt.tu..com.oomm wwt.auu..co/SSaaattttttto www.://haanggttogggnnn//eeeeeeee
Shangtong Zhang and Hengshuai Yao. Quota: The quantile option architecture for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5797- 5804, 2019.

---

Under review as a conference paper at ICLR 2022
A THEOREM 3 WITH PROOF
Theorem (Convergence and Contraction of Bellman operators the SN-MDP) Given a policy n, define the Bellman operator T : RR - I [1 under random and udversarial states noises by T," and To respectively,
Denote "merged" policy x' where x (a)as) - () and S(T) is policy given x.
Then we have:
(1) T, is contraction operator and can converge to Va,, ie.. T. VoNN VION Vea, where multiple policies E (S(m) might exist with Dete (a(a))) = (als))
(2) To a contruction with the convergence satisfying Ta VooN= miny VaON = ViaN*, where N is optimal adversarial noise strategy. If the optimal policy Ta exists, it satisfies Ta (alv" = x [a]s) (lalss.fo for each and o, where v" (s) is the adversarial noisy state manipulated by N (-1s)).
Proef. Our proof is partly based Theorem 1 and 2 in Zhang et al. 2020), but adds more analysis the converged policy especially under the random noisy states setting. The most important insight in the following proof that the noise transition can be merged into agent's policy, resulting a new "merged" policy x'.
Proof of (1) Firstly, the Bellman Equation under the random noisy states right the general form in Eq 6 it automatically satisfies that T, VaON VeeN when it converges. As for the proof of contraction, based on insight about the new "merged" policy x²' where sa Em.,) (( our Bellman Operator as:
rewrite
T, (s)
[a(ss) E p(s'ls,a) R(s, +7
(14)
R(s) + PreN(s
where R(s)= = E. x' (als) E. p(s a) R(s a, s)) and P E. (als))ss)sslss determined by the "merged" policy n¹. Then for two different value function VLoN and ViON we have:
||, Vel
VON
T, V.VoNlle 1oc0
max
b
Σ P'
PI
max
P Pi (H') - V22 (2))(
(15)
max Σ
P' Aia max (s') - (2)
=mmax >
P' P IIVi V2oNllo
=>|1V> TON V
Then according to the Banach fixed-point theorem, since (0, 1). VOON converges to unique fixed-point Val. However. even though the obtained policy s' satisfies that x' ra(ass) = En(s) N ( for each s, these equations can not necessarily guarantee unique especially when these equations behind this condition are underdetermined. In such scenario, multiple policies will exist as long as they satisfy the equations above.
Proof of (2) Firstly, based Theorem (Zhang veversriial noisy state setting for the convenience of exist, we assume that an optimal policy exists in the following analysis. Based on this assumption, we need to derive the explicit value function under the adversary. Inspired by (Zhang al. 2020). the proof insight is that the behavior of optimal adversary can be also viewed as finding another optimal polic,, yielding zero-sum two player game. Specifically, in the SN-MDP setting, the adversary selects action à E satisfying vs)), attempting maximize its state-action value function
2020) that shows optimal policy does always
12

---

Under review as a conference paper at ICLR 2022
Qua (s) Then the adversary's value function Vr, (s) can be formulated
Vr, (+)
max Qr, (s, a)
max Σ ( (,,,)))(((ss,,,)) 7V..
(16)
max v(a)
-
+. (4))
where 0(s'|s, the transition dynamics of the adversary. satisfying p(s'ls,)) E. <<clv(())p(( (s,ppss"sss,,, s,
from the perspective of the agent. R(s, à, 8') is the adversary's 's reward function while taking action â, which is the opposite number of R(s, a, 5') given the action a. addition, since both the adversary and agent can serve zero-sum two-player game, it indicates that (s) adversary setting. Then we rearrange the equation above as follows:
Vra (s) for the agent's value function Vr, in the
Vr. (s) -Va (s) ΣΣ
- min NG1e)
-Vee(''))
ΣΣ min o(3)
x' r(a)))066
(17)
VVe (s'))
ΣΣ min
# (als)p(s'|s.. 7 min EroN
Σ 14444 )
N(-la)
=nin (s)
Note that we optimize over N. which means consider (((ss) for each state Further, we derive the contraction of the Bellman operator Ta" We rewrite Bellman Operator T." as:
To" (x)N(() = min VRON (s)
min R(s) +7 Σ Paw
(18)
We firstly assume T," V.l (6) 2 T V2. (s), then have:
Ta. ViON (8) Tea (8)
< max (7
Σ Vi > pr Pr PPNN(())
N(-le)
Σ Pr.er Vi VHoN(s') VeoN(s'))
max
M(+x)
(19)
Σ p' N max (5') - v
7 max
N(1a)
P ViooN VilN 11o0
max M(-e)
21IV 1ox
where the first incquality holds as ming, f(x1) ming, g(x2) maxx(f(x) -x(x)) and we extends this incquality into the Wasserstein distance the proof of convergence of distributional RI. setting in Appendix
The last inequality holds since only P', depends on (-1ss) while the infinity norm is constant, which is independent with the current (--ss). Similarly, the other scenario be still proved. Thus, have:
|T. ( - T lbo S < V2
(20)
Thus, proved that T. is still contraction and converge minN VrON. We denote it V-ON- In addition. based the insight of the "merged" policy The have n/. Evcs N (((()))s) maalv" (s)) where the deterministic state o" (s) is the adversarial noisy state from the state
13

---

Under review as a conference paper at ICLR 2022
B PROOF OF THEOREM 1
Proof. Firstly, we will provide the properties of Wassertein distance dp in Lemma that we leverage in following convergence proof.
Lemma (Properties of Wasserstein Metric) consider distribution distance between random able U and Denote dg as the Wasserstein distance between two distribution defined in Eq. 4 For any sealar a and random variable A independent of U and V, the following relationships hold:
do(aU, aV) < (old,(U, d + A+ & V)(U,V) d,(AU,AV) & (IA)l,d,(U.V)
(21)
Further, let A1, A2,.. be set of random variables describing the partition of w. when the partition lemma holds:
d,(U, V) S Σ (AA.AAVVV).
(22)
Then, the following contraction proof is the maximal form of dp and we denote it as dy.
Proof (1) This contraction proof is similar the original (Bellemarc al. 2017) in the distributional RL without state observation noises. The only difference lics in the new transition operator P7, but it dose not change the main proof process. For two different random variables 2 and 2 about returns, we have:
(
= (T, (s)
supd,(R(s,a, omg 3,, (P.
(23)
S y 4 *. ''
y sup dz (')66',))),22( ''a''
sus d / (s. a). 2 (s,a)
=ddp zz²)
Thus, we conclude that म.- Z - Z is y-contraction in dp.
Proof of (2) Firstly, define the distributional Bellman optimality operator in MDP
IZ(s, 2 R S") + Tz()'))
(24)
where S ~ P(-ls, and Tz (S') arg max, E [S' a')] By contrast, in SN-MDP, Our greedy adversar-
ial rule Nº (-Isss) based on the greedy policy rule distributional Bellman optimality operator, which attempts to find adversarial N (-]s") in order to minimize E [ ,, a')], where d' ~ ++(VV(s)) and V ((s)== N (-1s"). We assume N (-(s") yields deterministic state s*, and thus the agent always takes action based on s", which denote A TT+]="), Therefore, can obtain the statc-action function Q (s, a) under the adversary
Qi- (s, a) min E (x, a)]
N
(25)
= E 2 (s.a)
where T² (-18) [[([s") for v's that follows the adversarial policy A".
14

---

Under review as a conference paper at ICLR 2022
Next, derive the contractive property of Tg. denote state-action valued distributions Z (s, a) and 2 (s, a). Then have:
N supd,
(s, 0), (s. (s,0),
ysup Σ I
2.a
= Σ P1
(26)
ysup
_ (s
ssup Σ x((c)5)dd(233$5)
S sup (s 23 s'aak
(x (x,,)) (
Thus, we conclude that 5T still contracctio dp.
C PROOF OF THEOREM 2
Proef. Firstly, we show the derivation details of the Histogram distribution loss starting from KL divergence between and qa. p, is the cumulative probability increment of target distribution TZa within the i-th bin, and Yo corresponds to a (normalized) histogram, and has density values [((x())) per bin. Thus, we have:
C(Zo. = - 1-² p(y) log go (v)dy
let p(v) log
fl (x(()) dy
1 (x(s))
(27)
log
(FEzo (Ie + wi) Fizo (1))
Pe log ff (x(s))
where the last equality holds because the width parameter uy can he ignored for this minimization problem.
15

---

Under review as a conference paper at ICLR 2022
Next, we compute the gradient of the Histogram distributional loss.
a
Dj log (x(s))
(x((s)
5-1
Σ Pj (x(() (x(((()))
=
& exp(x(s) (0) 0.)
Σ
Po (x()) (x(())
Σ
Σ p=1 exp(x(s) Tpp)
(28)
Σ Pj S. (x(s)) 0.)
==1
p3Oj Σ (x(()
Σ (ps (x(3)))
Then, as we have ||0:|| S 7 for Vi, we hound the norm of its gradient
Ps log fy (x(s))
(x(8) 1=1
[1pp- f (x(s)))0O1I
(29)
lp: (x(3))11111
< kl
The last equality satisfies because Ip: - / (x(s)) is less than I and even smaller. In summary, compared with the least squared loss in expectation-based RL, the histogram distributional loss in distributional RL can additionally enjoy kd-Lipschitz continuity with bounded gradient norm regarding the state features x(s). This upper bound of gradient norm can mitigate the impact the noises on state observations on the loss function, therefore yielding training robustness for distributional RL.
D TD CONVERGENCE UNDER NOISY STATE OBSERVATIONS
Let pa()) be the stationary distribution under the policy т and p(s'|s) (s) be the transition probability from to s' satisfying p(s"(s) = Σ. <<als)p(s') (s'ss,s,,, a). We analyze conditions of TD convergence when exposing state observation noises. Firstly, we recall the classical TD update step
W Wt + au (Re41 + yw: x W. Xx)x:
(30)
where is the step size at time f. Once the system has reached the steady state for any wi, then the expected
next weight vector be written E[w.41] w.] = w. + a.(b Aw.). where b B( 1dd
and A E. [x{d]]] E Rdxd_ The TD fixed point WTD to the system satisfies Awnp = b. From (Sutton & Barto 2018). we know that the matrix A determines the convergence in the linear TD setting. In particular, converges with probability one to the TD fixed point if A is positive definite. However, we add noises on either x, or X in Eq. 30 the convergence condition will be different. Theorem 4 (proof Appendix provides conditions for TD convergence in three different noisy state observation settings.
Theorem 4. (Conditions for TD Convergence under Noisy State Observations) Define P as the [S] x [S] matrix forming from p(s'(s), [s). D the |S] x [S] diagonal matrix with p(s) on its diagonal, and X the |S] x matrix with x(s) as rows, and E is the |S| perturbation matrix with each perturbation vector o(s) as its rows. The stepsizes ar E (0, 1] satisfy 2 ai = oc and Ene of = O. For noisy states, we consider the following three cases: (i) e(s) on current state features, i.e Xt + Xt + e.. (ii) e(s') on next state features, i.e. Xe+1 &- Xet1 et+1, (iti) the same on both state features. We can attain that converges TD fixed point if the following conditions are satisfied, respectively.
16

---

Under review as a conference paper at ICLR 2022
Case (i): both A and (X E) DDE are positive definite. Case (ii): both A and -X DDDE are positive definite. Case (iti): positive definite.
From the convergence conditions for the three cases in Theorem 4 it clear that (ii)) the mildest. This the same condition as that in the normal TD learning without noisy state observations. Note that the case (iii) can be viewed as the SN-MDP setting, whose convergence has been already rigorously analyzed in Section P In Section | our experiments demonstrate that both expectation-baeed and distribution RI. are more likely to converge case (iii) compared with case and (ii).
cases (i) and (ii), the positive definiteness of X DPE+ E DPE and X DPE is crucial. We partition (X + E) DPE into X DPE + DPE, where the first term has the opposite positive definiteness to X DPE, and the second term is positive definite (Sutton & Barto 2018). Based on these observations, we discuss the subtle convergence relationship in cases (i) and (ii):
(1) If -x DPE is positive definite, which indicates that TD convergent in case (ii). TD can still converge in case (i) unless the positive definiteness of E DPE dominates in X DPE + +E' DPE.
(2) If X DPE is negative definite, TD is likely to diverge in case (ii). By contrast, TD will converge in case ().
summary, there exists a subtle trade-off TD convergence in case (i) and (ii) if we approximately ignore the term E' DPE in case (i). The key of it lies in the positive definiteness of the matrix X DPE, which heavily depends on the task. In Section we empirically verify that the convergence situations for current and next state observations normally different. Which situation is superior is heavily dependent on the task.
E SENSITIVITY ANALYSIS BY INFLUENCE FUNCTION
Next, we conduct an outlier analysis by the influence function, key facet in the robust statistics (Huber 2004), The influence function characterizes the effect that the noise in particular observation has an estimator, and can utilized to investigate the impact of one particular state observation noise on the training reinforcement learning algorithms. Specifically, suppose that Fe is the contaminated distrihution function that combines the clear data distribution F and an outlier x. The distribution Fe can be defined as
F.
e) + cd..
(31)
where 62 is probability measure assigning probability 1 to x. Let 0 be a regression estimator. The influence function of A at F, +0 X' - r'is defined as
(1) lim (FF(())) C-
(32)
Mathematically, the influence function the Gateaux derivative of 0 at F in the direction 62. Owing to the fact that traditional dalue-based RL algorithms, e.g., DQN (Mnih et al.] (2015). can be viewed as a regression problem (Fan et al. (2020), the linear TD approximator also has strong connection with regression problems. Based on this correlation, in the following Theorem [1] learning in the case of linear function approximation.
we quantitatively evaluate the influence function of TD
Theorem (Influence Function Analysis in TD Learning with linear function approximation) Denote X. x E Rd, and A B [xdd]] E [dxxd, Let Fa be the data distribution generated from the environment dynamics given policy n. Consider an outlier pair (Xt, Xx41) with the reward Re41, the influence function ub of this pair the estimator is derived as
t/ (Xe, X++1) = E(A A) - xx (Re de w).
(33)
Please refer to Appendix G for the proof. Theorem 5 shows the quantitative impact an outlier pair (x,, x on the learned parameter w. Moreover, a corollary can be immediately obtained to make a precise comparison of the impacts of perturbations on current and next state features.
Corollary 1. Given the same perturbation n on either current next state features, i.e., Xs, and X 1, at the step I, approximate m87 x; and 1u1) as O as is small enough, the following relationship between the resulting variations of influence function, Ax. sp and Ax.-13), holds:
7xx, v Axyt 22denxx (Re+1- w).
(34)
We provide the proof of Corollary D in Appendix Under this equation, the sensitivity of noises on x, and X.+1. measured by Ax, vi and Ax.41 v. present trade-off relationship as their weighted sum is definite. However, there is not an ordered relationship between Ax. 1o and Ax v.. In summary, we conclude that the sensitivity of current and next state features against perturbations is normally divergent, and the degree of sensitivity is heavily determined by the task. These conclusions similar to those we derived in the TD convergence part.
17

---

Under review as a conference paper at ICLR 2022
F PROOF OF THEOREM 4
Proof. To prove the convergence of TT under the noisy states, the results from (Borkar & Meyn 2000 that require the condition about stepsizes as holds: Σ Or = oo and Ei=o of D. Based on (Sutton & Barto (018). the positive definiteness of A will determine the TD convergence. For linear TD(0), in the continuing case with 7 < 1. A can be re-written as:
A Σ ((s) Σ T(ass) Σ s(r,,'lo,, a)x, (x. 7x++1)
Σ u(s)
(xi -1--1)
(35)
((5)xx(Xx E [((61s))x11)
X DX - DyPx
X D(I-
Then we use A. to present the convergence matrix the case (i) where the perturbation vector e, is added onto the current state features, i.c, X₁ f X; + ex, while we use A4+1 and Ar. present the counterparts in the case (ii) and (iii), respectively. Based on Eq. 35 in the case (iii), we have:
A (X + E) D(X E) - (X E) Dyp((
(36)
= (X E) D(1-
From (Sutton & Barto 2018), we know that the inner matrix D(I PP) is the key to determine the positive definiteness of A. It we assume that A is positive definite, which also indicates that D(I - -P) is positive definite equivalently. As such, Ar.+1 is positive definite automatically, and thus the liner TD would converge the TD fixed point. Next, the case (i) have:
A. (X E) D(X E) (X E) D,PX
A X DE E DX E DE E D,PX
(X E) D(I PP)(X E) (X E) DDPPE
(37)
Au.t + + E) DPE
Au.4 +7(X D,PE E DyPE)
Similarly, in the case (ii), we can also attain:
A = X DX X DyP(X + E)
(38)
= A A-72 DPE DPE
We know that the positive definiteness of and At,++1 only determined by the positive definiteness of the inner matrix D(I - PP) we assume the positive definiteness A, i.e. the positive definiteness Ar,41 and DII PP,, as ry what only need to focus the positive definiteness of X DPE+ + E DPE and x DPE. If they are positive definite, TD learning will converge under their cases, respectively.
G PROOF OF THEOREM 5 AND COROLLARY
Prouf. We combine the proof of Theorem 5 and Corollary together. The TD fixed point to the system satisfies Awm b. Thus. the TD convergence point, i.c., TD fixed point, be attained by solving the following regression problem:
min b Awl2
(39)
To derive the influence function, consider the contaminated distribution which puts a little more weight on the outlier pair (Xe, x++1):
w arg min(1
e)BB Aw) T(b Aw)]+
(40)
<(3h x w) (yo x(w).
where = Re41 Ne and u5 = dxx,. We take the first condition:
(1 - C2A Aw 2A Ta)
2cr 27 w)
(41)
18

---

Under review as a conference paper at ICLR 2022
Then arrange this equality and obtain:
c)E A ZAT)) -1--))E(A (b) ELAYE
(42)
Then we take the gradient and let 0, then we have:
(-B(A A) + Jw. E(A A),..F A)) -B(A (b)
(43)
We know that under the least square estimation, the closed-form solution of w. is I. A (ATA AFAA Thus, after the simplicity, finally attain:
4o Po (Xt, x = E A T A) (y6 x(w)
(44)
= E. 1 dxx, x. Rut+1 de w).
Next, prove the Corollary. We only need focus the item d,x; x, di w). which denote th. Then we use Az, 1 and Axyy1 d to represent the change of y after adding perturbations 7 on X, and X++1; respectively. In particular, since approximate 1m) x; and ryn w as then we have that the change of influence function for the perturbation 7 on the current state feature Xx:
Az, 2 = (d: + (x, Xx + 2n x. ( di w - w) U/o
= -dx Xcnj w + 2d,77 xx (Re+1 di w) + +n-x; Xx (Re+1 dg w)
(45)
2den x. [Re4 of w) 1 (rdxx, x.1 w mxx x xHe41 di w)).
Then the influence function for the perturbation 17 on the next state feature xe+1 1 is:
Arit v = (de )))xx (R dj w 777 w) 4/0
(46)
2 7dxx X w 712, x, (Re+1 - d, w).
Finally, it is easy to observe that the following relationship holds:
A v/ = 2yderrr (Re+1 d, w) Areti v.
(47)
L
H EXPERIMENTAL SETUP
After linear search, QR-DQN, We set for Huber quantile loss across all tasks due its smoothness.
Cart Pole After linear search, in the QR-DQN, we set the number of quantiles N to be 20, and evaluate both DQN and QR-DQN 200,000 training iterations.
Mountain Car After linear search, in the QR-DQN, we set the number of quantiles N to be 2, and evaluate both DQN and QR-DQN 100,000 training iterations.
Breakout and Qbert After a linear search, in the QR-DQN, we set the number of quantiles to be 200, and evaluate both DQN and QR-DQN on 12,000,000 training itcrations.
I SN-MDP SETTING WITH RANDOM NOISES
From Figure we can easily observe that all DQN and QRDQN algorithms under various strength of random state noises converge, although they eventually obtained different average returns. This empirical observation is consistent with theoretical analysis in Section 3 where both classical and distributional Bellman operators are contractive and thus convergent. In addition. Figure 5 also manifests that the eventual performance that algorithms attained has a decreasing tendency as the perturbation strength, ie., standard deviation, increases especially for DQN. Remarkably, the final performance of QRDQN is more robust than DQN against different perturbation strength, especially in Mount Car and Breakout games, although both DQN and QRDQN are convergent in this SN-MDP-like setting.
19

---

Under review as a conference paper at ICLR 2022
Random
200
Mountain Car: Relurn
Retum
177
DOA poaltoo OIE
1:00
DRSCA LndDOsT
272.2
150.
123
100
620.7
784.5
GROON 2H-000
1800
8.0
Skyarh LP)
Sejrot aP5
Random
Random
Breakout: Return
00% aecket
D0A snaiis
Qbert: Retum
2003
7443.8
600
6093
458
111.6
10.0
Figure 5: Average returns of DQN and QRDQN against random state observation noises across four games. randX in the legend indicates random state observations with the standard deviation X. Both QRDQN (solid lines) and DQN (dashed lines) converge and the convergence level is determined by the perturbation strength.
Random Current
Random / Next
Random / Both
700
700
700
DIN
DGN CJ11
DGN
DGN DCT
DON U.!
DON
617.0
Breakout: Return
600
DON 0:25
DON 0:26
600
DON
GRDON
GRDON
GRDON
GRDON 3.01
GRDON
GRDON
CHIOON cas
GHIGN 0.05
CHUCNNO..CS
www.
$5.0
399.9
300
240.0
200
200
200
100
100
100
79.5
2.8
0.0
2.0
4.0
6.0
8.0
10.0
12.0
0.0
2.0
4.0
6.0
8.0
10.0
12.0
0.0
2.0
4.0
6.0
8.0
10.0
120
SlppeC 10f)
Steja(x
Stepe
Figure 6: Average returns of DQN and QRDQN against random state observation noises Breakout environment over 200 runs.
20

---

Under review as a conference paper at ICLR 2022
J EXTENSION ANALYSIS OF PERTURBATIONS ON CURRENT, NEXT AND BOTH STATE OBSERVATION
We provide the results of robust performance with different perturbation states in Breakout under random and adversarial noisy state observations in Figure respectively.
In the random state observation setting shown Figure 6 it turns that perturbations current state observations more robust than next state by comparing the first and second subfigures given the same pertur- bation strength. This empirical finding demonstrates the conclusion in Appendix D and B that the sensitivity of current and next states are normally divergent. The result where noises are added on both states seems to be most robust, consistent with the mildest TD convergence condition as shown in Theorem
Adversarial Current
Adversarial Next
Adversarial Both
DGN
Breakout: Return
600
600
500
300
500
495
0
4955
OREOON E1D1
493.2
400
316
300
238.0
300
300
255.9
200
200
200
100
100
100
64.0
2.0
10.0
120
Sign 10")
Stepa(X 10's
Stpei (x 10fy
Figure 7: Average returns of DQN and QRDQN against adversarial state observation noises on Breakout environment over 200 runs.
In the adversarial state observation setting as shown in Figure 7 it also turns out that perturbations on cur- rent state observations is more robust than next state by comparing the first and second subfigures given the perturbation strength, and algorithms converge easiest. These empirical results also demonstrates the conclusion made in Appendix [
21


Reviews:
[{"review_id": "tDdPwVjcOZr", "reviewer": "Reviewer_aTVu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. This paper studies the robustness of distributional RL agents, which were never done by previous works. The results are quite promising and show that distributional RL might be more robust than expectation-based RL agents under noises on state observations.\n\n2. The analysis includes both tabular and function approximation case, and an important theorem on the Lipschitz continuity of distributional RL is given.\n\n3. Experiments are relatively comprehensive.\n\nWeaknesses:\n\n1. The proposed SN-MDP seems to be very similar to SA-MDP (Zhang et al., 2020). I understand the analysis under the distributional RL setting is new, but it is not clear to me what is the main difference in terms of SA-MDP formulation. Can you explain?\n\n1. The paper is not well organized and writing can be improved. For example, there is no need to create 4 sub-sections for showing experimental results on 4 models. I think one subsection for random noise and one subsection for adversarial noise can be better. Theorems are not well motivated and the writing is hard to follow, especially in Section 4.2 and 4.3.\n\nQuestions:\n\n1. In Figure 3 the DQN lines (dotted red lines) look better than QRDQN, which do not match texts (\"QRDQN (solid lines) almost consistently outperforms\nDQN\"). Is it a plotting error?\n\n2. Is it possible to empirically measure the Lipschitz continuity (Theorem 2), the positive definitiveness of the matrix $X^T DPE$ (Theorem 3), and the influence function (Theorem 4) on a toy example?", "summary_of_the_paper": "This paper studies the robustness of distributional reinforcement learning, in particular the robustness on state observations, which have been demonstrated in a few papers on adversarial attacks to deep reinforcement learning. Compared to existing works on robust reinforcement learning on state observations, the main difference in this work is that it considers the distributional RL setting and also considers noise during training time. Theoretically, the authors find that distributional RL can be more robust under this setting, via the lens of Lipschitz continuity of the loss function and the influence function. The findings are also verified empirically on 4 benchmarks.", "summary_of_the_review": "I feel this paper does present an interesting study but my main concern on is its presentation. I feel this paper can become a good paper if the weakness and questions mentioned above can be addressed. I currently rate the paper at the borderline but I am willing to reevaluate the paper based on the authors' response.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Hiujprwqefj", "reviewer": "Reviewer_crGD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors extended State-Adversarial MDP to SN-MDP by considering more general noise generating mechanism. The theoretical contribution of this work seems clear, but I couldn't find out clear motivation and contribution from the experiments. The given experiments seems more focused on whether SN-MDP's Bellman operators and TD learnings can be used for various control problems or not and whether those are aligned with theoretical intuitions. However, I believe authors should have given experiments on the usefulness of this framework (e.g., authors may suggest more practical but simple problems where training observations are noisy as stated in the introduction). Also, only 3 runs are used for each experiments, and all results are reported without standard errors, which means that we cannot evaluate the statistical significance just by using the reported results. Therefore, I believe the empirical results should be refined for this work to be accepted. ", "summary_of_the_paper": "This work presented State-Noisy Markov Decision Process (SN-MDP), where there is a noise generating mechanism (either from the environment noise or from the adversary), and the theoretical properties (such as convergence and contraction) for corresponding (expected) Bellman operator and distributional Bellman operator were proved. The theoretical analysis was done for both tabular and linear funcion approximation settings. Especially in function approximation setting, authors characterized the robustness blessing of distributional RL based on histogram distributional loss and analyzed how the noise factor affects TD learning by using influence function that utilizes the perburbation method. Empirical analysis was done for DQN and QRDQN by varying noise standard deviations and the position of noise (state/successor state or both), which aims to support the authors' intuition coming from their theorems. ", "summary_of_the_review": "Although the theoretical contribution seems clear, I believe we need additional experiments to support the authors' claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EJnWIx-xMA3", "reviewer": "Reviewer_DaN6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of understanding the distributional robustness of distributional RL seems novel and has not been done before. Other than that, the technical contributions and insights are marginal, unclear, and disconnected: \n- Theorem 1 is a marginal extension of (Zhang et al., 2020): they impose a distribution $N$ on a disturbance function $v(s)$\n- Function approximation: It was actually a simple linear model with KL loss as the distributional loss for distributional RL, thus the Lipschipts smoothness there is very straightforward and is not really helpful for the real setting of distributional RL. Moreover, In RL, we often consider bounded reward, so if we consider the expected RL with squared loss function, it still enjoys all the properties of KL loss arising from distributional RL \n- Disconnected results: How were the analysis in Sections 4.2 and 4.3 related to distributional RL? After the sensitivity analysis, the paper did not give any actionable insights from their analysis except that \"the degree of sensitivity is heavily determined by the task\" . This conclusion is not helpful and does not need any such sensitivity analysis. \n- The paper says that QRDQN is more robust than DQN in MountainCar but Figure 3 tells the opposite. Plus, three experiments are of the same nature; thus I think it is not necessary to split it into three subsections to make them sound comprehensive. \n- Section 5.3: \"QRDQN eventually achieves similar performance as DQN, although QRDQN significantly reduces the sample efficiency [in Breakout]\". From my own experience with this experiment, I don't think so. \n\n\n**Minor comments**:\n- Eq (4) can be simplified \n- Section 3.1: \"the state space go to infinity\": unclear \n\n- ", "summary_of_the_paper": "This paper investigated the adversarial robustness of distributional RL against noisy/adversarial states. They proved the Lipschitz continuity of distributional RL , provided convergence condition for TD update under noisy states, and conducted sensitivity analysis", "summary_of_the_review": "The idea of exploring adversarial robustness of distributional RL is interesting and novel, but the analysis presented in the paper are marginal, disconnected, and does not support the understanding of the robustness of distributional RL. \n\n===== AFTER REBUTTAL ===    \nIncrease the score to 3", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Description of paper content:

A mixed theoretical and experimental paper that investigates the robustness of distributional RL to perturbations of state observations as compared to expectation-based value function learning. They provide sufficient conditions for TD’s convergence and prove the Lipschitz continuity of the loss of a histogram-based KL version of distributional RL with respect to the state features, whereas this is not true for expected RL. This continuity indicates a certain robustness of the loss with respect to perturbations of the state. The theory’s tie to experiment is weak in the sense that it is not predictive of the actual performance of any algorithm. The theoretical methods are based on a previously published paper SA-MDP.

Summary of paper discussion:
The reviewers raised concerns about the statistical significance of the experimental results, the clarity and organization of the writing, the novelty of the theoretical setting, and its usefulness for describing a real problem setting. The majority of reviewers rejected the paper and did not lift the scores after the rebuttal. 

(I personally wonder if the community would not benefit from conducting some of these kinds of theoretical analyses and experiments on LQR systems rather than Atari (etc.) environments.)
Decision: Reject