Exploring the Robustness of Distributional Reinforcement Learning against Noisy State Observations

In real scenarios , state observations that an agent observes may contain measurement errors or adversarial noises , misleading the agent to take suboptimal actions or even collapse while training . In this paper , we study the training robustness of distributional Reinforcement Learning ( RL ) , a class of state-of-the-art methods that estimate the whole distribution , as opposed to only the expectation , of the total return . Firstly , we propose State-Noisy Markov Decision Process ( SNMDP ) in the tabular case to incorporate both random and adversarial state observation noises , in which the contraction of both expectation-based and distributional Bellman operators is derived . Beyond SN-MDP with the function approximation , we firstly analyze the vulnerability of least squared loss in expectationbased RL , and by contrast theoretically characterize the bounded gradient norm of histogram-based distributional loss , accounting for the better training robustness of distribution RL . Finally , extensive experiments on the suite of games show that in SN-MDP-like setting both expectation-based and distributional RL can converge albeit corresponding to different levels under various state observation noises . However , distributional RL can enjoys better training robustness in the more complicated noisy state observation settings compared with its expectationbased counterpart . 1 INTRODUCTION . Learning robust and high-performance policies for continuous state-action reinforcement learning ( RL ) domains is crucial to enable the successful adoption of deep RL in robotics , autonomy , and control problems . However , recent works have demonstrated that deep RL algorithms are vulnerable either to model uncertainties or external disturbances ( Huang et al. , 2017 ; Pattanaik et al. , 2017 ; Ilahi et al. , 2020 ; Chen et al. , 2019 ; Zhang et al. , 2020 ; Shen et al. , 2020 ; Singh et al. , 2020 ; Guan et al. , 2020 ) . Particularly , model uncertainties normally occur in a noisy reinforcement learning environment where the agent often encounters systematic or stochastic measurement errors on state observations , such as the inexact locations and velocity obtained from the equipped sensors of a robot . On the other hand , external disturbances are normally adversarial in nature . For instance , the adversary can construct adversarial perturbations on state observations to degrade the performance of deep RL algorithms . These two factors lead to noisy state observations that influence the performance of algorithms , precluding the success of reinforcement learning in real-world applications . Existing works mainly focus on improving the robustness of algorithms in the test environment with noisy state observations . Smooth Regularized Reinforcement Learning ( Shen et al. , 2020 ) introduced a regularization to enforce smoothness in the learned policy , and thus improved its robustness against measurement errors in the test environment . Similarly , the State-Adversarial Markov decision process ( SA-MDP ) ( Zhang et al. , 2020 ) was proposed and the resulting principled policy regularization enhances the adversarial robustness of various kinds of RL algorithms against adversarial noisy state observations . However , both of these works assumed that the agent can access clean state observations during training , which is normally not feasible when the environment is inherently noisy , such as unavoidable measurement errors . Thus , the maintenance and formal analysis of policies robust to noisy state observations during training is a worthwhile area of research . On the other hand , recent distributional reinforcement learning algorithms , including C51 ( Bellemare et al. , 2017 ) , Quantile-Regression DQN ( QRDQN ) ( Dabney et al. , 2018b ) , Implicit Quantile Networks ( Dabney et al. , 2018a ) and Moment-Matching DQN ( MMD ) ( Nguyen et al. , 2020 ) , constantly set new records in Atari games , gaining huge attention in the research community . However , existing literature mainly focuses on the performance of algorithms , other benefits , including the robustness in the noisy environment , of distributional RL algorithms are less studied . As distributional RL can leverage additional information about distribution that captures the uncertainty of the environment more accurately , it is natural to expect that distributional RL with this better representation capability can be less vulnerable to the noisy environment while training , which motivates our research . In this paper , we investigate the robustness of distributional RL against various kinds of state observation noises encountered during training . Firstly , we propose a general State-Noisy MDP in the tabular setting , in which we prove the convergence of distributional Bellman operator . We further extend SN-MDP to the function approximation case by considering more complex noisy state observations . Notably , we analyze the vulnerability of classical RL and in contrast characterize the Lipschitz continuity blessing resulting from the Histogram distributional loss in distributional RL , which leads to a bounded gradient norm . This better behaved gradient mitigates the impact of noisy states on the objective function , accounting for the less vulnerability of distributional RL while training . Finally , extensive experiments demonstrate that both expectation-based and distributional RL algorithms can converge in SN-MDP-like settings . More importantly , distributional RL algorithms tend to achieve better robust performance in the presence of more complex state observation noises compared with its expectation-based counterpart that may even diverge in some cases . These empirical results in Section 5 echo our previous theoretical results in both Section 3 and 4 . Overall , the training robustness advantage of distributional RL algorithms we revealed facilitates their deployment especially in the noisy environment . 2 BACKGROUND . 2.1 DISTRIBUTIONAL REINFORCEMENT LEARNING . In the tabular setting without noisy states , the agent ’ s interaction with its environment can be naturally modeled as a standard Markov Decision Process ( MDP ) , a 5-tuple ( S , A , R , P , γ ) . S and A are the state and action spaces , P : S × A × S → [ 0 , 1 ] is the environment transition dynamics , R : S ×A× S → R is the reward function and γ ∈ ( 0 , 1 ) is the discount factor . Value Function vs Value Distribution . Firstly , we denote the return where st = s as Zπ ( s ) =∑∞ k=0 γ krt+k+1 , representing the cumulative rewards following a policy π , and rt+k+1 is reward scalar obtained in the step t + k + 1 . In the algorithm design , traditional expectation-based RL normally focuses on value function V π ( s ) , the expectation of the random variable Zπ ( s ) : V π ( s ) : = E [ Zπ ( s ) ] = E [ ∞∑ k=0 γkrt+k+1 | st = s ] . ( 1 ) In contrast , in the distributional RL setting , we focus on the value distribution , the full distribution of Zπ ( s ) , and the state-action value distribution Zπ ( s , a ) in the control problem where st = s , at = a . Both of these distributions can better capture the uncertainty of returns in the MDP beyond just its expectation ( Dabney et al. , 2018a ; Mavrin et al. , 2019 ) . Distributional Bellman Operator . In expectation-based RL , we update the value function via the Bellman operator T π , while in distributional RL , the updating is on the value distribution via the distributional Bellman operator Tπ . To derive Tπ , we firstly define the transition operator Pπ : Z → Z : PπZ ( s , a ) : D= Z ( S′ , A′ ) , S′ ∼ P ( ·|s , a ) , A′ ∼ π ( ·|S′ ) , ( 2 ) where we use capital letters S′ and A′ to emphasize the random nature of both , and : D= indicates convergence in distribution . For simplicity , we denote Zπ ( s , a ) by Z ( s , a ) . Thus , the distributional Bellman operator Tπ is defined as : TπZ ( s , a ) : D = R ( s , a , S′ ) + γPπZ ( s , a ) . ( 3 ) More importantly , Tπ is still a contraction for policy evaluation under the maximal form of the Wasserstein metric dp over the true and parametric value distributions ( Bellemare et al. , 2017 ; Dabney et al. , 2018b ) , where the p-Wasserstein metric dp is defined as dp = ( ∫ 1 0 ∣∣F−1Z∗ ( ω ) − F−1Zθ ( ω ) ∣∣p dω ) 1/p , ( 4 ) which minimizes the distance between the true value distribution Z∗ and the parametric distribution Zθ . F−1 is the inverse cumulative distribution function of a random variable with the cumulative distribution function as F . In the control setting , the distributional analogue of the Bellman optimality operator converges to the set of optimal value distributions , although it is in a weak sense and requires more involved arguments ( Dabney et al. , 2018b ) . 2.2 TWO KINDS OF NOISY STATE OBSERVATIONS . We investigate both random and adversarial training robustness , i.e. , the performance of RL algorithms under these two types of noisy state observations , between the expectation-based and distributional RL algorithms . We consider continuous state observations with continuous noises . In the random noisy state case , we apply Gaussian noises with mean 0 and different standard deviations to state features to simulate the measurement error stemming from various sources . In the adversarial state perturbation setting , we construct white-box adversarial perturbations on state observations for the current policy during training , following the strategy proposed in ( Huang et al. , 2017 ; Pattanaik et al. , 2017 ) that leveraged the gradient information of an engineered loss function . In particular , we denote atw as the “ worst ” action , with the lowest probability from the current policy πt ( a|s ) in the training step t. Thus , the optimal adversarial perturbation ηt , constrained in an -ball , can be derived by minimizing the objective function J : min η J ( st + η , πt ) = − n∑ i=1 pti log πt ( ai|st + η ) , s.t.‖η‖ ≤ , ( 5 ) where pti = 1 if i corresponds to the index of the least-chosen action , i.e . the w-th index in the vector a , otherwise pti = 0 . In other words , we construct a target one-hot action p t with 1 assigned to the index of the least-chosen action . Through this minimization in the form of the cross entropy loss , we can construct the state perturbations ηt that can force the policy to choose the least-chosen action atw in each t step . 3 TABULAR CASE : STATE-NOISY MARKOV DECISION PROCESS . In this section , we extend State-Adversarial MDP ( Zhang et al. , 2020 ) to a more general StateNoisy Markov Decision Process ( SN-MDP ) , and particularly provide a proof of the convergence and contraction of distributional Bellman operator in this setting . 3.1 DEFINITIONS . As shown in Figure 1 , SN-MDP is a 6-tuple ( S , A , R , P , γ , N ) , where the noise generating mechanism N ( ·|s ) maps the state from s to v ( s ) using either random or adversarial noise with the Marko- vian and stationary probability N ( v ( s ) |s ) . It is worthwhile to note that the explicit definition of the noise mechanism N here is based on discrete state transitions , but the analysis can be naturally extended to the continuous case if we let the state space go to infinity . Moreover , let B ( s ) be the set that contains the allowed noise space for the noise generating mechanism N , i.e. , v ( s ) ∈ B ( s ) . Following the setting in ( Zhang et al. , 2020 ) , we only manipulate state observations but do not change the underlying environment transition dynamics based on s or the agent ’ s actions directly . As such , our SN-MDP is more suitable to model the random measurement error , e.g. , sensor errors and equipment inaccuracies , and adversarial state observation perturbations in safety-critical scenarios .

Summary: This work presented State-Noisy Markov Decision Process (SN-MDP), where there is a noise generating mechanism (either from the environment noise or from the adversary), and the theoretical properties (such as convergence and contraction) for corresponding (expected) Bellman operator and distributional Bellman operator were proved. The theoretical analysis was done for both tabular and linear funcion approximation settings. Especially in function approximation setting, authors characterized the robustness blessing of distributional RL based on histogram distributional loss and analyzed how the noise factor affects TD learning by using influence function that utilizes the perburbation method. Empirical analysis was done for DQN and QRDQN by varying noise standard deviations and the position of noise (state/successor state or both), which aims to support the authors' intuition coming from their theorems.