Online Robust Reinforcement Learning with Model Uncertainty

1 Introduction . Existing studies on Markov decision process ( MDP ) and reinforcement learning ( RL ) [ Sutton and Barto , 2018 ] mostly rely on the crucial assumption that the environment on which a learned policy will be deployed is the same one that was used to generate the policy , which is often violated in practice – e.g. , the simulator may be different from the true environment , and the MDP may evolve over time . Due to such model deviation , the actual performance of the learned policy can significantly degrade . To address this problem , the framework of robust MDP was formulated in [ Bagnell et al. , 2001 , Nilim and El Ghaoui , 2004 , Iyengar , 2005 ] , where the transition kernel of the MDP is not fixed and lies in an uncertainty set , and the goal is to learn a policy that performs well under the worst-case MDP in the uncertainty set . In [ Bagnell et al. , 2001 , Nilim and El Ghaoui , 2004 , Iyengar , 2005 ] , it was assumed that the uncertainty set is known beforehand , i.e. , model-based approach , and dynamic programming can be used to find the optimal robust policy . The model-based approach , however , requires a model of the uncertainty set known beforehand , and needs a large memory to store the model when the state and action spaces are large , which make it less applicable for many practical scenarios . This motivates the study in this paper , model-free robust RL with model uncertainty , which is to learn a robust policy using a single sample trajectory from a misspecified MDP , e.g. , a simulator and a similar environment in which samples are easier to collect than in the target environment where the policy is going to be deployed . The major challenge lies in that the transition kernel of the misspecified MDP is not given beforehand , and thus , the uncertainty 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . set and the optimal robust policy need to be learned simultaneously using sequentially observed data from the misspecified MDP . Moreover , robust RL learns the value function of the worst-case MDP in the uncertainty set which is different from the misspecified MDP that generates samples . This is similar to the off-policy learning , which we refer to as the `` off-transition-kernel '' setting . Therefore , the learning may be unstable and could diverge especially when function approximation is used [ Baird , 1995 ] . In this paper , we develop a model-free approach for robust RL with model uncertainty . Our major contributions in this paper are summarized as follows . • Motivated by empirical studies of adversarial training in RL [ Huang et al. , 2017 , Kos and Song , 2017 , Lin et al. , 2017 , Pattanaik et al. , 2018 , Mandlekar et al. , 2017 ] and theR-contamination model in robust detection ( called -contamination model in [ Huber , 1965 ] ) , we design the uncertainty set using the R-contamination model ( see ( 4 ) for the details ) . We then develop an approach to estimate the unknown uncertainty set using only the current sample , which does not incur any additional memory cost . Unlike the approach in [ Roy et al. , 2017 ] , where the uncertainty set is relaxed to one not depending on the misspecified MDP that generates samples so that an online algorithm can be constructed , our approach does not need to relax the uncertainty set . • We develop a robust Q-learning algorithm for the tabular case , which can be implemented in an online and incremental fashion , and has the same memory cost as the vanilla Q-learning algorithm . We show that our robust Q-learning algorithm converges asymptotically , and further characterize its finite-time error bound . Unlike the results in [ Roy et al. , 2017 ] where a stringent condition on the discount factor ( which is due to the relaxation of the uncertainty set , and prevents the use of a discount factor close to 1 in practice ) is needed to guarantee the convergence , our algorithm converges without the need of such condition . Furthermore , our robust Q-learning algorithm converges as fast as the vanilla Q-learning algorithm [ Li et al. , 2020 ] ( within a constant factor ) , while being robust to model uncertainty . • We generalize our approach to the case with function approximation ( for large state/action space ) . We investigate the robust policy evaluation problem , i.e. , evaluate a given policy under the worstcase MDP in the uncertainty set . As mentioned before , the robust RL problem is essentially `` off-transition-kernel '' , and therefore non-robust methods with function approximation may diverge [ Baird , 1995 ] ( also see our experiments ) . We develop a novel extension of the gradient TD ( GTD ) method [ Maei et al. , 2010 , Maei , 2011 , Sutton et al. , 2008 ] to robust RL . Our approach introduces a novel smoothed robust Bellman operator to construct the smoothed mean-squared projected robust Bellman error ( MSPRBE ) . Using our uncertainty set design and online sample-based estimation , we develop a two time-scale robust TDC algorithm . We further characterize its convergence and finite-time error bound . • We conduct numerical experiments to validate the robustness of our approach . In our experiments , our robust Q-learning algorithm achieves a much higher reward than the vanilla Q-learning algorithm when being trained on a misspecified MDP ; and our robust TDC algorithm converges much faster than the vanilla TDC algorithm , and the vanilla TDC algorithm may even diverge . 1.1 Related Work . Model-Based Robust MDP . The framework of robust MDP was investigated in [ Iyengar , 2005 , Nilim and El Ghaoui , 2004 , Bagnell et al. , 2001 , Satia and Lave Jr , 1973 , Wiesemann et al. , 2013 ] , where the transition kernel is assumed to be in some uncertainty set , and the problem can be solved by dynamic programming . This approach was further extended to the case with function approximation in [ Tamar et al. , 2014 ] . However , these studies are model-based , which assume beforehand knowledge of the uncertainty set . In this paper , we investigate the model-free setting , where the uncertainty set is a set of MDPs centered around some unknown Markov transition kernel from which a single sample trajectory can be sequentially observed . Adversarial Robust RL . It was shown in [ Iyengar , 2005 ] that the robust MDP problem is equivalent to a zero-sum game between the agent and the nature . Motivated by this fact , the adversarial training approach , where an adversary perturbs the state transition , was studied in [ Vinitsky et al. , 2020 , Pinto et al. , 2017 , Abdullah et al. , 2019 , Hou et al. , 2020 , Rajeswaran et al. , 2017 , Atkeson and Morimoto , 2003 , Morimoto and Doya , 2005 ] . This method relies on a simulator , where the state transition can be modified in an arbitrary way . Another approach is to modify the current state through adversarial samples , which is more heuristic , e.g. , [ Huang et al. , 2017 , Kos and Song , 2017 , Lin et al. , 2017 , Pattanaik et al. , 2018 , Mandlekar et al. , 2017 ] . Despite the empirical success of these approaches , theoretical performance guarantees , e.g. , convergence to the optimal robust policy and convergence rate , are yet to be established . The main difference lies in that during the training , our approach does not need to manipulate the state transition of the MDP . More importantly , we develop the asymptotic convergence to the optimal robust policy and further characterize the finite-time error bound . In [ Lim et al. , 2013 ] , the scenario where some unknown parts of the state space can have arbitrary transitions while other parts are purely stochastic was studied . Adaptive algorithm to adversarial behavior was designed , and its regret bound is shown to be similar to the purely stochastic case . In [ Zhang et al. , 2020a ] , the robust adversarial RL problem for the special linear quadratic case was investigated . Model-free Robust RL . In [ Roy et al. , 2017 , Badrinath and Kalathil , 2021 ] model-free RL with model uncertainty was studied , where in order to construct an algorithm that can be implemented in an online and incremental fashion , the uncertainty set was firstly relaxed by dropping the dependency on the misspecified MDP that generates the samples ( centroid of the uncertainty set ) . Such a relaxation is pessimistic since the relaxed uncertainty set is not centered at the misspecified MDP anymore ( which is usually similar to the target MDP ) , making the robustness to the relaxed uncertainty set not well-justified . Such a relaxation will further incur a stringent condition on the discounted factor to guarantee the convergence , which prevents the use of a discount factor close to 1 in practice . Moreover , only asymptotic convergence was established in [ Roy et al. , 2017 ] . In this paper , we do not relax the uncertainty set , and instead propose an online approach to estimate it . Our algorithms converge without the need of the condition on the discount factor . We also provides finite-time error bounds for our algorithms . The multi-agent RL robust to reward uncertainty was investigated in [ Zhang et al. , 2020b ] , where the reward uncertainty set is known , but the transition kernel is fixed . Finite-time Error Bound for RL Algorithms . For the tubular case , Q-learning has been studied intensively , e.g. , in [ Even-Dar et al. , 2003 , Beck and Srikant , 2012 , Qu and Wierman , 2020 , Li et al. , 2020 , Wainwright , 2019 , Li et al. , 2021 ] . TD with function approximation were studied in [ Dalal , Gal and Szörényi , Balázs and Thoppe , Gugan and Mannor , Shie , 2018 , Bhandari et al. , 2018 , Srikant and Ying , 2019 , Cai et al. , 2019 , Sun et al. , 2020 ] . Q-learning and SARSA with linear function approximation were investigated in [ Zou et al. , 2019 , Chen et al. , 2019 ] . The finite-time error bounds for the gradient TD algorithms [ Maei et al. , 2010 , Sutton et al. , 2009 , Maei et al. , 2010 ] were further developed recently in [ Dalal et al. , 2018 , Liu et al. , 2015 , Gupta et al. , 2019 , Xu et al. , 2019 , Dalal et al. , 2020 , Kaledin et al. , 2020 , Ma et al. , 2020 , Wang and Zou , 2020 , Ma et al. , 2021 , Doan , 2021 ] . There are also finite-time error bounds on the policy gradient methods and actor critic methods , e.g. , [ Wang et al. , 2020 , Yang et al. , 2019 , Kumar et al. , 2019 , Qiu et al. , 2019 , Wu et al. , 2020 , Cen et al. , 2020 , Bhandari and Russo , 2019 , Agarwal et al. , 2021 , Mei et al. , 2020 ] . We note that these studies are for the non-robust RL algorithms , and in this paper , we design robust RL algorithms , and characterize their finite-time error bounds .

Summary: This paper considers the model-free robust reinforcement learning, where the uncertainty set is assumed to be unknown. The authors propose a robust Q-learning algorithm for tabular settings and a robust TDC algorithm for function approximation settings. The authors theoretically analyze the converge properties for these two algorithms. Numerical experiments demonstrate the robustness of the proposed methods.