Online Robust Reinforcement Learning with Model Uncertainty

1 Introduction . Existing studies on Markov decision process ( MDP ) and reinforcement learning ( RL ) [ Sutton and Barto , 2018 ] mostly rely on the crucial assumption that the environment on which a learned policy will be deployed is the same one that was used to generate the policy , which is often violated in practice – e.g. , the simulator may be different from the true environment , and the MDP may evolve over time . Due to such model deviation , the actual performance of the learned policy can significantly degrade . To address this problem , the framework of robust MDP was formulated in [ Bagnell et al. , 2001 , Nilim and El Ghaoui , 2004 , Iyengar , 2005 ] , where the transition kernel of the MDP is not fixed and lies in an uncertainty set , and the goal is to learn a policy that performs well under the worst-case MDP in the uncertainty set . In [ Bagnell et al. , 2001 , Nilim and El Ghaoui , 2004 , Iyengar , 2005 ] , it was assumed that the uncertainty set is known beforehand , i.e. , model-based approach , and dynamic programming can be used to find the optimal robust policy . The model-based approach , however , requires a model of the uncertainty set known beforehand , and needs a large memory to store the model when the state and action spaces are large , which make it less applicable for many practical scenarios . This motivates the study in this paper , model-free robust RL with model uncertainty , which is to learn a robust policy using a single sample trajectory from a misspecified MDP , e.g. , a simulator and a similar environment in which samples are easier to collect than in the target environment where the policy is going to be deployed . The major challenge lies in that the transition kernel of the misspecified MDP is not given beforehand , and thus , the uncertainty 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . set and the optimal robust policy need to be learned simultaneously using sequentially observed data from the misspecified MDP . Moreover , robust RL learns the value function of the worst-case MDP in the uncertainty set which is different from the misspecified MDP that generates samples . This is similar to the off-policy learning , which we refer to as the `` off-transition-kernel '' setting . Therefore , the learning may be unstable and could diverge especially when function approximation is used [ Baird , 1995 ] . In this paper , we develop a model-free approach for robust RL with model uncertainty . Our major contributions in this paper are summarized as follows . • Motivated by empirical studies of adversarial training in RL [ Huang et al. , 2017 , Kos and Song , 2017 , Lin et al. , 2017 , Pattanaik et al. , 2018 , Mandlekar et al. , 2017 ] and theR-contamination model in robust detection ( called -contamination model in [ Huber , 1965 ] ) , we design the uncertainty set using the R-contamination model ( see ( 4 ) for the details ) . We then develop an approach to estimate the unknown uncertainty set using only the current sample , which does not incur any additional memory cost . Unlike the approach in [ Roy et al. , 2017 ] , where the uncertainty set is relaxed to one not depending on the misspecified MDP that generates samples so that an online algorithm can be constructed , our approach does not need to relax the uncertainty set . • We develop a robust Q-learning algorithm for the tabular case , which can be implemented in an online and incremental fashion , and has the same memory cost as the vanilla Q-learning algorithm . We show that our robust Q-learning algorithm converges asymptotically , and further characterize its finite-time error bound . Unlike the results in [ Roy et al. , 2017 ] where a stringent condition on the discount factor ( which is due to the relaxation of the uncertainty set , and prevents the use of a discount factor close to 1 in practice ) is needed to guarantee the convergence , our algorithm converges without the need of such condition . Furthermore , our robust Q-learning algorithm converges as fast as the vanilla Q-learning algorithm [ Li et al. , 2020 ] ( within a constant factor ) , while being robust to model uncertainty . • We generalize our approach to the case with function approximation ( for large state/action space ) . We investigate the robust policy evaluation problem , i.e. , evaluate a given policy under the worstcase MDP in the uncertainty set . As mentioned before , the robust RL problem is essentially `` off-transition-kernel '' , and therefore non-robust methods with function approximation may diverge [ Baird , 1995 ] ( also see our experiments ) . We develop a novel extension of the gradient TD ( GTD ) method [ Maei et al. , 2010 , Maei , 2011 , Sutton et al. , 2008 ] to robust RL . Our approach introduces a novel smoothed robust Bellman operator to construct the smoothed mean-squared projected robust Bellman error ( MSPRBE ) . Using our uncertainty set design and online sample-based estimation , we develop a two time-scale robust TDC algorithm . We further characterize its convergence and finite-time error bound . • We conduct numerical experiments to validate the robustness of our approach . In our experiments , our robust Q-learning algorithm achieves a much higher reward than the vanilla Q-learning algorithm when being trained on a misspecified MDP ; and our robust TDC algorithm converges much faster than the vanilla TDC algorithm , and the vanilla TDC algorithm may even diverge . 1.1 Related Work . Model-Based Robust MDP . The framework of robust MDP was investigated in [ Iyengar , 2005 , Nilim and El Ghaoui , 2004 , Bagnell et al. , 2001 , Satia and Lave Jr , 1973 , Wiesemann et al. , 2013 ] , where the transition kernel is assumed to be in some uncertainty set , and the problem can be solved by dynamic programming . This approach was further extended to the case with function approximation in [ Tamar et al. , 2014 ] . However , these studies are model-based , which assume beforehand knowledge of the uncertainty set . In this paper , we investigate the model-free setting , where the uncertainty set is a set of MDPs centered around some unknown Markov transition kernel from which a single sample trajectory can be sequentially observed . Adversarial Robust RL . It was shown in [ Iyengar , 2005 ] that the robust MDP problem is equivalent to a zero-sum game between the agent and the nature . Motivated by this fact , the adversarial training approach , where an adversary perturbs the state transition , was studied in [ Vinitsky et al. , 2020 , Pinto et al. , 2017 , Abdullah et al. , 2019 , Hou et al. , 2020 , Rajeswaran et al. , 2017 , Atkeson and Morimoto , 2003 , Morimoto and Doya , 2005 ] . This method relies on a simulator , where the state transition can be modified in an arbitrary way . Another approach is to modify the current state through adversarial samples , which is more heuristic , e.g. , [ Huang et al. , 2017 , Kos and Song , 2017 , Lin et al. , 2017 , Pattanaik et al. , 2018 , Mandlekar et al. , 2017 ] . Despite the empirical success of these approaches , theoretical performance guarantees , e.g. , convergence to the optimal robust policy and convergence rate , are yet to be established . The main difference lies in that during the training , our approach does not need to manipulate the state transition of the MDP . More importantly , we develop the asymptotic convergence to the optimal robust policy and further characterize the finite-time error bound . In [ Lim et al. , 2013 ] , the scenario where some unknown parts of the state space can have arbitrary transitions while other parts are purely stochastic was studied . Adaptive algorithm to adversarial behavior was designed , and its regret bound is shown to be similar to the purely stochastic case . In [ Zhang et al. , 2020a ] , the robust adversarial RL problem for the special linear quadratic case was investigated . Model-free Robust RL . In [ Roy et al. , 2017 , Badrinath and Kalathil , 2021 ] model-free RL with model uncertainty was studied , where in order to construct an algorithm that can be implemented in an online and incremental fashion , the uncertainty set was firstly relaxed by dropping the dependency on the misspecified MDP that generates the samples ( centroid of the uncertainty set ) . Such a relaxation is pessimistic since the relaxed uncertainty set is not centered at the misspecified MDP anymore ( which is usually similar to the target MDP ) , making the robustness to the relaxed uncertainty set not well-justified . Such a relaxation will further incur a stringent condition on the discounted factor to guarantee the convergence , which prevents the use of a discount factor close to 1 in practice . Moreover , only asymptotic convergence was established in [ Roy et al. , 2017 ] . In this paper , we do not relax the uncertainty set , and instead propose an online approach to estimate it . Our algorithms converge without the need of the condition on the discount factor . We also provides finite-time error bounds for our algorithms . The multi-agent RL robust to reward uncertainty was investigated in [ Zhang et al. , 2020b ] , where the reward uncertainty set is known , but the transition kernel is fixed . Finite-time Error Bound for RL Algorithms . For the tubular case , Q-learning has been studied intensively , e.g. , in [ Even-Dar et al. , 2003 , Beck and Srikant , 2012 , Qu and Wierman , 2020 , Li et al. , 2020 , Wainwright , 2019 , Li et al. , 2021 ] . TD with function approximation were studied in [ Dalal , Gal and Szörényi , Balázs and Thoppe , Gugan and Mannor , Shie , 2018 , Bhandari et al. , 2018 , Srikant and Ying , 2019 , Cai et al. , 2019 , Sun et al. , 2020 ] . Q-learning and SARSA with linear function approximation were investigated in [ Zou et al. , 2019 , Chen et al. , 2019 ] . The finite-time error bounds for the gradient TD algorithms [ Maei et al. , 2010 , Sutton et al. , 2009 , Maei et al. , 2010 ] were further developed recently in [ Dalal et al. , 2018 , Liu et al. , 2015 , Gupta et al. , 2019 , Xu et al. , 2019 , Dalal et al. , 2020 , Kaledin et al. , 2020 , Ma et al. , 2020 , Wang and Zou , 2020 , Ma et al. , 2021 , Doan , 2021 ] . There are also finite-time error bounds on the policy gradient methods and actor critic methods , e.g. , [ Wang et al. , 2020 , Yang et al. , 2019 , Kumar et al. , 2019 , Qiu et al. , 2019 , Wu et al. , 2020 , Cen et al. , 2020 , Bhandari and Russo , 2019 , Agarwal et al. , 2021 , Mei et al. , 2020 ] . We note that these studies are for the non-robust RL algorithms , and in this paper , we design robust RL algorithms , and characterize their finite-time error bounds .

Summary: This paper focuses on model free robust Q-learning that can be implemented online. The authors estimate the unknown uncertainty set (R-contamination model) with current samples. Under the perfect duality condition they obtain the bellman operator which optimize their robust objectives. They propose two different algorithms: one for tabular case and another one with function approximation for problems with large state and action spaces. The robust algorithms presented in the paper converges as fast as their non-robust counterparts. They also provide empirical analysis to show the usefulness of their proposed methods.