Offline Reinforcement Learning with Resource Constrained Online Deployment

1 INTRODUCTION . There have been many recent successes in the field of Reinforcement Learning ( Mnih et al. , 2013 ; Lillicrap et al. , 2015 ; Mnih et al. , 2016 ; Silver et al. , 2016 ; Henderson et al. , 2018 ) . In the online RL setting , an agent takes actions , observes the outcome from the environment , and updates its policy based on the outcome . This repeated access to the environment is not feasible in practical applications ; it may be unsafe to interact with the actual environment , and a high-fidelity simulator may be costly to build . Instead , offline RL , consumes fixed training data which consist of recorded interactions between one ( or more ) agent ( s ) and the environment to train a policy ( Levine et al. , 2020 ) . An agent with the trained policy is then deployed in the environment without further evaluation or modification . Notice that in offline RL , the deployed agent must consume data in the same format ( for example , having the same features ) as in the training data . This is a crippling restriction in many large-scale applications , where , due to some combination of resource/system constraints , all of the features used for training can not be observed ( or misspecified ) by the agent during online operation . In this work , we lay the foundations for studying this Resource-Constrained setting for offline RL . We then provide an algorithm that improves performance by transferring information from the full-featured offline training set to the deployed agent ’ s policy acting on limited features . We first illustrate a few practical cases where resource-constrained settings emerge . System Latency A deployed agent is often constrained by how much time it has to process the state of the environment and make a decision . For example , in a customer-facing web application , the customer will start to lose interest within a fraction of a second . Given this constraint , the agent may not be able to fully process more than a few measurements from the customer before making a decision . This is in contrast to the process of recording the training data for offline RL , where one may take sufficient time to generate an abundance of features by post-processing high-dimensional measurements . Power Constraints Consider a situation where an RL agent is being used in deep space probes or nano-satellites ( ( Deshmukh et al. , 2018 ) ) . In this case an RL agent is trained on Earth with rich features and a large amount of sensory information . But when the agent is deployed and being used on these probes , the number of sensors is limited by power and space constraints . Similarly , consider a robot deployed in a real world environment . The limited compute power of the robot prevents it from using powerful feature extractors while making a decision . However , such powerful feature extractors can be used during the offline training of the robot ( Fig 1a ) . In the resource-constrained setting , one can simply ignore the offline features and only train the offline agent with the online features that are available during deployment . This strategy has the drawback of not utilizing all of the information available during training and can lead to a sub-optimal policy . To confirm this , we performed the following simple experiment . We consider an offline RL dataset for the OpenAI gym MuJoCo HalfCheetah-v2 environment and simulate the resourceconstrained setting by removing a fixed set of randomly selected features during deployment ( see Sections 5.1.1 , C.1 for more details ) . We train an offline RL algorithm , TD3+BC ( Fujimoto & Gu , 2021 ) using only the online features and collect online data in the environment using the trained policy . We repeat this assuming all features available during deployment , train a TD3+BC agent using the same offline dataset with all features , and collect online data in the environment . We plot the histogram of rewards in the two datasets in Fig 1b . We observe that the agent trained only with online features obtains much smaller reward than the agent trained with offline features . Traditionally , scenarios where the observability of the state of the system is limited are studied under the Partially Observable Markov Decision Process ( POMDP ) setting by assuming a belief over the observations ( Åström , 1965 ) . In contrast , we have an offline dataset ( which records rich but not necessarily full state transitions ) along with partially obscured ( with respect to the offline dataset ) observations online . Our goal is to leverage the offline dataset to reduce the performance gap caused by the introduction of resource constraints . Towards this , we advocate using a teacherstudent transfer algorithm . Our main contributions are summarized below : • We identify a key challenge in offline RL : in the resource-constrained setting , datasets with rich features can not be effectively utilized when only a limited number of features are observable during online operation . • We propose the transfer approach that trains an agent to efficiently leverage the offline dataset while only observing the limited features during deployment . • We evaluate our approach on a diverse set of tasks showing the applicability of the transfer algorithm . We also highlight that when the behavior policy used by the data-collecting agent is trained using a limited number of features , the quality of the dataset suffers . We propose a data collection procedure ( RC-D4RL ) to simulate this effect . 2 RESOURCE-CONSTRAINED ONLINE SYSTEMS . In the standard RL framework , we consider a Markov Decision Process ( MDP ) defined by the tuple ( S , A , R , P , γ ) where S is the state space , A is the action space , R : S × A → R is the reward function , P : S × A → ∆ ( S ) is the transition function , ∆ ( S ) denotes the set of all probability distributions over S , and γ ∈ ( 0 , 1 ) is the discount factor . We consider the discounted infinite horizon MDP in this paper . We consider the continuous control setting and assume that both S and A are compact subsets of a real-valued vector space . The transition at time t , is given by the tuple ( st , at , R ( st , at ) , st+1 ) . Each policy π : S → ∆ ( A ) , has a value function Qπ : S × A → R that estimates the expected discounted reward for taking action a in state s and uses the policy π after that . The goal of the agent is to learn the policy π that maximizes the expected discounted reward Eπ [ ∑∞ t=0 γ tR ( st , at ) ] . In online RL , this problem is solved by interacting with the environment . In offline ( or batch ) RL ( Lange et al. , 2012 ) , instead of having access to the environment , the agent is provided with a finite dataset of trajectories or transitions denoted by D = { ( si , ai , ri , s′i ) } Ni=1 . The data is collected by one or many behavior policies that induce a distribution µ on the space of S × A . The goal of the agent is to learn a policy using the finite dataset to maximize the expected discounted reward when deployed in the environment . In the resource-constrained setting , the agent does not have access to the full state space or features during deployment . Instead , the agent can only observe from Ŝ ( another bounded subset of the realvalued vector space ) that is different from S. It is assumed that the space S is rich in information as compared to Ŝ . For example , Ŝ might have fewer dimensions , or some entries may include extra noise ( see Figure 1a ) . We will use online/limited features , to refer to observations from the online space Ŝ , offline/rich features to refer to observations from the offline space S. We assume that both online features and offline features are available in offline data . The goal of the agent is to use the offline data and train a policy π : Ŝ → ∆ ( A ) . The agent can use the offline features from S during training but is constrained to only use the online features from Ŝ while making a decision . A similar paradigm of Learning Under Privileged Information ( LUPI ) ( Vapnik et al. , 2015 ) has been studied under the supervised learning setting , where the privileged information is provided by a knowledgeable teacher . 3 RELATED WORK . Offline RL There has been an increasing interest in studying offline RL algorithms due to its practical advantages over online RL algorithms ( Agarwal et al. , 2020 ; Wu et al. , 2021 ; Chen et al. , 2021 ; Brandfonbrener et al. , 2021 ) . Offline RL algorithms typically suffer from overestimation of the value function as well as distribution shift between the offline data and on-policy data . Buckman et al . ( 2020 ) and Kumar et al . ( 2020 ) advocate a pessimistic approach to value function estimation to avoid over-estimation of rarely observed state-action pairs . To constrain the on-policy data to be closer to offline data , several techniques have been explored , such as restricting the actions inside the expectation in the evaluation step to be close to the actions observed in the dataset ( Fujimoto et al. , 2019 ) , adding a regularization term during policy evaluation or iteration ( Kostrikov et al. , 2021 ) , ( Wu et al. , 2019 ) , ( Guo et al. , 2020 ) , adding a constraint of the form MMD ( µ ( ·|s ) , π ( s ) ) ( Gretton et al. , 2012 ; Blanchard et al. , 2021 ; Deshmukh et al. , 2019 ) on the policy ( Kumar et al. , 2019 ) , using behavior cloning ( Fujimoto & Gu , 2021 ) , adding an entropy term in the value function estimation ( Wu et al. , 2019 ) , and model-based approaches that learn a pessimistic MDP Kidambi et al . ( 2020 ) . A thorough review of these techniques is presented in an excellent tutorial by Levine et al . ( 2020 ) . To the best of our knowledge , there is no existing work that addresses the resource-constrained offline RL setting where there is a mismatch between the offline features and online features . Knowledge Transfer Knowledge transfer/distillation is widely studied in various settings including vision , language , and RL domains ( Gou et al. , 2021 ; Wang & Yoon , 2021 ) . In RL , under the domain transfer setting ( Taylor & Stone , 2009 ; Liu et al. , 2016 ) , the teacher is trained on one domain/task and the student needs to perform on a different domain/task ( Konidaris & Barto , 2006 ; Perkins et al. , 1999 ; Torrey et al. , 2005 ; Gupta et al. , 2017 ) . Li et al . ( 2019 ) train a model so that features from different domains have similar embeddings , and Kamienny et al . ( 2020 ) perturb the feature using a random noise centered at the privileged information . An offline RL algorithm for domain transfer has been proposed by Cang et al . ( 2021 ) . Policy distillation is studied in the setting where the knowledge from a trained policy ( teacher ) is imparted to an untrained network ( student ) ( Rusu et al. , 2015 ; Czarnecki et al. , 2019 ) . This leads to several advantages such as model compression and the ability to learn from an ensemble of trained policies to improve performance ( Zhu et al. , 2020 ) . One distinguishing feature of the resource-constrained setting that differentiates it from other transfer settings is that the teacher has access to the privileged information and student needs to adapt from the data available without interactive learning . In most of the existing approaches , the difference between teacher and student was either the network size ( which is also present in our setting due to the difference in input features ) or the dynamics ( as in the domain transfer case ) . To the best of our knowledge , we are the first to study policy distillation in the offline RL framework under the resource-constrained setting . Another interesting line of work is called Sim2Real ( Lee et al. , 2021 ; Traoré et al. , 2019 ) . In these papers , they train a model using a simulator and then transfer the knowledge to real data . However , this work requires an accurate simulator which results in a fairly expert teacher model . However , in the offline RL setting , depending on the data quality , the teacher itself might be weak . Partially Observable MDP POMDP generalizes the MDP framework where the agent does not have access to the full features and only partially observes the state space ( Åström , 1965 ; Kaelbling et al. , 1998 ; Ortner et al. , 2012 ) . More recently , Rafailov et al . ( 2021 ) studied a model-based offline RL algorithm for image data under the POMDP setup . Our setting resembles this setup , but our agent also has access to the full privileged features in the offline dataset while training . This availability of the offline dataset with privileged information differentiates our setting and enables the student to inherit the knowledge from the rich space while only using the limited features during deployment .

Summary: The paper proposes an offline RL algorithm in the resource-constrained setting, where the offline dataset contains richer features than provided by online interactions. The authors propose a transfer learning objective, where a teacher policy is learned from the rich features, then a policy is learned from the limited features by additionally fitting to the actions chosen by the teacher policy. The authors compare the policy learned via this transfer objective to a baseline that does not do transfer learning on D4RL tasks.