Offline Reinforcement Learning with Resource Constrained Online Deployment
Authors: Jayanth Reddy Regatti
Aniket Anand Deshmukh
Young Hun Jung
Frank Cheng
Abhishek Gupta
Urun Dogan
Venue: ICLR 2022 Submitted (2022)
Keywords: Offline Reinforcement Learning
Reinforcement Learning
Transfer Learning
Knowledge Transfer
Resource Constraints

Abstract: Offline reinforcement learning is used to train policies in scenarios where real-time access to the environment is expensive or impossible.
As a natural consequence of these harsh conditions, an agent may lack the resources to fully observe the online environment before taking an action. We dub this situation the resource-constrained setting. This leads to situations where the offline dataset (available for training) can contain fully processed features (using powerful language models, image models, complex sensors, etc.) which are not available when actions are actually taken online.
This disconnect leads to an interesting and unexplored problem in offline RL: Is it possible to use a richly processed offline dataset to train a policy which has access to fewer features in the online environment? 
In this work, we introduce and formalize this novel resource-constrained problem setting. We highlight the performance gap between policies trained using the full offline dataset and policies trained using limited features. 
We address this performance gap with a policy transfer algorithm which first trains a teacher agent using the offline dataset where features are fully available, and then transfers this knowledge to a student agent that only uses the resource-constrained features. To better capture the challenge of this setting, we propose a data collection procedure: Resource Constrained-Datasets for RL (RC-D4RL). We evaluate our transfer algorithm on RC-D4RL and the popular D4RL benchmarks and observe consistent improvement over the baseline (TD3+BC without transfer).

Under review as a conference paper at ICLR 2022
OFFLINE RL WITH RESOURCE CONSTRAINED ONLINE DEPLOYMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning is used train policies in scenarios where real- time access to the environment is expensive or impossible. As natural conse- quenee of these harsh conditions, an agent may lack the resources to fully observe the online environment before taking an action. We dub this situation the resource- constrained setting. This leads to situations where the offine dataset (available for training) can contain fully processed features (using powerful language mod- els, image models, complex sensors, etc.) which are not available when actions are actually taken online. This disconnect leads to an interesting and unexplored problem in offline RL: Is it possible to use a richly processed offline dataset to train policy which has access to fewer features in the online environment? In this work, we introduce and formalize this novel pnouucceconssrrinee problem setting. We highlight the performance gap between policies trained using the full offline dataset and policies trained using limited features. We ad advocate the use of transfer learning address this performance gap by first training a teacher agent using the offline dataset where features are fully available, and then transfering this knowledge to a student agent that only uses the resource-constrale evaluate the proposed approach on three diverse set of tasks: MuJoCo (continu- ous control), Atari 2600 (discrete control) and a real life Ads dataset. Our analysis shows the proposed approach improves over the considered basclines and unlocks interesting insights. To better capture the challenge of this setting, we also propose data collection procedure: Resource monstrinedd-Datasee for RL (RC-D4RL).
features. We
INTRODUCTION
There have been many recent successes in the field of Reinforcement Learning (Mnih et al., 2013; Lillicrap et al., 2015; Mnih et al., 2016; Silver et al., 2016; Henderson et al., 2018). In the online RL setting, an agent takes actions, observes the outcome from the environment, and updates its policy based on the outcome. This repeated access to the environment is not feasible in practical applica- tions; it may be unsafe to interact with the actual environment, and a high-fidelit simulator may be costly to build. Instead, offline RI., consumes fixed training data which consist of recorded inter- actions between one (or more) agent(s) and the environment to train a policy (Levine et al., 2020). An agent with the trained policy then deployed in the environment without further evaluation or modification. Notice that in offline RL, the deployed agent must consume data in the same format (for example,having the same features) in the training data. This is crippling restriction in many large-scale applications, where, due to some combination of resource/system constraints, all of the features used for training cannot be observed (or misspecified) by the agent during online opera- tion. In this work, we lay the foundations for studying this RL. We then provide an algorithm that improves performance by transferring information from the full-featured offline training set to the deployed agent's 's policy acting on limited features. We first illustrate a few practical cases where resource-constrainee settings emerge.
Construind setting for offline
System Latency A deployed agent is often constrained by how much time it has to process the state of the environment and make a decision. For example, in a austomer-facing web application, the customer will start to lose interest within fraction of second. Given this constraint, the agent may not be able to fully process more than a few measurements from the customer before making a decision. This is in contrast to the process of recording the training data for offline RL, where one

---

Under review as a conference paper at ICLR 2022
offline
online
Raw Inputs
1500
1000
500
Light-wight festure
Powerful Festure
0
Rule based features
Human Annotation
(b) Histogram of rewards in online data col- lected by agents trained with online versus of- fline features
Online Features
Offline Features
(a) Example of system constraints
Figure 1: Illustrations of the resource-constrained setting
may take sufficient time to generate an abundance features by post- post-processing high- digh--imeesional measurements.
Power Constraints Consider a situation where an RI. agent is being used in deep space probes or nano-satellites ((Deshmukh al., 2018)). In this case an RL agent is trained on Earth with rich features and large amount of sensory information. But when the agent is deployed and being used on these probes, the number of sensors is limited by power and space constraints. Similarly, consider robot deployed in real world environment. The limited compute power of the robot prevents it from using powerful feature extractors while making decision. However, such powerful feature extractors can be used during the offline training of the robot (Fig la).
In the resource-constrained setting, one can simply ignore the offine features and only train the of- fline agent with the online features that are available during deployment. This strategy has the draw- back of not utilizing all of the information available during training and can lead sub-optimal policy. To confirm this, performed the following simple experiment. We consider offline RL dataset for the OpenAl gym MuJoCo HalfCheetah-v2 environment and simulate the resource- constrained setting by removing fixed set of randomly selected features during deployment (see Sections 5.1.1, C.I for more details). We train an offline RL algorithm, TD3+BC (Fujimoto Gu, 2021) using only the online features and collect online data in the environment using the trained policy. We repeat this assuming all features available during deployment, train TD3+BC agent using the same offline dataset with all features, and collect online data in the environment. We plot the histogram of rewards in the two datasets in Fig lb. We observe that the agent trained only with online features obtains much smaller reward than the agent trained with offline features.
Traditionally, scenarios where the observability of the state of the system is limited are studied under the Partially Observable Markov Decision Process (POMDP) setting by assuming a belief over the observations (Aström, 1965). In contrast, we have an offline dataset (which records rich not necessarily full state transitions) along with partially obscured (with respect to the offline dataset) observations online. Our goal is to leverage the offline dataset to reduce the performance gap caused by the introduction of resource constraints. Towards this, we advocate using teacher- student transfer algorithm. Our main contributions summarized below:
We identify a key challenge in offline RL: in the resource-constrained setting, datasets with rich features cannot be effectively utilized when only limited number of features observable during online operation.
We propose the transfer approach that trains an agent to efficiently leverage the offline dataset while only observing the limited features during deployment.
We evaluate our approach on a diverse set of tasks showing the applicability of the transfer algorithm. We also highlight that when the behavior policy used by the data-collecting agent is trained using a limited number of features, the quality of the dataset suffers. We propose data collection procedure (RC-D4RL) to simulate this effect.
2

---

Under review as a conference paper at ICLR 2022
2 RESOURCE- RESOURCE-CONSTRAINE ONLINE SYSTEMS
In the standard RL framework, we consider Markov Decision Process (MDP) defined by the tuple (S, A. R, P.?) where S is the state space, A is the action space, S x A - R is the reward function, P S x A ((S) is the transition function, A(S) denotes the set of all probability distributions over S. and E (0, is the discount factor. We consider the discounted infinite horizon MDP in this paper. We consider the continuous control setting and assume that both S and A are compact subsets of a real-valued vector space. The transition at time t. is given by the tuple (st, ae, R(st, ae); St+1). Each policy т : S - A(A), has value function Q S A + R that estimates the expected discounted reward for taking action a in state s and uses the policy after that. The goal of the agent is to learn the policy л that maximizes the expected discounted reward E,, 1o R(s,, a.)l. In online RL, this problem is solved by interacting with the environment.
In offline (or batch) RL (Lange et al., 2012), instead of having access to the environment, the agent is provided with finite dataset of trajectories or transitions denoted by D = {(ss, 0i, ri b)) The data is collected by one or many behavior policies that induce distribution μ on the space of S x A. The goal of the agent is to learn policy using the finite dataset to maximize the expected discounted reward when deployed in the environment.
In the resource-comstrainee setting, the agent does not have access to the full state space or features during deployment. Instead, the agent can only observe from S (another bounded subset of the real- valued vector space) that is different from S. It is assumed that the space S is rich in information as compared to S. For example, S might have fewer dimensions, or some entries may include extra noise (see Figure la). We will use online/limited features, to refer to observations from the online space S. offline/rrchhfaatue online features and offline features are available in offline data.
to refer to observations from the offline space S. We assume that both
The goal of the agent is to use the offline data and train a policy क : S + ((A). The agent can use the offline features from S during training but is constrained to only use the online features from S while making a decision. A similar paradigm of Learning Under Privileged Information (LUPI) (Vapnik et al., 2015) has been studied under the supervised learning setting, where the privileged information provided by a knowledgeable teacher.
3 RELATED WORK
Offiine RL There has been an increasing interest in studying offline RL algorithms due to its practi- cal advantages over online RL algorithms (Agarwal et al., 2020; Wu et al., 2021; Chen et al., 2021; Brandfonbrener et al., 2021). Offline RL algorithms typically suffer from overestimation of the value function well as distribution shift between the offline data and on-policy data. Buckman et al. (2020) and Kumar et al. (2020) advocate pessimistic approach to value function estimation to avoid over-estimation of rarely observed aattenaction pairs. To constrain the on-policy data to be closer to offline data, several techniques have been explored, such as restricting the actions inside the expectation in the evaluation step be close the actions observed in the dataset (Fujimoto al., 2019), adding a regularization term during policy evaluation or iteration (Kostrikov et al., 2021), (Wu et al., 20u al., 2020), adding constraint of the form MMD(u(-Is), (s)) (Gretton et al., 2012; Blanchard et al., 2021; Deshmukh et al., 2019) on the policy (Kumar et al., 2019), using behavior cloning (Fujimoto & Gu, 2021), adding an entropy term in the value function estimation (Wu et al., 2019), and model-based approaches that learn pessimistic MDP Kidambi et al. (2020). A thorough review of these techniques is presented in an excellent tutorial by Levine et al. (2020). To the best of our knowledge, there is no existing work that addresses the RI. setting where there is mismatch between the offline features and online features.
offline
Knowledge Transfer Knowledge transfer/distillattinn is widely studied in various settings including vision, language, and RL domains (Gou et al., 2021; Wang & Yoon, 2021). In RL, under the domain transfer setting (Taylor& & Stone, 2009; Liu et al., 2016), the teacher is trained and the student needs to perform on a different domain/task (Konidaris & Barto, 2006; Perkins et al., 1999; Torrey al., 2005; Gupta et al., 2017). Li et al. (2019) train model so that features from different domains have similar embeddings, and Kamienny et al. (2020) perturb the feature using a random noise centered at the privileged information. An offline RL algorithm for domain transfer
one domain/task
3

---

Under review as a conference paper at ICLR 2022
has been proposed by Cang et al. (2021). Policy distillation is studied in the setting where the knowledge from a trained policy (teacher) is imparted to an untrained network (student) (Rusu et al., 2015; Czarnecki et al., 2019). This leads to several advantages such model compression and the ability to learn from an ensemble of trained policies to improve performance (Zhu et al., 2020).
One distinguishing feature of the resource-constrainee setting that differentiates it from other trans- fer settings is that the teacher has access to the privileged information and student needs to adapt from the data available without interactive learning. In most of the existing approaches, the differ- ence between teacher and student was either the network size (which is also present in our setting due to the difference in input features) or the dynamics (as in the domain transfer case). To the best of our knowledge, we are the first to study policy distillation in the offline RL framework under the essurcce-onssrainee setting. Another interesting line of work called Sim2Real (lee et al,, 2021; Traoré et al., 2019). In these papers, they train model using a simulator and then transfer the knowledge to real data. However, this work requires an accurate simulator which results in a fairly expert teacher model. However, in the offline RL setting, depending the data quality, the teacher itself might be weak.
Partially Observable MDP POMDP generalizes the MDP framework where the agent does not have access to the full features and only partially observes the state space (Aström, 1965; Kaelbling et al., 1998; Ortner et al., 2012). More recently. Rafailov et al. (2021) studied model-based offline RL. algorithm for image data under the POMDP setup. Our setting resembles this setup, but our agent also has access to the full privileged features in the offline dataset while training. This availability of the offline dataset with privileged information differentiates our setting and enables the student to inherit the knowledge from the rich space while only using the limited features during deployment.
PROPOSED ALGORITHM
In this section, we advocate the use of transfer algorithms to address the resource constrained setting. In particular, we discuss teacher-student knowledge transfer models.
4.1 CONTINUOUS CONTROL
We motivate the proposed algorithm by first providing a general approach to tackle the resource- constrained setting using policy distillation (Rusu et al., 2015; Czarnecki et al., 2019). This is modeling choice; alternative types of knowledge transfer, such as transfer of the value function's knowledge, are also possible. (Czarnecki et al., 2019).
We first train a teacher network using the full offline dataset. Let us denote the policy output by the teacher network as Tggesmer : S + ((A). The student network is denoted by a $ - ((A). The knowledge transfer is performed by using the pre-trained teacher network to compute regularization term that is added to the objective (policy iteration):
T + arg max E (6, - 3M (8), π Tagaachur (+)]]
Here, M can be any divergence metric on the policy distributions. We can compute the terms n. (8) and Tgloousies (s) simultaneously since we assume that the offline dataset contains the offline features and the corresponding online features š. If the teacher and learned policy functions are determin- istic, the divergence function can be the squared loss, and if they are stochastic, the divergence can be KL divergence, MMD (Maximum Mean Discrepancy), Wasserstein divergence, etc. This regu- larization term encourages the learnt policy to behave similarly to the teacher policy. An illustration of the proposed setup is given in Figure 6 in the Appendix.
Proposed Algorithm We now propose our transfer algorithm that adopts TD3+BC (Fujimoto & Gu, 2021) to the cesstrce-onsttrained setting. We add an additional regularization term during the policy iteration step, that tries to keep actions predicted by the trained policy close to the actions taken by the teacher policy. This knowledge can be imparted to the student policy through regularization.
N
arg max &
1
Σ
AQo, (o)))) B1 T ( -a) a) B2 Tatnscher (sx) To((i))
(2)
N
TD$+BC
Transfer
4

---

Under review as a conference paper at ICLR 2022
We weigh the two terms with weights 31. B2. The transfer term is beneficial in learning because the teacher (trained using offline RL algorithm on full features) often predicts a better action than those available in the dataset (depending
the quality of the dataset) as observed in earlier offline RL works (Kumar et al., 2020). To keep all terms in equation 3 at comparable magnitudes, we add constraint that 31 8/ 1. If f1 and B2 O, the proposed algorithm is equivalent the TD3+BC algorithm. Algorithm describes the complete steps.
4.2 DISCRETE CONTROT.
For discrete control tasks, Deep Q learning is a widely followed approach (Mnih et al., 2015) and more recently distributional versions of deep Q learning considered state of the (Dabney al., 2018). For the setting of Offline RL, CQL (Kumar et al., 2020) computes a conservative estimate of the function values minimize overestimation error. The objective minimized by CQL adds additional conservative loss term to the loss of DQN as follows
L = DoN (0) + LCQu(0)
where
L DoubleDDN : (T Q0 (st+1, argmax, Lesgmax,,ossttiiee (o(s(+1, Qu(ss, (((ss,a)))) at))
yrang
(c) aBBsDDloog Σ exp Qa(s,, a) Eu- QQ(ss,a)))
and 0 are the parameters of the Q function and are the parameters of the target Q function. The original CQL paper uses DQN instead of the DoubleDQN but following Seno & Imai (2021) we consider the Double DQN.
Proposed Method:
We use the CQL algorithm and propose the transfer algorithm by using the offline teacher's Q function (Qleacher) in computing Uuurget-
1/arget (1 B) Q0r argmax, Qu (o(+,,,, +B (0 argmax, Queacher (81+ a))
Argmax Student Q
Teacher
where 3 is fixed value between [0,1]. This enables the Yurget value to be queried at actions that are preferred by the teacher agent the current state. This imply updating more frequently the Q values of the state action pairs experienced by the teacher. When 3 = 0. this is the same as the CQL. algorithm, and when function.
only the arg max Qleacherss, used to query the student's target Q
5 EXPERIMENTAL RESULTS
We performed experiments on variety of domains to study the resource constrained setting. For continuous control tasks, we used the OpenAl gym MuloCo (Todorov et al., 2012); for discrete control tasks, we used the Atari-2600 environment (Bellemare et al., 2013) and for real world task we consider the task of Auto-bidding and use proprictary dataset.
5.1 MuJoCo
Before we discuss the resource-constraing setup used for the MuJoCo environments in Sec- tion 5.1.1, it is important to note that the data-collecting agent plays a vital role in determining the quality the dataset (coverage state-action pairs) and subsequently the performance of any offline RI. algorithm. In the D4RL suite (Fu et al., 2020), datasets were collected using behavior policies of varying expertise to simulate the variation in data quality. Each of these behavior poli- cies was trained online using the full feature set. Thus, these policies can use all of the information and explore the environment online. The quality of the offline dataset collected by these behav- ior policies is thus relatively high thereby improving the performance of offline RL algorithms them. In the resource-constraan does not have access to the full features online. The agent may only explore and navigate using the
setting, however, the behavior policy of the data-collecting agent
5

---

Under review as a conference paper at ICLR 2022
limited feature set during training. Using this behavior policy to collect data results in relatively lower quality dataset. To account for this limitation, and to illustrate its effect on our algorithm's performance, we used two different methods to generate offline datasets.
Offline RL datasets are collected by agents trained with limited features. We refer to these datasets as RC-D4RL (Resource-Constrrainnn Datasets for RL). We use the OpenAl gym Mu- JoCo locomotion environments Hopper-v2, HalfCheetah-v2,, and Walker2d-v2.
Offline RL datasets are collected by agents trained using the full features. We use the D4RL-v0 datasets (Fu et al., 2020) of the MuJoCo locomotion environments.
5.1.1 SIMULATION OF RESOURCE- CONSTRAINED SETTING
We simulate the resource-constrainee setting by reducing the feature space available during the deployment of the agent. We do this by dropping a fixed set of features from the full features available (this is possible in the system latency as well as the nano-satellite example). For instance, consider the MuloCo environment Hopper-v2 where the original state space 11 dimensional. We consider four scenarios where the online observable feature dimensions is reduced to 5, 7, 9 and 10 by randomly picking subset of the features of the given dimension. For each of these scenarios, we consider 5 random seeds to simulate different features getting dropped in each seed. We summarize this setting for the three environments in Table I. For the dataset collection using policies trained with limited features, we follow similar protocol asthe D4RL data collection procedure (Fu al., 2020). We have added all details to Appendix C.1.
Environment
Original Dim Resource-Conssrraned Dim
Number of Seeds
Hopper
11
5,7,9, 10
5
HalfCheetah, Walker2d
17
9, 11, 13, 15
5
Table I: Resource-Constrainne Simulation using gym MuJoCo tasks
5.1.2 TRAINING AND EVALUATION
We train a teacher agent using the full feature space in the dataset. We use the TD3+BC algorithm trained on the limited feature set as the baseline (unless otherwise mentioned). Then we train the student agent using the trained teacher following Algorithm 1. We train the baseline and student for three random seeds. To illustrate the effect of the transfer, we consider two representative settings, (i Transfer (0.5, 0.5) where 31 Bg = 0.5 which gives equal weight to behavior cloning and transfer, (ii) Transfer (0.0, 1.0) where 31 = 0, B2 = 1.0 where only transfer is performed. We adopted the base hyperparameters from TD3 since hyperparameter tuning in offline RL is difficult task without the access for the environment during training (discussed in Appendix C). We used the normalized score (Fu et al., 2020) for evaluating the algorithm (more details in Appendix C.2).
From Figures and 3, we can see that the proposed transfer algorithm significantly outperforms the baseline for both the RC-D4RL and D4RL datasets. In Figures 2a and 3a, we show the percentage of experiments (all the difficulties, dimensions, dataset seeds, algorithm seeds) where each of the proposed algorithms performs better than the baseline for each environment. We observe that the proposed algorithm performs similarly for both the datasets with the performance in D4RL being slightly better. In Figures 5d and 3b, we average the scores of all experiments for each environment and algorithm pair and compare the % improvement over the baseline. We also study the loss in performance by not using offline features Table 2 (more detailed results in Table 5) which shows that the transfer approach is more suitable to the low data quality regime. We perform thorough analysis of the results and summarize the findings in Appendix C.3, focusing on the effect of difficulties of the datasets and different (limited) feature dimensions.
Additionally, we also train two more baselines: (i) True-BC. agent that learns from the teacher policy directly through behavior cloning, (ii) predictive baseline where an autoencoder is first used to predict the offline features from the observed online feature and uses the predicted features during training and deployment (discussed in detail in Appendix C.4). We evaluated these algorithms

---

Under review as a conference paper at ICLR 2022
Transfer (0.5.0.5)
(0.5.0.5)
Transfer 0,100.0))
Transfer (0. (.5.5.5)
Transter (0.0.1.0)
Trunsfer 08 1.0)
Trandfer (0.5.0.5)
Transter (0.0.1.0)
basaline
2.0
v6 seesi better than baseline tx % B3
20
10
20
HC
HC-4
Environment
Environment
(a) % of succes over the baseline across all exper- iments.
(b) & improvement over the baseline across all experiments.
(a) % of succes over the baseline across all exper- iments.
(b) % improvement over the baseline score experiments.
Figure 2: Results RC-D4RL datasets.
Figure 3: Results D4RL datasets.
Baseline (TD3+BC)
Transfer (0.5,0.5)
Transfer (0.0,1.0)
True-BC
Predictive
Difficulty
Difficulty
expert
-29.8 %
-32.5 -22.4 %
-39.6
expert medium-replay
-32.1 % 30.7 % 31.0%
-18.5%
medium-replay -33.2 %
-14.1 %
We summarize the loss performance by not using the offline features for the Baseline (TD3+BC), Transfer (0.5,0.5) and (0.0,1.0) as % change over the Teacher score RC-D4RL HalfCheetah-v2.
Table 2:
Table 3: Average % improvement of
the proposed method over the True-BC and Predictive baselines on RC-D4RL HalfCheetah-v2 datasets
RC-D4RL HalfCheetab-v2 medium-replas and expert datasets and present the result summary in Table 3 (more detailed results Table 6).
5.2 ATARI-2600
We evaluate our proposed approach on the Atari 2600 suite of games Bellemare al. (2013), which is discrete control problem that is challenging especially due to the high dimensional visual input and delayed credit assignment. Since we are in the offline RL setting, we use the DQN replay dataset from Agarwal et al. (2020) which is standard benchmark to evaluate offline RL algorithms for discrete control problems. particular, we use the data for the following games: Pong, Qbert, Breakout.
5.2.1 DATASETS
The DQN replay dataset consists of 50M state transitions that are collected during the training of DQN agent in the online setting. Each transition consists of a tuple (state, action, reward, next state) where the state and next state are 84x84 images and the action space is discrete. Following earlier works, we consider environments with sticky actions (the agent takes current action with probability 0.25, otherwise repeats past actions). Additionally, we use stacking of 4 consecutive frames together as the state space which is a common technique used for the Atari suite. Instead of using the full 50M transitions, use 1M transitions in the dataset that collected during the end of the DQN online training and hence contain expert level trajectories.
To simulate the online and offline features, consider the following setup. We assume the actual image observation (4x84x84) as the offline features, and pixelated version of the images as online features. To simulate this, resize each (84x84) image 16x16 and then resize it back 84x84. This setup reasonably imitates the nano-satellite example where some cheap sensors need to be used online compared high resolution sensor data available offline.
We trained the algorithms with a batch size of 32 for 1M batches using the same network architecture as the DQN (Mnih et al., 2015) (which is also used in Agarwal al. (2020), Kumar et al. (2020)). We refer to it as (N-DQN). We evaluate during training at frequency of every 20k batches by assuming access to the environment, and consider the score as the average reward achieved in the
H-v2: Hopper-v2, HC-v2: HalfChectah-v2, W-v2: Walker2d-v2
H--v0: Hopper-v0, HC-v0: HaliChectah-v0,, W-vO: Walker2d-v0

---

Under review as a conference paper at ICLR 2022
last 10 evaluations. The algorithm (from Section 4.2) is implemented using d3rlpy (Seno & Imai, 2021) and use CQL (trained using online features) as the baseline.
In order to simulate the power constrained setting (as in the nano-satellite example), also sider the setup where the agent to be deployed online uses a different network architecture than the teacher network architecture (N-DQN). This enables the agent to utilize lesser memory and power during deployment. We refer to this setting as the power constrained setting. We call this network architecture as (S-DQN) by changing the size of features in the final layer. Specifically, we vary the feature dimension with values 256, 128, 64 (see Appendix B) for more details.
We observe from both the settings that there is significant drop in performance of the Baseline and Transfer agents using the online features as compared to the Teacher. The Transfer agent performs marginally better than the Bascline agent on all the three games for the N-DQN setting, For the S-DQN setting, the Transfer agent outperforms the Baseline for Pong on all the three encoder sizes considered. For Qbert, the Transfer agent outperforms the Baseline for one of the encoder sizes. is important to highlight that there is significant gap between the performance of the Teacher and the Transfer/Baselinee agents for both the settings, This suggests that more tailored transfer learning approaches are required for the Resource Constrained Offline RL problem.
Size of last layer (#Parameters of network)
Teacher
Transfer
Baseline
Game
Pong
512 (1.6 M)
8.2 + 1.89
-1.02
-2.81 + 1.39
Pong
256 (0.88 M)
-1.08 + 0.88
-1.74 + 4.83
Pong
128 (0.48 M)
-1.83 + 4.75
-7.99 + 1.52
Pong
64 (0.28 M)
-3.53 + 5.01
-5.25 :|2 7.95
Qbert
512 (1.6 M)
7890.25 1896.22
+ 4762.95 + 542.23
5833.58 + 584.74
Qbert
256 (0.88 M)
4710.00 = 73.86
Qbert
128 (0.48 M)
2524.80 717.82
2808.92 + 1577.74
Qbert
64 (0.28 M)
1336.85 = 566.79
1830.00 + 908.12
Breakout
512 (1.6 M)
106.63 + 8.64
6.07 + 0.08
4.65 + 0.22
Table 4: Results of the transfer algorithm on three Atari games using N-DQN architecture and the power constrained setting (using S-DQN). We present the mean and std for 3 random seeds
5.3 AUTOBIDDING ADS DATA
Purely simulated settings may not be representative for evaluating algorithm which is meant for real-world application. Thus, we studied agent performance in the auto-bidding task for online ad- vertising (Bottou et al., 2013). Online advertising is dynamic, stochastic environment. Advertisers have increasingly delegated decisions to machines in order to achieve increased return on invest- ment. Auto-bidding one of the critical components in this shift towards AI-driven optimization. Auto-bidding agents determine a unique bid for each opportunity in real-time. This problem is one of distributed, stochastic control in a partially observable, stochastic, non-stationary environment. This type of environment is extremely difficult to develop complex algorithms for, and it demon- strates our setting well in that there are myriad of computational constraints that limit the type of models that can be considered during online operation.
In this task, agents attempt maximize the number of ad clicks they collect during a day by bidding on queries (on which ads are shown). The set of queries which receive bid are set by the advertiser's s choice of bidded keywords. Agents are given a fixed daily budget to do this. They are charged according to some black-box auction mechanism only if their ad is clicked. Agents must balance between saving budget for future opportunities, and buying guaranteed ad space now. At each time step, an auto-bidding agent has some information available like time of the query, details of the ad text or bidded keyword, available budget, past spend, model-based estimate of how likely the consumer is to click ad, etc. This information is used takes an action (decides the bid) and gets feedback (if the ad was selected and/or clicked, and how much the click costs). If the ad was clicked, the "available budget" feature in the state is updated.
state. Based on the state, the agent
8

---

Under review as a conference paper at ICLR 2022
Transfer laamnng dada
3.0 35
Figure 4: FQE estimated rewards on the Ads data. Only the relative metrics are presented hide business-sensitive information. 10 different seed values are tried, and the average values are pre- sented. Transfer (0.0. I.0) outperforms the baseline, Transfer (1.0, 0.0), 7 times of 10.
The budget constraint in this problem means that it cannot be modeled using contextual bandits (as many Ads problems can), because the next state depends on the action taken. If the agent bids $10 given available budget of $100, can expect to have at least $90 to spend for the rest of the day. This $90 is part of the state for the next step. If the agent bids $1 in the same situation, it may expect have at least $99 for the rest of the day. This rest-of-day budget constraint state, as it restricts the trajectory of future states and actions thus, requiring an MDP formulation.
key part of the
We have pulled query-level Ads data for 10 days and 8,000 advertisers (from a proprietary dataset). The features that are available online and offline include (but not limited to) the time of the query, model-based estimate of how likely the consumer to click an ad, remaining budget, and past spend. The features that are only available offline are model-based embeddings of the query and the bidded keywords as they require extensive computational resources. There are 881 features available offline and 1 111 features available online. The main constraint limiting the types of models and features used in the online setting is computation time, as the user who generated the query will not accept wait time of more than fraction of a second before they want to see their search results. We trained on 80% of the advertisers using the proposed TD3+BC transfer algorithm and evaluated on the remaining 20% using the FQE (Le et al., 2019) algorithm.
In Figure 4, FQE estimates are normalized by the value estimate at the beginning of the first epoch (before policies are trained), which is the same value for all settings (since before training all policies are initialized as random with fixed seed). We report the average normalized metrics across 10 seeds. Normalization is done to mask the identities of advertisers, which business-sensitive. Transfer (0.0, 1.0) tends to perform better than other models including the baseline (Transfer (1.0, O.0). without transfer). Furthermore, out of 10 different seeds, Transfer (0.0, 1.0) outperforms Transfer (1.0. 0.0) 7 times suggesting the applicability of incorporating transfer learning.
6 CONCLUSION
In this work, we address an open challenge in Offline RL. In the resource-constrainee setting that is motivated by real world applications, the features available during training offline may be different than the limited features available online during deployment. We highlighted performance gap be- tween offline RL agents trained using only the online features and agents trained using all the offline features. To bridge this gap, we proposed a student-teacher-basee policy transfer learning approach. The proposed algorithm improves over the baseline significantly even when the dataset quality lacking in the resource-constrainee setting. The simplicity of the approach (with just one additional hyperparameter) makes easy to extend it to other offline RL algorithms. It would be interesting to study other transfer learning approaches (c.g., policy transfer with other divergence regularizations for stochastic policies) in the future. Moreover, we observe that the proposed approach benefits es- pecially when the dataset quality is low which is often the case with real world datasets. Despite this, the performance gap with the teacher is still high (in the low quality data regime) and this suggests more tailored approaches are required.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114, PMLR, 2020.
Karl Johan Aström. Optimal control of markov processes with incomplete state information i. Jour- nal of Mathematical Analysis and Applications, 10:174-205, 1965.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ-
ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. J. Mach. Learn. Res., 22:2-1, 2021.
Léon Bottou, Jonas Peters, Joaquin Quinonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research, 14 2013.
2013.
David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. arXiv preprint arXiv:2 (06i::2000000009, 2021.
Jacob Buckman, Carles Gelada, and Mare G Bellemare. The importance of pessimism in fixed- dataset policy optimization. arXiv preprint arXiv:2009.06799 2020.
Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl. arXiv preprint arXiv:2106.09119, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2106.01345 2021.
Wojeiech Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz. Swirsze7, and Max Jaderberg. Distilling policy distillation. In The 22nd International Conference Arti- ficial Intelligence and Statistics, pp. 1331-1340. PMLR, 2019.
Will Dabney, Mark Rowland, Marc G Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Aniket Anand Deshmukh, Srinagesh Sharma, James W Cutler, Mark Moldwin, and Clayton Scott. Simple regret minimization for contextual bandits. arXiv preprint arXiv:1810.07371, 2018.
Aniket Anand Deshmukh, Yunwen Lei, Srinagesh Sharma, Urun Dogan, James W Cutler, and Clay- ton Scott. A generalization error bound for multi-class domain generalization. arXiv preprint 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study on ppo and trpo. In International conference learning representations, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning.
arXiv preprint arXiv:216..00 066600 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 0052-2066. PMLR, 2019.
10

---

Under review as a conference paper at ICLR 2022
Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Shane Gu. Identifying co-adaptation of algorithmic and implementational innovations in deep reinforce- ment learning: A taxonomy and case study of inference-basee algorithms. arXiv preprint arXiv:2103. 17258, 2021.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):179-1811) 2021.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Berhhard Schölkopf, and Alexander Smola.
A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-715 2012.
Yijie Guo, Shengyu Feng, Nicolas Le Roux, Honglak Lee, Minmin Chen, et al. Batch reinforcement learning through continuation method. In International Conference on Learning Representations, 2020.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv: 1703.0299, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning. pp. 1861-1870. PMLR, 2018.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):9--34, 1998.
Pierre-Alexandre Kamienny, Kai Arulkumaran, Feryal Behbahani, Wendelin Boehmer, and Shi- mon Whiteson. Privileged information dropout in reinforcement learning. arXiv preprint 2XXv:22005.09200. 2020.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. arXiv preprint arXiv:2005.05951, 2020.
George Konidaris and Andrew Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 489-496, 2006.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. arXiv preprint 1906: 00949, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforce- ment learning, pp. 45-73. Springer, 2012.
Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712. PMLR, 2019.
Alex X Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond picke Tackling robotic stacking of diverse shapes. In 5th Annual Confer- ence on Robot Learning, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
11

---

Under review as a conference paper at ICLR 2022
Ang Li, Huiyi Hu, Piotr Mirowski, and Mehrdad Farajtabar. Cross view policy learning for street navigation. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pp. $100-8109, 2019.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (00i://500.02999 2015.
Lydia T Liu, Urun Dogan, and Katja Hofmann. Decoding multitask dqn in the world of minecraft. In The 13th European Workshop on Reinforcement Learning (EWRL.) 2016, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 5 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuogiu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Ronald Ortner, Daniil Ryabko, Peter Auer, and Rémi Munos. Regret bounds for restless markov bandits. In International conference on algorithmic learning theory, pp. 214-228. Springer, 2012.
Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.0905: 2020.
Theodore Perkins, Doina Precup, et al. Using options for knowledge transfer in reinforcement learning. Technical report, Citeseer, 1999.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168 PMLR, 2021.
Andrei A Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint arXiv: arXiv:1511.080 151.00699,. 2015.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement library. NeurIPS 2021 Offline Reinforcement Learning Workshop. December 2021.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484488, 2016.
Matthew E Taylor and Peter Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Lisa Torrey, Trevor Walker, Jude Shavlik, and Richard Maclin. Using advice to transfer knowledge acquired in one reinforcement learning task to another. In European Conference Machine Learning. pp. 412-424. Springer, 2005.
René Traoré, Hugo Caselles Dupré, Timothée Lesort, Te Sun, Natalia Díaz Modrigudr,g, and David Filliat. Continual reinforcement learning deployed in real-life using policy distillation and sim2real transfer. arXiv preprint arXiv: 1906.0442, 2019.
12

---

Under review as a conference paper at ICLR 2022
Vladimir Vapnik, Rauf Izmailov, et al. Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res., 16(1):2023-200 2015.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual in- telligence: A review and outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv:1911.11366 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey, arXiv preprint arXiv:2009.0788 2020.
13

---

Under review as a conference paper at ICLR 2022
A ALGORITHM
Background of TD3+BC We discuss a simple way to extend an existing offline RL algorithm to apply to the resource-constrainee setting. TD3+BC (Fujimoto & Gu, 2021) is recent offline RL algorithm which shows that by adding an additional behavior cloning term to an existing online RL. algorithm TD3 (Fujimoto et al., 201 8) (that learns deterministic policy). it possible to attain state- of-the-art performance on the D4RL benchmark datasets (Fu et al., 2020). The policy evaluation step in TD3+BC is the same TD3 where the minimum of two functions is used reduce the over-estimation bias of the value function. In the policy iteration step, the behavior cloning term (TT(s) - a)² regularizes the policy to take actions similar to the actions observed in the dataset. The policy iteration step is given by
(s, (s)(s)- -o)
T++11 arg max
N E (sn,as) Q(8s a:)l
where a is parameerr.ettrr.
Algorithm Policy Transfer with TD3+BC
83 2: Given: policy update frequency d; weighted average parameter noise parameter ō
1: Given: offline dataset D with full feature observations, the weights 31, 82 s.t.
Train an offline RL agent using TD3+BC which outputs the teacher model Tatasciu:
4: Initialize critic networks Qo Qo and an actor network T
5: Initialize target networks 0g 01, 02 02, Φ)
6: for = 1, ................................ T do
7: Sample mini-batch of N transitions (s, a, 8')
8: Online features for this transition is given as (8, a, 8')
9:
- T5 (8') min(ma c)) where N(0,5) min;=1,2 Qo (8',
10:
+ y +
11:
Update critics 0g + arg ming, 1 N if t mod d == then
E (y Qo, (8, a))
13:
Update é by deterministic policy gradient optimizing the objective
N
AQo, (8s, (Bi)) - 31 (i(T(((6 (T(())) - a) - -3 (T (8i) To((i)) 2
arg max
Σ 1=1
(3)
N
15:
Update target networks: 8 T0; + 8)
16:
17:
+ - TØ + T
18:
end if
19:
end for
B ATARI EXPERIMENT DETAILS
The architecture of N- N-DQN is given by [[32, 8, 4]. 164, 4, 2], 3,11, 512]. where 512 the size of the final layer. We simulate the power constrained setting by varying the final layer size as 256, 128, 64, thereby reducing the number of parameters of the network and hence the memory and power usage.
The hyperparameters are different from the implementation of CQL (available at We use QR-DQN Dabney et al. (2018) com- pute the DQN loss using 32 quantiles. We use learning rate 6.25 10 and used Adam optimizer with e = 1/32 x 10-2. The target update interval is set to 8000. The a in CQL loss set to 1.0. During evaluation, the agent follows greedy approach with 0.001.
We selected the value of 3 by choosing the value that resulted in the best performance on Qbert and used for the other environments. We did this separately for the N-DQN setting and the power constrained setting (for final layer 2izc=256). In particular, we use 3 0.8 for N-DQN (using 512 in final layer) and we use 3 0.95 for S-DQN (using 64, 128. 256).
14

---

Under review as a conference paper at ICLR 2022
(a) Pong original image (b) Pong pixelated image (c) Qhert original image (d) Qbert pixelated image
Figure 5: Simulation of high resolution and low resolution sensors.
During evaluation, the environment outputs pixelated images that are used by the Baseline and Transfer agents. The pixelated images can be see as shown in Figure 5.
C MuJoCo EXPERIMENT DETAILS
We used the open source implementation of TD3+BC available at TD3+BC for our experiments. We used the same network architectures, and all other hyperparameters in this implementation. We also normalized the offline RL dataset using the procedure described in Fujimoto & Gu (2021).
Recent work (Fujimoto & Gu, 2021) highlighted the difficulty in comparing algorithmic contribu- tions in Offline RL due to the high sensitivity of deep RL algorithms to implementation changes (Engstrom al., 2019; Furuta al., 2021). It is even more crucial in offline RL because hyper- parameter tuning by interacting with the environment defies the purpose of offline RL, and tuning without environment access algorithmic contributions, we adopt the base hyperparameters of TD3+BC for the baseline and the proposed algorithm.
well understood (Paine al., 2020). Therefore, highlight our
(obearved offinn)
Data
Tancher
Limited features
Student
Regaarzztion
Gradkents
Figure 6: Illustration of the proposed method
C.I DATASET COLLECTION PROCEDURE: RC-D4RL
For the dataset collection using policies trained with limited features, we follow similar protocol as the D4RL data collection procedure (Fu al., 2020). In the D4RL benchmark suite, offline datasets with different difficulty levels are collected based on the exploration capacity of the data-collecting policy or behavior policy. The difficulty levels are denoted as medium-replay, medium, medium- expert and expert. For each combination (environment, difficulty, dimension, seed) in Table we train an expert policy by training TD3 (Fujimoto et al., 2018) algorithm for million timesteps using the limited features, and using the base hyperparameters of the TD3 algorithm. We then train the medium policy by training the TD3 algorithm until the agent reaches roughly half of the reward achieved by the expert policy for this combination. These policies deployed in the simulator and the interactions are logged to generate the expert and medium datasets. Note that all the features are logged in the datasets. The medium-replay data comprises of the environment interactions recorded
15

---

Under review as a conference paper at ICLR 2022
during the training of the medium policy. The medium-expert dataset is the concatenation of the medium and expert policies. Thus, we generate in total 240 offline RL datasets. Note that we did not perform hyperparameter tuning to compute the expert data collection policy and only used the base hyperparameters used in the TD3 paper'. We can observe the variation in the quality of the datasets with the dimensionality of online features and the difficulty from Table C.6 which reinforces the need to evaluate the proposed algorithm with the RC-D4RL datasets.
C.2 EVALUATION
When reporting the performance of the algorithm, we assume online access to the gym MuJoCo sim- ulator for evaluation. The agents are restricted to using the limited features (defined by the combi- nation of the offline dataset used during training) to make a decision. Given a policy evaluate, we perform 10 different rollouts using this simulator with random initial states and compute the mean of the cumulative rewards. We perform one round of evaluation of the policy (student or baseline) dur- ing training at fixed frequency. We take the average value of the last 10 evaluation rounds during training and report the mean and the standard deviation of the score. Lastly, since we run multiple trials (for each configuration), we compute the mean and standard deviation of the results of the three random trials for the given configuration. In order to facilitate easier understanding and comparison of the results across datasets and environments, we adopt the normalized score computation' (Fu et al., 2020)
score score of random policy
normalized score = 100 x score of expert online policy score of random policy"
C.3 DETAILED ANALYSIS
We designed our experimental analysis to answer the following questions.
How does the improvement offered by the transfer algorithm vary with the dataset difficulty? See Figures 7a and 8a.
How does the improvement offered by the transfer algorithm vary with the number of offline features that are left out of the online setting? See Figures 7b and 8b.
We want to study how much performance is lost by not using the offline features (for the base- line), and what part of that performance is recovered by using the transfer algorithm (Table 5).
Dataset Difficulty: From Figures 7a and Sa, observe that majority of the performance gains have been observed for the medium or medium-replay datasets. The expert and medium-expent datasets consistently provided relatively lesser % improvement, and in some cases performed worse than the baseline. The trend is consistent across the different environments (although it is more significant for HalfCheetah-v2) With expert or medium-expee expert datasets, the coverage of the state- action product space is narrow but of good quality. When training with limited features, the trained policy can drift away from this distribution. Since the data distribution is narrow, it is difficult for the trained policy to distinguish between unsafe state-action pairs from the safe state-action pairs. Whereas for the datasets with poor quality (medium, medium-replay), the data distribution wide. The regularization from the teacher is stronger and also the agent can better discriminate unsafe state-action pairs due to the good coverage of the data.
From Fig 8b, we observe that the performance gains with the pro- posed algorithm follows an inverted "U" pattern where the performance gain is highest when the reduced dimension is neither too less nor too large. When the reduced dimension is small, the online and offline features might be less correlated and hence the transfer is not efficient. When the online feature dimension is already high, the baseline has decent performance and therefore the % per- formance improvement over the baseline is limited. This trend is more pronounced for the results on D4RL datasets but not very clear on the RC-D4RL datasets.
Reduced Feature Dimension:
'3 environments x 4 dimensions
seeds 4 difficulty levels.
"The expert data collection policy affects the quality of all the difficulties. 5 EEpeert online policy is fully trained online SAC policy (Haamoja et al., 2018)
16

---

Under review as a conference paper at ICLR 2022
Heapand
Hepponed
Treette Koa.st
Trurster X.5.M
Turskra
(a) % Improvement of the aggregated across the Dimensions. mr: medium- replay, m: medium, me: medium expert, e: expert
the baseline,
(b) Improvement of the the baseline, aggregated across the difficulties.
Figure 7: Results RC-D4RL datasets.
hoopperv
(a) % Improvement of the scores over the baseline, aggregated across the Dimensions. mr: medium- replay, m: medium, me: medium expert, e: expert
(b) % Improvement of the scores over the baseline, aggregated across the difficulties.
eeppe,,
Figure 8: Results D4RL datasets.
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
% Recovered (0.5,0.5)
% Recovered
(0.0,1.0)
medium-replay
9
-36.0 %
-25.7 %
-1.66 %
10.3 %
18.4 %
medium-replay
11
-40.3 %
-28.8 %
-18.0
11.5 %
22.3 %
medium-replay
13
-35.3 %
-27.8 %
-19.3 %
7.5 %
16.0 %
medium-replay
15
-21.5 %
-7.3 %
-1.6 %
14.1 %
19.9 %
medium
9
-77.9 %
-63.4 %
-6.66 %
14.5 %
21.4 %
medium
-67.1
-66.4
-0.55 %
0.7 %
6.6 %
medium
13
-37.2 %
-36.6 %
-30.1 %
0.5 %
7.1 %
medium
-12.4
-5.7
-4.3
6.7
8.1
medium-expert
9
-43.2 %
-31.8 %
-37.5 %
11.4 %
5.7 %
medium-expert
11
-43.6
-36.4 %
-33.3 %
7.3 %
10.3 %
medium-expert
13
-26.3 %
-19,2 %
-16.1 %
7.1 %
10.2 %
medium-expert
15
-3.6
-10.4
-12.3 %
-6.8 %
-8.8 %
expert
-6.55
-5..5
-64.3 %
0.0 %
-7.7 %
expert
11
-41.1 %
-39.3 %
-39.3 %
1.8 %
1.8 %
expert expert
-15.5
-23.9
-35.6
-8.5
-20.1
15
-6.3 %
-10.6 %
-19.2 %
-4.3 %
-12.9 %
Table 5: We summarize the loss in performance by not using the offline features for the Baseline, Transfer (0.5,0.5) and Transfer 10.0,1.0) percentage change the Teacher score. We also show the % of recovered performance by using the Transfer algorithm as compared to the Baseline in the last two columns.
% of Teacher's performance: Here, we present the analysis of the drop in performance by using only online features the baseline) by measuring the percentage difference compared to the teacher model (trained using full offline features). As we can observe, the baseline consistently under performs the teacher (as far as -70% in some cases). We also show the percentage drop in performance of the proposed transfer method trained using the teacher and that the transfer approach can recover some lost performance by not using the offline features. Interestingly, we observe that for higher quality datasets, the improvement offered by the transfer approach is limited. Moreoever, we observe that increasing the weight of the teacher has a positive impact on the performance (as also be seen for the Ads results and Atari results).
17

---

Under review as a conference paper at ICLR 2022
Difficulty
Dimension
True-BC
Predictive
Transfer (0.0,1.0)
medium-replay
8.1
8.1
11.2
medium-replay
8.1
8.4
11.4
medium-replay
13
13.9
12.1
15.1
medium-replay
15
13.9
15.0
18.8
expert
9
13.1
9.9
6.0
expert
11
14.4
9.9
9.9
expert
24.8
24.2
18.3
expert
15
34.0
31.5
28.2
Table 6: Performance of True-BC and Predictive baseline RC-D4RL HalfCheetah-v2 datasets
C.4 ADDITIONAL. BASELINES
C.4.1 PURE BEHAVIOR CLONING TRANSFER (FROM TEACHER)
In addition to the TD3+BC BC baseline that we consider to evaluate in the online features case, we also evaluate pure behavior cloning algorithm that takes teacher policy as input and learns imitate the teacher. The new policy is learnt by minimizing
arg min E (Treacher (T (B)
The policy as uses similar architecture as the teacher, with the exception that the input number of features reduced due to the online features available.
We evaluate the algorithm on HalfCheetah environment in the RC-D4RL datasets, and summarize the results the following table. The training and evaluation procedure similar as the main experiments.
C.4.2 PREDICTIVE MODEL
We also consider an additional baseline that predicts the missing features (offline features) from the available online features. do this by first training an autoencoder that takes the online features as input and predicts the offline features by minimizing the MSE loss between the predicted offline features and the actual offline features. The trained autoencoder is than passed to the offline RL algorithm (that is trained for deployment). During every step of training, the algorithm takes the online features, predicts the offline features using the autoencoder and uses the predicted features as the state observation. Similarly, during evaluation, the trained agent first predicts the features using online features and uses them to take an action.
We evaluate the algorithm on HalfCheetah environment in the RC-D4RL datasets, and summarize the results the following table. The training and evaluation procedure is similar as the main experiments.
C.4.3 RESULTS
We can observe from the results that the True BC agent is very effective in the expert dataset (which of high quality). whereas the proposed algorithm is effective in medium-replay dataset (which is of low quality). Similar conclusions hold true for the predictive baseline (autoencoder). Although the performance on the expert datasets is quite good for these baselines, real world datasets are often mixture of datasets from multiple policies and often noisy. Although these algorithms provide strong starting point, they may not be quite useful in real applications.
18

---

Under review as a conference paper at ICLR 2022
C.5
LEARNING CURVES
Figures 9, 10, 11 depict the learning curves during training in the RC-D4RL experiment.
2.0
122
1.00
6.4
6.6
beacher
baoofine
0.50
0.75
0.50
Seacher
Esseline
0.2
0.29
stusker_
62
Studer
0.00
150
200
(a) medium-replay
(b) mcdium
(c) medium-expeeee
(d) expert
Figure 9: Hopper
0.15
0.000
0.30
0.10
0.28
teacter
teache"
Haccher
teacher
0.05
boceliee
0.00
baseine
0.25
0.44
0.28
0005
0.000
150
(a) medium-replay
(b) medium
(c) mediuu--xppert
(d) expert
Figure 10: HalfCheetah
0.75
0.6
teacher
0.50
teacter
baseline
studert
0.25
Studont.1
6.0
0.2
100
(a) medium-replay
(b) medium
(c) medium-expert
(d) expert
Figure 11: Walker2d
C.6 DATASET SUMMARY
We present the summary of the RC-D4RL datasets.
19

---

Under review as a conference paper at ICLR 2022
Dataset
Reduced Dim:
10
medium-replay
47.4 23,4
41.0
186.0 76.1
medium
588.1 + 351.5
754.4 + 339.0
900.0 + 285.5
1357.8 + 423.6
medium-expert
660.6 + 335.7
936.7 + 395.1
1380.9 + 473.1
1929.7 + 469.3
expert
803.6 + 320.0
1196.9 + 402.8
2399.0 = 744.9
3068.5 + 617.1
Table 7: Hopper environment: Average reward per trajectory
Dataset
Reduced Dim: 9
11
13
15
medium-replay
-77.5 + 140.9
-31.6 141.2
183.7
288.9
206.2 + 276.5
medium
612.8 + 321.5
838.3 531.1
1389.1 = 637.9
1786.6 = 931.7
medium-expert
1013.2 + 519.6
1450.2 873.8
2471.6 1073.3
3404.9 1803.1
expert
1413.5 + 724.8
2062.2 + 1233.0
3554.1 + 1528.2
5023.2 + 2682.
Table 8: HalfCheetah environment: Average reward per trajectory
Dataset
Reduced Dim:
13
15
medium-replay
19.8 + 17.6
43.2 65.2
175.2 = 131.6
269.7 + 73.2
medium
307.2 + 106.2
446.8 = 182.9
797.8 = 208.4
1071.0 = 188.5
medium-expert
424.1 + 128.8
650.0 + 273.4
1173.3 + 367.4
1743.9 + 341.4
expert
691.9 + 302.2
1039.5 + 507.4
1837.0 735.4
3330.7 + 685.7
Table 9: Walker2d environment: Average reward per trajectory
C.7 MORE ANALYSIS
Figures 12, 13, 14 illustrate the results our experiments with RC-D4RL datsets. We plot the mean normalized score and the error bars represent the standard deviation across the random seeds.
Tables 10, 11, 12 contain the results of the RC-D4RL experiments. 13, 14, 15 contain the results of the D4RL experiments. In both cases, we report the mean normalized score and the standard deviation.
20

---

Under review as a conference paper at ICLR 2022
medium-replay
medium
medium-expert
expert
Dimension
1000
Dimension==
2000
1500
500
1
3
1
2
3000
Dimension=9
2000
1000
3
3000
Dimension=
2000
1000
data seed
data seed
data seed
data seed
baseline
teacher
Transfer (0.5,0.5)
Transfer (0.0,1.0)
Figure 12: RC-D4RL Hopper-v2 v2 experiments summary. We plot the mean of the rewards and the emvor bars represent the standard deviation across the 3 random seeds for given dataset.
21

---

Under review as a conference paper at ICLR 2022
medium-replay
medium
medium-expprt
expert
2000
Dimension=
1000
1
1
Dimension=
2000
1000
TP
tuft
1
3
1
3
1
1
2
Dimension=15
2000
Jl
6000
Dimension=115
4000
2000
0
4
data seed
data seed
data seed
data seed
baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
Figure 13: RC-D4RL HalfCheetah-v2 experiments summary. We plot the mean of the rewards and the error bars represent the standard deviation across the random seeds for given dataset.
22

---

Under review as a conference paper at ICLR 2022
medium-replay
medium- experrt
exp ert
Dimension=
1000
500
I.
1
3
1
1500
Dimension=11
1000
500
1
3
1
4000
Dimension=11
3000
1000
88
Dimension=15
4000
2000
0
1
data seed
data seed teacher
data seed
data seed
baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
Figure 14: RC-D4RL Walker2d-v2 experiments summary. We plot the mean of the rewards and the error bars represent the standard deviation across the 3 random seeds for given dataset.
23

---

Under review as a conference paper at ICLR 2022
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
5.5 + 1.9
13.4 9.6
9.0 + 6.4
medium-replay
8.4 + 6.4
9.0 = 6.2
12.9 9.2
9
20.0 + 3.6
21.6 = 5.6
26.9 9.5
10
22.7 + 6.3
30.2 + 11.6
37.7 16.6
19.1
10.5
15.0 + 10.7
16.7 11.6
medium
7
14.9
- 11.6
12.6 + 13.0
9.4 + 9.5
9
31.0 + 8.5
34.5 + 14.2
45.5 + 21.8
10
48.0 21.1
51.1 + 24.4
50.9 + 25.3
5
23.3 - 10.6
24.2 = 8.6
19.0 = 10.0
medium-expert
7
20.1 + 18.0
25.8
+ 21.6
25.1 + 17.3
52.5 = 24.6
61.9
+ 29.4
57.0 = 34.5
10
67.4 21.8
75.8
+ 27.2
74.7 23.8
5
19.2 + 12.1
21.5
+ 12.2
20.5 + 12.7
7
26.0 = 17.5
25.1 + 18.4
24.8 + 19.0
expert
69.1
16.3
83.2
+ 12.7
86.6 17.8
10
60.8 + 34.5
62.2
+ 32.7
49.2 + 29.6
Table 10: Results RC-D4RL Hopper-v2 dataset
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
8.7 + 1.5
10.1 2.0
11.2
= 2.8
medium-replay
8.3 4.8
9.9 = 4.8
11.4
- 5.0
13
12.1 + 4.6
13.5 = 4.7
15.1
+ 5.3
15
15.0 + 7.6
17.7
18.8
+ 6.9
3.2 4.7
5.3 5.4
6.3
+ 6.5
medium
11
5.0 + 4.2
5.1 5.4
6.0
5.6
13
11.5 + 7.7
11.6 = 9.6
12.8
+ 10.6
15
18.3 + 8.4
19.7 8.7
20.0
+ 9.7
10.0 + 6.1
12.0 = 5.7
11.0 + 7.3
medium-expert
11
9.3 + 5.5
10.5 = 6.1
11.0 + 6.3
13
18.8
+ 11.5
20.6 + 13.1
21.4 14.2
15
29.7
11.7
27.6 + 14.9
27.0 15.5
7.3
+ 7.2
7.3
8.0
6.0 + 8.9
11
9.6
+ 5.4
9.9
6.2
9.9 + 6.3
expert
13
24.0
+ 11.4
21.6 + 13.5
18.3 16.4
15
32.7
13.5
31.2 + 12.6
28.2 + 12.7
Table 11: Results RC-D4RL HalfCheetah-v2 dataset
24

---

Under review as a conference paper at ICLR 2022
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
0.8 = 1.5
1.8 = 2.5
4.1 = 5.4
medium-replay
2.2 3.1
1.4 + 2.6
1.9 + 3.7
13
8.5 + 7.8
13.0 + 12.4
14.9
+ 15.5
15
18.7 + 7.1
22.9 + 10.3
29.8 + 12.6
5.6 1.6
+ 4.6
6.5 + 3.4
medium
11
8.1
=
6.6 + 4.0
7.2 - 4.1
13
15.8 + 11.7
16.2 + 14.8
17.5
+ 17.8
15
32.2 8.1
40.0 + 12.8
44.8
16.2
7.9 3.5
8.9 5.1
10.6
- 5.7
medium-expert
11
6.5 + 6.9
8.7 6.4
11.4
+ 8.5
13
36.1 - 24.1
26.1
51.2
- 27.4
15
65.0
18.3
71.6 + 15.9
78.3
17.0
9
11.3 + 7.3
12.9 + 6.9
11.3 + 7.3
11
12.8 5.6
13.7 7.6
14.1
= 7.0
expert
13
42.7
+ 22.9
51.2 + 23.7
52.6 25.0
15
54.1 + 36.3
62.0
+ 38.4
59.9 + 29.6
Table 12: Results RC-D4RL Walker2d-v2 dataset
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
6.3 + 3.3
8.0 4.2
8.2 4.4
medium-replay
14.7 4.5
19.0 = 4.9
20.3
- 6.9
9
29.3 + 3.0
30.6 = 2.1
30.9 + 2.7
10
31.5 + 1.0
32.3 1.2
31.1 + 0.7
10.6
3.6
8.4 2.3
6.5 + 1.7
medium
7
29.6 + 7.0
35.5 + 19.1
50.4 + 30.7
9
43.4 + 12.7
73.0 + 17.1
88.6 + 18.5
10
66.9 + 28.7
87.6 + 18.3
99.6 + 0.5
5
6.6 2.8
7 3,4
8.2 3.0
medium-expert
7
29.2 + 15.4
35.5 + 22.0
43.0 + 26.5
9
85.6 + 29.0
104.6 7.3
101.2 + 5.3
10
107.9 + 3.3
111.0 1.3
109.0 + 2.2
5
9.1
+ 2.2
7.4 = 0.7
8.1 + 1.6
7
46.9
+ 33.6
38.1 + 37.6
33.1 + 33.3
expert
61.5
+ 50.6
58.9 + 42.5
60.4 39.7
10
70.8
+ 55.5
91.0 + 46.6
92.0 44.4
Table 13: Results D4RL Hopper-vO dataset
25

---

Under review as a conference paper at ICLR 2022
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
26.5 5.8
29.4 5.8
30.0 + 6.3
medium-replay
33.1 5.6
35.4 = 4.5
36.1 4.4
13
32.0 + 8.9
35.7 = 6.8
35.6 + 6.7
15
38.9 + 2.0
41.0 1.2
41.6 + 1.3
33.5 4.1
35.6 3.8
37.3
3.5
medium
11
36.1 + 2.5
39.1 = 0.9
40.5
+ 1.2
13
36.9 + 2.0
39.3 = 1.4
40.8
+ 1.2
15
40.2 + 1.2
41.5 1.0
42.5 + 0.7
14.7
- 8.6
13.1 8.0
11,9 + 5.9
medium-expert
11
21.4
+ 5.5
19.6 = 4.9
18.7 + 2.7
13
25.3
= 14.9
25.3 + 13.8
25.3 = 14.8
15
40.8
8.4
37.9 + 12.9
31.1 + 6.4
9
6.2 + 2.0
6,6 = 2.8 2.8
6.7 +: 2.9
11
8.8 + 6.5
9.1 = 7.6
12.8 = 10.5
expert
13
24.5
+ 25.7
25.0 + 29.8
24.7 28.8
15
59.4
+ 22.1
53.4 + 25.0
52.8 23.0
Table 14: Results on D4RL HalfCheetah-v0 dataset
Difficulty
Dimension
Baseline
Transfer (0.5,0.5)
Transfer (0.0,1.0)
2.3 + 2.0
3.7 + 4.7
7.4
= 3.6
medium-replay
3,0 4.4
5.2 - 4.4
8.3 = 4.3
13
6.9 + 1.6
10.2 + 3.4
12.5
+ 3.5
15
11.2 + 2.7
10.0
16.0 - 3.7
8.8 5.5
11.5 + 5.5
10.7 4.2
medium
11
12.8 + 6.0
20.3 + 8.9
17.1 + 8.3
13
25.9 + 13.1
37.0
+ 14.1
35.1 + 20.4
15
50.5 + 12.8
57.9 + 15.1
67.3 12.4
7.1 4.1
10.1 6.2
15.4
- 9.9
medium-expert
11
15.8 + 8.0
24.6 + 5.1
37.5
+ 20.7
13
39.5 + 17.4
50.6 27.8
67.4
+ 14.7
15
89.1
9.6
95.1 + 14.7
96.9
10.4
9
11.3 + 8.3
11.1 + 6.7
13.8
+ 8.1
11
21.9 + 8.6
30.1 + 10.5
30.2
+ 13.3
expert
13
48.9
+ 27.3
60.4 + 25.1
61.1
28.8
15
93.4
8.3
95.6 + 9.1
96.5
+- 7.8
Table 15: Results on D4RL Walker2d-VD dataset
26


Reviews:
[{"review_id": "HnZLfLcVaZ", "reviewer": "Reviewer_oGRd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Motivation & significance: this paper introduces an interesting and novel offline RL setting and considers a simple modification to an existing algorithm for improved performance. It lists two real applications as the motivating examples. While it seems to be the real problem for the two examples, I doubt how general the proposed setting will applly in practice, and therefore how broadly the research in this setting will benefit to the RL community. The authors do carry out extensive experiments on the three environments from D4RL, but I am disappointed that no experiments are done for any real application or a simulated setting that resembles some property of either of the motivating examples.\n\nNovelty: the proposed solution is quite simple and straightforward, adding an additional regularisation to a teacher policy with a higher quality. Therefore, there is very limited contribution to the methodology.\n\nClarity: the idea and experiments are very well presented in the paper. The algorithm design choice and experiment settings are explained in details and it is easy for reproducing the results.\n\nEmpirical evaluation: the experiments show the proposed method (both variants) is robust and provide improvement in most situations. I appreciate the authors makes a good effort to consider different experiment settings to get a comprehensive study about the proposed method and compare it to the baseline. However, I’m concerned that all the experiments are restricted to three simple simulated environments in D4RL and the results may not provide much evidence to how they will generalise to other environments or any real problems. Also, a very simple baseline is missing that trains a pure behaviour cloning policy to learn the teacher policy.\n\nOverall, I think it would make this work much more impactful if the authors could consider a real problem / simulated problem similar to a real problem, and study different approaches and show what the best performance gain we could obtain against an existing solution.", "summary_of_the_paper": "This paper considers a novel offline RL setting with resource-constrained online deployment. In that setting, the agent has access to less information about the state in the online deployment phase than the available information from the offline dataset. It proposes a two stage training strategy under this setting, first training a teacher agent with full features, and then training a student agent with regularisation to mimic the teacher agent. Experiments are carried on in three D4RL environments with a variety configurations in the data collection protocol, behaviour policy, feature constraints. Results show the advantage of the proposed distilled policy than a simple baseline.", "summary_of_the_review": "The new problem setting could be better motivated and supported by experiments. The proposed algorithm is a simple modification to the existing method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GTA4unYLCe", "reviewer": "Reviewer_QjjH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths** \n\n\n**S1:** The paper outlines a previously under-explored and an important class resource constrained setup for offline RL. Furthermore they define the data collection strategy inline with this setup which was not present in previously explored / popular datasets such as D4RL.  \n \n\n**S2:** They propose a minimal extension on TD3 + BC algorithm for the resource constrained settings and show that the proposed algorithm was able to close the performance gap due to the missing privileged information (I.e. the additional offline feature set).  \n\n \n\n**S3:** With a few exceptions (discussed below), The experiments presented are well designed.   \n \n\n \n \n**Weaknesses:** \n \n**W1:** The motivation behind the RC-D4RL dataset is not well articulated. As articulated in the paper, the available dataset D4RL is collected by a policy that has access to the privileged information, which in turn results in the behavioral policy having high quality trajectories. These high quality action choices should not be expected form a policy trained with no access to the privileged information. Hence it is important to collect the dataset with a resource constrained policy (no access to privileged information) in order to better quantify the advantages of being able to leverage privileged information.  \n \n**Q:** While the proposed setting makes sense in imitating real world settings, would it not be useful to have high quality trajectories in order to best leverage the action choices that come from these high quality offline features. Why not stick with D4RL?  \n**Q:** And if it indeed the case that D4RL does not represent real world settings why are we evaluating the given algorithm for both RC-D4RL and D4RL settings?  \n\n \n\n \n\n**W2:** The paper seems to be missing some natural baselines. While the proposed algorithm is quite straightforward and is able to beat a blind agent (blind to privileged information) it would be interesting to see how it compares to other natural baselines such as a predictive model baseline. \n\n \n\nPredictive mode baseline: Train a model to predict the privileged information and use this in tandem with a offline RL model that is trained with the privileged information.  \n\n \n\nAlso If I understand it correctly, the baseline would simply by equivalent to Transfer(1.0, 0). Then the question becomes did we just propose an algorithm and just compared the different hyperparameter settings?  \n\n \n**Q:** If a very natural baseline is comparable to the proposed algorithm, is the main contribution of the paper limited to the definition of the resource constrained framework?  \n\n   \n\n \n\n**W3:** More Datasets please: While this may come off as a \"knee-jerk\" reaction, I do believe that the class of problem defined here is important for the community. Hence it would definitely help if we are able to find real world examples of such scenarios and incorporate them here. Some new more natural additions on the dataset may include, (1) Atari benchmark dataset with semantic maps generated from internal bits. (2) D4RL dataset coupled with RGB frames, (3) Self driving / navigation dataset coupled with depth/semantic maps.\n\n\n\n\n**Clarity Issues/ Comments:** \n\n \n**CL1:** While the paper concludes that the performance gap between the use of online features and offline features have been highlighted, it is not well quantified in the paper, how much does the overall performance suffer due to these missing offline features. I am guessing this would amount to teacher performances – baseline performance from fig 11. It would be nice to have them in the main paper.  \n\n**CL2:** When we say we address the performance gap, it would be nice to produce numbers that represent exactly that. I.e. What percentage of the performance gap was the addition of offline features able to fill. \n\n**CL3:** The inverted U pattern mentioned for figures 4b and 5b are very hard to grasp, and basically nonexistent for 5b unless I am parsing the sentence incorrectly.  \n**CL4:** Figure 6 does not fully answer the question of how \"important\" the teacher's role is in training the student, were we expecting Transfer(0.5,0.5) to perform worse that Transfer(0, 1)? How do we maximize the teachers role in training without hurting performance?  ", "summary_of_the_paper": "The authors explore Learning Using Privileged Information (LUPI) paradigm for offline reinforcement learning and explore a teacher-student framework for transferring a policy learned under privileged information. The proposed algorithm is a minimal extension of recently proposed algorithm TD3+BC to leverages the offline/priviliged information for online deployment. ", "summary_of_the_review": "The framework introduced by the authors represent an important part of offline RL that remains under explored. The proposed algorithm is a minimal extension to a popular offline RL and the experiments/results are well motivated. However, at the current state I would lean towards rejection as the paper still lacks detailed investigation of possible approaches to solve this problem as well as results on a more compelling set of benchmarks(datasets).  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Ch86LBkh8fI", "reviewer": "Reviewer_pgM4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major Comments:\n\nAs far as I know, the authors consider a novel setting where the features that exist during online deployment are more limited than in the offline dataset. This is an interesting setting and likely relevant to many applications. However, the authors choose to model this by dropping a subset of dimensions between offline and online features. I personally would appreciate some discussion on why this is practical; I would imagine that it is more realistic to assume that certain features of the state are misspecified rather than missing, as it seems to more accurately match the motivation of offline human-annotated vs online heuristically-generated features.\n\nThe authors only consider adding their transfer learning objective on top of a specific offline RL algorithm, i.e. TD3 + BC. I feel that since the novelty of the paper is in the transfer learning objective, the authors should have considered adding their objective on top of other SOTA offline RL algorithms, i.e. CQL. Right now, I feel like since the new objective was only tested on top of one specific base algorithm, the generality of the approach is more limited. \n\nIn the experiments, the authors are able to show that adding their transfer learning objective improves upon not performing any transfer learning. While this is important to show, I feel that the result is not too surprising, as you are giving a lot of additional information to the proposed algorithm. I would personally also be interested to see how the learned policy compares to the teacher policy to see how much is actually lost by using the limited features. As mentioned in the previous paragraph, I also think the paper would be improved by considering other baseline algorithms than just TD3 + BC, as it can show that the transfer learning objective can improve different classes of offline RL algorithms.\n\nMinor Comments:\n\nIn Eq. 3 of Algorithm 1, I believe the \\pi(s_i) inside the Q-function should instead be using the limited features \\pi(\\hat{s}_i). \n", "summary_of_the_paper": "The paper proposes an offline RL algorithm in the resource-constrained setting, where the offline dataset contains richer features than provided by online interactions. The authors propose a transfer learning objective, where a teacher policy is learned from the rich features, then a policy is learned from the limited features by additionally fitting to the actions chosen by the teacher policy. The authors compare the policy learned via this transfer objective to a baseline that does not do transfer learning on D4RL tasks.", "summary_of_the_review": "Overall, the paper makes a first step in a novel and relevant setting. The actual approach appears to be general, as the transfer learning objective can be added to any existing offline RL algorithm. And though the results in the paper are perhaps unsurprising due to how much weaker the baseline is, it is understandable because, to my knowledge, no other algorithms that could leverage the offline dataset with rich features. Hence, I recommend that it be accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9w3YTbvnN4E", "reviewer": "Reviewer_CV2Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper does a good job at introducing a novel scenario of resource constrained deployment, although to my mind it is a slight over-exaggeration to say that this is the “key challenge” in offline RL. The paper is very clearly written and the contributions and the experimental settings are easy to follow. The approach is simple, does not have many hyperparameters and, thus, practical. The main concern that I have is regarding the novelty of the work. While this paper talks about offline-online RL, I think very similar issues arise in the online RL or in sim2real transfer when either more features are available during the training time than at the deployment time or when in simulation we have access to the privileged information that is leveraged for the deployment. I don’t think that the resource-constraint issue is specific to the offline RL and approaches to this problem in the online and sim2real cases are not discussed. Some example of related ideas where more information is available during the training than deployment are: Cross-View Policy Learning for Street Navigation, Li et al., Privileged Information Dropout in Reinforcement Learning, Kamienny et al, Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes. Lee at al., Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer, Traoré et al. Could the authors provide a more detailed overview of the related work in the online and sim2real RL? Are there other baselines that could be relevant to compare against? Then, it seems to me that the main contribution of the paper is rather empirical. The experiments are well designed, explained and executed on three control environments. However, the diversity of the environments is rather limited and I would like to see a broader set of experiments on varying tasks to appreciate the benefits of the method (maybe training policies from vision? maybe some discrete control? maybe some navigation tasks?)\n\nPros:\n\n- Interesting and well motivated setting of resource constrained deployment.\n\n- Simple and clear approach.\n\n- I appreciated the dataset construction component that helps to test the proposed algorithm in more realistic settings.\n\nCons:\n\n- Related literature is not covered completely. How similar problems are addressed in online RL and sim2real?\n\n- Limited novelty of the proposed approach.\n\n- The experiments are only conducted on a small set of environments of limited diversity.\n\nOther points and questions:\n\n- The way I read figure 1b only demonstrates that more features is better. It does not necessarily show that the limitation comes because of ignoring privileged information, it may well be just because limited features are available at the deployment time.\n\n- Do the authors mean “policy improvement step” instead of “policy iteration step” in the algorithm description?\n\n- Percentage of seeds better than the baseline is an unusual metric, what is the motivation of it? What is the advantage against a more standard percentage of improvement?\n\n- Could the authors explain better why the gain with a small number of available features is smaller or even negative? I understand that the performance in general would degrade, but I would expect that the baseline would suffer even more.\n", "summary_of_the_paper": "This paper presents a new under-explored problem in offline RL: the situation when during the online deployment some features that were present in the offline dataset are missing. The authors motivate this problem by the challenges in real applications. They demonstrate that a straightforward approach of training an offline policy on a set of restricted features suffers from a loss in performance. Then, they propose an extension of an existing offline RL algorithm to distil the teacher offline policy which has access to all features to a student policy which has access to a limited feature set. Finally, the authors conduct a set of experiments on 3 Mujoco control tasks where they vary different parameters of the problem, for example, the quality of the datasets (and the way they were collected), the number of dropped dimensions in the resource-constrained setting. The results show the benefits of the proposed method compared to the baseline.", "summary_of_the_review": "This paper proposes a simple but efficient approach to the resource constrained deployment of offline RL policies. I would like to hear more about the contributions of this work compared to prior works in similar settings in online RL and sim2real. I appreciate the experiments and new dataset construction, but I would like to see a broader set of experiments on a larger variety of tasks to be able to make conclusions about the efficiency of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose the resource constrained offline RL problem where the offline dataset contains extra features that are not available online. The goal is to use these extra features to improve performance during deployment. They propose a simple modification to TD3-BC in the continuous control setting and a simple modification to CQL in the discrete setting. They evaluate their proposed approaches on D4RL, RC-D4RL (a novel dataset that they introduce for resource constrained offline RL), Atari, and a proprietary real-life Ads problem.

Initial reviews identified the following concerns:
* While the exact problem is novel, the idea of having access to privileged features at training time that are not available at deployment has been explored in supervised learning and online RL. The reviewers were not clear how considering the offline RL setting interacts specifically with the privileged features to produce an interesting setting.
* The baseline simply trains on the limited feature set. Unsurprisingly, using the extra features can improve performance. In light of the previous point, reviewers asked for more substantial baselines, suggesting BC on the teacher and predicting the missing features as some possibilities.
* The set of tasks was too limited.

The authors provided a substantial response:
* Experiments on Ads data 
* Experiments on Atari with CQL as the base algorithm
* Additional baselines on RC-D4RL HalfCheetah-v2 datasets (BC on teacher and predictive)
* Additional analysis

I commend the authors on the hard work they did preparing this response. It is quite substantial and does improve the paper significantly. However, reviewers and I still have a number of concerns:
* The additional baselines are appreciated, however, the results are mixed. The additional baselines are a step in the right direction, but they need to be evaluated beyond a single dataset. It is hard to evaluate the results without reasonable baselines. I agree and think that even though the specific problem is novel, the idea of transfer learning is not, so it is reasonable to require that we have more extensive baselines. Furthermore, while the authors argue that their method has an edge on the more practical dataset, that is based on a very limited evaluation. Probing this further is important.
* The CQL modification is quite different than the TD3+BC modification. The performance of the modification for CQL is not significantly better than CQL. What should we make of this?
* For the Ads dataset, all hyperparameter settings except Transfer(0, 1) show the same performance. This seems surprising as even Transfer(0.1, 0.9) shows no difference. Finally, Transfer(0, 1) beating Transfer(1, 0) 7/10 times is not statistically significant.

At this time, the paper is not ready for publication, but the paper is moving in the right direction and I encourage the authors to submit a revised version to a future venue.
Decision: Reject