Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new under-explored problem in offline RL: the situation when during the online deployment some features that were present in the offline dataset are missing. The authors motivate this problem by the challenges in real applications. They demonstrate that a straightforward approach of training an offline policy on a set of restricted features suffers from a loss in performance. Then, they propose an extension of an existing offline RL algorithm to distil the teacher offline policy which has access to all features to a student policy which has access to a limited feature set. Finally, the authors conduct a set of experiments on 3 Mujoco control tasks where they vary different parameters of the problem, for example, the quality of the datasets (and the way they were collected), the number of dropped dimensions in the resource-constrained setting. The results show the benefits of the proposed method compared to the baseline.
main_review: The paper does a good job at introducing a novel scenario of resource constrained deployment, although to my mind it is a slight over-exaggeration to say that this is the “key challenge” in offline RL. The paper is very clearly written and the contributions and the experimental settings are easy to follow. The approach is simple, does not have many hyperparameters and, thus, practical. The main concern that I have is regarding the novelty of the work. While this paper talks about offline-online RL, I think very similar issues arise in the online RL or in sim2real transfer when either more features are available during the training time than at the deployment time or when in simulation we have access to the privileged information that is leveraged for the deployment. I don’t think that the resource-constraint issue is specific to the offline RL and approaches to this problem in the online and sim2real cases are not discussed. Some example of related ideas where more information is available during the training than deployment are: Cross-View Policy Learning for Street Navigation, Li et al., Privileged Information Dropout in Reinforcement Learning, Kamienny et al, Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes. Lee at al., Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfer, Traoré et al. Could the authors provide a more detailed overview of the related work in the online and sim2real RL? Are there other baselines that could be relevant to compare against? Then, it seems to me that the main contribution of the paper is rather empirical. The experiments are well designed, explained and executed on three control environments. However, the diversity of the environments is rather limited and I would like to see a broader set of experiments on varying tasks to appreciate the benefits of the method (maybe training policies from vision? maybe some discrete control? maybe some navigation tasks?)

Pros:

- Interesting and well motivated setting of resource constrained deployment.

- Simple and clear approach.

- I appreciated the dataset construction component that helps to test the proposed algorithm in more realistic settings.

Cons:

- Related literature is not covered completely. How similar problems are addressed in online RL and sim2real?

- Limited novelty of the proposed approach.

- The experiments are only conducted on a small set of environments of limited diversity.

Other points and questions:

- The way I read figure 1b only demonstrates that more features is better. It does not necessarily show that the limitation comes because of ignoring privileged information, it may well be just because limited features are available at the deployment time.

- Do the authors mean “policy improvement step” instead of “policy iteration step” in the algorithm description?

- Percentage of seeds better than the baseline is an unusual metric, what is the motivation of it? What is the advantage against a more standard percentage of improvement?

- Could the authors explain better why the gain with a small number of available features is smaller or even negative? I understand that the performance in general would degrade, but I would expect that the baseline would suffer even more.

summary_of_the_review: This paper proposes a simple but efficient approach to the resource constrained deployment of offline RL policies. I would like to hear more about the contributions of this work compared to prior works in similar settings in online RL and sim2real. I appreciate the experiments and new dataset construction, but I would like to see a broader set of experiments on a larger variety of tasks to be able to make conclusions about the efficiency of the method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an offline RL algorithm in the resource-constrained setting, where the offline dataset contains richer features than provided by online interactions. The authors propose a transfer learning objective, where a teacher policy is learned from the rich features, then a policy is learned from the limited features by additionally fitting to the actions chosen by the teacher policy. The authors compare the policy learned via this transfer objective to a baseline that does not do transfer learning on D4RL tasks.
main_review: Major Comments:

As far as I know, the authors consider a novel setting where the features that exist during online deployment are more limited than in the offline dataset. This is an interesting setting and likely relevant to many applications. However, the authors choose to model this by dropping a subset of dimensions between offline and online features. I personally would appreciate some discussion on why this is practical; I would imagine that it is more realistic to assume that certain features of the state are misspecified rather than missing, as it seems to more accurately match the motivation of offline human-annotated vs online heuristically-generated features.

The authors only consider adding their transfer learning objective on top of a specific offline RL algorithm, i.e. TD3 + BC. I feel that since the novelty of the paper is in the transfer learning objective, the authors should have considered adding their objective on top of other SOTA offline RL algorithms, i.e. CQL. Right now, I feel like since the new objective was only tested on top of one specific base algorithm, the generality of the approach is more limited. 

In the experiments, the authors are able to show that adding their transfer learning objective improves upon not performing any transfer learning. While this is important to show, I feel that the result is not too surprising, as you are giving a lot of additional information to the proposed algorithm. I would personally also be interested to see how the learned policy compares to the teacher policy to see how much is actually lost by using the limited features. As mentioned in the previous paragraph, I also think the paper would be improved by considering other baseline algorithms than just TD3 + BC, as it can show that the transfer learning objective can improve different classes of offline RL algorithms.

Minor Comments:

In Eq. 3 of Algorithm 1, I believe the \pi(s_i) inside the Q-function should instead be using the limited features \pi(\hat{s}_i). 

summary_of_the_review: Overall, the paper makes a first step in a novel and relevant setting. The actual approach appears to be general, as the transfer learning objective can be added to any existing offline RL algorithm. And though the results in the paper are perhaps unsurprising due to how much weaker the baseline is, it is understandable because, to my knowledge, no other algorithms that could leverage the offline dataset with rich features. Hence, I recommend that it be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors explore Learning Using Privileged Information (LUPI) paradigm for offline reinforcement learning and explore a teacher-student framework for transferring a policy learned under privileged information. The proposed algorithm is a minimal extension of recently proposed algorithm TD3+BC to leverages the offline/priviliged information for online deployment. 
main_review: **Strengths** 


**S1:** The paper outlines a previously under-explored and an important class resource constrained setup for offline RL. Furthermore they define the data collection strategy inline with this setup which was not present in previously explored / popular datasets such as D4RL.  
 

**S2:** They propose a minimal extension on TD3 + BC algorithm for the resource constrained settings and show that the proposed algorithm was able to close the performance gap due to the missing privileged information (I.e. the additional offline feature set).  

 

**S3:** With a few exceptions (discussed below), The experiments presented are well designed.   
 

 
 
**Weaknesses:** 
 
**W1:** The motivation behind the RC-D4RL dataset is not well articulated. As articulated in the paper, the available dataset D4RL is collected by a policy that has access to the privileged information, which in turn results in the behavioral policy having high quality trajectories. These high quality action choices should not be expected form a policy trained with no access to the privileged information. Hence it is important to collect the dataset with a resource constrained policy (no access to privileged information) in order to better quantify the advantages of being able to leverage privileged information.  
 
**Q:** While the proposed setting makes sense in imitating real world settings, would it not be useful to have high quality trajectories in order to best leverage the action choices that come from these high quality offline features. Why not stick with D4RL?  
**Q:** And if it indeed the case that D4RL does not represent real world settings why are we evaluating the given algorithm for both RC-D4RL and D4RL settings?  

 

 

**W2:** The paper seems to be missing some natural baselines. While the proposed algorithm is quite straightforward and is able to beat a blind agent (blind to privileged information) it would be interesting to see how it compares to other natural baselines such as a predictive model baseline. 

 

Predictive mode baseline: Train a model to predict the privileged information and use this in tandem with a offline RL model that is trained with the privileged information.  

 

Also If I understand it correctly, the baseline would simply by equivalent to Transfer(1.0, 0). Then the question becomes did we just propose an algorithm and just compared the different hyperparameter settings?  

 
**Q:** If a very natural baseline is comparable to the proposed algorithm, is the main contribution of the paper limited to the definition of the resource constrained framework?  

   

 

**W3:** More Datasets please: While this may come off as a "knee-jerk" reaction, I do believe that the class of problem defined here is important for the community. Hence it would definitely help if we are able to find real world examples of such scenarios and incorporate them here. Some new more natural additions on the dataset may include, (1) Atari benchmark dataset with semantic maps generated from internal bits. (2) D4RL dataset coupled with RGB frames, (3) Self driving / navigation dataset coupled with depth/semantic maps.




**Clarity Issues/ Comments:** 

 
**CL1:** While the paper concludes that the performance gap between the use of online features and offline features have been highlighted, it is not well quantified in the paper, how much does the overall performance suffer due to these missing offline features. I am guessing this would amount to teacher performances – baseline performance from fig 11. It would be nice to have them in the main paper.  

**CL2:** When we say we address the performance gap, it would be nice to produce numbers that represent exactly that. I.e. What percentage of the performance gap was the addition of offline features able to fill. 

**CL3:** The inverted U pattern mentioned for figures 4b and 5b are very hard to grasp, and basically nonexistent for 5b unless I am parsing the sentence incorrectly.  
**CL4:** Figure 6 does not fully answer the question of how "important" the teacher's role is in training the student, were we expecting Transfer(0.5,0.5) to perform worse that Transfer(0, 1)? How do we maximize the teachers role in training without hurting performance?  
summary_of_the_review: The framework introduced by the authors represent an important part of offline RL that remains under explored. The proposed algorithm is a minimal extension to a popular offline RL and the experiments/results are well motivated. However, at the current state I would lean towards rejection as the paper still lacks detailed investigation of possible approaches to solve this problem as well as results on a more compelling set of benchmarks(datasets).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers a novel offline RL setting with resource-constrained online deployment. In that setting, the agent has access to less information about the state in the online deployment phase than the available information from the offline dataset. It proposes a two stage training strategy under this setting, first training a teacher agent with full features, and then training a student agent with regularisation to mimic the teacher agent. Experiments are carried on in three D4RL environments with a variety configurations in the data collection protocol, behaviour policy, feature constraints. Results show the advantage of the proposed distilled policy than a simple baseline.
main_review: Motivation & significance: this paper introduces an interesting and novel offline RL setting and considers a simple modification to an existing algorithm for improved performance. It lists two real applications as the motivating examples. While it seems to be the real problem for the two examples, I doubt how general the proposed setting will applly in practice, and therefore how broadly the research in this setting will benefit to the RL community. The authors do carry out extensive experiments on the three environments from D4RL, but I am disappointed that no experiments are done for any real application or a simulated setting that resembles some property of either of the motivating examples.

Novelty: the proposed solution is quite simple and straightforward, adding an additional regularisation to a teacher policy with a higher quality. Therefore, there is very limited contribution to the methodology.

Clarity: the idea and experiments are very well presented in the paper. The algorithm design choice and experiment settings are explained in details and it is easy for reproducing the results.

Empirical evaluation: the experiments show the proposed method (both variants) is robust and provide improvement in most situations. I appreciate the authors makes a good effort to consider different experiment settings to get a comprehensive study about the proposed method and compare it to the baseline. However, I’m concerned that all the experiments are restricted to three simple simulated environments in D4RL and the results may not provide much evidence to how they will generalise to other environments or any real problems. Also, a very simple baseline is missing that trains a pure behaviour cloning policy to learn the teacher policy.

Overall, I think it would make this work much more impactful if the authors could consider a real problem / simulated problem similar to a real problem, and study different approaches and show what the best performance gain we could obtain against an existing solution.
summary_of_the_review: The new problem setting could be better motivated and supported by experiments. The proposed algorithm is a simple modification to the existing method.
review_title: null

Meta-review: The authors propose the resource constrained offline RL problem where the offline dataset contains extra features that are not available online. The goal is to use these extra features to improve performance during deployment. They propose a simple modification to TD3-BC in the continuous control setting and a simple modification to CQL in the discrete setting. They evaluate their proposed approaches on D4RL, RC-D4RL (a novel dataset that they introduce for resource constrained offline RL), Atari, and a proprietary real-life Ads problem.

Initial reviews identified the following concerns:
* While the exact problem is novel, the idea of having access to privileged features at training time that are not available at deployment has been explored in supervised learning and online RL. The reviewers were not clear how considering the offline RL setting interacts specifically with the privileged features to produce an interesting setting.
* The baseline simply trains on the limited feature set. Unsurprisingly, using the extra features can improve performance. In light of the previous point, reviewers asked for more substantial baselines, suggesting BC on the teacher and predicting the missing features as some possibilities.
* The set of tasks was too limited.

The authors provided a substantial response:
* Experiments on Ads data 
* Experiments on Atari with CQL as the base algorithm
* Additional baselines on RC-D4RL HalfCheetah-v2 datasets (BC on teacher and predictive)
* Additional analysis

I commend the authors on the hard work they did preparing this response. It is quite substantial and does improve the paper significantly. However, reviewers and I still have a number of concerns:
* The additional baselines are appreciated, however, the results are mixed. The additional baselines are a step in the right direction, but they need to be evaluated beyond a single dataset. It is hard to evaluate the results without reasonable baselines. I agree and think that even though the specific problem is novel, the idea of transfer learning is not, so it is reasonable to require that we have more extensive baselines. Furthermore, while the authors argue that their method has an edge on the more practical dataset, that is based on a very limited evaluation. Probing this further is important.
* The CQL modification is quite different than the TD3+BC modification. The performance of the modification for CQL is not significantly better than CQL. What should we make of this?
* For the Ads dataset, all hyperparameter settings except Transfer(0, 1) show the same performance. This seems surprising as even Transfer(0.1, 0.9) shows no difference. Finally, Transfer(0, 1) beating Transfer(1, 0) 7/10 times is not statistically significant.

At this time, the paper is not ready for publication, but the paper is moving in the right direction and I encourage the authors to submit a revised version to a future venue.
Decision: Reject