Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, authors formalized the problem of task-induced representation learning. It basically involves investigating various different ways for learning representations from the raw data (for ex. offline dataset) containing "privileged" information (task relevant) from many different tasks. The aim is to leverage task relevant information for learning representations that capture task-relevant information. 

The authors evaluate their idea on three different suite of environments (a) Distracting DMControl (b) ViZDoom (c) Autonomous Driving. The authors compare the proposed way of learning task-induced representation to various other methods for representation learning like reconstruction based approaches, mutual information based methods, prediction based methods etc. The results show that the proposed way of learning task-induced representations capture task-relevant information. 
main_review: Strengths of the paper:

- The paper is very well written.
- The authors evaluate the proposed way of learning task-induced representations to various other "task" agnostic ways of learning representations, and show that the proposed method is able to capture task-relevant information. 
- The ablations done in section 5.5 are very informative i.e., set of best practices while collecting datasets for learning task-induced representations 

Weakness:
-  For transferring information between different tasks, there are normally three different ways (a) by learning representations of the raw data (b) by learning policies/skills (c) by learning world models. In this work, paper compares to the proposed framework with the scenario when the information is transferred by learning representations of the raw data. It would be also useful to compared to methods/benchmarks which learn the representation of skills/dynamic models. 
- It may be useful to compare to methods which explicitly capture task relevant information like information bottleneck approaches [1, 2, 3].
- One other drawback is, that the paper only evaluates by modifying the visual perception by adding distractors. Since the paper uses dynamics model, it would probably be also meaningful to see as to what happens when some "irrelevant" dynamics are added during testing.

[1] Information asymmetry in KL-regularized RL, https://arxiv.org/abs/1905.01240
[2] InfoBot, https://arxiv.org/abs/1901.10902
[3] Learning invariant representations for reinforcement learning without reconstruction, https://arxiv.org/abs/2006.10742

After rebuttal: 

I've read the rebuttal. The paper makes it clear they are comparing to methods  that directly learn parametric model (either dynamics model or directly transfer policy)  dynamics models from the offline data (Pred-O and Pred-O+R in Figure 4) and others that directly transfer learned behaviors from the offline data (Policy transfer in Figure 4). I keep my original score.

summary_of_the_review: The reviewer  found the paper very well written, and tackling an important problem. I like the ablations done in the paper, and the focus on "constructing datasets" for learning task-induced representations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a representation learning approach for multi-task reinforcement learning. The idea is to pre-train a model using multi-task data either with rewards or behavior cloning. Then fine-tune it on a new task hopefully more efficiently. They contrast it to unsupervised representation learning.

Fine-tuning experimentation is performed with reasonable baselines on 3 different visual domains. The authors probe both what the representation looks like and what pretraining tasks give better performance.



main_review: strengths:
* experiments comparing with reasonable baselines
* paper is clear

weaknesses:
* i am not sure this paper addresses a real problem. In the limit of high empowerment e.g. human, an agent could achieve a very large number of goals and all potential parts of the screen should be relevant. In that sense this approach only makes sense when a precise goal is targeted and the task space in pre-training can be used to cover it.
* salience maps are not convincing to me. I am not sure what they are meant to convey and in which sense the TARP ones are better than other ones. Background / foreground prediction is also not very relevant in this context.

questions:
* for unsupervised representation learning methods are the unsupervised losses still optimized during fine-tuning. For a fair comparison they should be because it is possible and would make the claim of transfer stronger if they still don't get better results than TARP.
summary_of_the_review: The contribution and novelty are both very limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses the problem of learning meaningful representations that can efficiently focus only on the features that are relevant for downstream tasks. As opposed to unsupervised representation learning approaches that do not differentiate between task-specific and other information in a given dataset, the paper proposes a framework named “Task Induced Representation Learning” which leverages task information to guide the representation learning. Through this framework, the paper suggests three possible ways to induce task-relevance in representations: by predicting the value of a state, maximizing the discounted reward for a group of tasks, or by imitating an expert policy again for a group of tasks. Through a set of experiments in three domains: distracting DMControl, VizDoom and CARLA, the authors show higher task learning efficiency using the proposed framework as compared to standard approaches such as reconstruction, contrastive learning, state-reward prediction etc.  
main_review: Strengths:
1.	The paper presents an interesting discussion on task relevant supervision vs. purely unsupervised techniques. The empirical results are mostly in support of the proposed argument and contain diverse domains of tasks that the technique was tested on. 
2.	The paper follows up the main results with with an additional analysis and discussion of why TARP techniques perform better than the baselines, specifically, the role data plays in this performance boost. This analysis adds value by showing a closer look at the features and the kind of data that would result in better performance; and is valuable for the representation learning community in general.

Weaknesses:
1.	Firstly, the crux of the paper deals with a comparison between using task-based supervision to learn representations, and unsupervised representation learning techniques. While the argument that task supervision is more valuable compared to unsupervised learning is interesting, there is an obvious difference in the sense that task supervision directly allows for extracting task-relevant features in a representation. When unsupervised learning techniques do not account for any relevance, or there are no particular constraints as in techniques like beta VAE, it is obvious that those representations would not fare as well as a task-induced one in the absence of any additional supervision, constraints, hand crafted loss functions etc. In that way, it is not exactly an apples-to-apples comparison. This is made evident from the saliency map comparison, in the absence of any finetuning / pre-defined constraints, there is no way for an unsupervised learning method to know which features are important for an arbitrary task. 

This is why I feel that the discussion of novelty and the comparison with existing techniques is somewhat insufficient. For instance, it would have been more interesting and valuable to compare the proposed technique with algorithms such as DREAMER [1,2].  DREAMER also contains similar subgoals as TARP such as a value learning network. Although it is true that TARP proposes training value prediction or reward maximization on a ‘set’ of tasks as opposed to a single task, how is that better than, for instance, applying DREAMER itself to multiple tasks and expecting it to build a generalizable representation? Similarly, another relevant algorithm would be TIA (Task induced abstractions) [3] that proposes targeted reconstructions to solve issues with generalization as well as distractors and outperforms DREAMER. Comparing with other techniques in similar classes would help gauge the novelty and efficacy of the current method and localize it properly within the current literature. 

2.	Secondly, there are some parts of the evaluation I am not clear about. What is the main intuition for why the baseline representations are unable to focus on task relevance during the policy learning process? During the pretraining phase, an unsupervised model likely encodes both task-relevant and task irrelevant features in each scene – which gives rise to two possibilities. Either the model is failing to encode features in a disentangled way, which ruins performance during finetuning because it is not able to isolate task-relevant features from the others. If that is not the case, then it should be possible to isolate which of the encoded features are useful during finetuning for a task. If the baseline models can do the latter, then they should be applicable for diverse tasks similar to the way TARP is claimed to be. Hence, I think it would be very valuable to have some experiment that includes finetuning the representations as well while learning the policy. Speaking just conceptually, if we consider only a single task, it is not clear why TARP based learning is fundamentally different from finetuning a representation learnt in an unsupervised way for a given task, assuming that the representation is a well structured one. 


3.	I think the paper would also benefit from a discussion of how the proposed techniques could compare against the current increasing push towards unsupervised pretraining. Unsupervised pretraining through large “foundation models”, as well as reward-free pretraining are gaining traction in the world of representation learning and reinforcement learning, where specific tasks or rewards are not considered, and in cases were found to benefit downstream reinforcement learning or imitation learning [4]. In the current paper, the authors mention through empirical findings that it is more important to obtain (although less) data from a large number of tasks, as opposed to lots of data from few tasks. But this also means a lot of effort and handcrafting in setting up the tasks, the associated rewards and so on. 

Minor comments/clarification questions:

1. What is the downstream task corresponding to Figure 7? 
2. It is seen that the contrastive learning approach (ATC) performs really badly on distracting DMControl. How much of that is due to a lack of data augmentation? Would a contrastive learning approach perform significantly better if it was trained on a sufficiently large dataset with multiple distractors? 
3. In Figure 8, is there a significant difference in the size of datasets between expert and suboptimal?

[1] Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019). Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603.  
[2] Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020). Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193.  
[3] Fu, X., Yang, G., Agrawal, P., & Jaakkola, T. (2021, July). Learning task informed abstractions. In International Conference on Machine Learning (pp. 3480-3491). PMLR.  
[4] Yang, M., & Nachum, O. (2021). Representation matters: Offline pretraining for sequential decision making. arXiv preprint arXiv:2102.05815.
summary_of_the_review: This paper proposes a framework called Task-induced representation learning (TARP), with the claim that having task supervision during representation learning is a lot more beneficial than unsupervised, task-free representation learning. Moreover, TARP is able to use supervision from a set of multiple, perhaps diverse tasks so that the representations can benefit from different kinds of guidance and thereby encode features that are useful for downstream tasks. Through a set of experiments on DMC, VizDoom, CARLA etc. the authors show that TARP representations outperform usual unsupervised learning approaches. While the paper is well written and the discussion is interesting, it is well known that task induced supervision will induce more relevance compared to unconstrained, unsupervised learning, and I feel the paper is not strong enough because it does not compare to more relevant baselines such as DREAMER, Task Informed Abstractions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper attempts to answer the question of how to pre-train encoders in order to get task-relevant representations that are useful for a downstream task in each of three different environments (ViZDoom,  Distracting DMContro and Autonomous Driving). Three main approaches are compared (Behavioral Cloning, Value prediction only or another commonly used offline RL loss), along with a number of additional baselines (VAEs, predicting combinations of observations and rewards, contrastive methods and others), in a setting where an encoder is pre-trained offline on data coming from a certain task and its weights are frozen and used to provide representations to a policy head trained to solve an unseen task, that’s held out in the original dataset. The authors also provide an analysis using saliency maps to explore which parts of the input image are important for decreasing the loss of each method (task relevant vs background) and also show the performance of the representation on a task relevant and a task irrelevant task. Finally, the authors explore how the quality of the data (coming from many tasks vs coming from a single task) during the representation learning phase, and whether they come from an optimal, random or suboptimal policy, affects downstream performance. The main contribution of the paper, in my opinion, is this investigation of the different methods. In particular the comparison of performance in each of the three environments, the saliency analysis and predictability of state vs background information and finally the exploration of the effect of data on performance.
main_review: Advantages:

- I found the analysis combining saliency maps and state predictions vs background video classification interesting. It was nice to see the authors attempting to gain intuition by combining different techniques.
- The authors attempt to cover a number of interesting questions, from what should we predict, to how homogeneous or optimal the dataset should be. I found this interesting and informative.

Disadvantages:

- The authors only chose a single training-testing transfer pair of tasks from each environment type. Given how different the rankings of the performance of the methods are between each environment, I am curious whether there’s a high variance between different training-testing transfer pairs from the same environment. Though the results and narrative make sense, I am wondering whether the conclusions would hold in additional training-testing pairs within these three environments.
- I’m not clear what Section 5.4 and figure 6c show. What are the exact training-testing settings here? How do they differ from the experiments whose results are shown in figure 4?
- Though the authors attempt to answer many questions, they only provide analysis results on some of the methods. For example, how do saliency maps and state vs background predictions of every other method compare? Wasn’t it possible to undertake a similar analysis for non-TARP methods too?
- I recommend rewriting the contributions paragraph. Points 1 and 3 seem to be very similar and I don’t think the grouping provided by point 2 is necessary. Recognising that task relevant information is important for representation learning is a worthy contribution, but in my humble opinion “formalize task induced representation learning (TARP) as an alternative family of representation learning approaches that leverage task information from prior tasks“ is not really a separate contribution. 

summary_of_the_review: In summary, I think the paper is interesting, but importantly, I did not find many of the findings surprising enough to recommend for acceptance. There are additionally two main issues with the paper in its current form. First, the contributions section in the introduction should, in my opinion, be re-written (see disadvantages section above). Second, experimentally there are results on only a single training-testing transfer pair for each of three environments. I would expect a more thorough investigation of multiple of these pairs to assess the robustness of methods that make use of task information.
review_title: null

Meta-review: The paper is an interesting take on representation learning, using (prior) tasks to determine which information is important. The problem setting is somewhat difficult to pin down, so that that finding the correct comparisons is not obvious and opinions differ on many details of the setup. However, this is not a fault of the paper; it is a general problem the further one moves away from clean settings like classical supervised learning.

There was a lengthy and detailed back-and-forth between the authors and reviewers, where the authors clarified most of the points raised, extended their results, resulting in one reviewer switching from reject to accept.
Decision: Accept (Poster)