Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies fine-tuning vs freezing in the context of reinforcement learning. The authors argue that the layers that represent general-purpose features(mainly the early layers) could be frozen while those that represent task-specific features(mainly the last layers) should be fine-tuned to achieve higher performance in the downstream task.
strength_and_weaknesses: Strength:

- The paper is well-written and easy to follow.

Weaknesses:

- I am not an expert in reinforcement learning. However, the argument in the paper in terms of representation learning does not seem convincing and sufficient as an ICLR paper to me.  First, the author's argument about freezing vs finetuning in the introduction(namely the second and third paragraphs) holds for any neural network and downstream task. Posing it specifically as an RL problem does not seem rigorous to me.  Second, as it is mentioned by the authors themselves, the fact that the early layers of a convent capture general-purpose features while the later layers represent more task-specific features, is known and widely accepted. In fact, the pioneering work on fine-tuning such as Fast-RCNN achieves higher performance when the early layers are frozen, and the last layers are fine-tuned.

I can not evaluate the performance, but  I am not entirely convinced about the sufficiency of the contributions in. this paper. 
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written. However, suffers from a lack of novelty as I discuss above.
summary_of_the_review: In my opinion, this is a well-written paper that suffers from a lack of novelty and convincing arguments. Please see the above for more detailed feedback.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies transfer learning in the context of Reinforcement Learning. Using 3 diverse domains as a testbed (a toy platform game, some tasks from the DeepMind Control Suite, and some navigation tasks from habitat), it makes the following contributions:
- Contribution #1: Shows that when transferring representations, the best strategy involves freezing the deepest layers and fine-tuning the others
- Contribution #2: Provides a strategy to select which layers to freeze and which to finetune
- Contribution #3: Introduces an auxiliary self-supervised objective to further enhance the transfer performance.



strength_and_weaknesses: All 3 contributions outlined above are more or less independent from each other, as such this section will be divided into three parts discussing the strength and weaknesses of each of them.

## Contribution 1

The paper starts by analyzing the performance of two main transfer strategies, namely freezing the representations or finetuning them. This is compared to the baseline of training the representations from scratch on the target task (strategy nicknamed "de novo"). In a nutshell, the results can be summarized as follows:
- For the toy jump task as well as the Deepmind control suite, finetuning outperforms "de novo" while freezing representations substantially decreases performance
- For the Habitat tasks, both strategies outperform "de novo" but none seems strictly better than the other.

The paper provides a thorough and solid analysis of the failure of the freezing of the representations for the jump task, linking it to robustness issues.
However, in my opinion, this falls short of making a broader conclusion: the results above suggest a meaningful qualitative difference between the Habitat task and the other ones (in Habitat, freezing works while it doesn't in the others). The reason for this difference is not discussed in the paper, yet it seems to be somewhat at the crux of the matter. Does this difference comes from intrinsic characteristics of the environments or from other factors?
As for the other factors, we can list 2 main ones: 
- Network capacity/architecture: The jump task uses the original DQN network, which is rather primitive (no skip connections for example) and has a tiny capacity (around 100k trainable parameters). By contrast, the Habitat task uses a state-of-the art CNN, with 28M parameters, ie two orders of magnitude larger.
- Pre-training tasks: The convnext for Habitat is pre-trained on ImageNet, while in the other experiments the backbones are pretrained on related control tasks.

In a nutshell, the networks used for Habitat have received a much more varied pre-training and have a much higher capacity. It is well known in the vision community (see [1] for example) that both these properties improve robustness and transferability of the features, making the results presented in the paper somewhat unsurprising.

Overall, it is not clear to me if the "frozen" transfer strategy would still fail in the two tasks where it currently does, if the same convnext, with the same pretraining, was used as the feature extractor. Depending on the answer to that question, a slightly different conclusion could be made: similarly to trends in other domains (Vision, NLP, speech, ...) does RL benefit from larger models pre-trained on large, general datasources?

Finally, I believe the paper could discuss a bit the alternate method to full freezing and finetuning that is being privileged more and more both for vision and NLP: freezing the pre-trained weights but adding a small amount of trainable parameters to allow "adpatation" of the learned representation to the new task. See [2], [3] and references therein.
Another thread of work of interest is the idea of using a different learning rate per layer (typically lower for the deepest layers and increasing from there). See [4] for an example.


## Contribution 2

The paper proposes to resort to linear probing of the feature of each intermediate layer to determine where the cut-off between frozen and fine-tuned should be made.
While the method seems sound, several design choices are not really explained:
- What is the justification behind reducing the dimensionality with PCA first, as opposed to simply using a linear layer directly on the features?
- The paper states that the probing is using the output of a action-value function network. Would it be possible to alleviate the need to train such a model by simply predicting an empirical estimate (eg the mean) of the expected discounted return in each state?


Figure 4.b (and similar figures in the appendix) would benefit from error bars since it's not clear what the variance is. Why does Fig. 11 not contain the estimates for the fine-tuned features?

Finally, the main weakness of the approach is its computational cost. Section 5.1 states "For each layer, we measure the action value estimation error of pretrained and finetuned representations, and only freeze the first pretrained layers that closely match the finetuned one". This implies access to the said fine-tuned representations, which in-turn implies that a full RL training is required as a first step. As such, this doubles the cost of the method (both in wall-clock time as well as sample complexity), significantly reducing its practicality. For proper comparison, it would be necessary to see whether the "Finetuned" baseline from figure 5 is still lower than the "Frozen+Finetuned" if the former is trained for twice the number of steps.
Perhaps, a "softer" scheme akin to [4] would do the job equally well while side-stepping the computational cost entirely.

## Contribution 3

The last contribution is a self-supervised auxiliary objective aiming at improving the quality of the representation. Though it is inspired from findings from contribution #1, it is a contribution that is not specific to the transfer setting and could potentially be applied elsewhere.

Here are some questions and comments related to this contribution:
- What is the motivation for the projection layer in the SSL loss? The choice is not ablated in the paper. I understand the need for such a layer in methods that do direct embedding supervision (either contrastive loss or simple L2 loss for example), but in this case the loss directly operates on the output distribution. Similar vision method like DINO [5] don't use such projector
- It would be valuable to provide "De Novo + Pisco" as a baseline to disentangle the gains from the transfer scheme from those due to the additional objective
- Though comparison to CURL is provided, SPR [6] has been shown to outperform it significantly, so a comparison seems warranted.




[1] "Big Transfer (BiT): General Visual Representation Learning", Kolesnikov et al, ECCV 2020

[2] "AdapterHub: A Framework for Adapting Transformers", Pfeiffer et al, EMNLP 2020

[3] "Learning multiple visual domains with residual adapters", Rebuffi et al, Neurips 2017

[4] "Universal Language Model Fine-tuning for Text Classification" Howard et al, ACL 2018

[5] "Emerging Properties in Self-Supervised Vision Transformers", Caron et al, ICCV 2021

[6] "Data-Efficient Reinforcement Learning with Self-Predictive Representations", Schwarzer et al, ICLR 2021


clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written and easy to read.
I believe most technical details required to allow reproduction are provided, with one exception: I could not find specifications on the data augmentation used.


summary_of_the_review: My main critiques for each of the contributions are as follows:

- The analysis of finetuning vs freezing representations misses the discussion on the capacity and pre-training diversity of the network involved, which in my opinion makes the picture incomplete
- The proposed strategy to choose which layers to freeze seems too costly, in its current form, to be broadly applicable
- The auxiliary objective is not studied in depth, be it with respect to its own design choices (eg the projection layer) or with respect to current state of the art auxiliary objectives (eg SPR)

As such, I don't recommend the paper in its current form for publication. That being said, all the aforementioned limitations seem fixable, and I am willing to increase my score if they are.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors compare finetuning and freezing pretrained representations on various RL tasks.
They seek to understand under what conditions each of these can be effective, and when one should be preferred to the other.
On a simple task where the optimal policy is known, they show that the pretrained representation does contain enough information to regress the optimal actions, but finetuning still fails to recover the optimal policy in spite of this.
They use insights from these experiments to propose a new self-supervised objective for RL, and evaluate this objective on a variety of tasks.

strength_and_weaknesses: Strengths:
1. The proposed objective, PiSCO, seems elegant and promising. I am curious if the authors could say more about the projector $h(\cdot)$ that is used in the PiSCO loss. The intuition for PiSCO makes sense, but I am not sure why why should compare $\pi(\cdot|h_1)$ and $\pi(\cdot|z_2)$ instead of simply $\pi(\cdot|z_1)$ and $\pi(\cdot|z_2)$?
2. The paper is well-written, and the authors attempt to be insightful in their discussion.

Weaknesses:
1. The narrative of the paper feels a bit incoherent. It feels to me that the freezing vs finetuning investigation did not well-motivate PiSCO, and that PiSCO does not address the original problem statement that motivated said investigation. From what I can tell, PiSCO is an auxiliary loss to be used during RL, not a pretraining objective. I think this paper's contributions would be more clear if either: (a) the authors proposed a pretraining objective, or (b) the authors more directly motivated PiSCO, e.g. by discussing robustness or data augmentation/efficiency.
2. Related to the previous point, if the authors were to consider option (b), then I feel that PiSCO could benefit from a more thorough experimental evaluation (a greater variety of tasks/domains, with more emphasis paid to sample efficiency).
3. The authors make broad statements about finetuning vs freezing solely based on results from a toy environment, MSR jump. I understand the attractiveness of that domain (in that the optimal policy is known), but I'm wondering if it would be possible to conduct a similar study in other domains as well?


clarity,_quality,_novelty_and_reproducibility: Clarity: the writing is easy to follow, but the high-level clarity is a bit lacking (see the "Weaknesses" section).

Novelty: to the best of my knowledge, the contributions are novel.

Reproducibility: the authors provide both algorithm pseudocode and their actual code in the supplementary material. The paper contains many experimental details.

summary_of_the_review: I think that the proposed approach shows promise, but the motivation is unclear (mainly per Weaknesses #1). I would feel comfortable accepting this paper if this were addressed (and the experiments/evaluation may need to change accordingly).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The goal of the paper is to study randomly initialized, fine-tuned and frozen backbones for a set of interactive tasks. Inspired by the study, a new self-supervised objective (Pisco) is proposed that provides better representations. 

The paper is not written in a coherent way and it also suffers from lack of clarity and consistency. For example, (1) the paper mentions that "We regress the distance and optimal actions with linear models from evaluation task observations, using a mean square error and binary cross-entropy loss". It is not clear what the training data is. It seems it is trained on test data. It is not surprising that the task performance is 100% using either type of representation. (2) Some experiments are performed using a subset of the environments, some are done on all environments, and some are later added to the Appendix during the rebuttal period. These are just a few examples of lack consistency and clarity. There are various other issues similar to these.

Overall, the paper has many interesting analyses, and some of the approaches used for probing are interesting. However, it suffers from lack of coherency and clarity. There are several other weaknesses such as: (1) Some of the conclusions are presented as a general conclusion while the supporting experiments are done in a toy environment. (2) The computational cost of the approach is high (reviewer nmYq). (3) Some of the conclusions are known in the vision or RL communities (reviewer ZmQR). Despite the interesting analyses, the paper needs a major revision to address the coherency and clarity issues. Therefore, the AC recommends rejection.

justification_for_why_not_higher_score: The paper needs major revision to make the story coherent and make the experiments clear and consistent.
Decision: Reject