Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents an empirical study of language pre-training (GPT-2), image pre-training (iGPT), and training from scratch for offline RL. The study looks at the differences in activations, parameters, and gradients as well as the impact of the context length.
strengths_and_weaknesses: Strengths:
- The study considers an interesting question
- The experiments show interesting results (e.g., impact of context)

Weaknesses:
- The paper considers the number of aspects (e.g., activations, parameters, etc.) and reports interesting observations. However, it feels that it often stops too early and presents a hypothesis without trying to explore it further. For example, in the case of the gradient analysis the paper suggests that iGPT does not train well due to large gradients in early layers. It would be nice to explore this further and see if, e.g., this observation can inform a better training recipe for offline RL with iGPT pre-training.
- Writing and presentation are sometimes a bit hard to follow
questions: Suggestions:
- It would be good to confirm that the observed trends hold when using more than one seed

limitations: The paper discusses limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this work, the authors empirically investigate how pre-training on language and image data can affects fine-tuning of Transformers on offline reinforcement learning tasks. Previous studies reported that, for offline RL tasks, pre-training a Transformer-based model on image data either maintains or helps performance, whereas pre-training on vision data deteriorates performance. To understand the effect of fine-tuning, the paper compares between models that are randomly initialized, pre-trained with language data (GPT2), and with image data (iGPT). For that purpose they used GPT2 architecture. For offline RL tasks, the paper employs medium datasets of HalfCheetah, Walker2d, and Hopper environments. The authors analyze various network artifacts, like how layer activation and model weights change during training, initials gradient norms, also what type of language-trained model learns and utilizes for down-stream tasks, and others. The conclusion roughly is that, pre-trained model parameters do not change much during fine-tuning, though their layer representations change largely. Pre-trained models also do not acquire more knowledge about the input or the label during fine-tuning. The authors also hypothesize with empirical proofs that, the image-pre-trained model fails probably because of large gradients, and also that language pre-training probably gives the Transformer context-like information that can be taken advantage of during down-stream tasks.
strengths_and_weaknesses: Strengths:
- The topic investigated in this work is important and has room to be explored.
- The paper presents a well-written, detailed study.
- Limitations, mainly from the validity of the metrics used, is clearly pointed out.
questions: - Lines 181-182, "does not 182 encode more information about the input or the label". This may be because of limitation of mutual information, as the authors themselves pointed out in line 326. So should take the finding that, pre-trained Transformers do not acquire more knowledge about the input or the label during fine-tuning, with pinch of salt.
- In section 5.3, we can see parameters between pre and post-fine-tuning have changed for random initialization. Then in section 5.1, why representations for random initialization did not change much, i.e., the CKA scores for representations from random initialization are high?
- In line 263, I am guessing, fine-tune without context.
- In line 281, the authors noted that "other than a particular block". But by considering results from all three datasets, it is more like all blocks can contribute only in varying amount. That is, the observation is more in line with lines 303-304.
limitations: The limitations about the validity of the metrics used, is already pointed out in line 326.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an analysis of how the pretraining-finetuning paradigm works for Transformer models in the context of reinforcement learning. Pretrained Transformer models trained on two different modalities (GPT-2 for language, and iGPT for image) are applied to Mujoco offline RL tasks - Hopper, HalfCheetah and Walker2D. The paper examines how the representations change during finetuning - which is achieved by using several techniques such as activation similarity, distance between pretrained and finetuned weights, mutual information between training data and learned representations, gradient norms and magnitudes etc. Through this analysis, the paper uncovers some findings: primarily, how different language and image-based pretrainings fare during finetuning for RL, how different layers of the transformers exhibit different magnitudes of changes - demonstrating how pretraining knowledge trickles down to finetuning, and potentially why iGPT underperforms compared to the language GPT. It is also shown that the language GPT version achieves reasonable performance on RL tasks even under the case of constrained context length, thus highlighting the fact that it might already be aware of context-related information from pretraining. 
strengths_and_weaknesses: Strengths:

- The paper presents a very interesting and thought-provoking analysis of representation dynamics between pretraining and finetuning. Given the current interest in large foundational models being the starting point for perception and perception-action, the paper is a good contribution to the field, and can aid in understanding how large Transformers can transfer between different domains. 
- The paper identifies a set of relevant techniques to perform analysis of representation similarity, training dynamics through gradient analysis etc. which can be useful to apply to other large models as well. By applying these techniques to large Transformer models, the paper extends the findings in papers such as [1] to understand why or why not these models perform well at RL tasks. 

Weaknesses: 

- I believe the paper can benefit from a bit of reframing and rewrite. After reading through the paper, some of the initial points end up sounding a bit confusing. For instance, the paper states in several places that "pretrained models largely change their representation", and then "pretrained models do not change parameters much". It is a bit unclear exactly what is leading to what in terms of performance. 

1. In terms of parameters, for the pretrained models, it seems that the only significant changes are in the shallow layers, and to a lesser extent, in the layer norms (section 5.1, section 5.3). Whereas for the randomly initialized model, most of the changes are in the layer norms. Does this still explain the consistency of representation in the shallow to middle layers of random init? (section 5.1)

2. The analysis in 5.2, which investigates the amount of label information being encoded can benefit from a little more clarity. As it stands, it looks like the randomly initialized model does a better job at predicting the appropriate action given the states. Yet, random init does noticeably worse than GPT-2 on Hopper medium in table 2 or figure 9(a) (with K=20). It would be good to shed some light on exactly which objective ends up mattering the most for performance. 

3. In section 5.4, the paper hypothesizes that the fact that the gradient confusion is higher for the pretrained models, that makes them hard to train which explains the smaller changes in parameters. But this does not again seem to affect the performance in a detrimental way. 

- While I acknowledge that the analysis is meant build upon the findings from [1], one can't help but wonder that because reinforcement learning performance requires encoding state transitions, whether models that are trained on large corpuses of RL data (similar to the concepts in multi-game decision transformer, or Gato) would fare better at RL and shed light on pretraining-finetuning that's directly 

- The analysis being done in the paper is quite extensive and nuanced. It would also be useful to have a summary of the findings at the end of the results section that summarizes the main takeaway points from all the subsections combined, otherwise it can be hard to keep track. Similarly, the paper would benefit from a summary of the baseline results in the main text itself (moving Table 2 from appendix to section 5). 

[1] Machel Reid, Yutaro Yamada, Shixiang Shane Gu, "Can Wikipedia help Offline Reinforcement Learning?"
questions: - Have the authors experimented with changing the gradient norm clipping value for iGPT? It seems unlikely that it would cause a major shift in the performance of iGPT, but it might help with understanding the role of gradient magnitudes and gradient confusion in the final performance of these models. 

- Why does the random init model with no context (K = 1) perform significantly worse than the pretrained ones only in Hopper? For the other two environments, it actually performs a bit better than the pretrained ones. 
limitations: Mostly yes. The analysis could change a bit if the models were to be trained on a different corpus, such as not language or images, but rather trajectories from different RL tasks in a multi-task learning fashion.
review_title: null

Meta-review: metareview: The paper unanimously receives positive rates thanks to strong motivations and interesting results. As the reviews show satisfaction on the authors’ feedback, the final draft needs to respect it accordingly, for example, about the limitations of this research. 
confidence: Certain
Decision: Accept