On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
Authors: Shiro Takagi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Pre-training
Offline Reinforcement Learning
Transformer
Representation Analysis

Abstract: We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks. Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one. A closer look at the parameter changes of the pre-trained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping. To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information. Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize it to solve the downstream task.

On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning
Shiro Takagi
Independent Researcher takag146460gmail.. com
Abstract
We empirically investigate how pre-training on data of different modalities, such as language and vision, affects fine-tuning of Transformer-based models to Mujoco offline reinforcement learning tasks. Analysis of the internal representation reveals that the pre-trained Transformers acquire largely different representations before and after pre-training, but acquire less information of data in fine-tuning than the randomly initialized one. A closer look at the parameter changes of the pre- trained Transformers reveals that their parameters do not change that much and that the bad performance of the model pre-trained with image data could partially come from large gradients and gradient clipping. To study what information the Transformer pre-trained with language data utilizes, we fine-tune this model with no context provided, finding that the model learns efficiently even without context information. Subsequent follow-up analysis supports the hypothesis that pre-training with language data is likely to make the Transformer get context-like information and utilize to solve the downstream task.
Introduction
The past few years have witnessed the tremendous success of pre-trained Transformer-based models ① on versatile natural language processing (NLP) tasks. This because pre-training with a large corpus enables Transformer-based models to gain rich universal language representations transferable to vast downstream tasks 2 E. Remarkably, recent studies demonstrate that Transformers pre- trained with language data can efficiently solve even non-NLP tasks [5] 6
S g [C]] Because of its
versatility, has been suggested that Transformers pre-trained with large corpus could be called a kind of universal computation engines [5].
One area worthwhile attempting to study the benefits of pre- trained Transformer is reinforcement learning because the upplicability of pre- pre-training to this area still has room to be explored [1]21113 13 In this vein of research, Reid et al. studied the effect of pre-training of Transformers on offline reinforcement learning (offline RI.) tasks [9]. Their study demonstrates surprising contrast that pre-training with image significantly deteriorates performance, while that with language data does not negatively affect downstream tasks, rather it even improves performance for some datasets.
However, it is not yet known how the presence absence pre-training and its content leads to differences in the performance of downstream tasks. What information does/doesn't 't the model pre-trained on language data (language-pre gegguage--re-traate pre-ttrained model) leverage solve downstream tasks, and why does the model pre-trained on image data tinage-pre-fe pre-trained model) result in catastrophic performance? In this paper, we approach these questions by analyzing the inside of the Transformer
https:/t46.gihuu...o/
Code is available at hitps//tticooooooommmimiiiiiiiii
LaTeX source is available at http://swt..c.uooooooooeoeetenee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

that is tine-tuning to Mujoco [44 offline RL tasks. Analyzing the internals of Transformer, we study how randomly initialized, eeeguage-pre-trinaee and image-pre-traine pre-trained models acquire different representations, and what the differences could be attributed to. Our contributions are as follows:
We study the internal representation of pre-trained and randomly initialized Transformers and find that the pre-trained models largely change their representation, while they encode less information of data during fine-tuning.
The analysis of parameter change and gradient shows that the pre-trained models do not change parameters much perhaps due to gradient confusion and that the performance of the image-pre- trained model might be partly affected by gradient clipping the large gradient.
We examine context-dependence by fine-tuning models with giving no context information, Transformer efficiently learns even without context and
finding that the this could be the case by replacing just single Transformer block with pre-trained one.
Subsequent analysis of the change in how far away the model process the tokens of input supports the possibility that some contextual equivalent information has been positively transferred from language pre-training.
Related Works
The study of transferability of neural representation has been one of the most important research topics in machine learning research. A bunch of studies have investigated how pre- vre-training influences the downstream tasks, such as computer vision tasks 17 and NLP tasks [99 200 (2T). Recently, several studies have investigated the transferability of Transformers to various NLP tasks since understanding its high transferability is of great interest. Previous research reveals that Transformer representation encodes some syntax 22 semantics 2 cross-lingual information [24], and world
knowledge 255. Also, it has been pointed out that Transformers obtains not only language-speciie information but also more abstract inter-toke token dependencies pre-trained Transformers may apply to a wider variety of tasks than just NLP tasks.
[26 27 28 This finding suggests that
In recent years, studies have emerged that apply the Transformer to tasks other than NLP tasks, reporting the versatility of the Transformer even in such multi-mode modal cases. Such prior works have shown that language-pre-trained Transformers can efficiently learn image classification [5], symbolic mathematics (6), simple arithmetic 4 reinforcement learning tasks 7 [ and more generally, sequence modeling problems [5]. Our study is in line with the research that examines the applicability of pre-trained Transformers on reinforcement learning.
Most prior works applying pre-trained Transformer on reinforcement learning explicitly express goals, observations, or actions language 29 [Q] On the other hand, the work of applies pre-trained Transformers on the offline RL tasks that language is unrelated to (c.g. Mujoco still reports that pre-training is helpful. This study is interesting because suggests that language- pre-trained Transformers seem to exploit some similarity between the structure of natural language data and trajectory data, which is more abstract and fundamental than linguistic information. Also, exploring the applicability of pre-trained Transformers, which have already been successful, is of great engineering significance since pre-training offline RL has room for improvement compared to other modalities. Examining the internals of Transformers, we further delve into the findings from this study by Reid et al. and explores the applicability of Transformers to offline RI.
and
3 Background
3.1 Transformer
Transformers sequence-to-sequuene models composed of stacked identical blocks, called Trans- former blocks. Each block consists of self-attention [1. multi-layer perceptron, skip-connection 300, and layer normalization (layer norm) [BI. Denoting the linear projection of input sequence of length
n to query, key, values of length m by Q E Raxdo, K E Rmmxk,, and V E Rmxdy, respectively,
the self-attention is Attention(Q2 (Q, K, V) softmax (QK" V. We use GPTs 32 and hence the
Vdx
autoregressive language model, where queries head to only keys before their position.
2

---

3.2 Offline Reinforcement Learning and Decision Transformer
Reinforcement learning is the problem to learn to make optimal decisions in a dynamic environment. For ease of mathematical handling, the environment is usually descrihed as Markov decision process (MDP). The MDP consists of states S, actions a E .A, transition probability P(s' ss a), and reward function r(s, a): 8 R. The interaction of the agent in the environment is represented as K length trajectory, or sequence of state, action, and reward (80, dos ro, 81, 01, 11,71,-m ...,,. K ag, rk), where St, at, and 1t are state, action, and reward at time step t. The goal is find an optimal policy T (a) () to maximize the expected cumulative rewards E. - rc] for the trajectory.
Offline RL aims to achieve objectives of reinforcement learning from the trajectory data collected by some policy [33] Because offline RL is purely characterized by trajectory data, several studies have proposed formulating offline RL as a sequence modeling problem 34 335. Decision Transformer is a seminal work that attempts to use causal transformer (in other words, GPT architecture or decoder of BERT (2]) to solve such a sequence modeling problem [35]. In Decision Transformer, input data is a sequence = (Ro, 80: do, R1, 81, ...., RK, 8K ax) of length K, where return-to-go
R2 = - ri is the cumulative reward from time step t. The previous studies y 35 employ the above problem set up to discuss the applicability of Transformer for offline RL. We also follow these studies and use the setup for our analysis.
4 Experimental Setup
Unless otherwise noted, all of our experimental setups, including model configuration, dataset, and hyperaarameters follow the previous work [9] F
The sanity check that our trained models achieve comparable performance with the previous work is shown in Table ! Details are in Appendix A
We compare the model with GPT2 architecture t that is randomly initialized (randomly initialized model), pre-trained with language data (GPT2), and with image data (iGPT) study the effect of pre-training. Pre-trained models are from the Transformex: library from HuggingFace [36]. Like the previous study, the model code for GPT2 and iGPT are gpt2 and
For offline RL tasks, we use some of Mujoco tasks [14] in OpenAl Gym
37 provided by D4RL 38
Fl an offline RL dataset library. Specifically, employ medium datasets of HalfCheetah, Walker2d, and Hopper environments. Unless otherwise specified, we show the result for Hopper-medium. We
put the results for other environments in Appendix !
Following the previous study [. we train the models for 40 epochs, each of which consists of 2500 steps. In the following sections, we refer to the 40th epoch checkpoint the post-fine-tunile model and the Oth epoch model (the initial state) as the pre- pre-finetunning model. The training details, including hyperparameters and the optimizer, detailed in Appendix B
Table 1: Normalized mean return.
Dataset
Environment
GPT2 (
iGPT (
DT (9)
GPT2 (ours)
iGPT (ours)
Random Init (ours)
Hopper
1..11:1.1
5.7
67.6
79.5 19.5113 1.3
2.6 0.3
67.7 3.0
Medium
HalfCheetah
2.18
1.
42.6
48.4
1,3
48.6 0.2
Walker 2D
78.3 1.5
0.4
74.0
71.2.1..3 1.3
4.3 1.9
71.0 0.5
Results and Analysis
5.1
Activation Similarity
We first study how the pre-trained and randomly initialized models shape the representation in fine- tuning. To that end, we compare the centered kernel alignment (CKA) [39 of each layer's activation between pre and post-finettuinn, investigating how each layer's 's representation changes by learning
We use the following code for experiments: http://ghhhh..oooooooccocllaaaah https///ittub..om/aaa http://bbb.oomaaiibeeeeeeell/aaa
3

---

offline RL data The CKA has been used in various studies and has provided numerous insights into the understanding of neural networks 40
44 In particular, we compute the linear CKA, for a subset of the dataset. We use the activation of each layer from the last time step of the context for return-to-go, state, and action, respectively. We show the result of state, putting the results for the others in Appendix [.T] Following the previous study [%] we compute CKA of activation not only for Transformer block output but for all internal activation of the Transformer blocks. The definition of the CKA and the details are in Appendices D.2 and D.I
12
following previous studies 39 45
Randen at
Randon
Randoo
GP72
GPT2
80
60
03
100
20
40
60
100
20
190
(a) Hopper
(b) HalfCheetah
(c) Walker2D
Figure 1: CKA similarity of each layer between pre and post-fine- tosttiin-tuuing.
Fig. shows the result. The y-axis is each layer's CKA between representation at pre and post-fine- tuning. The x-axis the normalized layer index: because the number of layers of iGPT (216) is twice the others (108), we average CKA over two adjacent elements starting from the Oth layer for iGPT and align the x-axis for different models for comparison. Higher CKA means higher similarity.
We observe that the CKAs of pre-trained models (GPT2 and iGPT) are uniformly low across layers except for few layers, indicating that pre-trained models largely change representation. On the other hand, for the randomly initialized model (Random Init), the representations have a higher similarity to the initial state from the middle to the shallow layers. Also, the CKAs of the randomly initialized model seem to be higher than pre-trained models across layers. These observations indicate that the randomly initialized and pre-trained models structure their representations in very different ways. Further support for this conclusion is shown in Appendix D.4
To identify which layers have relatively higher CKA for the randomly initialized model, we obtain layer names with CKA values above a threshold (dashed line at 0.23 in Fig. (a)). Then, we find that most of them are layer normalization layers (the list of the layer names are put in Appendix D.3). Therefore, we can speculate that the difference in CKA similarity of layer normalization can be related to the effect of pre-training. This possibility could be supported by previous research [ that reports that layer norm parameters are the most important to fine-tune.
In uni-modal cases, where the statistical nature of the input data does not change significantly, previous studies have a consensus that the shallow layer acquires a more universal representation and the deep layer acquires a task-specific representation for wide range of architectures and tasks
42 44 The finding that the language-pre-ttn -reateine Transformer performs as well as better than the baseline but largely changes the representation suggests that in multi-modal cases, something different may be happening in forming the representation than in prior studies.
46
50
51
5.2 Mutual Information Between Hidden Representation and Data
In the previous section, we observe that pre-trained models drastically change their internal represen- tation during fine-tuning, attaining largely different representations. One possible consequence of this observation that the pre-trained model may have adapted better to the data in the downstream tasks. To explore this possibility, we calculate estimated values of mutual information between hidden representation and data and compare them among different models. Mutual information is a well-known measure of the mutual dependence between the two random variables. Many previous studies have used this metric to investigate the extent to which neural representation encodes input and label information and provided profound insight into how neural networks work 55 53
55 56
WW use the following code to compute resarcchhre/maaseeeqgeesssnaotot tttee/sssssstttttttiiiiititt 39
CKA:
https//atthbbbooooooogerrerreggo

---

In particular, we calculate the estimated mutual information IT);T) between some layer representa- tion I' and input data X, and that iY; (") between T and label Y. We employ Mutual Information Neural Estimation (MINE) [57] to estimate mutual information since this method can estimate mutual information
when random vectors are continuous and have different dimensions f Since Decision Transformers are trained to minimize the gap between predicted action d and actual action ae, we define the label as Y = a₁ Unlike non-sequential models, the causal Transformer exploits past context information. Thus, denoting the input token except for ak by Dt and the internal activation of Transformer block ( of the input by Tix we compute ( (xx)) as I(;;7) 2') and xax:: (x;)) as [(Y; for all token positions, t = 0, O.... K. We calculate these values for the shallow, middle, and deep Transformer blocks, respectively. We show the result of the middle block here and leave the remaining in Appendix 22 The other details are described in Appendix
Random
Random
GPT
IGPT
40
60
90
10
40
60
90
Context
Context
(a) I(X;T)
(b) TT
Figure 2: Estimated mutual information between data and hidden representation.
Fig. D is the result, where Fig. 2 (a) and (b) are results of ((::) and i(Y: :T), respectively. The markers triangle, circle, and cross represent return-to-go, state, and action, respectively. The x-axis the position of tokens in the context and the y-axis is the estimated value of mutual information.
We observe that the randomly initialized model consistently has higher mutual information both for input and label than pre-trained models. Given the catastrophic performance of iGPT, it is understandable that iGPT representations have almost information. The result comparing GPT2 to random initialization indicates that the changed representation observed in Section B.J does not encode more information about the input or the label than the randomly initialized model. That is, the Lunguage-pre-traiiee pre-trained model does well probably not because it acquires as much or more information about the data the random initialization. This result suggests that although the representation has changed, there remains something unchanged, and the language peruuge-----aeee pre-trained model may leverage it to solve the offline RL task. Note that this might be affected by limitations of mutual information.
We also notice that for label the maaguage-pre-rriingg model has consistently higher mutual infor- mation for a specific token type (return-to-go), while that is not always the case for the randomly initialized model. This result implies that the of specific token type each layer for prediction.
model may process the information
5.3 Parameter Similarity
In the previous section, we discuss the possibility that the language-pre-trainn Transformer exploits some pre-acquired information. To investigate this issue on a finer scale, we turn our eye from representation to parameter. We conduct parameter- similarity analysis and investigate to what extent parameter change between pre and post-fine- see--finetuniin tine-tuning. In concrete, we compute 12 distance and cosine similarity of parameters between pre and post-fine-tuning The details are in Appendix F.I
The result for 72 distance is shown in Fig. 3 and for cosine similarity Fig. where the top the result for GPT2, the middle is for iGPT, and the bottom is for random initialization. The x-axis is the parameter set index, where, from left to right, the ticks correspond to the shallowest to deepest layer parameters. For visibility, we do not show the labels for ticks and put them in Appendix
'e use the following code for mutual information estimation: http:://giiubboooom/gggeerrrnnnp
5

---

GP3
10
Pansrise of Lach Lawe
Figure 3: 12 distance of each parameter between pre and gost-fine-tuning.
cockmok contrri..
Figure 4: Cosine similarity of each parameter between pre and post-fine-tuning
Overall, We observe that [2 distance is small and cosine similarity is large for pre-trained models, meaning that the parameters of pre-trained models do not change that much. At first glance, this appears to contradict the results of Section 5.T However, prior research has reported that most the information in Transformers propagates through skip connections 44 This means that, as long as the parameters of the shallow layers change, the representation each layer may change significantly without changing the parameters of each layer. We indeed observe that the shallow layer 72 distance of the pre-trained model shows relatively large changes. The existence of unchanged parameters means that knowledge acquired through pre-training could be stored in those parameters.
5.4 Gradient Analysis
In Section 5.3 we observe that pre- traiied models' parameters don't change that much. We dig into this issue, investigating why GPT2 and iGPT do not change parameters. To this end, we compare the gradient norm and gradient confusion [58] the early phase of the training (1st epoch), discussing the training difficulty. When parameter gradients of loss for two different samples are negatively correlated, we say that there is gradient confusion. Empirically, we can measure this by the minimum gradient cosine similarity among all pairs of gradients: when the value is close to 0, gradient confusion low. The previous study showed that training is casier when gradient confusion is low [58]. Because gradient clipping is used in the previous work O and this study (the maximum norm is 0.25), compute the values for clipped gradients. The details are described Appendix G.I
Fig. 5 is the result for the gradient confusion and Fig. i that for gradient norm. Fig. shows that the minimum cosine similarity of iGPT and GPT2 is similarly smaller than the randomly initialized model. This high gradient confusion probably makes pre-trained models harder to train, which explains why they change parameters less than the randomly initialized model. Fig. 3 on the other hand, shows that the gradient norm of iGPT is concentrated around the clipping threshold (0.25), while that of others is below the bound. That is, the differences the magnitude of the gradient collapsed for iGPT, reducing the informational value of the gradients.

---

Mrnu cosdiriaall (u6))
0.20
0.22
9.18-
2.1
6972 1IP7
Zondon ki
G972
Rordon
Oriden Ride
Figure 5: Grad. confusion.
Figure 6: Grad. norm.
Figure 7: iGPT's grad. norm ratio.
To identify which parameter of iGPT dominates the gradient norm, compare the gradient norm of each parameter set. We calculate the gradient norm ratio of weights and biases all parameters in all Transformer blocks. The result is shown in Fig, for visibility, first layer norm module (weights 0. 0.In_1. 1n_gt.weight and biases O. 0.1n_1 bias) and lump the remaining
show the parameters of the
results together others. The full result is shown in Appendix G.2 We observe that weights and biases of the first layer norm module dominate the norm. This result implies that the difficulty of training iGPT might partially come from the of gradient clipping large gradients dominated by few parameters. A complementary analysis is described in Appendix G.3
5.5 Fine- Tine-Tunnig with No Context Information
In the sections up this point, we find the possibility that the language-pan peee--rreeerreeeee model may exploit some pre-acquired information. Then what information does the language-pres pre-trained model exploit to solve the tasks? One possibility is the ability handle context because Transformer skilled at handling the relations of tokens, which enables it to perform well in variety of NLP tasks [22]. Therefore, we train the language- pre-trained and the randomly initialized model with no access to the context (K 1) dig into this possibility. If model learns efficiently without context, the model is already likely to have context-equivalen information. Details of experiments are in Appendix H.]
language-pretrained
2000
6000
4000
Meen DORE 2000
Mean Retmr
1000
GPTS
1000-
GPT2
1000-
Krt
600
Plancors KX23
Planco Roecon 18t Kn!
Planices K-22
Reecon bot dert
g-
Rancoes oet Krn
Esoch
Eacoh
Eacoh
(a) Hopper
(b) HalfCheetah
(c) Walker2D
Figure 8: Mean return throughout fine-tuning when access to the context information is prohibited.
The mean return throughout training time is shown in Fig. and the normalized scores of return are shown Table 2 The solid lines in Fig. are results without context and the dashed lines for reference
Table 2: Normalized return of 1.
Dataset
Environment
GPT2
Random Init
are those with context. From Fig. S we observe that when the context is not provided (K = 1), the GPT2 reaches the high mean return much faster than the randomly initialized model. Furthermore, Table g reveals that the randomly initialized model fails to learn the trajectory for a dataset (Hopper), while the language-prg traitrained model consistently achieves comparable performance with that of the bascline result (K = 20). That is, we observe that the innguage-pre-trimee performs more efficiently when context is provided. This is somewhat surprising, given that previous studies have pointed out that Decision Transformer performs significantly worse with no context 55 The result that the language-pres pre-triined Transformer learns efficiently as well without
Hopper
83.73
-0.24
Medium
HalfCheetah
47.7
49.0
Walker 2D
69.0
69.1
Transformer consistently
FFlllowing the previous studies
B8
35
report normalized scores: 100 X experi
7

---

context as it does with context suggests that the model may have already acquired some kind of context-like information through pre-training and is utilizing it to solve the task.
5.6 More In-Depth Analysis of Context Dependence
5.6.1
Replacement by the Pre- Pre-Trained Block
In the previous sections, we find that the language- manguage-pre-trainee pre-trained Transformer exploits context-related information solve the task. To identify which block contains the useful information, we conduct block replacement experiment, where we replace Transformer block of the randomly initialized model with that of the language-pre- preatgieep-- model and then fine-tune the model without context. replacement of a block does not help training, the block probably did not learn effective information pre-training. Because we observe in Section 5.5 that the early phase characterizes the difference of the models, we focus on the epochs up to 10th epoch. We explain the other details in Appendix [.I.]
1.2
b 0
1500
Mock 0
1.0
0o0
1000
Nock
0.6
5 0000
750
Nock 5
900
0.2
BCX 10
00X 9
250
0
Back 10
Nock
3
10
001
2
10
Mock
Epoch
GPT2
Epoch
GPT2
(a) Action error
(b) Mean return
Figure 9: Learning curve when a single Transformer black is replaced with a pre-trained one.
The result is shown in Fig. 2 where (a) is the prediction error of action and (b) is the mean return To get rough trend, we take the exponential moving average. We observe that only when block is replaced with the pre-trained block, does the action error (Fig. P (a)) and the mean return (Fig. 9 (b)) behave similarly the randomly initialized model. Thus, we conclude that all blocks other than block contain context- -nuiix--qqivalennt useful information to solve tasks. The findings that replacing single block with the pre-trained one improves learning efficiency and that effective information is distributed across layers alone are surprising enough. This could be related to the dominant role of skip connections for information propagation
[ or the nature of non-monotonic change in the content each layer processes 20 in Transformers. We show the result for Hopper since the trend explained above most pronounced. For other environments, although the similarity between the random model and block1 is much weaker, the observation that block is not good is consistent. Results for other environments are in Appendix U.6.
5.6.2 Attention Distance Analysis
In previous sections, we find that the information improving learning efficiency is preserved in all blocks only in the varying amount and that is probably context-related. To further confirm that the context-like information is indeed preserved,
investigate how the model processes context by calculating the attention distance (59 of each Transformer block for the result of Section 5.3 Attention distance is the average distance between key and query weighted by attention weight. Previous studies successfully used this measure to reveal how locally Transformers process context Attention distance will elucidate how far away each model processes the tokens of input.
43 59
In particular, we compute the gap (epor(ppoch) - datt (0)) between attention distance at the initial state (0)) and that at an epoch (dau (Epoch)) to study to what extent the model preserves the way to utilize context. the gap is smaller, it means that the attention distance is more preserved. To clarify the difference between the language- peeerage-----tteee trained model and the randomly initialized model, compare the gap between epoch 0 and 4 since at epoch 4 for all environments the randomly initialized
Note that since the medium dataset we use is corrected by sub-optimal policy, a better action error does always result in better return. We explain this more in detail in Appendix O
8

---

model still produces low return, while the GPT2 achieves high return (Fig. setup and results for other epochs are shown Appendices I.2.
The detail of the
J.6.2
2712
0.30-
GP12
0712
0.25-
0.16
0.00-
0116-
0155
0010-
0100
0:00
0.00
Tranfoome Slock
Tranforrer Stock
Trandormr Block
(a) Hopper
(b) HalfCheetah
(c) Walker2D
Figure 10: Attention distance gap between epoch 0 and epoch 4.
Results shown in Fig. TO where the x-axis is the Transformer block index and the y-axis is the gap between attention distances. Fig. T0 (a), shows that the attention gaps of GPT2 except for block are small, indicating that attention distances are kept from the pre-training in these blocks. Taken together with the finding of Section 5.6.1 that blocks other than block 1 improve learning efficiency, these preserved attention distances likely positively affect fine-tuning. This result further supports the hypothesis that language language-pre-trainng pre-training brings the model context-lic -ike information and the use of this information allows efficient learning.
We also notice the outstanding contrast that gaps of unguage-pre-trainee models are almost zero except for block (and block for HalfCheetah), while those for the randomly initialized model large. This trend is clearer from middle to deep blocks (3 to 10) for all environments. Turning an eye Fig. (a), notice that the action error of these blocks decreases faster than the other block (0 to 2 and II). Thus, we conclude that the pre-trained context-like information could help the training by enabling the model to predict action more accurately.
We summarize findings of Section S with particular attention to the implication for performance:
5.1: Re-using representation is not the cause of the good performance of GPT2.
5.2: Fitting to data better is not the cause of the good performance GPT2.
5.3: The good performance GPT2 might come from some unchanged parameters.
5.4: Gradient clipping and large gradient might be cause of the bad performance of iGPT.
5.5: GPT2 can learn efficiently even without the context provided.
5.6.1: Even single pre-tree pre-trained Transformer block makes training more efficient.
5.6.2: cause of the good performance of GPT2 is pre-acquired way use context.
6 Discussion
Conclusion We have examined how pre-training on data of different modalities influences fine- tuning to offline RL. Internal representation analysis shows that pre-trained Transformer largely changes its representation while attaining less knowledge of the downstream task. By the analysis of change in parameters, we find that pre-trained Transformers do not change parameters that much and that the bad performance of the image- ponge--re-uiee pre- tre-trriied model might partially come from gradient clipping on the large gradient dominated by few parameters. Fine-tuning models with no context, we find that the aaguaage-pre-rre pre-trained model can efficiently solve offline RI. tasks even without context. Follow-up analysis supports the hypothesis that language pre-training probably gives the Transformer context-like information and the model exploits it to tackle the offline RI. tasks.
The finding Section 5.6.2 of the usefulness middle blocks is interesting, given that the contextual model's 's middle layers contain syntactic information [60] 61l and are most transferable 20 This implies that the syntactic capability of Transformers could be related to the performance on offline RI. The role of layer norm is also noteworthy we observe that it may characterize the behavior in fine-tuning (Sections 5. and 5.4). Studying the pre-training on other RI. tasks is also critical to understanding the benefit of pre-training in RL. Many studies of human evolution and
Discussion

---

Hence, studying development point to a close relationship between language and behavior 62 63 the relationship between these modalities matters create human-like intelligence that handles language. A more detailed analysis of this relationship is promising research direction.
Limitation and Impact We consider only a random seed and a few datasets from OpenAl Gym for analysis. Thus, using diverse data is important and checking if the claim holds with the average results with many more seeds is crucial. Also, the validity of the metrics we used should be critically examined since different results could be obtained by using more carefully designed ones. Because this work fundamental research, no immediate serious negative societal impact expected.
Acknowledgments and Disclosure of Funding
We thank DEEPCORE Inc. for providing the computational resource.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[3] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
[5] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.0222 2021.
[6] Kimia Noorbakhsh, Modar Sulaiman, Mahdi Sharifi, Kallol Roy, and Pooyan Jamshidi. Pretrained language models are symbolic mathematics solvers too! arXiv preprint 1rXiv:2110.33001 2021.
[7] Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang. Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01777 2022.
[8] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference Machine Learning, 2022.
[9] Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv:2201.12//1 2022.
[10] Allison C Tam, Neil C Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanic CY Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. arXiv preprint arXiv:2204.05080. 2022.
[11] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. International Conference on Learning Representations, 2021.
10

---

[12] Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential decision making. In Proceedings of the 38th International Conference on Machine Learning. 2021.
[13] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Proceedings of the 38th International Conference Machine Learning, 2021.
[14] Emanuel Todorow, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
[15] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, 2014.
[16] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[17] Yifan Ding, Liqiang Wang, and Boqing Gong. Analyzing deep neural network's s transferability via frechet distance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[18] Romain Orhand, Hiba Khodji, Amarin Hutt, and Anne -ainiinnr Quantification of the transferability of features between deep neural networks. Procedia Computer Science, 192:138-147. 147, 2021.
[19] Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, and Zhi Jin. How transferable are neural networks in NLP applications? In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, 2016.
[20] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[21] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What know about how bert works. Transactions of the Association for Computational Linguisties, 8:842- 866, 2020.
[22] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations, 2019.
[23] Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistcc diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48, 2020.
[24] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, July 2020.
[25] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) 2019.
[26] Ryokan Ri and Yoshimasa Tsuruoka. Pretraining with artificial language: Studying transferable knowledge language models. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), 2022.
11

---

[27] Cheng-Hau Han Chiang and Hung-yi Lee. On the transferability of pre-trained language models: A study from artificial datasets, In Proceedings of the AAAI Conference on Artificial Intelligence. 2022.
[28] Ethan A. Chi, John Hewitt, and Christopher D. Manning. Finding universal grammatical relations in multilingual BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, 2020.
[29] Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and explore: Language models for action generation in text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[30] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[31] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16x1::1607.06400 2016.
[32] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAl blog, 2018
[33] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164: 2020.
[34] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[35] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, 2021.
[36] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in System Demonstrations, 2020.
Natural Language Processing:
[37] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 000.//0000..5. 2016.
[38] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[39] Simon Komblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[40] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Representationss 2020.
[41] John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Similarity analysis of contextual word representation models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[42] Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? In Advances in Neural Information Processing Systems, 2020.
[43] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovit- skiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, 2021.
12

---

[44] Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic for- getting: Hidden representations and task semantics. In Proceedings of the 38th International Conference Learning Representations, 2021.
[45] Thao Nguyen, Maithra Raghu, and Simon Korbliihh. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference on Learning Representations, 2021.
[46] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl- Sohl-Dickstein. Sveca: Singular vec- tor canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, 2017.
[47] Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, 2018.
[48] Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representation, 2018.
[49] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems, 2019.
[50] Elena Voita, Rico Sennrich, and Ivan Titov, The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) 2019.
[51] Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020.
[52] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), 2015.
[53] Ravid Shwartz-Zii and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.0088 2017.
[54] Hassan Kafez-Kollhi and Shohreh Kasaci. Information bottleneck and its applications in deep learning. arXiv preprint S.K3v:19044037453 2019.
[55] Ziv Goldfeld and Yury Polyanskiy. The information bottleneck problem and its applications in machine learning. IEEE Journal on Selected Areas in Information Theory, 1(1):19-38, 2020.
[56] Bernhard C Geiger. On information plane analyses of neural network c IEEE Transactions on Neural Networks and Learning Systems, 2021.
[57] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference Machine Learning, 2018.
[58] Karthik Abinav Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparameteriizatio on gradient confusion and stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning, 2020.
[59] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 1021.
13

---

[60] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), 2019.
[61] Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.0228: 2019.
[62] K. S. Lashley. The problem of serial order in behavior. Cerebral Mechanisms in Behavior; The Hixon Symposium, pages 112 112-1466, 1951.
[63] Patricia M Greenfield. Language, tools and brain: The ontogeny and phylogeny of hierarchically organized sequential behavior. Behavioral and Brain Sciences, 14(4):531-551, 1991.
[64] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning. 2018.
[65] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[66] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. In Advances in Neural Information Processing Systems, 2007.
[67] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Appendix B
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Appendix K
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All existing assets we used are mentioned footnote References.
(b) Did you mention the license of the assets? [Yes] Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [Yes] Appendix !
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "j43DHndu18D", "reviewer": "Reviewer_mvwR", "summary": "The paper presents an empirical study of language pre-training (GPT-2), image pre-training (iGPT), and training from scratch for offline RL. The study looks at the differences in activations, parameters, and gradients as well as the impact of the context length.", "questions": "Suggestions:\n- It would be good to confirm that the observed trends hold when using more than one seed\n", "limitations": "The paper discusses limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The study considers an interesting question\n- The experiments show interesting results (e.g., impact of context)\n\nWeaknesses:\n- The paper considers the number of aspects (e.g., activations, parameters, etc.) and reports interesting observations. However, it feels that it often stops too early and presents a hypothesis without trying to explore it further. For example, in the case of the gradient analysis the paper suggests that iGPT does not train well due to large gradients in early layers. It would be nice to explore this further and see if, e.g., this observation can inform a better training recipe for offline RL with iGPT pre-training.\n- Writing and presentation are sometimes a bit hard to follow", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PWjgJYjylPh", "reviewer": "Reviewer_JfVS", "summary": "This paper presents an analysis of how the pretraining-finetuning paradigm works for Transformer models in the context of reinforcement learning. Pretrained Transformer models trained on two different modalities (GPT-2 for language, and iGPT for image) are applied to Mujoco offline RL tasks - Hopper, HalfCheetah and Walker2D. The paper examines how the representations change during finetuning - which is achieved by using several techniques such as activation similarity, distance between pretrained and finetuned weights, mutual information between training data and learned representations, gradient norms and magnitudes etc. Through this analysis, the paper uncovers some findings: primarily, how different language and image-based pretrainings fare during finetuning for RL, how different layers of the transformers exhibit different magnitudes of changes - demonstrating how pretraining knowledge trickles down to finetuning, and potentially why iGPT underperforms compared to the language GPT. It is also shown that the language GPT version achieves reasonable performance on RL tasks even under the case of constrained context length, thus highlighting the fact that it might already be aware of context-related information from pretraining. ", "questions": "- Have the authors experimented with changing the gradient norm clipping value for iGPT? It seems unlikely that it would cause a major shift in the performance of iGPT, but it might help with understanding the role of gradient magnitudes and gradient confusion in the final performance of these models. \n\n- Why does the random init model with no context (K = 1) perform significantly worse than the pretrained ones only in Hopper? For the other two environments, it actually performs a bit better than the pretrained ones. ", "limitations": "Mostly yes. The analysis could change a bit if the models were to be trained on a different corpus, such as not language or images, but rather trajectories from different RL tasks in a multi-task learning fashion. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper presents a very interesting and thought-provoking analysis of representation dynamics between pretraining and finetuning. Given the current interest in large foundational models being the starting point for perception and perception-action, the paper is a good contribution to the field, and can aid in understanding how large Transformers can transfer between different domains. \n- The paper identifies a set of relevant techniques to perform analysis of representation similarity, training dynamics through gradient analysis etc. which can be useful to apply to other large models as well. By applying these techniques to large Transformer models, the paper extends the findings in papers such as [1] to understand why or why not these models perform well at RL tasks. \n\nWeaknesses: \n\n- I believe the paper can benefit from a bit of reframing and rewrite. After reading through the paper, some of the initial points end up sounding a bit confusing. For instance, the paper states in several places that \"pretrained models largely change their representation\", and then \"pretrained models do not change parameters much\". It is a bit unclear exactly what is leading to what in terms of performance. \n\n1. In terms of parameters, for the pretrained models, it seems that the only significant changes are in the shallow layers, and to a lesser extent, in the layer norms (section 5.1, section 5.3). Whereas for the randomly initialized model, most of the changes are in the layer norms. Does this still explain the consistency of representation in the shallow to middle layers of random init? (section 5.1)\n\n2. The analysis in 5.2, which investigates the amount of label information being encoded can benefit from a little more clarity. As it stands, it looks like the randomly initialized model does a better job at predicting the appropriate action given the states. Yet, random init does noticeably worse than GPT-2 on Hopper medium in table 2 or figure 9(a) (with K=20). It would be good to shed some light on exactly which objective ends up mattering the most for performance. \n\n3. In section 5.4, the paper hypothesizes that the fact that the gradient confusion is higher for the pretrained models, that makes them hard to train which explains the smaller changes in parameters. But this does not again seem to affect the performance in a detrimental way. \n\n- While I acknowledge that the analysis is meant build upon the findings from [1], one can't help but wonder that because reinforcement learning performance requires encoding state transitions, whether models that are trained on large corpuses of RL data (similar to the concepts in multi-game decision transformer, or Gato) would fare better at RL and shed light on pretraining-finetuning that's directly \n\n- The analysis being done in the paper is quite extensive and nuanced. It would also be useful to have a summary of the findings at the end of the results section that summarizes the main takeaway points from all the subsections combined, otherwise it can be hard to keep track. Similarly, the paper would benefit from a summary of the baseline results in the main text itself (moving Table 2 from appendix to section 5). \n\n[1] Machel Reid, Yutaro Yamada, Shixiang Shane Gu, \"Can Wikipedia help Offline Reinforcement Learning?\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "549CGljMccL", "reviewer": "Reviewer_GdNs", "summary": "In this work, the authors empirically investigate how pre-training on language and image data can affects fine-tuning of Transformers on offline reinforcement learning tasks. Previous studies reported that, for offline RL tasks, pre-training a Transformer-based model on image data either maintains or helps performance, whereas pre-training on vision data deteriorates performance. To understand the effect of fine-tuning, the paper compares between models that are randomly initialized, pre-trained with language data (GPT2), and with image data (iGPT). For that purpose they used GPT2 architecture. For offline RL tasks, the paper employs medium datasets of HalfCheetah, Walker2d, and Hopper environments. The authors analyze various network artifacts, like how layer activation and model weights change during training, initials gradient norms, also what type of language-trained model learns and utilizes for down-stream tasks, and others. The conclusion roughly is that, pre-trained model parameters do not change much during fine-tuning, though their layer representations change largely. Pre-trained models also do not acquire more knowledge about the input or the label during fine-tuning. The authors also hypothesize with empirical proofs that, the image-pre-trained model fails probably because of large gradients, and also that language pre-training probably gives the Transformer context-like information that can be taken advantage of during down-stream tasks.", "questions": "- Lines 181-182, \"does not 182 encode more information about the input or the label\". This may be because of limitation of mutual information, as the authors themselves pointed out in line 326. So should take the finding that, pre-trained Transformers do not acquire more knowledge about the input or the label during fine-tuning, with pinch of salt.\n- In section 5.3, we can see parameters between pre and post-fine-tuning have changed for random initialization. Then in section 5.1, why representations for random initialization did not change much, i.e., the CKA scores for representations from random initialization are high?\n- In line 263, I am guessing, fine-tune without context.\n- In line 281, the authors noted that \"other than a particular block\". But by considering results from all three datasets, it is more like all blocks can contribute only in varying amount. That is, the observation is more in line with lines 303-304.", "limitations": "The limitations about the validity of the metrics used, is already pointed out in line 326.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The topic investigated in this work is important and has room to be explored.\n- The paper presents a well-written, detailed study.\n- Limitations, mainly from the validity of the metrics used, is clearly pointed out.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept