A Meta-Learning Perspective on Transformers for Causal Language Modeling (2024)

Abstract: The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process that may happen within the Transformer. Further, from within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments conducted on pre-trained large language models and real-world data.

heading: 1 Introduction
text: The Transformer architecture for neural networks is based on a self-attention mechanism (Vaswani et al., 2017) and has been widely used in natural language processing, computer vision, and scientific discovery (Devlin et al., 2019; Radford et al., 2019; Dosovitskiy et al., 2021; Vig et al., 2021), among other topical areas. Causal language modeling (CLM) aims to predict the next element in a sequence in an autoregressive manner and is one of the most important applications of the Transformer model. Large language models (LLMs) based on the Transformer architecture and CLM have shown impressive capabilities in many fields that have been considered difficult challenges in artificial intelligence (AI) (Liu et al., 2023; OpenAI, 2023). However, the underlying mechanisms of Transformer-based causal language models are still not well understood.
Meta-learning seeks to learn how to learn by leveraging common knowledge across various tasks, often through a bi-level optimization (Ravi and Larochelle, 2017; Finn et al., 2017; Franceschi
et al., 2018). Within this framework, an inner process mirrors a typical model optimization process such as stochastic gradient descent (SGD) (Bottou, 2010), whereas an outer process is employed to learn hyperparameters for the inner process.
Studying Transformer models from a metalearning perspective is an emerging research direction. Chen and Wang (2022) treat the Transformer model as a meta-learner to learn parameters for another model but did not identify a possible gradientbased inner optimization process undergone by the forward pass of the Transformer model, as we will show in the sequel. Some recent works of von Oswald et al. (2023a); Ahn et al. (2023); von Oswald et al. (2023b); Dai et al. (2023) have attempted to interpret in-context learning in Transformer-based language models from a meta-learning perspective by identifying a bi-level optimization process within in-context learning. However, they rely either on simplifications of the original Transformer model such as linear attention, or on special constructions of parameters. We make only limited assumptions to analyze a model that is as close to the Transformer used in practice as possible. However, we still do make some assumptions including zero initialization and omission of the impacts of normalization components, which makes the subject of our analysis different from Transformer models used in practice.
More importantly, rather than focusing on incontext learning that is regarded as an emergent capability of Transformer-based CLM models, we explore whether we can discover a meta-learning process within the training of Transformer-based CLM models. We attempt to give a plausible bilevel optimization process for a Transformer model trained for the CLM task. Additionally, to investigate the inner optimization process from the metalearning perspective, we visualize the evolution of the token representations within Transformerbased CLM models. We uncover a characteristic of
15612
the evolution of the norms of token representations learned in Transformer-based CLMs, which may indicate a special optimization trajectory. We further theoretically analyze this characteristic using a simplified linear model. Various linear simplifications have been used in theoretical analyses of Transformer models (von Oswald et al., 2023a; Ahn et al., 2023). Although the linear model is limited in being different from practical settings, it helps us initiate the analysis of this characteristic in an otherwise complicated Transformer model.
We summarize our contributions as follows:
• We derive a plausible inner optimization process for a Transformer model trained for the CLM task and specify its objective, which is also supported by evidence from our experiments on real-world LLMs, namely GPT-2, LLaMa-7B, and LLaMa-13B.
• Building on the inner optimization process, we present a meta-learning view of a Transformer-based CLM model.
• Through experimental investigation of the inner optimization process, we discover a special characteristic of norms of the token representations learned by the Transformer-based CLM model, which may indicate a special optimization trajectory. We analyze this characteristic and further investigate it through experiments on real-world LLMs and datasets.

heading: 2 Transformer Layer may Approximate Optimization Steps on CLM
text: Inspired by a bi-level optimization process discovered for in-context learning, we ask whether a bi-level optimization process also exists in training a Transformer for the CLM task. Let us first formulate a linear projection weight matrix in a Transformer model by expanding it via its gradient descent learning dynamics. A weight matrix W ∈ Rdout×din in a Transformer model trained by standard stochastic gradient descent using T training inputs x1, . . . , xT to optimize a loss function L via backpropagation can be represented as in Irie et al. (2023):
W = W0 − T∑
t=1
ηt∇ytL ⊗ xt (1)
where W0 ∈ Rdout×din is the weight initialization, ηt ∈ R is the learning rate, yt = Wtxt is the output
of the linear transformation of xt via Wt at step t and∇ytL ⊗ xt is the outer product between∇ytL and xt. We treat −ηt∇ytL ⊗ xt as an error signal. Note that with our expression, training on one single token corresponds to one training step.
For simplicity, we assume W0 is very small such that we can use a zero matrix to approximate it. Then, we have:
W ≈ − T∑
t=1
ηt∇ytL ⊗ xt. (2)
The assumption of zero initialization may be better approximated by larger models. It is a common practice to initialize weights with smaller values for larger dimensions to control the initial variance of the output, thereby fostering a stable learning process and mitigating problems like exploding gradients. Numerous initialization methods commonly employed, such as Xavier (Glorot and Bengio, 2010) and Kaiming (He et al., 2015), incorporate a scaling factor proportional to the inverse of the number of dimensions in either or both the input and output.
A Transformer model is built from several Transformer layers. Each layer consists of a multi-head self-attention (MHSA) sublayer and a feedforward network (FFN) sublayer. We investigate them in the following sections.

heading: 2.1 Multi-head Self-attention may Approximate an Optimization Step
text: Next, we assume the dimension of a token representation within a Transformer model to be dmodel. By following Vaswani et al. (2017), we write and expand the formulation of a multi-head attention module for the CLM task with z ∈ Rn×dmodel being the representations of current n sequential tokens (we omit the layer l of each component in the following formulations for simplicity):
MHSA(z1:n) = H∑
h=1
W hO
n∑
i=1
vhi softmax((k h)Tqhn)i
(3) where MHSA is the parameterized multi-head selfattention module. Here, W hO ∈ Rdmodel×dhead , W hV , W hK , and W h Q ∈ Rdhead×dmodel are output, value, key, and query projection matrices of head h respectively. We have dmodel = Hdhead and H as the total number of heads. The ith value vhi , key k h i , and query qhi of head h are computed by v h i = W h V zi, khi = W h Kzi and q h i = W h Qzi respectively. Further,
kh = [ kh1 , . . . , k h n ] is the key matrix of head h. The ith element of the softmax output is denoted softmax((kh)Tqhn)i. Throughout, we ignore the scalar factor within the softmax attention and any bias terms, to simplify analysis.
We expand W hO by (2) to get an approximation:
MHSA(z1:n) ≈ H∑
h=1
(− T∑
t=1
ηt∇ŷht L ⊗ v̂ h t )
n∑
i=1
vhi softmax((k h)Tqhn)i
(4)
where L is assumed to be the CLM loss throughout and we use ·̂ to denote something in the training history. Here, v̂ht refers to a historical value vector of head h at step t in the training history. Note that we omit the token positions for anything in the training history and t should not be confused with a token position within a sequence. As before, ŷht = W h O,tv̂ h t ∈ Rdmodel and ηht is the learning rate of the corresponding weight matrix W lhO,t of head h at step t.
Furthermore, by the chain rule, we have:
∇ŷht L = ( ∂ẑt
∂ŷht
)T ∇ẑtL (5)
where ẑt is a historical token representation at step t.
With (5), we can rewrite (4) as:
MHSA(z1:n) ≈ H∑
h=1
[ − T∑
t=1
( ∂ẑt
∂ŷht )Tηt∇ẑtL ⊗ v̂ht
]
n∑
i=1
vhi softmax((k h)Tqhn)i
= − T∑
t=1
ηt
H∑
h=1
wht A h t∇ẑtL
(6) where wht represents a weighting factor with wht = (v̂ h t ) T ∑n i=1 v h i softmax((k
h)Tqhn)i and Aht = (
∂ẑt ∂ŷht )T ∈ Rdmodel×dmodel is a transformation matrix.
We can interpret the formulation in (6) as a key-query retrieval of a value via the softmax selfattention and then, using the retrieved value as a query and the historical values as keys to obtain a weighted average of transformed historical gradients with respect to the historical inputs to the module. In principle, the weighted average of transformed gradients could implement an approximation of the transformed gradient with respect to
the current input zn by its neighborhood in a soft manner with closeness measured by the weighting factor. This multi-retrieval process is conducted for different heads and the final error signal is also aggregated over different heads. Many methods treat a transformed gradient as a pre-conditioned gradient that captures curvature information (Park and Oliva, 2019; Martens and Grosse, 2015; Grosse and Martens, 2016), such as Newton’s method for second-order optimization (Nocedal and Wright, 2006).
We can represent the nth token representation after the MHSA module with residual connection as follows:
zn ← zn + MHSA(z1:n)
≈ zn − T∑
t=1
ηt
H∑
h=1
wht A h t∇ẑtL
= zn − ¯A∇znL
(7)
where we exclude normalization components such as LayerNorm (Ba et al., 2019) or RMSNorm (Zhang and Sennrich, 2019) to isolate the core mechanisms and ¯A∇znL is an approximation of the transformed gradient with respect to the current input zn based on our previous interpretation.
From (7), we can see an implementation of an approximation of transformed gradient updates to the current token representation for optimizing the CLM loss L is possible from the mathematical formulations.

heading: 2.2 Feed-forward Network may Approximate an Optimization Step
text: We can perform a similar analysis of the feedforward network (FFN) layer. We define the FFN module as follows:
FFN(zn) = W2ϕ(W1zn) (8)
where W2 ∈ Rdmodel×dff and W1 ∈ Rdff×dmodel are weight matrices of the first and second layer in the FFN module and ϕ is a nonlinear activation function. Some popular choices of nonlinear activation functions are ReLU (Nair and Hinton, 2010), GELU (Hendrycks and Gimpel, 2023), and SwiGLU (Shazeer, 2020).
We can expand the formulation in (8) via (2) and
chain rule:
FFN(zn) ≈ −( T∑
t=1
ηt∇b̂tL ⊗ ât)an
= − [ T∑
t=1
( ∂ẑn
∂b̂t )Tηt∇ẑtL ⊗ ât
] an
(9a)
⇒ zn ← zn + FFN(zn)
≈ zn − [ T∑
t=1
ηt( ∂ẑt
∂b̂t )T∇ẑtL ⊗ ât
] an
(9b)
= zn − T∑
t=1
ηtwtBt∇ẑtL
= zn − ¯B∇znL (9c)
Here an = ϕ(W1zn) and L is the CLM loss, ât is a historical output of the first layer at step t. Further, b̂t = W2,tât ∈ Rdmodel and ηt is the corresponding learning rate at step t, wt = (ât)Tan is a weighting factor, Bt = (∂ẑt∂b̂t )
T ∈ Rdmodel×dmodel is a transformation matrix, and ¯B∇znL is an approximation of the transformed gradient with respect to the current input zn.
Similar to the previous analysis, we can interpret the FFN module as a query-key retrieval of a weighted average of transformed historical gradients with respect to the historical module input. In principle, the FFN module with the residual connection could approximate a transformed gradient descent update to the current token representation for optimizing the CLM loss L.
Overall from its mathematical formulations, a Transformer layer can be understood as approximately performing two types of transformed gradient descent updates via the MHSA and the FFN modules, respectively. Particularly, the token representations are normalized after each type of transformed gradient descent update. Thus, based on this understanding, the forward pass of a series of Transformer layers trained for the CLM task minimizes an approximate CLM loss L̄ via an inner optimization process. So far, we have shown the plausibility of an inner optimization process within a Transformer trained for the CLM task from its mathematical formulations. This can serve as a first step toward our meta-learning perspective, which will be discussed in the following section. In Section 5.2, we will present empirical evidence based on real-world LLMs and data that supports this understanding.

heading: 3 A Meta-Learning View of Transformer
text: We generally view the meta-learning process as a “learning to learn” process based on a variety of tasks. The “learning to learn” process is usually formulated as a bi-level optimization process. We have discovered an optimization process inner to the Transformer’s training process within its forward process, which forms a bi-level optimization process. The language data utilized for the CLM task typically spans a broad range of topics, inherently encompassing various tasks. It has long been recognized that a language model trained on such data can proficiently undertake multiple tasks. (Radford et al., 2019).
From our perspective illustrated in Figure 1, a Transformer layer takes the context tokens as the input data and uses them to form the weightings to approximate the gradients as discussed in Section 2. This can be considered as a way of learning by approximating the learning signal, the gradients from close points, different from the classical stochastic gradient descent (SGD), where the gradients are computed from a loss. Then, the layer can work as a meta-optimizer to learn to arrange the points and compute the weightings based on current data to reflect expected closeness for efficiently solving a certain task. Particularly, these close points could potentially come from different tasks such that ongoing learning could leverage knowledge from other tasks. Then, the approximate gradients are used to update the current token representation to be a better one for minimizing the CLM loss. Particularly, a Transformer layer or a meta-optimizer is trained using the CLM loss by the backpropagation from an outer optimization process. The token representations can be viewed as model parame-
ters from an optimization perspective. Through a series of optimization steps performed by layers, the CLM loss is expected to decrease over layers. Particularly, we view the final layer along with the linear language head as a predictor model that takes a token representation as its parameter vector.
So far, we have established a meta-learning view of the Transformer trained for the CLM task. Interestingly, unlike standard gradient descents commonly used in meta-learning, we found that in principle, the Transformer could utilize a transformed gradient descent within its inner optimization process, which could be more efficient than a standard gradient descent (Nocedal and Wright, 2006). We will leave investigations of these differences to future work.

heading: 4 Optimization Trajectory
text: In Section 2, we developed an understanding of the forward pass of a Transformer-based CLM model as an inner optimization process. One further question to ask is whether there are any characteristics of this optimization process. A token representation is continuously updated across layers, which has a similar role to the model parameters being updated by optimization steps. An optimization trajectory can be characterized by the evolution of a token representation. Many widely used learning algorithms consist of two parts: optimization and regularization. Investigation of a single aspect won’t give us a complete picture of the learning algorithms. L2-norm regularization is commonly used with SGD as a part of an optimization process. Since the token representation is the subject of optimization, this inspires us to study the property of the norm of the token representations like that of model parameters to give a more comprehensive view of the inner optimization process.
From the visualizations and quantitative analysis of the intermediate token representations in Section 5.3, surprisingly, we find the norm of a token’s vector representation approximately follows an incremental trend across layers in sequence. Notice that the loss value has a decreasing trend at the same time across layers in sequence based on Section 5.2. This is interesting because it indicates the inner optimization process may follow a specific type of optimization trajectory in general instead of an arbitrary one. In this section, we perform some analysis regarding this characteristic.
We propose the following conjecture.
Conjecture 4.1 A Transformer model trained for the causal language modeling task learns a solution such that the norm of the current token vector representation is approximately non-decreasing with increasing layer number l in the Transformer model (except for the last layer).
We do not consider the last layer because it is a part of the predictor model from our meta-learning view discussed in Section 3. This conjecture is supported by both qualitative and quantitative analysis performed in Section 5.3. We can also study this characteristic from a vector transformation perspective. Due to the complexity of the nonlinear transformation of a Transformer layer, we initiate studies on this characteristic by using a simplified linear model. We begin by simplifying the softmaxbased attention mechanism as a linear attention mechanism as done in some prior works (von Oswald et al., 2023a; Dai et al., 2023) such that the MHSA module becomes:
MHSAlinear(z1:n) = H∑
h=1
W hO
n∑
i=1
WV zi(W h Kzi) TW hQzn
=
H∑
h=1
n∑
i=1
W hOWV zi(zi) T(W hK) TW hQzn
= WMHSAzn
(10)
by following conventions from (3). We can now treat the MHSA module as a linear transformation using matrix WMHSA.
Similarly, we obtain a linear version of the FFN module by removing its nonlinear activation function:
FFNlinear(zn) = W2W1zn = WFFNzn. (11)
We develop a linear model corresponding to a Transformer layer by omitting the layer normalization and combining everything and residual connections:
zl+1n = (I +WFFN)(I +WMHSA)zn = Wlinearzn (12)
where I ∈ Rdmodel×dmodel is an identity matrix. We view each Transformer layer as a metaoptimizer, which should be responsible for this special characteristic. Therefore, we conduct studies on the weight matrices of these meta-optimizers with a focus on their eigenvalues to see if there are any possible hidden mechanisms leading to this
special characteristic. Then, we present the following proposition for the norm of the current token representations across layers in the linear model.
Proposition 1 Let W ∈ Rdin×dout be a linear transformation matrix, and x ∈ Rdin and y ∈ Rdout be the input and output of the linear transformation. We can have the Gram matrix of W decomposed by eigendecomposition as WTW = UΛU−1. Let a = UTx and b = U−1x. Given ∑dout i=1 λiaibi ≥ ∑dout i=1 aibi, we have ∥y∥ ≥ ∥x∥, where λi is the ith eigenvalue of the Gram matrix.
Proof 1 Since WTW is a symmetric matrix and has non-negative real eigenvalues, we can decompose WTW by eigendecomposition: WTW = UΛU−1.
∥y∥ = √ yTy = √ xTWTWx
= √ xTUΛU−1x = √ aTΛb
=
√√√√ dout∑
i=1
λiaibi ≥
√√√√ dout∑
i=1
aibi
= √ xTUU−1x = √ xTIx
= √ xTx = ∥x∥
(13)
With Proposition 1, we have another conjecture.
Conjecture 4.2 One of the intrinsic characteristics of Wlinear in the aforementioned linear model constructed from a Transformer model trained for the CLM task is to have large enough eigenvalues for its Gram matrix such that its output has norm no less than that of its input.
We perform experiments to verify Conjecture 4.2 in Section 5.3 with results shown in Table 2. A way to theoretically guarantee the non-decreasing property of the norm is to transform the current token representation via the linear model that we constructed, so as to meet the condition in Proposition 1.

heading: 5 Experiments
text: In this section, we demonstrate experimental results to provide evidence for our theoretical analysis presented in Sections 2 and 4.

heading: 5.1 Experimental Setting
text: Our experiments are based on three popular Transformer-based causal language models: GPT2 (Radford et al., 2019) (released under a modified
MIT license), LLaMa-7B, and LLaMa-13B (Touvron et al., 2023; Geng and Liu, 2023; Computer, 2023) (both released under a Meta license). Their parameter counts are 1.5 billion (GPT-2), 7 billion (LLaMa-7B), and 13 billion (LLaMa-13B) respectively. We use test sets of Wikitext-103 (Merity et al., 2017) and 1 Billion Words (1BW) (Chelba et al., 2013), two common language benchmark datasets in our experiments. The wikitext-103 and 1BW datasets were constructed from Wikipedia articles and news crawl data respectively, so they follow different data distributions. To have enough context for the next token prediction task, we only keep data with lengths more than or equal to 10 words. Due to limitations on computational resources, we only randomly chose 5,000 samples from the test set of 1BW. All experiments are performed using an NVIDIA A100 GPU.

heading: 5.2 Inner Optimization Process
text: To investigate the possible optimization process that we discovered in our theoretical development, we perform experiments over all three models on the two datasets. We attempt to study the behavior of the inner optimization process by an approximation of the inner optimization loss with respect to the current token representation zln at the lth layer as follows:
Llinner,n ≈ CrossEntropy(softmax{LH [ TFL(zl1:n) ] }, yn)
(14) where LH is the linear head and TFL is the last Transformer layer, together forming a predictor model, and yn is the true label of the current token. Note that we assume TFL only outputs the representation for the current token zln. As discussed in Section 2, the inner optimization loss in our understanding is a direct approximation of the CLM loss, so using the CLM loss based on true labels would be a good approximation of the inner optimization loss of interest.
A sequence of the approximate losses for a single token is computed for a sequence of layers up to the L − 1 layer since the Lth layer is the last layer and a part of the predictor. In this experiment, we only study the fifth or later position in a textual sequence to have enough context for the prediction task, and due to large variances, we filter out loss sequences with a final value greater than 1 to only focus on those cases in which the models are able to make a good prediction. In Figure 2, we observe
a decreasing trend of the approximate loss on both LLaMa models across all of the datasets, mirroring an optimization process. We hypothesize the reason why GPT-2 doesn’t have a obvious decreasing trend is that the zero initialization assumption isn’t approximated well by a small model. We indeed observe that as the zero initialization assumptions better met by larger models, the larger models show a clearer decreasing trend. This indicates the Transformer model indeed conducts an inner optimization process for an objective similar to the CLM objective layer-by-layer and gives evidence that supports our understanding of Transformerbased CLM models proposed in Section 2. See more experimental results on the 1BW dataset in the appendix.

heading: 5.3 Studies of Token-level Representation
text: We investigate the inner optimization trajectory by studying how the current token vector representation evolves during the inner optimization process. We visualize the current token vector representations across layers in sequence for different samples in a 3D space by using principal component analysis (PCA) for dimension reduction. Note that we do not consider the last layer here because we treat it as a part of the predictor model as mentioned in Section 3. We display the optimization paths of different samples using different colors by connecting the token representations from neighboring layers. We randomly selected 50 samples for GPT-2 and 150 samples for LLaMa-7B and LLaMa-13B to make the visualizations.
Surprisingly, we observe that almost all of the samples follow a specific type of optimization trajectory instead of arbitrary ones. From Table 1, we find that for the majority of samples, the norms of their current token representations are non-decreasing along the optimization trajectory and this characteristic is also not hard to see in the
visualizations in Figure 3. To study this characteristic, we measure the percentage of the optimization trajectories with non-decreasing norms (sequencelevel) and the percentage of pairs of token representations at neighboring layers in a sequence whose norms are non-decreasing (pair-level). Let us assume the decreasing and non-decreasing happen uniformly at random: then the probability for a sequence of n layers to have such non-decreasing property is 2−n. Considering n = 11 for GPT-2, n = 31 for LLaMa-7B, and n = 39 for LLaMa13B, the chance is exceedingly small. Across different models and datasets, we consistently observe a high percentage at the sequence level compared to the null random ensemble. Moreover, the pair-level percentage is almost perfect. This gives strong evidence for our Conjecture 4.1. We present additional experimental results on the 1BW dataset in the appendix. Based on the theoretical analysis in Section 4, we construct the linear transformation matrices for GPT-2 for different layers and samples. From Table 2, the eigenvalues of the constructed matrices perfectly meet the condition proposed in Proposition 1, which means the matrix can effectively stretch the current vector representation resulting in an incremental norm. Note that we only conduct this experiment on GPT-2, because LLaMa uses a special FFN module with a gating mechanism instead of a standard one.

heading: 6 Related Work

heading: 6.1 Meta-Learning
text: Meta-learning aims to acquire the skill of learning itself by tapping into shared knowledge among different tasks, typically through a process involving bi-level optimization. Many meta-learning methods have shown their great efficiency in learning from few-shot examples (Ravi and Larochelle, 2017; Finn et al., 2017; Franceschi et al., 2018).
Our work focuses on studying Transformer-based causal language models from a meta-learning perspective instead of developing a new meta-learning method.

heading: 6.2 Mechanistic Interpretability
text: Mechanistic interpretability involves analyzing the weights of a trained neural network to reverse engineer the algorithms learned by the model (Meng et al., 2022; Elhage et al., 2021; Nanda et al., 2023; Ilharco et al., 2022; Conneau et al., 2018). Our research also endeavors to explore the hidden mechanisms of Transformer models, albeit from a distinct perspective compared to existing studies in this field.

heading: 6.3 Representations of Transformer Models
text: Many works have attempted to study various properties of internal representations learned by Transformer models (Thompson and Mimno, 2020; Chen et al., 2021; Reif et al., 2019). Even though some works (Fayyaz et al., 2021; Tenney et al., 2019; Niu et al., 2022) have shown increasing
prediction power across Transformer layers, especially following a BERT architecture (Devlin et al., 2019), our work attempts to provide reasons for this phenomenon instead of simply presenting the existence of it. Our work also studies a special characteristic about norms of the token representations learned by the Transformer-based CLM model, which has not been explored by existing works.

heading: 7 Conclusion
text: In this work, we show the plausibility of an inner optimization process in a Transformer model trained for the CLM task by both mathematical derivations and empirical evidence. We also provide a meta-learning view of the Transformer model trained for the CLM task, which may provide useful insights into the learning dynamics of Transformer-based causal language models. In addition, from our experiments on the inner optimization process, we discover a special characteristic about norms of the token representations learned by the Transformer-based CLM model and investigate this characteristic by theoretical analysis and experiments on real-world large language models. Our overarching objective in this study is to provide a new perspective for studying Transformer-based CLM models as well as some empirical evidence to encourage further research in this direction. The
perspective may also give more insights into the design and training of more advanced Transformer models by having a clearer view of their internal mechanism.

heading: 8 Limitations
text: This work is limited to enhancing the understanding of the hidden mechanisms of Transformerbased CLMs. More effort can be made by inspiration from these understandings to improve existing models and propose more advanced algorithms. Our current study verifies our discovered mechanisms of Transformer-based CLMs on limited models and datasets. It is interesting to study these mechanisms in more diverse scenarios.

heading: A Additional Experimental Results
text: We show additional experimental results on the inner optimization process and studies of tokenlevel representation below.

heading: Future Work
text: We will leave investigations of these differences to future work