Order-Level Attention Similarity Across Language Models: A Latent Commonality
Authors: Jinglin Liang
Jin Zhong
Shuangping Huang
Yunqing Hu
Huiyuan Zhang
Huifang Li
Lixin Fan
Hanlin Gu
Venue: NeurIPS 2025 poster ()
Keywords: Exploration of language model commonalities
Language model adapter transfer
Primary area: deep_learning
TL;DR: We discover commonalities across different language models and leverage them to achieve training-free cross-model adapter transfer.

Abstract: In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities?
While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities.
In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer.
In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities.
Furthermore, we discover an implicit mapping between OLA and syntactic knowledge.
Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method.
Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input.
Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates.
Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. 
Code is available at \url{https://github.com/jinglin-liang/OLAS}.

Order-Level Attention Similarity Across Language Models: A Latent Commonality
Jinglin Liang', Jin Zhong', Shuangping Huang Yunqing Hu", Huiyuan Zhang'. Huifang Li Lixin Fan", Hanlin Gu
South China University of Technology. Pazhou Laboratory, 3 Zhuzhou CRRC Times Electric Co., "China Telecom Research Institute, WeBank, 'hhe Hong Kong University of Science and Technology, edu.cn, aeljl@mall scut. edu. en, eehspescut. edu. edu.. cn
Abstract
In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities, In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across L.Ms exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA representation and train an adapter that takes OLA as imput. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at / tith....... com/jingling liang/OLAS
unified syntactic feature
Introduction
With the rapid development of large language models (LMs), their exceptional capabilities have profoundly impacted human society [1]. In practical applications, practitioners often fine-tune models to meet the demands of specific tasks 22 3] However, due to the lack of efficient methods for knowledge transfer between different LMs, results fine-tuned
model cannot be directly reused on another, significantly increasing development costs. Research in knowledge distillation [4] and representation learning [6] suggests that when different models share common representational spaces, efficient knowledge transfer becomes possible. This inspires us to ponder question: Do pretrained LMs possess commonality that could enable cross-model knowledge transfer?
approach this through the lens of attention mechanisms. Although I.Ms differ in architecture, training data, and other factors, mainstream transformer-bast based LMs rely on attention mechanisms [7] to aggregate context for prediction mechanisms, different LMs trained on large corpora may converge to optimal attention pattern for the same text, resulting in commonalities in their contextual aggregation behavior.
9.. Given the similarity in training objectives and attention
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

MHA1
MHAZ
MHAA
MHA1
MHAA
layer1
layer2
MHAZ
(a) Structure of Simplified Twe--Lyer LM
(b) Zero aggregation path (c) Singlee aggregattnn paths (d) Double aggreglttogggooonn path
Figure Information flow path decomposition. (a) Simplified LM showing only Multi-Head Attention (MHA) modules (others omitted); (b) Path via residual connections across all layers; (c) Paths through MHA in one layer, residual connections elsewhere; (d) Path through MHA in all layer.
Although some studies [10 individual LMs, their focus has been on attribution analysis, aiming to quantify the contribution of different tokens to the output [3 [4. However, focusing solely on attribution analysis of a specific model may lead to an overemphasis on its unique characteristics, without considering the potential commonalities across different LMs. To our knowledge, the commonality of contextual aggregation patterns across different LMs has not yet been explored.
11
12 have investigated the context aggregation mechanisms within
In this work, we unveil this commonality and propose a training free cross-model knowledge transfer approach based on it. While attention pattern similarities among LMs seem intuitive, inherent differences in layer numbers and attention heads endow their attention weights with distinct meanings, making it challenging to identify attention similarities the layer or head level. To unify attention weights across models into comparable representations, we propose Order-Level Attention (OLA). Specifically, as shown in Figure I we decompose information flow into multiple paths, with the context aggregation effects from paths sharing the same aggregation count being represented as an OLA that specific order. For instance, first-order OLA captures aggregation effects from paths containing one aggregation step, as shown in Figure c)) Mathematically. OLA equates to order-wise decomposition of Attention Rollout [IO] (detailed motivations and derivations in (3.I). OLA unifies the meanings of attention across different models, establishing foundations for analyzing attention similaritics. Extensive experiments on 12 LMs demonstrate significant similarities in the same-order OLA across different LMs. We refer to this phenomenon Order-Level Attention Similarity (OLAS). Furthermore, to investigate the linguistic implications OLA, we conduct experiments demonstrating that syntactic dependencies (15) can be predicted solely from OLA representations using an auxiliary model. This finding suggests that OLA inherently encodes syntactic knowledge of the input text.
Based on these findings, we propose the Transferable OLA Adapter (TOA), enabling cross-LM adapter transfer without requiring tuning. Training-free cross-LM adapter transfer a valuable task with many potential applications ($4). but poses significant challenges. Adapters process features specific to individual LMs, yet different LMS exhibit divergent feature spaces. Even when training the same model twice with different initialization parameters, the resulting feature spaces remain distinct [6). causing adapters to tightly couple with source LMS and limiting transferability. To address this, leverage OLA as a unified syntactic representation across LMs and train adapters using OLA for downstream tasks. Since OLA exhibits similarities across LMs, the trained adapter can be directly transferred other LMs without any parameter updates or training data. We evaluate TOA four tasks: relation extraction (RE), named entity recognition (NER), dependency parsing (DP), and part- of-speech tagging (POS). Extensive experiments demonstrate significant performance improvements when transferring TOA from a source I.M to an unseen target I.M. For example, transferring TOA trained on LLaMA3-3B Owen2-1.50 elevates Qwen2's s relation prediction accuracy from 7.69% (zero-shot baseline) to 34.90%.
In summary, contributions are follows:
1) We propose OLA 3... which unifies the attention mechanisms of different LMs into comparable representations.
2) Based on extensive qualitative and quantitative experimental analysis, we propose two key findings: significant similarities in OLAA across different L.Ss ($3.2 and $33)). and OLA's inherent encodingg syntactic knowledge ($3.4).
and
3) Building on the above findings, we introduce TOA ($4.) which enables training-free cross-LM adapter transfer. Extensive experiments demonstrate that transferring TOA trained
source LM to an unseen target LM significantly enhances its performance (44.2

---

2 Related Work
Although some studies have explored the context aggregation mechanisms of LMs and adapter transfer for LMs, different LMs and leverage these to achieve cross-LM adapter transfer without requiring tuning.
the first to investigate the commonalities in context aggregation across
Context Aggregation Mechanisms in LMs. While deep learning has advanced various fields in recent years [6. [7], our understanding of deep models remains limited [8] [9]. This has motivated extensive research on model interpretation, including work on attribution [20] 21 and
feature interactions [22] 23]. Among these, some methods focus specifically on analyzing the
attention patterns of transformer-
models. For instance, early works [24 25 26
27 aimed
to understand the nature of attention by performing intuitive visualizations or statistical analyses
in classic models such Bert (28 and GPT2 [29]. Subsequently, some studies [3
14
30 31
explored the explainability of attention by perturbing attention weights and observing changes in outputs. Additionally, some studies 32 33 have investigated the identifiability of attention weights.
Unlike the aforementioned works that analyze a specific layer or an individual attention head, the study [10] conducts a comprehensive analysis of multi-layer attention and residual connections using matrix multiplication and maximum flow algorithms [34] Recently, a series of studies based on norm-based methods [35] investigate the contribution of each input token the output. Specifically, they decompose the model's 's output into multiple terms, each associated with particular token, and then estimate the contribution of the token based on the norm of each term its deviation from the output. Some studies [36 [ primarily consider the attention blocks responsible for context further incorporate the feed-forward blocks that map the features of each token into consideration. While existing studies focus on attention explainability and its use in model prediction attribution, the commonalities underlying LM context aggregation mechanisms remain unexplored.
aggregation, while others 37 38
Adapter Transfer Across LMs. Freezing the LM's parameters and only training adapters is common paradigm for applying LMs to specific tasks [39 3]. These adapters come in various forms, such as LoRA [2] and soft prompts [40]. To reduce the resouree consumption associated with repetitive learning, some studies explore adapters transfer. Some works [41 42 13 investigate the cross-lingual transfer of adapters, meaning they use adapters trained on a source language in target language. The work [4 proposes training delta LM that assembles outputs with the pretrained LM to enable cross-mis model knowledge transfer. However, their method transfers text-input delta LMs, differing fundamentally from our feature-space adapter transfer approach. The work (45) is the only study that studies cross-model adapter transfer, applying the representation learning [46 method proposed in [6] to soft prompts. However, this approach requires training during transfer and suffers significant performance degradation. To our knowledge, we are the first to achieve training- cross- model adapter transfer, which allows adapters trained on the source model to be directly applied to the target model without any additional training.
3 Order-level Attention
In this section, we first describe the derivation of OLA (3..). Then, we present qualitative ($3.2) and quantitative ($..) experimental evidence to reveal the phenomenon of OLAS. Finally, we demonstrate the implicit mapping between OLA and syntactic knowledge 3.4).
3.1 Order-Level Decomposition of Attention Rollout
Although different LMs exhibit certain structural variations, typical models such as Bert 28 and Llama [48] feature layers composed of an attention block followed by feed-forward block, as illustrated in Figure 2 Since the feed-forward block and the normalization layers only perform trans- formations on the features of individual tokens without aggregating information from other tokens, is actually the multi-head attention module that facilitates contextual information aggregation at each layer. Additionally, the attention module is paired with a residual connection, creating a shortcut for the information to bypass the attention module. As result, the context aggregation matrix for the i-th layer can be expressed as (A4() + 7), where (6) E R is the average attention matrix across all heads the i-th layer, is the token sequence length, and is the identity matrix representing the shorteut created by the residual connection. By multiplying the matrices of each layer, we obtain the
3

---

Input
Input
Output
(a)
MHA
Muei Head
Attention
FFN
Fend
Forward
UN
LayorNorm
Input
Attention Block
Feod Forward Blocs
Figure 2: Structure of each layer in typical LMs. (a) Llama and Qwen. (b) Bert, Roberta, and Electra. (c) Gemma.
attention rollout [10]. expressed as:
N Á = II ( + +1,
(I)
where Å is the attention rollout, with N the number of layers the language model.
However, when we input different texts into LMs and visualize their attention rollout, as shown in Figure a), we observe that the attention rollout exhibits a consistent pattern across different texts, with nearly all attention concentrated on a few less important tokens. More visualizations are presented in SA showing this phenomenon is prevalent across various LMs. This phenomenon has been observed before and termed "Attention Sinks" [49]. The reason is that the softmax function prevents attention scores from being exactly zero, which forces each token to aggregate information from other tokens. However, when a token has already aggregated sufficient contextual information and longer needs to aggregate information from other tokens, offloads its attention other less important tokens, causing Attention Sinks [49] This may imply that although LMs consist of many layers, the number of effective aggregation steps may be fewer than the number of layers.
Due to Attention Sinks, the Attention Rollout exhibits similar responses across different texts, lacking distinctiveness. We posit that Attention Sinks occur because each layer's s attention module in LM is paralleled with a residual connection, allowing information to flow both through the attention module and the residual connection. Consequently, shown in Figure | an N N-layer LM creates 2 potential information pathways. The Attention Rollout represents the contextual aggregation matrix resulting from all these paths. As previously mentioned, the number of effective aggregations fewer than the number of layers, causing some path components to become ineffective due to excessive aggregation. This results in similar biases and diminishes the distinctiveness of the Attention Rollout. To address this, we separately analyze the contextual aggregation effects from paths with varying aggregation counts, illustrated in Figure (c)).. Mathematically, this involves performing an order-level decomposition of the Attention Rollout, expressed as:
N
-=
10
Σ
A + + A(N-), A
ISKJEN
where each term corresponds to the sums of contextual aggregation effects across paths with identical aggregation steps. For instance, the Oth-order term the identity matrix I, which means that information flows through residual connections at all layer, as illustrated in Figure [bb). The first- order term sums effects across all paths containing one aggregation step (i.e., one attention module traversal and N-1 residual connections), with (?) such paths, as shown Figure (c.. Similarly, the k-th order term aggregates effects across (2) paths where aggregation occurs k times.
We normalize each term to obtain the OLA of each order. For example, the first-order OLA is defined A4) ΣN E A(i), and the Attention Rollout À is reformulated weighted sum of the OLA:
N
A
(3)
where 10) denotes the i-th order OLA.
In summary, given that different LMs share similar optimization objectives and context aggregation mechanisms, it is intuitive that their attention mechanisms exhibit commonalities. To verify this,

---

Text A full stationary tank that is right circular cylinder has redius of 100 feet and height of 35 feet, oilis pumped from the tank has right circular cyfinder truck tank complatuly filid.
Qwen2 1.58
LLaMA3.2
Teat2: Developing omotional intallgennc enhaanees your ability navigate relationchips, communicate affectively, and marage
both professional settings, leading probiem salving skills dedison makking
Owen2
Dant de i.
de Hel
1.50
LlaMA8.2
38
1000
P00 11
(c)
(e)
Figure 3: Visualization results of the Attention Rollout and first- and second-order OLA obtained by inputting two texts into Qwen2-1.5b and Llama3.2-33 (a) Attention Rollout. (b) First-order OLA. (c) First-order OLA with row-wise maximum values set to zero. (d) Second-order OLA. (e)
Second-order OLA with row-wise maximum values set to zero.
propose OLA, which unifies attention across LMs into comparable representations with equivalent semantics to enable cross-mas cross-model comparisons. Below, we analyze OLA's cross-model similarity through quantitative and qualitative perspectives, and explore its linguistic implications.
3.2 Qualitative Empirical Evidence of OLAS
To qualitatively analyze OLA similarity across LMs, we input two distinct texts to Qwen2-1.5b and Llama3.2-3b, visualizing Attention Rollout and first/second--rrde OLA in Figure (a), (b), and (d). Since extreme maximum values obscure distribution patterns, we mask these values (zeroing maxima for firs--seccondoorde OLA in Figure (c) and (e), and for Attention Rollout in A revealing clearer structural features. From the figures, we can derive the following insights: 1) The same-order OLA from different LMS for the same text is highly similar. This can be seen by comparing the first and second, as well as the third and fourth rows in Figures (c) and (e). It is also evident in the third-order OLA visualizations in $B] We term this phenomenon Order-Level Attention Similarity (OLAS). 2) OLA from different texts show a clear distinction. This can be concluded by comparing the first and third rows, as well as the second and fourth rows in Figures 3 (c) and (e). This suggests that OLA could potentially serve as a feature representation for sentences. 3) Attention sinks in low-order OLA are less pronounced than in higher-order OLA. Attention Rollout (weighted sum of all-order OLAss) exhibits the most significant attention sinks, with each row's attention focused on unimportant tokens, such as Llama's '<bos>' token. First-order OLA exhibits the least sinking, while second-order OLA shows more. This suggests that higher-order OLA may represent ineffective components with similar biases. Comprehensive visualizations for more LMs are in $ and
(a),
3.3 Quantitative Empirical Evidence of OLAS
To comprehensively quantify the similarity of OLA across different LMs, we design two innovative evaluation methods: the first relies on visual model to replace human assessment of OLAA visual similarity (33.3.1. the second adopts image retrieval framework ($3..2).
5

---

Table I: Results of quantitative evaluation on cross-mode similarity for OLA and baselines based on visual classification model. Entries represent accuracy (unit: %), averaged over three experiments, reflecting source- target LM similarity (higher more similar). The terms 1st, 2nd, 3rd denote first-, second-, and third-order OLA. Best performance is bolded.
(a) CL.M Results. Q-1b5, Q-7b, G-2b, G-9b, L-3b, and 1.-8h denote Qwen2-1.50 1,5b, Qwen2-7b, Gemma2-2b, Gemma2-9h, Llama3.2-3b and Llama3.
(b) ML.M Results. B-h, B-I, R-b, R-1, E-b, and E-1 denote Bert-base, Bert-large, Roherta-base, Roberta- large, Electra-base, and Electra-large, respectively.
L-3b, L-8b,
L-3b, L-8b,
Q-1b5, Q-7b, G-2b, G-9b
R-b, R-I, E-b, E-1
B-b, B-I, E-b, E-1
B-b, B-I, R-b, R-1
Source
Source
G-2b, G-9b
Q-1b5, Q-7b
Target
Q-1b5 Q-7b
G-2b
G-9b
L-3b
L-8b
Target
B-b
B-1
R-b
R-1
E-b
E-I
Rollout
10
27.90
7.70
52.60
26.00
66.10
59.70
Rollout
44.30
42.80
13.20
36.60
13.60 9.10
IRNL
36
18.50
11.90
58.10
67.20
77.20
73.60
IRNL.
60.00
4.10
20.80
15.00
55.20
38.10
ALTI
22.60
15.50
69.30
71.80
85.60
79.80
ALTII
90.30
80.30
73.80
48.10
86.80
90.30
52.60
49.20
93.10
92.40
94.60
94.10
1st
91.90
92.40
80.40
69.90
95.20
95.80
2nd
67.10
49.90
89.30
86.20
90.70
91.90
2nd
88.80
62.70
60.70
28.00
76.30
82.90
3rd
76.50
48.90
79.90
78.80
88.60
60
86.70
3rd
80.40
67.70
38.10
31.90
58.10 68.30
3.3.1 Quantitative Analysis Based on Visual Model
We employed an image classification model as a proxy for human evaluation to objectively assess the similarity of OLA maps generated by different LMs given identical text inputs. Specifically, we trained an image classifier (ResNet-18 18 (50)) using OLA maps generated by source LMs, where all OLA maps produced by different LMs for the same text were assigned to the same category, with the optimization objective defined as:
= arg min ((a.i) [CcE(Fo(a). Dtrain = ((af') i) i [I.. j [1. (4)
where M denotes the count of texts, S the number of source LMs, ai the OLA map from the j-th source LM for the i-th text, Dtrasn the training dataset, 0 the classifier parameters, LcB the cross-entropy loss, and Fe (a) the classifier's 's predicted text index for OLA map a. The trained classifier was evaluated on the dataset Dtest composed of OLA generated by target LMs on the same set of texts, where Diest- { (ā., i) i E [1..M]]. with à; denoting the OLA produced by the target LM for the i-th text. Higher accuracy indicates stronger alignment between source and target LMs' OLA, as the classifier more reliably classifies their OLA generated for the same text into the same category. Experiments were conducted on 12 LMs, including 6 Causal Language Models (CLMs) and 6 Masked Language Models (MLMs), detailed descriptions of these LMs are provided in $C Further implementation details, including dataset construction and preprocessing, are provided in
Baselines. To analyze whether the performance of existing context aggregation analysis methods exhibits similarities across different LMs, we also validated them within the experimental framework. The methods include Attention Rollout [TO]. IRNL (36), and ALTI [.]. Among these, Attention Rollout is the most similar to our method, while IRNL and ALTI are common norm-based attribution methods focused on attention block analysis. Additionally. IRNL and ALTI require deriving the expression between the model's output and input tokens. However, their derivations were only conducted for MLMs and cannot be directly applied to CLMs due to structural differences. Therefore, we derive the expressions for CLMs and present the process in SD
Main Results. From the Table J we can draw the following insights: 1) As concluded in $32 the OLA obtained from the same text across different LMs exhibits significant similarity, while OLA from different texts shows clear distinctions. In experiments with both MLMs and CLMs, the first-, second-, and third-order OLA achieved high classification accuracy, particularly the first-order OLA, which exceeded 90% accuracy under multiple settings. The fact that OLA can be used for classification indicates that OLA from different texts are distinguishable. Furthermore, the ability of a classifier trained on the source LMs' OLA to generalize to the target LM suggests that OLA across the source and target LMS are highly similar. 2) Existing methods also exhibit similar performance across different LMs, but not as prominently as the simpler OLA. This may arise because Attention Rollout incorporates higher-order OL-A components with weaker distinguishability. Other norm-based methods for attribution analysis primarily focus on the relationships between individual

---

Table 2: Retrieval- based quantitative evaluation of first-order OLA cross-model similarity. Row headers denote source LMs. Column headers denote target LMs. Entries indicate evaluation metrics Hits Hits @5 (unit: %).
(a) CLM Results.
(b) MLM Results.
Ste\Tg[
Q-1b5
G-2b
L-3b
Sre\Tgt
B-b
R-b
E-b
Q-1b5
83.60 89.40
95,90 97 00
B-b
51.90/ 58.80 91.60 94,90
G-2b
83.20 89.30
95.30 / 99.10
R-b E-b
/3.90/83.90
71.20
L-3b
92.90 96.10
/ 96.50
92.40 96.00 67.40 72.90
Table 3: Results of syntactic dependency parsing using OLA predicted by LMs. Entries indicate UAS/LAS (unit: %). Best performance is bolded.
(a) CLM Results.
(b) MLM Results.
LMs
Q-1b5
G-2b
L-3b
LMs
B-b
R-b
E-b
Rollout
50.53/29.79
44.24/22.04
53.77/35.57
Rollout
46.20/30.69
35.77/17.94
50.35/34.02
1st
63.58/48.24 62.25/45.95
62.98/48.19
1st
81.29/72.16
80.00/70.44
81.23/72.63
60.58/43.90
57.28/38.88
58.93/42.94
2st
72.86/61.05
72.68/60.
77.47/66.78 78
3rd
55.19/36.82
51.89/32.25 51.00/33.35
3rd
66,44/53.17
36.99/18.67
50.72/33.90
tokens in the feature space, which, compared to the contextual aggregation patterns we emphasize, are more reflective of the model's ' individual characteristics. 3) The similarity of OLA varies across different source-target LMs combinations. This suggests that the similarity between different LMs is relative and influenced by various factors during training, such as data and architecture. OLA has the potential serve as an indicator for evaluating the similarity between LMs.
Controlled Experiments. To verify the reproducibility our findings across diverse data settings, conducted controlled experiments on dataset and preprocessing. The OLAS phenomenon persists under varying data configurations, demonstrating its robustness (results and analysis in ŞH).. To ensure observations reflect inherent properties of LMs (i.e., the commonality of contextual aggregation patterns learned from large corpora), we perturbed the model parameters and observed that the OLAS phenomenon disappeared in the perturbed models. This indicates OLAS is intrinsically tied to pre-trained model parameters, not experimental biases or data artifacts (results and analysis in $
3.3.2 Quantitative Analysis Based on Image Similarity Retrieval
We propose image serrevval-based quantitative evaluation method for OLA similarity. Specifically, we feed the text from Section I into the source LM and target LM to generate corresponding OLA maps. Using the target LM's OLA maps queries, compute SSIM [51] (a standard image similarity metric) between each query and source LM's OLA, then rank the results by SSIM scores. Retrieval correctness is determined by whether the ground-truth candidate appears in the retrieved results. The ground-truth is defined as the source LM's OLA whose original text matches the query OLA's original text. We evaluate Hits@I (the probability of the correct source LM OLA ranking first) and Hits @5 (the probability of the correct OLA appearing in the top five), with first-order results shown in Table and second- and third-order results shown in the (Table [2]. From the tables, observe remarkably high retrieval success rates. For example, in the first-order CLM results, even the lowest Hits @5 surpasses 89%, while the highest exceeds 97%. Though MLM performance is weaker than CLM, it remains substantial. These findings further support the OLAS phenomenon.
3.4 Relation between OLA and Syntactic Knowledge
The OLAS phenomenon suggests unified attention pattern across different LMs in OLA. To explore the nature of OLA, we designed an experiment to investigate whether OLA contains syntactic knowl- edge. Specifically, we utilize the OLA of training texts to train an additional syntactic dependency parsing network, where the input is the OLA and the target the corresponding syntactic dependency annotations. After training, evaluate the accuracy of this network the OLA of test texts. If
7

---

the network can successfully predict the original text's syntactic dependencies using only OLA, it suggests that OLA encodes syntactic knowledge. More implementation details can be found in [
We conducted experiments on first-, second-, and third-order OLA, respectively. Additionally, since Attention Rollout is equivalent to the weighted sum of OLA across all orders, we included it in our analysis to investigate whether higher-order OLA encode syntactic knowledge. From the results presented in Table 3 we derive the following insights: 1) OLA encodes syntactic knowledge. Using first-, second-, and third-order OLA to predict syntactic dependencies achieves promising performance. Notably, first-order OLA achieves over 80% Unlabeled Attachment Score (UAS) across all MLMs and over 60% UAS for CLMs. 2) Lower-order OLA exhibit more prominent syntactic features than higher-order OLA. Across all LMs, we observe a consistent trend where higher-order OLA yield lower performance. Furthermore, Attention Rollout (as the aggregation of all all-order OLA) exhibits significantly lower accuracy compared lower-order OLA, this suggests that higher-order components contain less prominent syntactic features.
4 Training-free Cross-LM Adapter Transfer
4.1 Transferable OLA Adapter
Due to the large number of parameters in LMs, directly fine-tuning them is often prohibitively expensive. As a result, freezing the parameters of the LMs and training an adapter tailored to downstream tasks has become a common approach for applying LMs to specific tasks [39 3]. However, adapters are typically tied to the specific LMs they are trained on, which limits their flexibility and reusability. To address this limitation, we investigate how to transfer adapters across LMs. This is a valuable question with many potential upplications. For example, we could first train an adapter on a smaller model and then transfer it to a larger model, significantly reducing the resource cost of tuning the adapter for the larger model. another example, we could train an adapter on an open-source model and transfer it to closed-source model. This enables the closed-souree model to learn knowledge from the data while keeping the model and data isolated, thereby protecting the privacy of both the model and the data.
However, this is challenging problem, especially in the context of training-free free transfer. This difficulty arises because adapters are designed to process the features of LMs, and different LMs often have sigmif- icantly different feature distributions and dimensions. As result, the adapter becomes tightly coupled with the source model, making it difficult directly trans- fer it to other models. To the best of our knowledge, no existing work has achieved training-free free cross-LM adapter transfer.
Training Text
Adapter
Loss
OLA
Training Stage Testing Stoge
Direct Transfer
Testing Text
Target
Adapter
Predict
OLA
forward
backward
learnable
Inspired by our findings that OLA from different LMs exhibits similarity and that OLA encodes syn- tactic knowledge, we propose the Transferable OLA Adapter (TOA), which enables adapter transfer across models without requiring training. Specifically, we treat OLA as unified syntactic feature representation across models and train an adapter that takes OLA as input for downstream tasks. Due to the cross-model similarity of OLA, the trained adapter can be directly transferred to other LMs without requiring additional training, as illustrated in Figure In the experiments reported later, we use stacked first- and second-ords -rder OLA as input features for the adapter. However, this is not the only option. Other configurations. such as using only first-order OLA or combining different orders, can be chosen based on task requirements.
Figure Overview of TOA. In the training phase, the source LM is frozen and an adapter is trained for the downstream task using OLA as input. In the testing phase, the adapter is directly transferred to the target LMM
4.2 Experiments
We evaluated the cross-model transfer capability TOA on four foundational NL.P tasks: Relation Extraction (RE) [52), Named Entity Recognition (NER), Dependency Parsing (DP), and Part-of-
8

---

Table 4: Cross-Modes Transferability of TOA the RE Task. Column headers indicate source LMs, row headers indicate target LMs, and entries represent relation prediction accuracy (unit: %). Best performance is bolded; scores exceeding the zero-shot baseline are underlined.
(a) CLM Results.
(b) MLM Results.
Ste\Tgt Q-1b5 Q-7b G-2b
G-9b
L-3b
L-8b
Ste\Tgt
B-b
B-1
R-b
R-1
E-b
E-1
Zero-shot
Zero-shot
7.69
8.58
5.01
18.22
14.65 12.99
2.69
0.48
0.04
7.18
5.78
0.04
TOA (Ours)
TOA (Ours)
Q-1b5
34.90
26.33
30.95
25.98
31.08
29.46
B-b
36.19
29.90
23.90
23.60
25.70
28.28
Q-7b
27.58
31.48
25.25
21.60
25.41
25.44
B-1
22.13
32.29
19.33
23.16
18.63
17.45
G-2b
21.17
19.92
34.73
23.33
23.49
22.64
R-b
25.63
18.70
32.63
21.61
26.40
22.50
G-9b
18.63
17.42
22.12
26.28
20.99
20.73
R-1
25.59
28.98
25.15
32.73
24.12
25.15
L-3b
30.49
22.35
33.49
22.19
35.57
33.03
E-b
36.01
26.99
31.96
25.77
41.27
36.97
L-8b
28.24
22.28
30.03
25.80
32.03
33.43
E-1
31.85
31.89
30.63
24.15
32.99
38,18
Table 5: Cross-Model Transferability of TOA the NER Task. Entries represent Fl score (unit: %).
Best performance is bolded; scores exceeding the zero-shot baseline are underlined.
(a) CLM Results.
(b) MLM Results.
Sre/igt Q- lb5
Q-7b G-2b
G-9b
L-3b
L-8b
SreVigt B-b
B-1
R-b
R-I
E-b
E-1
Zero-shot
Zero-shot
5.35
28.21
1.45 53.82
13.24 22.12
0.00
0.00
0.00
0.00
0.00
0.00
TOA (Ours)
TOA (Ours)
Q-1b5
53.81
30.99
21.08
9.58
29.24
26.94
B-b
68.47
46.00
29.74
16.62
39.14
39.74
Q-7b
36.28
54.85
15.56
10.01
26.17
28.30
B-1
36.40
68.74
9.69
14.14
14.54
13.21
G-2b
23.12
12.47
54.53
24.31
22.64
15.80
R-b
41.66
36.02
60.99
40.60
39.85
42,90
G-9b
12.80
9.94
31.15
55.36
16.65
12.74
R-1
31.43
36.12
40.31
64.60
20.50
27.35
L-3b
27.48
21.79
24.79
14.05
54.51
45.24
E-b
36.28
25.53
20.83
12.21
65.14
46.87
L-88h
27.93
25.33
20.63
16.34
47.84
51.68
E-1
24.62
23.98
17.71
8.38
34.51
67.08
Speech Tagging (POS). Since TOA is directly transferred to the target LM without any training (it is trained solely the source LMM, compared its performance against the zero-shot capability of the target LM to assess the utility of TOA. Specifically, the TOA transfer process involves training an adapter using the OLA generated by the source LM on the training set and evaluating its performance on the OLA produced by the target LM on the test set. For zero-shot evaluation, we guided LMs' predictions using manually designed prompts, with prompt templates and implementation details provided in KK The results for RE and NER are presented in Tables 4 and S while those for DP and POS are included (Tables [5 and 16). Detailed implementation for TOA transfer (adapter
architectures, datasets, metrics) are elaborated in SG
Main Results. From Tables 15 and T6 we derive the following insights: 1) In most settings, TOA consistently surpasses the zero-shot baseline, demonstrating its practical utility. For MLMs, TOA surpasses zero-shot performance across all source-target LM pairs all four tasks. For CLMs, TOA exceeds zero-shot performance in all source-target LM pairs for RE and DP tasks, but underperforms zero-shot in a small fraction of NER and POS cases (less than 6% of total scenarios). These underperforming cases primarily involve larger, high-capacity target LMs (c.g.. Gemma2-9B). This suggests that TOA may not improve performance for large. high-capacity models on some tasks but delivers significant gains for smaller models like Qwen2-1 1we52-1.5BB and Llama3.2-23 3.BBBB. 2) Cross-model transfer incurs insignificant performance degradation compared to self-transfer. The diagonal entries in the tables represent self- transferneer results, where the source and target models are identical (i.e., adapters are trained and tested on the same model). These entries serve as an upper-bound baseline for TOA's transfer capability. Remarkably, cross-mode mossl performance remains close to this baseline. For example, transferring TOA from BERT-base to BERT-large achieves 29, 29.90% accuracy on the RE task, reaching 93% of BERT-large's self-transee transfer performance (32.29%). This indirectly

---

supports the existence of the OLAS phenomenon. 3) TOA achieves stronger performance with MLMs than CLMs. This may stem from the bidirectional attention in MLMs, which captures richer contextual information compared to the unidirectional attention in CLMs. 4) TOA performs better on syntax-dependent tasks (DP, POS) than on semantics-driven tasks (RE, NER). This suggests that OLA primarily encodes syntactic structures rather than semantic knowledge.
5 Conclusion
In this work, we introduced a novel perspective for analyzing LMs by focusing on the commonalities in context aggregation patterns. We revealed the significant similarity in OLA across different LMs, marking key discovery in understanding the shared mechanisms of LMs. Furthermore, we explored the linguistic implications OLA and found that it encodes syntactic knowledge. Building on these findings, we proposed the TOA, achieving training-free cross-LM adapter transfer. Extensive experiments demonstrate that TOA trained on other LMs can be transferred to unseen LMs to enhance their performance, yielding promising results.
6 Acknowledgement
The research is partially supported by National Natural Science Foundation of China (No. 62576139, 62176093, 61673182), National Key Research and Development Program of China (No (N0.2023YFC3502900), Guangdong Emergency Management Science and Technology Program (No.2025YJKY001)
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[3] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 1e.e24443..60.... 14608, 2024.
[4] Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings the IEEE/CVF conference 2022.
computer vision and pattern recognition, pages
[5] Junjie Yang, Junhao Song, Xudong Han, Ziqian Bi, Tianyang Wang, Chia Xin Liang, Xinyuan Song. Yichao Zhang, Qian Niu, Benji Peng, et al. Feature alignment and representation transfer in knowledge distillation for large language models. arXiv preprint arXiv:2504.1 13855, 2025.
[6] Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023.
[7] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[8] Ang Lv, Yuhan Chen, Kaiyi Zhang, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms factual recall in transformer-based language models. arXiv preprini arXiv:2403. 2024.
[9] Ulme Wennberg and Gustav Eje Henter. The case for translation-invarian self-attention in transformer-basee language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 130-140, 2021.
10

---

[10] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-41977 2020.
[11] Javier Ferrando, Gerard Gállego, and Marta R Costa-jussà. Measuring the mixing of contextual information in the transformer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8698-8714, 2022.
[12] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Analyzing ceed-forward blocks in transformers through the lens of attention maps. In The Twelfth International Confer- ence on Learning Representations, 2024.
[13] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 3543-3556, 2019.
[14] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lunguage P Processing (EMNLP-IJCNLA) pages 11-20, 2019.
[15] Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajic, Christopher D Manning. Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. Universal dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language 2020.
Resources and Evaluation Conference, pages
[16] Gang Dai, Yifan Zhang, Quhui Ke, Qiangya Guo, and Shuangping Huang, One-dm: One-shot diffusion mimicker for handwritten text generation. In European Conference on Computer Vision, pages 410-427. Springer, 2024.
[17] Gang Dai, Yifan Zhang, Yutao Qin, Qiangya Guo, Shuangping Huang, and Shuicheng Yan. Beyond isolated words: Diffusion brush for handwritten text-line generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19054-19064, 2025.
[18] Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018.
[19] Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus-Robert Müller, Matteo Valleriani, and Grégoire Montavon. Building and interpreting deep similarity models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1149 191-1199111111 2020.
[20] Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T Schütt, Klaus- Robert Müller, and Grégoire Montavon. Higher-order explanations of graph neural networks via relevant walks. IEEE transactions on pattern analysis and machine intelligence, 44(11):7581- 7596, 2021.
[21] Alexandros Vasileiou and Oliver Eberle. Explaining text similarity in transformer models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationnal Linguistics: Human Language Technologies (Volume Long Papers), pages 7852-7866, 2024.
[22] Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104): 1-54, 2021.
[23] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. Kernelshap-iq: Weighted least square optimization for shapley interactions. In International Conference Machine Learning, pages 14308-14342. PMLR, 2024.
[24] Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1264-1274, 2018.
[25] Kevin Clark. What does bert look at? an analysis of bert's attention. arXiv preprint 1rXiv:1966.04341, 2019.

---

[26] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pages 63-76, 2019.
[27] Joseph F DeRose, Jiayao Wang, and Matthew Berger. Attention flows: Analyzing and comparing attention mechanisms in language models. IEEE Transactions on Visualization and Computer Graphics, (7): 160-1170, 2020.
[28] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18/01///0000 11..880..48555 2018.
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
[30] Sofia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931-2951, 2019.
[31] Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M Khapra, Bal- aji Vasan Srinivasan, and Balaraman Ravindran. Towards transparent and explainable attention models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2206-421,, 2020.
[32] Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability transformers. 8th International Conference on Learning Representations (ICLR 2020avir International Conference on Learning Representations, 2020.
[33] Kaiser Sun and Ana Marasović. Effective attention sheds light on interpretabillty. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages
2021.
[34] H Thomas et al. Introduction to algorithms, 2009.
[35] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
[36] Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Incorporating residual and normalization layers into analysis of masked language models. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 4547 2021.
[37] Ali Modarressi, Mohsen Fayyaz, Yadollah Yaghoobzadeh, and Mohammad Taher Pilehvar. Globenc: Quantifying global token attribution by incorporating the whole encoder layer in transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 258-271, 2022.
[38] Ali Modarressi, Mohsen Fayyaz, Ehsan Aghazadeh, Yadollah Yaghoobzadeh, and Moham- mad Taher Pilehvar. Decompx: Explaining transformers decisions by propagating token de- composition. Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 2649-2664, 2023.
[39] Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide 2023.
to parameter-efficient fine-tuning, arXiv preprint
[40] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, 2021.
[41] Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, 2020.
12

---

[42] Xuming Hu, Aiwei Liu, Yawen Yang. Fukun Ma. S Yu Philip. Lijie Wen, et al. Enhancing cross-lingual natural language inference by soft prompting with multilingual verbalizer. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 1361-1374, 2023.
[43] Yiran Zhao, Wenxuan Zhang. Huiming Wang, Kenji Kawaguchi, and Lidong Bing. Adamergex: Cross-lingual transfer with large language models via adaptive adapter merging. arXiv preprint arXiv:2402. 18913, 2024.
[44] Jiayi Wu, Hao Sun, Hengyi Cai, Lixin Su, Shuaiqiang Wang, Dawei Yin, Xiang Li, and Ming Gao. Cross-model control: Improving multiple large language models in one-time training. In The Thihty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[45] Zijun Wu, Yongkang Wu, and Lili Mou. Zero-shot continuous prompt transfer: Generalizing task semantics across language models. In The Twelfth International Conference on Learning Representations, 2024.
[46] Gang Dai, Yifan Zhang, Qingfeng Wang, Qing Du, Zhuliang Yu, Zhuoman Liu, and Shuangping Huang. Disentangling writer and character styles for handwriting generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5977-5986, 2023.
[47] Jinglin Liang, Jin Zhong, Hanlin Gu, Zhongqi Lu, Xingxing Tang, Gang Dai, Shuangping Huang, Lixin Fan, and Qiang Yang. Diffusion-driven data replay: novel approach to combat forgetting in federated class continual learning. In European Conference on Computer Vision, pages 303-319. Springer, 2024.
[48] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arX1v 2023.
[49] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024.
[50] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[51] Jim Nilsson and Tomas Akenine-Möller Understanding ssim. arXiv preprint 18844 2020.
[52] Jinglin Liang, Yutao Qin, Shuangping Huang, Yunqing Hu, Xinwu Liu, Huiyuan Zhang, and Tianshui Chen. seowledge-embeddee graph representation learning for document-level relation extraction. Expert Systems with Applications, 128872, 2026.
[53] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[54] Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. Conll-2012 shared task: Modeling multilingual unrestrieted coreference in ontonotes. In Joint conference on EMNLP and CoNLL-shared task, pages 1-40, 2012.
[55] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi-way way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38, 2010.
[56] EF Tjong Kim Sang and S Buchholz. Introduction to the conll-2000 shared task: Chunking. In
Proceedings of the Fourth Conference on Computational Natural Language Learning and of the Second Learning Language Logic Workshop (CONLI/LLL 2000) Lissabon, Portugal, 13-14 september 2000, pages 127-132. ACL, 2000.
[57] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the asxociation for computational linguistics: Human language technologies, pages 142-150.
2011.
13

---

[58] Jonathan Ho. Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multicimensional transformers. arXiv preprint arXiv:1912.12180, 2019.
[59] Rrubaa Panchendrarajan and Aravindh Amaresan. Bidirectional Istm-crf for named entity recognition. In Proceedings of the 32nd Pacific Asia conference on language, information and computation, 2018.
[60] Alireza Mohammadshahi and James Henderson. Recursive non-autoregressive graph-to- to to-graph transformer for dependency parsing with iterative refinement. Transactions of the Association for Computational Linguistics, 9:120-138, 2021.
14

---

A Visualization of Attention Rollout
We present the visualization results of Attention Rollout on six LMs, including three MLMs and three CLMs. Specifically, we show the visualizations of Attention Rollout obtained from different text inputs in Figure [ (a) and (c), and the results after setting outliers in Attention Rollout zero in Figure 5 (b) and (d). Outliers are defined as values greater than the mean of the row plus three times standard deviation. The detailed calculation process is provided in S
From the figures, we observe that Attention Rollout exhibits a significant attention sink phenomenon, with attention being highly concentrated few unimportant tokens. Additionally, the Attention Rollout of different sentences lacks distinguishability. indicating that it is challenging to use it as feature representation for sentences.
Telt: ANM statonory oifank thst pricomude side has of 1ED lott and wightt of pumped Foom the statonwry anot
snight
cased
base
Olwen2
E
E
(a)
(b)
Tert? Beeekeiing www.eroooooooneooooooooooooo wnligeoen weaarorcs sia saiggee vninncig,, canoeaiii www.iiiioee...o.oo..ooo www.eielllyy.... www.ooogg westic int.ssceeeiiiitttooooo proseee and profusionall seztnng, kuding improwed probumssshre dacakon mulling
Bert-base-cased
Roberta-baad base
Electra-base
Qwwnn2 1.58
Gemma2 28
LIaMA3.2 38
E
(c)
l:
(d)
50 00 00
Figure 5: Visualization results of Attention Rollout obtained from two texts input into Bert-base- cased, Roberta-bass Electra-base, Qwen2-1.5b. Gemma2-2h, and Llama3.2-3h (a) Attention Rollout for Text 1. (b) Attention Rollout for Text 1 with outlier values set to zero. (c) Attention Rollout for Text 2. (d) Attention Rollout for Text 2 with outlier values set zero.
15

---

B Visualization of OLA
We visualize the first-, second-, and third-order OLA obtained by inputting two texts into LMs, as well as the results after setting outlier values in the OLA to zero. The visualizations for CLMs are presented in Figure 6 and those for MLMs are shown in Figure 7 Outliers are defined as values greater than the mean of the row plus three times the standard deviation. The detailed calculation process is provided in F The comprehensive visualization results further support our conclusions in 3.2
Teat1: A full stacionary oll tank that is right droular cylinder has radius of 100 feet and height of 25 feet. all pumped from the stationary tank to an truck that has tank that right cicular cylnner until the truck: tank completely filied.
Owen22
FET
Gemma2
38
Developing emotional intelligecee enhances your ability navigate relationships, communicate effectivel,, manage emotions in both personal and professional settings, leading to improved problem solving skills and better deciion making.
Qwen22
777 PPPPET
Gemma2
b
LLaMA3.2
FET
FT1
(a)
(b)
(c)
(d)
(e)
Figure 6: Visualization results of finst-order, second-order and third-order OLA obtained by inputting two texts into Qwen2-1.5b 1.5b, Gemma2-2b and (lama3.2-3b (a) Finst-order OLA. (b) First-order OLA with outlier values set zero. (c) Second-order OLA. (d) Second-order OLA with outlier values set to zero. (e) Third-order OLA. (f) Third-order OLA with outlier values set to zero.
16

---

Teat:: full stationary bank that right circuler offinder has redius 100 feet and height feet, oilis pumped from the stationary cylinder uniil trudi'' tank completaly filied.
Bert-base-
Roberta
Electra-base
Tedt2: Developing ematioral intelliennce enhances your ability to navegate relationships, communicate effectivly, and manage emotions both personal and grofossional settings. leading to improved problem solving skils and better dectioon making.
Bert base- cased
Roberta-tan
Electra-base
(a)
(b)
(c)
(d)
(e)
(f)
Figure 7: Visualization results of first-order, second-order and third-order OLA obtained by inputting
two texts into Bert-base-cased, Roberta-base and Electra-base. (a) First-order OLA. (b) First-order
OLA with outlier values set zero. (c) Second-order OLA. (d) Second-order OLA with outlier
values set to zero. (e) Third-order OLA. (f) Third-order OLA with outlier values set to zero.
17

---

C Language Models we used
We conducted experiments on twelve commonly used LMs, including six CLMs: Qwen2-1. Qwen2-7B Gemma2-2HH Gemma2-9B Llama3.2-3E and Mama3.11883 and six MLMs: Bert- base-cased bert-large-casee
Electra-basd and Electra-large For Electra, which consists of generator and discriminator, we use the generator because its training objective aligns with traditional MLMs such as Bert and Roberta. The architectural hyperparameters, training data size, and vocabulary size of these models are detailed as Table and Table 7
Roberta-bat baserta-basaa Roberta-large
Table Details of masked language models.
Models
Bert-hase-cased Bert-large-cased Roberta-hase Roherta-large Electra-base Electra-large
Hidden Size
768
1,024
768
1,024
768
1,024
Layers
12
24
12
24
12
24
Attention Heads
12
16
12
16
4
Head Size
64
Vocabulary Size
28,996
28,996
50,265
50,265
30,522
30,522
Trained Tokens
3.3B
3.3B
3.3B
33B
Table 7: Details of causal language models.
Models
Qwen2-1.5B
Qwen2-7B Gemma2-2B Gemma2-9B Llama3.2-38 Llama3. 1-8B
Hidden Size
1,536
3,584
2,304
3,584
3,072
4,096
Layers
28
28
26
42
28
32
Query Heads
12
28
8
16
24
32
Key Value Heads
8
Head Size
128
128
256
256
128
128
Vocabulary Size
151,936
152,064
256,000
256,000
128,256
128,256
Trained Tokens
7T
7T
2T
8T
9T
15T
"Qwen2-1.5B:
Instruct
http:///hhtaeeee///////////////0 insttrr Instruct
Gemmm2-225 http://s//igggii/oooooooo/o///// 2-2 1t
GGemma2-99 9B:
6 Llama3.2-333
3.2 3B Instruct
"Llama3.1-88
Instruct
$ (Bert-base-cased:
htap:c//de//caea//ooo/////e///oo base- cased
lBrrt-large-cased
http://wwww.pp...///...////////r cased
Rooertaa-base:
larerta-large::
Fleetra-base:
Electra-large:
18

---

D Derivation of Norm-based Decomposition for CLMs
The norm- momm-based method requires decomposing the features of the language model into terms associ- ated with each token, and this decomposition process is highly tied to the model's structure. Since ALTI and IRNL only derive the decomposition for MLMs, and due to the structural differences between CLMs and MLMs, the decomposition expression in ALTI and IRNL cannot be directly applied to CLMs. To adapt ALTI and IRNL to CLMs, we derive the decomposition for CLMs.
D.1 Decomposition for Llama and Qwen
The structures of Llama and Qwen are shown in Figure aa. important note that their LN module is not the standard LayerNorm, but instead Root Mean Square Layer Normalization (RMSLN)
[53] expressed as:
1 d j=1
RMSLN((;)) RMS(x;))* Ts: where RMS(x;)
(5)
Here, Hy E Rixd represents the input feature of the i-th token, d is the hidden dimension of the language model, H. represents the output feature of the RMSLN for the i-th token, RMS(1,) denotes the root mean square of 2ú, denotes the i-th element of, and is a learnable parameter.
Then, analyze the decomposition of MHA, which expressed as follows:
2, cat 2/ 24 .... 23 Wo.
(6)
Σ 4 Wh.
H
Σ
Σ
Whh Aly 8j W.t w/.
Al By W. //)
Σ
-E RMS(xr) Σ xz W w/b
Here, 3; represents the feature of the i-th token in the output of MHA, A denotes the i-th row and j-th column element of the attention matrix for the h-th head, 2/ represents the output after aggregating the values of other tokens for the i-th token in the h-th head. W. E Rdx(MxE) is the value projection matrix of MHA, where M is the number of heads for the keys and values, and E is the dimension of each head. W, E R (H E) represents the matrix obtained by replicating W, of H /MM times and then concatenating, and H is the number of heads for the query. W/ E RRdx represents the h-th block of W.. W, E R( xE represents the output projection matrix of MHA, and W e I Exd represents the h-th block of Wo.
Since there is a residual connection in parallel with MHA and LN, the output of the attention block can be expressed as:
H x. Wh wh + xi.
Σ
Σ Aly
y
RMS(2,)
h
where ys represents the output feature of the i-th token in the attention block. In summary, the decomposition expressions for Llama and Qwen as follows:
H
A My Wh Wh + Jy A zy Wh Wh
ifi=j
T(x,)) =
RMS( RMS((a))
(8)
if
where (x represents the term associated with xy obtained after decomposing the output feature of the attention block for the i-th token.
19

---

D.2 Decomposition for Gemma
As shown in Figure 2 c), Gemma2's differences from Llama and Qwen are mainly in two aspects: 1) Its LN is also RMSLN, but its learnable parameters differ, as it includes fixed bias of size 1; 2) Its attention block has two LNs, one before and one after MHA. Therefore, the expression for the first I.N is:
1+71 RMS(x;) Zs:
1 d j=l
Σ z 2)
23 RMSLL((;;))
where RMS(x:)
(9)
Here, 71 represents the learnable parameter of the first LN.
The decomposition of MHA is as follows:
4 Σ
1+7 RMS(ay)
Σ
A
Iz
W whb
(10)
The output the attention block can be expressed as:
y: RRMLL (a,) + + RMSLN
Σ
1+
Σ Al my W w
+ 22 (11)
RMS(2,)
(1+m) (1+ 722
H
Σ
Aly 23 W W
+ xss
RMS(1.) RMS(2,)
where represents the learnable parameter of the second LN.
In summary, the decomposition expressions for Gemma is as follows:
(1+xy)-(11a)) Σ A W Wh + Iss
if j
T((,,)
RMS(3:) (AMs(F:)-RMSS)) RMS(a;)
h
(12)
(147))-(1+7))
Σn
A Wh
W if
RMS(3;) (RMM))
20

---

E Datasets
This paper employs the following five datasets, which are introduced below.
CoNLL2012 [4 It is a multilingual, multi-genre, and multi-tast dataset. It supports tasks such as Named Entity Recognition (NER), part-of-speech tagging, coreference resolution, and more. We utilize the corpus of this dataset in the quantitative analysis experiments of OLAS ($3.3.1) and employ both the corpus and its NER annotations in the experiments exploring the cross-model transferability of TOA on the NER task ($4.2.. The dataset contains 1,940 documents in the training set and 222 documents in the test set. Each document has an average of 39 sentences.
UD-English-EWT v2.15 [5]. It is a subset of the Universal Dependencies project serves as a key resource for dependency parsing. This corpus primarily consists of English texts sourced from web-based content, including blogs, reviews, and social media posts. We utilize this dataset in the experiments investigating the relationship between OLA and syntactic knowledge ($3.4). and the experiments exploring the cross-model transferability of TOA on the DP task (44.2). ). Its training set contains 12,544 sentences, and the test set contains 2,077 sentences.
SemEval-2010 Task (55]. It a widely adopted benchmark dataset for relation extraction, specifically designed for multi-way classification of semantic relations between pre-identified entities. Each instance in the dataset is annotated with two entities and the semantic relation between them that requires classification. We use the dataset in the experiments exploring the cross-model transferability of TOA on the RE task ($4.2). Its training set contains 8,000 sentences, and the test set contains 2,717 sentences.
CoNLL2000 [56]. It serves as widely adopted benchmark for POS tagging and text chunking tasks. We use the dataset in the experiments exploring the cross-model transferability of TOA on the POS Its training set contains 8,937 sentences, and the test set contains 2,013 sentences.
task 2
IMDB 577 It is widely used dataset for binary sentiment classification, containing movie reviews its corpus. In the controlled experiments of the quantitative analysis for OLAS ($H), we utilized the corpus from this dataset. Both its training and test sets contain 25,000 sentences each.
F Implementation Details of Quantitative Analysis of OLAS
We sequentially sampled 1000 sentences from the CONI.L-2012 154 dataset, i.e., M in Equation is 1000. Details about the dataset are in SEE] We focus on the case where the source and target LMs do not belong to the same series, i.e., S in Equation f is 4. For example, we use models of different sizes from the Llama and Qwen series as the source models, and models from the Gemma series as the target models. In this setup, there are significant differences in the research institutions, training data, model architectures, and tokenizers of the source and target models. To mitigate the effects of sentence length bias, we retained only the first 50 words of each sentence, including punctuation marks. Each sentence assigned unique identifier ranging from 1 to 1000, which used to train the image classification model. For the image classification model, we utilized ResNet-18 18 (50) with the input channels modified to 1 and the output layer dimension set 1000. Due small number of outliers that deviate noticeably from the overall distribution in both OLA and baseline results, the overall feature distribution is not clearly visible, as shown in Figure 3 (a), (b), and (d). To address this, we apply a consistent preprocessing procedure. Specifically, we calculate the mean and standard deviation of each row, outliers greater than μ + 30 zero, and then normalize the values by dividing by the row sum. Since the number of tokens obtained from sentence tokenization may not exactly correspond the number of words in the original sentence, we resize all OLA maps to 50 x 50 dimension. Additionally, as the OLA of CLMs is a lower triangular matrix, resizing introduces non-zero values in the upper triangular area. To avoid leaking tokenized sentence length information, we multiply the OLA by lower triangular mask. Moreover, we employed common data augmentation techniques, including Gaussian noise, temperature scaling perturbations, and random highlighting. For each experimental setup, we trained with three learning rates: le-2, 5e-3, and 3e-3, and report the best perf-perf results. All experiments were conducted on single 40GB NVIDIA A100 GPU.
httpg:///ntveeraallleedddeeeeeee org/
21

---

G Implementation Details of Transferable OLA Adapter
In this section, we present the implementation details of TOA for the four tasks: RE, NER, DP, and POS ($4.2) Since there are overlaps in the implementation details across these tasks, we consolidate their common aspects in the subsection below titled "Common Implementation Details". Additionally, due to the significant overlap between the experiments investigating the relationship between OLA and syntactic knowledge ($3.4) and the implementation of TOA on the DP task, these are discussed within the DP task section.
G.1 Common Implementation Details
OLA Preprocessing. We apply the preprocessing operations described $ remove outliers and normalize both the OLA and baseline outputs. The processed results are then concatenated along the channel dimension form the final input tensor. This tensor undergoes data augmentation using the operations described in {F)
Feature Extractor in the Adapter: Considering that the semantics of attention distributed across rows and columns, where the i-th row represents the weights assigned to other tokens when the i-th token, query; aggregates other tokens, and the i-th column represents the weights assigned the i-th token as a key when it is aggregated by other tokens, we use an axial transformer (58), which can extract semantics across rows and columns, to compose the feature extractor our OLA adapter. The specific architecture of the feature extractor based on the axial transformer is as follows: the input map X Rx & is first passed through a convolutional layer with kernel size of x to increase its dimensionality to 768. It is then processed by several layers of axial transformers to produce the feature map Fn E RR 708x I, where C represents the number of channels and L represents the number of tokens. The diagonal features of are extracted to form the feature sequence Fi E IR 76768
Hyperparameters. For the CLMs experiments, the axial transformer consists of layers, while for the MLMs experiments, it consists of 3 layers. The number of epochs is set to 15. For each experimental setup, we trained with three learning rates: Ie-4, 3e-5, and Ie-5, and report the best- performing results.
G.2 Task-Specific Implementation Details
RE. This task requires predicting the relationship between two entities in sentence. Therefore, the adapter is structured to extract features of entity 1 and entity from the feature extractor's 's output Fi based on their annotated positions, concatenate these features, feed them into fully connected layer, and produce the output E 19 where 19 denotes the number of relationship categories. The evaluation metric is the relationship classification accuracy.
This task requires identifying entities sentence, where we convert the NER task into a sequence labeling task using BIO tagging [59]. Therefore, the adapter is structured to pass the feature extractor's output Fj through a fully connected layer, producing the output y E x, where 37 denotes the number of BIO tag categories. The evaluation metric is the Fl score.
NER.
DP. This task requires identifying the head (governor) each word in a sentence and its corresponding dependency relation. Therefore, the adapter is structured to pass the feature extractor's 's output Fi through two separate MIP+biaffine modules [60), producing two outputs: 3/1 LR for predicting the head of each token, and E R55x Lx for predicting the dependency relation between each token and its head. Here, 55 denotes the number of dependency relation categories. The evaluation metrics are UAS (Unlabeled Attachment Score, measuring accuracy of head prediction without relation labels) and LAS (Labeled Attachment Score, measuring accuracy both head and relation prediction).
POS. This task requires identifying the POS for each word in a sentence. Therefore, the adapter is structured to pass the feature extractor's output F; through a fully connected layer, producing the output y R. L where 45 denotes the number of POS categories. The evaluation metric the POS classification accuracy.
22

---

H
Experimental Results with Controlled Dataset and Preprocessing
Controlled Dataset Experiment. When validating OLAS in $3..11 sequentially selected the first 1,000 sentences of length 50 from CoNLL-2012. This approach avoids subjective bias through manual selection and ensures reproducible stability compared to random sampling. To verify whether the data selection generalizes to broader scenarios, we extended the experiments presented in Table with the following three settings: (1) Replacing CoNLL-2012 with the IMDB dataset [57], (2) Increasing the sentence count from 1,000 to 2,000, and (3) Extending the sentence length from 50 to 80 words. As shown in Table [ while experimental results vary across different data settings-primaree inherent differences in data similarity between datasets- consistently high performance under all configurations further validates the OLAS findings reported in the main text, demonstrating their reproducibility across diverse data seftings.
due to
Data Augmentation Ablation Study. Our data preprocessing pipeline consists of three main steps: (1) Outlier Removal: Remove data points beyond three standard from row means. As shown in Figure 3] this prevents attention map suppression by extreme values in both OLA and baselines, ensuring stable training of the visual model. (2) OLA Size Standardization: Unify OLA dimensions across text sources to eliminate size-related information leakage, forcing the visual model to rely solely on visual features. (3) Data Augmentation: In our qualitative analysis, OLA generated by different LMs for the same text are grouped into one class for training the image classifier, resulting in only four samples per class (equal to the number of source models). Common augmentation overfitting in the visual model. Both outlier removal and OLA size standardization are fair and necessary operations. To verify whether the optional data augmentation introduces unintended bias, we conducted an ablation study (T. Comparisons with Table the main text reveal that while augmentation moderately reduces overfitting and improves classifier accuracy, removing it does not alter our core conclusion-O11 LLAs still exhibit the most pronounced similarity patterns.
applied to alleviate
Table 8: Quantitative OLAS analysis under three data settings. Rows 4-6: Results of dataset sub- stitution (CoNI.L-2012 ~> IMDb); Rows 8-10: Sentence count adjustments (1k 2k); Rows 12-14: Sentence length extensions (50 + 80 words).
Table 9: OLAS Quantitative Evaluation Results Without Data Augmentation.
L-3b, L-8b,
L-3b, L-8b, 1-5,, Q-7b
Q-1b5, Q-7b.
Source
G-2b, G-9b
G-2b, G-9b
L-3b, L-8h,
L-3h, L-8b,
Q-1h5, Q-7b, G-2b, G-9b
Target
Q- 1b5
Q-7b
G-2b
G-9b
L-3b
L-8b
Source
G-2h, G-9b
Q-1b5, Q-7b
Dataset
Target
Q-115
Q-7b
G-2b
G-9b
L-3b
L-8b
1st
80.00
79.50
84,00
84.10
97.00
96.20
Rollout
20.30
7.10
31.30
16.10
48.20
37.30
2nd
70.60
65.00
77.90
73.00
91.50
92.60
IRNL
8.90
5,00
47.10
57.50
70.60
64.80
3rd
66.90
57.30
73.10
68.70
89.20
88.30
ALTI
4.60
3.20
55.60
56.90
64.00
68.20
Data Num
1st
41.00
40.00
86.20
60.20
91.80
89.60
1st
72.80
64.35
86.25
83,05
93.55
94.15
2nd
50.30
40.80
86.50
76.80
85,90 ,0
83.90
2nd
61.00
45.40
79.40
77.30
88.05
87.25
3rd
59.30
44.20
77.40
75.20
86.20
81.90
3rd
63.10
33.85
77.05
74.40
85.50
84,95
Data Len
1st
62.60
60.30
91.20
87.90
90.10
87.20
2nd
58.60
42.20
97.70
96.60
95.00 93.80
3rd
53.80
25.50
85.70
84.90
92.80 92.00
23

---

I Experimental Results with Controlled Parameters
To ensure that the results of our quantitative analysis experiments ($3.3.) reflect the inherent properties of LMs (i.e., the commonality of contextual aggregation patterns learned from large corpora) rather than confounds caused by data or other experimental biases, we conducted the following two controlled experiments:
Perturbations on Source Model Parameters: We perturbed the parameters of the source LMs while keeping the target LM unchanged and repeated the OLAS quantitative analysis experiment (Table T The perturbations included two types: Random (randomizing model parameters) and Disorder (shuffling the order of model layers). As shown in Table I0, nearly no OLA similarity was observed between perturbed and normal models. This indicates that the OLAS phenomenon is intrinsically tied to pre-trained model parameters, rather than arising from other experimental biases.
Exploring OLAS Across Structurally Identical Models with Varied Parameters: Taking Qwen2-1.558 as an example, the source models included four perturbed variants: Q-r1, Q-r2 (randomized parameters under different random seeds), and Q-dI, Q-d2 (disordered layers under different random seeds). The target models included the unperturbed Qwen2-1.55D and additional perturbed variants (Q-d3 and Q-r3). Under the configuration of the source and target LMs,, we conducted the OLAS quantitative analysis experiments similar those presented in T Because the source and target models share identical architectures, this stricter experimental setup isolates the impact of parameter variations. We further extended this analysis to Gemma2-9p -B and LLaMA3.2-3B. As shown in Table [T] no OLA similarity observed between source and target models in any scenario, further supporting the conclusion that OLAS depends on pre-trained parameters and eliminating confounds from other experimental setups.
Table 10: Results of Perturbations on Source Model Parameters. Rows 4-6: Random (parameter randomization); Rows 8-10: Disorder (layer shufiling).
L-3b, L-8b,
L-3b, L-8b, 0-1b5. Q-7b
Q-1b5, Q-7b, G-2b, G-9b
Source
G-2b, G-9b
Target
Q-1b5
Q-7b
G-2b
G-9b
L-3b
L-8b
Random
1st
0.90
0.90
0.30
0.20
0.90
0.50
2nd
1.20
1.00
0.90
1.60
0.40
1.30
3rd
1.30
0.90
1.00
1.40
0.90
1.10
Disorder
1st
0.70
0.60
1.20
1.30
1.10
0.80
2nd
1.30
0.60
1.20
1.40
0.90
1.40
3rd
0.80
1.00
0.90
1.50
1.10
1.60
Table 11: Results of exploring OLAS across structurally identical models with varied parameters.
Q-rl, Q-r2,
G-rl, G-r2,
L-rl, L-r2, L-d1, L-d2
Source
Q-d1, Q-d2
G-d1, G-d2
Target
Q-1b5
Q-r3
Q-d3
G-2b
G-r3
G-d3
L-3b
L-r3
L-d3
0.60
1,90
1.30
0.30
1.80
3.80
0.40
1.20
2.50
2nd
1.50
3.10
1,30
1.80
1.40
3.60
0.40
2.50
2.70
3rd
2.70
1.80
2.80
1.20
1.80
3.90
0.10
3.40
3.30
24

---

J Additional Result of Quantitative Analysis Based on Image Similarity Retrieval
We conducted retrieval-based quantitative evaluations on the cross- moss-model similarity of OLA across first-, second-, and third-order configurations, with the results presented in Table T2
Table 12: Retrieval- based quantitative evaluation of first- third-order OLA cross-model similarity. Rows denote source LMs, columns represent target LMs, with entries reporting Hits@) / @ metries.
(a) CLM Results.
(b) MLM Results.
Ste\Tgt
Q-1b5
G-2b
L-3b
SreATgt
B-b
R-b
E-b
1st
1st
Q-1b5
83.60 / 89.40 95.90/97.0
B-b
51.90/ 58.80 91.60/94 94.90
G-2b
83.20
02.0/89..0 89.30
95.30/ 97.10
R-b
75.90 83.90
71.70/80.20
L-3b
92.90
96.10 /4.10/ 96.50
E-b
92.40 / 96.00 67.40 72.90
2nd
2nd
Q-1b5
75.20 / 83.60
91.50 95.60
40.10 48.00
92.20 92.20/95.00
G-2b
74.20/ 82.50
93.60 / 96.40
R-b
51.20/68.22
56.50/ 68.80
L-3b
85.40 / 92.60 91.80/95.20
E-b
88.10 93.50 45.00 53.90
3rd
3rd
Q-1b5
57.10 / 72.00 69.90 88.40
B-b 18.10/63.99
25.60 / 36.00 85.70/ 93.10/93.10
G-2b
58.30 71.90
89.90 94.60
R-b
53.80 66.80
L-3b
64.40/ 81.60
/6.40 92.90
E-b
85.70 / 93.00 26.90 40.10
25

---

K
Implementation Details of LLM Zero-Shot Evaluation
We use hand-crafted prompts guide LMs in generating formatted outputs, then structure these outputs according to rules to extract their predictions. The prompt templates for RE and NER tasks are presented Table 13 while those for DP and POS tasks are shown in Table T4
Table 13: Prompt templates for RE and NER tasks.
Task
Model type
Prompt template
Act as a relation extraction tagging tool. Find the relationship between el and c2 in the given sentence by choosing the correct option number from [REL_LABELS __TR).
MLM
RE
Sentence: (sentence).
el: [el]
e2: (e2)
Response:
The relationship number is {mask_sttr
Act as a relation extraction tagging tool. Find the relationship between el and in the given
sentence according to these rules:
Choose the correct option number from {REL_ LABELS_SS |_RR].
2. Do not explain or add extra text. Only provide the option number.
CLM
Sentence: (sentence).
el: [el]
e2: [e2]
Response:
Act as a named entity recognition tagging tool. Given the sentence: "(sentence)", determine whether the span "(span)" is named entity.
not a named entity, respond strictly with "none". If it is named entity, select the correct category from LABELS_SSSSSS
MLM
NER
and respond only with the corresponding number.
Response: mmask_str).
Act named entity recognition tagging tool. Find all entities and their classes a sentence
according to these rules:
1. Choose the correct named entity class from
ANER_LABER (RELS.
2. Do not explain or add extra text.
CLM
Sentence: (sentence).
Response as tuples, and each tuple must have exactly two elements: first element is the named entity text (as string), second element is the named entity class (as string), e.g. (centityl>, <class>>), (centity2>, <class2>), Response:
26

---

Table 14: Prompt templates for DP and POS tasks.
Task
Model type
Prompt template
Act as dependency relation analyzing tool. Find the head and dependency relation of the given word in a sentence according to these rules:
1. Choose the correct head number from (words_ _aads
2. Choose the correct dependency relation from (REL MABEELSSSRR
MLM
DP
Sentence: {" ".join(words))
Word: {word)
Response: the head number (mask_str), the dependency relation is (mask_str).
Act dependency relation analyzing tool. Find the head and dependency relation of the given word in sentence according these rules:
1. Choose the correct head number from (words_mapp
2. Choose the correct dependency relation from (REL LABELS_ )..R).
3. Do not explain or add extra text.
CLM
Sentence: {" ".join(words))
Word: [word)
Response as tuple which has exactly two elements: first element is the head number (as int), second element is the dependency relation (as str), e.g. (<head>, <relation>)
Response:
Act as part-of-speech (POS) tagging tool. Find the POS tag number the given word in the given sentence by choosing the correct option number from (POS_LABELS_STE
MLM
POS
Sentence: (sentence).
Word: (word).
Response: The POS tag number is {mask imask_str).
Act part-of-speech (POS) tagging tool. Find the POS tag of the given word the given sentence according to these rules:
1. Choose the correct option number from COOS_LABBLSSSTEE
2. Do not explain or add extra text. Only provide the option number.
CLM
Sentence:
(sentence).
Word: (word)
Response:
27

---

L Additional Experimental Result of TOA
We investigate the cross-model generalization ability of TOA on the DP task and POS task. The experimental results are presented in Table 15 and Table 16 The comprehensive experimental results further support our conclusions in $4.2
Table 15: Cross-Model Transferability of TOA on the DP Task. Entries represent UAS/LAS score
(unit: %). Best performance is bolded; scores exceeding the zero-shot baseline are underlined.
(a) CLM Results.
SreNTgt Q- Q-5
Q-7b
G-2b
G-9b
L-3b
L-8b
Zero-shot
6.17/0.43
8.66/2.57
7.36/2.41
8.24/2.97
7.38/1.15
8.79/3.38
TOA (Ours)
Q-1b5
65.44/50.48
45.96/27.32
45.32/26.83
38.97/18.77
49.02/31.11
45.24/26.
Q-7b
52.63/33.69
61.93/47.24
35.94/17.44
31.45/13.74
45.87/24.88 88
47.95/26.93
G-2b
40.00/21.61
32.47/14.85
62.53/46.13
50.27/31.54
48.77/30.54
43.36/24.62
G-9b
36.93/16.81
30.48/12.61
51.62/32.75
61.05/44.27
44.49/24.78
40.18/20.35
L-3b
48.02/29.54
43.82/23.83
50.22/31.61
40.82/19.83
62.95/47.40
57.46/40.48
L.-8h
48,44/30.35
47.03/27.83
46.87/28.05
35.08/16.23
58.53/42.86
60.99/45.77
(b) MLM Results.
SrelTgt
B-b
B-1
R-b
R-I
E-b
E-1
Zero-shot
0.47/0.00
0.60/0.00
0.86/0.00
1.65/0.00
4.33/0.00
3.82/0.00
TOA (Ours)
B-b
81.15/72.07
71.19/57.66 62
58.02/38 10
58. 18.14//7.90 98
71.11/55,99 90
71.02/55,2 28
B-1
67.38/52.89 80.95/71.15
43.77/20.54
54.31/32.38
59.05/37.95
62.52/41.88 8.77
R-b
65.70/51.30
52.50/36.55
78.35/67.51
63.90/45.66 6.64
68.49/54.21
69.64/55.15
R-1
64.58/49.80
58.91/43.45
64.11/48.20
78.92/68.23
59.01/42.04
55.65/37.11
E-b 72.15/58.83
62.84/46.75
62.52/45.14
56.66/36.49
80.33/70.93
76.84/65.45
E-1
68. 68.88/55.76
63.26/47.87
64.45/48.27
53.35/33.46
73.92/61.98
80.77/71.24
Table 16: Cross-Model Transferability of TOA on the POS Task. Entries represent accuracy (unit: %). Best performance bolded; scores exceeding the zero-shot baseline are underlined.
(a) CLM Results.
(b) MLM Results.
SrelTgt Q-1b5 Q-7b G-2b
G-9b
L-3b L-8b
Sre\Tet| B-b
B-1
R-b
R-1
E-b
Zero-shot
Zero-shot
3.88
22.76 7.77
59.77 22.05 39.00
0.44
0.66
0.23
0.65
0.67
0.60
TOA (Ours)
TOA (Ours)
Q-1b5
74.30
54.63
50.94
34.83
53.82
52.21
B-b
85.29
67.91
57.46
40.70
65.10
68.70
Q-7b
60.13
73.49
42.80
31.62
47.46
50.83
B-1
61.84
83.37
37.86
37.62
49.55
47.16
G-2b
53.03
41.79
73.78
57.45
51.52
47.92
R-b
58.31
52.13
83.35
60.63
54.16
58.20
G-9b
43.81
39.22
62.90
73.02
48.62
46.64
R-1
51.40
50.12
66.92
80.31
47.81
51.58
L-3b
56.96
50.48
57.18
45.09
73.20
68.35
E-b
66.56
53.13
54.36
37.03
84.31
74.53
L-8b
54,89
53.42
51.37
41.05
67.85
70.99
B-1
62.47
55.61
54.26
38.64
69.65
85.28
To conduct more comprehensive evaluation of TOA's performance, we compare it with few-shot prompting and Cross-Model Control (CMC) 44. The few-shot templates were constructed by adding the first five samples from the training set as demonstrations to the zero-shot templates ( CMC trains delta LM and integrates its output with that of LLM to enhance the latter's performance. Although this delta LM shares the same input-output format as the L.LM and does not process features (thus not qualifying adapter), has demonstrated considerable capability in
28

---

cross-model knowledge transfer. Therefore, we include it in the comparison. For the experimental setup, we adopt the widely used NER task and CLMs for evaluation. The results are presented in Table 17
Table Comparative experimental results on the NER Task. Entries represent Fl score (unit: %).
Best performance is bolded.
Sre\igt Q-1b5
Q-7b
G-2b
G-9b
L-3b
L-8b
Zero-shot
5.35
28.21 1.45 53.82 13.24
22.12
Few-shot
9.21
29.33
7.10
54.23
15.31
24.71
CMC
Q-1b5
8.99
21.09
15.90
44.78
14.96
19.99
G-2b
1.03
7.19
29.78
39.96
1.71
1.93
L-3b
13.79
22.13
18.42
42.18
18.51
23.51
TOA (Ours)
Q-1b5
53.81
30.99
21.08
9.58
29.24
26.94
G-2b
23.12
12.47
54.53
24.31
22.64
15.80
L-3b
27.48
21.79
24.79
14.05
54.51
45.24
To better analyze the reasons why LLMs underperform on fundamental NLP tasks, have showcased several cases below using the NER task as an example. For Case 1, we show the full prompt (including the template from KK. For other cases, display only the input sentence and model output; their prompts were generated by inserting the input after the template's 'Sentence:'.
Casel (Q-7b):
Prompt: Act as a named entity recognition tagging tool. Find all entities and their classes in sentence according to these rules: 1. Choose the correct named entity class from [PERSON, NORP, FACILITY, ORGANIZATIOM, Geo-Political Entity, LOCATION, PRODUCT, DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL, EVENT, WORK_DF OF [EK__F_AART_ LAW, LANGUAGE]. 2.Do not explain or add extra text. Sentence: Well I especially want to know ha how the two of you found out the news the day of the accident Response
tuples, and each tuple must have exactly elements: first element is the named entity text (as string), second element the named entity class (as string), e.g. (sentityl>, <class1>), (<entity2>, <class2>),
Response:
Sentence: Well I especially want to know ha how the two of you found the Lables: (two, CARDINAL)
the day of the accident ?.
Outputs: (I, PERSON), (two of you, GROUP), (the news, PRODUCT), (the day, DATE), (the accident, EVENT)
Case2 (L-8b))
From perspective as media I think it was precisely by going through those breaking handling such an emergency today effective announcements and notifications and instructions Lables: (today, DATE)
Sentence:
trials that could be calm in and give everyone most
Outputs: ('our perspective', 'EVENT') ('as media', 'FACILITY') (II² 'PERSDI") ('it', 'PERSON') ('thoae breaking trials', 'EVENT') ('today', 'DATE') ('ah', (PERSON')
Case3 (Q-7b):

---

Sentence: Well like Professor Zhou also received this ha through a mobile phone SMS Lables: (Zhou, PERSON)
Outputs: ('Professor Zhou', PERSON), ('news', EVENT), ('mobile phone SMS', PRODUCT)
Case4 (L-8b):
Sentence: think in comparison to China we should say that urbanization in foreign countries developed earlier and is more widespread Lables: (China, Geo-Political Entity)
Outputs: ('China', 'Geo-Political Entity'), ('"rr"))()rbuuiiiiea"" ('urbanization', 'EVENT"), ( ('foreign countries', 'LOCATION")
Analysis of the zero-shot and few-shot performance. 1) Using NER as an example, LMs may not fully grasp certain nuances of the task. For instance (Cases 2), they often incorrectly predict pronouns like 'I' 'you' named entities. They also exhibit counterintuitive erros, such as labeling phrases like "our perspective' as an 'EVENT' 2) LLMs sometimes generate outputs that seem reasonable but diverge from dataset labels (Cases 3 & 4). This discrepancy resembles a mismatch with human labeling preferences. For example, where the dataset annotates 'Zhou' as 'Person', LMs prefer the full span 'Professor Zhou', Similarly, LMs may classify 'mobile phone SMS' as 'PRODUCT', while the original labels do not. These represent comprehensible deviations from dataset conventions, rather than fundamental errors. 3) Few-shot prompting improves modestly zero-shot, but gains remain limited. TOA typically outperforms both (Table 17). Smaller models (Qwen-1.5BB Gemma-2B, LLaMA-3B) benefit more from few-shot examples than larger counterparts (Qwen-7B, Gemma-9B, LLaMA-8B), likely because the limited information from only five examples adds negligible value large models with extensive pre-trained knowledge.
Analysis of CMC Results: 1) CMC enhances an LM's performance by training a delta LM that concurrently reasons the same input text alongside the source LM and integrates their output logits. Furthermore, this delta LM can be applied to enhance other target LMs, demonstrating promising results in instruction-followiii LM (LLaMA2-7B) with identical tokenizers, and they only explored scenarios where the delta LM's tokenizer differed from the target LM's (e.g.. LLaMA2-70B. Mistral-7B) via token mapping based on metrics like edit distance. We observed that when the source LM (e.g.. Qwen- 1,5B) and its delta LM (e.g.. Tlaaa employ different tokenizers, the knowledge transfer from the delta LM to target LMs faces challenges, leading to suboptimal performance. This performance gap likely arises from mismatches in tokenizer mapping between the source/target LMs and the delta L.M, which can cause a distributional discrepancy in the tokens processed by the delta LM during both training and inference. Since the OLA of two LMs with different tokenizers remain similar, did not encounter this problem. 2) We found that incorporating CMC may introduce previously absent formatting issues, such as nested parentheses, extra spaces, or superfluous commas. These artifacts somewhat lowered its overall score.
tasks. However, they utilized a delta LM (TinyL.lama) and source
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims make in the abstract and introduction accurately reflect the contribution and scope the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We analyze the limitations in $42
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
31

---

Justification: In $3.2 nomenon, and in 3.4 we provide proofs that OLA encodes syntactic knowledge.
and $3.3 we formally demonstrate the existence of the OLAS phe-
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
we provide comprehensive description of our implementation details. We will open-source our code to ensure stable and straightforward reproducibility.
Justification: $ and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
32

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper uses publicly available open-source models and datasets, but does not provide open access to the code used for the experiments. We plan to release code upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides SoodeSubmissioooolieee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips http:ccccccccccgccccceecccccccee wwweeoomiieiioooooooooooo ierbbbmmssioooooolomcoo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: In and G we present comprehensive implementation details including model architectures, hyperparameters, evaluation metrics, and preprocessing procedures. The dataset configurations documented in SE with @CCttaaling the LMs deployed in our framework.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In 33.3 report averaged results across three independent trials with distinct random seeds for the quantitative analysis of OLAS.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computational resources required for the experiments in $F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrpp: wttpouurrppcccpppuupeeeee
Answer: [Yes]
Justification: Yes, the research conducted in the paper fully conforms to the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The research explores commonality in LLMs, with no discernible direct effects on societal contexts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
34

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Our paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the paper properly credits the creators original owners of assets and respects the license and terms use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve humans in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs are a primary research subject in this paper, with detailed descriptions of the methodology and implementation provided in and $
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy http:e///ourpsssccc//ooerreeeeee for what should or should not be described.
37


Reviews:
[{"review_id": "qHk3C6Zllv", "reviewer": "Reviewer_aRHQ", "summary": "This paper investigates the similarity of context aggregation patterns across different language models. The authors introduce Order-Level Attention (OLA) as a method to characterize these patterns. Specifically, the i-th order OLA is defined as the sum of the i-th order terms in the expansion of the attention rollout. The authors demonstrate that OLA exhibits notable similarities across models and is associated with syntactic knowledge. Building on these observations, they propose the Transferable OLA Adapter (TOA), claimed to be the first approach enabling cross-LM adapter transfer without additional tuning. By leveraging OLA as a unified syntactic representation across LMs, the adapters are trained with OLA for downstream tasks. Since OLA appears to be similar across models, the trained adapters can be directly applied to other LMs without further adaptation.", "questions": "1.\tCan TOA generalize to more challenging tasks that require deeper semantic understanding, such as generative tasks like question answering?\n2.\tI am puzzled as to why 7B-scale instruct models (e.g., Qwen2-7B, LLaMA 3.1-8B) perform poorly on these tasks, given that models of this size have achieved strong results on more difficult mathematical benchmarks. Could the authors provide further analysis or case studies to explain this behavior? How does TOA performance compare to few-shot prompting?\n3.\tWhat is the rationale for employing visual models or image similarity metrics to assess similarity, rather than simpler measures such as vector similarity?", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.\tThe proposed TOA enables transfer across different language models without requiring additional training.\n2.\tThe authors employ a range of qualitative and quantitative analyses to examine OLA similarity across LMs.\n3.\tThe experiments are conducted on models of varying sizes and architectures.\n\nWeaknesses\n1.\tThe evaluation is limited to NLP tasks involving syntactic and simple semantic understanding; more complex tasks are not considered.\n2.\tThere is no comparison against related methods, such as Cross-model Control: Improving Multiple Large Language Models in One-time Training.\n3.\tThe approach requires modifying the adapter structure and training separate adapters for different tasks, limiting its universality.\n4.\tThe cross-model generalization capability appears limited; transferring to other models often results in performance degradation exceeding 10 percentage points.\n5.\tSome descriptions lack clarity. For example, Sec 3.2 refers to “zeroing maxima for first-/second-order OLA,” while Appendix B mentions “setting outlier values in the OLA to zero,” creating inconsistencies. Additionally, Section G.1 contains an incomplete phrase: “The diagonal features of …”", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful suggestions.\n\n# 1. TOA’s Limitation in Applicability (w1, q1):\n\nAs stated in Sec 4.2, while OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. **It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks** (e.g., text generation). \n\n**Developing cross-model unified semantic representations is a key direction for our future work.** We're integrating the technique from [1] into OLA by representing each word's semantics as its relevance to a set of Anchor Words. This approach holds promise to enhance OLA's semantic capacity while preserving its cross-model consistency.\n\nMoreover, **a significant portion of our contribution is reflected in the discovery of OLAS**, which reveals universal attention patterns across diverse LMs, a finding acknowledged by other reviewers.\n\n# 2. Supplemental Comparative Methods and Further Performance Analysis for Zero-shot (w2, q2)\n\nIn the subsequent version, **we will supplement comparative methods including Cross-Model Control (CMC) [2] and few-shot prompting.** Below we present the partial results for the NER task (Sec. 4.2). The few-shot templates were constructed by adding the first five samples from the training set as demonstrations to the zero-shot templates (Sec. K).\n\n**Table 1: Comparative experimental results (F1 score). Best performance bolded.**\n|method|src\\tgt|Q-1b5|Q-7b|G-2b|G-9b|L-3b|L-8b|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|few shot|-|9.21|29.33|7.10|**54.23**|15.31|24.71|\n|CMC|Q-1b5|8.99|21.09|15.90|44.78|14.96|19.99|\n||G-2b|1.03|7.19|29.78|39.96|1.71|1.93|\n||L-3b|13.79|22.13|18.42|42.18|18.51|23.51|\n|TOA|Q-1b5|**53.81**|**30.99**|21.08|9.58|29.24|26.94|\n||G-2b|23.12|12.47|**54.53**|24.31|22.64|15.80|\n||L-3b|27.48|21.79|24.79|14.05|**54.51**|**45.24**|\n\n\n**Analysis of the zero-shot and few-shot performance:**\n1. Using NER as an example, LMs may not fully grasp certain nuances of the task. For instance (Cases 1 & 2), they often incorrectly predict pronouns like 'I' or 'you' as named entities. They also exhibit counterintuitive errors, such as labeling phrases like 'our perspective' as an 'EVENT'.\n2. LLMs sometimes generate outputs that seem reasonable but diverge from dataset labels (Cases 3 & 4). This discrepancy resembles a mismatch with human labeling preferences. For example, where the dataset annotates 'Zhou' as 'Person', LMs prefer the full span 'Professor Zhou'. Similarly, LMs may classify 'mobile phone SMS' as 'PRODUCT', while the original labels do not. These represent comprehensible deviations from dataset conventions, rather than fundamental errors.\n3. Few-shot prompting improves modestly over zero-shot, but gains remain limited. TOA typically outperforms both (Table 1). Smaller models (Qwen-1.5B, Gemma-2B, LLaMA-3B) benefit more from few-shot examples than larger counterparts (Qwen-7B, Gemma-9B, LLaMA-8B), likely because the limited information from only five examples adds negligible value to large models with extensive pre-trained knowledge.\n\n**case study:**\n\nFor Case 1, we show the full prompt (including the template from Section K). For other cases, we display only the input sentence and model output; their prompts were generated by inserting the input after the template’s 'Sentence:'.\n\nCase1（Q-7b）:\n> **Prompt**:\n> Act as a named entity recognition tagging tool. Find all entities and their classes in a sentence according to these rules:\n> 1.Choose the correct named entity class from \\[PERSON, NORP, FACILITY, ORGANIZATION, Geo-Political Entity, LOCATION, PRODUCT, DATE, TIME, PERCENT, MONEY, QUANTITY, ORDINAL, CARDINAL, EVENT, WORK_OF_ART, LAW, LANGUAGE\\].\n> 2.Do not explain or add extra text.\n> Sentence: Well , I especially want to know , ha , how the two of you found out the news on the day of the accident ?\n> Response as tuples, and each tuple must have exactly two elements: first element is the named entity text (as a string), second element is the named entity class (as a string), e.g. (\\<entity1\\>, \\<class1\\>), (\\<entity2\\>, \\<class2\\>), ...\n> Response:\n> \n> **Sentence**: Well , I especially want to know , ha , how the two of you found out the news on the day of the accident ?\n> \n> **Lables**: (two, CARDINAL)\n> \n> **Outputs**: (I, PERSON), (two of you, GROUP), (the news, PRODUCT), (the day, DATE), (the accident, EVENT)\n\nCase2（L-8b）:\n> **Sentence**: From our perspective as media , I think it was precisely by going through those breaking - in , trials that we could be so calm in handling such an emergency today , ah , and give everyone the most effective announcements and notifications , and instructions .\n> \n> **Lables**: (today, DATE)\n> \n> **Outputs**: ('our perspective', 'EVENT') ('as media', 'FACILITY') ('I', 'PERSON') ('it', 'PERSON') ('those breaking - in , trials', 'EVENT') ('today', 'DATE') ('ah', 'PERSON')\n\nCase3（Q-7b）:\n> **Sentence**: Well , like Professor Zhou , I also received this news , ha , through a mobile phone SMS .\n> \n> **Lables**: (Zhou, PERSON)\n> \n> **Outputs**: ('Professor Zhou', PERSON), ('news', EVENT), ('mobile phone SMS', PRODUCT)\n\nCase4（L-8b）:\n> **Sentence**: I think , in comparison to China , we should say that urbanization in foreign countries developed earlier and is more widespread .\n> \n> **Lables**: (China, Geo-Political Entity)\n> \n> **Outputs**: ('China', 'Geo-Political Entity'),('urbanization', 'EVENT'),('foreign countries', 'LOCATION')\n\n\n**Analysis of CMC Results:**\n\n1. CMC enhances an LM's performance by training a delta LM that concurrently reasons on the same input text alongside the source LM and integrates their output logits. Furthermore, this delta LM can be applied to enhance other target LMs, demonstrating promising results in instruction-following tasks. However, they utilized a delta LM (TinyLlama) and source LM (LLaMA2-7B) with identical tokenizers, and they only explored scenarios where the delta LM's tokenizer differed from the target LM's (e.g., LLaMA2-70B, Mistral-7B) via token mapping based on metrics like edit distance. We observed that when the source LM (e.g., Qwen-1.5B) and its delta LM (e.g., TinyLlama) employ different tokenizers, the knowledge transfer from the delta LM to target LMs faces challenges, leading to suboptimal performance. This performance gap likely arises from mismatches in tokenizer mapping between the source/target LMs and the delta LM, which can cause a distributional discrepancy in the tokens processed by the delta LM during both training and inference. Since the OLA of two LMs with different tokenizers remain similar, we did not encounter this problem.\n2. We found that incorporating CMC may introduce previously absent formatting issues, such as nested parentheses, extra spaces, or superfluous commas. These artifacts somewhat lowered its overall score.\n\n# 3. TOA require training independent adapters for downstream tasks. (w3)\n\nYes, but learning separate adapters for different downstream tasks is a common practice in adapter-based methods [3,4]. Moreover, **once TOA is trained, it can be directly used by other LMs without retraining, making it efficient.**\n\n# 4. Performance Degradation After Transferring TOA. (w4)\n\nTraining-free cross-model transfer is profoundly challenging due to the absence of target model information. This inevitably leads to performance degradation after TOA transfer; nevertheless, **its performance remains superior to the target model's zero-shot capability, indicating successful knowledge transfer to some extent.** Notably, this limitation also persists in other transfer methods [2,5].\n\n# 5. Unclear Expression (w5)\n\n**Thank you for the feedback. We will clarify this expression in subsequent revisions.**\n\nBy 'outliers,' we refer to points within a row that are significantly larger than the others (Line 583). In the example in Sec. 3.2, only the maximum value in each row is considered an outlier. For ease of understanding, we explicitly stated to 'set the maximum value to zero'.\nIn Sec. B, multiple outliers may exist per row. Therefore, we used the term 'outliers'.\n\nWe will remove the preposition 'of' from the phrase 'The diagonal features of…'.\n\n# 6. Metrics for Measuring OLA Similarity (q3)\n\n**We used a 2D similarity metric (visual model, SSIM) instead of 1D vector metrics because different LMs employ distinct tokenizers.** Even when resized to identical dimensions, corresponding pixels do not consistently represent the same key-value pairs (Figure 3). Thus, a 2D structural indicator is more appropriate than pixel-to-pixel metrics.\n\n**For comprehensive evaluation, we employed simpler vector similarity metrics in our image similarity retrieval experiment (Section 3.2.2), including the Cosine Similarity and Mean Squared Error (MSE).** It can be seen from the results in Tables 2 and 3 below that all tested metrics reveal the cross-model similarity of OLA.\n\nA more detailed discussion can be found in R4-#2.\n\n**Table 2: First-order OLA cross-model similarity using Cosine Similarity (Experiment settings match main text Table 2)**\n\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|83.40/88.70|94.20/96.80|\n|G-2b|78.70/85.80|-|97.40/98.60|\n|L-3b|90.60/93.90|96.70/97.80|-|\n\n**Table 3: Results using the MSE metric.**\n\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|84.20/89.20|94.50/97.10|\n|G-2b|78.40/85.10|-|98.00/98.80|\n|L-3b|88.40/92.50|97.00/98.10|-|\n\n# References:\n\n[1] Relative representations enable zero-shot latent space communication.\n    \n[2] Cross-model Control: Improving Multiple Large Language Models in One-time Training.\n    \n[3] Scalable Language Model with Generalized Continual Learning.\n\n[4] Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters.\n\n[5] Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models."}, {"review_id": "fKCE84X4Vq", "reviewer": "Reviewer_LYHK", "summary": "This paper investigates whether different large language models (LLMs) exhibit common patterns in how they aggregate contextual information through attention mechanisms. The authors propose a novel attention decomposition method called Order-Level Attention (OLA), which breaks down cumulative attention rollout into multiple “orders” (e.g., 0th-order: residual path only, 1st-order: single-layer aggregation, and so on). They observe that OLA representations of the same order across different models show a high degree of similarity, a phenomenon they term Order-Level Attention Similarity (OLAS). Building on this insight, the authors introduce the Transferable OLA Adapter (TOA), a training-free adapter mechanism that enables cross-model transfer. Experiments conducted on 12 different models and multiple downstream tasks—including relation extraction, part-of-speech tagging, and named entity recognition—demonstrate the effectiveness of their approach.", "questions": "- The current experiments do not clearly establish the advantage boundaries of TOA. Could you add comparisons with other representation transfer methods, such as Representation Editing, Middle-layer Alignment, or Hidden-State Transfer? Also, how does TOA perform relative to trained adapters like LoRA , or Prefix Tuning? Is there a trade-off between accuracy and training cost?\n- Have you evaluated OLA or TOA on models with significantly different architectures, such as encoder-only versus decoder-only Transformers? How well does the method generalize across these divergent settings?\n- How does OLA behave in long-sequence settings? In such cases, what changes (if any) occur in TOA's training complexity, inference latency, and accuracy? Does high-order OLA risk overestimating distant token contributions in longer contexts?\n- The current experiments focus exclusively on tasks suitable for encoder-only models (RE, NER, DP, POS). Can you expand the evaluation to include tasks like text continuation or summarization, which are more natural for causal language models (CLMs), to validate OLA and TOA in that setting?\n- The paper reports the performance differences across different OLA orders (first-order, second-order, and third-order) in Table 2 (retrieval task) and Table 3 (dependency parsing task), with notable variations observed across both CLM and MLM model families. However, the authors do not provide sufficient analysis or discussion to explain these differences.  For example:\n  - In Table 3(a), the Qwen2-1.5B model achieves a UAS/LAS of 63.58/48.24 using first-order OLA on the dependency parsing task, which is significantly higher than the 55.19/36.82 performance obtained with third-order OLA.\n  - Moreover, in the ablation study on data augmentation in Appendix Table 9, the cross-model transfer results vary across different OLA orders, showing inconsistent trends across different models.\n  - This raises the question of whether the choice of OLA order has a complex, model- and task-dependent impact on performance.", "limitations": "OLA performs an order-wise decomposition of attention rollout. However, attention rollout itself has been primarily applied in encoder-based models such as BERT, where it serves to quantify input token importance across layers. Its applicability to causal language models (CLMs) remains unclear and is not addressed in the paper. In addition, although the paper presents a wide range of experiments, many of them remain relatively superficial. The baselines used are fairly basic, and several important issues in the experimental data are not thoroughly examined or discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- OLA introduces a novel perspective for unified attention representation by decomposing attention rollout into order levels from a path-based view, facilitating interpretability and structural alignment across different language models.\nThe authors measure attention structure similarity using image-based methods and support their OLAS observation with thorough theoretical analysis and empirical validation.\n- TOA leverages attention representation for transfer and offers a simple, training-free mechanism that outperforms baseline models in multiple tasks.\n- The paper covers both theoretical formulation and practical application, showing strong breadth and depth.\n\n**Weaknesses**\n- OLA bears substantial similarity to existing methods like Attention Rollout or path tracing (e.g., Abnar & Zuidema, 2020). Attention Rollout is already a widely used method for quantifying input importance at any layer of encoder-based models like BERT. Its applicability to causal language models (CLMs) remains under-discussed. Furthermore, the proposed order-wise decomposition may amplify the overestimation of distant token importance in long sequences—an issue not addressed in the paper.\n- In the experiments, the authors did not adequately compare against other interpretability methods (e.g., LogitLens, ACDC) or transfer approaches (e.g., LoRA, P-Tuning). The main baseline used is zero-shot, which is a relatively weak comparison. No comparisons are made to representation transfer methods such as hidden-state alignment or DSKD, leaving the true advantage of TOA unclear.\n- The claim of being the “first” to achieve training-free adapter transfer is overstated, as similar ideas have been explored in prior works.\nTheoretical properties, generalization boundaries, and computational efficiency of high-order OLA are not clearly discussed.", "was_revised": true, "final_justification": "I have no further concerns and believe that 4 is an appropriate score for this paper.", "rebuttal": "Thank you for your constructive suggestions and positive feedback. Your insights will help us improve the work, and comments such as 'a novel perspective' and 'strong breadth and depth' are a great encouragement.\n\n# 1. The Innovativeness of OLA and its Relation to Interpretability Methods (w1, w2)\n\nAlthough OLA derives from Attention Rollout via order-level decomposition (Sec. 3.1), it differs from Attention Rollout and path tracing, which focus on quantifying input importance. Instead, our focus is on uncovering common attention patterns across different LMs, a novel aspect unaddressed by these methods. As shown in Secs. 3.2 and 3.3, identifying these cross-model commonalities is difficult with Attention Rollout. **In contrast, OLA, achieved through this straightforward decomposition, reveals significant similarities, highlighting its unique value from a novel perspective.**\n\nRegarding the relationship to interpretability methods: OLA originates from Attention Rollout, an interpretability method, and thus retains the ability to explain model behavior. However, this work is not aimed at enhancing model interpretability per se. **Our primary goal is to discover latent commonalities in attention patterns across LMs. Consequently, we did not include interpretability methods like LogitLens or ACDC (which analyze individual model behavior and don’t study cross-model commonalities) in our comparative analysis.**\n\n# 2. Is there a trade-off between training cost and performance for TOA? (w1)\n\nYes, **there is a trade-off between TOA's training cost and performance**. As shown in Table 1 below, performance increases with more training steps, peaking at ~149,004 steps (taking ~7 hours). Beyond this, gains slow and performance eventually plateaus.\n\n**Table 1:  TOA's F1 scores on NER with increasing training steps. (the source and target LM are Qwen2-1.5B)**\n\n|steps|49,668|99,336|149,004|198,672|248,340|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|F1|38.66|45.19|53.81|53.75|52.97|\n\n# 3. Applicability of OLA to CLMs and Long Sequences (w1,q3)\n\nAs shown in Section 3, OLA reveals significant commonalities across both CLMs and MLMs. As demonstrated in Section 4, TOA achieves cross-model knowledge transfer on both CLMs and MLMs. **This proves OLA's applicability to both CLMs and MLMs.**\n\nFor long sequences, we conducted two experiments:\n\n1. **We validated OLA's cross-model similarity on long sequences.** We sampled 1000 sentences of length 256 from CoNLL-2012 and repeated the experiment corresponding to Table 2 in the main text. Results are shown in Table 2 below. Retrieval accuracy improves with longer sentences (vs. main text Table 2), because increased sentence length enhances structural diversity, making OLA patterns between different sentences more distinct, thereby boosting retrieval success.\n2. **We validated TOA's transfer performance, training time, and inference speed on long sequences.** For the NER task (Section 4.2), we created length-256 sequences by randomly concatenating original sentences. Results are shown in Table 3 below. TOA's F1 score shows comparable performance vs. main text Table 5, indicating its suitability for long sequences. Regarding training complexity and inference latency, Table 4 shows both increase with sequence length. This issue can potentially be addressed in the future through architectural optimization of TOA.\n\nRegarding whether OLA overestimates the importance of distant tokens in long sequences: As observed in Fig. 3(b) and (d), OLA tends to overestimate the importance of distant tokens, particularly the first token, due to the attention sink phenomenon (Section 3.1). However, **this does not appear to affect OLA's core commonality findings or TOA's performance, since OLA patterns still exhibit high similarity across different models - including this shared tendency to overestimate distant token importance.**\n\n**Table 2: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity using long-sequence.**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|92.90/95.50|97.00/98.60|\n|G-2b|94.00/96.20|-|91.10/93.90|\n|L-3b|96.70/98.00|91.60/93.60|-|\n\n**Table 3: Cross-LM transferability of TOA on the NER task using long-sequence.**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|55.84|19.18|25.37|\n|G-2b|27.40|55.81|19.31|\n|L-3b|36.26|31.13|52.11|\n\n**Table 4: TOA's training time and inference latency across sequence lengths on the NER task.**\n|length|training time (s/step)|inference latency (s)|\n|:-:|:-:|:-:|\n|50|0.17|0.01|\n|256|3.74|0.76|\n\n\n# 4. Supplemental Comparative Methods (w2,q1): \n\nOur approach focuses on training-free adapter transfer, operating without accessing the target model during transfer. **While the knowledge distillation methods you mentioned, such as hidden-state alignment [1] or DSKD [2], offer valuable insights (and will be discussed in future version), they require training student models, falling outside our training-free scope. Consequently, they were excluded from our baselines.**\n\nRegarding LoRA and P-tuning methods:\n1. P-tuning: We adapted this as a baseline by directly transferring soft prompts trained on the source LM to target LMs. When encountering dimensional mismatches in word embeddings, we performed interpolation to resize prompts to the target LM's dimensions. \n2. LoRA: It cannot be directly applied in cross-LM transfer due to architectural incompatibilities (e.g., differing layer counts and hidden dimensions between models).\n\nP-tuning results (Table 5 below) show that transferred soft prompts achieved near zero-shot performance on Qwen models but failed completely on Gemma and LLaMA. This indicates that:\n1. This poor performance, approaching zero-shot levels or failing entirely, confirms P-tuning's unsuitability for cross-model transfer.\n2. This discrepancy may stem from tokenizer differences: Qwen's tokenizer lacks a BOS token, making it robust to noisy soft prompts occupying the first token position. In contrast, Gemma and LLaMA require a BOS token as the first token to function correctly.\n\n**Table 5: Cross-Model Transferability of P-tuning.**\n|src\\tgt|Q-1b5|Q-7b|G-2b|G-9b|L-3b|L-8b|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Q-1b5|74.36|28.84|0.00|0.00|0.00|0.00|\n|G-2b|6.15|27.86|81.39|0.00|0.00|0.00|\n|L-3b|5.32|30.20|0.00|0.00|53.00|0.00|\n\n[1] IMPROVING LANGUAGE MODEL DISTILLATION THROUGH HIDDEN STATE MATCHING.\n[2] Dual-Space Knowledge Distillation for Large Language Models.\n\n# 5. Overstatement of 'first' (w3)\n\nThank you for noting this. We are the first to identify the OLA similarity phenomenon. Regarding cross-model transfer, a limited number of studies have investigated this. **We will refine our expression.**\n\n# 6. Discussion on High-Order OLA (w3)\n\n**In the main text, we primarily analyze 1st to 3rd-order OLA because they have already demonstrated significant cross-model similarity and possess lower computational complexity.** The computational complexity of higher-order OLA is higher: for k-th order OLA, it is necessary to calculate $\\binom{N}{k}$ paths worth of information aggregation matrices, where N denotes the number of LM layers. Therefore, we currently focus less on higher-order OLA.\n\nTable 6 below presents quantitative results for fourth-order OLA similarity (settings match main text Table 2). We discover that fourth-order OLA still reveals significant commonalities. In future work, we will explore higher-order OLA to formalize theoretical generalization boundaries.\n\n**Table 6: Fourth-order OLA cross-model similarity retrieval scores. Entries: Hits@1 / Hits@5 (%)**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|48.50/69.60|58.80/80.30|\n|G-2b|45.90/66.00|-|89.30/94.00|\n|L-3b|53.70/76.80|88.70/93.30|-|\n\n# 7. OLA Similarity Analysis between CLM and MLM (q2)\n\nTable 7 below shows first-order OLA similarity between encoder-only (MLM) and decoder-only (CLM) Transformers, mirroring main text Table 2's settings. Since CLM OLA matrices are lower triangular but MLM ones are not, we masked the MLM matrices to lower triangular form for SSIM calculation.\n\nAs shown, **OLA similarity between CLMs and MLMs is lower than within CLM or MLM pairs, but still present in some degree.** This indicates shared contextual information aggregation mechanisms across these architectures.\n\n**Table 7: Retrieval-based quantitative evaluation of first-order OLA between MLMs and CLMs.**\n\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|B-b|12.40/26.90|12.40/24.30|11.30/26.90|\n|R-b|13.30/36.40|16.10/37.20|17.70/34.70|\n|E-b|6.60/20.20|3.80/13.00|6.70/16.80|\n\n# 8. TOA’s Limitation in Applicability (q4):\n\nAs stated in Sec 4.2, while OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation). **Developing cross-model unified semantic representations is a key direction for our future work.**\n\n# 9. Selection of the OLA Order (q5)\n\nThe OLA similarity does vary to some extent for different model pairs and orders, but the overall trend shows high similarity from orders 1 to 3 across most pairs (line 217), proving that OLA similarity is a widespread phenomenon. \n\nTo investigate how OLA order selection would affect TOA's performance, we tested TOA's performance under different orders for the NER Task. Results in Table 8 below show no substantial performance differences across orders 1 to 3. Therefore, **given that order-1 OLA has the lowest computational complexity, it is generally sufficient in most scenarios.**\n\n**Table 8: TOA Transfer Performance Under Different Orders. Source LM is Qwen2-1b5.**\n|Orders\\Tgt|Q-1b5|Q-7b|G-2b|G-9b|L-3b|L-8b|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|1st|54.12|32.69|22.31|9.89|27.65|27.23|\n|2nd|52.79|31.23|20.88|9.67|24.12|20.92|\n|3rd|51.52|27.20|20.35|9.48|13.59|11.55|"}, {"review_id": "JQiWwXpcVS", "reviewer": "Reviewer_uyDY", "summary": "From the perspective of information flow, this paper decomposes the information flow formed by multi‑head attention (MHA) and residual connections, proposing Order‑Level Attention (OLA). Through a series of qualitative and quantitative experiments, it demonstrates the validity of OLA, reveals its potential as an invariant shared across different large language models (LLMs), and then introduces Transferable OLA Adapter (TOA). Experiments on four basic NLP tasks further verify TOA’s effectiveness and, by extension, the soundness of OLA.", "questions": "1. Have you tried applying TOA to transfer for text‑generation tasks?\n\n2. Why are there no experiments for fourth‑order OLA and above? Is it because higher‑order features are heavily affected by the attention‑sink phenomenon?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "> Strengths\n\n- The paper offers a novel viewpoint by decomposing the information flow generated by MHA and residual connections, uncovering a common contextual aggregation pattern among LLMs and proposing OLA.\n\n- Experiments are thorough and well designed. Qualitative studies first reveal the similarity of same‑order OLAs (OLAS) across different LLMs for the same text; this is then substantiated through carefully crafted image‑classification and image‑similarity‑retrieval experiments, as well as quantitative analyses of “Relation between OLA and Syntactic Knowledge.”\n\n- The proposed methods are simple yet effective; on quantitative tasks based on image classification and image‑similarity retrieval they substantially outperform a range of baselines.\n\n> Weaknesses\n\n- The TOA method is restricted to syntax‑related tasks and cannot be directly applied to text‑generation or other semantics‑heavy tasks.", "was_revised": true, "final_justification": "The authors have addressed my concern regarding whether the TOA method can be applied to text generation. I have decided to keep the score unchanged.", "rebuttal": "Thank you for your valuable feedback and kind recognition of our work. Your comments such as 'The paper offers a novel viewpoint,' 'Experiments are thorough and well designed,' and 'The proposed methods are simple yet effective' have been extremely encouraging to us.\n\n# 1. TOA’s Limitation in Applicability. (w1,q1)\n\nWhile OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation).\n\nWe previously considered using OLA for text generation tasks and conducted exploratory experiments. When attempting to reconstruct original sentences using OLA, we observed unsatisfactory results. Consequently, we simplified the task by employing OLA for part-of-speech (POS) tagging of the original sentences, which yielded well-received outcomes. Since POS correlates strongly with syntactic structures, this realization directed our research toward syntax-oriented exploration. **Although OLA currently primarily encapsulates syntactic knowledge, we will continue pursuing a unified semantic feature representation across diverse LLMs.**\n\n# 2. Discussion on Higher-Order OLA (q2)\n\nIn the main text, we primarily analyze 1st to 3rd-order OLA because they have already demonstrated significant cross-model similarity and possess lower computational complexity. The computational complexity of higher-order OLA is higher: for k-th order OLA, it is necessary to calculate $\\binom{N}{k}$ paths worth of information aggregation matrices, where N denotes the number of LM layers. Therefore, we currently focus less on higher-order OLA.\n\nIn Table 1 below, we present the quantitative analysis results for fourth-order OLA cross-model similarity (settings mirror main-text Table 2). **We discover that fourth-order OLA still reveals significant commonalities. In future work, we will explore higher-order OLA to formalize theoretical generalization boundaries.**\n\n**Table 1: Retrieval-based quantitative evaluation of fourth-order OLA cross-model similarity. Entries indicate evaluation metrics Hits@1 / Hits@5 (unit: %)**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|48.50/69.60|58.80/80.30|\n|G-2b|45.90/66.00|-|89.30/94.00|\n|L-3b|53.70/76.80|88.70/93.30|-|"}, {"review_id": "GhNNGHdK3K", "reviewer": "Reviewer_dT4c", "summary": "This paper investigates context aggregation patterns across LMs by using Order-Level Attention (OLA). The authors find that OLA patterns are consistent across models and align especially well with syntactic patterns. Leveraging this, they introduce a Transferable OLA Adapter (TOA) that shows some transfer abilities to unseen models without further training.", "questions": "- Do tokenizers need to be aligned for this to work? It appears that subtokens were aggregated in Figure 3. How was the aggregation performed?\n- In Section 3.3, four LMs were using during training as the source LMs. Why did you decide to take several and how do the results depend on training on less source models, i.e. only one? \n- “This suggests that OLA primarily encodes syntactic structures rather than semantic knowledge.” (l. 326) If I understand correctly, OLA maps do not directly encode representations but relationships of representations. This should be discussed more clearly. What are the limitations of this focus on relatedness over representations?", "limitations": "Some specific limitations are discussed in Section 4.2 regarding the experimental results of the TOA experiments, but no general methodological limitations are discussed as part of the main paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The authors address an interesting and relevant problem of (i) analyzing higher-order patterns in LMs and (ii) investigate good agreement of resulting patterns across different models.\n- The methods are clearly defined and appear to be reproducible with reasonable efforts. Ablations, i.e. perturbation of the model parameters, further help to  support the results\n- Extensive experiments on a variety of language models (CLMs, MLMs) and various tasks.\n\n**Weaknesses**\n- The main theoretical contribution appears to be a reformulation of attention rollout from a multi-order perspective.\n- “We employed an image classification model as a proxy for human evaluation to objectively assess the similarity of OLA maps“  (l.182-183). I do not see how this is related to human evaluation. I was surprised to not see a more direct measure of relatedness, i.e. using correlation or distance measures between attention matrices (also in relation to the methods of representational alignment/representational similarity analysis). The image-based retrieval (Section 3.3.2 is a good step in this direction).\n- This evaluation makes it difficult to infer the main result that “LMs exhibit significant similarity”, as it is accessed by a classification into the same group. It does not directly measure that the patterns across LMs are comparable or similar (as suggested in Figure 3). It could further be that different source models exhibit different patterns and the classification step merely matches it to one of these.\n- The practical utility of the transfer setting (TOA) is not clearly demonstrated. While the authors show that training on the source model OLAs allows transfer to other LMs with improved performance over zero-shot, overall, performance is not competitive, e.g. for the NER task on CoNLL 2012 current state-of-the-art is at 80-84% with best scores on TOA is at around 69%. \n- Missing related works on higher-order explanations that encode structural information such as messages between sequence elements (see relevant related works below).\n- Lack of clear discussion of methodological limitations (see also questions below)\n\n\n**Relevant related works:**\n\nHigher-order explanations for DNNs:\n\n- [Tsa18] M. Tsang, D. Cheng and Y. Liu, \"Detecting statistical interactions from neural network weights\", Proc. Int. Conf. Learn. Representations, 2018.\n- [Ebe20] Eberle, O., Büttner, J., Kräutli, F., Müller, K. R., Valleriani, M., & Montavon, G. (2020). “Building and interpreting deep similarity models”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3), 1149-1161.\n- [Jan20] J. D. Janizek, P. Sturmfels and S. Lee. \"Explaining explanations: Axiomatic feature interactions for deep networks\", CoRR, 2020.\n- [Sch21] Schnake, T., Eberle, O., Lederer, J., Nakajima, S., Schütt, K. T., Müller, K. R., & Montavon, G. (2021). Higher-order explanations of graph neural networks via relevant walks. IEEE transactions on pattern analysis and machine intelligence, 44(11), 7581-7596.\n- [Vas24] A Vasileiou and O Eberle. “Explaining Text Similarity in Transformer Models”, NAACL 2024.\n- [Fum24] Fumagalli, F., Muschalik, M., Kolpaczki, P., Hüllermeier, E., & Hammer, B. (2024). KernelSHAP-IQ: Weighted Least-Square Optimization for Shapley Interactions. ICML 24.", "was_revised": true, "final_justification": "This was a good rebuttal and I appreciated the clarifications. The additional evaluations using Cosine Similarity and MSE are appreciated and confirm the reported benefits of their approach. The authors commit to completing the related work section on higher-order explanations to better contextualize their contribution in the broader research community. I would also suggest to clarify the focus on syntactic structural information, as this is crucial to consider for specific applications.\n\nI have maintained my score of (4). I think overall this is a useful paper, but it may be a borderline accept as the originality and significance is limited. Overall, the paper appears ready for publication; if at NeurIPS remains to be seen.", "rebuttal": "Thank you for your constructive suggestions and positive feedback. Your suggestions will help us refine this work, and comments such as 'addresses an interesting and relevant problem' and 'has clearly defined methods that appear reproducible with reasonable effort' serve as encouragement to us.\n\n# 1. OLA is a reformulation of attention rollout from a multi-order perspective.(w1)\n\nAlthough OLA is derived via order-level decomposition from Attention Rollout (Sec. 3.1), it differs from Attention Rollout and path tracing, which focus on quantifying input importance. Our focus, instead, is on uncovering common attention patterns across different LMs, an aspect not previously explored by these methods. **As demonstrated in Secs. 3.2 and 3.3, identifying cross-model attention commonalities proves difficult using Attention Rollout. In contrast, OLA, achieved through this straightforward decomposition, reveals significant similarities. This highlights the unique contribution of OLA from a novel perspective.**\n\n# 2. Evaluation of OLA Similarity. (w2, w3)\n\n**Regarding the question of whether the OLA generated by different LMs are comparable or similar, the answer is similar.** We visualized a large number of samples and observed that the OLA generated by different models for the same text do exhibit noticeable visual similarity (as shown in Fig. 3).\n\nFor quantitative evaluation, we employed two methods: **one leveraging a visual model to assess the similarity of high-level features between different OLA**, and **the other evaluating low-level similarity using Image Similarity Retrieval** (Sec. 3.3).\n\nThe rationale for introducing the visual model as a proxy for human evaluation is that it can assess the similarity of high-level features in images.\n\nTo more directly demonstrate the cross-LM similarity of OLA, we conducted the following experiments:\n\n1. We replaced the SSIM metric used in the OLA similarity evaluation experiment based on image retrieval (Sec. 3.3.2) with Cosine Similarity and Mean Squared Error (MSE). As shown in Tables 1 and 2 below, **using these simpler and more direct metrics, OLA still exhibit significant similarity.**\n2. Using the data from Sec. 3.3.2, we calculated the similarity metrics (SSIM, Cosine Similarity, MSE) between OLA generated by the source LM and target LM for the same text input. We then computed the mean value across all texts. In simpler terms: For each of the 1000 texts, we generated an OLA pair (one from the source LM, one from the target LM), and we calculated the mean similarity metric across all 1000 OLA pairs. As shown in Table 4 below, **OLA generated for the same text by different LMs exhibit high similarity scores.**\n\n**Table 1: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity. Settings identical to Table 2 in the main text. The similarity metric has been replaced with Cosine Similarity metric.**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|83.40/88.70|94.20/96.80|\n|G-2b|78.70/85.80|-|97.40/98.60|\n|L-3b|90.60/93.90|96.70/97.80|-|\n\n**Table 2: Retrieval-based quantitative evaluation of first-order OLA cross-model similarity. The similarity metric is MSE.**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|84.20/89.20|94.50/97.10|\n|G-2b|78.40/85.10|-|98.00/98.80|\n|L-3b|88.40/92.50|97.00/98.10|-|\n\n**Table 3: Mean similarity of OLA for the same text under different LM combinations. Values in the table represent (SSIM/Cosine Similarity/MSE). Higher SSIM and Cosine Similarity scores indicate greater similarity (maximum = 1). Lower MSE values indicate greater similarity (minimum = 0).**\n|Src\\Tgt|Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Q-1b5|-|0.8029/0.9605/0.00016|0.8315/0.9663/0.00013|\n|G-2b|0.8029/0.9605/0.00016|-|0.8713/0.9754/0.00010|\n|L-3b|0.8315/0.9663/0.00013|0.8713/0.9754/0.00010|-|\n\n# 3. The practical utility of TOA is not clearly demonstrated. (w4)\n\nTOA's performance has not yet reached the level of current state-of-the-art (SOTA) methods. As such, its practicality does have certain limitations. Nevertheless, other cross-model knowledge transfer approaches, such as the method in [1], face analogous challenges. **We maintain that this work represents a promising beginning and lays groundwork for future research.**\n\n[1] Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models.\n\n# 4. Related work is incomplete. (w5)\n\nThank you for your feedback. **We will supplement these references in the subsequent version and include a discussion on them.**\n\n# 5. What are the limitations of this focus on relatedness over representations? (w6,q3)\n\nOLA fundamentally captures relationships between tokens, rather than token representations themselves. Our analysis (Sec. 3.4) demonstrates that OLA contains syntactic knowledge. We therefore utilize it as a specialized syntactic representation serving as input for TOA.  Leveraging OLA's cross-model similarity, TOA enables direct cross-model transfer. The limitation, however, is that compared to features generated by traditional learnable encoders, OLA suffers from information loss. **It primarily captures syntactic structural information, containing relatively limited semantic information. Consequently, this limitation restricts TOA's performance to semantics-dependent tasks.**\n\n# 6. Do tokenizers of different LMs require alignment? Should subtokens be aggregated? (q1)\n\n**Alignment of different LMs' tokenizers is not required, nor is aggregation necessary.**\n\nAs evidenced in Fig. 3, despite Qwen and LLaMA having different tokenizers, their OLA for the same text remains similar.\n\nWe did not aggregate subtokens. The axes in Fig. 3 display raw subtokens.\n\n# 7. Why does Sec. 3.3.1 use 4 source LMs instead of 1? (q2)\n\nEmploying four source LMs rather than fewer is intended to mitigate overfitting in the visual model.\n\nWe treat OLA generated from the same text by different LMs as belonging to the same class (line 183). This implies that the number of training samples per class equals the number of source LMs. **If only one source LM were used, each class would contain just a single sample. This would lead to overfitting in the visual model, preventing it from learning comprehensive features of OLA. Using four source LMs helps mitigate this issue.**\n\nTable 4 below presents the results of cross-model similarity evaluation for OLA based on the visual model when using only a single source LM (analogous to Table 1 in the main text), which validates the above perspective.\n\n**Table 4: Results of quantitative evaluation on cross-model similarity for OLA based on visual model using only a single source LM.**\n|src\\tgt|Q-1b5|Q-7b|G-2b|G-9b|L-3b|L-8b|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Q-1b5|-|95.70|19.10|6.60|8.10|9.80|\n|G-2b|9.30|8.90|-|74.60|39.70|36.50|\n|L-3b|10.50|7.70|48.00|41.40|-|96.50|"}, {"review_id": "bo4VEM7FoP", "reviewer": "Reviewer_U4s6", "summary": "The paper presents a novel study exploring the commonalities in context aggregation patterns among LMs. First, it unifies the attention weights of the models into a comparable representation and introduces the concept of Order-Level Attention (OLA) derived from the decomposition of Attention Rollout. Both quantitative and qualitative methods reveal significant similarities in the same-order OLA across different LMs, and it is found that OLA encodes rich syntactic information. Based on this finding, the researchers propose the Transferable OLA Adapter (TOA), a training-free cross-model adapter transfer method. The experimental results demonstrate that TOA can effectively enhance the performance of LMs on syntactic-related tasks and facilitate cross-model knowledge transfer.", "questions": "1. What is the principle of linking OLA with syntactic knowledge? \n2. Intuitively, the authors' findings stem from the observation that the OLA mappings of the same sentence in different models are highly similar. However, does this phenomenon also occur in sentences with similar semantic/syntactic structures? If semantically similar sentences do not show similar OLA mappings, but syntactically similar sentences do, then the motivation for this association may hold.\n3. The TOA method shows ineffective results on G-9b, and I noticed that the OLA's ability to predict syntactic dependencies for G-9b is not reported in Table 3. Could there be different conclusions for G-9b? For example, does G-9b's OLA fail to accurately predict syntactic dependencies, or does it achieve better performance in predicting syntactic dependencies compared to other models? This could further explain the applicability of the TOA method.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The research questions explored in this paper are far-reaching; exploring commonalities between different LMs can support cross-model knowledge transfer.\n2. The research methods are comprehensive and insightful. From discovering common structural aspects to applications, the authors clearly explain the motivations and thought processes involved in the whole process, which will benefit similar studies.\n3. This paper conducts extensive experimental verification and theoretical proof, which fully proves the rationality of the OLAS phenomenon.\n\nWeaknesses:\n\n1. The motivation for linking OLA with syntactic information is vague.\n\n\n2. The applicability of the TOA method is limited.\n\nOn the one hand, TOA seems to focus mainly on syntax-related tasks, and on the other hand, it is only suitable for models that originally lack the ability in syntactic structure-related tasks. From the results of Gemma2-9B, TOA does not seem to achieve effective results when the model performs well. This may be because the improvement in TOA performance mainly comes from OLA, which contains rich general syntactic structure information. When the model already has strong capabilities in syntactic structure-related tasks, OLA's capabilities may not be enough to support its original capabilities, thus losing its advantages.", "was_revised": true, "final_justification": "The author effectively addresses my concerns about his motivations and analysis of the results. I hope this will be reflected in the new version.", "rebuttal": "Thank you for your insightful suggestions and positive feedback. Your input will help us refine this work, and comments such as 'The research questions are far-reaching' and 'The research methods are comprehensive and insightful' serve as strong encouragement to us.\n\n# 1. What is the principle of linking OLA with syntactic knowledge? (w1,q1,q2)\n\nThank you for your insightful suggestions. Based on the two prompts below, we used GPT-4o to generate two sets of sentences. Group A contains 100 pairs of sentences with identical grammatical structures but opposite meanings. Group B contains 100 pairs of sentences with identical meanings but different grammatical structures. We used three CLMs to generate the OLA for these sentence pairs and computed their mean SSIM. As shown in Table 1 below, **sentence pairs with identical syntax but different semantics exhibit significantly higher similarity than pairs with identical semantics but different syntax. This further supports the view that OLA captures syntactic knowledge.**\n\n> 1. Prompt for generating Group A (syntactically identical, semantically opposite):\n> Please generate one hundred pairs of English sentences with identical grammatical structures but opposite meanings. The semantic opposition should be achieved by replacing one or more words in a sentence with words of the same part of speech but opposite meanings, resulting in sentences that convey contrary ideas. Each sentence must be exactly 50 words in length.\n> 2. Prompt for generating Group B (semantically identical, syntactically different):\n> Please generate one hundred pairs of English sentences with different grammatical structures but identical meanings. Each pair should convey the same semantic content, describing the same event or scenario, with each sentence strictly containing exactly 50 words.\n\n**Table 1: Image Similarity of OLA for Group A (100 sentence pairs with identical syntax/different semantics) and Group B (100 sentence pairs with identical semantics/different syntax). Values represent SSIM scores; higher values indicate greater image similarity (maximum = 1).**\n||Q-1b5|G-2b|L-3b|\n|:-:|:-:|:-:|:-:|\n|Group A|0.8180|0.8346|0.8086|\n|Group B|0.6474|0.6497|0.6339|\n\n# 2. TOA’s Limitation in Applicability (w2)\n\nWhile OLA exhibits cross-model consistency, it has not yet achieved a perfect feature representation. It primarily captures syntactic information and encodes semantic information weakly. Thus, it is well-suited for syntax-related tasks but remains ineffective for semantics-heavy tasks (e.g., text generation). **We will continue refining it in subsequent work, aiming to uncover a shared semantic feature representation across LMs.**\n\n# 3. Does G-9b's OLA show significantly different performance in the dependency parsing task compared to other LMs? (q3)\n\nDue to space constraints, not all models' syntactic dependency parsing performances were included in Table 3 in the main text. Table 2 below shows the performance of Q-7b, G-9b, and L-8b's OLA on syntactic dependency parsing. It can be seen that **the performance of G-9b's OLA on this dependency parsing task is comparable to that of other LMs, showing neither unusually high nor low results.**\n\nThe reason for TOA's weaker transfer performance on G-9b might lie in the similarity between G-9b's OLA and that of other models being slightly lower compared to other model combinations.\n\n**Table 2: Results of syntactic dependency parsing using OLA predicted by CLMs.**\n||Q-7b|G-9b|L-8b|\n|:-:|:-:|:-:|:-:|\n|rollout|49.47/30.83|43.61/20.00|49.95/25.87|\n|1st|60.50/45.24|60.32/43.54|59.33/42.85|\n|2nd|56.74/40.36|56.00/37.96|54.25/36.81|\n|3rd|52.69/35.31|50.85/30.47|50.36/27.21|"}]

Author rebuttal:

Meta-review: The paper investigates commonalities in the representations between different LMs. It proposes Order-Level Attention (OLA), which allows authors to compare attention modules from different LLMs. It is argued that OLAs are similar across different LMs. Based on these findings a training free cross-LMs adaptor (TOA) is proposed. These techniques are studied empirically. 

Generally, reviewers believe that the paper is ready to be accepted. In the final version the authors should improve the related work section and better contextualize their contribution in the broader research community, per Reviewer dT4c request.
Decision: Accept (poster)