CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model
Authors: Kaiyan Zhang
Ning Ding
Biqing Qi
Xuekai Zhu
Xinwei Long
Bowen Zhou
Venue: EMNLP 2023 Main (2023)
Keywords: large language models
transfer learning
representation similarity
model privacy

Abstract: Instruction tuning has recently been recognized as an effective way of aligning Large Language Models (LLMs) to enhance their generalization ability across various tasks. However, when tuning publicly accessible, centralized LLMs with private instruction data, privacy concerns are inevitable. While direct transfer of parameterized modules between models is a plausible approach to address this, its implications and effectiveness need further exploration. This paper focuses on Offsite-Tuning (OFT), a representative technique that transfers transformer blocks between centralized LLMs and downstream emulators. Given the limited understanding of the underlying mechanism of OFT, we perform an empirical analysis on LLMs from the perspectives of representation and functional similarity. Interestingly, our findings reveal a unique modular structure within the layers of LLMs that appears to emerge as the model size expands. Simultaneously, we note subtle but potentially significant changes in representation and intermediate predictions across the layers. Inspired by these observations, we propose CRaSh, involving Clustering, Removing, and Sharing, a training-free strategy to derive improved emulators from LLMs. CRaSh significantly boosts performance of OFT with billions of parameters. Furthermore, we investigate the optimal solutions yielded by fine-tuning with and without full model through the lens of loss landscape. Our findings demonstrate a linear connectivity among these optima falling over the same basin, thereby highlighting the effectiveness of CRaSh and OFT.

CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without
Full Large Language Model
Kaiyan Zhang', Ning Ding Biqing Qi Xuekai Zhu'
Xinwei Long', Bowen Zhoul
Department of Electronic Engineering, Tsinghua University, Beijing, China
Frontis.AM AI, Beijing, China
School of Astronautics, Harbin Institute of Technology, Harbin, China zhang-ky22@maills tsinghua, edu.
zhoubowen@tsingguu. edu. cn
Abstract
125M
5 10 15 20
350M
1.3B
18
Instruction tuning has recently been recognized effective way of aligning Large Language Models (LLMs) to enhance their generalization ability across various tasks. However, when tuning publicly accessible, centralized LLMs with private instruction data, privacy concerns are inevitable. While direct transfer of parame- terized modules between models is a plausible approach to address this, its implications and effectiveness need further exploration. This paper focuses on Offsite- Tuning (OFT), rep- resentative technique that transfers transformer blocks between centralized LLMs and down- stream emulators. Given the limited under- standing the underlying mechanism OFT, we perform empirical analysis LLMs from the perspectives representation and functional similarity. Interestingly, our find- ings reveal a unique modular structure within the layers LLMs that appears to emerge as the model size expands. Simultaneously, note subtle but potentially significant changes representation and intermediate predictions across the layers. Inspired by these observa- tions, we propose CRaSh, involving Clustering, Removing, and Sharing, training-free strat- egy derive improved emulators from LLMs. CRaSh significantly boosts performance of OFT with billions of parameters. Furthermore, we investigate the optimal solutions yielded by fine-tuning with and without full model through the lens of loss landscape. Our findings demonstrate linear connectivity among these optima falling over the same basin, thereby highlighting the effectiveness of CRaSh and OFT. The source code publicly available at https://gi thub. wowm/ssinguuuC131 TiinghaaC3//CRaS
8
0.8
3
12
5
10
15 20
6
12
18
24
32
6.7B
13B
308
24
30
36
0.4
16
20
24
32
10
20
30
40
12
24
36
48
Layer
Figure 1: Emergence of modular structure of repre- sentations of LLMs. The table displays representation similarity among layers of OPT models (Zhang al., 2022) on ARC dataset (Clark et al., 2018) with instruc- tion format. The lighter colors indicate higher similarity and same color scale used in all plots.
GPT-4 (OpenAI, 2023) due to their potential to- wards advanced intelligent systems. By employing techniques like prompt learning (Ding et al., 2022a; Wei et al., 2022b) and instruction tuning (Sanh et al. 2022; Wei et al., 2022a; Wang et al., 2022b,c), the behavior of LLMs can be aligned with human intent using small amount of data. (Taori et al., 2023; Chiang al., 2023; Zhou et al., 2023).
However, the centralization of LLMs poses a significant challenge concerning the trade-off be- tween high performance and user data (Li et al., 2022a). For instance, OpenAl provides fine-tuning APIs* that allows users to upload personal data for further fine-tuning of davinci models family, which has gained popularity in the industry, like Ope- nAI GPT-4 (OpenAI, 2023), Google Bard', and Anthropic Claude*, Safeguarding the privacy of both LLMs and downstream user data an urgent concern. One viable approach involves the direct transfer of parameterized modules between mod- els, such as Federated Learning (FL) (McMahan
1 Introduction
Nowadays, there is a growing interest in large lan- guage models (LLMs) such as PaLM (Chowdhery et al., 2022), LLaMA (Touvron et al., 2023) and
httpps://platfrmm openai. cataii..cmm/de wtt.//ooo/////eeeeeee/ /om////egguises hhttps://bard. google. com/ hhttps://www wtt.rrpppc.ccopppuuuuouee
"Corresponding author

---

al., 2017; Lin al., 2022) and Split Learning (SL) (Vepakomma et al., 2018; Thapa al., 2022), where limited exploration are conducted on LLMs with billions of parameters. Recently, Xiao et al. (2023) propose Offsite-Tuning (OFT) for transfer learning that operates independently of full LLMs with sizes exceeding 1B. OFT entails compressing the LLM into smaller model known emulator by layer dropping, followed by fine-tuning the em- ulator using user data. Finally, the parameterized modules are transferred from emulator and seam- lessly integrated into LLM a single turn. Despite the promising results obtained by OFT, there is still limited understanding of its underlying mechanism.
the relationship of optimal solutions in CRaSh, we visualize the optima using loss surfaces and mode connectivity (Li et al., 2018; Frankle et al., 2020). This study offers valuable insights into the effec- tiveness of CRaSh. In summary, our main contri- butions can be summarized as follows:
We discover emergence of modular structure within layers along with size of model in- crease in decoder-only models (e.g., OPT and LLaMA), which shows clusters of similar rep- resentations across layers (Section. 2).
We propose CRaSh, training-free strategy enhance layer dropping compression (Sec- tion. 3). CRaSh improves the performance of fine-tuning without full model and even out- performs knowledge distillation on multiple datasets (Section. 4).
In this paper, we conduct a detailed analysis of LLMs from the perspective of representation and functional similarity (Kornblith et al., 2019; Bel- rose et al., 2023) to enhance our understanding of OFT. Through comparing the similarity of hidden states across layers, we observe emergence of mod- ular structure of representations within LLMs. As depicted in Figure 1, models with a size less than 1OB exhibit uniform representations across all lay- ers. However, for models of size 13B and 30B, modular structure becomes apparent in the repre- sentation similarities. For the 13B model, high similarities are observed between layers 5 and 20, forming light-colored block-diagonal structure (i.e., modular structure), well between layers 25 and 40. Additionally, we note subtle changes the representation between adjacent layers. We further analyze functional similarity to explore the intermediate predictions of each layer, which en- hances these findings.
We analyze the optima of fine-tuning with and without full model through the lens loss surface and mode connectivity, which shows the two minima fall over the same basin and are connected linearly in parameter space. This observation explains the effectiveness of CRaSh (Section. 5).
Empirical Analysis
In this section, we analyze the inherent similar- ity of layers in LLMs from two complementary perspectives: (i) representation similarity (Sec- tion. 2.1), which examines the differences in activa- tions among intermediate layers, and (ii) functional similarity (Section. 2.2), specifically the variations in predictions among intermediate layers.
Building upon our findings, we propose com- pletely training-free strategy to enhance fine-tuning without relying full model. This strategy con- sists of three steps, namely Clustering, Removing, and Sharing (CRaSh). In the initial step, we cluster adjacent layers based on their similarity using vari- ous hierarchical clustering algorithms (Murtagh and Contreras, 2012). We then remove layers within the same cluster to obtain an emulator. The remaining layers are shared as supplement to the removed layers. Finally, selected layers of the emu- lator updated using downstream data and trans- ferred be seamlessly integrated into LLMs, re- sulting in improved performance. Through the uti- lization of CRaSh, we significantly enhance the per- formance of fine-tuning LLMs without full models across multiple datasets. In order to comprehend
2.1 Representation Similarity
Given LLM, which produces word representa- tions (i.e., hidden states) at each layer and ag- gregates them into sentence representations. We measure the similarity of sentence representations among layers using linear centered kernel align- ment (CKA) (Kornblith et al., 2019). CKA em- phasizes the distributivity of information. If two layers exhibit similarity across all their neurons, the similarity will be higher, even if individual neu- rons do not have similar matching pairs are not well represented by all neurons in the other layer. The representation similarity reveals correlations between layers within LLMs. The inherent sim- ilarity in the representation of layers across two

---

1.00
125M
10 35 20
350M
1.38
16 24 32
6.78
138
48
308
12 18
20 30
36
24
8
10
12
6
12
10
15
20
12
18
24
8
16
24
0
10
30
40
12
24
36
48
1.0
1.0
1.0
1.0
1.0
Similarity
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2g
0.2
0.2
24
0.2
10
0.2
12
Layer
Layer
Layer
Layer
Layer
Layer
Figure 2: This table presents the representation similarity among all layers (shown above the heatmaps) and the similarity between adjacent layers (represented by the line charts) on the Wikitext corpus (Merity et al., 2017).
datasets are demonstrated in Figure I and Figure 2. We provide detailed explanation of our two main findings as follows:
among layers. In addition to modular structure, we can also observe high similarity along the di- agonal of the similarity matrix. This indicates that intermediate layers may exhibit significant similar- ity to their adjacent layers. We directly examine the representation similarity between adjacent layers in Figure 2, where point at 2; means similarity be- tween layer i and layer + 1. In addition to the low similarity observed in the bottom layers, starting from about the 5th layer, there is significantly higher similarity compared to adjacent layers.
Finding 1: Emergence of modular structure. As shown in Figure and Figure 2, we observe that as the number of parameters increases, distinct blocks or modules with high similarity emerge in the representations of different layers. We refer to these blocks of high-similarity representations modular structure. The emergence of modular structure be seen self-organizing behav- ior during the pre-training of LLMs, where the internal representations gradually differentiate into modules with specific functions or semantics. This phenomenon has been briefly examined in previous studies (Phang et al., 2021; Merchant et al., 2020; Chen et al., 2021; Chiang et al., 2020). To the best of our knowledge, no study has investigated this question LLMs with billions of parameters.
Furthermore, we broaden the scope of these find- ings to include more LLMs and datasets, uncover- ing a consistent trend of modular structure emer- gence. The details of results, as well as implemen- tation details, are provided in Appendix A.1.
2.2 Functional Similarity
In addition, a complementary perspective to rep- resentation similarity involves extracting specific concepts from the hidden states. For example, we can convert the hidden states at each intermediate layer into probability distribution over the vocab- ulary (nostalgebraist, 2020; Belrose al., 2023). This approach facilitates deeper comprehension of the functional aspects of the layers. For the anal- ysis of functional similarity, we utilize the state- of-the-art tool Tuned-Lens to obtain predictions from the intermediate layers. Additional details are provided in Appendix A.2.
Additionally, our initial findings indicate that there are no evident clusters of layers in pre-trained models with millions of parameters, which may indicate insufficient capacity to comprehend tasks in zero-shot setting. However, when the model size reaches certain threshold, referred to as mod- ular point, modular structure emerges in LLMs, resulting in the formation of distinct clusters. Our experiments have shown that the specific modular point varies across tasks. Furthermore, we observe that the modular point is larger for harder tasks, but smaller for easier tasks, such as language model- ing. For example, in the case of the ARC dataset (a question answering task with instruction format), modular point is observed to be 10B, as depicted in Figure 1. On the other hand, for the Wikitext corpus (a language modeling task), modular point is found to be 1.3B, as illustrated in Figure 2.
Finding 3: Removing layers within cluster maximizes behavior invariance. In the sub-figure labeled full model of Figure 3, we observe subtle changes as the depth increases and note that adjacent layers display high similarity in hidden predictions, resulting in the formation of clusters.
Finding 2: Subtle changes in representation

---

ful model
un form (remmnry lover di Tho
Inteclattr (remmun lager
highekueeer Swemooing layer 20 220
Men Proae Py Byyppe
This figure shows results of Tuned-lens on OPT-1.3B and three variants of layer dropping, The entire input text consists of "One Ring to rule them vi One Ring to find them, One Ring to bring them alla and the darkness bind them", where tokens from position 14 to 21 are indicated.
Figure 3:
To assess the redundancy of layers, we eliminate multiple layers from the cluster based on represen- tation similarity in Figure and compare the pre- dictions when these layers are uniformly dropped. Based on the results, we also observe all the re- moval strategies yield predictions that exhibit over- all similarity the internal layers of the full model, showing only minor differences in probability, as indicated by subtle variations in color shades. In comparison to the fuull model, the @uni form strategy exhibits notable differences in the middle layers of token one, high layers of token four, and the last two tokens, indicating the presence of im- pure predictions in these positions. Conversely, the @llow-cluster and @hiighclusteer strate- gies display closer aligmment with the full model these locations. This phenomenon leads to the conclusion that adjacent layers with high represen- tation similarity also demonstrate analogous func- tional similarity. Furthermore, removing layers from these clusters results in maximum behavior invariance compared to uniformly dropping them.
LLM owner, and vice versa, the LLM owner cannot share their model with the data owner.
As shown in Figure 4a, OFT involves compress- ing the LLM into smaller emulator by layer drop- ping and fine-tuning it using privacy data (i.e., E))mulltor Fine-tuning). Subsequently, the block weights are transferred and plug-in the LLM for inference (i.e., PPlugiin). The main objective of OFT is for the plug-in to exceed the performance of both full zero-shot on the LLM and fine-tuning on the emulator a > O, 3), thereby ensuring the overall effectiveness of OFT. Additionally, OFT aims to optimize the performance of the plug-in to closely approximate the results achieved by full fine-tuning on the LLM
2)
The key to OFT lies in identifying an emula- tor that closely resembles the original LLM and subsequently performing fine-tuning to replace the LLM. Drawing from our findings on the empirical analysis of LLMs in Section 2, propose CRaSh, which designed to optimize layer dropping and yield superior sub-layer emulators from LLMss as illustrated in the following sections.
3 Methodology
3.2 CRaSh
Empirical analysis of LLMs can inspire fine-tuning approaches, contributing to privacy protection for both centralized LLMs and user data. In this sec- tion, we begin by revisiting Offsite-Tuning (OFT), a representative method for fine-tuning LLMs with- out full model. Subsequently, we introduce our proposed CRaSh method inspired by Section.
In this section, we provide detailed description of the three steps of CRaSh, namely Clustering, Re- moving, and Sharing presented in Figure 4b. The concept of Clustering and Sharing are supported by Finding which suggests that layers within a cluster share similar functions. Additionally, Find- ing and Finding 3 support the idea of removing layers from clusters, as it helps minimize changes in representations and functions.
3.1 Revisit Offisite Offsite-Tuning
In this section, provide a brief introduction the settings and evaluation metrics of Offsite- Tuning (OFT) (Xiao al., 2023). The primary con- cern of OFT to maintain the privacy of both the LLM and user data. Specifically, the data owner is unable to share their labeled training data with the
Clustering Drawing inspiration from the pres- ence of layer clusters in representations and the gradual change layer behavior in Section. 2, it is observed that adjacent layers within a cluster may have similar functions. These layers can be

---

Zerochet
Comgrenn
Transher
OMI Pre-toring
Piggin
(a) Fine-tuning and inference strategy.
(b) Overview CRaSh Strategy
(c) Adjacent Clustering
Figure 4: Overview of Pffsite-Tunnig and CRaSh strategy.
grouped together into single cluster and subse- quently replaced by the cluster center. As shown Figure 4c, we propose a process for clustering the layers of LLMs based on the similarity of repre- sentations between adjacent layers. Initially, each individual layer is considered as a separate clus- ter. Subsequently, the closest clusters are selected and merged into new cluster based on the CKA metric of their output representations. This step is repeated until the desired number of clusters is reached. This process bears resemblance to hierar- chical clustering (Murtagh and Contreras, 2012), the key distinction lies in the fact that we exclu- sively cluster the adjacent layers.
implement layer sharing within the remaining model to achieve optimal results.
Experiments
4.1 Setup
Dataset. We evaluate CRaSh on three tasks and eight datasets which are used in Offsite- Tuning (Xiao et al., 2023), including Multi- Choice QA: OpenBookQA (Mihaylov et al., 2018). PIQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), RACE (Lai et al., 2017); Closed-Book QA: EaRChassyChallleng (Clark et al., 2018), WebQuestion (Berant et al., 2013); and Sentence Completion: HellaSwag (Zellers et al., 2019). To enhance zero-shot performance, we organize source and target text in instruction format as 1m- evaluation- utilize it to evaluate our models and report the curacy on all benchmarks. For clustering step, we utilize data from the same task as support dataset to maintain the privacy of target dataset, including BoolQ (Clark et al., 2019). TriviaQA (Joshi et al., 2017), and CoPA (Wang et al., 2019). We show details about statistic of datasets in Appendix B. Models. We perform empirical analysis range of models, including OPT (from 125M 30B) (Zhang al., 2022) and LLaMA (from 7B to 30B) (Touvron et al., 2023). Due to limited com- putational resources, we primarily conduct main experiments on OPT-1.3b, with plans to scale up to OPT-6. 7BB and LLaMA-7B. As our primary base- lines, we consider models from OFT (Xiao et al., 2023), including a knowledge distillation emula- tor and uniform 2-8-2 configuration for OPT- 1.3b, 2-18-2 for OPT-6.7B and LLaMA-7B. Here, 1-c-r denotes setting the parameters of bottom and top r layers as learnable, while c layers are uniformly selected from original LLM and kept
Removing Following the clustering of interme- diate layers, only layers located the cluster cen- ters are retained, while the rest within the clusters are removed. Subsequently, a selection of layers is chosen for fine-tuming where parameters of the bottom and top layers are learnable in OFT (Xiao et al., 2023) (with 2). Considering the signifi- cance of knowledge contained in the middle layers for downstream tasks, we uniformly select set of n layers from the remaining layers after removal. The exploration of skill layers (Sajjad et al., 2023; Jordao et al., 2023) based on module cruciality is left as potential area for future research.
(Gao et al., 2021). We
Sharing In the original OFT approach, the model with the remaining layers is considered as an emulator and utilized for fine-tuning on down- stream user data. This allows users to experiment with different strategies to optimize the emulator's s performance. For downstream fine-tuning, we pro- pose straightforward strategy to enhance the per- formance of emulator. Considering the emergent abilities of LLMs, certain features, such as chain- of of-thought (Wei et al., 2022b), may not be present in shallow layer models. Drawing inspiration from the concept of layer sharing (Lan et al., 2020),

---

Setting
OpenBookQA
ARC-E
ARC-C
WebQs
PIQA
SciQ
RACE
HellaSwag
Full Large Language Model
Zeto shot (ZS)
23.4%
56.9%
23.5%
4.6%
71.6%
84.4%
34.2%
41.5%
Fine uning (FT)
27.7%
75.2%
92.5%
42.7%
Knowledge Distillation (Coutinual Training)
Emulator ZS
19.4%
53.9%
21.5%
1.3%
68.7%
80.9%
33.0%
35.1%
Emulutor
37.055
Plug-in (Nia al., 2023)
29.0%
59.4%
27.8%
26.2%
74.5%
92.9%
38,9 %
43.3%
Uniform Strategy (Training-free)
Emulator 2S
19.0%
Emulator FT
24. 6%
50.4%
21.2%
21.8%
69.3%
89. 4%
36.5%
32.7%
Plug-in (base)
26.4%
58.3%
23.0%
21.4%
72.7%
90.8%
37.9%
41.2%
with uniform Inarmable layers
Emulator
24.2%
69.3%
89.3%
33.8%)
Plug in
27.6%
58.8%
24.8%
16.4%
72.6%
92.1%%
39.7%
41.2%
CRaSh Strategy (Treining-free)
Emulutor 7S
35,
18.5%
4.7%
84.3%
Fruulator FT
25.0%
50.0%
21.5%
21.8%
68.9%
88.9%:
38.9%
33.6%
Plug-in (our)
14.8
=1.8
13.3
73.2% 20.5
-3.a
19.0
41.9% m0.7
w/o layer sharing
Emulator Z8
35.9%
18.5%
57.0%
43.3%
25.99 25.9%
Emulator FT
23.8%
50.0%
21.5%
24.3%
68.8%%
89.6 59.6%%
36.2%
31.2%
Plug-in
24.6%
41.3%
Table I: Results of CRaSh OPT-1.3B. The values in red font indicate increase compared to the uniform strategy, highlighting the superiority of CRaSh. Remarkably, CRaSh outperforms the strategy with knowledge distillation (KD) on several datasets. is worth noting that can further enhance the performance CRaSh.
frozen during fine-tuning. We present implementa- tion details about experiments in Appendix B.2.
ments on OpenBookQA and 1.0% on RACE. The knowledge distillation method obtains the emulator by continuously pre-training it the first block of the Pile corpus (Gao al., 2020), which aims to align the intermediate representation between emulator and full model. Therefore, CRaSh is com- plementary to knowledge distillation, which can further improve performance by leveraging the im- proved emulator initialization provided by CRaSh. CRaSh works better while scaling up model. As depicted in Table 2, CRaSh remains effective even when scaling up the model from 1B to 7B. The benefits and effectiveness of CRaSh not restricted to specific model sizes, making it valu- able strategy for enhancing OFT outcomes across models of varying scales. Meanwhile, as the depth increases, layers are more prone to redundancy in LLMs. Therefore, by employing clustering step, CRaSh eliminates redundant network layers and effectively improves the performance of OFT com- pared to the uniform strategy.
4.2 Experimental results
We present the main results in Table 1 and provide analysis as follows.
OFT and CRaSh does work well. CRaSh satis- fies the condition of OFT, where the performance of plug-in better than the zero-shot performance of full model and the fine-tuning performance of emu- lator. The results prove that OFT effectively works for fine-tuning without full model and is benefi- cial for protecting the privacy of L.LMs. Addition- ally, due to pereparrameteiizationn of LLMs (Agha- janyan et al., 2021; Ding et al., 2022b), , it may not be necessary to optimize all parameters. There- fore, the performance of plug-in can surpass that of directly fine-tuning on LLMs, particularly on datasets like SciQ and RACE.
CRaSh is better than uniformly dropping startegy. Compared to the uniform layer drop- ping strategy used in OFT (Xiao et al., 2023), CRaSh is effective method to boost performance that requires low additional cost and is completely training-free, where lifting effects are indicated in red font. Additionally, CRaSh can outperform the knowledge distillation models OFT settings on several datasets, such as achieving 1. 1.2% improve-
4.3 Ablation study
Impact of clustering and sharing steps. This sec- tion discusses the significance of clustering and sharing steps in CRaSh. We compare the results with and without the clustering step, as shown Table 1. OFT (Xiao et al., 2023), the top and

---

Setting
OpenBookQA
ARC-E
ARC-C
WebQu
PIQA
SeiQ
RACE
HellnSwag
OPT-6.7B
Full
27.6%
65.6%
30.6%
8.8%
76.2%
90.1%
38.2%
50.5%
Emulator ZS
21.4%
55.6%
23.9%
1.5%6
57.0%
84.1%
31.1%
28.4%
Emuuaatrr FT
29.0%
60.1%
31.1%
22.1%
75.6%
88,49
36.2%
43.4%
Plug-in (Niao al., 2023)
33.8%
66.8%
33,9%
23.9%
71.7%
91.9%
44.1%
52.1%
Plug-in (CRaSH)
38.8%
16.0
70.7% 18.9
36.3% 12.4
26.1% 12.2
78.0% 19.3
95.3% 14.2
45,2% 11.1
53.4%
1oMM--BB
Full ZS
28.2%
67.3%
38.2%
0.0%%
78.3%
89.7%
40.0%
56.4%
Emulator ZS
15.0%%
44.3%
23.5%
0.0%
65.7%
57.6%
30.2%
36.2%
Emulator FT
25.4%
60.0%
28.8%
25.7%
73.6%
91.8%
40.8%
45.0%
Plug-in (Uniform)
33.0%
69.6%
39.0%
27.3%
78.8%
93.5%
44.0%
57.4%
Plug-in (CRaSH)
34.6%
71.3%
11.7
41.8%
12.8
29.8% %
19.5
80.0%
11.2
95.1%
11.0
45.6%
13.6
58.4% 23.0
Table 2: Results OPT-6.7B 7 and LLaMA-7B. CRaSh continues to perform effectively the model size scales up, and even achieves further improvement, benefiting from the emergence of modular structure.
Datnset
OpenBookkQA
ARC-E
ARC-C
WehQs
downstream data. Therefore, we solely utilize data from the support task for clustering, which shares similar task type with the downstream task. As presented in Table 3, we compare this approach with using public general task and directly down- stream dataset in order to examine the impact of data type on clustering and the resulting plug-in performance. By avoiding direct use of the down- stream task, which may compromise privacy, we achieve comparable performance by utilizing data solely from similar support task. We leave it for future work to explore the identification of the most relevant support task for clustering based on downstream task information.
58.5%
Support Task
30.2%
60.0%
24.8%
23.7%
Downstream Task 29.4%
Table 3: The table presents the plug-in results of CRaSh, considering different data types for the clustering step.
0.35
Iatenoste: litteetrigg
20
Numbor Syors
Figure 5: The accuracy varies as the number of layers be dropped changes OPT-6.7B model.
Number of layers for emulator. To ensure fair comparison, we set the number of layers of emulator to 12 and in main experiments, respec- tively. for 1B and 7B LLMs. To investigate the impact of an extensive range of layers, we varied the number of clusters from 2 to the maximum number of total layers. The performance of plug-in increases in accordance with the number of layers for the emulator, depicted in Figure 5. Addition- ally, we observed that the performance of plug-in can achieve comparable effect to full fine-tuning when the layers comprise only 50% ~ 60% of the LLMs. However, there is still room for further ex- ploration when transferring fewer layers of LLMs.
bottom two layers are set to be learnable, while the four layers be updated are uniformly cho- sen in our CRaSh strategy. Hence, run the Uniform Strategy in "with uniform learnable lay- ers" and compare with CRaSh Strategy without layer sharing (e.g., "w/o layer sharing"). In cases where only the layers of the emulator differ, we find that the clustering step provides slight utility compared to the uniform strategy. This trend may be attributed to the potential loss of hidden informa- tion caused by layer dropping. However, this loss be mitigated by incorporating layer sharing and muning resulting in improved performance of the final CRaSh approach. The results in Table also indicate the significance of layer sharing steps, as they enable the emulator to have sufficient depth to handle challenging tasks.
Discussion
Loss landscape and mode connectivity. In der to comprehend the effectiveness of CRaSh, we conduct an analysis of the relationship between op- timal minima derived from plug-in and full model fine-tnning Firstly, based on Figure 6a, we observe that the initialization resides in a low basin and per-
Impact of data type for clustering. In our main experiments, we take into account the privacy of

---

openbookqa
aperiboktge
Maay
Intopp allos
10
GFT' hd
2.0
150
75
St
2.6
2A
contictent
conficont
(a) Loss surface
(b) Model interpolation
Figure 6: (a) The initialization weights and optima obtained through CRaSh full model fine-tuning are located within the same basin. (b) Model interpolation weights from CRaSh, OFT, and full fine-tuing.
forms well across various datasets, which benefits from the over-parameterization and generalization of LLMs. Consequently, by optimizing the LLM at a relatively low cost on the target dataset, it can effectively perform on this dataset. This observa- tion highlights the efficacy of delta-tuning (Ding al., 20226) and offsite-tuning. Secondly, the so- lutions derived from CRaSh and full fine-tuning reside within the same basin. Utilizing the clus- tering and sharing steps, CRaSh exhibits a closer proximity to full fine-tuning compared to OFT (re- fer to Figure 6b). Finally, through the interpolation of various solutions, we observe their mode con- nectivity in the parameter space, wherein CRaSh exhibits smoother transition towards full fine- tuning compared OFT, which undergoes more dramatic changes. Based on the visualization the loss landscape, there potential to enhance CRaSh for better performance in future directions.
in CRaSh is viewed as the most effective recon- struction technique. However, it does not repli- cate the performance of the original model entirely. As indicated in Table 2, the emulator fine-tuned with CRaSh does not achieve the full model's zero- shot performance. But, when integrated, there's s marked improvement in performance across mul- tiple datasets, especially ARC-E/C, PIQA, RACE, and HellaSwag. (2) crucial factor the recon- struction challenge is the number of transferred layers; fewer transferred layers complicate the re- construction. Figure 5 demonstrates that accuracy varies with the number of layers omitted. Interest- ingly, emulators with merely 6 layers (2 frozen and 4 fine-tuned, compared to the 32 layers in the full model) continue to boost the primary LLM's per- formance when incorporated. Undoubtedly, CRaSh could be enhanced by incorporating federated learn- ing technologies, including homomorphic encryp- tion (Lee et al., 2022; Chen et al.. 2022) and dif- ferential privacy (Yue et al., 2021), making the original model's reconstruction more challenging.
Cnrrmter-Efffcieen Fine-tuning (PEFT). PEFT has gained popularity as a method for adapting LLMs with minimal parameter updates or additions. For instance, in the case of LoRA, only 590K parameters need to be updated for OPT-1.3B, while OFT requires updating 201 M parameters and the full model requires updating 1208M parameters. By applying LoRA to the transfereed layers, CRaSh achieves parameter- and resource-efficient fine-tuning. Further details on this topic can be found in Appendix C.2.
6 Related Work
Delta-tuning (Ding al., 2022b) methods effi- ciently update a subset of parameters compared to the entire LLMs (Houlsby et al., 2019; Ben Za- ken et al., 2022: Hu et al., 2022; Dettmers et al.. 2023). However, these methods require feeding data into the entire LLMs, which not resource- efficient and raises concerns about the privacy of LLMs. Black-box tuning (Sun et al., 2022b,a), upplied centralized LLMs, attempts to learn pa- rameters based input or output text (Cui et al., 2022), which helps protect LLMs but poses risks to user data. Directly manipulating model pa- rameters instead of transferring data has found wide applications in federated learning (McMa-
Reconstruct full model from emulator. When transmitting the emulator downstream, an impor- tant consideration emerges: given the emulators, how challenging would it be reconstruct the orig- inal model? The core question the attainable performance level with just the emulator. Our find- ings, detailed Section 4, discuss the complexi- ties involved in reconstruction: (1) Layer sharing

---

han et al., 2017), split learning (Vepakomma et al.. 2018; Thapa et al., 2022), distributed gradient de- scent (Huo et al., 2018; Xu et al., 2020; Ni et al., 2023), branch-trainn-ie interpotttrooonioonnn (Li et al., 2022b; Wortsman et al., 2022), and collaborative machine learning development (Raffel, 2023; Kandpal et al., 2023). These methods either involve model sharing between servers and clients or require multi-turn communication to achieve coverage, which poses limitations when applied to LLMs with billions of parameters. Offsite- Offsite-Tuning (Xiao et al., 2023) is a notable method that addresses these challenges by selectively sharing layers of LLMs with bil- lions of parameters between servers and clients in a single turn. On the other hand, previous studies have focused on gaining insights into the interme- diate representation of neural networks for better fine-tuning (Kornblith et al., 2019; Merchant et al.. 2020; Wu et al., 2020; Raghu et al., 2021). In terms transformers, Phang et al. (2021) observed clus- tering of layer representations in fine-tuned BERT models(Devlin et al., 2019), supporting the notion of layer redundancy (Dalvi et al., 2020). In contrast, we find that clustering also emerges in pre-trained LLMs as the model size increases, which helps Offsite-Tuning LLMs. detailed introduction related work is presented in Appendix D.
as Pythia). Due to limited computing resources. our main experiments were conducted on LLMs with a model size of less than 10B.
Although we hypothesize that larger models with redundant layers and CRaSh may lead to improved performance, further exploration is necessary to validate this hypothesis. We utilize representation similarity as clustering metric, and although it demonstrates satisfactory performance in our ex- periments, we have encountered challenges and ob- served instability, Consequently, we intend to inves- tigate functional similarity as an alternative, which may necessitate additional preprocessing time.
Given the complexity involved in validating the plug-in, further research on behavior predicting is required to enhance CRaSh (which tends to be heuristic strategy) and transform it into an auto- mated learnable strategy (such as reinforcement learning). For future work, we plan to do analy- sis using more similarity methods and broaden the application of this strategy to encompass a wider range of instructional data and LLMs.
Ethics Statement
Our research on Offsite-Tuning (OFT) and the pro- posed CRaSh strategy aligns with the ethical guide- lines outlined by the ACL Ethics Policy. We recog- nize the importance of addressing privacy concerns when fine-tuning publicly accessible, centralized Large Language Models (LLMs) with private in- struction data.
7 Conclusion
this paper, we uncover block structure of repre- sentation within the intermediate layers of LLMs, indicating the clustering of layers in depth models. Based on these observations, we propose a com- pletely training-frre frree strategy to enhance fine-tuning without full model. The strategy consists of three steps: Clustering, Removing, and Sharing (CRaSh). CRaSh boosts the performance of Offsite-Tuning for LLMs on various datasets. Further analysis of the loss surface and mode connectivity provides insights into the effectiveness of CRaSh.
The primary objective of our work is to enhance the generalization ability of LLMs across various tasks while safeguarding the privacy of both the LLMs and the instruction data. We acknowledge the potential risks associated with the direct transfer of parameterized modules between models and the need for further exploration to fully understand its implications and effectiveness.
Our research adheres to ethical principles by promoting privacy protection, transparency, and responsible use of LLMs. We are committed to continuous ethical evaluation and will contribute to the ongoing discourse on the ethical implications of language model fine-tuning.
Limitations
This paper primarily focuses on fine-tuning LLMs without using the full model, thereby safeguard- ing the privacy of both centralized model and data. Our empirical analysis on LLMs has inspired straightforward yet effective improvement that out- performs previous methods.
Acknowledgements
This work is supported by the National Key R&D Program of China (No. 20227D01 19101). We extend our gratitude to the anonymous reviewers for their insightful feedback.
However, We have only conducted experiments using two types of LLMs (OPT and LLAMA), leav- ing a wide range of other LLMs unexplored (such

---

References
and Furu Wei. 2022. THE-X: transformer inference with homomorphic encryption. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3510-3520, Dublin, Ireland. Association for Computational Linguistics.
Armen Aghajanyan, Sonal Gupta, and Luke Zettle- moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Pro- ceedings of the 59th Annual Meeting of the Associa- sion,oor Computational Linguisties and the I 11th Inter- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages 7319-7328, Online. Association for Computational Linguistics.
Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Has- san Awadallah, Zhangyang Wang. and Yu Cheng- 2023. Dsee: Dually sparsity-embeadeed efficient tun- ing of pre-treaiee pre-trained language models.
Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky. Lev McKinney, Stella Bider- man, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens.
Cheng Mang-HHn Chiang, Sung-Fem Feng Huang, and Hung-yi yi Lee. 2020. Pretrained language model embryology: The birth of ALBERT. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 0813-6828. On- line. Association for Computational Linguistics.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficien fine-tuning for transformer-based masked language-models, In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguistics.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1533-1544, Seattle, Wash- ington, USA. Association for Computational Linguis- tics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Nlya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Lan- guage models can explain neurons in language mod- els.
Christopher Clark, Kenton Lee, Ming-Wo Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguistics.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference Artificial Intelli- gence, AAAI 2020, The Thirty-Secone Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question swering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.05457.
Boli Chen, Yao Fu, Guangwei Xu, Pengjun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. 2021. Probing BERT in hyperbolic spaces. In 9th Inter- national Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Adrián Csiszárik, Péter Korösi-Szabó, Ákos K. Matszan- gosz, Gergely Papp. and Dániel Varga. 2021. Similar- ity and matching of neural network representations. In Advances in Neural information Processing Sys- tems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 5656-5668.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained BERT networks. In Advances in Neural In- formation Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 2020, virtual.
Ganqu Cui, Wentao Li, Ning Ding, Longtao Huang, Zhiyuan Liu, and Maosong Sun. 2022. Decoder tun- ing: Effiiientt language understanding as decoding. ArXiv preprint, atts/2212.044/
Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li,

---

Language Processing (EMNLP). pages 4908-4926, Online. Association for Computational Linguistics.
Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black. Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang. Laria Reynolds, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
Luke Zettlmmyer. 2023. Qlora: Efficient finetuning of quantized llms.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning transfer learning. In Proceedings of the Sth Workshop on Representation Learning for NLP, pages 143-155, Online. Association for Com- putational Linguistics.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022a. OpenPrompt: An open-source framework for prompt-lcarning. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 105-113, Dublin, Ireland. Association for Computational Lin- guistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015, Distilling the knowledge neural network.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NL.P In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang. Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang. Juanzi Li. and Maosong Sun. 2022h. Delta tuming: A comprehensive study of parameter efficient methods for pre-trained language models.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,
Angela Fan, Edouard Grave, and Armand Joulin. 2020. Reducing transformer depth on demand with struc- tured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Zhouyuan Huo, Bin Gu, Qian Yang, and Heng Huang- 2018. Decoupled parallel backpropagation with con- vergence guarantee. In Proceedings of the 35th In- ternational Conference on Muchine Learning. ICML 2018, Stockholmsmässan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2103-2111. PMLR.
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference Learn- ing Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. 2020. Linear mode con- nectivity and the lottery ticket hypothesis. In Pro-
Artur Jordao, George Correa de Araujo, Helena de Almeida Maia, and Helio Pedrini. 2023. When layers play the lottery, all tickets win at initialization.
ceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Vir- tual Event, volume 119 of Proceedings of Machine Learning Research, pages 3259-3261 PMLR
Mandar Joshi, Eunsol Choi. Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), pages 1601-1611, 1, Vancouver, Canada. Association for Computational Linguistics.
Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Mas- sive language models can be accurately pruned in one-shot.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, 2023. Gptq: Accurate post-training quantization for generative mre-trained transformers.
Nikhil Kandpal, Brian Lester, Mohammed Mugeeth, Anisha Mascarenhas, Monty Evans, Vishal Baskaran. Tenghao Huang, Haokun Liu, and Colin Raffel. 2023. Git-theta: A git extension for collaborative develop- ment of machine learning models. ArXiv preprint. hls/2306.04529,
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn

---

Max Klabunde, Tobias Schumacher, Markus Strohmaier. and Florian Lemmerich. 2023, Similarity of neural network models: A survey of functional and represen- tational measures. ArXiv preprint, abs/2305 06329,
Yuanxin Liu, Fandong Meng, Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang. and Jie Zhou. 2022. Learning to win lottery tickets in BERT transfer via task-agnostic mask training. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Techmologies, pages 5840-5857, Seat- tle, United States. Association for Computational Linguistics.
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of Machine Learn-
the 36th International Conference ing. ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pages 3519-3529, PMLR.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agúera y Arcas. 2017. Communication-effiiiii learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statisticx, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL. USA, volume 54 of Proceedings of Machine Learning Research, pages PMLR.
Guokun Lai, Qizhe Xie. Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785- 794, Copenhagen, Denmark. Association for Compu- tational Linguistics.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick. and Ian Tenney. 2020. What happens to BERT embed- dings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP. pages 33 44, Online. Association for Computational Linguistics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th Inter- national Conference Learning Representutions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture mod- els. Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. Open- Review.net.
Garam Lee, Minsoo Kim. Jai Hyun Park, Scung- won Hwang, and Jung Hee Cheon. 2022. Privacy- preserving text classification on BERT embeddings with homomorphic encryption. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 3169-3175. Seattle, United States. Association for Computational Linguistics.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. 2023. Language models implement simple word2vec-style vector arithmetic.
Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Li, Peng Qi. Bo Liu, Shuai Di, Jingen Liu, Jiquan Pei, Jinfeng Yi, and Bowen Zhou. 2022a. Trustworthy ai: From principles to practices.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31: Annual Conference on Neu- ral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 6391-6401.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jef- frey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint a3Xiv:1301.3781.
Margaret L., Suchin Gururangan, Tim Detimers, Mike Lewis, Tim Althoff, Noah Smith, and Luke Zettle- moyer. 2022b. Branch-train menne: Embarrassingly parallel training of expert language models. ArXiv preprint, 00122000.0306....
Niklas Muennighoff. 2022. Sgpt: Gpt sentence embed- dings for semantic search.
Fionn Murtagh and Pedro Contreras. 2012. Algorithms for hierarchical clustering: an overview. Wiley Inter- disciplinary Reviewx: Data Mining and Knowledge Discovery. 2(1):6699) 97.
Bill Yuchen Lin, Chaoyang He, Zihang Ze. Hulin Wang. Yufen Hua, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Aves- timehr. 2022. FedNLP: Benchmarking federated learning methods for natural language processing tasks. In Findings of the Association for Compu- tational Linguistics: NAACL 2022, pages 157-175, Seattle, United States. Association for Computational Linguistics.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021. Do wide and deep networks learn the same things? uncovering how neural network represen- tations vary with width and depth. In 9th Inter- national Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

---

Zanlin Ni, Yulin Wang. Jiangwei Yu, Haojun Jiang. Yue Cao, and Gao Huang. 2023. Deep incubation: Training large models by divide-and-conguering.
Empirical Methods in Natural Language Processing. pages 3916-3930, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
nostalgebraist. 2020. interpreting gpt: the logit lens. LessWrong.
Tranxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022b. Black-box tuning for sataaag--mmodll------eeeeee
In International Conference on Machine Learning, ICML 2022, 17-23 July 2022. Baltimore, Maryland. USA, volume 162 of Proceedingy of Machine Learning Research, pages PMLR.
OpenAI. 2023. Gpt-4 technical report.
Jason Phang, Haokun Liu, and Samuel R. Bowman. 2021. Fine-tuned transfommers show clusters of simi- lar representations across layers. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and interpreting Neural Networks for NLP. pages 529- 538, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and mem- ory efficient transfer learning, ArXiv preprint, 2bs/2206.06552.
Colin Raffel. 2023. Building machine learning models like open source software. Communications of the ACM, 66(2):38-4) 30.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang. Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression generative pre-trained language mod- els via quantization. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Longg Papers), pages 4821- 4836, Dublin, Ireland. Association for Computational Linguistics.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers see like convolutional neural net- works? In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 1021, NeurIPS 2021, December 6- 6-14, 2021, virtual, pages 12116-12128.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023, Stanford alpaca: An instruction-followinn llaaa model. https: // gi thub. cww www.ssslabbsttaoorr....
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping layers of pre-trained transformer models. Computer Speech &amp Language, 77:101429.
Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. 2022. Splitfed: When federated learning meets split learn- ing. In Thirty- AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence EAAI 2022 Vir- tual Event, February 22 March 2022, pages 8485- 8493. AAAI Press.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani. Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. ArXiv preprint, abs/2302.13971 13971..
Praneeth Vepakomma, Otkrist Gupta, Tristan Swedish, and Ramesh Raskar. 2018. Split learning for health: Distributed deep learning without sharing raw patient data. ArXiv preprint, abs/181 1b.0544..
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP) pages 4323 4332, Hong Kong, China. Association for Com- putational l Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275.
Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. 2022a. BBTv2: Towards a gradient-free future with large language models. In Proceedings of the 2022 Conference on

---

Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. 2022a. Finding skill neurons in pre-trained transformer-based language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1133-11152, Abu Dhabi, United Arab Emi- rates. Association for Computational Linguistics.
Guangxuan Xiao, Ji Lin, and Song Han. 2023. Offsite- tuning: Transfer learning without full model. arXiv.
An Xu, Zhouyuan Huo, and Heng Huang. 2020. On the acceleration of deep learning model parallelism with staleness. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 2085 2094. IEEE
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022b. ielf-instruct: Aligning lan- guage model with self generated instructions. ArXiv preprint, abs/2212..n 10560.
Xiang Yue, Minxin Du, Tianhao Wang, Yaliang Li. Huan Sun, and Sherman S. M. Chow. 2021. Dif- ferential privacy for text analytics via natural text sanitization. In Findings of the Association for Com- putational Linguisticx: ACI-IJCNLPP 2021, pages 3853-3866, Online. Association for Computational Linguisties.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phami Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022c. Super-Natural Instrrtonss General- ization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4791-4800, Florence, Italy. Association for Computational Linguisties.
Minjia Zhang and Yuxiong He. 2020. Accelerating training of transformer-based language models with progressive layer dropping. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M. Dai, and Quoe V. Le. 2022a. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representa- fions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. ArXiv preprint, abs/2305. 12206.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201.119003
Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 94-106, Copenhagen, Den- mark. Association for Computational Linguistics.
Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Michael Rabbat, and Ari Morcos. 2022. fo-fi: distributed fine-tuning without communication. ArXiv preprint, abs/2210.11 1994..
John Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Dur- rani, Fahim Dalvi, and James Glass. 2020. Similar- ity analysis of contextual word representation mod- els. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4638 4655, Online, Association for Computational Linguistics.

---

A Empirical Analysis
Additionally, we provide results on more datasets and models in Figure 11, Figure 12, and Figure 15. While we employ weighted mean pool- ing to obtain sentence representations, we also in- clude results using mean pooling in Figure 13 and Figure 14.
A.1 Representation Similarity Details
To compute the similarity of representations be- tween pairs of layers within LLMs. we employ the technique known as Centered Kernel Analysis (CKA), as introduced by Kornblith et al. (2019). For every layer within LLMs, the hidden state out- puts encompass characteristics derived from the input tokens with the shape of (N, L, II,, denoted as hg. Here, N signifies the batch size, L denotes the input length, and H represents the hidden size. After getting features of all imput tokens, a pool- ing strategy used to get sentence embedding for similarity computing, where max pooling is taken. We evaluate CKA between each pair of layers in the LLM to be compared. For layer and j. which denoted as and sj. both with the shape of (N, H). the linear CKA is given by:
A.2 Functional Similarity Details
Logits-Lens was proposed by (nostalgehraist, 2020) as a method to gain insights into the internal work- ings of GPT-2, with a specific focus on analyzing the logits, which represent the raw outputs gen- erated by the model before applying the softmax function to obtain probabilities. Logits-Lens aims to examine the logits various layers of GPT mod- els in order enhance understanding of prediction process LLMs. Through the inspection of these logits, researchers and developers can gain valuable insights into decision- makinn process of LLMs and potentially discover underlying patterns or biases. Due to the instability of Logits-Lens which fails to function effectively in larger and deeper mod- cls, (Belrose al., 2023) introduces Tuned-Lens. where each layer of the model is trained using an affine transformation on a pre-training corpus.
5 1
CKA(s;:
|15 sile
where || 1| denotes the Frohenius norm.
In contrast to BERT-style models, which incorpo- rate CLS token for sentence embedding, GPT-style models focus solely on preceding tokens. There- fore, the last token within GPT models often holds the most significant semantic representation. Nev- ertheless, according to the findings in the experi- mental study by Muennighoff (2022), employing the weighted mean pooling strategy, which assigns more importance to later tokens, leads to supe- rior results in sentence-level semantic expression and ensures stability even as the depth of layers increases. Consequently,
B Main Experimental Details
B.1 Dataset Details
We present statistical results of both downstream tasks (target tasks) and support tasks in Table 4.
B.2 Implementation Details
For empirical analysis, we randomly seleet 512 samples from the evaluation set each dataset for CKA computation. Additionally, we perform the clustering step by considering only layers be- tween the lth bottom layers and the last rth top lay- ers, taking into account the coarse changes among them. However, we ensure that the layers of the emulator remain the same as in Offsite- Tunning. We perform a learning rate tuning process on a grid of values and report the runs with the high- est emulator performance, where {le - 4,2c 4, 3e - 4, 2e - 5, 5e 5, Se 5} for OPT-1.3B
adopt the weighted mean pooling strategy to obtain the sentence vector within the hidden layers:
L. wp hg, W/
i
R
1==1
where h3 is the ktth hidden state and R mcans the sentence embedding of ith layer.
We organize the source and target text samples into an instruction format, as illustrated in Table 5 and Table 6.
and (5c 6, 8c 6. 1c 5. 2c 5, 5e 5] for U.T-6.7B and LLaMA-7B. We also select the best adapter layers in both settings, with and without sharing, indicating that the sharing step an op- tional component for local clients. Furthermore, the decision to repeat fine-tuned layers is dependent on the available local resources. The experiments
For CKA, representation similarity is computed across all input samples. Due computational constraints, we use set of 512 samples for com- putation, which has been found to be sufficient and stable for analyzing the modular structure.

---

Task
Multi-Choiee Choti-Choice QA
Closed-Book QA
SentComp
Dutaset
OpenBookQQ Sci.Edu
PIQA Physical
SeiQ Sed
RACE Exam
BoolQ
ARC-E
ARC-C
WehQu
TrivieQA Gen.
HellaSwag
CoPA
Gen.
Train.Size
4,957
11,679
62,445
3,589
87.622
39,905
Size
500
1,838
1,000
3,451
3,270
570
299
189
11,313
10,042
100
Test.Size
500
3,084
1,000
3,498
3,245
2,376
1,172
2,032
10,832
10,003
Avg.Come Cog.entt
13.10
110.27
143.96
49.18
Avg.Tareet
4.87
24.28
3.68
8.68
2.0
5.93
7.28
4.24
4.48
29.52
7.34
AvgaTotal
17.97
38.64
113.95
416.58
145.96
33.95
38.43
19.78
28.71
78.69
16.03
Table 4: Statistics are collected for all datasets, with the last one of each task serving as a support task for clustering (e.g. BoolQ, TriviaQA, and CoPA). Abbreviations: SeniComp = Sentence Completion, Sci.Edu Science Enducation, Edu..xxmm Enducation and Examination, Gen. = General domains.
clustering results tend to stabilize as the data size increases up 128. Based these findings, down- stream users can now upload only small amount of data relevant to their task, resulting in improved clustering.
2aker des
123
C.2 Combined with parameter-efficient tuning
16/10
Figure This figure presents clustering results on OPT: 1.3B with the Wikitext Corpus, showcasing the influence of varying example numhers (8 to 1024). Adjacent layers within the same cluster are color-coded (red or blue). Emulator layers are marked with and v, where the latter indicates lcarnable.
OFT is independent of parameter-efficient tuning methods, including Adapter (Houlsby et al., 2019), BitFit (Ben Zaken et al., 2022), and LoRA (Hu et al., 2022). For example, in the case of LoRA, only 590K parameters need to be updated for OPT- 1.3B, compared to 201M for OFT and 1208M for the full model. This demonstrates the parameter- and ssource-efficiennyy achieved through OFT. Based on results presented in Figure 8, we observe that CRaSh is also compatible with LoRA and ex- hibits superior performance. Nevertheless, there is still noticeable performance decline various datasets that requires further exploration in future studies.
are conducted using the NVIDIA RTX 3090 GPUs for OPT-1.3B and A6000 GPUs for OPT-6.7B and LLaMA-7B.
Additional Experiments and Analysis
C.1 Impact of data size for clustering
As illustrated Section. 3.2, the CKA metric used in clustering step is computed based on the similar- ity of representation in intermediate layers.
C.3 Sharing among different layers
Layer sharing is a beneficial approach to en- hance model capacity when faced with limited re- sources (Lan al., 2020). The downstream user has the option to either share weights among layers or not, depending on the emulator's performance. In order to investigate the impact weight sharing across different layers, we substitute each layer LLMs with another layer. As illustrated in Figure 9, the heatmap above displays the loss on the valida- tion dataset when replacing layer with layer j. The diagonal represents the original LLMs, where the loss the lowest, and locations near the di- agonal also exhibit low loss. Additionally, in the heatmap below, we compare the representations from the last layer of the original LLM and the layer sharing models, which demonstrates a similar
According to previous studies (Wu et al., 2020; Csiszárik al., 2021; Phang et al., 2021), increas- ing the amount of data used results in higher accu- racy for the CKA metric. However, due to limited computing resources, it is challenging to load all examples into memory once for computation. To reduce memory consumption during computation, Nguyen et al. (2021) proposed computing linear CKA by averaging HSIC scores based on mini- batches. However, being limited with computing resource, it's hard feed all examples into mem- ory for computing. In contrast to their approach, we focus on the clustering process itself and ana- lyze the results using varying sizes of examples, as shown in Figure 7. Our findings indicate that the

---

efficient fine-tuning, Houlsby et al. (2019) intro- duced task-specific modules called adapters, such as bottleneck networks, into LLMs. Taking into account the intrinsic space of LLMs (Aghajanyan et al., 2021), LoRA (Hu et al., 2022) optimizes LLMs in low-rank space, where additional ma- trices are updated and can be directly added to the original parameter matrix. Except for inserting new parameters, other approaches, BitFit (Ben Za- ken et al., 2022) and skill neurons (Wang et al., 2022a), aim to identify task-related parameters within LLMs and update them to incorporate task- specific knowledge while keeping the remaining model parameters frozen. Another approach is black-box tuning (BBT) (Sun et al., 2022b,a), which involves LLMs that are inaccessible to users. In this method, the continuous prompt prepended to the input text optimized using derivative-free optimization techniques. However, in these meth- ods, the full models are required to participate forward propagation, and they do not provide any assistance in protecting the privacy of LLMs and data. Another approach is resource-efficient fine- tuning, which involves evolving only a subset of parameters during the fine-tuning process (Sung et al., 2022; Chen et al., 2023). These methods solely focus on transferring knowledge in one di- rection to downstream tasks and do not facilitate the transfer updated weights back to the original model.
Model
60
Full
LoRA
Uniform+LoRA
GRash
Plug Accuarty
CRahh LoRA
30-
10
openbookga
Art-eaxy
art-chullenge
mebb-quesiion
Figure 8: The table presents the plug-in accuracy for four datasets when combining LoRA with different strategies. It evident that CRaSh performs well with LoRA compared to the uniform strategy. Although LoRA significantly reduces the number learnable pa- rameters by optimizing the model in a low-rank space, it negatively impacts performance in almost all datasets.
phenomenon. These analyses collectively indicate that weights sharing between adjacent layers can maintain comparable performance to the primary model. This finding supports the effectiveness of the layer sharing steps in CRaSh.
C.4 Comparing layer clustering to module cruciality
As demonstrated in Figure 10, showcase mod- ule cruciality through the process of rewinding and removing layers. The findings reveal that the ma- jority of important neurons are concentrated in the middle and high layers of the model. This sup- ports our decision to uniformly select learnable layers. However, further investigation is required for a more comprehensive understanding of this phenomenon.
Fine-tuning without using full models is moti- vated by the need for privacy protection, which is not adequately addressed by current transfer learning methods. Federated learning (FL) (McMa- han et al., 2017) involves distinguishing server and client models, where training is performed locally on the client and then aggregated. However, the client and server retain the same model, which com- promises the privacy of LLMs. Another approach decoupled learning, which decomposes the end- to-end optimization problem of neural training into smaller subproblems. This objective is achieved through various techniques, such as distributed gra- dient descent (Huo et al., 2018; Xu et al., 2020), model assembly (Ni et al., 2023), branch-train- interpolation (Li et al., 2022b; Wortsman et al., 2022), and collaborative development of machine learning (Raffel, 2023; Kandpal et al., 2023). How- ever, these methods primarily focus on training from the beginning rather than during the fine- tuning phase or risk leaking the full model. To ad-
Related Works
In this section, we first review the adaptation meth- ods for LLMs, then introduce technique called fine-tuning without full models. After that, we discuss the compression methods for LLMs. Fi- nally, we present similarity methods for interpret- ing LLMs to enhance adaptation.
Parameter- and resource-efficiene fine-tuning has emerged a popular method for adapting LLMs. Instead of directly updating pre-trained models for downstream tasks, these methods focus on updating or adding a minimal number of pa- rameters, sometimes not requiring the involvement of the entire parameter set. Regarding parameter-

---

arc_challenge
arc_easy
openbookga
web__quettion
69-33
LAM
Figure 9: Analysis layer sharing. (i,j) means replacing layer with layer j. i.e. sharing layer j with layer i. Above: loss on validation set. Below: representation similarity compared to last layer.
dress this issue, Xiao et al. (2023) propose Offsite-
Tuning, novel approach that involves transferring compressed versions of LLMs to downstream tasks and performing fine-tuning on privacy-sensitive user data. The updated weights are then transferred back and incorporated into LLMs. This method shows promise for protecting the privacy of LLMs and user data. This work is highly relevant to our research, and we propose a training-free strategy to enhance offsite-tuning by leveraging the similari- ties among neural networks.
4.50
4.99
Compression methods for LLMs To address resource-constrained scenarios, various technolo- gies such as knowledge distillation (Hinton et al., 2015; Sun et al., 2019; Gordon et al., 2020), prun- ing (Chen et al., 2020; Liu et al., 2022), and quan- tization (Tao et al., 2022) have been employed in pre-trained models such as BERT, T5, and GPT-2. However, with the increasing size of LLMs, such GPT-3 which consists of 96 layers, knowledge dis- tillation becomes challenging. With the increasing popularity of post-quantization, methods such as GPTQ (Frantar et al., 2023) and QLoRA (Dettmers et al., 2023) have emerged to reduce the computa- tional resources required by LLMs. However, these methods still encounter challenges related to loss of precision and model Icakage, posing threats the privacy and safety of LLMs. SparseGPT (Fran- tar and Alistarh, 2023) demonstrated, for the first time, the possibility of pruning large-scale gener- ative pretrained transformer (GPT) family models to achieve at least 50% sparsity in a single opera-
Figure 10: Given a full finetuned model, we evaluate specific layer importance via rewinding weights of layer to initialization (above) and directly removing layer from finetuned model (below).

---

tion. In comparison the aforementioned methods, LayerDrop is an effective technique for reducing the parameters of LLMs while maintaining com- parable performance. Fan et al. (2020) explored the use of LayerDrop to reduce the depth of trans- formers, but this approach relies on a specialized training strategy. Building on this, Zhang and He (2020) proposed the concept of layer drop ex- pedite training. These studies primarily focus on training with layer drop during the training pro- cess rather than during post-training Sajjad al. (2023) investigated various strategies for imple- menting layer drop on pre-trained models such as BERT, XLNet, and GPT-2, with particular empha- sis on the application of heuristic rules. Building upon the concept of "lottery tickets" (Frankle and Carbin, 2019), which involves discovering sparse subnetworks within trained dense networks that can achieve comparable accuracy, Jordao al. (2023) confirmed the presence of winning tickets when layers are pruned in vision models. Insufficient exploration has been conducted on the application of layer dropping in large-depth neural networks such as LLMs.
pre-trained models, comparing them within various model families (Wu al., 2020), and have inves- tigated the dynamics of embeddings during fine- tuning (Merchant et al., 2020). Phang et al. (2021) discovered the clustering of layer representations in fine-tuned transformers, providing evidence for the presence of layer redundancy in LLMs (Dalvi et al., 2020). As the size increases, observe that clustering also occurs in original pre-trained mod- els, not just in specialized models. Recognizing the wide applicability of this concept in various fields, recent studies have aimed to gain insights into the functional aspects of LLMs. nostalgebraist (2020) directly mapped the hidden states of intermediate layers to the final classification layer to obtain hid- den predictions. Considering the phenomenon of representation drift within hidden layers, especially in lower layers, Belrose et al. (2023) propose tuned- lens, a method that trains projections for each layer using a small amount of pre-trained corpus. These methods indicate that the functional behaviors of hidden layers also undergo subtle changes as the depth of the layers increases. Building upon these observations, Merullo et al. (2023) discovered that language models implement simple word2vec-style vector arithmetic (Mikolov al., 2013), which sheds light on the prediction process of LLMs. In- stead of using the transformation layer for decod- ing, Bills et al. (2023) attempted to interpret GPT-2 hidden states using GPT-4 (OpenAI, 2023), open- ing up the possibility of automatically aligning the behavior of models with AI.
Neural network similarity primarily encom- passes representation similarity and functional sim- ilarity. Representation similarity refers to the simi- larity of hidden states across all layers, while func- tional similarity indicates that two models exhibit the same behavior on the given inputs (Klabunde al., 2023). By measuring the similarity of neu- ral network representations, we can gain insights into the reasons behind variations in model be- havior. Kornblith et al. (2019) revisited previous methods for comparing neural network represen- tations and introduced centered kernel alignment (CKA), technique that measures the relationship between representations without being affected by high dimensionality or the number of data points. Extensive exploration has been conducted in the field of computer vision. comparison to CNN models, Raghu et al. (2021) discovered significant differences in the representation structure between Vision Transformers (ViTs) and convolutional net- works. VITs exhibit highly similar representations across all layers, whereas ResNet models demon- strate lower similarity between lower and higher layers, indicating the absence of block structure in the representation of transformer models. In the study of large language models, researchers have analyzed representation similarity across different

---

OpenBookQA
Which organism cannot specialize?
protozeu
person can grow cabbage in January with the help of what product? Green house
PIQA
Question: To fight Ivan Drago in Rocky for sega master system.
Answer:
You have to defeat Apollo Creed and Clubber Lang first.
Question: Make outdoor pillow.
Answer:
Blow into trash bag and tie with rubber band.
SciQ
A wetland is an area that is wet for all or part of the year. Wetlands are home to certain types of plants Question: What is an area of land called that is wet for all or part of the year?
Answer:
werland
Question: Surface waters are heated by the radiation from?
Answer:
the sun
RACE
Article: I am psychologist. first met Timothy, a quiet, overweight eleven-year-old buy, when his mother brought him to me to discuss his declining grades. A few minutes with Timothy were enough to confirm that his self-esteem and general happiness were falling right along with thirty so he could reach his school by cight and arrived home around four thirty each afternoon, Hc then had a quick snack, followed by either piano lesson or a lesson with his math tutor. He finished dinner at 7 pm, and then he sat down to do homework for two to three hours. Quickly doing the math in my head, I found that Timothy spent an average of thirteen hours day at a writing desk.
asked about Timothy's 's typical day. He awoke every morning at six
What if Timothy spent thirteen hours a day at sewing machine instead of a desk? We would immediately be shocked, because that would be called children being horribly mistreated. Timothy was far from being mistreated, but the mountain of homework he faced daily resulted in similar consequence -he was being robbed of his childhood. In fact. Timothy had time to do anything he truly enjoyed, such as playing video games, watching movies, playing bourd games with his friends. Play. however, is a crucial part of healthy child development. It affects children's 's creativity, their social skills, and even their brain development. The absence of play, physical exercise, and freefrom social interaction takes a serious toll many children. It can also cause significant health problems like childhood obesity, sleep problems and depression.
Experts in the field recommend the minutes children spend on their homework should be more than ten times the number of their grade level. As fifthgrader, Timothy should have more than fifty minutes day of homework (instead of three times that amount). Having extra two hours an evening play, relax, see friend would soundly benefit any child's 's life quality.
Question: According to the passage, how long should thirdgrader spend day doing homework?
Answer:
No more than thirty minutes.
BoolQ <support set>
Phantom pain- Phantom pain sensations are described as perceptions that an individual experiences relating to limb an organ that is not physically part the body. Limb loss result either removal by amputation or congenital limb deficiency. However, phantom limh sensations can also occur following nerve avulsion or spinal cord injury. Question: is pain experienced a missing body part or paralyzed area? Answer: yes
Table 5: Instructions format of Multi-Choice QA task.

---

ARC-E
Question:
Which technology was developed most recently?
Answer:
cellular telephone
Question: A student hypothesizes that algae are producers. Which question will best help the student ceetermine if this
conrect?
Answer:
Do algae use sunlight to make food?
ARC-C
Question: Juan and LaKeisha roll a few objects down a ramp. They want to see which object rolls the farthest. What should they do they can repeat their investigation? Answer:
Record the details of the investigation.
Question: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense.
What will most likely result if a high-pressure system remains in an area for long period of time? Answer: drought
WebQS
Question: what is the name of justin bieber brother?
Answer:
Jazmyn Bieber
Question: what character did natalie portman play in star wars?
Answer:
Padmé Amidala
TriviaQA <support
Question:
What star sign is Jamie Lee Curtis?
Answer:
Scorpio
Question:
Which Lloyd Webber musical premiered in the on 10th December 1993?
Answer:
Sunset Boulevard
HellaSwag
Roof shingle removal: A man is sitting on roof. He starts pulling up roofing on roof.
Clean and jerk: A lady walks to harhell. She bends down and grabs the pole. The lady stands and lifts the weight over her head.
CoPA csupport set>
The man turned on the faucet therefore water flowed from the spout.
The girl found bug her cereal therefore she lost her appetite.
Table Instructions format of Closed- -Book QA and Sentece Completion task.

---

1,0
125m, openbookqa
250m, openbookqa
1.3b, openbookqa
6.7b, apenbookqa
13b. openbookqa
30b, apenbookqa
16 24
10
36
225m, ch_cchlllegeeeee 350m, arc_ challenge 2.3b, arc_ challenge
28.7b, arc_ challenne
13b, arc.arccccao challeege
30b, arc_ PCC_chalengge
0.8
2
26.7b questinsssnnsss 13b, web_questtion
125m, reb__qusttiosee 950m, web_questtonss... questtons
30b, web__qesttions
56
0.7
15
12
125m. piqa
350m, piqa
1.3b, piga
6.7b, piqa
13b. piqa
30b, piqa
0.6
X
2
125m, sciq
350m, sciq
1.3b, sciq
6.7b, sciq
13b, sciq
30b, sciq
24
X
0
0.4
125m, race
350m, race
1.3b, race
6.7b, race
13b, race
30b, race
1
16
X
0.3
2
12
10
20
18
24
32
10
20
30
36
48
125m, hellaswag
350m, hellaswag
1.3b, hellaswag
6.7b, hellaswag
13b, hellaswag
30b, hellaswag
0.2
24
36
21
3
6
5
12
10 20
12
28
24
10
20
30
36
48
Figure 11: The table displays the representation similarity among layers of OPT models (Zhang et al., 2022) with different sizes, across five datasets with weighted mean pooling strategy. The rows the table represent the layer similarity of each dataset across the 125M, 1.3B, 6.7B, and 13B OPT models. Notably, the characteristies of the Wikitext dataset differ from the rest because it a language modeling task similar to the pre-training objective. (Phang al.. 2021) introduced that fine-tuned transformers exhibit clusters of similar representations across layers, which explains the emerging block structure of the Wikitext dataset even in small language models. In contrast, the remaining four datasets are associated with question answering tasks, which pose challenges smaller models. Consequently, the representation of these tasks demonstrates abnormal similarity in the intermediate layers of the 125M OPT model, suggesting relatively lower level of comprehension in it.

---

1.0
125m, openbookqa
1.0
350m, openbookga
1.0
1.3b, openbookqa
1.0
6.7b. openbookqa
1.0
13b. openbookga
1.0
306. openbookqa
Similarity
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.2
9
12
0.2
10 20
0.2
0
6
12
24
0.2
0.2g
40
0.2
36
48
1.m.5m are challenge, 1.0 arc challenge,
1.3b. arc challenge
6.7b,
moc_chaleenge
13b, arc challenge
30b, arc challenge
1.0
1.0
1.0
1.0
Simiarisy
0.8
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.2
9
12
0.2
10 20
0.2
0
6
12
18
24
0.2
0.2
40
0.2
36
48
125m, web questions, 1,500, web questions
1.30, web questiosssnsss
6.7b, web questions
13b, web questions
30b, quesstioss
1.0
1.0
1.0
1.0
Simllrtty
0.8
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.2
0.2
0.2 0
0.2
0.2
0
3
6
9
12
10 15 20
6 12
18
24
8
16
24
32
10 20
30
40
12
24
36
48
125m, piga
350m, piga
1.3b, piqa
6.7b, piga
13b. piqa
30b,
1.0
1.0
1.0
1.0
Similatty
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.2 0
3
6
9
12
0.2
0
10 15 20
0.2 0
6
12
18
24
0.2
8
16
24
32
0.2
10 20
30
40
0.2
12
24
36
48
sciq
sciq
13b. sclq
306,
1.0
1.0
1.0
1.0
Simlarity
0.8
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.2 0
3
6
9
12
0.2
0
10
15 20
0.2 0
6
12
18
24
0.2
0
8
16
24
32
0.2
10
20
30
40
0.2
12
24
36
48
1.30,
6.7b.
1.0
1.0
1.0
Similrit
0.8
0.8
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0
3
6
9
12
0.2
0
10 15
20
0.2 0
6
12
18
24
0.2
0
8
16
24
32
0.2
10
20
30
40
12
24
36
48
hellaswag
hellaswag
hellaswag
hellaswag
hellaswag
30b,
1.0
1.0
Simllaiiy
0.8
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0
3
6 Layer
9
12
0.2
0
10 15 Layer
20
0.2
0
6
12 Layer
18
24
0.2
8
16 Layer
24
32
0.2
10
20 Layer
30
40
12 24
36 Layer
48
Figure 12: The table displays the representation similarity among layers of OPT models (Zhang al., 2022) with different sizes, across five datasets with weighted mean pooling strategy.

---

1.0
125mm openbookga
350m, openbookqa
1.3b, openbookqa
66.7b, openbookga
13b, openbookqa
30b, openbookqa
18
12
8
24
6
10
6
12
5
10
15
20
18
24
8
16
24
20
30
12
36
48
225m, arc_challenge 350m, arc_ challenge J.3b, arc challenge
6.7bb arc arC_challenge
13b, arc challenge
300, arc cic_chaleenge
0.8
10
3
6
12
5
30
15
20
18
24
8
16
24
32
20
30
40
12
36
48
125m, web_ pel_gquestionsaae web questions] 3b, web_ reeb_qustions 6.7b. web questions
J3b, geeb_questions
20b. web
wttpwwe__eueee questions
30
30
20
10
10
6
12
5
10
15
20
0
12
18
24
11
16
24
32
10
20
30
40
12
24
36
40
125m, piqa
350m, piqa
1.3b, piqa
6.7b, piqa
13b. piqa
30b, piqa
01
8
12
5
10
15
20
32
18
24
8
1.6
24
32
10
20
30
40
12
24
36
40
125m, sciq
350m, scig
1.3b, sciq
6.7b, sclq
13b, sciq
30b, sciq
408
R.
0.4
12
5
10
20
18
16
20
30
12
36
48
125m,
350m, race
1.3b,
6.7b,
13b,
30b,
12
20
20
50
45
m25m. hellaswag
350m, hellaswag
hella
6.7b, hellaswag
hellaswag
30b, hellaswag
24
18
12
10
12
20
20
40
Figure 13: The table displays the representation similarity among layers of OPT models (Zhang et al., 2022) with different sizes, across five datasets with simple mean pooling strategy.

---

125m, openbookqa
1.0
350m, openbookga
1.3b, openbookqa
6.7b. openbookqa
13b. openbookga
306. openbookqa
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.8
Similarit
0.6
0.6
0.6
0.6
0.6
0.4
12
0.2
10
20
0.2
0
6
12
24
0.2
0.2
40
0.2
36
48
125m, arc chw.lehaaeee.ccooo 1.0
arc challenge,
1.3b. arc challenge
6.7b,
ch_challeenge
13b, challenge
30b, challenge
1.0
1.0
1.0
1.0
1.0
Simiarisy
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.2
9
12
0.2
10
20
0.2
0
6
12
18
24
0.2
0.2
40
0.2
36
48
125m, web questions, 1,500, web questions
1.30, web questioss
6.7b. web questions
130, web questions
30b, questiinss
1.0
1.0
1.0
1.0
Simllntty
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2 0
0.2
0.2
0.2
0
3
6
9
12
10
15 20
6 12
18
24
8
16 24
32
10
20
30
40
12
24
36
48
125m, piga
350m, piga
1.3b, piga
6.7b, piga
13b.
30b,
1.0
1.0
1.0
1.0
Similntty
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2 0
3
6
9
12
0.2
0
10 15 20
0.2 0
6
12
18
24
0.2
8
16
24
32
0.2
10 20
30
40
0.2
12
24
36
48
sciq
sciq
13b. sciq
306.
1.0
1.0
1.0
1.0
Simlarity
0.8
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0
3
6
9
12
0.2
0
10
15 20
0.2 0
6
12
18
24
0.2
0
8
16
24
32
0.2
10
20
30
40
0.2
12
24
36
48
1.30.
6.7b.
1.0
1.0
1.0
1.0
1.0
Similrit
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0
3
6
9
12
0.2
0
10
15
20
0.2 0
6
12
18
24
0.2
0
8
16
24
32
0.2
10
20
30
40
0.2
12
24
36
48
hellaswag
hellaswag
hellaswag
hellaswag
hellaswag
30b, hellaswag
1.0
1.0
1.0
Smilarity
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0
3
6 Layer
9
12
0.2
0
10 Layer
15
20
0.2
0
6
12 Layer
18
24
0.2
0
8 16 Layer
24
32
0.2
10 20 Layer
30
40
0.2
12 24
36 Layer
48
Figure 14: The table displays the representation similarity among layers of OPT models (Zhang al., 2022) with different sizes, across five datasets with simple mean pooling strategy.

---

7b, wikitext
13b, wikkext
30b, wikitext
wikitext
wikktext
wikitext
5
8
R
X
8
15
15
24
22
10
20
45
10
7b, cayy
easy
arc_casy
eac_
30b, are_eaty
0.8-
10
7b, challenge
J.3b, challenge
30b, challenge
7b. arc challenge
130. challenge
30b, challenge
30
10
0.6
7b, piga
13b, piqa
30b, piga
7b, piga
13b, piça
30b, piça
20
20
30
10
24
10
7b, sciq
13b, sciq
30b, scia
7b, sciq
13b, scio
30b, sciq
20
0.4
20
0.4
10
15
24
72
0
10
20
22
45
59
15
24
10
45
60
7b, race
7b, race
13b, race
300, race
8
R
0.2
0.2
14
24
10
40
30
60
7b, hellaswag
13b. hellaswag
30b, hellaswag
7b. hellaswwg
13b. hellaswag
hellaswag
10
14
10
(a) Weighted mean pooling on LLAMA.
(b) Mean pooling on LLAMA.
Figure 15: This table illustrates the presence of modular structure in LLAMA models. We observe that weighted mean pooling performs well on LLAMA models, while direct mean pooling may lead to information loss and abnormal structures, particularly the case of 30B models.


Reviews:
[{"review_id": "CBirfEvNTu", "reviewer": "Reviewer_CWER", "summary": "", "questions": "See reasons to reject.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive feedback and positive remarks we have received. We welcome the chance to delve deeper into discussions with the reviewers concerning the propensity of LLM providers to distribute compressed emulators and the practicality of restoring the original model from such emulators.\n\n> This is not necessarily a reason to reject: the paper assumes that LLM providers (e.g. OpenAI) would be willing to share the compressed emulators. However, it does not provide a compelling argument or incentive for why they would do so, if they do not want to share the entire model. \n\nOffering compressed emulators bridges the gap between safeguarding proprietary LLMs and ensuring data privacy, a compelling compromise LLM providers might find beneficial.\n\nDespite the acknowledged capabilities of models like ChatGPT and GPT-4 across diverse tasks, there are specific domains and particularly complex tasks where their effectiveness is constrained. In such instances, in-context learning and prompting techniques occasionally do not meet expectations, underscoring the importance of fine-tuning with specialized datasets. Notably, OpenAI acknowledged this limitation and rolled out a fine-tuning API for `gpt-3.5-turbo`, enabling users to harness their distinct datasets to achieve superior performance.\n\nYet, with the increasing adoption of such APIs comes heightened concerns about data privacy. These concerns could potentially negate the commercial advantages sought by LLM providers. The proposed compressed emulators stand as an exemplary solution, striking a balance between retaining the proprietary nature of the LLMs and addressing data privacy issues. \n\nWe are confident that OFT and our newly introduced CRaSh, offer a novel resolution to the pressing privacy challenges faced by LLM providers. \nFurthermore, as the domain advances, we anticipate the development of even more innovative solutions beyond CRaSh and OFT.\n\n> ... warrants consideration: given the emulators, how challenging would it be to reconstruct the original model?\n\nYour question about model attack and defense is certainly thought-provoking. The central issue concerns the level of performance that can be achieved using only the emulator. Our findings in the manuscript delve deeper into this topic, preliminarily outlining the challenges associated with reconstructing the original model.\n\nFirstly, we regard 'layer sharing' in CRaSh as the most promising method for reconstruction. However, it cannot completely match the performance of the original model. As shown in Table 2, the emulator fine-tuned with CRaSh falls short of the full model's zero-shot performance (denoted as 'Full ZS'). Nevertheless, after plugged in, there is a significant improvement in performance across several datasets, specifically ARC-E/C, PIQA, RACE, and HellaSwag.\n\nSecondly, a pivotal aspect of the challenge is dependent on the number of layers transferred; transferring fewer layers increases the complexity of reconstruction.  As illustrated in Figure 5, accuracy fluctuates with the changing number of layers dropped.  Notably, emulators with just 6 layers (2 frozen and 4 fine-tuned, relative to the 32 layers in the full model) continue to enhance the performance of the primary LLM when plugged-in. \n\nCertainly, techniques like OFT and CRaSh could benefit from the integration of federated learning technologies, such as homomorphic encryption and differential privacy, making the reconstruction of the original model more challenging. We look forward to more research addressing these open questions and intend to include these discussions in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper conducts an in-depth empirical analysis of LLMs from the perspectives of representation and functional similarity. This analysis provides valuable insights into the underlying mechanisms of OFT.\n2. The proposed CRaSh strategy is novel and interesting, and the authors present compelling evidence of its effectiveness in improving the performance of OFT with billions of parameters.\n3. The paper's exploration of privacy concerns in the context of tuning publicly accessible, centralized LLMs with private instruction data is an important and interesting topic.\n4. The paper is well-structured and clearly presents complex concepts, making it accessible to readers with varying levels of familiarity with the topic.\n", "reasons_to_reject": "This is not necessarily a reason to reject: the paper assumes that LLM providers (e.g. OpenAI) would be willing to share the compressed emulators. However, it does not provide a compelling argument or incentive for why they would do so, if they do not want to share the entire model. Actually, another intriguing question emerges, which, while perhaps extending beyond the scope of this paper, warrants consideration: given the emulators, how challenging would it be to reconstruct the original model?", "questions_to_authors": "See reasons to reject.", "paper_topic_and_main_contributions": "This paper explores the implications and effectiveness of Offsite-Tuning (OFT), a technique for transferring transformer blocks between centralized large language models (LLMs) and downstream emulators, particularly in the context of privacy concerns related to the use of private instruction data. The authors conduct an empirical analysis of LLMs from the perspectives of representation and functional similarity, revealing a unique modular structure within LLM layers that emerges as the model size expands. They also note subtle yet potentially significant changes in representation and intermediate predictions across layers. Based on these observations, they propose a training-free strategy, CRaSh (Clustering, Removing, and Sharing), to derive improved emulators from LLMs. They demonstrate that CRaSh significantly enhances the performance of OFT with billions of parameters. Additionally, they examine the optimal solutions yielded by fine-tuning with and without the full model through the lens of the loss landscape, demonstrating a linear connectivity among these optima.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PvjrKn7Exy", "reviewer": "Reviewer_ZSZD", "summary": "", "questions": "- Across various baseline methods and the proposed CRaSh method, can you provide comparison in computation budget such as overall GPU-hours or number of updates for obtaining downstream models?\n- It is very interesting that just plug in fine-tuned emulator into centralized model can lead to better performance. Can you provide performance of emulator before and after plugged into the original LLM?\n- It is possible to use clustering and removing part to just \"prune\" some layers in the LLM. Then, we can apply some existing baseline methods such as OFT to pruned LLM to obtain the model for downstream task. Can you provide some results about this \"pruned\" LLM model? ", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your positive feedback. Below, we have paraphrased your comments as we've interpreted them and provided our responses. Kindly inform us of any misinterpretations or if further clarifications are required.\n\n**Q1: Can our proposed CRaSh method reduce inference cost and be effective in the long run?**\n\nPerhaps, our proposed CRaSh method might indeed have the potential to reduce inference costs and maintain its effectiveness over time.\nWe are looking forward with anticipation to further research and optimization of this method, hoping to obtain more definitive answers in future experiments.\n\nWe initially present the motivation behind CRaSh, drawing inspiration from three findings centered on the functional and representational similarity of LLMs. These insights into the redundancy of LLMs can enhance our understanding of them and pave the way for future research.\nRegarding our proposed CRaSh, it stands as an efficacious strategy for deriving compact emulators from LLMs. \n\nBased on recent research, we believe that CRaSh holds significant promise in diverse inference scenarios beyond efficient and privacy-centric fine-tuning.\n\n- Context compression. \n  Regarding the instructed LLMs (ChatGPT, Vicuna, and LLaMA-2-Chat), long prompts not only increase inference costs but also introduce redundant computations for the same instructions.\n  Mu et al. (2023)[1], Chevalier et al. (2023)[2], and Ge et al. (2023)[3] investigate the compression of prompts/contexts into concise gist/memory/slot tokens (termed as soft prompts) to accelerate inference. However, they continue to employ LLMs of the same scale for compression.\n  Motivated by the success of CRaSh in emulating LLMs, we posit that emulators derived from CRaSh can serve as effective initializations for compact compressors, subsequently reducing inference costs in context compression scenarios. \n\n- Speculative decoding[4][5] has been introduced to reduce both end-to-end latency and computational demands. In this approach, a draft model performs approximate inference for \"easier\" steps, while a target model subsequently verifies and rectifies errors. \n  Given the pronounced functional and representational similarity between emulators and LLMs, we posit that emulators derived from CRaSh may serve as superior draft models, further aiding in the efficient inference of large models[6].\n\n- Model Pruning. \n  As outlined in Appendix D concerning LLM compression methods, layer pruning stands as a straightforward yet efficacious approach for model compression [7][8][9][10]. \n  However, there remains limited investigation into the application of layer dropping within LLMs. We posit that CRaSh might catalyze the development of more sophisticated strategies in the realm of model pruning.\n\n**Q2: Performance of emulator before and after plugged into the original LLM, and results of this \"pruned\" LLM model.**\n\nConsidering the intricate configuration of OFT, we wish to clarify the multiple performance metrics depicted in Figure 4(a) that could be a potential source of confusion for reviewers. \nFor the centered LLM, we present three performance measures: zero-shot (\"Full ZS\"), full fine-tuning (\"Full FT\"), and plug-in with the emulator (\"Plug-in\"). Regarding the emulator itself, we display both the zero-shot (\"Emulator ZS\") and the fine-tuned (\"Emulator FT\") performance metrics.\nThe prerequisites for implementing OFT are to ensure that the \"Plug-in\" outperforms both \"Emulator FT\" and \"Full ZS\". This necessitates both fine-tuning with downstream data and integration into the centered LLM. \nOur primary objective is to approximate \"Full FT\" through plug-in, especially in scenarios where the Full LLM cannot directly access downstream data.\n\nAs depicted in Table 1 and Table 2, the performance of the emulator, before and after plugged into the original LLM, corresponds to \"Emulator FT\" and \"Plug-in\", respectively.  Notably, the former consistently lags behind the latter, underscoring the validity of OFT. \nThe \"pruned\" LLM, in essence the emulator, displays its performance in both zero-shot and fine-tuning conditions.  While this \"pruned\" LLM falls short of the original LLM in a zero-shot setting, it surpasses the zero-shot LLM once fine-tuned in most scenarios.\nWe apologize for any confusion, and will incorporate additional details and context regarding OFT to ensure our work is more comprehensive.\n\n**Q3: Comparison in computation budget such as overall GPU-hours or number of updates for obtaining downstream models.**\n\nWe approach this issue from two perspectives: 1) obtaining initialized downstream emulators, and 2) obtaining fine-tuned downstream emulators.\n\n- Regarding 1), both the uniform strategy in OFT and our CRaSh are training-free. This contrasts with the knowledge distillation (KD) approach in OFT, which is pre-trained on the initial 30 chunks of The Pile corpus and is significantly more time-intensive than downstream fine-tuning. In comparison to the uniform strategy, CRaSh requires only additional inference on a minimum of 128 examples for clustering, as indicated in Figure 7.\n\n- Regarding 2), the number of updates is contingent upon the specific task and final performance, but generally, the relationship is KD < CRaSh ≈ uniform strategy. Indeed, CRaSh, benefiting from layer sharing, consumes the same GPT memory as the uniform baseline. While layer sharing renders CRaSh's forward computation less expedient compared to both uniform and KD, it simultaneously provides a robust initialization, facilitating model convergence in fewer steps.\nUsing OPT 1.3B on OpenbookQA as an illustrative example, we observe that the steps and accuracies for KD, uniform strategy, and CRaSh are approximately ~400 steps (acc=0.29), ~600 steps (acc=0.264), and ~520 steps (acc=0.302), respectively. \n\nWe appreciate your astute observation and will incorporate this detailed information into the revised manuscript.\n\n**Q4: Typos Grammar Style And Presentation Improvements**\n\nThank you for highlighting the incorrect bold formatting; we will address this in the revised manuscript.\n\n**References**\n\n[1] Jesse Mu, Xiang Lisa Li, and Noah Goodman. \"Learning to compress prompts with gist tokens.\"\n2023.\n\n[2] Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. \"Adapting language models to\ncompress contexts.\" 2023.\n\n[3] Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. \"In-context autoencoder for context\ncompression in a large language model.\" 2023.\n\n[4] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. \"Fast inference from transformers via speculative decoding.\" ICML 2023.\n\n[5] Chen, Charlie, et al. \"Accelerating large language model decoding with speculative sampling.\" 2023.\n\n[6] Miao, Xupeng, et al. \"SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification.\" 2023.\n\n[7] Angela Fan, Edouard Grave, and Armand Joulin. \"Reducing transformer depth on demand with structured dropout.\" ICLR 2020.\n\n[8] Minjia Zhang and Yuxiong He. \"Accelerating training of transformer-based language models with progressive layer dropping.\" NeurIPS 2020.\n\n[9] Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. \"On the effect of dropping layers of pre-trained transformer models.\" 2023.\n\n[10] Artur Jordao, George Correa de Araujo, Helena de Almeida Maia, and Helio Pedrini. \"When layers play the lottery, all tickets win at initialization.\" 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The empirical results is very surprising. CRaSh even wins full-finetuning for some datasets (Table 1 RACE, SciQ). \n- The analysis on redundancy in LLMs is interesting (Figure 1). \n- The idea is simple and intuitive.\n", "reasons_to_reject": "- The method can effectively reduce the number of parameters to train. However, it cannot reduce inference cost. Hence, I don't know if it will be very effective in the long run.\n- I want to see more analysis and ablation on the method. Please check questions.", "questions_to_authors": "- Across various baseline methods and the proposed CRaSh method, can you provide comparison in computation budget such as overall GPU-hours or number of updates for obtaining downstream models?\n- It is very interesting that just plug in fine-tuned emulator into centralized model can lead to better performance. Can you provide performance of emulator before and after plugged into the original LLM?\n- It is possible to use clustering and removing part to just \"prune\" some layers in the LLM. Then, we can apply some existing baseline methods such as OFT to pruned LLM to obtain the model for downstream task. Can you provide some results about this \"pruned\" LLM model? ", "paper_topic_and_main_contributions": "- This paper is following work of Offsite-Tuning (OFT) technique which efficiently transfers ability of centralized LLMs into smaller models (emulators) for downstream tasks without full access to centralized LLMs. \n- The authors find out that LLMs which have more than 1B parameters have modular structure within their layers.\n- The authors suggest a novel approach CRaSh which enhances the OFT by cleverly utilizing redundancy in layers.\n- The empirical results show that CRaSh consistently outperform baseline methods in various type of tasks and datasets.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "wrong bold in Table 1.\nplug-in (base) of uniform strategy has 21.4% performance which is smaller than 21.8 of emulator FT."}, "extra_text": {}}, {"review_id": "jNIPgEJcD2", "reviewer": "Reviewer_TamK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the thoughtful feedback requesting additional motivation and background on our use of Offsite-Tuning (OFT). In the revised manuscript, we will elaborate further in the introduction and background sections, as you've suggested. \n\nInitially, we delineate the reasons for the limited background presented in the current manuscript. Due to page constraints, our emphasis was on showcasing empirical results, analyses, and enhancements to underscore the overarching significance of our proposed CRaSH method (as detailed in our response to Q1 from Reviewer ZSZD).\n\nWe agree that incorporating more OFT details would make our innovations easier to interpret in context. To address this, we will add concise yet comprehensive coverage of OFT's core concepts and relevance to our method.  Although centered on OFT, we believe our empirical insights and techniques may generalize to other application scenarios as well.\n\nHere, we preliminarily clarify the motivation for basing the proposed approach on OFT, particularly in relation to the two aspects you emphasized.\n\n>  ... explicitly discussing the advantages of OFT as a privacy-preserving tuning approach\n\nThe primary concern of OFT is to maintain the privacy of both the LLM and user data. Specifically, the data owner is unable to share their labeled training data with the LLM owner, and vice versa, the LLM owner cannot share their model with the data owner.\nThe benefits of OFT encompass an innovative approach to ensuring data and model privacy without significant performance degradation, while also achieving parameter and resource efficiency.\n\nAs elucidated in \"Section 6 Related Work\", the strategy of directly transferring model parameters, as opposed to transferring data, serves to enhance the privacy protection of user data. This approach has been widely adopted in various domains, including federated learning[1], split learning[2], distributed gradient descent[3], branch-train-interpolation[4], and collaborative machine learning development[5]. Nevertheless, many of these methodologies either necessitate the sharing of the full models between servers and clients or mandate multiple rounds of communication to achieve comprehensive coverage. Such practices may inadvertently breach the privacy of centralized LLMs and introduce challenges, especially when implemented on LLMs with billions of parameters.\n\nIt's worth noting that OFT[6] offers a commendable solution to these issues by judiciously sharing specific layers of LLMs, with billions of parameters, between servers and clients in a single communication round.\nIn a manner analogous to model-sharing in federated learning, OFT safeguards user data privacy by distributing part of the model instead of transferring user data to the cloud. Concurrently, OFT circulates select layers of the centralized LLM (referred to as emulators) to downstream clients, rather than the full model, to preserve the privacy of the complete LLM. While there exist reservations about the potential reconstruction of the original model, as highlighted by Reviewer CWER, this risk can be mitigated by limiting the number of shared layers, making it a viable avenue for exploration.\n\nTo address the reviewer's suggestions, we will add a motivational paragraph highlighting OFT's well-aligned strengths to extend introduction, including preservation of model and data privacy. \n\n> ... how it enables analyzing and adapting large pretrained models without access to the full model\n\nOFT employs a substitute model known as the \"Emulator\", which is derived from the LLM using a layer dropping strategy. Subsequently, OFT optimizes specific parameters within the \"Emulator\", which comprises multiple layers. Finally, the optimized parameters are transferred and plugged into the LLM for inference. \n\nHere, we provide a comprehensive introduction accompanied by illustrative figures in the paper. As shown in Firgure 4(a), rather than fine-tuning directly on LLM (i.e., \"②Full Fine-tuning\") which leak both privacy of LLM and downstream data, OFT compress LLM into an small emulator, fine-tuning it with privacy data (i.e., \"③Emulator Fine-tuning\"), then transfering block weights and plug-in LLM for inference (i.e., \"④Plug-in\"). The objective of OFT is to optimize performance of plug-in to approximate to full fine-tuning on LLM, and subject to better than full zero-shot of LLM and fine-tuning on emulator, which ensure the effectiveness of OFT.\n\nWe will also provide more background details on OFT in \"Section 3.1 Revisit Offsite-Tuning\". We will summarize OFT's core concepts such as the adapter/emulator architecture and how it enables practical adaptation of large models without exposing full weights. \n\n----\n\nThe key to OFT lies in identifying an emulator that closely resembles the original LLM and subsequently performing fine-tuning to replace the LLM.\nDrawing from our findings on the functional and representational similarity of LLMs in \"Section 2 Empirical Analysis\", we propose CRaSh, which is designed to optimize layer dropping and yield superior sub-layer emulators from LLMs, as illustrated in Figure 4(b).\n\n-----\n\nWe believe these additions motivate and contextualize our use of OFT more clearly. By highlighting OFT's strengths, explaining how it adapts large models privately, and positioning our work as targeted innovations, we demonstrate deep understanding of OFT and reinforce how it provides an ideal aligned basis for our privacy-first approach.\n\nWe sincerely appreciate you taking the time to offer constructive feedback. Your expertise and guidance have been invaluable for improving the manuscript's clarity within space constraints. Please advise if you would like us to expand on any specific OFT aspects further. We are happy to provide any clarification needed to positively address your comments.\n\n**References**\n\n[1] Brendan McMahan, Eider Moore, et al. \"Communication-efficient learning of deep networks 898 from decentralized data.\" AISTATS 2017.\n\n[2] Chandra Thapa, Mahawaga Arachchige Pathum Chamikara, Seyit Camtepe, and Lichao Sun. \"Splitfed: When federated learning meets split learning.\"  AAAI 2022.\n\n[3] Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, and Gao Huang. \"Deep incubation: Training large models by divide-and-conquering.\" 2023.\n\n[4] Mitchell Wortsman, Suchin Gururangan, Shen Li, et al. \"lo-fi: distributed fine-tuning without communication.\" 2022.\n\n[5] Colin Raffel. \"Building machine learning models like open source software.\" Communications of the ACM 2023.\n\n[6] Guangxuan Xiao, Ji Lin, and Song Han. \"Offsite-tuning: Transfer learning without full model.\" 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides an in-depth empirical analysis of large language models (LLMs) from the perspective of representation and functional similarity. This sheds light on the internal workings of LLMs as model size increases.\n2. The paper proposes a practical training-free strategy called CRaSh that builds on these insights to improve Offsite Tuning, a method for fine-tuning LLMs without access to the full model.\n3. Experiments demonstrate CRaSh can boost Offsite Tuning performance across multiple datasets. The analysis of loss landscapes provides theoretical justification for why CRaSh works well.", "reasons_to_reject": "This paper builds heavily upon the Offsite-Tuning (OFT) method. However, the motivation for basing the proposed approach on OFT could be clarified further. In particular, explicitly discussing the advantages of OFT as a privacy-preserving tuning approach and how it enables analyzing and adapting large pretrained models without access to the full model would strengthen the paper. Additionally, providing more background details on OFT would make the current work more self-contained. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "Instruction tuning is an effective way to enhance the generalization ability of LLMs. However tuning publicly accessible, centralized LLMs with private data raises privacy concerns. The paper focuses on Offsite-Tuning (OFT), which transfers transformer blocks between LLMs and downstream emulators to address this issue. Through an empirical analysis of LLMs, the authors find a unique modular structure emerges within layers as model size expands, along with subtle representation and prediction changes across layers. Inspired by these observations, they propose CRaSh - a training-free strategy involving clustering, removing, and sharing to derive improved emulators from LLMs, significantly boosting OFT performance. Furthermore, analysis of the loss landscape shows fine-tuning optima with and without the full model exhibit linear connectivity within the same basin, highlighting the effectiveness of CRaSh and OFT.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Offsite-Tuning (OFT) transfers transformer blocks between LLMs and downstream emulators to address this issue of privacy when instruction tuning LLMs with private data. The authors conducted an empirical study of LLMs, focusing on their representation and functional similarity. They discovered that as the model size grows, a unique modular structure emerges within LLM layers. They proposed a training-free strategy, CRaSh (Clustering, Removing, and Sharing), to derive improved emulators from LLMs that significantly enhances  the performance of OFT with billions of parameters.


Pros:
- This paper analyzes LLMs from the perspective of representation and functional similarity, shedding light on how they work as model size increases.
- CRaSh is a novel and effective strategy for improving the performance of OFT with billions of parameters.
- They demonstrate empirically that CRaSh improves performance across multiple datasets.

Cons:
- The paper will improve by clearly motivating how OFT ensures privacy. Adding the answers provided by the authors during rebuttal will improve the paper.
Decision: Accept-Main