Venue: NAACL 2025

# KS-Lottery: Finding Certified Lottery Tickets for Multilingual Transfer in

Large Language Models

 Fei Yuan\({}^{1}\), Chang Ma\({}^{2}\), Shuai Yuan\({}^{1}\), Qiushi Sun\({}^{2}\), Lei Li\({}^{3}\)

\({}^{1}\) Shanghai Artificial Intelligence Laboratory

\({}^{2}\) The University of Hong Kong, \({}^{3}\) Carnegie Mellon University

yuanfei@ejlab.org.cn, cma@cs.hku.hk

syuanafconnect.ust.hk, qiushisun@u.nus.edu, leili@cs.cmu.edu

###### Abstract

The lottery ticket hypothesis posits the existence of "winning tickets" within a randomly initialized neural network. Do winning tickets exist for LLMs in fine-tuning scenarios? How can we find such winning tickets? In this paper, we propose KS-Lottery, a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. Our key idea is to use Kolmogorov-Smirnov Test to analyze the distribution shift of parameters before and after fine-tuning. We further theoretically prove that KS-Lottery can find the certified winning tickets in the embedding layer, fine-tuning on the found parameters is guaranteed to perform as well as full fine-tuning. Comparing KS-Lottery with other tuning algorithms on translation tasks, the experimental results show that KS-Lottery finds a much smaller set of parameters for fine-tuning while achieving the comparable performance as full fine-tuning LLM. Surprisingly, we find that fine-tuning 18 tokens' embedding of LLaMA suffices to reach the fine-tuning translation performance 1.

Footnote 1: [https://github.com/CONE-MT/KS-Lottery.git](https://github.com/CONE-MT/KS-Lottery.git)

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 9077–9090

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Can we find an ultra-small subset of a well-trained Large Language Model (LLM; Touvron et al., 2023, 2023, 2022; OpenAI, 2023; Chowdhery et al., 2022) such that fine-tuning these few parameters suffices to achieve the same performance as full tuning? The lottery tickets hypothesis Frankle and Carbin (2019) states that a small subnetwork (less than 10-20% of the whole model size), referred to as "winning tickets", in a large, randomly initialized neural network can achieve comparable performance to the original network with the same amount of training. Yet, the existence of winning tickets is not investigated for fine-tuning scenarios. Prior work Aghajanyan et al. (2021) presents evidence that there are a small number of additional parameters corresponding to an intrinsic dimension Li et al. (2018) on which fine-tuning leads to good performance. However, it remains an unsolved challenge to uncover such a small subset of fine-tuning efficient parameters within the _original_ model.

In this paper, we show that there exist key parameters - winning tickets, for transferring LLM to multiple new languages. As shown in Figure 1 (KS-Lottery), we found that as tuning as few as 18 (\(18/32000=0.0006\)) token embeddings of a well-trained LLM could achieve test performance comparable to full tuning on machine translation tasks. Based on the observation, we state the lottery ticket hypothesis for multilingual fine-tuning.

Generally, the fine-tuning process can be represented by a transition in \(M\) parameters from \(\mathbf{\theta}=[\theta_{0},\theta_{1},\cdots,\theta_{M}]\) to \(\widetilde{\mathbf{\theta}}=[\widetilde{\theta}_{0},\widetilde{\theta}_{1},\cdots, \widetilde{\theta}_{M}]\), where \(\mathbf{\theta}\) and \(\widetilde{\mathbf{\theta}}\) denote the sets of parameters that characterize the LLM \(f(\cdot,\mathbf{\theta})\) before and after the fine-tuning process, respectively. Typically, the entire set of parameters \(\mathbf{\theta}\) is adjusted during fine-tuning to enhance the model's ability to represent and learn the new tasks more effectively.

Figure 1: Existence of “winning tickets”. KS-Lottery identifies a small subset of embedding parameters of LLaMA-7B to maintain the translation performance of en\(\rightarrow\)ca on Flores-101.



The Fine-Tuning Lottery Ticket Hypothesis.n_A _pre-trained neural network contains a small subset of parameters (\(\widetilde{\boldsymbol{\theta}}^{D}=[\widetilde{\theta}_{0},\widetilde{\theta}_{1}, \cdots,\widetilde{\theta}_{D},\theta_{D+1},\cdots,\theta_{M}]\), where \(D\ll M\)) that is initialized such that--when fine-tuned in isolation--it can match the performance of full tuning._

In this work, we examine the fine-tuning lottery ticket hypothesis on LLMs in a multilingual transfer scenario, and explore the following inquiries:

* _Existence of Winning Tickets:_ Is it certain that every LLM in multilingual transfer encompasses a compact subset of winning tickets? And how to quickly identify the winning tickets?
* _Efficiency of Winning Tickets:_ How minimal can this subset be in terms of size?
* _Interpretability of Winning Tickets:_ Do these winning tickets reflect the unique architectural characteristics of the multilingual LLM?

Identifying Winning Tickets.We propose KS-Lottery, a method to identify a winning ticket by merely fine-tuning the embedding layer of an LLM. And then fine-tune these identified tickets, keeping the remaining parameters frozen. The whole KS-Lottery consists of three steps:

1. [leftmargin=*]
2. Fine-tuning the embedding layer of an LLM \(f(\cdot,\boldsymbol{\theta})\) to obtain \(f(\cdot,\widetilde{\boldsymbol{\theta}}^{D})\).
3. Run Kolmogorov-Smirnov Test between \(\boldsymbol{\theta}\) and \(\widetilde{\boldsymbol{\theta}}^{D}\) to select the winning ticket \(\widetilde{\boldsymbol{\theta}}^{D}_{a}\).
4. Tuning the \(\widetilde{\boldsymbol{\theta}}^{D}_{a}\) within \(\boldsymbol{\theta}\) on downstream tasks.

The core idea of our method is to heuristically identify parameters with large distribution changes before and after fine-tuning. Here we use Kolmogorov-Smirnov Test to determine whether two sample variables stem from the same underlying distribution. Also, we simplify and accelerate the Kolmogorov-Smirnov Test process by focusing on the embedding layer, which constitutes the major change in parameters, due to the inductive bias of multilingual tasks. To this end, our approach, KS-Lottery, is a surprisingly straightforward yet effective technique for pinpointing winning tickets in LLMs. The Kolmogorov-Smirnov Test has the advantage of not assume the distribution, which is particularly useful when the data does not conform to a normal distribution. Furthermore, a theoretical framework is developed to certify the effectiveness of our method. Inspired by randomized smoothing techniques Zhao et al. (2021, 2022), we illustrate that parameters with Kolmogorov-Smirnov distance bounded by a small value before and after fine-tuning do not impact prediction. This provides a way for giving a certified lower bound for the performance of partial tuning on winning tickets. Our analysis also proves that KS-Lottery can find a small set of winning tickets when the original prediction model shows little uncertainty. This gives us a strong foundation for asserting that KS-Lottery can be an effective tool for finding winning tickets.

* an ultra-small subset of parameters that are sufficient to fine-tune on to achieve that of Full Tuning.
* Theoretically, we prove that KS-Lottery finds certified winning tickets.
* Empirically, we demonstrate that fine-tuning as few as 18 identified winning tickets (the token embedding) of LLMaMA-7B using en\(\rightarrow\)ca data achieves surprisingly good performance in translation tasks. This will result in a new standard in multilingual transfer of LLMs.

## 2 Related Work

Lottey Tickets Hypothesis.The Lottery Tickets Hypothesis suggests the presence of 'winning tickets' or beneficial subnetworks within a modelFrankle and Carbin (2018); Malach et al. (2020). These subnetworks, discovered during pruning, are believed to be specifically suited to the learning task Frankle and Carbin (2018). In relation to this, Zheng et al. (2022) fins that these 'winning tickets' are more sparse in the later layers of the BERT model on GLUE tasks Wang et al. (2018). While much research focuses on model pruning, there's also work on efficient parameter tuning Ding et al. (2023). For example, it's shown that the performance of fine-tuned parameters can indicate both the task's inductive biases and the model's inherent structure Ding et al. (2023).

Certified Methods for Transfer Learning.Certification is crucial in transfer learning, aiming to measure a model's generalization and capabilities Raghunathan et al. (2018); Jia et al. (2019). Research has introduced certified robustness accuracy as a defense against adversarial attack Raghunathan et al. (2018); Jia et al. (2019); Muravev and Petiushko (2021); Zhao et al. (2022); Lecuyer 

et al., 2019). Randomized smoothing, a model-independent certification technique, assesses how input changes affect predictions Lecuyer et al. (2019); Muravev and Petiushko (2021). Our approach focuses on model parameter variations. Other studies have certified fairness in models Ruessos et al. (2020); Peychev et al. (2021) and robustness against data selection Wang et al. (2023).

Multilingual Large Language Model.Large Language Models LLMs; OpenAI (2023); Zhang et al. (2022); Brown et al. (2020); Chowdhery et al. (2022); Touvron et al. (2023a, b)_inter alia_) excel in English but underperform in other language. Studies have fine-tuned LLMs using monolingual or multilingual data to enhance their multilingual capabilities Zhu et al. (2023); Li et al. (2023); Jiao et al. (2023); Cui et al. (2023); Yang et al. (2023). Embedding Tuning can activate multilingual abilities in certain languages, suggesting that the intrinsic dimension of these abilities may lie within the embedding layer Li et al. (2018); Yuan et al. (2023b). The intrinsic dimension, the minimum parameters needed for a specific objective function, is estimated using heuristic methods and random subspace training due to computational constraints Li et al. (2018); Aghajanyan et al. (2021).

## 3 Certified Winning Tickets via KS-Lottery

In this section, we first introduce KS-Lottery Fasano and Franceschini (1987) (SS 3.1). Then we apply KS-Lottery to find certifiable winning tickets in a multilingual transfer scenario (SS 3.2). Finally, the theoretical result of SS 3.2, as well as the experimental result (SS 3.3), validate the effectiveness of KS-Lottery.

### KS-Lottery

Guided by the hypothesis that parameters undergoing substantial changes during fine-tuning are crucial for making predictions Levin et al. (2022), we exam the distribution \(p_{i}\) of each parameter \(\mathbf{\theta}_{i}\) (with the parameter \(\theta_{ij}\) of token \(j\) drawn from \(p_{i}\)) before and after the fine-tuning. Although various metrics exist for pinpointing essential parameters Li et al. (2016); Dalvi et al. (2019); Meng et al. (2022), these often depend on specific cutoff values, and determining the necessary number of parameters in advance is challenging. We advocate for a new approach: _actively seeking out "lottery tickets" that are guaranteed to achieve similar fine-tuning outcomes with a high level of confidence._ This calls for a more principled approach and the Kolmogorov-Smirnov Test stands out. In this section, we introduce the test and then theoretically explain its effectivenes in the following section.

We propose a probing strategy that employs the Kolmogorov-Smirnov Test, which is a statistical method used to compare two sample distributions and determine whether they are drawn from the same underlying distribution. The Kolmogorov-Smirnov Test is an exact test, meaning that distribution does not depend on the underlying cumulative distribution function being tested. Specifically, we view the embedding of each LLM token \(j\) as a distribution i.e. \(\theta^{E}_{ij}\sim p_{i}\). The cumulative distribution function (CDF) of \(\theta^{E}_{i}\) and \(\widetilde{\theta}^{E}_{i}\) could be denoted by \(\Phi_{i}(\theta)\) and \(\widetilde{\Phi_{i}}(\theta)\). The Kolmogorov-Smirnov distance between the two CDFs is \(D_{i}=\sup_{\theta}|\widehat{\Phi_{i}}(\theta)-\Phi_{i}(\theta)|\).

Now we wish to determine whether a token embedding before and after fine-tuning comes from the same distribution. Formally, we state Kolmogorov-Smirnov Test as:

**Theorem 1**.: _(Kolmogorov-Smirnov Test, Fasano and Franceschini (1987)) The test statistic for this Kolmogorov-Smirnov Test can be defined in terms of two hypotheses: \(H_{0}\): \(\theta_{i}\) and \(\widetilde{\theta}_{i}\) come from the same distribution. \(H_{1}\): two samples aren't from the same distribution. If test T: \(D_{i}>\tau(\alpha)\) is passed, then \(H_{1}\) holds with confidence \(1-\alpha\), where \(D_{i}=\sup_{\theta}|\widehat{\Phi_{i}}(\theta)-\Phi_{i}(\theta)|\), \(\tau(\alpha)=c(\alpha)\sqrt{\frac{2}{d}}\), the value of \(c(\alpha)\) is given in the reference table Karson (1968), and \(d\) is the parameter dimension._

Based on the Kolmogorov-Smirnov Test, we came to propose our method, KS-Lottery. A parameter is designated as a "winning ticket" if it meets the criterion of rejecting the null hypothesis (no difference in the distribution of the embedding before and after fine-tuning) and the alternative hypothesis \(H_{1}\) (indicating a significant distributional change) is accepted. Kolmogorov-Smirnov Test ensures that if the distribution of parameter \(\theta\) does not change after fine-tuning, then \(\mathbb{P}\left[D_{i}>\tau(\alpha)\right]<\alpha\), ensuring the majority of crucial token embeddings would be chosen by the test.

### Finding 1: KS-Lottery finds certifiable winning tickets in multilingual transfer.

There exists a set of winning tickets \(\mathbf{\theta}^{E}_{a}\) within the token embeddings \(\mathbf{\theta}^{E}=[\theta_{0},\theta_{1},\cdots,\theta_{|V|}]\), where 

\(\mathbf{\theta}^{E}=[\mathbf{\theta}_{a}^{E},\mathbf{\theta}_{b}^{E}]\), \(\widetilde{\mathbf{\theta}}^{E}=[\widetilde{\mathbf{\theta}}_{a}^{E},\mathbf{\theta}_{b}^{E}]\) is the parameters of embedding layer before and after tuning. If tuning on parameters \(\mathbf{\theta}_{a}^{E}\) would achieve similar performance to full tuning on \(\mathbf{\theta}^{E}\) downstream tasks, then we refer to \(\mathbf{\theta}^{E}\) as the winning tickets. For convenience, we discuss the downstream task as a next-token prediction problem, which is central to text generation, such that for an LLM \(f(\cdot,\mathbf{\theta}^{E})\), given input context \(x\), generate the most probable token in vocabulary set \(\mathcal{Y}\).

\[g(x,\widetilde{\mathbf{\theta}}_{a}^{E},\mathbf{\theta}_{b}^{E})=g(x, \widetilde{\mathbf{\theta}}_{a}^{E},\widetilde{\mathbf{\theta}}_{b}^{E}),\\ \text{where }g(x,\mathbf{\theta}_{a}^{E},\mathbf{\theta}_{b}^{E})=arg \max_{c\in\mathcal{Y}}\mathbb{P}\left[f(x,\mathbf{\theta}_{a}^{E},\mathbf{\theta}_{ b}^{E})=c\right] \tag{1}\]

Suppose that when the LLM \(f(\cdot,\mathbf{\theta}_{a}^{E},\mathbf{\theta}_{b}^{E})\) predicts any given input \(x\), the most probable token \(c_{A}\) is returned with probability \(p_{A}\). Given that small changes to parameters in a smooth subspace do not affect decision boundary [26, 30], we could probably guarantee that training on KS-Lottery selected token embeddings could achieve the same performance as full layer Embed Tuning at high confidence. The intuition of the theory is illustrated in Figure 2. Our main theoretical results are as follows, the proofs can be found in Appendix B:

**Theorem 2**.: _(Certified Winning Tickets) Let \(f(\cdot,\mathbf{\theta}_{a},\mathbf{\theta}_{b}):\mathbb{R}^{d}\rightarrow\mathcal{ Y}\) be any random or deterministic function, and let \(g\) be defined as in Equation 1. For any input \(x\), suppose \(c_{A}\in\mathcal{Y}\), the bounds of prediction based on random variable parameters \(\widetilde{\mathbf{\theta}}_{a}\), \(\widetilde{\mathbf{\theta}}_{b}\), \(\underline{p_{A}},\overline{p_{B}}\in[0,1]\) satisfies_

\[\begin{split}&\mathbb{P}\left(f(x,\widetilde{\mathbf{\theta}}_{a}, \widetilde{\mathbf{\theta}}_{b})=c_{A}\right)\\ &\geq\underline{p_{A}}\geq\overline{p_{B}}\geq\max_{c\neq c_{A}} \mathbb{P}\left(f(x,\widetilde{\mathbf{\theta}}_{a},\widetilde{\mathbf{\theta}}_{b}=c )\right)\end{split} \tag{2}\]

_If the set of parameters \(\mathbf{\theta}_{b}\) satisfies_

\[D(\widetilde{\theta}_{b}^{i},\theta_{b}^{i})<\tau(\alpha)<\frac{\underline{p_{ A}}-\overline{p_{B}}}{2}, \tag{3}\]

_for all \(i\in V_{b}\), the generator partial-tuned on parameters \(\mathbf{\theta}_{a}\) always return token \(c_{A}\), i.e. \(g(x,\widetilde{\mathbf{\theta}}_{a}^{E},\mathbf{\theta}_{b}^{E})=c_{A}\)._

_Remark 3_.: (About \(\mathbf{\theta}_{a}\)) Theorem 2 immediately holds for _Partial Transfer_ setting as specified in SS 3.3. As for _Partial Tuning_ setting, we need to use the hypothesis that the value of \(\widetilde{\mathbf{\theta}}_{a}\) in the Embed Tuning setting and partial-tuning setting are the same for Theorem 2 to hold. This is due to \(\mathbf{\theta}_{a}\) taking major effect during Embed Tuning despite small changes in \(\mathbf{\theta}_{b}\).We also show with empirical analysis in SS 4.2 that \(\widetilde{\mathbf{\theta}}_{a}\) in two tuning settings share the same distribution.

_Remark 4_.: (About \(\alpha\)) Practically, \(\tau(\alpha)\) is small. In the case of fully embedding tuned LLM that has optimal performance and produces confident prediction, \(1>p_{A}\gg p_{B}>0\), we can choose small \(\alpha=\tau^{-1}(\frac{p_{A}-p_{B}}{2})\) while maintaining performance. In this case, the Kolmogorov-Smirnov Test samples fewer token embeddings for fine-tuning.

_Remark 5_.: (About Applicability) This theorem could be used for certifying fine-tuning lottery tickets for any black-box model and input. Note that KS-Lottery could be applied to the entire model, though we only perform testing on the embedding layer for convenience and training stability.

### Finding 2: 18 identified winning tickets (18 tokens) achieves remarkable performance.

There are two different ways to verify the effectiveness of winning tickets: Partial Tuning (sufficient

Figure 2: Illustration of \(f(\cdot,\mathbf{\theta}_{a}^{E},\mathbf{\theta}_{b}^{E})\) in 2 dimensions. **Left:** The concentric circles are the density contours of embedding parameters before and after tuning, and the colored landscape is the decision boundaries of \(f(\cdot)\). **Right:** the distribution \(\mathbb{P}\left[f(x,\widehat{\theta}_{a}^{E},\widetilde{\theta}_{b}^{E})\right]\) and \(\mathbb{P}\left[f(x,\widetilde{\theta}_{a}^{E},\theta_{b}^{E})\right]\). \(\underline{p_{A}}\) is the probability \(\mathbb{P}\left[f(x,\widetilde{\theta}_{a}^{E},\widetilde{\theta}_{b}^{E})\right]\) predicts \(x\) to be token \(c_{A}\) (color blue), and \(\overline{p_{B}}\) as the probability of second most likely prediction (color red). \(D_{ks}\) denotes the Kolmogrov-Smirnov distance between distributions before and after tuning. We choose the set of token embeddings for tuning as those with little distribution overlap before and after fine-tuning, which may be critical to prediction.

 

condition) and Partial Transfer (necessary condition). **Partial Tuning.** As shown in Figure 3, only train winning tickets, keeping the remaining parameters frozen. **Partial Transfer.** Given a model trained by Embed Tuning, we select the winning tickets from the embedding layer and use them to replace the corresponding parameters in the original model, thus curating a new model.

As shown in **Table 1**, just tuning winning tickets can achieve results on par with Embed Tuning. Meanwhile, by only modifying the winning tickets, it manages to retain \(86.9\%(31.3/36.1)\) of the performance. The experimental result demonstrates that a low-dimensional subspace exists, guided by the winning tickets, that can achieve comparable performance as optimizing all parameters in the embedding-tuned model. The result empirically verifies the effectiveness of winning tickets.

## 4 Analysis

KS-Lottery.There are three different ways to use the winning tickets. **Partial Tuning, Partial Transfer** and **Frequency Tuning**, which only train the high-frequency tokens in the corpus.

Other Baselines.Original Model Directly using the LLAMA-7B [20]Mistral-7B [25] weight on test data, without any tuning. **Random Tuning** Only the tokens randomly selected in the embedding layer are fine-tuned, while the remaining parameters are kept frozen. **Full Tuning** is an approach to transfer learning where the weights of a pre-trained entire model are trained on new data. **Embed Tuning** Merely fine-tuning the embedding layer of a model keeping the remaining param

\begin{table}
\begin{tabular}{l|c c c c c||c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{4}{c||}{**LLaMa-7B**} \\  & **\# Token** & **en–rcróg** & **en–rsè** & **en–ndêe** & **en–raça** & **Avg.** & **\# Token** & **en–rvø** & **en–resè** & **en–de** & **em–va** & **Avg.** \\ \hline Original Model & & & 3.5 & 4.8 & 4.8 & 5.7 & 4.6 & & & & & & & & \\ Full Tuning & - & 28.3 & 23.5 & 22.5 & 34.9 & 27.3 & - & 33.4 & 25.1 & 29.1 & 37.8 & 31.4 \\ Embed Tuning & & 28.7 & 25.5 & 25.8 & 36.1 & 29.0 & & & **34.2** & **28.3** & **33.2** & 40.2 & **34.0** \\ \hline Random Tuning* & \textless{}18 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & & & & & & & \\ Partial Tuning & += & 20.7 & **26.7** & 26.4 & **37.7** & 27.9 & \(<\)169 & 16.2 & 13.9 & 15.3 & 20.3 & 16.4 \\ Partial Transfer & (p-value \(<\) 0.05) & 23.4 & 22.6 & 16.1 & 31.1 & 23.4 & (p-value \(<\) 0.05) & 26.7 & 27.3 & 30.5 & 37.5 & 30.5 \\ \hline Random Tuning* & \textless{}100 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & 0.1 & & & & & & & \\ Partial Tuning & += & 26.9 & 27.3 & 27.2 & 37.4 & 29.7 & \(<\)170 & 17.3 & 13.9 & 15.3 & 20.4 & 16.7 \\ Partial Transfer & (p-value \(<\) 0.25) & 25.1 & 26.5 & **30.4** & 34.4 & 29.1 & (p-value \(<\) 0.25) & 26.7 & 27.3 & 30.5 & 37.5 & 30.5 \\ \hline Random Tuning* & \textless{}800 & 5.8 & 8.7 & 5.2 & 11.8 & 7.9 & & & & & & & & & \\ Partial Tuning & \textless{}800 & 29.4 & 27.3 & 30.1 & 37.7 & 31.1 & \(\leq\)180 & 16.3 & 13.9 & 15.4 & 20.3 & 16.5 \\ Partial Transfer & & **30.2** & **26.7** & **30.4** & 37.6 & **31.2** & & & & & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Empirical verification of winning tickets with **Partial Tuning** and **Partial Transfer** with bilingual translation data. * denotes that the random results from three distinct seeds.

Figure 3: Overview of **Partial Training**, one application of KS-Lottery. It consists of two steps: (1) finding the winning tickets in the embedding layer by Kolmogorov-Smirnov Test; (2) one way to use these winning tickets is partial tuning these tokens ensuring other parameters keep frozen.

 

eters frozen. **LoRA**Hu et al. (2022) utilizes low-rank matrices for approximating parameter updates. **QLoRA**Dettmers et al. (2023) reduces memory usage enough to efficiently fine-tune. **Prefix-Tuning**Li and Liang (2021) introduce a lightweight prefix module into the input layer and each transformer layer, enabling efficient training over these modules.

Training and Evaluation.To ensure a fair comparison, we apply various parameter-efficient settings on LLaMA-7B using Lego-MT Yuan et al. (2023) 10k data. For full tuning, training with LoRA, and Embed Tuning, we set the learning rate to \(2e-5\) and the number of epochs to \(3\). For partial tuning, prefix-tuning, and KS-Lottery, we set the learning rate to \(1e-2\) and the number of epochs to \(5\). All other parameters are kept consistent across all settings. We test each model on the Flores-101 Goyal et al. (2022) devtest, which offers human-written translation pairs across 101 languages. In alignment with Flores-101, we employ the same evaluation metric, sentence piece BLEU (spBLEU) on beam size\(=4\), to assess multilingual capabilities.

### KS-Lottery Certification

_Certified Accuracy_, when using certified winning tickets Embed Tuning, is measured as the proportion of correct predictions from an embedding tuned model (reference model) that is certified to be correct at a significance level of \(\alpha\). The certification process follows Theorem 2 and is stated in Algorithm 1, where for each prediction based on input sequence \(x\), we compare the probability gap between two most-likely tokens by the original LLaMA, i.e. \(\frac{p_{A}-\overline{p_{B}}}{2}\) and \(\tau(\alpha)\). \(\tau(\alpha)\) is a static value that could be obtained through the equation, though we use the value estimated with Scipy Kolmogorov-Smirnov test to be precise.

To check the validity of our certification method and test the empirical tightness of the certification bound, we experiment on Flores-101 devtest. A model, developed using partial tuning, and Embed Tuning, utilizes both the instruction and a partial sequence of reference tokens as input. Temporarily both settings of certification only use the first twenty token prediction results for calculation for a fair comparison and don't let overlong sentences dominate the results. It subsequently predicts the next token in the partial reference sentence. The "prediction accuracy" is ascertained by comparing the predicted token, generated by the partially tuned model, with the reference token.

Our theoretical discovery provides certification for the lottery tickets hypothesis.In Figure 4 that plots certified accuracy as a function of \(\alpha\), the certified accuracy always increases gradually until reaching the same value as the prediction accuracy. This is due to the decrease in \(\tau(\alpha)\) as \(\alpha\) increases and reaches 0 when \(\alpha=1\). At any \(\alpha\), the empirical estimation of certified accuracy is a lower bound of prediction accuracy, providing a performance guarantee for tuning on the lottery tickets.

Certification lower bound is tighter for larger \(\alpha\) .When \(\underline{p_{A}}\gg\overline{p_{B}}\), there must exist a \(\alpha\) that satisfies the certification constraint. Since \(\underline{p_{A}}\) and

Figure 4: Certified experiment under **Partial Tuning** setting. **A**: Estimation of \(\tau(\alpha)\) w.r.t different \(\alpha\) by running Kolmogorov-Smirnov Test between the distribution of LLaMA-7B embedding and fine-tuned embedding on different datasets. **B C D**: Comparison between _Certified Accuracy_ and _Empirical Prediction Accuracy_ w.r.t. different \(\alpha\) on 3 datasets. More results are shown in Appendix C.

 

\(\overline{p_{B}}\) are input and model dependent, we empirically assessed the tightness of our bound by comparing the estimated value of certified accuracy with prediction accuracy. As shown in Figure 4, the bound is tighter when \(\alpha\to 1\), and the gap is larger when \(\alpha\to 0\). This is due to when fewer token embeddings are chosen, the certification expects the performance to be worse, though in actual tuning even zero-shot performance is quite good due to pre-training. The gap is quite small in practical cases, the gap is about 2% on average for \(\alpha=0.05\) and around 1.5% on average for \(\alpha=0.5\). Therefore, we recommend using \(\alpha\geq 0.05\) significance rate for guaranteed.

KS-Lottery Efficiency and Interpretability Partial Tuning can serve as a parameter-efficient tuning method.We proceed to evaluate it as a method that optimizes parameter usage, in comparison with another similar method. As illustrated in Table 2, in contrast to other methods such as LoRA Hu et al. (2022), QLoRA Dettmers et al. (2023) and Prefix-tuning Li and Liang (2021), Partial Tuning eliminates the need for an additional model structure. Remarkably, our method not only matches but frequently surpasses the performance of these alternate approaches.

Comparing with other selection methods, KS-Lottery is parameter-efficient.There are other ways (more details in Appendix C) to select critical parameters, such as Cos, Absolute, Relative, Ratio, and KL. In Table 3, we selected only 18 selective tokens by KS-Lottery on en\(\rightarrow\)ca bilingual data with extremely stringent requirements (p-value \(<\) 0.05). At this time, we used other evaluation methods to measure the importance of these tokens separately: the evaluation results of these tokens based on distribution changes (such as Cos, KL) are highly consistent with KS-Lottery; those evaluation methods based on value changes (Absolute, Relative, and Ratio) tend to give these tokens very low importance evaluation results. If we use different methods to select the top 18 important tokens for **Partial Transfer** experiments, the performance of different selective methods is 31.3 (KS-Lottery), 28.7 (Cos), 5.8 (Absolute), 5.8 (Relative), 5.8 (Ratio) and 1.6 (KL). Based on the experimatl results, we find that except for the results of Cos, which are close to the performance of KS-Lottery (but still -2.6 points), the translation performance of other methods is very poor.

Winning tickets are high-frequency tokens in the corpus.In the embedding layer, each dimension is associated with a meaningful token index. By referencing the vocabulary, we can decode the text representation corresponding to this index, as illustrated in Table 3 (Str). To examine the distinct characteristics of these tokens, we retrieve 50k sentences in ca language from the MC4 Raffel et al. (2019) dataset, then employ LLaMA's tokenizer to segment all sentences, tallying the occurrence of each token within the corpus. Table 3 (Freq.) indicates winning tickets commonly associated with the most frequently occurring tokens in the corpus. Based on this finding, we start the training only with the high-frequency tokens in the corpus. The summary of different tuning setting, as shown

Figure 5: When selective tokens are restricted from being updated, the model’s fine-tuning process for downstream tasks loses its effectiveness.

\begin{table}
\begin{tabular}{c|c c c c c c c} \hline \hline
**Setting** & **en\(\rightarrow\)ca** & **en\(\rightarrow\)da** & **en\(\rightarrow\)de** & **en\(\rightarrow\)ce** & **em\(\rightarrow\)np** & **Avg.** \\ \hline Embedding Tuning & 36.1 & 32.7 & 25.8 & 25.5 & 19.5 & 27.9 \\ Partial Tuning & **37.7** & **33.3** & 30.1 & **27.3** & **19.8** & **29.6** \\ Frequency Tuning & **37.7** & 30.0 & **30.7** & 26.1 & 19.6 & 28.8 \\ \hline \hline \end{tabular}
\end{table}
Table 4: The best result from Frequency Tuning, Partial Tuning, and the result of Embed Tuning.

 

in Table 4, suggests the training is sufficient with frequency token tuning.

**Sensitivity in the significance level Selection (\(\alpha\)).** Our theoretical framework provides a way for heuristically finding an effective \(\alpha\), as mentioned in Remark5. Since the original fully-tuned prediction model is available, we could directly predict the range of \(\frac{pA-pB}{2}\) as [\(s_{min},s_{max}\)] for all predictions in the dataset. An ideal \(\alpha\) would satisfy \(\tau(\alpha)<s_{min}\), under this condition, every data point in the dataset is guaranteed to make the same prediction as the original prediction model.

In practice, we don't need a full certification of the dataset to make the KS-Lottery work, as this would result in quite large \(\alpha\) values. As we can see in the en\(\rightarrow\)ca example below: for \(\alpha\) larger than 0.05, the verified percentage is an acceptable 94.3%, though not fully verified, and the empirical accuracy proves acceptable. In practice, the acceptable range of \(\alpha\) varies between datasets, though usually \(\alpha\) within 0.05-0.4 guarantees a verified percentage 95%.

**The distribution of winning tickets under both Partial Tuning and Partial Transfer is largely identical.** By conducting KS-Lottery with varying \(\alpha\) values, we obtain different winning tickets (denoted as \(\boldsymbol{\theta}_{a}\)). Furthermore, we perform a Kolmogorov-Smirnov Test on the winning tickets tuned with partial tuning and Embed Tuning, and compute the number of tokens that exhibit a significant difference (\(\widetilde{\boldsymbol{\theta}}^{\prime}_{a}\)). The ratio of unchanged tokens is calculated using the formula \(1-\widetilde{\boldsymbol{\theta}}^{\prime}_{a}/\widetilde{\boldsymbol{\theta }}_{a}\). As shown in Table 6, the distribution of winning tickets before and after tuning (Partial Transfer vs Partial Tuning) remains largely consistent.

**Keeping the parameters of winning tickets frozen while tuning the remaining parameters could lead to the collapse of the Embed Tuning.** Our previous experiments have highlighted the significance of winning tickets in training, while it's intriguing to consider whether such an important function can be replaced by other tokens. To delve into this problem, we freeze the winning tickets, and fine-tune the reset parameters on bilingual data. As shown in Figure 5, a small amount of disabled winning tickets is acceptable for tuning. However, as the number of disabled tokens increases, the entire tuning process crashes. The process reveals a remarkable finding: for a vocabulary size of 32k, 1k is a very small, yet fewer than 1k winning tickets play a crucial role in Embed Tuning.

**Applying KS-Lottery on the whole model.** After training all model parameters using en\(\rightarrow\)ca bilingual sentence pair from Lego-MT Yuan et al. (2023), we utilize the KS-Lottery to identify a small parameter set for multilingual transfer by comparing the parameters before and after tuning. Interestingly, no parameters exhibite significant changes before and after tuning. Concurrently, the study by Yuan et al. (2023) revealed that fine-tuning specific layers, including the embedding layer, can yield results comparable to those of full-tuning. Figure 6 reveals that following finetuning on the Lego-MT Yuan et al. (2023) en\(\rightarrow\)ca bilingual dataset, a subset of token embeddings exhibit significant changes in their parameters. Additionally, a small number of parameters (fewer than two for each layer with a significance level of \(\alpha<0.05\)) demonstrate substantial changes within LayerNorm. However, for multilingual transfer, the impact of LayerNorm varies and is not uniform across the lower and higher layers.

## 5 Conclusion

This work presents a novel method, KS-Lottery, which applies the lottery ticket hypothesis to

\begin{table}
\begin{tabular}{c|c c c c c c c} \hline \hline \(\alpha\) & **ro** & **es** & **de** & **ca** & **ptd** & **da** & **no** & **bs** \\ \hline
0.05 & 0.56 & 0.67 & 0.81 & 0.78 & 0.82 & 0.33 & 0.64 & 0.82 \\
0.1 & 0.75 & 0.78 & 0.72 & 0.80 & 0.85 & 0.71 & 0.79 & 0.81 \\
0.2 & 0.78 & 0.75 & 0.68 & 0.73 & 0.75 & 0.79 & 0.83 & 0.81 \\
0.3 & 0.69 & 0.82 & 0.58 & 0.77 & 0.86 & 0.76 & 0.81 & 0.80 \\
0.4 & 0.75 & 0.92 & 0.89 & 0.83 & 0.83 & 0.78 & 0.82 & 0.84 \\
0.5 & 0.83 & 0.87 & 0.82 & 0.80 & 0.85 & 0.86 & 0.84 & 0.90 \\
0.6 & 0.83 & 0.90 & 0.88 & 0.85 & 0.86 & 0.86 & 0.85 & 0.92 \\
0.7 & 0.83 & 0.91 & 0.88 & 0.86 & 0.90 & 0.86 & 0.89 & 0.88 \\
0.8 & 0.82 & 0.90 & 0.87 & 0.87 & 0.90 & 0.87 & 0.88 & 0.88 \\
0.9 & 0.82 & 0.86 & 0.84 & 0.85 & 0.88 & 0.84 & 0.86 & 0.85 \\ \hline \hline \end{tabular}
\end{table}
Table 6: With varying values of \(\alpha\), the distribution of winning tickets remains largely consistent across both Partial Tuning and Partial Transfer.

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline
**paralueCo** & **Verified Percentage** & **Original** & **Certified** & **Empirical** \\  & **(\% \(D_{i}>r(\alpha)\))** & **Accuracy** & **Accuracy** \\ \hline
0.05 & 94.3 & 75.86 & 63.77 & 78.87 \\
0.10 & 95.1 & 75.86 & 63.98 & 78.85 \\
0.20 & 95.6 & 75.86 & 66.15 & 78.93 \\
0.30 & 95.8 & 75.86 & 66.19 & 78.98 \\
0.50 & 96.5 & 75.86 & 66.35 & 78.97 \\
0.70 & 96.8 & 75.86 & 66.48 & 79.03 \\
0.90 & 97.5 & 75.86 & 66.66 & 79.09 \\
1.00 & 100.0 & 75.86 & 67.23 & 78.46 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Impact of Different \(\alpha\) on the en\(\rightarrow\)ca. For \(\alpha\) values greater than 0.05, the verification percentage reaches 94.3%, indicating substantial but not complete verification. The empirical accuracy under these conditions is also satisfactory.

 

LLMs fine-tuning. By employing the Kolmogorov-Smirnov Test, KS-Lottery first analyzes the shift in the parameter distribution before and after finetuning, and then identifies a small but effective subset of LLM parameters. Theoretical evidence confirms that KS-Lottery can pinpoint certified winning tickets within the embedding layer, thereby ensuring performance equivalent to full tuning. Notably, KS-Lottery surpasses other parameterized tuning algorithms by identifying fewer parameters for fine-tuning while maintaining similar performance levels.

## References

* A. Aghajanyan, S. Gupta, and L. Zettlemoyer (2021)Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 7319-7328. External Links: Link Cited by: SS4.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D Kaplan, P. Dhariwal, Arvind Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1674-469X, Document, Link Cited by: SS4.
* A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. (2022)Palm: scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. Cited by: SS4.
* Y. Cui, Z. Yang, and X. Yao (2023)Efficient and effective text encoding for chinese llama and alpaca.. arXiv preprint arXiv:2304.08177. Cited by: SS4.
* F. Dalvi, N. Durrani, H. Sajjad, Y. Belinkov, A. Bau, and J. Glass (2019)What is one grain of sand in the desert? analyzing individual neurons in deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, pp. 6309-6317. External Links: Link Cited by: SS4.
* T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer (2023)Qlora: efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, Vol. 36, pp. 10088-10115. External Links: Link Cited by: SS4.
* N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C. Chan, W. Chen, J. Yi, W. Zhao, X. Wang, Z. Liu, H. Zheng, J. Chen, Y. Lui, J. Tang, J. Li, and M. Sun (2023)Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence5 (3), pp. 220-235. External Links: Link Cited by: SS4.
* G. Fasano and A. Franceschini (1987)A multidimensional version of the Kolmogorov-Smirnov test. Monthly Notices of the Royal Astronomical Society225 (1), pp. 155-170. External Links: Link Cited by: SS4.
* Jonathan Frankle and Michael Carbin (2018)The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. Cited by: SS4.
* J. Frankle and M. Carbin (2019)The lottery ticket hypothesis: finding sparse, trainable neural networks. In International Conference on Learning Representations, Cited by: SS4.
* N. Goyal, C. Gao, V. Chaudhary, P. Chen, G. Wenzek, D. Ju, S. Krishnan, M. Kanezato, R. Guzman, and A. Fan (2022)The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics10, pp. 522-538. External Links: Link Cited by: SS4.
* E. J. Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen (2022)LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS4.
* R. Jia, A. Raghunathan, K. Goksel, and P. Liang (2019)Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 4129-4142. External Links: Link Cited by: SS4.
* A. Q. Jiang, A. Sablayrolles, A. Menesch, C. Bamford, D. Shahplot, D. de Las Casas, F. Brassand, G. Lengyel,

Figure 6: Applying KS-Lottery on whole LLAMA-7B which is single-layer tuned on Lego-MT en\({}_{-}\)ca 10k data. Each layer is trained in isolation and is analyzed by KS-Lottery to identify the parameters with significant changes (as indicated by scatter points above the red line). We find that within each Transformer layer, changes are primarily focused on LayerNorm, while other notable changes occur in the embedding layer.

 

Guillaume Lample, Lucile Saulnier, Lelio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothe Lacroix, and William El Sayed. 2023. Mistral 7b. _CoRR_, abs/2310.06825.
* Jiao et al. (2023) Wenxiang Jiao, Jen tse Huang, Wenzuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Parrot: Translating during chat using large language models tuned with human translation and feedback. In _Findings of EMNLP_.
* Karson (1968) Marvin Karson. 1968. Handbook of methods of applied statistics. volume i: Techniques of computation descriptive methods, and statistical inference. volume ii: Planning of surveys and experiments. i. m. chakravarti, r. g. laha, and j. roy, new york, john wiley; 1967, SS9.00. _Journal of the American Statistical Association_, 63(323):1047-1049.
* Lecuyer et al. (2019) Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. 2019. Certified robustness to adversarial examples with differential privacy. _Preprint_, arXiv:1802.03471.
* Levin et al. (2022) Roman Levin, Manli Shu, Eitan Borgnia, Furong Huang, Micah Goldblum, and Tom Goldstein. 2022. Where do models go wrong? parameter-space saliency maps for explainability. _Advances in Neural Information Processing Systems_, 35:15602-15615.
* Li et al. (2018) Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. 2018. Measuring the intrinsic dimension of objective landscapes. In _International Conference on Learning Representations_.
* Li et al. (2023) Jiahuan Li, Hao Zhou, Shujian Huang, Shanbo Cheng, and Jiajun Chen. 2023. Eliciting the translation ability of large language models via multilingual finetuning with translation instructions. _Preprint_, arXiv:2305.15083.
* Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding neural networks through representation erasure. _arXiv preprint arXiv:1612.08220_.
* Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 4582-4597, Online. Association for Computational Linguistics.
* Malach et al. (2020) Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. 2020. Proving the lottery ticket hypothesis: Pruning is all you need. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 6682-6691. PMLR.
* Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. _Advances in Neural Information Processing Systems_, 35.
* Muravev and Petiushko (2021) Nikita Muravev and Aleksandr Petiushko. 2021. Certified robustness via randomized smoothing over multiplicative parameters of input transformations. _arXiv preprint arXiv:2106.14432_.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Peychev et al. (2021) Momchil Peychev, Anian Ruoss, Mislav Balunovic, Maximilian Baader, and Martin T. Vechev. 2021. Latent space smoothing for individually fair representations. _CoRR_, abs/2111.13650.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv e-prints_.
* Raghunathan et al. (2018) Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. 2018. Semidefinite relaxations for certifying robustness to adversarial examples. _CoRR_, abs/1811.01057.
* Ruoss et al. (2020) Anian Ruoss, Mislav Balunovic, Marc Fischer, and Martin T. Vechev. 2020. Learning certified individually fair representations. _CoRR_, abs/2002.10312.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Bapitste Rozieere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _Preprint_, arXiv:2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355, Brussels, Belgium. Association for Computational Linguistics.
* Wang et al. (2023) Yudong Wang, Chang Ma, Qingxiu Dong, Lingpeng Kong, and Jingjing Xu. 2023. A challenging benchmark for low-resource learning. _arXiv preprint arXiv:2303.03840_.
* Yang et al. (2023) Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bigtrans: Augmenting large language models with multilingual translation capability over 100 languages. _arXiv preprint arXiv:2305.18098_.
* Yuan et al. (2023a) Fei Yuan, Yinquan Lu, Wenhao Zhu, Lingpeng Kong, Lei Li, Yu Qiao, and Jingjing Xu. 2023a. Lego-MT: Learning detachable models for massively multilingual machine translation. In _Findings of the Association for Computational Linguistics: ACL 2023_, 

pages 11518-11533, Toronto, Canada. Association for Computational Linguistics.
* Yuan et al. (2023) Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. 2023b. How multilingual is multilingual llm? _arXiv preprint arXiv:2311.09071_.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhouhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koruz, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models. _Preprint_, arXiv:2205.01068.
* Zhao et al. (2021) Haiteng Zhao, Chang Ma, Qinyu Chen, and Zhifong Deng. 2021. Domain adaptation via maximizing surrogate mutual information. _arXiv preprint arXiv:2110.12184_.
* Zhao et al. (2022) Haiteng Zhao, Chang Ma, Xinshuai Dong, Anh Tuan Luu, Zhi-Hong Deng, and Hanwang Zhang. 2022. Certified robustness against natural language attacks by causal intervention. In _International Conference on Machine Learning_, pages 26958-26970. PMLR.
* Zheng et al. (2022) Rui Zheng, Bao Rong, Yuhao Zhou, Di Liang, Sirui Wang, Wei Wu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2022. Robust lottery tickets for pre-trained language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2211-2224, Dublin, Ireland. Association for Computational Linguistics.
* Zhu et al. (2023) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. _arXiv preprint arXiv:2308.04948_.

 



#### Acknowledgments

Authors of this paper would like to thank Zixian Huang for giving many helpful comments on previous versions of this paper. We deeply express our gratitude to Shanghai AI Laboratory. This work is partially supported by the National Key R&D Program of China (NO.2022ZD01601022030).

## Appendix A Limitations and Broader Impacts

Limitations.In the scope of this research, our proposed KS-Lottery mainly targets multilingual research and has been thoroughly validated in bilingual translation tasks. However, we believe that our certified theoretical framework is generalizable to other types of tasks, and we leave these explorations for future work.

Broader Impacts.This paper presents work whose goal is to advance the field of machine learning and multilingual research. We do not anticipate it will cause negative societal impacts, such as potential malicious or unintended uses.

## Appendix B Proofs

**Theorem 3**.: (Certified Winning Tickets) Let \(f(\cdot,{\mathbf{\theta}^{a}},{\mathbf{\theta}^{b}}):\mathbb{R}^{d}\to\mathcal{Y}\) be any random or deterministic function, and let \(g\) be defined as in Equation 1. For any input \(x\), suppose \(c_{A}\in\mathcal{Y},p_{\underline{A}},\overline{p_{B}}\in[0,1]\) satisfies

\[\mathbb{P}\left(f(x,\widetilde{{\mathbf{\theta}}}^{a},\widetilde{{\mathbf{\theta}}}^{ b})=c_{A}\right)\geq p_{\underline{A}}\geq\overline{p_{B}}\geq\max_{c\neq c_{A}} \mathbb{P}(f(x+\tau))\]

If the set of parameters \({\mathbf{\theta}^{b}}\) satisfies

\[D(\widetilde{\theta}_{i}^{b},\theta_{i}^{b})<\tau(\alpha)<\frac{p_{A}- \overline{p_{B}}}{2},\]

for all \(i\in V_{b}\), the classifier partial-tuned on parameters \({\mathbf{\theta}^{a}}\) always return class \(c_{A}\), i.e. \(g(x,\widetilde{{\mathbf{\theta}}}_{a}^{E},{\mathbf{\theta}}_{b}^{E})=c_{A}\).

Proof SketchFix an input \(x\), we study the change of prediction w.r.t. change in the distribution of \({\mathbf{\theta}^{b}}\). Let \(A=\{{\mathbf{\theta}|g(X,\widetilde{{\mathbf{\theta}}}^{a},{\mathbf{\theta}})=c_{A}}\}\), \(B=\{{\mathbf{\theta}|g(x,\widetilde{{\mathbf{\theta}}}^{a},{\mathbf{\theta}})\neq c_{A}}\}\). Using definition, \(\mathbb{P}(\widetilde{{\mathbf{\theta}}}^{b}\in A)\geq\underline{p_{A}}\), \(\mathbb{P}(\widetilde{{\mathbf{\theta}}}^{b}\in B)\leq\overline{p_{B}}\). Since \(D(\widetilde{{\mathbf{\theta}}}^{b},{\mathbf{\theta}}^{b})=\max_{i}D(\widetilde{ \theta}_{i}^{b},\theta_{i}^{b})\leq\tau(\alpha)\), the minimum overlapping cumulative probability \(\mathbb{P}(\widetilde{{\mathbf{\theta}}}^{b}\in S\cap{\mathbf{\theta}}^{b}\in S)\geq 1 -\tau(\alpha)\), where \(S\) is any contour set \(S=\{{\mathbf{\theta}||{\mathbf{\theta}}-\widetilde{\mu}^{b}}|\leq s,s\geq 0\}\).

Now we compare the probability that \(f(x,\widetilde{{\mathbf{\theta}}}^{a},{\mathbf{\theta}}^{b})\) predicts \(c_{A}\) or other classes. The probability that the partial-tuned \(f\) predicts \(c_{A}\) is

\[\mathbb{P}({\mathbf{\theta}^{b}}\in A) \geq\mathbb{P}({\mathbf{\theta}^{b}}\in A\cap{\mathbf{\theta}^{b}}\in S)\] \[\geq\mathbb{P}\left[(\widehat{{\mathbf{\theta}}}^{b}\in S\cap{\mathbf{ \theta}^{b}}\in S)-(\widetilde{{\mathbf{\theta}}}^{b}\in S\cap\widetilde{{\mathbf{ \theta}}}^{b}\not\in A)\right]\] \[\geq(1-\tau(\alpha))-(1-\underline{p_{A}})=\underline{p_{A}}-\tau(\alpha)\]

Similarly, we can prove that \(\mathbb{P}({\mathbf{\theta}^{b}}\in B)\leq\overline{p_{B}}+\tau(\alpha)\). Thus when \(\tau(\alpha)<\frac{p_{A}-\overline{p_{B}}}{2}\) and we partial-tune all parameters \({\mathbf{\theta}^{a}}\) that fails to pass the KSFest, then the partial-tuned model \(g(x,\widetilde{{\mathbf{\theta}}}_{a}^{E},{\mathbf{\theta}}_{b}^{E})=c_{A}\), i.e. always give the same prediction as the embedding fully-tuned model. 

## Appendix C More Analysis

The observed trend in the value of \(\alpha\) suggests the presence of an optimal size for the set of selective tokens.In Figure 8, we examined the translation performance on the Flores-101 dataset under varying thresholds. It's important to note that by default, the threshold creates an interval that is closed on the left and open on the right. Our observations indicate that even a minimal threshold is capable of preserving approximately 80% of the performance that is achieved through comprehensive embedding fine-tuning, as highlighted in green. Interestingly, each plot exhibits the same pattern characterized by an initial increase in performance, followed by a subsequent decline. This pattern suggests the existence of an optimal point that maximizes performance.

Another Selective MethodsThere are five commonly used methods for determining parameters: Cos-Similarity (cos), Absolute Value (absolute), Related Value (relative), Related Ratio (ratio), and KL Divergence (KL). Each method emphasizes different aspects and requires a heuristic value \(p\) for selection.

\begin{table}
\begin{tabular}{l l|l l l l l l l} \hline \multicolumn{7}{l}{\(\alpha\)**} \\  & **Setting** & **



**Cos-Similarity:** This method determines the similarity between \(\theta_{i}^{E}\) and \(\widetilde{\theta}_{i}^{E}\) from the perspective of vector dot product. A lower cos-similarity suggests that it is the target of fine-tuning adjustment.

**Absolute Value:** This is calculate as \(|\widetilde{\theta}_{i}^{E}-\theta_{i}^{E}|\), which is concerned with the magnitude of the change. A larger value indicates that it is the target of fine-tuning.

**Related Value:** This is calculated as \(\widetilde{\theta}_{i}^{E}/\theta_{i}^{E}\), which focuses on the rate of change before and after fine-tuning. A larger ratio suggested that it is the target.

**Related Ratio:** This method considers the initial value and calculates the result as \((\widetilde{\theta}_{i}^{E}-\theta_{i}^{E})/\theta_{i}^{E}\). A larger ratio indicates that it is the key adjustment in fine-tuning.

**KL Divergence:** This is a statistical distance, measuring the distance between \(\widetilde{\theta}_{i}^{E}\) and \(\theta_{i}^{E}\), with a focus on the distribution itself.

### When Fine-tuning on different languages, the winning tickets contain the same tokens.

The winning tickets across various languages overlap with a subset of tokens, as shown in Table 8, {index id = 13 (token="n"), index id=278 (token='the'), index id=29871 (token='\(\sim\)'), and so on}, which are frequently used and common across various languages.

## Appendix D AI Assistants

We just use AI assistants for our sentence-level polishing.

## Appendix E Used Scientific Artifacts

Below lists scientific artifacts that are used in our work. For the sake of ethic, our use of these artifacts is consistent with their intended use.

* _Stanford Alpaca (Apache-2.0 license)_, a project that aims to build and share an

Figure 8: The translation performance of updated models with replacement method on Flores-101 with different token numbers. We observe the minimal threshold is sufficient to maintain approximately 80% of the performance achieved through embedding tuning (marked in orange). Meanwhile, for each plot, there exists a trend of initial increase followed by a decline, with the presence of an optimum point.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**en –ca** & **en –es** & **en –ro** & **en–da** & **en –de** & **en –ptt** \\ \hline

[MISSING_PAGE_POST]

 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Across various languages, the winning tickets share some common pattern.

Figure 7: Certified experiment under **Partial Transfer** setting. **A**: Estimation of \(\tau(\alpha)\) w.r.t different \(\alpha\) values by running KS-Test between the distribution of LLaMA-7b embedding and fine-tuned embedding on different datasets. **B C D**: Comparison between _Certified Accuracy_ and _Empirical Prediction Accuracy_ w.r.t. different \(\alpha\) values on three datasets.



instruction-following LLaMA model.
* _Lego-MT (MIT license)_, a dataset for machine translation.
* _Transformers (Apache-2.0 license)_, a framework that provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.