RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Cross Model Control
Given the observation that the logits shifts before and after fine-tuning in different LLMs are remarkably similar, we introduce a parameter-efficient, portable tiny language model designed to learn to alter the output logits of LLMs, which we call the “delta model”. Subsequently, we introduce a lightweight token mapping strategy called PM-MinED. Empowered by PM-MinED, the delta model becomes applicable to other LLMs with varying parameter sizes and vocabularies. An overview of the cross-model control is illustrated in Figure 3 .
Training the portable delta model to alter the output of LLM
In Section 2 , we found that the logits shifts before and after fine-tuning LLM are very similar. Therefore, we attempt to fit the logits shifts of LLM using a delta model. Specifically, as shown in figure 3(a) , we train the template LLM and delta model together, keeping the template LLM frozen and the delta model tunable. During forward propagation, we add the logits of LLM and the logits of the delta model together to obtain the final logits, aiming to teach the delta model to alter the logits of LLM. Considering that the logits of different LLM outputs have different scales, to enhance the delta model’s applicability during the inference stage after training, we apply a LogSoftmax to the logits of LLM to project them into logarithmic space.

Formally, we denote the template LLM as \mathcal{M}_{t} , the delta model as \mathcal{M}_{d} , a training data sample as x , the logits output by \mathcal{M}_{t} and \mathcal{M}_{d} at the final layer as \zeta_{t} and \zeta_{d} respectively, and the final logits used for calculating the loss as \zeta_{final} .

\zeta_{t}=\mathcal{M}_{t}(x),\quad\zeta_{d}=\mathcal{M}_{d}(x) (3)

\zeta_{final}=LogSoftmax(\zeta_{t})+\zeta_{d} (4)

We do not apply LogSoftmax to the logits \zeta_{d} of the delta model because it would lead to poorer performance, which we will explain in Section 3.2 .
Sharing the portable delta model with other LLMs
After the delta model has acquired the capability to alter LLM logits, we share it with other user LLMs to achieve fine-tuning effects. During the inference stage, the interaction between the LLM and the delta model remains consistent with the training stage. We apply LogSoftmax to the logits output by the LLM, projecting them into logarithmic space, and then add them to the logits output by the delta model to obtain the final logits used for decoding. When applying the delta model to LLMs with different vocabularies, we employ a token mapping strategy called PM-MinED to map tokens from the delta model’s vocabulary to tokens in the LLM’s vocabulary. The implementation details of this strategy will be presented in Section 3.3 .

Formally, we represent the user LLM as \mathcal{M}_{u} , a prompt as [x_{0},x_{1},...,x_{t-1}] . At time step t , the logits output by \mathcal{M}_{u} and \mathcal{M}_{d} at the final layer are denoted as \zeta_{u}^{t} and \zeta_{d}^{t} respectively, where

\zeta_{u}^{t}=\mathcal{M}_{u}(x<t),\quad\zeta_{d}^{t}=\mathcal{M}_{d}(x<t) (5)

The logits output by the user LLM assisted by the delta model at time step t are represented as

\zeta_{final}^{t}=LogSoftmax(\zeta_{u}^{t})+\alpha\cdotTokenMap(\zeta_{d}^{t}) (6)

Here, TokenMap represents the PM-MinED strategy, and \alpha denotes a strength coefficient that can adjust the intensity of alteration made by the delta model to the LLM outputs, which will be demonstrated in Section 4.4 . We do not apply LogSoftmax to the delta model’s logits in both Equations 4 and 6 . This is because the outputs of LogSoftmax are all negative. If a token from the LLM cannot find its corresponding delta model token in the token mapping, the logits of this token will not be able to receive a negative adjustment.
Token Mapping Strategy PM-MinED
To enable the delta model to be adapted to user LLMs with varying vocabularies , it is imperative to establish a mapping relationship between the vocabularies of the user LLM and the delta model. In previous token mapping strategies, Fu et al. ( 2023 ) implemented an exact match method, which involves finding tokens in the delta model’s vocabulary that are completely identical to the tokens of the user LLM; if a perfect match cannot be found, the matching attempt is abandoned. This method’s limitation is that the logits of tokens that cannot be perfectly matched will not be adjusted by the delta model. Building upon this, Wan et al. ( 2024 ) introduced the minimum edit distance strategy, aiming to utilize tokens that cannot be perfectly matched by finding the token in the delta model’s vocabulary with the smallest edit distance to the user LLM’s token. However, this method might lead to matching tokens with the smallest edit distance but irrelevant semantics, such as erroneously matching “ultimate” with “estimate”.

To overcome the aforementioned issues, we propose a new mapping strategy: Prefix-Match with Minimal Distance (PM-MinED). This strategy not only considers the edit distance between tokens but also introduces the concept of prefix matching to enhance the accuracy and semantic relevance of the mapping.
As illustrated in Figure 3(c) , in the process of matching the token “ultimate” with the corresponding token in the delta model vocabulary, we initially create a candidate set consisting of tokens that either have “ultimate” as a prefix or are prefixes of “ultimate”, which includes elements such as [“ultimately”, “ult”, “ul”, “u”]. Subsequently, within this candidate set, by comparing the edit distances, the token “ultimately” is identified as the one most closely matching “ultimate”.
Experiment
To evaluate the effectiveness of our method, we conducted experiments on two optimization tasks commonly required for LLMs: instruction tuning and unlearning, with the results presented in Sections 4.1 and 4.2 . Furthermore, we analyzed the impact of different sizes of delta models and the strength coefficient \alpha on performance in Sections 4.3 and 4.4 . Finally, the outcomes of the ablation studies are demonstrated in Section 4.5 , where we investigate the necessity of applying LogSoftmax to the logits of LLM outputs and employing prefix match during token mapping.
Experiment on Instruction Tuning
Instruction tuning refers to the process of fine-tuning pre-trained models to better understand and follow human natural language instructions.
Training Dataset
We utilized the GPT4-Alpaca dataset (Peng et al., 2023 ) to train our delta model. This dataset consists of 52k instruction-following examples, with instructions sourced from Stanford Alpaca data (Taori et al., 2023 ) and responses generated by GPT-4.
Evaluation Method
We employed the AlpacaEval benchmark (Li et al., 2023b ) to evaluate the instruction-following ability of our method. This benchmark includes 805 instructions, with GPT-4 serving as the annotator to compare the output of the tested model against Davinci003’s output, using win rate as the evaluation metric.
Implementation
We introduce tinyllama-110M 2 2 2 https://huggingface.co/nickypro/tinyllama-110M as the delta model, which is based on the Llama (Touvron et al., 2023 ) architecture with a parameter size of 110M, featuring 12 Transformer decoder layers and a hidden size of 768. We use Llama2-7b as the template LLM to guide the delta model in learning to modify the output of the LLM. Llama2-13b , Llama2-70b , and Mistral-7B are selected as user LLMs to assess the effectiveness of the delta model on LLMs with different parameter sizes and vocabularies.
Baseline Methods
a) LoRA fine-tuning (Hu et al., 2022 ) : A parameter-efficient fine-tuning method that freezes the pre-trained model weights during training and incorporates trainable rank decomposition matrices into the Transformer layer to reduce the number of trainable parameters. As an immovable method, LoRA is not suitable for direct comparison with our approach but serves as a performance upper bound for reference. b) Proxy-tuning (Liu et al., 2024a ) : A method that does not directly fine-tune the model itself but selects a smaller-scale model within the model family as an anti-expert, fine-tunes it as an expert, and leverages the difference in logits between the expert and anti-expert during decoding for larger models. To facilitate a fair comparison, we use tinyllama-110M as the anti-expert, fine-tuned on GPT4-Alpaca as the expert. This approach cannot be directly applied to models with different vocabularies .
Analysis
The results of instruction tuning are shown in Table 1 , where we observe that:

A single portable delta model can enable LLMs with different parameter sizes and vocabularies to achieve the ability to follow instructions. Furthermore, this ability is not constrained by the template model’s capabilities, and as the user model’s capabilities increase, the ability of the user model to follow instructions also increases. This indicates that the logits transformation for enabling a model to follow instructions can be applied to a wide range of LLMs, which is consistent with the findings in Section 2 .

Our approach achieves better performance than Proxy-tuning with the same trainable parameters and fewer inference costs, as our delta model can alter the outputs of LLMs through training with the template model, while Proxy-tuning is constrained by the performance of the anti-expert model.

Our method’s performance is not as good as LoRA’s, as LoRA incorporates new tunable parameters in each Transformer layer, enabling deep interactions between the LoRA module and the model. However, this approach also prevents the LoRA module from being used as a portable neural network for models in different parameter spaces.
Experiment on Unlearning
Unlearning refers to the process of making a model forget specific information from the training data in order to prevent privacy leakage.
Evaluation Method
We utilized the TOFU benchmark (Maini et al., 2024 ) to evaluate our approach. The test data consists of four subsets of QA pairs, namely Forget Set, Retain Set, Real Author, and World Fact. The Forget Set and Retain Set contain fictitious author information, with the Forget Set representing the information to be forgotten and the Retain Set representing the information to be retained. 3 3 3 The test data for Forget Set and Retain Set are paraphrased and perturbed training data, serving as the ground truth answer and incorrect answer, respectively. Real Author and World Fact are used to test the impact of unlearning on other knowledge within the model. Following TOFU’s setting, we employed the following three evaluation metrics: ROUGE-L evaluates the matching degree between the output of the tested model and the ground truth answer; Probability assesses the conditional probability of the tested model outputting the correct answer; Truth Ratio calculates a ratio that compares the likelihood of its correct answer to an incorrect answer.
Implementation
Initially, we trained Llama2-7b-chat , Llama2-13b-chat , and Mistral-7B-Instruct on the Forget Set and Retain Set of TOFU to memorize fictitious author information, resulting in Llama2-7b-TOFU , Llama2-13b-TOFU , and Mistral-7B-TOFU .
We selected tinyllama-110M as the delta model, with the model learning fictitious author information, Llama2-7b-TOFU , serving as the template LLM, and Llama2-13b-TOFU and Mistral-7B-TOFU serving as user LLMs. During the training phase, we employed a gradient difference strategy, specifically conducting gradient ascent on the Forget Set and gradient descent on the Retain Set. The loss function can be expressed as:

\mathcal{L}_{diff}=-\mathcal{L}(S_{F})+\mathcal{L}(S_{R}) (7)

Here, \mathcal{L} represents the cross-entropy loss, S_{F} denotes the Forget Set, and S_{R} denotes the Retain Set.
Baseline methods
a) LoRA fine-tuning , which involves training each model separately. b) \delta -Unlearning (Huang et al., 2024 ) fine-tunes two Llama2-7b-chat models in both directions to prevent and encourage information leakage, and guides Llama2-13b-chat during decoding to avoid outputting private information using the changes in logits between the two. This method necessitates inferring three models during the inference, incurring substantial computational costs.
Analysis
The results are shown in Table 2 . The delta model trained with Llama2-7b-TOFU enables models with varying parameter scales and vocabularies to achieve unlearning effects. Its performance is comparable to LoRA while offering the capability of improving multiple models in a single training session. Furthermore, compared to \delta -Unlearning , our approach achieves superior performance at less than half the inference cost.
Impact of Parameter Size of Delta Model on Performance
We experimented with delta models of smaller parameter sizes, specifically the tinyllama-42M 4 4 4 https://huggingface.co/nickypro/tinyllama-42M and tinyllama-15M 5 5 5 https://huggingface.co/nickypro/tinyllama-15M models, and tested their performance on the first 50 data points of AlpacaEval. The results are presented in Table 3 . In most cases, we find that reducing the delta model’s parameter size leads to a decrease in performance, but the performance does not rapidly decline as the parameter size decreases. When the delta model is applied to Llama2-70b , a larger delta model size may have a negative effect. We believe this is because the 70B LLM, with a massive parameter size, does not need to make significant adjustments to the output to acquire the ability to follow instructions, and an overly large parameter size in the delta model may lead to overfitting. This indicates that a delta model used for adjusting LLM output logits does not necessarily require a large number of parameters. It also demonstrates the significant potential for small models to assist large models.
Impact of Strength Coefficient on Performance
We experimented with different strength coefficients \alpha and observed their impact on performance.

For instruction tuning, we tested values within the range of [0.5, 2] and evaluated them on the first 50 data points of AlpacaEval. As shown in Figure 4(a) , we found that the performance was optimal when the \alpha value was 1.0, and increasing or decreasing \alpha resulted in decreased performance. This is similar to causing the delta model to overfit or underfit, so we explored whether adjusting the \alpha value during inference could counteract the overfitting and underfitting during training. Specifically, the delta model performed best after training for four epochs, and we selected the checkpoint after training for two epochs as the underfitting scenario and the checkpoint after training for eight epochs as the overfitting scenario. We found that adjusting the \alpha value could counteract the overfitting or underfitting during training. The underfitting delta model had a win rate of 54 when \alpha was 1.0, which increased to 64 when \alpha was 2.0. Similarly, the overfitting delta model had a win rate of 48 when \alpha was 1.0, which increased to 64 when \alpha was 0.75. This phenomenon indicates that our training method offers a considerable degree of fault tolerance, even if incorrect epoch parameters are set during the training phase, it is still possible to achieve the desired performance by adjusting the strength coefficient \alpha .

For unlearning, we found that adjusting the value of \alpha can serve as a balance between forgetting and retaining . As shown in Figure 4(b) , increasing the value of \alpha can prevent the model from outputting more sensitive information, but it may also lead to the loss of necessary information. Conversely, decreasing the value of \alpha can allow the model to retain more essential information, but it may result in the model outputting more sensitive information. Users of LLM can flexibly adjust the \alpha value based on the actual circumstances.
Ablation Study
In the ablation study, we examined the impact of not applying LogSoftmax to the logits output by the LLM and the effect of omitting prefix matching during token mapping on performance. As demonstrated in Table 4 , our findings reveal that the removal of LogSoftmax results in a decline in performance. This suggests that projecting the logits output by the LLM into the logarithmic space can reduce the gap between training and inference. Similarly, the absence of prefix matching in token mapping diminishes the supportive effect of the delta model on the LLM. This indicates that prefix matching has a higher priority than selecting tokens with the minimal edit distance. Eliminating prefix matching could lead to the aggregation of token logits that are semantically unrelated.
Related Work
In this paper, we focus on modifying the outputs of the base language model. The related work can be broadly categorized into two approaches: training-intensive methods and decoding-time methods.
Training-Intensive methods
One of the most effective methods for adapting large language models (LLMs) to specific downstream tasks involves updating only a subset of the model parameters, rather than the entire set.
For instance, LoRA (Hu et al., 2022 ) achieves parameter updates by decomposing them into trainable low-rank vectors.
Prefix-tuning (Li and Liang, 2021 ) introduces a series of continuous, task-specific vectors at the beginning of the input sequence for task adaptation.
Adapter-tuning (Houlsby et al., 2019 ) integrates compact, trainable modules within the layers of a pre-trained model to facilitate transfer learning.
BitFit (Zaken et al., 2021 ) selectively updates individual bias vectors in the model’s parameters.
Although these methods enhance performance in downstream tasks, they still require significant computational resources.
Decoding-Time methods
To reduce training costs, some researchers have explored decoding-time methods.
For example, DExperts (Liu et al., 2021 ) enhances or suppresses certain text attributes by integrating expert and anti-expert models during decoding.
GeDi (Krause et al., 2021 ) employs smaller LMs as generative discriminators to steer the output of larger LMs, enhancing safety and control.
Proxy-tuning (Liu et al., 2024a ) modifies the predictions of an untuned larger model towards a desired outcome.
ITI (Li et al., 2024 ) adjusts activation values during inference to generate more accurate responses.
EFT (Mitchell et al., 2023 ) employs two smaller LMs to emulate the behavior of a larger LM without the need for additional training.
Although these methods are effective, they often require the use of two additional language models during inference, increasing inference costs.
Conclusion and Limitation Discussion
In this paper, we introduced a training method named CMC, designed to improves multiple LLMs in one-time training, enabling LLM owners who lack data and computational resources to improve their models at a lower cost.
Through comparative analysis of the logits shifts in different LLMs before and after fine-tuning, we observed a similarity in these shifts. Based on this observation, we introduced a portable tiny delta model to fit the logits shifts of LLMs, enabling the adjustment of outputs for LLMs of varying sizes and vocabularies. Our experiments in instruction tuning and unlearning tasks have demonstrated the effectiveness of our method.

However, our approach has certain limitations. The performance of the delta model is constrained by the scope of its vocabulary. If the vocabulary of the user’s LLM contains tokens from languages not covered by the delta model’s vocabulary, then the logits of these linguistically diverse tokens will not be adjusted accordingly.
A.1 Instruction Tuning
During the training of the delta model, we set the learning rate to 2e-4, batch size to 64, and trained for 4 epochs.

For LoRA fine-tuning, for both Llama2-7b and Mistral-7B models, we set r to 140 and alpha to 280, while for Llama2-13b , r is set to 90 and alpha to 180. For all models, the learning rate is 2e-5, batch size is 32, LoRA target are [q_proj,k_proj,k_proj], and the training lasted for 2 epochs.

When training the expert models for Proxy-tuning, the learning rate was set to 2e-4, batch size to 64, and trained for 16 epochs. We also experimented with varying the number of training epochs and found that performance was best with 16 epochs.
A.2 Unlearning
In training the delta model, we set the learning rate to 1e-4, batch size to 16, and trained for 20 epochs.

For LoRA fine-tuning, across all models, r was set to 32, alpha to 64, with a learning rate of 1e-4, batch size of 16, and training spanned 5 epochs.
A.3 Preliminaries
The hyperparameter settings for Section 2 were consistent with those outlined for instruction tuning.
Figure 1 : Cross-model control could apply the fine-tuning outcomes of one model to other models; Cross-model control could apply the fine-tuning outcomes of one model to other models
(a) Logits shifs on Llama2-13b and Llama2-7b .; (b) Logits shifs on Mistral-7B and Llama2-7b .; (c) Logits shifs on Mistral-7B and Llama2-13b .; Logits shifs on different models exhibit a high degree of similarity.
(a) Training stage; (b) Inference stage; (c) Token mapping; Overview of cross model control
(a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Instruction tuning results on AlapcaEval (Win %). In cross-model control, all base models incorporate the same delta model, which is trained using the Llama2-7b as the template model.
Unlearning results on TOFU benchmark. All chat models incorporate
the same delta model, which is trained using the Llama2-7b-TOFU as the template model. Better scores are bolded.
Different delta model size on first 50 data points of AlpacaEval (win %).
Ablation study
Average Distance between logits shifts. Smaller distances mean more similarities