Introduction
In recent years, there has been an increasing number of large language models (LLMs) with varying parameter scales and vocabularies, whose outstanding performance has significantly impacted human society (Achiam et al., 2023 ; Zhao et al., 2023 ) . At the same time, although large pre-trained language models have gained the ability to handle various natural language processing tasks after pre-training, they generally face a series of common optimization needs to meet specific application requirements or ethical standards. For instance, in the case of instruction following (Wei et al., 2022 ) , after pre-training, vanilla models typically require instruction tuning to develop the capacity to comprehend user instructions accurately.
Alternatively, unlearning and detoxification are also necessary considerations (Gehman et al., 2020 ; Chen and Yang, 2023 ) . During the training process, models may encounter data containing real-world personal privacy information or content that is harmful, offensive, or prejudiced. This could lead them to output these information during the inference stage (Wei et al., 2023 ; Huang et al., 2022 ) . When deploying to a large number of users, it is crucial to avoid outputting these content.

However, existing methods usually could only optimize a single target model at a time, such as fine-tuning (Hu et al., 2022 ; Lester et al., 2021 ; Liu et al., 2024b ) , retraining (Keskar et al., 2019 ) , or activation editing (Li et al., 2023a ; Leong et al., 2023 ) . These methods require altering the model’s parameters or adding new parameters, which must align with the original parameters. These newly added or modified parameter values cannot be applied to models with different structures. Furthermore, although some guided decoding methods (Liu et al., 2021 ; Krause et al., 2021 ; Liu et al., 2024a ) can be used for a few models of different scales within the same model family, they cannot be applied to models with other vocabularies. Moreover, these methods introduce a significant inference burden. For in-context learning (Dong et al., 2022 ) , although this method can alter the behavior of models through natural language prompts, it falls short in satisfying the requirements of complex tasks such as instruction following or unlearning, which are better accomplished through fine-tuning.
Consequently, a critical question arises: When model owners face constraints on data and computational resources, preventing them from directly fine-tuning their models, can they effectively leverage the fine-tuning outcomes of other LLMs at a lower cost to improve their models?

To solve this problem, we sought to explore the similarities in the fine-tuning effects across models with different parameter scales and vocabularies. We define the effect of fine-tuning as the change in the model’s output logits after the fine-tuning process, compared to the logits before fine-tuning. We discovered that the shifts in logits across different models exhibit a high degree of similarity. It further inspires us to think: Could a portable neural network model be utilized to alter the output logits of various models? Thereby enabling a diverse range of models to achieve their optimization requirements through this neural network model.

In this paper, we propose Cross-model Control (CMC), a method that could improve multiple LLMs in one-time training with a portable tiny language model. As demonstrated in Figure 1 , we introduce a tiny language model with significantly fewer parameters than mainstream LLMs. This model is trained alongside a frozen template LLM, enabling the tiny language model to alter the logits output by the LLM. Subsequently, to facilitate its application across models with different vocabularies, we introduce the strategy of prefix match with minimum edit distance (PM-MinED), a lightweight approach for aligning the vocabularies of the user LLM and the tiny language model at the token level. Through this approach, we achieve the training of a single model that concurrently improves multiple models. We conducted extensive experiments on instruction tuning and unlearning tasks, demonstrating the effectiveness of CMC, where a tiny language model with only 15 million parameters can empower a large model with 70 billion parameters, which is thousands of times larger. Our contributions can be summarized as follows:

To the best of our knowledge, we are the first to propose a training method that improves multiple models in one-time training. This approach facilitates the multiple utilizations of fine-tuning outcomes, enabling LLM owners who lack data and computational resources to improves their models. Showcasing a novel method for model enhancement.

We conducted a detailed analysis of the similarities in fine-tuning across different models and discovered that the shifts in logits for the same task are similar across various models.

Through extensive experimentation, we have demonstrated the effectiveness of our proposed method. Moreover, we discovered that language models with minimal parameter sizes possess significant potential in assisting LLMs.