Venue: NAACL 2025

# Transferable Post-training via Inverse Value Learning

Xinyu Lu\({}^{1,2}\), Xueru Wen\({}^{1,2}\) Yaojie Lu\({}^{1}\), Bowen Yu\({}^{3}\), Hongyu Lin\({}^{1}\), Haiyang Yu\({}^{3}\), Le Sun\({}^{1,*}\), Xianpei Han\({}^{1}\), Yongbin Li\({}^{3,*}\)

Corresponding authors. \({}^{1}\)Chinese Information Processing Laboratory Institute of Software, Chinese Academy of Sciences

\({}^{2}\)University of Chinese Academy of Sciences

\({}^{3}\)Alibaba Group

{luxinyu02021,wenxueru2022,luyaojie,hongyuyu,sunle,xianpei}@iscas.ac.cn

###### Abstract

As post-training processes utilize increasingly large datasets and base models continue to grow in size, the computational demands and implementation challenges of existing algorithms are escalating significantly. In this paper, we propose modeling the changes at the logits level during post-training using a separate neural network (i.e., the value network). After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference, enables them to achieve similar capability enhancements. We systematically investigate the best practices for this paradigm in terms of pre-training weights and connection schemes. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes within the same family, models undergoing continuous pre-training within the same family, and models with different vocabularies across families. In certain cases, it can achieve performance comparable to full-parameter fine-tuning. Furthermore, we explore methods to enhance the transferability of the value model and prevent overfitting to the base model used during training.1

Footnote 1: Our code is open-source at [https://github.com/icip-cas/inverse-value-learning](https://github.com/icip-cas/inverse-value-learning)

_Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 4436–4447

April 29 - May 4, 2025 \(\copyright\)2025 Association for Computational Linguistics

## 1 Introduction

Post-training of Large Languge Models (LLMs), which can be referred to as fine-tuning a pre-trained model for helpfulness, honesty, and harmlessness Askell et al. (2021), is becoming increasingly important Dubey et al. (2024). A well-designed post-training process can maximize the potential injected during pre-training while remaining aligned with human intentions and values.

However, post-training has become increasingly intricate Dubey et al. (2024); Yang et al. (2024); Adler et al. (2024). A well-designed post-training procedure typically consists of two main stages: Supervised Fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. (2022). Each stage involves iterative data synthesis, quality control, and training pipelines Cao et al. (2024), which collectively introduce significant complexity to the model training workflow. Furthermore, as model sizes continue to grow, the processes of both sampling and training larger-scale models are increasingly resource-intensive.

To address these challenges, we propose a novel framework for transferable post-training that efficiently adapts models of varying sizes and families with minimal reconfiguration. This framework distills the post-training process to another network by leveraging the _logits space_ as a shared interface for model adaptation and capability transfer. In contrast to parameter or representation spaces, the logits space is more consistent across different models and can serve as a universal channel for communication Hinton (1999). By adjusting the base model's logits during decoding, we can adapt any base models to new tasks without modifying

Figure 1: Illustration of Inverse Value Learning. V.M. denotes Vocabulary Mapping.



their original parameters. This allows for efficient model adaptation, avoiding the resource-intensive process of full fine-tuning.

Specifically, we propose _transferable posttraining via inverse value learning_, a framework that captures changes at the logits level using a separate neural network, referred to as the value network. This network is trained on a lightweight base model using demonstration data to capture the necessary adaptations of the logits in the post-training process. During inference, the value network can be seamlessly integrated with various pre-trained base models, enabling them to adopt the learned adjustments without requiring further training, thus facilitating efficient model transfer of model capabilities.

We systematically investigate practical implementations for this framework. Specifically, we first study the importance of pre-trained weights in modeling the residual logits. Then, we examine two schemes for integrating the value network with pre-trained models: Cascade and Residual connections. Our experiments reveal that the residual connection scheme, where the value network predicts the delta logits based solely on previous text inputs, demonstrates superior transferability and efficiency. To improve the generalization ability of the value network, we incorporate regularization techniques, such as norm constraints, to mitigate overfitting to the base model during training. Additionally, we introduce a vocabulary mapping algorithm to facilitate effective cross-vocabulary transfer.

We assess our approach on multiple datasets and tasks, including general instruction following, zero-shot capabilities, and few-shot learning scenarios. Our results show that the value network exhibits broad transferability across pre-trained models of varying parameter sizes within the same model family, models undergoing continual pre-training, and even across different model families. In certain cases, our framework reaches close performance with full-parameter finetuning. This highlights the potential of our method for efficient and practical application.

Our main contributions are as follows:

1. To address the complexity and resource demands of the post-training process, we propose a transferable post-training framework based on the logits space. By modeling the residuals in the logits space, we can transfer post-training capabilities across models of varying sizes and families without altering the original parameters or requiring additional training.
2. We systematically studied the implementations of this framework, including pretrained weights, connection schemes, regularization methods, and cross vocabulary transfer algorithm. Through comprehensive experiments across diverse settings and datasets, we demonstrate the effectiveness of our proposed framework and broad transferability of the trained value models.

## 2 Problem Statement and Background

The soaring computational costs of full-parameter updates with growing model sizes have driven researchers to explore efficient alternatives like low-rank parameter updates Hu et al. (2022); Dettmers et al. (2024) or representation interventions Wu et al. (2024) while freezing pre-trained weights. However, inconsistencies between the parameter spaces and representation spaces across different models stand as a key challenge for transferable post-training. In contrast, the logits space has better sharing properties and can serve as a communication channel for model interaction, enabling the transfer of capabilities between models Hinton (1999). Formally, the impact of post-training on the logits of a pretrained model can be formally expressed as:

\[\log p_{\text{post}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})= \log p_{\text{base}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})\] \[+ \boxed{\log p_{\Delta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}\]

where \(\log p_{\Delta}\) represents the change in the logits space during post-training. A series of inference-time proxy tuning studies focus on exploring \(\log p_{\Delta}\). Mitchell et al. (2023) and Liu et al. (2024) obtained \(\log p_{\Delta}\) by taking the difference between an existing instruction-following model and its corresponding base pre-trained model, i.e., \(\log p_{\Delta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})=\log\frac{p_{\pi^{*}}(y_{t}|\mathbf{ x},\mathbf{y}_{<t})}{p_{\text{base}}(y_{t}|\mathbf{x},\mathbf{y}_{<t})}\). This difference can be regarded as an advantage function Mitchell et al. (2023), or an implicit reward signal Rafailov et al. (2024).

Another line of research using logits-based steering focuses on controlling model outputs to satisfy predefined attributes (e.g., style, toxicity, length). A key characteristic of these works is their reliance on trajectory-level rewards, such as scores from toxicity or topic classifiers. These studies investigate the 

conversion of trajectory-based rewards into token-level dense guidance (Khanov et al., 2024; Krause et al., 2020; Yang and Klein, 2021; Liu et al., 2021) to achieve attribute-compliant text generation.

Compared to the logits arithmetic works, we provide a more practical training-based solution that reduces the number of models needed during inference from three to two. Furthermore, in contrast to controllable text generation methods that rely on predefined outcome-based reward signals, we learn the reward function implicitly by modeling using a separate neural network trained with demonstrations, thus efficiently achieving the transfer of post-training capabilities.

## 3 Preliminary Study

### Task Definition

Let denotes an input text, denotes the model generated tokens. We utilize a pre-trained model with fixed parameters that outputs logits :

To adapt this model to a new task without altering , we introduce a delta model with parameters , producing steering logits:

The final logits are computed by combining the pre-trained model's logits with the delta model's:

where stop_gradient indicates that gradients are not propagated through having during back-propagation.

Let represent the label probabilities (e.g., one-hot next-token distribution in supervised fine-tuning). The training objective is to minimize the Cross-Entropy loss function:

By optimizing , we aim to align with the target label distribution , thereby enhancing performance on the new task while keeping the pretrained parameters unchanged. Importantly, this process is equivalent to inversely learning an action-value function based on demonstrations and , where the states are and the actions are . In this context, the value function learned from training demonstrations assigns higher scores to actions that align with the demonstrated behaviors at each state . Therefore we refer to this process as _inverse value learning_.

AdvantagesThis task formulation has a set of advantages, including:

1. The decoupling of delta logits allows for a more thorough investigation when errors occur, enabling us to attribute them to either the pre-training process or the post-training process.
2. This framework enables both Weak-to-Strong Burns et al. (2024) and Strong-to-Weak generalizations by introducing different scales of models to parameterize and [Mitchell et al., 2023]. Since the value model can be trained with small base models and plugged into the larger ones in the Weak-to-Strong setting (), the overall training time can be reduced compared to training a strong model directly.
3. The formulation is compatible with a variety of loss functions, including KL-based teacher-student distillation, pairwise optimization, and standard cross-entropy adopted in this paper.
4. Operating in the logit space allows for cross-model guidance in a plug-and-play manner, enabling capability transfer across models with the same or different vocabularies, thus minimizing the need for redundant training.

LimitationsThis formulation can introduce additional inference costs, as it requires obtaining logits from both the pre-trained and value models, resulting in extra computational overhead. Nevertheless, with slightly more computational resources and by adapting residual architecture, which will be discussed in Section 4.1, this will not introduce additional inference latency. Moreover, techniques like speculative decoding (Chen et al., 2023; Leviathan et al., 2023) could be employed to further optimize inference time. We leave these optimizations to future work.

### Linear Probes are Insufficient for Inverse Value Learning

Given the recent prominence of the surface alignment hypothesis (Zhou et al., 2023a; Lin et al., 2024), which posits that language models acquire 

the majority of their knowledge during pre-training. We have the reason to investigate whether a minimal set of parameters could sufficiently model the transformation between pre-trained and fine-tuned logits. To address this question, we conducted a preliminary experiment utilizing a single linear layer (see Appendix A.1 for details) to model \({\mathbf{z}_{\Delta}}\). We trained this model on ShareGPT [20], a typical instruction-tuning dataset, and evaluated its efficacy.

As illustrated in Figure 2, we find that a single linear probe is insufficient to effectively model such a transformation (more than a 3-point drop in MTBench scores). While the combined model successfully mimics the assistant's conversational style, it fails to effectively apply the task-specific knowledge needed for problem-solving (typical failure modes shown in Figure 5 in the Appendix). This finding suggest that the relationship between pretrained and fine-tuned logits is more sophisticated than initially hypothesized.

We further investigate whether the performance gap could be bridged by increasing model capacity while maintaining random initialization. Specifically, we employed a randomly initialized 7B Llama architecture as \(\pi_{\Delta}\), following the identical training procedure to guide the alignment of an 1.1B pre-trained Llama model. While this model marginally outperforms the shallow linear probe in instruction-following capabilities, it still significantly underperforms compared to full fine-tuning. These results suggest that leveraging pre-trained weights is crucial for effective delta logits prediction.

## 4 Methods

### Connection Schemes

Based on the formulation and observation in Section 3, we design two distinct architectures to transform a pre-trained model into a value network and integrate it with another pre-trained model, namely **Cascade** and **Residual**, as illustrated in Figure 2(a). It is important to note that both approaches maintain the internal structure of the LLM unchanged, focusing solely on the connection schemes between the pre-trained model and the value network.

Cascade.In the cascade architecture, the value model receives both the logits information from the pre-trained model and the text embedding as the inputs. The fusion of these inputs is formulated as:

\[\mathbf{h}_{\text{final}}=p_{\text{base}}\mathbf{W}_{e}\oplus\mathbf{h}_{\text{embed}}\]

where \(p_{\text{base}}\) represents the pre-trained probabilities, \(\mathbf{W}_{e}\) is the pre-trained embedding matrix, and \(\mathbf{h}_{\text{embed}}\) denotes the original text embedding, \(\oplus\) is the fusing operator (set as addition in this work).

An alternative hypothesis suggests that the textual embedding input may be redundant, as the value model could potentially achieve comparable performance by exclusively utilizing the logit information from the pre-trained model. In this case:

\[\mathbf{h}_{\text{final}}=p_{\text{base}}\mathbf{W}_{e}\]

The cascade model takes \(\mathbf{h}_{\text{final}}\) as input and outputs the value scores:

\[\mathbf{z}_{\Delta}(y_{t}|\mathbf{x},\mathbf{y}_{<t})=\pi_{\Delta}(\mathbf{h}_{\text{final}}; \theta_{\text{cascade}}).\]

Full Residual.The residual architecture specifies that the value model has no access to the logit information of the pre-trained model. Instead, it predicts the residual for the next token position based solely on the previous tokens. This design allows the value model to begin forward without waiting for the pre-trained model's inference to complete. However, this approach may potentially limit transferability, as it cannot steer predictions based on logits from the inference-time pre-trained model which it is plugged into, which may differ significantly from the training-time one.

Figure 2: MT-Bench scores of models with different sizes using linear probes for modeling \(z_{\Delta}\) and full fine-tuning.

 

The full residual model takes previous tokens \({\mathbf{x}}\) and \({\mathbf{y}}_{<t}\) as inputs and outputs the value scores:

\[{\mathbf{z}}_{\Delta}(y_{t}|{\mathbf{x}},{\mathbf{y}}_{<t})=\pi_{\Delta}({\mathbf{ x}},{\mathbf{y}}_{<t};\theta_{\text{residual}}).\]

### Plug-and-play transferability

Once a value model is trained on one backbone, it can be plugged into other pre-trained models without further training, potentially achieving similar performance and behaviors as if it were post-trained. This capability stems from the feature of logits space steering Mitchell et al. (2023). To further expand the boundaries of transferability, we investigate techniques to mitigate overfitting to the training-time backbone and methods for cross-vocabulary model transfer.

#### 4.2.1 Regularization via Norm Constraint

Inevitably, the value model tends to overfit the backbone it is trained with, thereby reducing its generalization capability on other pre-trained models. Applying appropriate regularization techniques during training can mitigate this issue. For example, we can directly constrain the sparsity of delta logits by incorporating an L1 norm term in the loss function. Specifically, this can be expressed as:

\[\mathcal{L}=\operatorname*{CE}({\mathbf{z}}_{\text{post}},p_{l})+\lambda\lVert{ \mathbf{z}}_{\Delta}\rVert_{1}\]

where \(\lambda\) is a hyperparameter controlling the strength of regularization.

#### 4.2.2 Vocabulary Mapping

To adapt value-guided decoding to pre-trained and value models trained on different vocabularies and tokenizers, such as plugging a Llama-2-based value model to a Llama-3 family pre-trained model, a vocabulary mapping algorithm is essential. We adopt the MinED mapping algorithm Wan et al. (2024) for per-step mapping of the logits, which identifies mapping relationships between two sequences by minimizing their edit distance. Based on these mapping relationships, we can derive a vocabulary mapping matrix, which is then normalized and regularized to ensure the mapped logit values are not biased. During each decoding step, this mapping matrix is employed to transform the vocabulary space as follows:

\[{\mathbf{z}}_{\text{post}}(y_{t}|{\mathbf{x}},{\mathbf{y}}_{<t}) ={\mathbf{z}}_{\text{base}}(y_{t}|{\mathbf{x}},{\mathbf{y}}_{<t}){\bf W}\] \[+{\mathbf{z}}_{\Delta}(y_{t}|{\mathbf{x}},{\mathbf{y}}_{<t}). \tag{5}\]

where \({\mathbf{W}}\in\mathbb{R}^{n\times m}\) is the obtained transition matrix to map the size-\(n\) pre-trained vocabulary to the size-\(m\) vocabulary of the value model. The details of the algorithm and corresponding evaluation are shown in Appendix B.

Figure 3: Cascade and Residual connection schemes for Inverse Value Learning. (a) Illustration of the two connection schemes. (b) MT-Bench scores for 1.1B models trained on the 1.1B base and generalized to the 7B base model using different connection schemes. “Cascade+” refers to the cascade value model where logits information from the pre-trained models is fused with the original token embeddings (marked as optional in Figure (a)) and used as input for the value model. The final unembedding layer maps the residual stream from hidden space to the logits space, equivalent to the Linear Layer in the original Transformer literature Vaswani et al. (2017).

 

Experiments

### Training Datasets

The experiments are primarily conducted based on the following two training datasets:

ShareGPTis a dialog dataset collected from sharegpt.com, a website collecting user-shared dialog with ChatGPT. This dataset encompasses a wide range of topics, making it suitable as a general instruction tuning set focusing on the instruction-following ability of models. While ShareGPT is the training set of Vicuna (Chiang et al., 2023), the exact data has not been released. For our study, we utilize a version comprising approximately 86K dialogs.

InfinityInstruct-7M (Zhang et al., 2024a)is a large-scale instruction tuning dataset specifically designed to enhance foundational reasoning abilities in code and math. This collection consists of 7M carefully curated instructions, aggregated and filtered from a variety of open-source datasets.

### Pre-trained Backbones

We employed TinyLlama (Zhang et al., 2024b) and the Llama-2 series (Touvron et al., 2023) as backbones for the 1.1B, 7B, 13B, and 7OB parameter scales. To validate broader transferability, we also conduct tests on CodeLlama (Roziere et al., 2023), a model based on Llama-2 that undergoes additional continual pertaining on 20B of code related data. Furthermore, we use Llama-3 (Dubey et al., 2024) to test cross model family generalization ability.

The training details of the value models are shown at Appendix C.

### Tasks

General Instruction Following.We employ MTBench (Zheng et al., 2023) as an open-ended instruction following benchmark and use GPT-4 to judge the response from the models.

Zero-shot capabilities.We select representative datasets in Tulu-2 (Ivison et al., 2023) evaluation suite, including IFEval Zhou et al. (2023b), ToxiiGen Hartvigsen et al. (2022), GSM8K Cobbe et al. (2021), MBPP Austin et al. (2021), HumanEval Chen et al. (2021). For MBPP and HumanEval, we report the Pass@1 scores. We apply corresponding chat templates for the instruction-following models.

Few-shot capabilities.We select representative few-shot datasets originally employed in Open LLM Leaderboard v12 for testing the few shot capabilities, including ARC (25-shot) Clark et al. (2018), TruthfulQA (0-shot) Lin et al. (2021), MMLU (5-shot) Hendrycks et al. (2020), Winograde (5-shot) Sakaguchi et al. (2019), Hellaswag (10-shot) Zellers et al. (2019), .

Footnote 2: [https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/archive](https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/archive)

### Results

Which connection scheme enables better transfer for Inverse Value Learning?Prior to conducting extensive transfer testing, we first evaluate which of the proposed connection schemes is more conducive to modeling \(\mathbf{z}_{\Delta}\). Using an 1.1B model fine-tuned as the value model, we study the effects of the two schemes in Figure 2(a) on both the 1.1B backbone (used during training) and the 7B backbone (used for inference-time transfer). As shown in Figure 2(b), for the "Cascade+" and "Residual" schemes, difference in connection schemes did not result in significant variations in general instruction-following ability, especially when transferring to other pre-trained models. However, the token embedding serve as essential input features for the value model's functionality ("Cascade" _v.s._ "Cascade+"). Meanwhile, integrating logits from the base model slightly improves performance on the training backbone. However, when transferring to a larger backbone, there is a decrease in performance compared to the full residual structure. We attribute this to two potential factors: 1) Excessive input signals introduce noise to the cascade scheme, and 2) The value model in the cascade scheme no longer maintains a purely autoregressive form (i.e., it predicts the \(\mathbf{z}_{\Delta}\) based on the pre-trained logits at the same time step), widening the gap between pre-training and fine-tuning. Consequently, we adapt the full residual scheme for subsequent experiments in Table 1, given its superior performance and efficiency.

Wide-ranging transferability.The inverse learned value model demonstrated broad generalizability across models of different parameter scales, consistent with the findings of Mitchell et al. (2023). When an 1.1B parameter value model, trained on an 1.1B parameter backbone, is transferred to a 7B parameter base model, it retains large portion of its instruction-following capability 

compared to full finetuning. Additionally, when a 7B parameter value model, jointly trained with a 7B base model, was transferred to a 13B parameter model, the instruction-following performance matched that of full finetuning. This transferability can similarly be scaled up to 70B.

Table 1 further illustrates the potential of our approach in continued pre-training scenarios within the same model family. For instance, in the case of Codellama, the 1.1B value model can be directly applied to guide the base model in acquiring instruction-following capabilities while preserving its strong performance in coding tasks.

The gap between inverse value learning and full fine-tuning narrows with the introduction of more direct supervision and larger value models.Encouragingly, when we evaluate the inverse value learning paradigm in the context of code and math post-training, which requires training on several million instruction-response pairs, we observe corresponding performance gains. Notably, these gains are transferable, even though the 1.1B backbone models perform poorly on code and math tasks. For instance, 1.1B value models, trained with 1.1B base models on InfinityInstruct, substantially improve the performance of 13B models on benchmarks such as GSM8K, MBPP, and HumanEval, outperforming other value models and full finetuning methods on the ShareGPT dataset. Furthermore, the overall performance of inverse

\begin{table}
\begin{tabular}{c|c|c c c c|c c c c c} \hline \multirow{2}{*}{Methods} & IF & \multicolumn{5}{c|}{Zero-Shot} & \multicolumn{5}{c}{Few-Shot} \\ \cline{2-14}  & MT-Bench & IFEval & GSM8K & Toxicgeni & MBPP & HumanEval & ARC & Hellaswag & MMLU & TruthfulQA & Winogrande \\ \hline \multicolumn{14}{c}{Tinylama-1.1B} \\ \hline Full & **3.84** & **16.1** & 1.97 & 26.8 & 14.1 & 5.37 & 27.6 & 40 & **26.4** & **34** & 57.8 \\ \hline
1.1B & 3.62 & 15.2 & 2.43 & **21.1** & 10.6 & 8.17 & 28.4 & 44.1 & 25.2 & 30.4 & 57.5 \\
1.1B norm & 3.52 & 14.6 & **2.65** & 62.8 & **17.9** & **15.4** & **31.4** & **44.9** & 25.6 & 30 & **59.6** \\ \hline \hline \multicolumn{14}{c}{Llama-2.7B} \\ \hline Full & 5.93 & 23.8 & 10.6 & 49.8 & 26 & 17.6 & **51.9** & 59.5 & **50.1** & **41.5** & **74.3** \\ \hline
1.1B & 4.62 & 15.5 & 4.7 & **44.3** & 15.8 & 11.3 & 42.7 & 55 & 43.2 & 31.1 & 66.2 \\
1.1B norm & 4.55 & 15.3 & 5.61 & 62.8 & 20.4 & 15.4 & 48.5 & 57.9 & 44.2 & 31.8 & 72.5 \\
1.1B infinity & 4.06 & 17 & **27.2** & 65.3 & **30.9** & **18.3** & 48.3 & 58.6 & 45.8 & 30.2 & 71.2 \\ \hline
7B & 5.96 & 24.4 & 13.4 & 58.3 & 21.4 & 13.9 & 51 & **61** & 49.3 & 40.1 & 72 \\
7B norm & **6.03** & **32.7** & 13.5 & 45.2 & 23.4 & 14.8 & **51.9** & 60.9 & 49.6 & 39.9 & 70.6 \\ \hline \hline \multicolumn{14}{c}{Llama-2.13B} \\ \hline Full & 6.23 & 36.3 & 18.7 & **5.03** & 24.3 & 22.8 & 49.2 & 60.6 & **53.9** & **41.8** & 72.3 \\ \hline
1.1B & 4.74 & 15.5 & 6.67 & 39.7 & 16.4 & 13.6 & 46.1 & 57.2 & 52 & 29.3 & 70.1 \\
1.1B norm & 4.65 & 14.8 & 10.6 & 55.8 & 28.5 & 15.9 & 49.1 & 59.1 & 52.9 & 31.8 & **76** \\
1.1B infinity & 4.45 & 18.7 & **30.3** & 77 & **33.1** & **23.7** & 52.2 & 61.2 & 53.7 & 30.3 & 74.6 \\ \hline
7B & **6.34** & **26.3** & 18.3 & 56.8 & 24.9 & 19.6 & 56.1 & **63.6** & 53.4 & 37.9 & 74.4 \\
7B norm & 6.22 & 25.3 & 17.5 & 39.1 & 27.7 & 15.9 & **56.6** & **63.6** & 53.6 & 37.7 & 74.8 \\ \hline \hline \end{tabular} 
\begin{tabular}{c|c|c c c c c|c c c c} \hline \multirow{2}{*}{Full} & **7.08** & **46.6** & 35.6 & **0** & 36.6 & **41.9** & **62.9** & 67.8 & **69.3** & **48.4** & 80.5 \\ \hline
1.1B & 4.92 & 16.6 & 12.4 & 42.1 & 27.9 & 20.4 & 52.6 & 60.5 & 68.4 & 31.6 & 77.3 \\
1.1B norm & 5.37 & 16.8 & 24.9 & 70 & 37.8 & 28.6 & 62.8 & 66.7 & 67.9 & 37.1 & **82.9** \\
1.1B infinity & 4.86 & 20.1 & **44** & 71 & **42.9** & 33 & 58.6 & 66.4 & 68.5 & 36.1 & 81 \\ \hline
7B & 6.56 & 33.3 & 27.9 & 43.6 & 29.4 & 25.8 & 62.1 & **68.2** & 63.6 & 42.1 & 78.6 \\
7B norm & 6.48 & 34.6 & 28.8 & 27.5 & 33.1 & 25.1 & 62.8 & **68.2** & 64.1 & 41.9 & 79.3 \\ \hline \hline \multicolumn{14}{c}{Codellama-7B} \\ \hline Full & 5.47 & **33.4** & 15.4 & 83.2 & **39.4** & **43.6** & 40.6 & 48.3 & 41.6 & 34.4 & 64.5 \\ \hline
1.1B & 3.94 & 15.7 & 3.79 & 48.2 & 28.1 & 24 & 35.6 & 45.9 & 35.2 & 29.1 & 59.6 \\
1.1B norm & 3.99 & 15.9 & 4.85 & 62.3 & **39.4** & 31 & 39.4 & 46.9 & 38.1 & 29.5 & 64.4 \\
1.1B infinity & 3.51 & 19.8 & **27.2** & 65.2 & 31 & 18.4 & 38.9 & 49.1 & 38.5 & 29.6 & 64.7 \\ \hline
7B & **5.64** & 31.2 & 12 & 56.8 & 30.9 & 26.8 & **47.9** & 53.9 & 49.2 & **38.2** & **68.4** \\
7B norm & 5.54 & 31.2 & 16.8 & **40.6** & 33.2 & 25.1 & 47.8 & **54.1** & **49.7** & 37.4 & 68.3 \\ \hline \end{tabular}
\end{table}
Table 1: Performance comparison of full fine-tuning and inverse value learning on instruction following, zero-shot and few-shot tasks. Note that value models of various parameter scales are trained together with base models of corresponding scales. For instance, “1.1B” indicates a 1.1B value model trained on a 1.1B backbone model. We further use a blue cell tag to denote a transfer setting. “]“ symbol indicates that lower values are better.

 

value learning generally increases with the parameter scale of the value models, demonstrating the scalability of this paradigm.

Additional norm constraint can effectively prevent overfitting to the weak base models.Table 1 also reveals that the value models trained with the norm term, when generalizing to stronger models, typically maintain more robust reasoning and knowledge capability while sacrificing a small degree of instruction-following capability. This phenomenon is particularly pronounced when there is a significant disparity in parameter scale between the value model and the pre-trained model. Consequently, the incorporation of a simple normalization term during the training of small value models can mitigate overfitting to weak base model and enhance weak-to-strong generalization performance.

## 6 Discussions

Can multi-model curriculum fine-tuning improve weak-to-strong generation?Given that all models in the Llama-2 series share the same vocabulary, we designed a curriculum to train an 1.1B value model sequentially on the 1.1B, 7B, and 13B versions, and then evaluated its generalization performance on the powerful 70B model. We maintained consistent training data volume and iteration counts across experiments. As shown in Table 2, our findings indicate that this multi-model curriculum training can also mitigate over-fitting to some extent, functioning similarly to regularization techniques. However, this method necessitates longer training periods and more complex model scheduling processes. Consequently, it does not offer significant advantages in weak-to-strong generation over simpler normalization methods in our observation.

Feasibility of cross-vocabulary transfer.We further explored the transferability of model capabilities across different model families. We transfers Llama-2-vocabulary-based 1.1B parameter value models to Llama-3 using vocabulary mapping at decoding time, comparing it with Llama-3.2-1B guided and full fine-tuning performance. Interestingly, among all settings of the trained value model, only the curriculum-trained value model yielded acceptable generation ability in the cross-model and cross-vocabulary setting. However, as Figure 4 shows, while the transferred model grasped basic conversational abilities, it still underperformed compared to same-vocabulary value models and full fine-tuning. In addition to the generation ability of the value model, we attribute this performance gap partly to tokenization differences: source tokens can be mapped to multiple consecutive target tokens, which are unfamiliar to the source model in the following decoding steps, potentially degrading performance during decoding. While we've demonstrated the feasibility of cross-vocabulary value transfer, improving vocabulary mapping algorithms and cross-family model guidance remain promising areas for future research.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline Llama-2-70B + & 1.1B & 1.1B norm & 1.1B curriculum \\ \hline MT-Bench & 4.92 & 5.37 (+0.45) & 5.14 (+0.22) \\ IFEval & 16.6 & 16.8 (+0.2) & 15.7 (-0.9) \\ GSM8K & 12.4 & 24.9 (+12.5) & 15.8 (+3.4) \\ Toxicen \(\downarrow\) & 42.1 & 70 (-27.9) & 40.9 (+1.2) \\ MBPP & 27.9 & 37.8 (+9.9) & 31.7 (+3.8) \\ HumanEval & 20.4 & 28.6 (+8.2) & 25.5 (+5.1) \\ ARC & 52.6 & 62.8 (+10.2) & 57.7 (+5.1) \\ Hellasavg & 60.5 & 66.7 (+6.2) & 64.3 (+3.8) \\ MMLU & 68.4 & 67.9 (-0.5) & 68.4 (+0.0) \\ TruthfulQA & 31.6 & 37.1 (+5.5) & 36.2 (+4.6) \\ Winogrande & 77.3 & 82.9 (+5.6) & 73.2 (-4.1) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison between Llama-2-70B base model guided by the standard 1.1B value model, the 1.1B model trained with a curriculum across 1.1B, 7B, and 13B models, and the model trained with a L1 normalization term.

Figure 4: MT-Bench scores of Llama-3-8B fully fine-tuned, Llama-3-8B-base plugged with the TinyLlama 1.1B value model (cross vocabulary), and Llama-3-8B-base plugged with the Llama 3.2 value models (same vocabulary).

 



## 7 Conclusions

In this paper, we introduced a novel method for transferable post-training called inverse value learning, which models residual logits adjustments using a separate value network. We systematically explored various aspects of this paradigm, including network architectures, pre-training weights, and connection schemes. Additionally, we addressed challenges related to overfitting and cross-vocabulary transfer by incorporating regularization techniques and a vocabulary mapping algorithm. Our results confirm that the value network exhibits broad transferability across models of different sizes and families, highlighting the potential of logits-space adjustments for efficient model adaptation.

### Limitations

We've discussed the limitation of additional computational resources in Section 3.1. In addition, inverse value learning is not applicable when logits of models are unavailable, and there is a theoretical performance degradation when integrate value models to pre-trained models with different vocabularies.

## Acknowledgments

We sincerely thank the anonymous ARR reviewers for their insightful comments and valuable suggestions. This work was supported by Beijing Natural Science Foundation (L243006), the Basic Research Program of ISCAS (Grant No. ISCASZ-D202401), the Youth Talent Program of ISCAS (Grant No. SYQ2022-3).

## References

* A. Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. (2021)A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. Cited by: SS1.
* J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Johan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021)Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: SS1.
* Collin Burns, P. Izmailov, J. Hendrik Kirchner, B. Baker, L. Gao, L. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, I. Sutskever, and J. Wu (2024)Weak-to-strong generalization: eliciting strong capabilities with weak supervision. In Forty-first International Conference on Machine Learning, Cited by: SS1.
* B. Cao, K. Lu, X. Lu, J. Chen, M. Ren, H. Xiang, P. Liu, Y. Lu, B. He, X. Han, L. Sun, H. Lin, and B. Yu (2024)Towards scalable automated alignment of lms: a survey. arXiv preprint arXiv:2406.01252. Cited by: SS1.
* Charlie Chen, S. Borgeaud, G. Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper (2023)Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. Cited by: SS1.
* M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pode De Oliveira Pluto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021)Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: SS1.
* W. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing (2023)Vicuna: an open-source chatbot impressing gpt-4 with 90%* chatgpt quality. External Links: Link Cited by: SS1.
* Peter Clark, Isaac Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tarjford (2018)Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457. Cited by: SS1.
* Karl Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021)Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: SS1.
* Tim Dettmers, Artidoro Pagnoni, A. Holtzman, and L. Zettlemoyer (2024)Olora: efficient finetuning of quantized llms. Advances in Neural Information Processing Systems36. Cited by: SS1.
* A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahale, A. Letman, A. Mathur, A. Schellen, A. Yang, A. Fan, et al. (2024)The llam 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: SS1.
* T. Hartvigsen, S. Gabriel, H. Palangi, M. Sap, D. Ray, and E. Kamar (2022)ToxFiGen: a large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 3309-3326. Note: Link Cited by: SS1.



* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Hinton (1999) GE Hinton. 1999. Products of experts. In _1999 Ninth International Conference on Artificial Neural Networks ICANN 99.(Conf. Publ. No. 470)_, volume 1, pages 1-6. IET.
* Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. 2023. Camels in a changing climate: Enhancing Im adaptation with tulu 2. _arXiv preprint arXiv:2311.10702_.
* Khanov et al. (2024) Maxim Khanov, Jirayu Burapachep, and Yixuan Li. 2024. ARGS: Alignment as reward-guided search. In _The Twelfth International Conference on Learning Representations_.
* Krause et al. (2020) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2020. Gedi: Generative discriminator guided sequence generation. _arXiv preprint arXiv:2009.06367_.
* Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In _International Conference on Machine Learning_, pages 19274-19286. PMLR.
* Lin et al. (2024) Bill Yuchen Lin, Abhilasa Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2024. The unlocking spell on base LLMs: Rethinking alignment via in-context learning. In _The Twelfth International Conference on Learning Representations_.
* Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. _arXiv preprint arXiv:2109.07958_.
* Liu et al. (2024) Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. 2024. Tuning language models by proxy. _arXiv preprint arXiv:2401.08565_.
* Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021. DEXerts: Decoding-time controlled text generation with experts and anti-experts. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6691-6706, Online. Association for Computational Linguistics.
* Mitchell et al. (2023) Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning. 2023. An emulator for fine-tuning large language models using small language models. _arXiv preprint arXiv:2310.12962_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: your language model is secretly a reward model. In _Proceedings of the 37th International Conference on Neural Information Processing Systems_, NIPS '23, Red Hook, NY, USA. Curran Associates Inc.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Sakaguchi et al. (2019) Keiskue Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. An adversarial winograd schema challenge at scale. _arXiv preprint arXiv:1907.10641_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. LIama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wan et al. (2024) Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. In _The Twelfth International Conference on Learning Representations_.
* Wu et al. (2024) Zhengxuan Wu, Argaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. Reft: Representation finetuning for language models. _arXiv preprint arXiv:2404.03592_.
* Yang et al. (2024) An Yang, Baosong Yang, Bingyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. _arXiv preprint arXiv:2407.10671_.
* Yang and Klein (2021) Kevin Yang and Dan Klein. 2021. FUDGE: Controlled text generation with future discriminators. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3511-3535, Online. Association for Computational Linguistics 

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? _arXiv preprint arXiv:1905.07830_.
* Zhang et al. (2024a) Bo-Wen Zhang, Yan Yan, Lin Li, and Guang Liu. 2024a. Infinitymath: A scalable instruction tuning dataset in programmatic mathematical reasoning. _arXiv preprint arXiv:2408.07089_.
* Zhang et al. (2024b) Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024b. Tinylama: An open-source small language model. _Preprint_, arXiv:2401.02385.
* Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-jadge with mt-bench and chatbot arena. _Advances in Neural Information Processing Systems_, 36:46595-46623.
* Zhou et al. (2023a) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023a. LIMA: Less is more for alignment. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Zhou et al. (2023b) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023b. Instruction-following evaluation for large language models. _arXiv preprint arXiv:2311.07911_.

## Appendix A The details of the Linear Probes

### Probe Architecture

In this work, we adapt the MLP layer from the Llama Touvron et al. (2023) architecture as a linear probe, which can be formally expressed as:

\[{\bf y}={\bf W}_{\text{down}}(\sigma({\bf W}_{\text{gate}}{\bf x}+{\bf b}_{ \text{gate}})\odot({\bf W}_{\text{up}}{\bf x}+{\bf b}_{\text{up}}))+{\bf b}_{ \text{down}}\]

where \(\sigma(\cdot)\) is the activation function, and \(\odot\) denotes the element-wise product.

### Failure modes

We demonstrated two representative failure case of using Linear layer to model the \(\mathbf{z}_{\Delta}\) in Figure 5.

```
0: Base tokenizer \(T_{b}\) with vocabulary \(V_{b}\) Value tokenizer \(T_{v}\) with vocabulary \(V_{v}\) Training data \(\mathcal{D}=\left\{\text{tex}_{1},\text{tex}_{2},\dots,\text{tex}_{N}\right\}\)
0: Normalized token alignment matrix \(M\in\mathbb{R}^{|V_{b}|\times|V_{v}|}\)
1: Initialize count matrix \(C\leftarrow{\bf 0}_{|V_{b}|\times|V_{v}|}\)
2:for each text \(\in\mathcal{D}\)do
3: Tokenize text using \(T_{b}\) to obtain sequence \({\bf t}_{b}\)
4: Tokenize text using \(T_{v}\) to obtain sequence \({\bf t}_{v}\)
5: Remove special prefixes from tokens in \({\bf t}_{b}\) and \({\bf t}_{v}\)
6: Compute cost matrix based on edit distances between tokens in \({\bf t}_{b}\) and \({\bf t}_{v}\)
7: Perform Dynamic Time Warping (DTW) to find the optimal alignment path \(\mathcal{P}\)
8:for each aligned token pair \((t_{b,i},t_{v,j})\in\mathcal{P}\)do
9: Increment count: \(C[t_{b,i},t_{v,j}]\gets C[t_{b,i},t_{v,j}]+1\)
10:endfor
11:endfor
12: Normalize and sparsify the count matrix \(C\) to obtain the mapping matrix \(M\)
13:return\(M\)
```

**Algorithm 1** Vocab Alignment

### Training Details

The majority of the training is performed on an 8 x A100-80G node. We adapt the hyperparameter configuration as outlined in Table 5.

## Appendix D Additional Results

Comparison with Proxy TuningAs a complement to our related work analysis, we conduct performance comparisons with proxy tuning on a 7B model, with results shown in Table 4. The empirical results demonstrate that our IVL-trained model achieves comparable performance to proxy tuning on the 7B model. Furthermore, IVL demonstrates higher inference efficiency as it does not require an additional 1B reference model during inference.

Inference CostsAs shown in Table 3, we conduct experiments to measure inference time for generating sequences of different lengths using various combinations of base models and value models. All experiments were performed using bfloat16 precision and flash-attention v2 for acceleration on NVIDIA H800-80G GPUs. For models 

smaller than 13B parameters and their combinations, pipeline parallelism was not employed. We did not implement any additional optimizations for value model inference. The reported results are averaged over 5 independent runs.

\begin{table}
\begin{tabular}{l|c c} \hline \hline Llama-2-7B+ & 1.1B & 1.1B+1.1B \\ \hline MT-Bench & 4.62 & 4.65 \\ ARC & 42.7 & 40 \\ Hellaswag & 55 & 50.4 \\ MMLU & 43.2 & 44.4 \\ TruthfulQA & 31.1 & 33.4 \\ Winogrande & 66.2 & 66.7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison between Inverse Value Learning (7B+1.1B) and Proxy Tuning (7B+1.1B+1.1B).

Figure 5: Failure modes for linear probes.

\begin{table}
\begin{tabular}{l l r r} \hline \hline \multirow{2}{*}{Model Config} & \multicolumn{2}{c}{\#GPU} & \multicolumn{2}{c}{Time/Peak Memory} \\ \cline{3-4}  & & 128 tokens & 512 tokens & 1024 tokens \\ \hline
1.1B & 1 & 1.99s/2.09GB & 6.99s/2.09GB & 13.36s/2.10GB \\
7B & 1 & 2.87s/12.65GB & 10.98s/12.84GB & 22.06s/13.09GB \\
1.1B + 7B & 1 & 5.13s/14.79GB & 20.83s/15.33GB & 49.74s/15.91GB \\
13B & 1 & 3.48s/24.51GB & 12.98s/24.79GB & 26.09s/25.18GB \\
1.1B + 13B & 1 & 5.73s/26.66GB & 26.10s/27.30GB & 61.61s/28.14GB \\
70B & 4 & 11.76s/32.41GB & 20.36s/32.42GB & 40.36s/32.44GB \\
1.1B + 70B & 4 & 16.12s/33.08GB & 33.07s/33.31GB & 75.37s/33.78GB \\ \hline \hline \end{tabular}
\end{table}
Table 3: Time and Peak Memory per GPU for different model configurations and generate sequence lengths.