Portable Reward Tuning: Towards Reusable Fine-Tuning across Different Pretrained Models
Authors: Daiki Chijiwa
Taku Hasegawa
Kyosuke Nishida
Kuniko Saito
Susumu Takeuchi
Venue: ICML 2025 poster (2025)
Keywords: inference-time tuning
reward maximization
Primary area: deep_learning->foundation_models

Abstract: While foundation models have been exploited for various expert tasks with their fine-tuned parameters, any foundation model will be eventually outdated due to its old knowledge or limited capability, and thus should be replaced by a new foundation model. Subsequently, to benefit from its latest knowledge or improved capability, the new foundation model should be fine-tuned on each task again, which incurs not only the additional training cost but also the maintenance cost of the task-specific data. Existing work address this problem by inference-time tuning, i.e., modifying the output probability from the new foundation model by the outputs from the old foundation model and its fine-tuned model, which involves an additional inference cost by the latter two models. In this paper, we explore a new fine-tuning principle (which we call portable reward tuning; PRT) that reduces the inference cost by its nature, based on the reformulation of fine-tuning as the reward maximization with Kullback-Leibler regularization. Specifically, instead of fine-tuning parameters of the foundation models, PRT trains the reward model explicitly through the same loss as in fine-tuning. During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels) through the formulation of reward maximization. Experimental results, including both vision and language models, show that the PRT-trained model can achieve comparable accuracy with less inference cost, in comparison to the existing work of inference-time tuning.

Portable Reward Tuning:
Towards Reusable Fine-Tuning across Different Pretrained Models
Daiki Chijiwa"
1
Taku Hasegawa"
22
Kyosuke Nishida Kuniko Saito
Susumu Takeuchi
Abstract
gence (Bommasani et al., 2021). They are typically large- scale neural networks pretrained on massive amounts of data from the Internet, which makes them generalizable to vari- ous downstream tasks, such as CLIP (Radford et al., 2021) for visual recognition and LLAMA-Series (Touvron et al., 2023a;b) for language generation. Even though foundation models are somewhat already capable of handling down- stream tasks, we can further bring out their potential ability by fine-tuning for each task, i.e., additional optimization of the foundation models on supervised data. Furthermore, fine-tuning can also be used for aligning the behavior of foundation models to follow human instructions or prefer- ences, by instruction tuning or reinforcement learning from human feedback (RLHF; Christiano et al. (2017)).
While foundation models have been exploited for various expert tasks through fine-tuning, any foun- dation model will become outdated due its old knowledge or limited capability. Thus the un- derlying foundation model should be eventually replaced by new ones, which leads to repeated cost of fine-tuning these new models. Existing work addresses this problem by inference-time tuning, i.e., modifying the output probabilities from the new foundation model with the outputs from the old foundation model and its fine-tune fine-tuned model, which involves an additional overhead in inference by the latter two models. In this paper. we propose a new fine-tuning principle, Portable Reward Tuning (PRT), that reduces the inference overhead by its nature, based on the reformulation of fine-tuning as the reward maximization. Specif- ically, instead of fine-tuning parameters of the foundation models, PRT trains the reward model explicitly through the same loss function as in fine-tuning. During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels) through the for- mulation of reward maximization. Experimental results, covering both vision and language mod- els, demonstrate that the PRT-trained model can achieve comparable accuracy to the existing work of inference-time tuning, with less inference cost.
Although fine-tuning has been the standard principle for tuning foundation models, there still remains an overlooked problem in the long term: the underlying foundation models should be eventually replaced by other (often newer) foun- dation models for various reasons, such as their outdated knowledge or limited capability, which requires the new foundation models be fine-tuned on every task again. For example, (i) in the case of open- CLIP models (Schuh- mann et al., 2021; 2022), an early model was initially trained on a dataset with 400 million image-text pairs, but later it was replaced by new versions trained on 2 billion and 5 bil- lion pairs; (ii) in the case of the LLAMA series, both major updates (e.g., LLAMA-2 to LLAMA-3) and minor updates (e.g.. LLAMA-3.1 I to LLAMA-3.2 -322) have occurred. Every time such update occurs, one may have to fine-tune the new foundation model again, which poses an important issue of the repeated training cost.
1. Introduction
One promising approach for this challenge is inference-time tuning (Mitchell et al., 2024; Liu et al., 2024a), which em- ulates fine-tuning of the new foundation model even with possibly different architecture, by mixing three output probabilities from the old foundation model, its fine-tuned model, and the new foundation model. More specifically, the approach first defines an implicit reward for each task as the logarithmic density ratio of the output probabilities from the (old) foundation and its fine-tuned models. Then, it reinterprets fine-tuning as maximization of the implicit reward with penalizing the deviation from the foundation
Foundation models, or simply pretrained models, play a central role in the recent development of artificial intelli-
'Equal contribution 'NTT Computer and Data Sci- ence Laboratories, NTT Corporation 2NTT Human Infor- matics Laboratories, NTT Corporation. Correspondence to: Daiki Chijiwa dbalkichhjiaa@ntt.cmm. Taku Hasegawa Sackk.hasggaa@@ntt.com
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Portable Reward Tuning
Previous Approach (EFT)
Our Approach (PRT)
Training
Fine Tunnnn
Reward Tuning
Reward (
Tuned
Inference with other pretrained models
Implicit Rewand:
(O Tog
Model molk): argnuax Ey- (r(s.y1]
PRT Model mir1s::
Des ( ky)
anymaa Ey- i jo (x.0))))) Das
Figure 1: An overview of our approach of portable reward tuning (PRT) compared with the previous work of inference- tuning, emulated fine-tuning (EFT). training phase, we tune the reward model re (zt, y) instead of tuning given pretrained model. through the same loss and dataset, which leads to the reduced cost in inference with another pretrained model.
model in Kullback-Leibler (KL) divergence, whose closed- form solution (Ziebart, 2010) can be computed efficiently and coincides with the fine-tuned model itself in this for- mulation. Based on this viewpoint, inference-time tuning is performed by solving the reward maximization, with the foundation model in the KI. penalty replaced by the new foundation model. This approach promising since it can avoid the repeated fine-tuning associated with replacement of foundation models, but instead it introduces another issue of inference cost, i.e., requiring three models to run at each inference step.
havior by replacement of the underlying foundation model in terms of KL divergence, (iii) generalization analysis from the PAC- -ayysian perspective.
Using both vision and language models, we conduct experiments inference-time tuning in realistic sce- narios like updating the pretrained knowledge and up- scaling the size of foundation models. We confirmed that PRT models can achieve comparable accuracy to the baseline of emulated fine-tuning, even with less in- ference cost in terms of both speed and memory usage.
In this paper, we propose a new principle as an alternative to fine-tuning, called Portable Reward Tuming (PRT), that more suitable for inference-time tuning. The proposed principle is straightforward from the above viewpoint of fine- tuning as reward maximization: introduce an auxiliary model as an explicit reward, and redefine a fine-tuned model, which we call a PRT model, as the closed-form solution that maximizes the explicit reward with KL regularization to a given foundation model. For training, instead of directly optimizing the foundation model itself, the explicit reward model is trained that the corresponding PRT model min- imizes the same cross-entropy loss as in usual fine-tuning. For inference, the PRT model is (re- ree))contteeeed from a pair of the explicit reward model and a given (possibly different) foundation model to serve as the corresponding fine-tuned model. By its nature, PRT enables us to freely update the underlying foundation model only with a sin- gle model overhead, while the existing methods based on standard fine-tuning still require additional two models to compute the implicit reward for inference-time tuning.
2. Background
In this section, we summarize the background of this work. In Sec. 2.1, we review the basics of KL-regularized Reward Maximization from the literature of Decision-Making The- ory and Reinforcement Learning. In Sec. 2.2, we briefly explain previous work on inference-time tunngg with its formulation based on reward maximization.
2.1. KI. Kl.-Regularized Reward Maximization
Let S be a state space, A be an action space, and a policy model (aass) be the probability that the action a A is chosen by some probabilistic mapping S - A given the consider reward function r(s, a) S x A - IR and a reference model (a)ss Then the goal of KL-regularized reward maximization is to maximize the expected reward Egsa(als) [r(s, a)] with soft constraint that keeps π (als)) close to the reference model Feef (als), in the sense of KL. divergence, as follows:
state S. Furthermore,
Our contributions are as follows:
max E. Bawx(els) [r (s. =(188)
We derive portable reward tuning (PRT) as an alter- native to conventional fine-tuning, and established its basic theoretical properties including (i) a natural in- terpretation of its training objective as reward learning, (ii) evaluation of how the PRT model changes its be-
ADKl (T((-s) || Trer(-ls)),
(I)
where DKL. (T(-s) || Teer(-Is)) = E. x (als) lag(s)))s)/
(a)s)) The closed-form solution of this problem is well- known (Ziebart, 2010; Korbak et al., 2022) as an extension
2

---

Portable Reward Tuning
of maximum entropy principle (Ziebart et al., 2008), given by
emulation result of fine-tuning the new pretrained model pe (v/z) through the implicit reward the specific task. Also, Liu et al. (2024a) proposed almost the same approach called proxy-tuning.
1
Trel (a|s) exp ((=(s_a)):
T(a|s)= ((s)
(2)
where ((s) : Enea Teet (als) exp (+r(s, is the nor-
3. Portable Reward Tuning
malization factor.
In this section, we develop new fine-tuning framework, called portable reward tuning (PRT), with both training and inference algorithms based on KL-regularized reward maximization. Throughout this section, we follow the same classification setting as in Sec. 2.2, which includes both image classification and language generation tasks.
2.2. Inference-Time Tuning
Let (y) be classification model defined (v(aa) := softmax(f (x; f)), where E X' is an input, E (ci, , a classification label, [x;; 0) E R a neural network parameterized by E IR,, and the softmax func- tion'. Let (v/zx) == Ram (y/(z) a pretrained model, and Tu (y/2)) := Tgt. (y/lz) its fine-tuned model on some specific task. Typical examples include (i) in image classification, is image, is its corresponding label and fz; 0) is CNN (Bengio et al., 2017) or Vision Transformer (Doso- vitskiy et al., 2021); and (ii) in language generation, x is sequence of tokens t ..... fx. is the next token fx+1 and fo(x) a decoder-only Transformer (Radford et al., 2018).
3.1. Training of PRT
Setup. Let r(æ; 0) = (r1 (z E R be a neural network with L-ddimenional outputs. We refer to the i-th component r(x; 0) as the reward value, denoting ro c.) == rı(2;; 0) for an input x and the i-th label 6; E y We also assume that a pretrained model Tpt and dataset of input-label pairs S = {(11,(())) v1), (2) v(s) for some specific task are given.
Based on the theory of KLLregularized reward maximization, Mitchell et al. (2024) proposed emulated fine-tuning (EFT) which views the fine-tuned model 7. (y((r) the solution of the following problem:
Formulation. In our PRT framework, we optimize the reward model ro(a,, y), instead of directly optimizing the given pretrained model Te (v/zr) For a little while, assume that we already have the learned reward model ra(2, y) for the given specific task. Then, the desired classification model To (y) (z:) is defined the solution of reward maxi- mization with KL-constraint to the pretrained model:
Ta Tn (3/l3c)
]
max E (x(x)) <(-(a)
log
Tpt (3) (z))
DKL. (x
Tpt (-1z:),
(3)
where log (Tn(yl=) (y)l=)))))) is called implicit reward, expected to be the reward function that reflects the task- specific preference for y E V. Indeed, by applying (2), the closed-form solution क (yl+c) of (3) is given by
max Eg Eawn(yla) [re( (3] (+|))
- ADKL (T(-|x) || Tp((-1z))
(5)
As we already discussed in Sec. 2.1, the closed-form solu- tion for this maximization problem is provided by
1
Tie (v(xx x))
π (y)æ²) =
[xx) To (x)lx) exp
log
Tpt (y/lr)
Ta (v)
(e(2z) = Zo Tpe (B(()) exp (Fro(e.v)) (6)
Building on this fact, Mitchell et al. (2024) also proposed what they call scale decoupling, which replaces the pre- trained model in the KL constraint by different pretrained model from the one appeared in the implicit reward. In other words, they consider the following problem:
where (z(x) Ey ppt (y ar) exp(ro (x,,rr malization factor. We call this To (y)lr) a PRT model (for training) with the reward ra( y) and the pretrained model Tpe (y/z).
the nor-
Although it is somewhat obvious, the following proposition guarantees that the above PRT models are equivalent to standard fine-tuned models in terms of their expressiveness:
mh (y/lz))
max E. ())r(()>) log m(+ax)
Tpp (y) (//z)/
- DKL (=((xx)
mpt (-1x)),
(4)
Proposition 3.1. There is a one-to-one correspondence between fine-tuned models and rewards, which preserves their accuracy:
where (-1x)) is another pretrained model that is different from (-)(- possibly with different network architecture. The closed-form solution of (4) can be considered as the
{T (ul=)) fine-tuned models}
"softmax(yl,"" .U)) = (e / EE,", eh ... , / )
- fr(y/o)) rewards satisfying Ey a) (erluss))) =1}
3

---

Portable Reward Tuning
where (n(()))) is mapped the implicit reward log(m Ta (pa
Algorithm Pseudocode for Training of PRT
1: Given: training data S {(21 V1). (Im vm)}.
reward model r(x; 0),
Proof. The mapping preserves accuracy since the PRT model (6) with the implicit reward recovers the given model
3: pretrained model Tpt (-(x) softmax(f(et) (p;:p)))
4: Initialize 0.
Te (v(z). The invertibility also holds since the implicit re- ward of the PRT model (6) recovers the reward itself. LI
5: for i = 1.... m do
6: Vg t log softmax( (p)) + r(x;)))
7: Pa f soft
In other words, if there is a fine-tuned model that achieves some accuracy, there is also the corresponding reward whose PRT model achieves the same accuracy. Thus, the reparam- eterization in PRT (6) does not restrict its expressiveness, even compared to standard fine-tuning.
Lo < CF(po.y))
Update with the gradient of Ca.
10: end for
11: return 0.
Algorithm 2 Pseudocode for Inference of PRT
Training objective. The reward model ro(2, y) is trained by simply optimizing the same loss function C(p. y²* as in standard fine-tur finettunin, with the true label y" for the input x and the output distribution p = Th (-|a:) of the PRT model. (See also lines 6-9 in Algorithm 1.) In particular, throughout this paper, we minimize the cross-entropy loss [(P. y²) == CE ("))" == - log (v over a given dataset S C X' x y to train the reward model:
1: Given: an input 2; the trained reward r(a; 0),
a pretrained model p (y/z) = softmax (f;;; pp)),
3: V + log softmaxx( ppt) + r(x;
4: P f soft
5: return P as the output probability conditioned by x.
reward model r(2; 0). although it can be arbitrary neu- ral network model, we assume that it is modeled using the same network architecture as the pretrained model [(;; ppe) and that 0 initialized with Opr throughout this paper. Lines 6-7 compute the PRT model (6) in the logit space to avoid numerical instability. Here, we the coefficient À 1 in (6) since the reward model can automatically learn the scaling factor.
arg min
Σ
v(P.v")
S
(a.y) JES
arg max
Σ
log (yy
(ESR)SS
arg max
Σ
Te(2,8" Vo(x),
(x.0 JES
where Vo(x) = A Aogg Zo) Notably, this maximization can be reinterpreted in terms of reward training. Indeed, by applying Jensen's 's inequality, we obtain
3.2. Inference of PRT
Let ro(x, v) and Tpe (yla)) be the reward and pretrained model introduced in Section 3.1. Let 7 (y/x² be another pretrained model whose label space ) (or vocabularies for language models) is the same as the one for Tpt (y/zr) Ex- amples of (3)2)) include model pretrained on larger or more recent dataset than that of Tip (y) and a pretrained model with more parameters.
re(x,y") Vz(z)
= =rolz,y) (1),,y") Alog B (ula) exp(re(v\=)/)
<ro(F, v²) AEy= (vlm) log exp((o(y) x)
= =ro(z,y)) (
(9)
The inference model (y(y)))) for the reward ro(x,, y) and the specified pretrained model 7po. (ulz)) can be derived by replacing Te (y|z) xx) in (6) with Tpt (y/l). Specifically, given an input æ E X, the prediction for its label y is performed by the following model that maximizes the reward re (z. y) while minimizing the deviation from the specified pretrained model m (y]ar):
Therefore, maximization of (8), i.e., training the reward model using the cross-entropy loss, leads to an increase in the reward for the ground-truth y" while decreasing rewards for the average outcomes y from the pretrained model Tpp (x) This interpretation can be seen anal- ogous to the training of the Mandley-Terry reward model in RLHF (Christiano et al., 2017) with pairs of preferred- dispreferred sentences, where the reward model learns to evaluate the preferred one higher than the dispreferred one.
To (y(a)))) = arg max Bawa((yla) [ro v)]
=(|z)
ADRI. (T(-|() pp((-1x)
Implementation Details Algorithm presents the pseu- docode for training PRT models. The training data is a set of laput-label pairs just like in standard training. For the
1
The (y(2)) exp ( (Frolo.w). (10)
(z(

---

Portable Reward Tuning
where (x(x) = Ey (p)()) exp(ra(x; v) The imple- mentation of inference by this PRT model (10) is straight- forward described in Algorithm 2, with À = 1 as in
The KL divergence (2 P) is not computationally tractable because the pretrained distribution P itself is not tractable and also the underlying space of pretrained models is too vast. Nevertheless, the result implies that the gen- eralization capability of PRT models can be captured by the closeness of the PRT model Ao (y/xr compared to the underlying pretrained model (pr.
training.
Now the following question naturally arises: How does the choice of pp a)) affect the behavior of the inference model iT [y]==)? Intuitively. if (pp(y/z)) does not deviate from the original (p)()))), the inference model Tpa (y/lz) also keeps to behave similarly to the training time. This intuition can be formalized as follows:
In particular, we can easily see that the KL divergence term vanishes if the reward ro y) is constant value given each v and then the equality To (o(() = Fpa.vllz) holds. This can be seen as the case that the exponential distribution Pe (ulz:) == cxp(re(2; (z. y)) exp(ro y)) maximizes its entropy. Thus, the generalization capability can be further improved by regularizing the exponential distribution of the reward ra (z, y) during training of re (1z, y), while it may hurt the optimization quality instead.
Proposition 3.2. Suppose that ppt (y(z)) is close to ptt (y/zz) i.e.. DKL (Tpe(-1z) |I (-(x)) S s. Additionally, we as- sume that the maximum and mean value ratio of the ex-
ponential reward, i.e,, maxy exp expro(z.0) B)/Ey exp expro (x. y). is bounded by some constant C. Then, the PRT models To Te(y)z)) and To (y)())) are also close as distributions:
The above analysis motivates us to consider the following regularization, which we call Entropy Maximization (EM), to optionally enhance the generalization capability across various pretrained models:
DKL (To(y)z) [o((yll)) O(VE).
Proof. See Appendix A.
3.3. A PAC-Bayesian Perspective
Σ(0) C())
Σ (x.U)ES
(-)((=)))), (12)
SST
We can suppose that the pretrained models for both train- ing and inference, i.c., Tp (y/lz) and ppt (y/zz), are chosen from some distribution PP over the set of pretrained models. Then the PRT models for inference, Ro(y)z)) combined with the sampled pretrained model Te (ulz) ~ P, form a new distribution Qs.
where L()) is the original loss (7) for training the reward model, H (po(-l=)) == - -Ene Pe (v(yx) log po(ylx) is the entropy of the exponential distribution pe(y(æ), and a e
R20 is the hyperparameter to control the regularization. As we can see larger a enforces the reward ro y) to be closer to some constant for each input x E X.
This formulation of PRT models naturally fits into the PAC- Bayes framework established in McAllester (1999), which enables us to analyze the generalization error of posterior distributions over classifiers, in comparison to a fixed prior distribution. Specifically in our setting, the pretrained dis- tribution P can be seen as a prior distribution, and the PRT distribution Qo as a posterior distribution.
4. Experiments
In this section, we evaluate the performance of portable reward tuning (PRT) for inference-time tuning, with vari- ous pretrained models including both vision and language models. The main comparison is between PRT and its cor- responding baseline, emulated fine-tuning (EFT; Mitchell et al. (2024)), which differ only in whether they maximize explicit implicit reward during inference-time tuning.
Let [(I, π) be finitely bounded loss function, e.g., one that returns the error rate, for given input and classifier æ. Assume that the input follows some distribution D. The following generalization bound can be obtained as a direct consequence of Theorem in McAllester (1999):
Setups for vision experiments. (1) Pretrained models: We employed CLIP models (Ilharco et al., 2021) pre- trained on various datasets including OpenAl's proprietary dataset (Radford et al., 2021), LAION-400M (Schuhmann et al., 2021), LAION-2B (Schuhmann et al., 2022), and DataComp-1B 1B (Gadre et al., 2024). (2) Fine-tuning: For each fine- finegrained dataset, such as Cars (Krause et al., 2013) and CUB (Wah et al., 201 1), we first constructed and fixed the classification layer of each pretrained model for zero- shot classification, and then fine-tuned (or reward-tuned) its feature extractor on the train set. (3) Evaluation: We evaluated models on the test set of each dataset where the training set was used for tuning the models.
Proposition 3.3. Let S = (I1,' ,Tm) ~ mDm be i..dd m training samples from the data distribution D. Then, with probability at least 5, we have
En~O, [Mx, m]] S Banes -
1
Σ
(x,,, x)
m
(2x [1 + log( ) + 5 log m + 8
(11)
2m 1
for the posterior distribution Qo with the reward re trained on S.
5

---

Portable Reward Tuning
On
On
30
12
14000 4190
1200928
1018
1469 (019-1011
LAC9
120.01
19045
(a) ResNet-50 (OpenAD)
(b) ConvNext (LATON-400M)
(c) VIT-B-16 (OpenAT)
(d) VIT-B- (TTB-I6LAAOON-00M
(c) VIT-R-16-- (L-AION2)))))
Figure 2: Evaluations of inference-time tuned models for vision tasks. Each subcaption refers the source pretrained model, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot classification by each target model as baseline, and FT means the fine-tuned target model as an oracle result.
qsmBkk
@sm8k
@sm
qsm
138
38
76
108
Target Modd
Targst Sookl
Target Mood
Taipst Modd
1:38
3.2-18
540
3.1-80
30
7B
100
Turpet Model
Target
TorpetNocn
Turpet
(a) Llama2-7B
(b) Llama3.2-1B
(c) Qwen2.5-0.5B
(d) Falcon3-1B
Figure 3: Evaluations of inference-tie time instruction tuned models on GSM8k and IFEval benchmarks. Each subcaption refers the source pretrained model, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot inference by each target model baseline, and Instruct means the instruct-tune tundd target model as oracle result.
Setups for language experiments. (1) Pretrained models: We employed pretrained language models of the decoder- only Transformers such as LLAMA series (Touvron et al., 2023b), Qwen series (Yang et al., 2024b;a) and Falcon series (Team, 2024). (2) Fine-tuning: We performed in- struction tuning on these pretrained models with Tulu v2 dataset (Ivison et al., 2023), large-scale dataset consisting of demonstrations for following given instructions. (3) Eval- uation: We evaluated models on downstream benchmarks, particularly on the GSM8k benchmark (Cobbe et al., 2021) for reasoning ability and IFEval benchmark (Zhou et al., 2023) for instruction- sostuutioo--oolooinng ability.
4.1. Results
Figures 2, 3 and 4 show the results of inference- time tuning using PRT and EFT, from source pretrained model, i.c., the one used in tuning the (either explicit or implicit) reward, to a target pretrained model, i.e., the one never used in tun- ing the reward. Here PRT refers to the vanilla one without regularization for fair comparison. We also compare them with the zero-shot performance of pretrained models them- selves baselines, and with their fine-tuned performance oracles. Note that, in the case that the source and target are the same pretrained model (i.e., the leftmost one in x-axis), the results correspond to the ones without inference-time
6

---

Portable Reward Tuning
gsm8k
Prompt
Exacc Match 99
Pntraned
James has 6 candies than Robert. John twice as many candies as Robert. John has 54 candics, how many more candies does John have than James?
Question:
2.5.0.58
2.5-1.58
2.5:38
2.5-J4B
Answer:
ifeval
Protraintt
Model Response
Auntappe (N)
Intnec
Let R be the number of candies Robert has. John has 2R candies. James has + candies. We know that John has 54 candies, 2R 54. R 27. James has 6 + 27 33 candies. John has 54 33 21 more candies than James.
2.0.58
2.5.0.58
2.5-1.58
2.5:56
2.5-78
2.5-148
Figure 4: Inference-time tuning from Qwen2-0.5B to the Qwen2.5 models with various sizes.
tuning. Overall, these results support our main claim in this paper, i.c., PRT achieves comparable accuracy to EFT with less inference cost. (See also Section 4.4)
(a) Next Token Candidates
(b) Next Token Candidates
Following "... John has 2R Following
R 27.
candies. James
James has
Particularly, the pairs of the source/target pretrained models be categorized into either of the following two scenar- ios: (i) upscaling the source model to larger target models, and (ii) updating the pretrained knowledge of the source model to the target model with better pretraining data. The results for the former scenario (i) are shown in Figures 2a- 2b, 3a-3d, where the source and target models are pretrained on the same dataset, and thus the only difference between them is the network architecture. The results for the lat- scenario (ii) shown in Figures 2c-2e and Figure 4. In both scenarios, we observe that PRT successfully lever- ages the improved capabilities of target pretrained models, by reusing the fixed reward model trained with the source pretrained model.
Figure 5: Changes in next-token prohahilities by PRT.
Frwres
Flaer
(a) Accumcy of PRT (solid) and reward-only (dotted) during training.
(b) Inference- tuning with various regularization coefficients o.
Figure 6: Analysis how the EM regularization affects performance PRT with various coefficient a. Reward-on only refers to the exponential distribution pa (y(lzz) in Sec 3.3.
4.2. Qualitative Analysis
GSMBK
GSMBK
To analyze the behavior of PRT in more detail, we examine the tokens generated by the model. In this analysis, we used Llama3-8B as the target model, while Llama-3.2-1 1B was used for training and reward-model initialization. Figure presents the output obtained under these settings. We ob- serve that the PRT output correctly produces the chain of reasoning that leads the final answer of 21.² Additionally, Figures 5a and 5b show the top-5 prediction probabilities for the tokens that follow each highlighted sentence. From these distributions, we see that the target pretrained model's predictions are altered by the PRT model. In Figure 5b, instance, the pretrained model attempts to output the answer "33" directly, whereas the PRT outputs "6" which re- flects the step-by-ster reasoning capability acquired through nife instruction-tuning
EM score (96)
EM score (96)
322-8
3.1-8B
1.58
(a) Llama3
(b) Qwen2.5
Figure 7: Performance of PRT GSM8k with various regular- ization coefficients a.
4.3. Effects of Entropy Regularization
Here we employ Entropy Maximization (EM) regularization in PRT training, with its coefficient E R> In Section 3.3, the PAC- PAC-Bayesian analysis indicated that the generalization capability of PRT models is affected by the entropy of the reward distribution po po(y)z) ()) defined by the reward model
By contrast, the pretrained Llama3-8B output only "18." For details, see Appendix D.

---

Portable Reward Tuning
ra ra(a, y). In Figure 6 and 7, we empirically analyze its effect by varying a. First of all, we observed that the EM regular- ization generally suppresses the accuracy of the reward itself (Fig. 6a) as a increases, which can be naturally expected. However, interestingly, the performance of PRT (Fig. 6b, 7) is not degraded even with such rewards, but also sometimes boosted regardless of the accuracy of rewards themselves. The regularization itself may not be yet practical since the tasks and pretrained
in our work. In particular, Deng & Raffel (2023) proposed to control language models with reward models similarly to our work, but which are trained with a loss specific to text classification tasks, while our method employs standard loss for fine-tuning and thus can be nat- urally applied to broader domains and tasks such as vision classification and instruction tuning.
optimal tends to be dependent models, but these analyses provide valuable insights for generalization capability of PRT models, which may lead to future exploration of more practical regularization.
Tuning by Editing Parameters or Activations. Another possible approach for inference- -ireeneee-mmeeeeee tuning would be di- rectly editing the parameters or activations of pretrained models, instead of their predictions. A bunch of research on parameter editing, including (Ilharco et al., 2023; Gueta et al., 2023; Ortiz-Jimenez al., 2024; Yadav et al., 2024; Chijiwa, 2024; Daheim et al., 2024), addresses inference- time tuning by leveraging existing fine-tuned results. How- ever, most of these work aimed to tune given pretrained model for multiple tasks, and thus cannot be used to tune a newly provided pretrained model. Although Chijiwa (2024) tackled the challenge of inference-time tuning with different pretrained models, it still requires the pretrained models to share the same architecture, which may be fundamental limitation of the approach by editing parameters. Similarly, there is a line of research (Dathathri et al., 2020; Hernan- dez et al., 2023; Chuang et al., 2024; Li et al., 2024) on tuning by editing the activations, but they also require the model architecture to have the same dimension for acti- vations. In contrast to these approaches, the approach by refining predictions in this paper would be more promising since it is completely free from the choice of the model architectures. Also, in cases of language models, automatic prompt tuning (Zhou et al., 2022: Pryzant et al., 2023; Wang et al., 2024b) can also be considered as an approach of tun- ing models by editing input prompts and reusable to other language models, though there still have been difficulties coming from both the discrete nature of prompts and their limited expressiveness.
4.4. Memory and Speed Analysis
Since PRT introduces an auxiliary model as the reward for both training and inference, we investigate how the memory usage and speed increase or decrease, compared to standard -inettuning in training and EFT in inference. For training, Table 1 in Appendix shows that, while training time slightly increases due to the auxiliary model, the increase in memory usage is relatively negligible because the pretrained model in PRT does not require back- prock-rpppgttion. For inference, Tables 2 and 3 show that PRT successfully reduces both inference time and memory usage compared to EFT, which highlights the benefit of employing explicit reward models.
5. Related Work
Tuning by Refining Predictions. In this paper we focused on inference-time time tuning accomplished by refining predic- tions from the underlying pretrained model. In particular, our baseline is the emulated fine-tuning (EFT; Mitchell et al. (2024)) which established the interpretation of inference- time tuning based on KL-regularized reward maximization. Parallel work by Liu et al. (2024a) also proposed an es- sentially same method called proxy-tuning. While these previous work focused on inference-time tuning with pre- trained models that only differ in their model scale, we examined a more general setting that the pretrained models may differ even in their architectures or pretraining datasets. Also, the previous work assumed the fine-tuned model was prepared in advance, and explored how to exploit it for a new pretrained model. In contrast, we reexamined the as- sumption and explored an alternative to fine-tuning that is more suitable for inference-time tuning. As a consequence, at the cost of a little overhead in training, our approach successfully halves the overhead in inference-time tuning.
Reward for Language Models. For language models, the notion of rewards has been exploited mainly in two lines of research: (i) Reinforcement Learning from Human Feed- back (RLHF; Christiano al. (2017); Jaques al. (2017); Ouyang et al. (2022)), and (ii) multi-step reasoning (Cobbe et al., 2021; Uesato et al., 2022). In RLHF, reward models are first trained on a dataset of human feedbacks, such as pairs of nrefered-dispreferedd responses, and then used for training LLMs by reinforcement learning. Although recent methods (Rafailov et al., 2024a; Calandricllo et al., 2024; Ethayarajh et al., 2024; Rafailov et al., 2024b) successfully bypass the explicit use of reward models, they still learn- ing human preferences through implicit rewards. Several work (Liu et al., 2024b; Chakraborty et al., 2024; Khanov et al., 2024) also integrate the idea of inference-time de- coding into RLHF, but they assume the reward model is already prepared by RLHE In multi-step reasoning, reward
The literature of controlled text generation (Krause et al., 2021; Yang& & Klein, 2021; Pascual et al., 2021; Li et al., 2023; Deng & Raffel, 2023) is also related to our work, but they have explored specific methods for attribute- conditioned text generation, which requires classifier for some attributes, rather than fine-tuning for general tasks as
8

---

Portable Reward Tuning
models are trained to evaluate the intermediate process of reasoning by LLMss, from datasets with process or outcome supervision, and then used as verifiers for sampling such as Best-of-N sampling (Lightman et al., 2024; Wang et al., 2024a) or self-consistency decoding (Wang et al., 2023; Luo et al., 2024). In contrast to these lines of work, where rewards are used for either training LLMs or verifying infer- ence, the reward model in our approach is directly trained for classification generation from ground-truth labels or tokens through the same loss and dataset for standard fine-tuning, which may lead to broad range of applications in various domains including language generation.
man alignment of large language models through online preference optimisation. In International Conference on Machine Learning, pp. 5409-5435, PMLR, 2024.
Chakraborty, S., Ghosal, S. S., Yin, M., Manocha, D., Wang, M., Bedi, A., and Huang, F. Transfer q-star Princi- pled decoding for LLM alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http//pppeeeie.eeeeeeeeee net/
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint #rXiv:2107.03774 2021.
6. Conclusion
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-188; 2017.
In this paper, introduced new fine-tuning principle called portable reward tuning (PRT) an alternative to standard fine-tuning, based on the interpretation of fine- tuning as reward maximization with KL regularization. PRT naturally fits into the framework of inference-time tuning as the reward maximization. We theoretically analyzed its basic properties of both training and inference, revealing how the reward evolves in training and how the choice of pretrained models affects inference-time tuning. Also we empirically confirmed that PRT can achieve comparable accuracy to previous work of inference-time tuning, even with less computational overhead.
Chijiwa, D. Transferring learning trajectories of neu- ral networks. In The Twelfth International Conference Learning Representations, 2024. URL https:// openreview net / forum? i 1d=
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J. R., and He, P. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: htttt::://ppeeeeetee /wwpnreveeeeeeeeeeeeeeeeeeeeeewe net/ i (d=Th6NyL0770
Impact Statement
This paper investigates basic properties of proposed prin- ciple of fine-tuning which aims advance the field of Machine Learning. There may be potential societal conse- quences of our work, but discussing about them is out of scope of this work.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 100:::::1000000000 00545,, 2018
References
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., and Sutton, C. Program synthesis with large language models. arXiv preprint arXiv:2108.07733: 2021.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 1r 141168 2021.
Bengio, Y., Goodfellow, I., and Courville, A. Deep learning, volume 1. MIT press Cambridge, MA, USA, 2017.
Daheim, N., Möllenhoff, T., Ponti, E., Gurevych, I., and
Khan, M. E. Model merging by uncertainty- based gradi- ent matching. In The Twelfth International Conference on Learning Representations, 2024. URL https: openreview wetpi/ewwweeeeeeeeeeeeeeeeeeeeee forum? ? ddD
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint SXXi:2118800255 2021.
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play lan- guage models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020. URL ht ptttpp::/oppereviest net: forum? 1== 1eHHedeKKDD..
Calandriello, D., Guo, Z. D., Munos, R., Rowland, M.,
Tang, Y., Avila Pires, B., Richemond, P. H., Le Lan, C., Valko, M., Liu, T., Joshi, R., Zheng, Z.,, and Piot, B. Hu-

---

Portable Reward Tuning
Deng, H. and Raffel, C. Rewarrd-augmented decoding: Ef- ficient controlled text generation with a unidirectional reward model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11781-11791, 2023.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Car- lini, N., Taori, R., Dave, A., Shankar, V., Namkoong. H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, July 2021. URL https: //t::///do... org/10.
5281 /zenddo 51 43773.
Ilbarco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L.,, Ha- jishirzi. H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https:// openreview net forumm wtte:oooonnnnddto-oeee 21
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https::/ openreview net forum? d=ldd=iiiiiii cbFaNTTy..
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang. J., Wadden, D., Smith, N. A., Beltagy, I. et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702. 2023.
Dubey. A., Jauhri, A.. Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiw:2407.21783, 2024.
Jaques, N., Gu, S., Bahdanau, D., Hermández-Lobato, M., Turner, R. E., and Eck, D. Sequence tutor: Conser- vative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, pp. (645-1654, PMLR, 2017.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In International Conference on Machine Learning, pp.
PMLR, 2024.
Khanov. M., Burapacheep. J., and Li, Y. ARGS: Alignment as reward-guided search. In The Twelfth International Conference on Learning Representations, 2024. URL https::/p//// /ttp////ppeeeeeeeee/e/ee/e. / id id=shgx0egdwS.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smymiss G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., et al. Datacomp: In search of the next generation of multimodal datasets. Advances Neural Information Processing Systems. 36. 2024.
net/ net.ieeww.eeee/eeee.eeeeee forum?
Korbak, T., Perez, E., and Buckley, C. RI with kl penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, 2022.
Gueta, A., Venezian, E., Raffel, C., Slonim, N., Katz, Y., and Choshen, L. Knowledge is a region in weight space for fine-tuned language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:://// pestps:///perrrviiiee wwt. forum? id id=vg4BnrPyPb.
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. GeDi: Gen- erative discriminator guided sequence generation. In Moens, M.-F. Huang, X., Specia, L., and Yih, S. W.- t. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4929 November 2021. httpst h/acc//cllaaccc /aa/aallatthhhee ogy org 2021. E findings Seiiings-ennlp... www.p.nn.p..2.... 424.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d ob- ject representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Hernandez, E., Li, B. 7.,, and Andreas, J. Inspecting and editing knowledge representations in language models. arXiv preprint arXiv:2304.00740. 2023.
Li, F.FF., Andrecto, M., Ranzato, M., and Perona, P. Caltech 101, 2022.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang. L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Li, K., Patel, O.,, Viégas, F., Plister, H., and Wattenberg, M. Inference-tine time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36, 2024.
10

---

Portable Reward Tuning
Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L.,, and Lewis, M. Con- trastive decoding:
Ortiz-Jimenez, G., Favero, A., and Frossard, P. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
Open-ended text generation as opti- mization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12312, 2023. URL https: aaaclannhooogge ogy. wrg. 2023 acl acl-long 687.
Ouyang, L.,. Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277444 2022.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. In The Twelfth International Conference on Learning Representations, wttp:/iiwwwwewneenneeeeeeee net/forum?
Pascual, D., Egressy, B., Meister, C., Cotterell, R., and Wattenhofer, R. A plug-and-play method for controlled text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3973- 3997, 2021.
2024. URL https:://// htt.ttttt/ppeereeeieee. http///ppppwpwe.ept/ttp/eeoe./ep
1d=v810pN6E01.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 2021.
Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., and Zeng, M. Automatic prompt optimization with "gra- dient descent" and beam search. In The 2023 Conference on Empirical Methods Natural Language Processing, 2023. URL https: btttp:://peereveieee /wpp//eepeeeeeeeeeeeeeeeeeeeeewe net fooum?? id=wRYhaSrThy.
Liu, A., Han, X., Wang, Y., Tsvetkov, Y. Choi, Y., and Smith, N. A. Tuning language models by proxy. arXiv preprint arXiv:2401.084: 1.0565,,.6 2024a.
Liu, T., Guo, S., Bianco, L., Calandriello, D., Berthet, Q., Llinares, F., Hoffmann, J., Dixon, L.., Valko, M., and Blondel, M. Decoding realignment of language models. arXiv preprint arXiv:2402.02992, 2024b.
Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. Improving language understanding by generative pre-training. 2018. URL https: / //cdn. wtt.:ppppaa.ccco/eeeerrttaaaeaee covers/ Languagggunssueervisedd/t Lwngguuga.dddagggg.eee bawprrttannnng_pappeeee
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11976-1199999 11986, 2022.
Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natu- ral language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y. Shu, L., Zhu, Y., Meng, L., Sun, J., et Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint arXiv:2406.06592 2024.
Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft. Technical report, 2013.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024a.
McAllester, D. A. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 164-1700 170, 1999.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to Q": Your language model is secretly Q- function. In First Conference Language Modeling, 2024b. URL https:/ BttLtttts//ppeeeeeeeee /wwp///vpeeeeee.eeoeeeeeee http:ewwwwwwwwwefeooeeeeee net/fertuuu i id=kEVeNxIqXk.
Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Man- ning, C. D. An emulator for fine-tuning large language models using small language models. The Twelfth International Conference on Learning Representations, httpap/////ppeeeeeeeeeeeee/rrrrr id-Eo7kv0sllr.
2024. URL www.h:ttp/..cm pecttps://peereeeee wttp:/iwwwweeeffoorureee
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J.. Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311. 2023.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEFE, 2008
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and

---

Portable Reward Tuning
Komatsuzaki, A. Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint arXiv:2111 1.11111,1444 14, 2021.
A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574, 2024c.
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 2022.
Yadav, P., Tam, D., Choshen, L.,, Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merg- ing models. Advances in Neural Information Processing Systems, 36, 2024.
Yang, A., Yang. B., Hui, B., Zheng, B., Yu, B., Zhou, C.,, Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J., Yang. J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu,
Team, F.-L. The falcon 3 family of open models, De- cember 2024. URL blog/falll falcon3.
K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang,
P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang,
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang. X., Wei, X., Ren, X., Liu, X., Fan, Y. Yao, Y., Zhang. Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z.,, Zhang. Z.,, Guo, Z.,, and Fan, Z. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhos- ale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint (xXix:2377.09288 2023b.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115. 2024b.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.
Yang, K. and Klein, D. Fudge: Controlled text gen- eration with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 351 1-1155333, 2021.
2022.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. Caltech-uesd birds-200-2011. Technical Report CNS- TR-2011-001. California Institute of Technology, 2011.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint Seki:::190.0000 2019.
Wang, P., Li, L.., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce Ilms step-by-step without human annota- tions. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426-9439, 2024a.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloving evaluation for large language models. arXiv preprint arXiv:2311.0911 2023.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human- level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
Wang, X., Wei, J., Schuurmans, D., Le. Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self- consistency improves chain of thought reasoning in lan- guage models. In The Eleventh International Conference on Learning Representations, 2023. URL. https:/ // openreview. for/m 1.d--PPLIIIMMM
Ziebart, B. D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mel- lon University, 2010.
Wang, X., Li, C., Wang, Z., Bai, F., Luo, H., Zhang, J., Jojic, N., Xing. E., and Hu, Z. Promptagent: Strategic plan- ning with language models enables expert-level prompt optimization. In The Twelfth International Conference on Learning Representations, 2024b. URL https: // //ppenreview.cc.c new.eeww.eeeeeeee / wit.ummmdddd2amaaaom
Ziebart, B. D.,, Maas, A. I.,, Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008.
Wang, Y., Ma, X., Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., et al. Mmlu-pro:
12

---

Portable Reward Tuning
A. Proof of Proposition 3.2
Proof. Let (x) == By (1l)) (xxp(ra (3r, and similarly (xr)) = (olsn v))] for convenience.
DEL (-1 #o (-(()>))
no T xr)
Σ no (u) log
No (y)lax)) xr)
Tpp exp(ro
Tea (1 xx) To (y x))
log
(z(
Tpt (y)la) exp((o(xx,)) (1)
Zo(x)
y
Tpt (yll) (pp((y/z)
Zo(x))
x
log
+ log
Zz(x))
1
Σ exp (ro(2,, v)) xpp log
Tpt (1/ 2x) Tpt (y/ 2z)
(x(r)
(x(
+ log
(x(
max, exp ro (I. y)
Egr-r(())))) exp (o v) DKL (Tp(at((-aar) |l Tp (-1x))
(z(x)
log
The first term can be bounded by C's by combining the assumptions. The second term can be evaluated as follows:
(x(
(x(r)
Σ,
En Tpa (v/(²) exp
= 1 + E, (T (r)) Tp expro v)
Ey T (2) exp (e( v)
max, exp (xr, y)
< 1
E By>apt(u)a) exp ro (z. 8) Σ
[pp(()))) (21x))
51+ Σ ())(())))= - Tpe (y/xx)
y
(by the assumption)
< C" V DKi, |l काp (-1x)
(by Pinsker's inequality)
< =1+C'VE.
Since log(1 + CE) = O(VE) holds asymptotically, finally we have (To((())))) (o(vl=))) s (((E).
13

---

Portable Reward Tuning
B. Experimental Setup
B.1. Image Classification Tasks.
Training Setups In training of either standard fine-tuning or PRT, we used the same hyperparameters following existing
work (Ilharco et al., 2023) as follows: learning rate 10-6, batch size 128, number of iterations 2000, optimizer Adam, cosine annealing with 500 warmup iterations. All models are trained on a single A100 GPU.
Fine-Tuning and Evaluation Dataset
We consider the following image classification tasks:
Aircraft (Maji et al., 2013): dataset with 100 classes of airerafts, 100 images per class.
Caltech101 (Li et al., 2022): A dataset with 101 classes of objects, 40 800 images per class.
Cars (Krause et al., 2013); A dataset with 196 classes of various cars.
CIFAR-100 (Krizhevsky et al., 2009): dataset with 100 classes 32x32 color images.
Country211 (Radford et al., 2021): A dataset of photos taken 211 different countries.
CUB (Wah et al., 2011) 1): A dataset of images with 200 bird species for fine-grained classification.
Flowers (Nilsback & Zisserman, 2008); The Oxford 102 Flower Dataset, containing images of 102 flower categories for fine-grained classification.
RESISC45 (Cheng et al., 2017): A dataset of images with 45 scene classes.
Models
We employed the following model architectures for the vision feature extractors of CLIP models:
ResNets (ResNet-50, ResNet-101; He et al. (2016))
ConvNext (Liu et al., 2022)
Vision Transformers (ViT-B-16, ViT-L-14;; Dosovitskiy et al. (2021))
B.2. Language Modeling Tasks.
Training Dataset and Settings For instruction tuning, used the Tulu v2 dataset (Ivison et al., 2023), which is large-scale dataset designed to improve the instruction-following capabilities of language models. The dataset includes diverse set of instructions and corresponding responses, covering wide range of topics and tasks. The training conditions are as follows: learning rate = 2 x 10 10-5, batch size = 128, number of epochs = 2, optimizer = Adam, warmup ratio = 0.03, and learning rate scheduler = linear. We conducted all training on NVIDIA 1000 GPUs.
Evaluation Dataset
We consider the following language modeling tasks: GSM8k and IFEval.
GSM8K (Cobbe et al., 2021): A dataset for evaluating the ability of models solve grade-school math problems. Evaluation based on the exact match metric, which measures whether the model's final answer exactly matches the correct answer.
IFEval (Zhou et al., 2023): A dataset for evaluating the ability of models to perform information extraction tasks. It employs four evaluation metrics: instruction-leva strict accuracy, instruction-level relaxed accuracy, prompt-level strict accuracy, and prompt-leve level relaxed accuracy, which comprehensively assess the model's ability to follow complex instructions. In this paper, we report the average of these four scores as the evaluation metric.
14

---

Portable Reward Tuning
Models
We evaluated the proposed PRT method the following models:
LLAMa 2 Series (Touvron et al., 2023a): A family of large-scale language models developed by Meta AI, available in various sizes, including 7B, 13B, and 70B parameters, designed for wide range of natural language processing tasks.
LLAMA Series (Dubey et al., 2024): family of large-scale language models developed by Meta AI, introduced in April 2024 with 8B and 70B parameter variants. Compared to Llama it features improvements in tokenizer efficiency, training data scale, and overall model optimization. The subsequent update, LLAMA 3.1, released in July 2024, further enhanced performance by refining pretraining methodologies while maintaining the same parameter sizes. In September 2024, LLAMA 3.2 introduced lightweight text models with IB and 3B parameters, optimized for efficiency in anrource-constraingg environments such as mobile and edge devices.
Qwen 2 Series (Yang et al., 2024b;a): A series of large-scale language models developed by Alibaba Cloud's Qwen team, designed for various natural language understanding and generation tasks. The initial Qwen 2 models were released with parameter sizes such as 1.5B and 3B, focusing on high-quality training data and diverse applications. The subsequent Qwen 2.5 series expanded the model range, introducing sizes from 0.5B to 72B parameters, with both base and instruction-tuned variants, further improving performance and efficiency.
Falcon Series (Team, 2024): A series of open-source large language models developed by the Technology Innovation Institute (TII) in Abu Dhabi, designed to provide accessible and efficient AI solutions. Released in December 2024, Falcon 3 models are available in 1B, 3B, 7B, and 10B parameter sizes, each offered both Base and Instruct variants. The Base models are tailored for general-purpor text generation, while the Instruct models are fine-tuned for conversational applications.
C. Memory and Speed Benchmarks
See Table for training-time time benchmarks, and Tables 2 and 3 for inference-time benchmarks.
Models
FT
PRT
Peak GPU memory
12.84 GB
13.08 GB
ResNet-50
Average time per batch
38.2643.62 ms
46.3643.34 ms
Peak GPU memory
20.17 GB
20.58 GB
VIT-B-16
Average time per batch
116.10.0.43 ms
151.1340.48 ms
Table 1: Memory usage and average time per batch in training with batch size 128.
Source Models
Target Models
Target FT (Oracle)
EFT
PRT
Peak GPU memory
1.46 GB
2.42 GB
2.18 GB
ResNet-50
ViT-B-16
Average time per butch
3.52 + 0.17 ms
10.27 = 0.22 ms
7.00 +: 0.12 ms
Peak GPU memory
2.96 GB 5.85 0.11 ms
3.44 GB
3.20 GB 9.43 + 0.19
ResNet-50
ViT-L-14
Average time per batch
12.52 + 0.21
Peak GPU memory
1.46 GB
2.29 GB 9.18 + 0.12 ms
1.88 GB 6,33 +: 0.07 ms
ViT-B-16 (LAION-400M)
(L.IOO)))
Average time butch
3.52 + 0).17 ms
Peak GPU memory
2.96 GB
3.79 GB
3.38 GB
VIT-B-16 (LAION-2B)
ViT-L-14 (LAION-2B)
Average time per batch
5.85 0.11 ms
11.73 - 0.15 ms
8.89 + 0.11 ms
Table 2: Memory usage and average time per batch in inference with batch size 128.
15

---

Portable Reward Tuning
Average Time
Token (ms)
Source Models Target Models
Llama2-7B
Llama3.2-1B
Llama3.2-3B
Llama2-13B
Llama3-8B
Pretrained
26.0
17.1 i1.1
EFT
39.8 (x 0.65)
24.4 (x0.7)
30.4 x0.56)
PRT
27.8 (x 0.93)
22.7
23.1 x0.74)
Table 3: Inference speed of Pretrained, EFT and PRT. The number following "x" in brackets indicates each method's token generation speed relative to the Pretrained model, whose speed is set to 1.0.
D. Qualitative Analysis
To analyze the behavior of PRT more detail, we examine the tokens generated by the model. In this analysis, we adopt Llama3-8B as the target model. In this analysis, used Llama3-8B the target model, while Llama-3.2-1B was used for training and reward-model initialization. Figures 8 and present the output obtained under these settings.
Pronge
Question: Junes has 6 more cundiess than Roben, John has twice candles Beben. If Joha has 54 cundies, how many mom candies does Juhn have than Jame??
Asswer
Respenve of Pretesined
Response
lat it he the ofmbrre cedies Rehert has. John hos "It candies lemes has 6 candies, We know that John has 54 canllio, so 28 54. RT Bamus toac 27- canlice John has 56. - 21 nom candics tlen Jamos
0.8
Pretrained
Pribust
6.
0.0
mone" Fa"
pkn
22
Next Candiduees Following" "..
Following
candies. Jumes bas
Jumes hzs"
Figure 8: Example of model response.
Prompt
Question: Genil and Julia divided $100 the nato 3:2. If Genald spent 510 on book. how much money did be have left?
Asswer:
Response of Pretroined
Response
Let be the anvent of money Julia hod. Thes 3x 10 X. So 20) Thersfors Genid had
10 left
Fretaiied
FRT
20
0.2
150
Canditisss
Tikis Condiiats
Thssfins
Figure 9; Example model response.
16

---

Portable Reward Tuning
E. Ablation Study: Impact of Source Model Choice PRT
In this section, we investigate the impact of source model differences on PRT by comparing GSMSK and IFEval scores when using Llama 3.2-1B and 3B, Qwen 2.5-0.5B and 1.5B, and Falcon 3-1B and 3B source models, All models were trained under the same conditions using the training data and hyperparameters specified in Appendix B. The results for each case shown in Figures 10 and Figures 11.
gsm8k
gsm8k
9sm8k
Pntrained
Frezened
Intraind
Exact Match (%)
Exact Match (96)
Bonut
Exact Match
30
3,2-18
3.2-3B
3-85
3.1-88
0.5B
7B
14B
3B
10B
Target Model
Target Model
Target Model
Llama-3.2-1B
(b) Qwen2.5-0.5B
(c) Falcon3-1B
gsm8k
gsm8k
gsm@k
Pettrred
Exact Makch (96)
PN
Exact Match
8
3-8B Target
1.58
Target
Target
(d) Llama-3.2
(c) Qwen2.5-1.5B
(f) Falcon3-3B
Figure 10: Evaluations of inference-time, instruction-tuned models on the GSM8k benchmark. Each subcaption refers to the source pretrained model, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot inference by each target model baseline, and Instruct means the instruct-tuned target model as an oracle result.
ifeval
ifeval
ifeval
Petrained
46-
Frexened
Inttained
Average (%6)
Average
Average
3.2-3B
Target
Targat
Target Model
Llama-3.2-1B
(b) Qwen2.5-0.58
(c) Falcon3-1B
ifeval
ifeval
ifeval
EP
Average age (%))
PF
INT
Average
3.2-3B
3-3B Target Model
3.1-88
1.58
38
7B
14B
38
78 Target Model
10B
Target Model
(d) Llama-3 3.2-3B
(e)
Qwen2.5-1.5B
(I) Falcon3-3B
Figure 11: Evaluations of inference-time instruction models on the IFEval benchmark. Fach subcaption refers the source pretrained model, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot inference by each target model baseline, and Instruct means the instruct-tuner target model as oracle result.
17

---

Portable Reward Tuning
F. Ablation Study: Instruction-Tunine with Multiple Random Seeds
We conducted additional experiments using the Qwen-2.5 0.5 model as the source model, where three independent training runs are carried out with different random seeds. As summarized in Table the variance of evaluation scores across seeds nearly identical between our method (PRT) and standard fine-tuning, indicating that PRT is as stable as standard fine-tuning for language-modee training.
0.5B
1.5B
3B
7B
14B
EFT
26.79 + 3.79%
45.94
4.40%
53.37
4.77%
66.14
2.43%
71.01 = 3.16%
PRT 26.69 + 1.45%
51.73 + 1.84%
62.37 + 2.14%
71.34 + 0.60%
77.23 + 0.20%
Table Performance comparison of EFT and PRT with standard deviations across model sizes.
G. Ablation Study: Robustness to Distribution Shifts in Inputs
Here we examine the robustness of PRT input distribution shifts. For this purpose, conducted experiments on CIFARI00 with Gaussian noise. We employ reward model (ResNet50, untuned) trained on clean data by PRT, and then additionally tune it by PRT on the noisy data (ResNet50, tuned) for only a few iterations. In Table 5, we observed that (1) PRT degrades its performance on noisy data (as well as standard FT), but (2) additional PRT training can recover its performance.
ResNet50
ResNet101
VIT-B-16
PRT on clean data (ResNet50, untuned)
71.70%
72.36%
79.5%
PRT on noisy data (ResNet50, untuned)
21.37%
34.77%
53,47%
PRT on noisy data (ResNet50, tuned)
71.60%
72.18%
78.06%
Table 5: Performance of PRT under clean and noisy training conditions across different backbone models
H. Ablation Study: PRT with LoRA
Here we conducted additional experiments of Instruction-tuninn by PRT with LoRA (Hu et al., 2022). Table presents the results of instruction-tunel models with LoRA, evaluated on GSM-8k, showing that LoRA actually works with PRT as well FT.
0.5B
1.5B
3B
7B
14B
LOR+ (0.5B)
29.72%
53.15%
65.28%
56.41 %
53.37%
PRT+L (0.5B)
21.83%
50.11%
63.15%
74.91%
73.62%
Table 6: Comparison of EFT+LORA and PRT+LoRA across model sizes
I. Empirical Evaluation for E and C'
We conducted additional experiments to measure KL-divergence between pretrained models and the constant C appeared in Proposition 3.2. In Table 7, reported averaged results over inputs from the dataset. These results indicate that (1) KI. divergences between similar models are small as expected, (2) KL divergences are affected by differences in pretraining datasets, rather than model architectures, and (3) the constant C is finitely bounded in this setting.
J. Evaluation on Various Vision Tasks
Figure 12 shows an extensive evaluation of inference-time tuning on various vision datasets.
18

---

Portable Reward Tuning
Fpt
Tt
C
RN50 (OpenAI)
RN50 (OpenAl)
0.0
19.41
RN50 (OpenAl)
RNIO1 (OpenAl)
0,0016
VIT-B (OpenAl)
0.0030
VIT-B (LAION-400M)
0.0107
VIT-L (OpenAI)
0.0044
VIT-L (LAION-400M)
0.0158
Table 7: Empirical Evaluation of and C' in Proposition 3.2.
K. Evaluation on Code Generation Tasks
To evaluate the code generation capabilities of PRT, we conducted a comparison using HumanFval (Chen et al., 2021).
HvmanEval A dataset for evaluating the ability of models to generate correct code based on natural language descriptions.
Result As shown in Figure 13, the effectiveness of PRT was confirmed for Llama2, Llama3, and Qwen2.5, On the other hand, for Falcon3 with target model sizes of 7B and 1OB, the score actually declined compared to the Pretrained model. We believe this is because, during training, the same 1B model was used the target, which exhibited little improvement the Pretrained model in code generation. As result, the PRT model did not acquire code generation capabilities during training; consequently, when applied to larger model sizes, it acts noise rather than providing a benefit.
L. Evaluation on Various NLP Tasks
To investigate the effectiveness of PRT, we conducted comparisons using diverse of evaluation datasets. addition GSM8k, IFEval, and HumanEval, used the following datasets for evaluation.
Academic/ / General Knowledge MCQ
ARC (Clark et al., 2018): The A12 Reasoning Challenge (ARC) dataset is designed to test a model's ability to answer grade-school level science questions.
-MMLU (Hendrycks et al., 2020): The Massive Multitask Language Understanding (MMLU) benchmark tests
model's s ability to perform wide range of language understanding tasks.
MMML Pro (Wang et al., 2024c): An extension of the MMLU benchmark with more challenging tasks.
Code Generation Programming
MBPP (Austin et al., 2021): The Mostly Basic Programming Problems (MBPP) dataset is designed to test a model's ability to solve basic programming problems.
Commonsense Reasoning
Hellaswag (Zellers et al., 2019): A dataset for evaluating commonsense reasoning and natural language inference.
GPQA (Rein al., 2023): A dataset for evaluating general- cenproe-purpose question answering capabilities of models.
Truthfulness
-TruthfulQA (Lin et al., 2021): dataset for evaluating the truthfulness of answers generated by models.
In Figure 14, compared PRT with the baseline using the Llama2 series these benchmarks. Each method employs the same models used in the NLP experiments in Section 4, which were trained with Tulu v2.
19

---

Portable Reward Tuning
On
ten
33
SADVION AvSE
IWOR-IDE
AlOVT
LAK4-2
IROWN
P1
30
P1
SACKTOO
INOITTI LANK
1208
19038
1200020
Eestight
Tramty211
210
0241
STEL
VELS
VELS
JASSCS
NABECAS
(a) ResNet-50 (OpenAD
(b) ConxNext (LA1ON-400M)
(c) VIT-B-16 (OpenAD)
(d) VIT-B-16 (LAION-400M)
(c) VIT-B-16 (LAION-2B)
Evaluations of inference-time tuned models for vision tasks, Each subcaption refers to the source pretrained model, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot classification by each target model baseline, and FT means the fine-tuned target model oracle result.
Figure 12:
20

---

Portable Reward Tuning
Target Mode
Taayet Modn
Brreet Wasel
(b)
Qwae2.5
(6)
Figure 13: Comparison of PRT and EFT on HvmanEval.
arc easy
arc_ arc_challenge
89
Accuracy (%)
Accuracy
EFT
ETT
Instract
78
13B
78
138
78
138
Target Model
Target Model
Target Model
(a) ARC Easy
(b) ARC Challenge
(c) GPQA Diamond
gsm8k
hellaswag
Prstuined
Netiaree
Acturacy (96)
Match
20-
Instuct
Pass:1
13B
78
13B
Target Model
Target Model
Target Model
GSM8k
Hellaswag
(f) HumanEval
ifeval
mmlu
FFT
Average (966
(96)
PRI
13B
78
138
78
138
Targot
Model
Tainget
IFEval
(h) MBPP
(i) MMLU
mmlu pro
truthfulqa mcl
Exact Match %
Prenened
Petehet
Intrac
78
138
78
3388
Target Model
Target Model
(i) MMLU PRO
(k) TruthfulQA
Figure 14: Evaluations of inference-time instruction-tunee models Llama 2 Series. Each subcaption refers the task name, and the labels in x-axis are target pretrained models. Pretrained means the zero-shot inference by each target model as baseline, and Instruct means the instruct-tuned target model an oracle result.
21


Reviews:
[{"review_id": "rithMyQ8hs", "reviewer": "Reviewer_8QF5", "summary": "This paper introduces a novel approach to reward tuning that can be transferred across model architectures. Instead of modifying a model parameters directly, the proposed approach trains an explicit reward model using the same objective as fine-tuning. At inference time, the reward model can be applied to any compatible foundation model without additional retraining. Experiments on vision and language tasks demonstrate that the proposed algorithm achieves comparable accuracy to traditional fine-tuning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, and finding our research direction promising. Here we would like to address the questions.\n\n> There is a mismatch between algorithm 1 and the text in lines 171-206. Reward models are simply fine-tuned on a given task with cross entropy loss. However, algorithm 1 seems to be training a reward model that trains on the difference between pre-training model's predictions and the true distribution from the task.\n\nSorry for the confusion. We reviewed this part and confirmed there is actually no mismatch. Here let us explain that. From the text, we wrote `The reward model $r_\\theta(x,y)$ is trained by simply optimizing the same loss function $L(p, y^*)$ as in standard fine-tuning` and the loss function is expanded as $L(p, y) = ... = -\\log \\pi_\\theta (y | x)$. This description corresponds to line 8 in Algorithm 1. Also importantly, we note that $\\pi_\\theta (y|x)$ is defined in eq. (6), as the product of the pretrained model $\\pi_\\mathrm{pt}(y|x)$ and the exponential of the reward model $r_\\theta(x,y)$. This corresponds to lines 6-7 in Algorithm 1, where we compute the product in the logarithmic space and then take softmax to obtain $\\pi_\\theta (y|x)$. Nevertheless, as these correspondences are not explicitly described in the text, we would like to add such annotations in our revised paper. Thank you for pointing out them.\n\n> In the first scenario, the reward function that will be used in algorithm 2 will be a log ratio of the fine-tuned model's prediction and original pre-trained model's prediction. On the other hand, in second scenario, the reward model can be used directly. Please clarify which version is used in the experiments. Furthermore, please clarify how many models need to be maintained to perform PRT at inference?\n\n(cont.) Here we note that our reward model is expected to play the same role as the log ratio of the two models. In other words, Algorithm 1 automatically learns a single model $r_\\theta(x,y)$ that implements this role, and that is the point of PRT. So the answer for the last question is: two models (new pretrained model & reward model) are used in PRT inference, while three models (new pretrained model & old pretrained model & old FT model) are used in the previous work.\n\n> In figure 3, why is Instruct's performance lower than others? Isn't Instruct a model that has been fine-tuned on the task itself? Please clarify the differences between FT and Instruct in figures 2 and 3.\n\nFirst of all, let us clarify the differences between FT (in vision experiments) and Instruct (in language experiments). FT means a model fine-tuned and evaluated on the same expert dataset (e.g., Cars, CUB, etc). On the other hand, Instruct means a model fine-tuned on a general dataset (i.e., the instruction-following data) and evaluated on **other** expert datasets like math and coding. So the answer for the second question is yes. Thus, the performance of Instruct on each expert benchmark may degrade on some dataset due to implicit bias occurred in insturction tuning.\n\n> In proposition 3.1, when defining r(y|x), shouldn't be utilized for sampling y, i.e. $E_{y\\sim \\pi_{\\mathrm{ft}}} r(x,y) = 1$?\n\nSorry for the confusion. Thanks to this comment, we have noticed a (non-crucial) typo. The probability for sampling y, $\\pi_{\\mathrm{pt}}$, is actually correct. However, we should write the condition for $r(x,y)$ should be $E_{y\\sim \\pi_{\\mathrm{pt}}} \\exp r(x,y) = 1$.\n\nLet us also explain about this condition briefly. By the mapping in proposition 3.1, $\\pi_\\mathrm{ft}$ is mapped to $r(x,y) := \\log(\\pi_\\mathrm{ft}(y|x) / \\pi_\\mathrm{pt}(y|x))$. This satisfies $$E_{y\\sim \\pi_{\\mathrm{pt}}} \\exp r(x,y) = \\sum_y \\pi_{\\mathrm{pt}}(y|x) \\exp r(x,y) = \\sum_y \\pi_{\\mathrm{pt}}(y|x) (\\pi_{\\mathrm{ft}}(y|x) / \\pi_{\\mathrm{pt}}(y|x)) = \\sum_y \\pi_{\\mathrm{ft}}(y|x) = 1,$$\ni.e., $E_{y\\sim \\pi_{\\mathrm{pt}}} \\exp r(x,y) = 1$ rather than $E_{y\\sim \\pi_{\\mathrm{ft}}} \\exp r(x,y) = 1$.\n\n> How is the regularization term applied? Is it simply applying an entropy regularization on the log likelihood ratio between the pre-trained model and the model that is being fine-tuned? What are the differences of this regularization term from the KL divergence loss between the pre-trained model and fine-tuned model's predictions?\n\nThe application of the regularization term is as described in eq. (12). The difference between the two KL divergence (eq. (5) and eq. (11)) is an important point. On one hand, KL divergence in eq. (5) measures discrepancy of two distributions over the label/vocabulary space. On the other hand, KL divergence in eq. (11) measures discrepancy of two (meta-)distributions over the set of probability models. As discussed in lines 267-274 and 220-229, since the latter KL divergence is computationally intractable, we proposed the entropy maximization regularization (eq. (5)).", "claims_and_evidence": "The authors conduct convincing experiments and comparisons to support the strengths of their proposed algorithm.", "methods_and_evaluation_criteria": "The experimental settings selected in the paper clearly support the claims made by the authors in the paper.", "theoretical_claims": "I quickly glanced the proofs for the theoretical claims to  support the design of their algorithm and they look reasonable.", "experimental_designs_or_analyses": "The experiment designs are sound and valid, relevant to the claims made in the paper.", "relation_to_broader_scientific_literature": "The paper studies an important question on adaptable and reusable reward models that can utilized across pre-trained models. The proposed algorithm is simple, with a theoretical basis.", "essential_references_not_discussed": "The authors have extensively discussed the important references.", "other_strengths_and_weaknesses": "The strength of the paper lies in its motivation to make fine-tuning more efficient by introducing Portable Reward Tuning (PRT), which allows reward models to be reused across different pretrained models. The authors carefully discuss the key principles behind their approach, framing fine-tuning as reward maximization with KL regularization, which reduces inference costs while maintaining performance. Additionally, the paper provides theoretical insights into how reward models adapt to different pretrained models. Overall, this paper presents a promising direction for making model adaptation more efficient and reusable.\n\nPlease find my questions below.", "other_comments_or_suggestions": "I don't find any typos.", "questions_for_authors": "There are a few questions that I would like the authors to clarify during the rebuttal.\n\n- There is a mismatch between algorithm 1 and the text in lines 171-206. Reward models are simply fine-tuned on a given task with cross entropy loss. However, algorithm 1 seems to be training a reward model that trains on the difference between pre-training model's predictions and the true distribution from the task. In the first scenario, the reward function that will be used in algorithm 2 will be a log ratio of the fine-tuned model's prediction and original pre-trained model's prediction. On the other hand, in second scenario, the reward model can be used directly. Please clarify which version is used in the experiments. Furthermore, please clarify how many models need to be maintained to perform PRT at inference?\n\n- In figure 3, why is Instruct's performance lower than others? Isn't Instruct a model that has been fine-tuned on the task itself? Please clarify the differences between FT and Instruct in figures 2 and 3.\n\n- In proposition 3.1, when defining r(y|x), shouldn't $\\pi_{\\text{ft}}$ be utilized for sampling y, i.e. $E_{y \\sim \\pi_{\\text{ft}}} r(x, y) = 1$?\n\n- How is the regularization term applied? Is it simply applying an entropy regularization on the log likelihood ratio between the pre-trained model and the model that is being fine-tuned? What are the differences of this regularization term from the KL divergence loss between the pre-trained model and fine-tuned model's predictions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8i38ccb1Ka", "reviewer": "Reviewer_BmQN", "summary": "This paper proposes Portable Reward Tuning (PRT), a new fine-tuning paradigm that decouples the “reward” from the foundation model itself, thereby making it “portable” to other foundation models of the same architecture family (with shared label or token vocabulary). Overall, the method aims to reduce repeated training costs and extra inference costs when one’s underlying foundation model is replaced or upgraded.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We are really encouraged by the positive feedback to our research direction. Here we would like to address your concerns or questions.\n\n> The authors only do a certain limited ablation around, e.g., the effect of different source vs. target architectures or training stable reward networks. More discussions on how stable the training is across random seeds or large scale might be helpful.\n\nThank you for the suggestion. Indeed, in vision experiments (e.g. Fig 2), we have already plotted one standard deviation (by black bars) over three random seeds in training, which indicates very small variance in PRT training/inference. Moreover, although training language models with multiple seeds is computationally heavy and thus not standard in previous literature, we additionally conducted PRT/Instruct tuning of Qwen2.5-0.5B model with three random seeds, and evaluated them with other pretrained models. These results show the stability of PRT is at the same level as standard FT, also in language experiments. We would like to add such discussions on training stability with respect to random seeds in our revision.\n\n||0.5B|1.5B|3B|7B|14B|\n|---|---|---|---|---|---|\n|EFT|$26.79\\pm3.79\\%$|$45.94\\pm4.40\\%$|$53.37\\pm4.77\\%$|$66.14\\pm2.43\\%$|$71.01\\pm3.16\\%$|\n|PRT|$26.69\\pm1.45\\%$|$51.73\\pm1.84\\%$|$62.37\\pm2.14\\%$|$71.34\\pm0.60\\%$|$77.23\\pm0.20\\%$|\n\n> The authors position PRT as a solution for “cross-model generalization of fine-tuned solutions.” They might connect more with methods in model distillation or universal prompt offsets, but that is less critical.\n\nThank you for suggestions. We agree that automatic prompt tuning can also be considered as inference-time tuning (but only for language models), and thus we would like to add a discussion about them. Also we would like to survey related literatures from the field of model distillation.\n\n> Weaknesses: Some details about how robust the reward model is to massive distribution shifts remain underexplored.\n\n> [...] Another possible future direction is combining PRT with partial re-training to reduce any performance gap.\n\nHere we will proceed by assuming `massive distribution shifts` refers to input distributions. To examine this, we conducted experiments with noisy data, i.e., CIFAR100 with gaussian noise ($\\sigma^2=0.01$). Here we employ a reward model (ResNet50, untuned) trained on clean data by PRT, and then additionally tune it by PRT on the noisy data (ResNet50, tuned) for only $1/10$ iterations. The results show that (1) PRT degrades its performance on noisy data (as well as standard FT) but (2) additional PRT training can recover its performance.\n\n|           |ResNet50|ResNet101 |ViT-B-16|\n|---|---|---|---|\n|PRT on clean data (ResNet50, untuned)       |71.70%|72.36%|79.5%|\n|PRT on **noisy** data (ResNet50, untuned)       |21.37%|34.77%|53.47%|\n|PRT on **noisy** data (ResNet50, **tuned**)|71.60%|72.18%|78.06%|\n\n> Could we do something approximate if the new model’s vocabulary is only partly changed?\n> How big a mismatch can we handle between old and new model vocabularies or label sets? Could partial alignment or token mapping methods mitigate that?\n\nWe sincerely agree that such a research direction is important for inference-time tuning, i.e., for both EFT and PRT. We decided not to treat this topic in this PRT paper as it should be focused on questions specific to PRT. Also handling different vocabularies is actually a highly non-trivial problem in language models [1, 2, for e.g.], and thus it should deserve another paper, which may require to match different vocabularies using, for e.g., edit distance, optimal transport, etc.\n- [1] Xu et al. \"Bridging the Gap between Different Vocabularies for LLM Ensemble\"\n- [2] Wan et al. \"Knowledge Fusion of Large Language Models\"\n\n> Do you observe any stability concerns or require special hyperparameters to get consistent results in PRT training?\n\nAs described in Appendix B, we basically used the same hyperparameters for both PRT and (E)FT in training and inference, for fair comparison. Some experiments (Fig.6, 7) includes EM regularization, but we found that the regularization coefficient does not affect the stability of final results.", "claims_and_evidence": "Overall, the paper offers sound conceptual reasoning plus decent experiments on a variety of image and NLP tasks. While the final accuracy typically remains below a fully re-fine-tuned new model, PRT competes well with the existing “emulated fine-tuning” baseline and indeed uses fewer inference resources.", "methods_and_evaluation_criteria": "The authors design PRT by explicitly parameterizing a reward function r_\\theta(x,y). They then train it via the same cross-entropy style objective normally used for fine-tuning, ensuring that the closed-form “max reward + KL constraint” policy solution matches the fine-tuned distribution. They measure classification accuracy in vision tasks, code generation pass rates, or language understanding metrics. The baselines are: (1) the original, non-fine-tuned “pretrained” model, (2) a normal “fine-tuned” model for that new architecture (oracle), (3) the existing “emulated fine-tuning” approach (EFT), and (4) PRT.\n\nThe evaluation is thorough for a wide set of classification and NLP tasks. However, exploring truly large language models in real production settings or more open-ended tasks would bolster real-world relevance.", "theoretical_claims": "The authors rely on the standard derivation of KL-regularized maximum entropy RL. They show that “fine-tuning is the closed-form solution to a certain reward objective.” They provide a fairly standard PAC-Bayesian argument in the appendix to justify generalization.", "experimental_designs_or_analyses": "The paper’s results cover classification (eight standard fine-grained or broad datasets), instruction following tasks (GSM8k, IFEval, MMLU, etc.), and code generation (HumanEval). This breadth is good.\n\nThe authors only do a certain limited ablation around, e.g., the effect of different source vs. target architectures or training stable reward networks. More discussions on how stable the training is across random seeds or large scale might be helpful.", "relation_to_broader_scientific_literature": "Fine-tuning with KL constraints is widely used in RL from human feedback. The authors cite prior works that interpret instruction tuning or RLHF in such terms. The authors position PRT as a solution for “cross-model generalization of fine-tuned solutions.” They might connect more with methods in model distillation or universal prompt offsets, but that is less critical.", "essential_references_not_discussed": "No major references are obviously missing.", "other_strengths_and_weaknesses": "Weaknesses: \tSome details about how robust the reward model is to massive distribution shifts remain underexplored. And the final performance is typically still below a real full re-fine-tune. If one can afford re-training, PRT is only partial.\n\nOverall, it’s a promising approach.", "other_comments_or_suggestions": "Could we do something approximate if the new model’s vocabulary is only partly changed? Another possible future direction is combining PRT with partial re-training to reduce any performance gap.", "questions_for_authors": "How big a mismatch can we handle between old and new model vocabularies or label sets? Could partial alignment or token mapping methods mitigate that? Do you observe any stability concerns or require special hyperparameters to get consistent results in PRT training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlSJGXzNoZ", "reviewer": "Reviewer_RJqU", "summary": "The paper proposes Portable Reward Tuning (PRT), a fine-tuning pipeline to enable efficient and reusable fine-tuning across different foundation models (FMs). It is especially useful when the old FMs are replaced by new FMs with different pre-training dataset and even different model architectures. Previous fine-tuning methods, i.e. inference-time tuning methods or emulated fine-tuning (EFT) in this paper, require running multiple models during inference, increasing overhead. The proposed PRT claims to avoid this by training an explicit reward model instead of modifying the pretrained model for each fine-tuning task. Experiments on vision and language models show that PRT achieves comparable accuracy to EFT with lower inference costs (evaluated by memory usage and time usage).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed feedback. Here we focus on answering major concerns/questions due to space limitation. However, we sincerely appreciate other feedback and will reflect them to our revisions, as well as discussions below.\n\n### Methods And Evaluation Criteria\n> However, I believe the evaluation metrics are not sufficient.\n\nDue to the space limitation, please refer to our answer to Reviewer BmQN on standard deviations.\n\n### Theoretical Claims\n> When replacing a Resnet foundation model with ViT, the ε should be larger than replacing a Resnet with a Restnet. I am not sure how drastic the ε could be in practice.\n\n> I am not sure how large the C could be and if this assumption is realistic for practice or not.\n\nThe practical validity of our assumptions is indeed an important perspective. Thus we conducted additional experiments to measure $\\epsilon$, KL-divergence between pretrained models, and the constant $C$ on CIFAR100:\n\n|$\\pi_{pt}$|$\\widetilde{\\pi}_{pt}$|ε|C|\n|-|-|-|-|\n|RN50 (OpenAI)|RN50 (OpenAI)|0.0|19.41|\n|RN50 (OpenAI)|RN101 (OpenAI)|0.0016|-|\n||ViT-B (OpenAI)|0.0030|-|\n||ViT-B (LAION-400M)|0.0107|-|\n||ViT-L (OpenAI)|0.0044|-|\n||ViT-L (LAION-400M)|0.0158|-|\n\nHere we reported averaged results over inputs from the dataset. These results indicate that (1) KL divergences between similar models are surprisingly small, (2) KL divergences largely depend on pretraining datasets, rather than model structures, (3) the constant C is seemingly finitely bounded.\n\n### Experimental Designs Or Analyses\n> [...] there are other lightweight efficient FT methods available as baseline methods, such as LoRA and adapters. It weakens the point that PRT is a superior alternative to FT.\n\n> FT method performs better than EFT and PRT in terms of accuracy and speed in most of the cases, yet FT is not a baseline. The impact is the same with the last issue.\n\nFirst of all, we would like to emphasize that our paper **does not** claim PRT is `a superior alternative to FT` unconditionally. If we have enough resources that we can ignore costs for training models or maintaining training data, we should employ repeated FT rather than one-time PRT. Rather, our paper claims that PRT is a better alternative in cases that we don't want to retrain new pretrained models for various reasons. In such cases, FT models are unavailable and thus should be viewed as unknown oracles for PRT/EFT. Hence **the fact that FT is better than PRT/EFT in accuracy/efficiency does not diminish their motivation.**\n\nAlso, we consider that the lack of comparison with efficient FT methods like LoRA **does not weaken our claims** because: (1) Both PRT and EFT are not efficient FT methods. Rather, they enable us to reuse a once-tuned model for (inference-time) tuning other pretrained models with different sizes or updated knowledge. (2) Since efficient FT methods like LoRA are designed to apply to FT, they should also apply to PRT. The following additional experiments (instruct-tuned with LoRA, eval. on GSM8K) verify that LoRA actually works with PRT as well as FT.\n\n||0.5B|1.5B|3B|7B|14B|\n|-|-|-|-|-|-|\n|EFT+LoRA (0.5B)|29.72%|53.15%|65.28%|56.41%|53.37%|\n|PRT+LoRA (0.5B)|21.83%|50.11%|63.15%|74.91%|73.62%|\n\n> It lacks analysis of Figure 11-12 in HumanEval Results (...). For example, PRT underperforms on Falcon-3 models when the target model is 7B and 10B. Things are different with other target models. [...]\n\nActually, PRT performs well on the GSM8K benchmark even with Falcon3 7B/10B models, which implies PRT indeed transfer the instruction-tuned ability to these models. The problem on HumanEval is mainly due to the difficulty of controlling downstream accuracy in instruction-tuning in some models, since the instruction dataset contains many data irrelevant to downstream tasks.\n\n> Other issues not discussed in the paper a) The reward model scalability. [...] b) What are the limits of the \"same label space\" assumption? And what if the label is partially different?`\n\na): Figure 13 shows the reward model scalability. Overall, a larger reward model leads to better accuracy with new pretrained models. Also, we additionally evaluated how inference time will be changed by scailing model sizes (1B → 3B). The results show the increase in average time per token is only 2% with PRT, while 25% with EFT.\n\n||Llama3.2-1B w/ Llama3-8B|Llama3.2-3B w/ Llama3-8B|\n|-|-|-|\n|EFT|24.4 ± 0.2 ms|30.4 ± 0.0 ms (× 1.25)|\n|PRT|22.7 ± 1.8 ms|23.1 ± 1.0 ms (× 1.02)|\n\nb): See our answer to Reviewer BmQN.\n\n### Questions:\nQ1: The instruction-tuned models are actually evaluated on downstream tasks like math and coding. However, as you pointed out, tasks that require feature extraction, rather than label distributions, are out of scope in this paper. This may be an interesting direction for future work.\n\nQ2: See above discussion.\n\nQ3: We confirmed that the reward model can be retrained and then reused with other pretrained models when input distribution changed. See our answer to Reviewer BmQN.", "claims_and_evidence": "As my understanding, with the proposed PRT method, this paper makes three claims and provides proof to support each claim. \nClaim 1: PRT maintains accuracy comparable to EFT. Claim 2: PRT reduces inference overhead compared to EFT. Claim 3: During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels).\nFor Claim 1, Figures 2–4 and Appendix E,F show PRT matches EFT on vision and language tasks. For Claim 2, the authors provide Tables 2-3 in Appendix. For Claim 3, the above experiment results can prove it. For example, for CLIP, we see generalization across ResNet and ViT, for LLAMA we see results from 1B model to 8B model, and for Qwen (from 0.5 B to 72 B), etc.", "methods_and_evaluation_criteria": "The proposed methods: PRT trains a reward model using cross-entropy loss, reformulating fine-tuning as reward maximization with KL regularization. The reward is combined with new models at inference via closed-form policy (Eq. 10).\n\nThe experiments are designed on broad coverage of FMs and benchmarking datasets. The paper provides evaluation on proposed method, baseline methods (EFT) and FT for both vision tasks and language tasks. However, I believe the evaluation metrics are not sufficient. For example, for vision tasks, only one quantitative metric, accuracy (Figs. 2–4) is used. The accuracy focuses on means with no standard deviations or confidence intervals. Is it possible to have more metrics to provide statistical significance of differences between PRT and baselines?", "theoretical_claims": "Following are theoretical claims I have checked.\n\nProposition 3.1: Establishes a one-to-one mapping between fine-tuned models and rewards. Comment: Valid, assuming rewards are scaled properly.\n\nProposition 3.2: Bounds the KL divergence between inference models when source and target pre-trained models are ε-close. \nComment: 1) When replacing a Resnet foundation model with ViT (just like the experiment does. See Figure 2 (a)), the ε should be larger than replacing a Resnet with a Restnet. I am not sure how drastic the ε could be in practice. 2) The authors assume that the maximum and mean value ratio of the exponential reward is bounded by some constant C. I am not sure how large the C could be and if this assumption is realistic for practice or not.\n\nProposition 3.3: I am not familiar with the PAC Bayes framework so I lack understanding of this.", "experimental_designs_or_analyses": "1. Quantitative experiments on Vision experiments and Language experiments\na) The settings of baselines. For each experiment, the comparison is limited within FT, EFT and PRT, and there are other lightweight efficient fine-tuning methods available as baseline methods, such as LoRA and adapters. It weakens the point that PRT is a superior alternative to fine-tuning.\nb) FT method performs better than EFT and PRT in terms of accuracy and speed in most of the cases (Fig. 2 and 10. Table 1 and 2), yet FT is not a baseline. The impact is the same with the last issue.\nc) About the training and inference speed, the proof is provided in the Appendix. I feel it is also an important proof because it is one of the keys to proving the efficiency of the proposed method. Is it possible to have them or part of them in Section 4? \n\n2. Qualitative experiments\na) It lacks analysis of Figure 11 - 12 in HumanEval Results (Appendix F). For example, PRT underperforms on Falcon-3 models when the target model is 7B and 10 B. Things are different with other target models. What does this phenomenon mean? Does it imply that PRT’s reliance on source model quality, which is not discussed in the main paper?\n\n\n3. Other issues not discussed in the paper\na) The reward model scalability. What factors can impact the scale of the reward model and how costs would change with its scale?\nb) What are the limits of the \"same label space\" assumption? And what if the label is partially different?", "relation_to_broader_scientific_literature": "This work, if solid, may contribute to Foundation Models of both computer-vison and natural language processing.", "essential_references_not_discussed": "To my knowledge, the current related work is comprehensive.", "other_strengths_and_weaknesses": "S1. The paper provides an interesting idea to bridge an old FM and a new FM with a tuned reward, without training from the beginning every time for a new FM. \nW1. However I believe the motivation of using PRT is not persuasive enough because 1) the direct FM performs well. 2) Other efficient fine-tuning methods are not compared to prove the high efficiency of PRT.", "other_comments_or_suggestions": "1. The ‘flower’ dataset used for experiments in Figure 2 is not mentioned in section 4 but only appears in the Appendix.\n2. Section 4.4 ‘Tables 2 and 3 show that PRT successfully reduces both inference speed and memory usage’ should be ‘...increase inference speed’ (increase speed or reduce time).\n3. For Table 1, and Table 2 in the Appendix, ‘Speed per batch’ should be ‘Time usage per batch’.", "questions_for_authors": "Q1. We have seen PRT is capable of handling different datasets and even model architectures. Is it possible to test PRT method on different downstream tasks, and how to do that? For example, for vision tasks, there are different downstream tasks including object detection, image segmentation, etc.\n\nQ2.  Is it possible to have a comparison of PRT with other latest effective fine-tuning methods?\n\nQ3. Could the trained reward model of PRT be ‘outdated’ and require re-training too?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Portable Reward Tuning (PRT), a new paradigm that reformulates fine-tuning as reward maximization with KL regularization. Rather than modifying the foundation model (FM) directly, the proposed method trains a reward model that can be reused across models with shared label or vocabulary spaces. This enables inference-time adaptation of newer FMs using previously learned reward models, reducing the need to retrain or run older models alongside new ones. This paper is a well-motivated, methodologically sound, and practically valuable contribution. I recommend acceptance.
Decision: Accept (poster)