Logits are All We Need to Adapt Closed Models
Authors: Gaurush Hiranandani
Haolun Wu
Subhojyoti Mukherjee
Sanmi Koyejo
Venue: ICML 2025 poster (2025)
Keywords: Distribution Shift
Black-box Model
Reweighing
Decoding
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: We propose a “Plugin” framework that uses token-logit reweighting at inference to adapt closed-source LLMs to new domains without retraining or accessing model weights.

Abstract: Many commercial Large Language Models (LLMs) are often closed-source, limiting developers to prompt tuning for aligning content generation with specific applications. While these models currently do not provide access to token logits, we argue that if such access were available, it would enable more powerful adaptation techniques beyond prompt engineering. In this paper, we propose a token-level probability reweighting framework that, given access to logits and a small amount of task-specific data, can effectively steer black-box LLMs toward application-specific content generation. Our approach views next-token prediction through the lens of supervised classification. We show that aligning black-box LLMs with task-specific data can be formulated as a label noise correction problem, leading to Plugin model -- an autoregressive probability reweighting model that operates solely on logits. We provide theoretical justification for why reweighting logits alone is sufficient for task adaptation. Extensive experiments with multiple datasets, LLMs, and reweighting models demonstrate the effectiveness of our method, advocating for broader access to token logits in closed-source models. We provide our code at this https URL.

Logits are All We Need to Adapt Closed Models
Gaurush Hiranandani Haolun Wu 23
Subhojyoti Mukherjee
Sanmi Koyejo 2
Abstract
Next Inferance
Plugin Model
rice ackitie
0.1
Many commercial Large Language Models (LLMs) are often closed-source, limiting devel- opers to prompt tuning for aligning content gen- eration with specific applications. While these models currently do not provide access token logits, we argue that if such access were available, it would enable more powerful adaptation tech- niques beyond prompt engineering. In this paper, we propose a token-level probability reweighting framework that, given access to logits and a small amount of task-specific data, can effectively steer black-box LLMs toward application-specifi con- tent generation. Our approach views next-token prediction through the lens of supervised classi- fication. We show that aligning black-box LLMs with task-specific data can be formulated as a la- bel noise correction problem, leading to Plugin model an autoregressive probability reweighting model that operates solely on logits. We provide thcoretical justification for why reweighting logits alone is sufficient for task adaptation. Extensive experiments with multiple datasets, LLMs, and reweighting models demonstrate the effectiveness of our method, advocating for broader access to token logits in closed-soures source models. We provide our code at this https URL.
0.1
0.3
Bluck-bor
niw
0.11
This shit
black
adides
shit
rice
0.1
0.7
Figure I. Inference phase of the Plugin model. The token proba- bilities are a product of the probabilities from the black-box model and reweighting model that denotes label transitioning.
Despite these constraints, closed-source LLMs are widely adopted across applications ranging from travel itinerary generation to tax advice, with developers largely relying on prompt optimization to achieve Noomain-spectiee outputs.
However, this reliance on prompt engineering insufficient for specialized tasks, e.g., those requiring brand-specific tone style. Consider a content writer aiming generate product descriptions that reflect brand's 's unique identity. Black-box LLMs, trained on broad datasets, often fail to meet such nuanced requirements. With access limited to gen- crated tokens, developers resort to zero-shot (Kojima et al., 2022) or few-shot (Song et al., 2023) prompting techniques. However, if model weights were accessible, advanced tech- niques like Parameter- Earamittr-Efficientt Fine- Fine-Tuning (PEFT) using LoRA (Hu et al., 2021), QLoRA (Dettmers et al., 2024), pre- fix tuming (Li & Liang, 2021), or adapters (Hu et al., 2023a) could be employed for fine-tuning. Yet, due to intellectual property concerns and the high costs of development, most commercial LLMs remain closed-source, source, and even with PPl-based fine-tuning options, concerns over data privacy discourage developers from sharing proprietary data.
1. Introduction
The rise of Large Language Models (LLMs) has revolution- ized generative Artificial Intelligence, yet the most capable models are often closed-source or black-box (Achiam et al., 2023; Bai et al., 2022a). These models generate text based on input prompts but keep their internal weights and training data undisclosed, limiting transparency and customization.
In this paper, we propose middle ground between general- purpose LLM creators and developers seeking application- specific alignment. We argue that providing access to token logits, in addition to generated text, would enable more ef- fective customization for downstream tasks. Viewing next- token prediction as a classification problem, we draw an analogy between LLMs and supervised classification mod- els. Since decoder-only L.MSs are trained to predict the next token given preceding tokens, aligning black-box LLMs to domain-specine data can be reframed as a label noise cor- rection problem in supervised classification. In this analogy, the LLM''s broad training data serves as proxy labels, while
WWrk done while visiting Stanford University. !Wrk done while PhD Candidate at UW Madison. 'Typeface "Stanford Univer- sity Miila- Quebec AI Institute Adobe Research. Correspondence to: Gaurush Hiranandani <gaurush @ typefaccc.aix Haolun Wu <haolunwu& stanford.cduu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Logits are All We Need to Adapt Closed Models
application-speeifie data represents true labels. This can be interpreted distribution shift scenario. For example, in "label shift" (Lipton et al., 2018), certain tokens may ap- pear more frequently in application-specifi data than in the LLM's original corpus. In "class-dependent or independent label noise" (Patrini et al., 2017), synonymous expressions or stylistic variations application data may diverge from those seen during model training.
2. Preliminaries
We begin by establishing the notation. The index set is de- noted as [c] (I.....c} for any positive integer c. Vectors are represented in boldface, for example, v, while matri- ces are denoted using uppercase letters, such as V. The coordinates of vector are indicated with subscripts, for instance, vj. The all-ones vector is denoted by 1. with its size being clear from the context. The c-dimensional sim- plex is represented as - C 1] Finally, a sequence (Ie, Tt-1,-..; 211 of size is denoted by Tt:1-
Inspired by the label noise correction method of Patrini et al. (2017), which estimates a transition matrix to correct class-dependent noise, we adapt this idea to black-box LLM alignment. Unlike prior work that modifies the loss and retrains the model, we lack access to the LLM's training data and cannot retrain the model. Instead, we estimate an autoregressive transition matrix from ieplicatiionspeciffe data and use it to reweight token probabilities at inference.
We assume access to language data for the target task, while the black-box LLM, trained broad world knowledge, is treated as having learned from a noisy version of this data. We seek to adapt the black-box model to align with the task-specific distribution. To formalize this, we extend the label-noise framework from supervised classification (Pa- trini et al., 2017) to the decoder-only language modeling-
This autoregressive extension is novel, as it accounts for de- pendencies on previously generated tokens when adjusting logits for the next token. By adapting label noise correction techniques to autoregressive language modeling, we present a practical method to align black-box LLMs using only log- its-without requiring access model weights or original training data.
Decoder-only models are trained using a next-token predic- tion objective. At each step, this setup resembles super- vised classification problem with [V] classes, where V is the vocabulary of tokens. Formally, the label space at step / is X₁ = (e' : E [[V]], where e' denotes the i-th standard canonical vector in RR RVV, i.e., e' E {0, 1)1V,, 11 = 1. The task at each step t to predict the next token æ; (de- noted as one-hot vector) given a sequence of tokens 11-1:1
Our contributions are summarized as follows:
1.
We formulate the problem of adapting black-box LLMS for upplication-specca content generation as a loss correction approach, requiring only token logits at each generation step. This bridges label noise correction in supervised classification with autoregressive language modeling (Sections 2 and 3).
One observes examples (x),, Zt-1:1) drawn from an unknown distribution p" (2,, p' (2))=x-111p" (x;-1:1 over V x vl--), with ex- pectations denoted by E Cross-entropy loss is typically used for training over the vocabulary tokens. As- suming access to token logits. and thus the softmax outputs, from the black-box LLM, we interpret the softmax output as vector approximating the class-conditions probabilities p' (aza (2.,-:1)). ), denoted as b(æ, (m:-1a) E AIV--
2. We propose an autoregressive probability reweight- ing framework, enabling token-level probability adjust- ment during inference. The resulting Plugin model dynamically reweights logits to align generation with task-specific data (Section 4).
To quantify the discrepancy between the target label x; = e' at step and the model's 's predicted output, we define a loss function |V| x A IVi-1 + R. A common choice in next-token prediction tasks the cross-entropy loss:
3. We provide theoretical guarantees, showing that under mild assumptions, the Plugin model consistently aligns probability estimates with the target distribution given sufficient application-specific samples. To our knowl- edge, this is the first work to establish such consistency in an autoregressive label noise setting (Section 5).
E(ee, (( = -e'') ) log b(x (1)):))
= log b(x, = e (l( (1)
With some abuse of notation, the loss in vec- tor form e AII-1 on every possible label is ((((z) (E(e", b(xz/ ((
RIV, computed
4. We conduct extensive experiments across four lan- guage generation datasets and three black-box LLMs. Our results, supported by multiple ablations, demon- strate that the Plugin model outperforms baselines in adapting black-box LLMs for domain-specific content generation (Section 7). Based on our results, we ad- vocate for publishing token logits alongside outputs in closed-source I.MMs.
3. Loss Robustness
We extend label noise modeling to the autoregressive lan- guage setting, focusing on asymmetric or class-conditional
2

---

Logits are All We Need to Adapt Closed Models
noise. At each step i, the label ae in the black-box model's training data is flipped to De E V with prob- ability p" (2x), while preceding tokens (21-1:1) re- main unchanged. As result, the blxck-box model ob- serves samples from a noisy distribution: p" (#:, (1-1a) = Ea, p" (Z)| (2. (x1-1:1).
3.1. Estimation of Transition Matrix
We assume access to small amount of target data for the task. Given that the black-box model is expressive enough to approximate p" ( 0t-1:1) (Assumption (2) in Theorem 3 of Patrini et al. (2017)), the transition matrix T₃ can be estimated from this target data. Considering the supervised classification setting at step t, let X represent all target data samples where 2; = e' and the preceding tokens are (x1-1:1). naive estimate of the transition matrix is: Tu, = (i; e |a., e¹) [ ΣxEx; b(ăr e 1i)) While this setup works for a single step t, there are two key challenges in extending it across all steps in the token prediction task:
We define the noise transition matrix T; E [0, || at step t, where each entry Tú, = pāx = = e²| = e*) represents the probability of label flipping. This matrix is row-stochastic but not necessarily symmetric.
To handle asymmetric label noise, we modify the loss e for robustness. Initially, assuming known Tr, we apply loss conrection inspired by (Patrini et al., 2017; Sukhbaatar et al., 2015). We then relax this assumption by estimating T3 directly, forming the basis of our Plugin model approach.
1. Limited sample availability: The number of samples where I; = and the preceding tokens (æt-1..... X1) match exactly is limited in the clean data, especially with large vocabulary sizes (c.g.. [V| = (100K) for LLaMA (Dubey et al., 2024)). This necessitates model- ing the transition matrix as a function of features derived from 1:-1: akin text-based autoregressive models.
We observe that a language model trained with no loss correction would result in a predictor for noisy labels (()1)_)) We can make explicit the dependence on Tr. For example, with cross-entropy we have:
b( p2 11)) = - log = e² (2122-11))
2. Large parameter space: With a vocabulary size of
[V|
[V = -(100K), the transition matrix T, has approx- imately 10 billion parameters. This scale may exceed the size of the closed-source L.MM and cannot be effectively learned from limited target data. Thus, structural restric- tions must be imposed on T, to reduce its complexity.
= =olog logg
p" (at e , e² )b(z,, e² cd(01-1a1)
j=l
IV| Tey, b(x:: ed 11)x)
= =-log log
or in matrix form
To address these challenges, we impose the restriction that the transition matrix T, is diagonal. While various con- straints could be applied to simplify the problem, assuming Tr is diagonal offers two key advantages. First, it allows the transition calttix-effectiveely vector in this case to be modeled using standard autoregressive language models, such as a GPT-2 model with k transformer blocks, LLaMA model with d-dimensional embeddings, or a fine-tuned GPT- 2 small model. These architectures can be adjusted based on the size of the target data. Second, a diagonal transition matrix corresponds to a symmetric label noise setup, where æ; = e flips to any other class with equal probability in the training data. This assumption, while simplifying, remains realistic within the framework of label noise models.
E(b(a, = log T, b(2://)-1a))
This loss compares the noisy label a; to the noisy predic- tions averaged via the transition matrix Ti at step f. Cross-
entropy loss, commonly used for next-token prediction, is proper composite loss with the softmax function its inverse link function (Patrini et al., 2017). Consequently. from Theorem 2 of Patrini et al. (2017), the minimizer of the forwardly-corrent corrected loss in Equation (3) on noisy data aligns with the minimizer of the true loss on clean data, i.c.,
class-independent
argmin E m 1:1 [(x1, T, b( x 11))
= -argmin ES 1:11: l(xx, b(x,, ((0))))))1))))
Enforcing a diagonal structure ensures efficient estimation of the transition matrix while maintaining practical applica- bility within our framework. Next, outline our approach for adapting closed-source language models to target data.
where w are the language model's weights, implicitly em- bedded in the softmax output b from the black-box model. This result suggests that if T2 were known, could trans- form the softmax output b( 11)1:1) using T.. use the transformed predictions as final outputs, and retrain the model accordingly. However, since T, is unknown and train- ing data is inaccessible, estimating Te from clean data is essential to our approach.
4. Proposed Method: The Plugin Approach
To estimate the autoregressive transition vector, we train an autoregressive language model on target data, which oper-
3

---

Logits are All We Need to Adapt Closed Models
ates alongside the black-box model during inference. This model acts autoregressive reweighting mechanism, adjusting the token probabilities produced by the black-box model. The combined approach, integrating probabilities from the black-box and reweighting models, is referred to as the Plugin model. The term Plugin is inspired by clas- sification literature, where plugin methods reweight proba- bilities adapt to distribution shifts (Koyejo et al., 2014: Narasimhan et al., 2015; Hiranandani et al., 2021). We now detail the training and inference phases, summarized in Algorithm (Appendix A) and illustrated in Figure 1.
5. Theoretical Analysis
We establish the convergence properties of Plugin, showing that after tokens, it accurately estimates the autoregressive noise transition matrix. Modeling the matrix as a function of an unknown parameter 0., prove that optimizing the autoregressive loss over token sequences enables consistent estimation of 0, with high probability. To our knowledge, this is the first finite-time convergence analysis for transition matrix estimation under autoregressive noisy loss.
Let F denote the history of selected tokens up to time 1- Let an unknown parameter 0. O C R governs the transition dynamics of label flipping between token pairs. The transition matrix at time 4, denoted as T, ([,, depends on O. and all previously observed tokens. Before proving our main result, we first make few assumptions.
4.1. Training the Plugin Model
During each training iteration, a sequence s of m tokens is passed through both the black-box model and the reweight- ing model to obtain token probabilities {b1, be..... bm } and (ri, T2,...,, m'm. respectively, where each b,, rs E A The final token probability from the Plagin model is computed by normalizing the element-wise product of these probabilities:
Assumption 5.1. Let Tr. (i, component of the transition matrix, and let be the transition function that de- termines the transition from Iy to Zy. where Ie is the 3; token selected at time t. Let TG E Rdd. We assume that f,, (0-; Ft-1)
denote the
b, O T2
p.
b O rilli
(4)
Ao and 2 fr, (0.; T, Ft-)) < A for some constant An > 0, A > 0 and for all steps t.
The sequence-level tross-entropy loss is given by:
Assumption 5.1 assumes the transition matrix depends on the distory-dependeer function fi, (-) with bounded gradient and Hessian, similar to assumptions in (Singh et al., 2023; Zhang et al., 2024) for other deep models.
l, = M
m
log(p:) O ej,
(5)
m
where the j-th token appears at the i-th position in the se- quence 8. During backpropagation, only the reweighting model parameters are updated, while the black-box model remains frozen. This formulation extends naturally to batch training, refining r; over iterations to approximate the tran- sition vector governing label shifts in the target data.
A Assumption 5.2. We assume the cross-entropy loss (5) is clipped by e >> 0 and upper bounded as e gelipped < CIVI fr. (0.; Tj Ft for any time i, where Y, is the predicted token class, fr, determines the true class and satisfies Assumption 5.1, and C'> 0 is a constant.
Assumption 5.2 ensures that the clipped log loss is up- per bounded by a smoother squared loss. For the re- maining of this section we refer to this squared loss at time 1. as (.()). Let the Plugin model minimize the loss e, (0), la (0),, ((()) over iterations. Let O, argmingee Ei=1 (0). At every iteration f, the Plugin algorithm looks into the history Fi and samples a token ot ~ Po. = =,,00 Or..
4.2. Inference from the Plugin Model
Given a fully trained reweighting model and access to the black-box model, token generation proceeds autore- gressively. At the first step. the black-box model pro- duces token probabilities b1, while the reweighting model outputs ri. The Plugin model selects the first token as x1 = argmax, (b, @ r1). For subsequent steps, given the previously generated tokens 1:11:11 we obtain prohabilities b, from the black-box model and r, from the reweighting model. The Plugin model then predicts the next token as: x,- = argmaxx (b, C rt).
Let (,) E. (0) and its expectation L)(0)
1 E Ea,-P (0) We impose regularity and smoothness assumptions on the loss function (0) as stated in Assumption B.1 (Appendix B). We are now ready to prove the main theoretical result of the paper.
The process continues until stopping criterion is met. Note that, this manuscript focuses on greedy decoding for infer- ence. Other decoding strategies, such as temperature scaling, top-p sampling, or beam search, can be incorporated by nor- malizing the element-wiss product of probabilities and using them as the final token distribution, as in Equation (4).
Theorem 1. Suppose (())).... ,.(()) : RV + R are loss functions from distribution that sat- isfies Assumptions 5.1. 5.2, and B.I. Define L.(
Sei-EEn,u 1 Ba,-Pa_,, (0) where,
4

---

Logits are All We Need to Adapt Closed Models
0. argmingeo E.=1 (0). If l is large enough such max (1))))
on token logits, enabling adaptation without modifying the underlying model. Thus, we emphasize that the Plugin model is not alternative to fine-tuning, but rather ap- proach that uniquely stands for adapting black-bor box LLMs which only provide logit access.
that Ingg(d))
min C.C.
C2
then for constant 2 2, universal constants C₁ C2
we have that
Steering and Aligning LLMs. LLM alignment methods primarily use reinforcement learning or instruction tuning. RLHF and DPO (Christiano et al., 2017; Quyang al., 2022; Rafailov et al., 2024) optimize model behavior via human preferences, with DPO eliminating reward model- ing. Constitutional AT (Bai et al., 2022b) aligns models us- ing self-generated principles, while instruction tuning (Wei et al., 2021; Sanh et al., 2022) adapts them via task-specific demonstrations. Unlike our approach, these methods require model weights and training data, limiting their applicability as baselines in our setup.
(1 - pb) o C7
S SB (0.) - L. (0.))
t 12/2
oi
max dE
(c) (0.))
$ (1 + po)
t
t7
where of = E 2 V (0.)
|
2
(v=c,(0.))
and Pt
(C,C² + 27) v log(dt)
Theorem I bounds the difference between the estimated and true average loss functions, showing that this gap diminishes as the number of training tokens increases. Since 0: argmingge E (0), the Plugin model progressively refines its estimation of the unknown parameter 0.. As the transition matrix (:(0:; (1. Ft-1) is derived from Jn (O.; J. I( which depends on Oz. training on sufficiently many tokens ensures an accurate estimation of each component of (1..
Calibration of LLMs. LLM calibration methods aim to align model confidence with predictive accuracy and adjust confidence scores but do not alter token predictions (Ulmer et al., 2024; Shen et al., 2024; Huang et al., 2024; Kapoor et al., 2024; Zhu et al., 2023; Zhang et al., 2023). In con- trast, our method reweights token probabilities at inference, enabling adaptation of black-box LLMs without modifying the model or requiring fine-tuning,
Our proof reformulates the problem as a sequen- tial hypothesis testing setting to estimate the aver- age loss function (0)) using the sequence of losses & (O)...., (0) (Naghshvar & Javidi, 2013; Lattimore & Szepesvári, 2020). Unlike prior work (Frostig et al., 2015; Chaudhuri et al., 2015), which assumes i.idd losses, the loss at time in our setting depends on all previous losses. Ad- ditionally, Mukherjee et al. (2022) study a different active regression setting without considering cross-entropy loss or transition noise matrices as in Patrini et al. (2017). We pro- vide brief overview of the proof technique in Remark B.9 (Appendix B), highlighting key novelties.
Black-box LLMs. Prior work explores various ap- proaches for adapting black-box LLMs without fine-tuning, though they differ fundamentally from our method. (Gao et al., 2024) infer user preferences through interactive ed- its but do not adapt models based on past language data. Diffusion-LM (Li et al., 2022) formulates text generation as a non-autoregressive denoising process, whereas our ap- proach reweights token probabilities autoregressively with- out requiring black-box model weights. Diseriminator- based methods (Dathathri et al., 2020; Mireshghallah et al., 2022; Yang & Klein, 2021; Krause et al., 2021) control generation based on predefined attributes, contrasting with our method, which enables free-form text adaptation. DEx- perts (Liu et al., 2021: 2024) combines expert and anti- expert probabilities; we incorporate a similar probability combining strategy in a modified baseline without a de- expert component. In-context learning (Long et al., 2023; Dong et al., 2024) offers a common adaptation technique for black-box models and serves as a baseline in our setup.
6. Related Work
Farameter-Efficient Fine-Tuning (PEFT). PEFT meth- ods adapt LLMs to downstream tasks while minimizing computational overhead. LoRA (Hu et al., 2021) and QLoRA (Dettmers et al., 2024) introduce low-rank updates and quantization for efficient fine-tuning, while prefix tun- ing (Li& & Liang, 2021), adapters (Hu et al., 2023b), and soft prompting (Lester et al., 2021) modify task-specific repre- sentations through trainable layers or embeddings. Torroba- Hennigen et al. (2025) further explore the equivalence be- tween gradient-based transformations and adapter-based tuning. However, these methods require access to model weights, gradients, or architecture details, making them unsuitable for closed-source LLMs and inapplicable as base- lines our setup. In contrast, our approach operates solely
7. Experiments
We divide this section into four parts. Section 7.1 evaluates Plugin on four text generation datasets across three black- box language models. Since the Plugin model is trained on top of black-box models, we refer to black-box mod- els interchangeably as base models. Section 7.2 discusses how Plugin can be applied on top of any prompt-tuning method as a wrapper, when logits are accessible. Section 7.3 presents ablation studies analyzing the impact of black-box
5

---

Logits are All We Need to Adapt Closed Models
Table 1. Performance comparison E2E NLG dataset. We show mean and standard deviation of the metrics over five seeds.
Model
Method
BLEU
Rouge-
Rouge-2
Rouge-L.
METEOR
CIDEr
NIST
GPT2-M
Zeroshot
0.0247
0,3539
0.1003
0,2250
0,3015
0.0156
0.6133
GPT2-M
ICL-1
0.0543
0.3431
0.1
0.2280=a.047 0.129920.033
0.34.340.051
0.0260z0.0e2
0.7767=0.090
GPT2-M
ICL-3
0.075020.035
0.395510.02e
0.167620.001
0.264920.000
0.397720.00s
0.025220.00 059
0.8993_a.0ro
GPT2-M
NewModel
#..011
0.5049+0.014
0.2742::0.013
0.3902
0.452L+0.016
0.3938
1.1927 20027
GPT2-M
WeightedComb
0.1709 10.008
0,4817.10.022
0.2447 10.011
0.3720
0,4071
10.025
0.332910.027
1.0864_0.000
GPT2-M
TempNeet
0.3425 00.0000
0.152610.002
0.2735 L0.010
1.0615
10..06
0.2826_0..00 10.007
GPT2-M
Plugin (Ours)
0.1863+0.010
0.5227+0.011
0.2612-+0.033
0.3728-0.608
0..057+0.012
0.3544=0.013
1.1241-0.000
GPT2-XL
Zeroshot
0.0562
0.4013
0.1636
0.2862
0.3697
0.0187
0.5338
GPT2-XL
ICL-1
0.0686gg.032
0.4016g0:012
0.140420.052
0.2745.0.022
0.3503
0.0353-20.015
0.7944_0.067
GPT2-XL
ICL-3
0.0980
0.41884o.0a0
0.1923
0.291220.681
0.3925
0.0250
0.9390_2..oa
GPT2-XL
NewModel
0.2372+0.011
0.5049+0.014
0.2742++0.01
0.3902-0.cos
0.4521+0.016
0.3938
1.1927-0.000
GPT2-XL
WeightedComb
0.118410.010
0.423710.01
0.185811..00
0.300410.010
0.3776
0.1818 023
1.0261 10.057
GPT2-XI.
TempNet
0. 031325
0.4642+0.017
0.25 0.2516
0.3021 =0.022
0.4126+0.025
0.3627 oss
0.8027 =0.047
GPT2-XL
Plugin (Ours)
0.2470xa.000
0.5536++.007
0.3084
0.4213
0.5057+0.0m
0.5455 0100
1.2736_00.01
LLaMA-3.1-8B
Zeroshot
0.3226
0.6917
0.4050
0,5004
0,6041
0,9764
1.1310
LLaMA-3.1-8B
ICL-1
0.3301
0.691420.027
0.412610.02z
0.5023_0.015
0.6037 20.015
0.9715
1.1735_0.006
LLaMA-3.1-8B
ICL-3
0.3527
0.693640.0mm
0.4217
0.5127=0.017
0 0.6202+o.009
0.9927
1.1672=0.047
LLaMA-3.1-8B
NewModel
0.2452:20.006
0.5347 005
0.2905
0,4097 200000
0,4812 0000
0,457
20:001
1.228120.00
LLaMA-3.1-8B
WeightedComb
0.351710.004
0.7040
0,4249 10.00s
0,5181
0.6206
1.0947
10.010
1.1737_0.015
LLaMA-3.1-8B
TempNet
0.3502+0.028
0.6927+0.006
0,4216+0.023
0,5027
0.6124+0.019
0.9625
1.1713=0.027
LLaMA-3.1-8B
Plugin (Ours)
0.3691 10.013
A.711310.00
0.4374 14477410.000
0.5247 L0.002
0.63921o 10.000
1.1441 00.030
1.1749 00.034
Table 2. Performance comparison on Web NLG dataset. We show mean and standard deviation of the metrics over five seeds.
Model
Method
BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
CIDEr
NIST
GPT2-M
Zeroshot
0.0213
0.2765
0.1014
0.1872
0,2111
0,0479
0,2340
GPT2-M
ICI-1
0.031710013
1..08810.021
0.131810.010
0,2346_0.019
0.287610.042
0.2715 0.0732.1..5a
GPT2-M
ICI-3
0.0461
0,3388+0.018
0.1378220.01
0.229 12291100 1g0000
0.3408+0.027
0.0748
0.3283_0.087
GPT2-M
NewModel
0.1071+0.005
0,3260g0.010
0.1496:20.04
0.2724g0.013
0.2642+00006
0.4327 023
0.2916=0.001
GPT2-M
WeightedComb
0.0692.0.007
0.359340.010
0.1568
0.2834_0.015
0.2379
0.1916.0.028
0.2996
GPT2-M
TempNet
0. 10.4
0.1 0.1526 0.352620.014
0,2731 20.018
0,3326
0,4237 :00 033
0.3002
GPT2-M
Plugin (Ours)
0.1280
0.4590 10.05
0.2226 005
0.3515 0.000
0.3832
0,7280 039
0.3060 000017
GPT2-XL
Zeroshot
0.0317
0,2992
0.1321
0,2417
0.1969
0.0491
0,1826
GPT2-XI.
ICL-I
0.051010.024
0.022310.026
0.152610.00
0.256210.880
0.2591 10.009
0.1336
0.2235 L0,033
GPT2-XL
ICL-3
0.074420.016
0.3383+0.056
0.1682++0.01
0.2651=0.028
+0.014
0.1675
0.2550
GPT2-XL
NewModel
0.107110.005
1o.00
0.1496.0.00
0,2724_0.013
0,26442 10.008
0.4327 10.023
0.2916
GPT2-XL
WeightedCornb
0.063610.006
0,3453
0.166620.003
0.2782
0,2871+0.000
+0.006
0.2460
0.2981 =0.018
GPT2-XL
TempNet
0.0925.0.008
0.3357
0.1663
0.2764_0.011
0,3025
0.4226:0.015
0.2837
GPT2-XL.
Plugin (Ours)
0.1673+a.004
0.4616+0.007
0.2527
0.3757
0.3895
+0,007
0.8987
0.26466...com
LLLMA-3.I-8B
Zeroshot
0.1453
0.5278
0.3030
0.3982
0,4314
0.6991
0,2684
LLaMA-3.1-8B
ICL-I
0,2166+0.na:
0.5944+0.ozz
0.3706
0.4667
0.5651#0.045
1.5719+0.00
0.2462
LLaMA-3.1-8B
ICL-3
0,203110.022
0.5937 10:019
0,3821 0.382110.015 10.015
0,4653
Lo.03
0,5682
10.046
1.3826
0.2469_ L0.045
I.IMMA-3.--BB
NewModel
0.1284.0.005
0.350610.009
0.1673.10.007
0.2879L..com
20.00
0.2921
10.008
0.4999
10,030
0.2973
LLaMA-3.1-8B
WeightedComb
0.1922:20.012
0,5986+0.01
0.3612-0.0122
0.4659220.008
000000
1.1855
0,2575m0.0om 0.000
LLaMA-3.1-8B
TempNet
0.2315
0.591640.016
0.3794=a.012
0.46202...00
0.5581 40.006
1.4826
0,2513
=0.00.00000 0200
LLaMA-3.1-8B
Plugin (Ours)
0.2542:0.004
0.6375+o.our
0.3873
0,4869 0,4869=0.007 =00007
0.5724
1.5911
0.2590 -0 00.03
model quality, Plugin's 's complexity, and architecture choices. Section 7.4 shows qualitative analysis and case studies.
7.1. Text Generation Performance Comparison
We evaluate Plugin on the text generation task using only the validation and test splits of all four datasets, reserving the train split for ablation studies (Section 7.3). Plugin and baseline models are trained on the small validation set, with performance measured on the test set. Additionally, we allocate 40% of the validation data as hyper-validation for cross-vale vooss-alidation of hyperparameters.
We evaluate Plugin on four text generation benchmarks: (a) E2E NLG (Dušek et al., 2020), (b) Web NLG (Gardent et al., 2017), (c) CommonGen (Lin et al., 2020), and (d) the Adidas product description dataset (adi, 2023). For the first three datasets, we use the rain-validation-tes, splits from the Transformers library (Wolf, 2020). To introduce distribution shifts, we filter Web NI.G's training data to include only infrastructure descriptions, while validation and test sets re- tain person descriptions. Similarly, CommonGen's 'ss training set is restricted to samples having man, while validation and test sets remain unchanged. Details of this setup are in Sec- tion 7.4. The Adidas dataset is split into validation and test sets. Data statistics are provided in Table 6, Appendix C.
Performance is reported using seven standard natural lan- guage generation metrics: (a) BLEU (Papineni et al., 2002). (b) ROUGE-1 (Lin, 2004), (c) ROUGE-2 (Lin, 2004), (d) ROUGE-L (Lin & Och, 2004), (c) METEOR (Banerjee &
Lavie, 2005). (f) CIDEr (Vedantam et al., 2015), and (g)
NIST (Doddington, 2002). All experiments are repeated
6

---

Logits are All We Need to Adapt Closed Models
Table Performance comparison CommonGen dataset. We show mean and standard deviation of the metries over five seeds.
Model
Method
BLEU
Rouge-1
Rouge-2
Rouge-L.
METEOR
CIDEr
NIST
GPT2-M
Zeroshot
0.0153
0.2216
0.0409
0.1527
0.2848
0.0001
0,3686
GPT2-M
ICL-1
0.01572a003
0.2580go.024
0.0362:2..om
0. 0.1388
0.2871
0.0222
0.3704_a..0
GPT2-M
ICL-3
0.055220.010
0.361020.010
0.1248
0.2680
0.40792o.138
0.136620. 125
0.5340
GPT2-M
NewModel
0.1260
0.4106+0.016
0.1683++0.013
0.3740
0.3600
0,4570
0.7113
GPT2-M
WeightedComb
0,056710.008 10.000
0.391 100000
0.1353
10.000
0.3280
0.2929 10.016
0.2623
10.042
0.4353 _0.008
GPT2-M
TempNet
0.1248
10.048
0.1528 10.015
0.3526_0.004
10.017
0.4492
10.020
0.4037 _0.058
GPT2-M
Plugin (Ours)
0.1366+0.008
0.4533+0.007
0.1878
0.3934=0.006
0.4095+0.011
0.5572+0.022
0.6395-m0.00
GPT2-XL
Zeroshot
0.0317
0,2992
0.1321
0.2417
0.1969
0.0491
0.1826
GPT2-XL
ICL-1
0.0508g0.023
0.3201
0.1526
0,2562-20.10
0.2591
0.1336:0.00
0.2235_0.000
GPT2-XL
ICL-3
0.0044
0.3383
0.1682
0.2651 =0.072
0,307 +0.073
0,1675
0.2550ma.oor
GPT2-XL
NewModel
0.1260+0.007
0.4106+0.016
0.1683+0.013
0.3740=a.coo
0.3600+0.024
0,4570
0.7113
GPT2-XL
WeightedComb
0.001410.020
0.336410.024
0.1347 009
0.29691..000
0.2921
10.018
0.2763.10 010
0.3352
GPT2-XI.
TempNet
0. 00.
0.3937
#0.026
0.14822#0.or
0.3625=0.018
0.3389 #0.019
0,4376
0.5927
=0.047
GPT2-XL
Plugin (Ours)
0.1091
0.4932+0.007
0.2288 00.004
0.4347 =0.007
0.4702
0.7283 20.012
0.6554-o.oes
LLaMA-3.1-8B
Zeroshot
0.0643
0.2776
0.1181
0,2488
0,3857
0,3155
0,3347
LLaMA-3.1-8B
ICL-1
0.0615
0.2697
0.115820.o0z
0.246920.0om
0.38222..00e
0.3005
0.3059 =0.00400004
LLaMA-3.1-8B
ICL-3
0.0635
0.2748+0.024
0.1225
0.3120
0,4012+0.029
0.3250 2..022
0.3794_a.oss
LLaMA-3.1-8B
NewModel
0.075320.004
0,371644000 005
0.1 0.1122:10.00
0.3404...00
0,2665 000
0.191920.011
0.6900.z..omr
LLaMA-3.1-8B
WoightedComb
0.178910.00
0,3485
0.1797.10.00
0.2981 1 100012
10.011
0,5503
10.00000000
0.5450
LLaMA-3.1-8B
TempNet
0.1524
0.3372+0.016
0.1524=00000
0.3298=0.017
0.3676+0.016
0,3986+0.08s
0.5286_a.coa
LLaMA-3.1-8B
Plugin (Ours)
0. 0.266510.00
0.5800 10.002
0.3139 1.0.00
0.5037 L0.004
0.5829 10.003
1.0876
0.7031 0.007
Table 4. Performance comparison on Adidas dataset. We show mean and standard deviation of the metrics over five seeds.
Model
Method
BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
CIDEr
NIST
GPT2-M
Zeroshot
0,0046
0,2488
0.0189
0.1353
0,1653
0.0312
0.6860
GPT2-M
ICI-1
0.008810.0a4
10.047
0.024710.66
0,1358 L0.044
10.022
0.046410.099
0.6793_.0.or
GPT2-M
ICI-3
0.0.21
0.2693+0.028
0.0262-20.044
0. 1..470.0.oa0
O. 1806200030
0.0415
0.7037=0.061
GPT2-M
NewModel
0.0515+0.016
0,2690go.014
0.0637:20004
0.1697
0.191820.013
0.0550
0.6682
GPT2-M
WeightedComb
0.0565
0.0495 0.2630
0. 0.1565=0.013
0.138:00.019
0. 0.6456 0.0585
GPT2-M
TempNet
0.044220.017
0.267220.010
0.0482.10.0z 022
0. 0,158222.020
0. 00000
0.0525 051
0.6533 008
GPT2-M
Plugin (Ours)
0.048610.006
0.276610.002
0.0515 007
0.1684 0.005
0.199410.000
0,0626
0.7919 -00 024
GPT2-XL
Zeroshot
0.0075
0.2309
0.0278
0.1438
0.1487
0.0184
0,4956
GPT2-XI.
ICL-I
0.010910.000
0.256711006s
0.0265 L..06510.004
0,1519 L0.89s
0.1649 10.052
0.031810.171
0.5133 _0.102
GPT2-XL
ICL-3
0.0295+0.057
0.2509+0.071
0.0395-20.043
0.153620.030
0.1658+0.om
0.0321 10.
0.5176
GPT2-XL
NewModel
0.0515
0.26901o.014
0.0637110.02
0,1697 20.00s
0,1918 10.013
0.055010.090 10.096
0.66822.0.00
GPT2-XL
WeightedCorn
0.0567
0.22102o.oz7
0.0714:4.015
0.1550.0.024
O. 0.1674
0.0183
0.4105_0..00
GPT2-XL
TempNet
0.053920.018
0,2598
0.0686
0.156220.019
0.1863
0.0462
0.5263
GPT2-XL.
Plugin (Ours)
0.0600+a.017
0.2710+0.a2n
0.0722
0.1725
0.1995
0.1195
0.6375
LLLMA-3.I-8B
Zeroshot
0.0120
0,2470
0.0318
0.1493
0.1526
0.0424
0.5285
LLaMA-3.1-8B
ICL-I
0,0220xo.044
0.2472+0.072
0.0405
0,1434=0.0s5
0.1686+0.041
0.0555
0.5078
LLaMA-3.1-8B
ICL-3
0.017710.04:
0,2385
12388510.00
0.036410.071
0,1408
Lo.830
0.1712 10.029
0.058710.100
0.5775 10.145
I.IMMA-3.--BB
NewModel
0.050610.011
0,270016.011
0.0634ga.008
0.1749 L0.006
0.1995 10.009
0.0575.0.001
0.6570_0.072
LLaMA-3.1-8B
WeightedComb
0.0357
0,2583+0.011
+0.015
0.1560.a.om
0.1706
0.0745 01145520.00 0s0
0.5927=0.077
LLaMA-3.1-8B
TempNer
0.0472:.0.16
00.04
0.0625 20.012
0.1625
20.000
0.1857
0.0586+0.100
0.5926_0.117
LLaMA-3.1-8B
Plugin (Ours)
0.0611
0.2714
0.0742:=0.021
0.1759
0. 0.190+0.020
0.6361 0.1293
over five random seeds, and we report the mean and standard deviation for each metric.
learns a global temperature per input and uniformly scales logits during generation. Since the black-box model weights are inaccessible, fine-tuning- plicable in our setting. Nonetheless, we include a compar- ison with LoRA in Appendix C.4 for completeness. This highlights Plugin's 's competitiveness despite operating under stricter access constraints than required for PEFT.
approaches are not ap-
We compare Plugin with the following baselines: (a) Ze- roshot: The black-box model directly performs text genera- tion without additional adaptation. (b) ICI.-1 (Long et al., 2023): One randomly selected validation sample is used as an in-context example. (c) ICL-3 (Long et al., 2023): Three randomly selected validation samples are used as in-context examples. (d) NewModel: A new language model is trained using the validation data. (e) WeightedComb (Liu et al., 2021): A new model is trained alongside the black-box model, with token probabilities computed as an + (1 1 a bb, where represents the probabilities from the new model and is cross-validated in (0.25, 0.50, 0.75). (f) Temp- Net (Qiu et al., 2024), recent logit-scaling approach that
All methods use the same prompts where applicable (Ap- pendix C.2) and employ greedy decoding. The base (black- box) models used are GPT2-M (Radford et al., 2019), GPT2- XL (Radford et al., 2019), and LLaMA-3.1 (Dubey et al., 2024). NewModel, WeightedComb, and the reweighting model in Plugin share the same architecture. For GPT-based models, these use Transformer encoder with one hidden layer and default configurations. For LLaMA- based mod-
7

---

Logits are All We Need to Adapt Closed Models
BLEU
Rouge-L
0.50
Table 5. Performance comparison of BDPL and BDPL. Plugin.
0.30
Base
Baze
Datasel
Method
BLEU
Rouge-L
METEOR
CIDE NIST
0.25
0.40
BDPL
0.2287
0.3922
0.4628
0.4216 0.8625
E2E NI.G
0.20
0.30
RDPI. + Plugin 0.4527
0.6027
0.6214
0.7002 200877
0.15
0.20
WEB NLG
BDPL
BDPL. Phugin 0.2137
0.1024
0.3017
0.5928
0.3527
0.5766
04421
1.8266
0.6142
0.2631
0.10
CommonGien
BDPL
0.1023
0.2936
0.3362
0.2517
0.4226
0.05
0.10
BDPL. Plugin 0.2614
BDPL
0.0417
0,1710
0.5241
0.1826
0.5016
0.0861 0.6034
0.8251
0.9261
Adidas
0.00
0.00
BDPL Plugin 0.0623
0.1759
0.2148
0.1325 0.7024
GPT2-M GPT2-M
GPT2-M
GPT2-M (5FT)
GPT2-M GPTZ-M GPTZ-M GPTZ-M
(zerosho)) (1FT) (2FT)
(seroshot) (1FT) (2FT) (9FT)
and ICL-3 using LLaMA-3.1 as the base model. Evaluators
Figure 2. Plugin with increasingly fine-tuned GPT2-M models on the E2E NLG dataset. Results demonstrate that as the quality of the base model improves, the performance of the Plugin improves,
select the prediction closest to the ground truth, with Plugin preferred in 81 % of cases. Details are in Appendix C.7.
BLEU
Rouge-L
7.2. Plugin as a Wrapper
0.25
0.40
If logit access is available, Plugin can be applied on top of any prompt-based method using its best-found prompt. For example, our Zeroshot prompt is reused across methods. We also apply Plugin to the Black-box Discrete Prompt Learning (BDPL) approach from Diao et al. (2022), follow- ing their recommended 75 API call budget. Table 5 shows results on all datasets with GPT2-XL as the base model. Plugin outperforms BDPL (see Tables 1-4), and their com- bination yields further gains, underscoring the utility of logit-level access in strengthening prompt-based methods.
0.20
0.30
Value
0.16
0.20
0.10
0.05
0.10
0.00
1000/0000 2-11 200 "AN 2200 2013 snlt Choice of the reweig hiing module
Choice of the reweightling module
Figure 3. Performance of GPT2-M with varying reweighting model complexities on E2E NLG (BLEU, ROUGE-L). A single- layer reweighting model yields significant gains, while additional layers degrade performance due to overfitting. Initializing with GPT2-Small as the reweighting model improves performance, demonstrating the benefits of leveraging small pretrained models.
7.3. Ablation Study
We now show ablation studies that reflect various aspects of the Plugin model. We display the results using GPT2-M as base model on the E2E NLG dataset. The observation is similar on other base models and datasets (Appendix C.5).
els, the architecture consists of a Transformer encoder with one hidden layer, 256 hidden size, 1024 intermediate size, and one attention head. Learning rate and weight decay are cross-validated over {le - 5, 5e - 5, le - 5e - 4, le 3, 5e 3} and (0.01,0, 0.1, 1, 10), respectively. Models are trained using Adam W with warmup followed by linear de- cay, and early stopping is applied if the hyper-validation loss does not decrease for five consecutive epochs.
Impact of Base Model Quality. We fine-tune GPT2-M for varying epochs, denoted as IFT (one epoch), 2FT (two epochs), and SFT (five epochs), and train Plugin model for each. Figure 2 shows that as the base model's ss task-specific quality improves, the Plugin's 's performance improves.
As shown in Tables 1-4 (the best is bold, the second best Is underlined), Plugin outperforms baselines across nearly all datasets, black-box models, and evaluation metrics. New- Model occasionally achieves higher NIST scores due to increased repetition of less- fess-frequent input tokens, but this comes at the cost of coherence, as reflected by other met- rics. WeightedComb does not perform well, indicating one combination for all tokens is not a good modeling choice. TempNet, which learns a single temperature per input and uniformly scales logits during generation, also underper- forms. In contrast, Plugin reweights logits at each timestep, enabling finer, sontext-sensitive adjustments.
Complexity of the Reweighting Model in Plugin. We train Plugin models with reweighting architectures varying from I to 12 transformer layers while keeping other config- urations unchanged. Additionally, we train a variant where the reweighting model is initialized with GPT2-Small. As shown in Figure 3, single-layer reweighting model yields significant improvements over the base GPT2-M model, while additional layers (c.g.. 2, 4. 8. 12) offer diminishing returns and slight performance decline due to overfitting on the small validation set of E2E NLG. This suggests that more data is required for learning complex reweighting models. Notably, initializing with a pretrained GPT2-Small substantially improves performance, underscoring the ad- vantage of using small pretrained models for reweighting due their inherent autoregressive properties.
We note that the absolute numbers may not appear com- petitive with state-of-the-an art results, because (a) we restrict to greedy decoding (Section 4.2), and (b) Web NLG and CommonGen use distributioo--siiteed subsets.
7.4. Qualitative Analysis and Case Study
We also conduct a human evaluation on 100 Adidas dataset samples, where three subjects compare outputs from Plugin
Plugin adapting to distribution shift. We evaluate Plu- gin on saaiitt Web NLG and CommonGen us-
8

---

Logits are All We Need to Adapt Closed Models
Given Product Attributes
Ground truth Reference
Input: Given the following attributes of product, write descriptioon.
Description: More of worith, less of the ueight. This adidas running tee gives you Light, stretchy layer to wear while training and recing, AENOREADY wicks noisture, stay dry before it gets cold, Slip the thumbhole sleeves your hands for extra coverage. This product is made with Prixegreen, a series high-perfomancee recycled raterials.
name [Adizero 1/2 2ip Long Steeve Teel, Category [Clathing), Price [68), Color [Black]
Plugin Model
Base Model
Description: adizero
Description: It's tong steeve tee that is made in the adidas style. It keeps all your busy days short to keep you Teeling fresh, energized and ready for whatever comes next. This product belongs on this page:Stay confortable through our anbition-drrien Line of high-pertortance recycted naterials.
long sleeve tee is great wey to keep your feet wart and dry. The fabric
reweighting
been designed with confort in wind.
Decoding Step 23, Generated Tokeno keep'
Decodling Step Generated Tokenn comfortaable"
Decoding Step Cenersted "recccled"
220-
606
634
632
0.23
69
Figure 4. Comparison of the adaptation ability between the base model and Plagin on Adidas dataset. Plugin, enhanced with a reweighting model, generates text that better aligns with the "Adidas domain". The hottom row illustrates token probabilities for key Adidas-related words at different decoding steps, showing how the reweighting model influences token selection.
ing LLaMA-3.I-SB as the base model. Web NLG training data contains only Infrastructure concepts, while validation and test sets include Person concepts. Similarly, Common- Gen training data features man, whereas validation and test sets contain both man and woman. The base model is fine- tuned on the training data, and Plugin is trained on validation data using the fine-tuned model as the base. These settings reflect different degrees of domain shift, even adversarial to some extent as the training distributions induce biases (e.g., overemphasis on infrastructure male-related concepts), and Plugin is tasked with correcting them during inference.
and Plugin (right); the third row visualizes model probabili- ties for "Adidas domain" words at three decoding steps.
As seen in Figure 4, Plugin dynamically reweights proba- bilities to align with domain-specific language. At step 23, "keep" is significantly upweighted. At step 48, "comfort- able" and "dry" gain prominence over "fit," which the base model favors. At step 59, "recycled" is preferred by Plugin, aligning with the ground truth, while the base model favors "running" and "products". This demonstrates that Plugin effectively steers generation toward domain-speciin termi- nology, whereas the base model, trained on broad corpora, lacks inherent domain preference.
Using GPT-4o (Hurst et al., 2024) as an evaluator, the fine- tuned Web NLG model generates only 17.99% Person- related sentences, while Plugin increases this to 71.34%. On CommonGen, the fine-tuned model generates 10.37% Woman-related sentences, whereas Plugin improves this to 31.92%. These results highlight Plugin's 's ability to adapt un- der distribution shift and mitigate biases in the base model.
Unlike methods that prune or suppress tokens, Plugin softly reweights token probabilities without eliminating any vo- cabulary candidates. This preserves full coverage while amplifying domain terms. To quantify this, we mea- sure the total occurrences of the top-50 "Adidas domain" words in generated outputs: Plugin includes 25.6% of these terms compared to 13.8% in the base model, indicating substantially improved alignment with domain language.
Case study: Plugin adapting to domain (extreme dis- tirbution shifts). We examine token probabilities during inference for LLaMA-3.1-8P and Plugin to assess domain adaptation in the Adidas dataset, which features product- centric language and a brand-specific tone that diverge significantly from the general pretraining distribution of the black-box box LLMs. This experimental setup can also be viewed as extreme distribution shift. Removing stopwords, we extract the t0p-50 most frequent words, defining the "Adi- das domain". Figure 4 illustrates this adaptation: the first row shows product attributes and ground- truth references: the second row compares outputs from the base model (left)
8. Conclusion
We propose Plugin, a token-level probability reweighting framework that adapts black-box LLMs using only logits and small task-specific data. Framing next-token prediction as a label noise correction problem, we demonstrate both theoretical guarantees and empirical effectiveness across multiple datasets and models. Our findings highlight the potential of logit-based adaptation and advocate for broader access to token logits in closed-source LLMs.

---

Logits are All We Need to Adapt Closed Models
Acknowledgements
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin-
non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint urXiv:2212.08073. 2022b.
HW acknowledges support by Fonds de recherche du Québec Nature et technologies (FRQNT) and Borealis AL SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, the MacArthur Foundation, Stanford HAI, OpenAI, and Google.
Banerjee, and Lavie, A. Meteor: An automatic metric for evaluation with improved correlation with human judgments. In ACL workshop on intrinsic and extrin- sic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Impact Statement
This work introduces a powerful middle ground between fully black-box APIs and fully white-box access to large lan- guage models (L..Ms), addressing a critical constraint faced by developers: the inability to adapt models when weights and architecture inaccessible. By leveraging token-level logits-withowe requiring access to model weights chitecture- our approach enables meaningful adaptation of closed-source LLMs for domain-specific tasks. This has far-reaching implications for both research and indus- try: empowers developers to customize models within crivacy-preservve IP-sensitive environments while ensur- ing greater control, transparency, and safety. Our findings advocate for broader logit access as a scalable, secure, and effective interface interface-bridging the gap between usability and protection of proprictary models and open new possibil- ities for equitable, context-aware language generation in real-world applications.
Chaudhuri, K., Kakade, S. M., Netrapalli, P., and Sanghavi, S. Convergence rates of active learning for maximum likelihood estimation. Advances in Neural Information Processing Systems, 28, 2015.
Christiano, F., Leike, J., Brown, T. Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play lan- guage models: A simple approach to controlled text gen- eration. In International Conference on Learning Repre- sentations, 2020.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
While Plugin effectively adapts black-box LLMs, it has some limitations, too. Since it only reweights token proba- bilities without modifying internal representations embed- dings, it may struggle with tasks requiring deep structural adaptations, such as executing complex reasoning. Further research on this aspect is needed. Additionally, although Plagin avoids full fine-tuning, training separate reweight- ing model introduces computational overhead compared to prompt tuning or in-context learning, with efficiency de- pending the complexity of the reweighting model and the availability of task-specific data.
Diao, S., Huang, Z., Xu, R., Li, X., Lin, Y., Zhou, X., and Zhang, T. Black-box prompt learning for pre-trained lan- guage models. arXiv preprint arXiv:2201..m 00020111111,,,. 2022.
Doddington, G. Automatic evaluation of machine trans- lation quality using n-gram co-occurrence statisties. In International conference on Human Language Technol- ogy Research, pp. 138-145, 2002.
Dong, Q., Li, L.,, Dai, D., Zheng, C., Ma, J., J., Xia, H., Xu, J., Wu, Z.,, Chang, B., et al. survey on in-context learning. In Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, 2024.
References
Adidas retail products
dataset.
Kag-
gle, 2023. URL attps://wwi kaggle.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXi::2407.21738 2024.
com/dataset com/dattasets//henaaaaa htwp///eeaaaaaee///////////
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint 07X1v:2303.08874, 2023.
Dušek, O., Novikova, J., and Rieser, V. Evaluating the state- of-the-art of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language, 59: 123-156, 2020.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05886 2022a.
Frostig, R., Ge, R., Kakade, S. M., and Sidford, A. Com- peting with the empirical risk minimizer in a single pass. In Conference on learning theory, pp. 728-763. PMLR, 2015.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
10

---

Logits are All We Need to Adapt Closed Models
Gao, G., Taymanov, A., Salinas, E., Mineiro, P.,, and Misra, D. Aligning IIm agents by learning latent preference from user edits. arXiv preprint urXiv:2404.1522 15269, 2024.
Krause, B., Gotmare, A. D., McCann, B., Keskar, N. S., Joty, S., Socher, R., and Rajani, N. F. Gedi: Generative discriminator guided sequence generation. In Findings of the Association for Computational Linguistics; EMNLP 202/, pp. 4929-4952, 2021.
Gardent, C., Shimorina, A., Narayan, S., and Perez- Beltrachini, L. Creating training corpora for nig micro- planning. In Annual Meeting of the Association for Com- putational Linguistics, pp. 179-188, 2017.
Lattimore, T. and Szepesvári, C. Bandit algorithms. Cam- bridge University Press, 2020.
Hiranandani, G., Mathur, J., Narasimhan, H., Fard, M. M., and Koyejo, S. Optimizing black-box metries with itera- tive example weighting. In International Conference on Machine Learning, pp. 2239-4249. PMLR, 2021.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning, In Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 2021.
Hsu, D., Kakade, S., Zhang, T., et al. A tail inequality for quadratic forms of subgaussian random vectors. Elec- tronic Communications in Probability, 17, 2012.
Li, X., Thickstun, J., Gulrajani, I., Liang, P. S., and Hashimoto, T. B. Diffusion-Im improves controllable text generation. Advances in Neural Information Process- ing Systems, 35:4328 4343, 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continuous prompts for generation. In Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4582-4597, 2021.
Hu, Z., Wang, I.,, Lan, Y., Xu, W., Lim, E.PP., Bing, I.,, Xu, X., Poria, S., and Lee, R. LLM-adapters: An adapter fam- fine-tuning of large language models. In Conference on Empirical Methods in Natural Language Processing, pp.
Lin, B. Y., Zhou, W., Shen, M., Zhou, P., Bhagavatula, C., Choi, Y., and Ren, X. Commongen: A constrained text generation challenge for generative commonsense reason- ing. In Findings of the Association for Computational Linguistics, pp. 1823-1840, 2020.
ily for parameter-efficieet
2023a.
Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.PP., Bing, Xu, X., Poria, S., and Lee, R. Alm-adapters: An adapter fam- ily for efffeieette fine-tuning of large language models. In Conference on Empirical Methods in Natural Language Processing, pp. 5254-5276, 2023b.
Lin, C.-Y. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pp. 74, 2004.
Lin, C.-Y. and Och, F. J. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statisties. In Annual meeting of the asso- ciation for computational linguistics, pp. 605-612, 2004.
Huang, Y., Liu, Y., Thirukovalluru, R., Cohan, A., and Dhin- gra, B. Calibrating long-form generations from large lan- guage models. arXiv preprint wrXiv:2402.06544 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Lipton, Z., Wang. Y.-.., and Smola, A. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122- 3130. PMLR, 2018.
Kapoor, S., Gruver, N., Roberts, M., Pal, A., Dooley, S., Goldblum, M., and Wilson, A. Calibration-tuning Teach- ing large language models to know what they don't know. In Workshop on Uncertainty-Aware NLP, pp. 1-14, 2024.
Liu, A., Sap. M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. Dexperts: Decoding-ting -ime controlled text generation with experts and anti-experts. In Annual Meeting of the Association for Computational Linguistics, 2021.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 2022.
Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Tuning language models by proxy. arXiv preprint arXi::2401.0856 2024.
Koyejo, O., Natarajan, N., Ravikumar, P. K., and Dhillon, S. Consistent binary classification with generalized performance metries. Advances in neural information processing systems, 27, 2014.
Long, Q. Wang, W., and Pan, S. Adapt in contexts: Retrieval-augmenned domain adaptation via in-context learning. In Conference on Empirical Methods in Natural Language Processing, pp. 6525-6542, 2023.

---

Logits are All We Need to Adapt Closed Models
Mireshghallah, F., Goyal, K., and KirggKiikpatrick, T. Mix and match: Learning-free controllable text generationus- ing energy language models. In Annual Meeting of the Association for Computational Linguistics, pp. 401-415, 2022.
calibration for large language models. In International Conference on Machine Learning, 2024.
Singh, S. P., Hofmann, T., and Schölkopf, B. The hes- sian perspective into the nature of convolutional neural networks, arXiv preprint arXiv:2305.0 090088 2023.
Mukherjee, S., Tripathy, A. S., and Nowak, R. Chernoff sam- pling for active testing and extension active regression. In International Conference on Artificial Intelligence and Statistics, pp. 7384-7432. PMLR, 2022.
Song, C. H., Wu, J., Washington, C., Sadler, B. M., Chao, W.-L., and Su, Y. Lim Aim-llanner Fhw-shot grounded plan- ning for embodied agents with large language models. In IEEE/CVE International Conference on Computer Vision, pp. 2998-3009, 2023.
Naghshvar, M. and Javidi, T. Active sequential hypothesis testing. 2013.
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L.,, and Fer- gus, R. Training convolutional networks with noisy labels. In International Conference on Learning Representations, 2015.
Narasimhan, H., Ramaswamy, H., Saha, A., and Agarwal, S. Consistent multiclass algorithms for complex perfor- mance measures. In International Conference on Ma- chine Learning, pp. 1398-24011 PMLR, 2015.
Torroba- Brrooba-Hennigen, L.., Lang, H.,, Guo, H., and Kim, Y. On the duality between gradient transformations and adapters. arXiv preprint a 3811,, 2025.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Ulmer, D., Gubri, M., Lee, H., Yun, S., and Oh, S. J. Cal- ibrating large language models using their generations only. arXiv preprint arXiv:2403.05577 240005593. 2024.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: method for automatic evaluation of machine translation. In Annual meeting of the Association for Computational Linguistics, pp. 31 1-318, 2002.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider:
Consensus- based image description evaluation. In IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Patrini, G., Rozza, A., Krishna Menon, A., Nock, R., and Qu, L. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
Wei, J., Bosma, M., Zhao V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. In International Conference on Learning Representations, 2021.
pp.
Wolf, T. Transformers: State-of-the-an natural language processing. arXiv preprint
Qiu, Z.-H., Guo, S., Xu, M., Zhao, T., Zhang, L., and Yang, T. To cool or not to cool? temperature network meets large foundation models via dro. arXiv preprint arXiv:2404.0455 04575, 2024.
2020.
Yang, K. and Klein, D. Fudge: Controlled text generation with future discriminators. In Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 3511- 3535, 2021
Radford, A., Wu, J., Child, R. Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Zhang, H., Zhang, Y.-F. Yu, Y., Madeka, D., Foster, D., Xing. E., Lakkaraju, H., and Kakade, S. A study on the calibration of in-context learning. arXiv preprint arXiv:2312.04001, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Zhang, Y., Chen, C., Ding, T., Li, Z., Sun, R., and Luo, Z.. Q. Why transformers need adam: A hessian perspective. arXiv preprint arXiv:2402. 16788. 2024.
Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika. L., Alyafeai, Z., Chaflin, A., Stiegler, A., Raja, A., Dey, M., et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
Zhu, C., Xu, B., Wang, Q., Zhang, Y., and Mao, Z. On the calibration of large language models and alignment. In Findings of the Association for Computational Linguis- tics: EMNLP 2023, pp. 9778-9795, 2023.
Shen, M., Das. S., Greenewald,
K., Sattigeri, P., Wornell, G. W., and Ghosh, S. Thermometer: Towards universal
12

---

Logits are All We Need to Adapt Closed Models
A. Algorithm Details
We provide summarized form of the training and inference algorithm for the Plugin model below.
Algorithm Training and Inference for the Plugin Model
Input: Black-box model B, reweighting model Rt, clean training data D, vocabulary
Output: Plugin model predictions 21.7 for a given sequence
1: Training Phase:
2: for each sequence & DD do Compute token probabilities {61, b b using B.
4: Compute token probabilities {r1, r2..... ..... Tm using R.
5: Combine probabilities: P; 1b,
h or for i E [m].
6: Compute sequence loss = -11 E Σ M log(p:) @ e,.
7: Update parameters of R using back-propagation. Freeze B.
8: end for
9: Inference Phase:
10: Initialize sequence #1:T = 0.
11: for each token position f = to T do
12: Compute token probabilities b, using B.
Compute token probabilities 7, using R.
Combine probabilities: P:- [b
or
15:
Predict token: æ; = argmaxy (p:)
16:
Append 1, to #1:7.
end for
18:
Return: #1:7
B. Proof of Main Convergence Theorem
We define the following assumption on the smoothness and regularity of the loss function.
Assumption B.1. We assume the following assumptions hold with probability 1:
1. (Convexity of t.): The loss function é, convex for all time s E [t].
2. (Smoothness (.): The fg smooth such that the first, second, and third derivatives exist at all interior points in e.
3. (Regularity Conditions):
(a) @ is compact and f, (0) is bounded for all and for s [4].
(b) 0. an interior point in θ.
(c) (e.) is positive definite, for all s E [t]
(d) There exists neighborhood B of 0, and constant C₁, such that (0), C₁ -Lipschitz. Hence, have that
(0), vee, (0))l. - C. | o' t,), for 0, 0' in this neighborhood.
4. (Concentration at 0.): We further assume that V[, (6. )l ((.))(( S C2 hold with probability one.
Lemma B.2. (Proposition of (Hsu al., 2012)) Let uI..... ur be a martingale difference vector sequence (ie..
u, Ui...., U 0 for all I,..., n such that
[u: S b 11-1 s and
for all i = 1,...., n. almost surely. For all t > 0
n
Pr
Σ
/ + / + (4/3)bt Se-
i=1
13

---

Logits are All We Need to Adapt Closed Models
ey log(dl)
Lemma B.3. The probability that [V (V=2(0))) 1 crosses the threshold
bounded by
P [
C21
ey (ot(
1
toy
Proof.
Define Us = V(Y, - Sr, (0,33;,, Fe 1) Then we have 11, U2 ................................ Uy as random vectors such that
E
U1
= E
Σ ==1
U
(02
un
u
< tt02
(o)))
Also we have that |us|| S C2. Finally we have that
V Ug] - Σ Pe. (f,, By, Fu-1) - f1, (0,; 2,. F (0, F(-11
Then following Lemma B.2 and by setting ey log(d() can show that
1
4C2
P
E
11 usl
81CBB+ +
i #=1
(0221
(122)10)))-
7
3e
8e +
4C2
P
u
> Ct +22
(0=c((00))-
3e
8E
t
ey log (dt)
<P
Σ ==1
2
((22)10)))-
>
C2p
tta
=P P
Σ U1
12
((22
> 4C21
t
8=1
1
07
- <exp(-crlog(tt)) (ot( =
<
dt
tc
The claim the lemma follows.
Lemma B.4. Let the j-th row and k-th column entry in the Hessian matrix V Vo (C)( denoted as 1o (L)) Then we have that
o (c.(0)) =2 0fi. (0; r,, R,,,,,,,,. Fo (0; T4, F+
(., + 2f. By, Fo (0, - Yr)) 02 82 (0: x,, Tj.
00,
00
00
Proof. This lemma follows from Frostig et al. (2015); Mukherjee al. (2022) adapted to our setting for the squared loss, and transition function fr, (0. 24, Ty. Ty..F F*-1). We want to evaluate the Hessian V2 (f, (6)) at any 0' E O. We denote the j-th row and k-th column entry in the Hessian matrix as [ (L, (0)) Then can show that
0 (0; x4, x1, F.- Y2)²
2(fr. (0; 21 F., F F - Ya) 0.,, 10;
V2 o (4.(0))) =
00,
00,
00,
00
2 (0; Fj Fj. Fo-1) 011, (0; Tz, (0; F
-2, 0fr, (0; Li, Ii F
00;
00
00
2 0... (0; T. Fj F 0ft. (0: F4,
0 fl, (0: Ij, 2j F
+ 2/1, (0; Tg F
00,
00,
00,00%
02 fr, 0 F1, Lj, F
0fi. (0, F F3-1) 0Y,
-2Y,
00
00,
30g
00,, (0; F= 0fr R0 F--1)
S1 (0,,,, FT-1)
+ [1, n, (0; Fo-1) ) -Y.)
00,
00,
00,00,
14

---

Logits are All We Need to Adapt Closed Models
The claim of the lemma follows.
Lemma B.5. Let the j-th row and k-th column entry in the Hessian matrix V o (B((=(0))F=-1) be denoted as Then we have that
[VI
Ōfi, 21, 0fr, (0; F
Vo ([((0)) -
> Po...()) i=1
00g
00k
+2 (0, (0. F+-1) -,, 0 fI, (0: Ii. T. F
00
Proof. This lemma follows from Frostig et al. (2015); Mukherjee et al. (2022) adapted to our setting for the squared loss, transition function fr, (0x; Te, F).11 and the sampling distribution P We show it here for completeness. Now we want evaluate the Hessian V V (0) any 0' O. We denote the j-th row and k-th column entry in the Hessian matrix as [ v= (BlL, (0) 5) Then can show that
V 0= [O) = VoLo (fh. (0;
2i (0;
F=ー1)=----- |[
210 F If,, (0; 2is T.. Fo-1))
IVI
V 0=
Po._. (i)
10 F5 + 12 00; Ti,
+
2fr, (0,,(0.;2s, Ty F 15,, (0; R,, F5-1
2
IVI
2
po._1 (f (,,,,,,,,11,,,,,)),000000,0,,,,,
+
i=1
[V|
2 0
Po. (i) (fi Fo-1) -
(6)
We now denote the j-th row and k-th column entry of the Hessian Matrix (Urr (0; Fe-1) f. 2i, 20, F+-1))²² as V ((fr, (0; z. Jy, fr. FF. jkb
Then we can show that
V a. (ff. (0; zi, -j F 1) fı. 00: Ij. 1) s/
9
afrr, (0; Fi, F.j ) fr, 0 Tj Fj (o-1))²
a
00,
0
2ff1. (0: z4, Fj. Fe-1) fl, (0. (0,; 24, Fy, ))) 0fi, (0: F(-1)
00,
00g
0
0f,, (0: 2i, Zj. F
Ofr, (0; xi, Fj FF
2fr, (0; Fí, Tj. F
(0.)
00,
00
00g
0fr, (0: Di, Fj. F
0f,, (0; Fí. Bj Fo-1)
+ 2f,,, Z4, I,, F* 02 f1, (0; Ni, T.. Fo-1)
00;
00
00,0%
-2f1, (0,)20 T, F 0 fr, 0 Fg Fj. F--1)
0ff. (0,I,, Zj. F 0fr, (0.; Iy Bj. F
00,02
00,
00
2 0f. D4, Fj. F 0f,, 0; Fi. Fe-1)
02 fr,
(0: v4 F
+ 2 (0, F+ - (0, Fo Fo-1))
00
a0x
00,00,
Plugging this back in Equation (6) we get that
@fr, (0, Fo-1) 05r, (0; ni Jj. Fro1)
V 1[00))1) 2 Σ Po (0)
(i)
jk
00,
00p
82 (011
(0; 2,. Fo
+2 (fr, 20
Fo-1) -f,,
Fo-1))
00,00%
15

---

Logits are All We Need to Adapt Closed Models
Lemma B.6. The sum of the difference of the Hessians Σ == lo (0) E V pr |
is given by
2(Y, - -1,, (0; xi, My ))- f1, Ii, T.jy Fo
V o o es (0) - EE ( re-1
00,00k
0,, 10 2jj
0fi. (0: It, Di, F*.
+ 2
00,
00
M
05,, (0: Fú, J. F.. 05,, Jz. Tyz T,, Fo-1)
Pa. (1)
00;
00
Proof. This lemma directly follows from Lemma B.4 and Lemma B.5. First note that the difference E V o (0)
V (0)
_e-1] gk is given by
V2 o l, (0) V L & (0) [=
(a)
fr, (0; F., Uz F 0ft, (0; Zi, Jy J. F+-1)
(r,,
- -Y.) 8 (0;,,0; xi, Iy, II. F+-1)
00,
d0
00,00%
M
0fr, (0; Jy. Fo-1) 05,, 0i, Fo-1)
2
Σ
Po. (i)
10,,, f1, (0. Ti, Ti, (.)
00,
00k
1=1
8² 5r,(0;
Ni, Tj. F=-1)
00,00:
= - f1. (0: Fí, TJ, 1)-1)) 8
F.. F
0fi. (0; Tú, TJ. F+-1)
+
00,00s
00,
00
I
-2 - Po. (1) 01. (0; F.
Uy. Bj, Fo-1) 0f,, (0; T,,
Tj, Fo-1) F
00,
002
where, (a) follows from Lemma B.4 and Lemma B.5. Plugging this equality in Equation (7) below get
V 6 0 (0) - E Lo (0) F+
2(Y, f1. (0; xz, Jj F fr, (0; x,, 21.
Ofr, (0; R,,T,, "1. (0,,00010000 (0;z,, Jy, F F*
+
00
00,
00,
8=1
[V]
0f1, (0, F+-1) 0f,, (0; 1i Jj Fo-1)
Σ Po. (i)
-2 (fi, 00 F,,
F Fy.
fr, (0,; 23 F)-1)
00,
00
821 fr, (0; Fi, j. Fo-1))
00,00
The claim of the lemma follows.
Lemma B.7. Let (.(0))= = Et. (o.) and V²L.(O.) = E.- (O.]). I..
Then we can bound the
P Xmnax (V2.x( V2
1 Xfer log(dl)
2
(dt)>)
where e is a constant.
16

---

Logits are All We Need to Adapt Closed Models
Proof. This lemma is different than Frostig et al. (2015); Mukherjee al. (2022) as it requires different concentration bound to take into account the squared loss Assumption 5.2 and the vocabu- lary size. Recall that (0.) + (o.) and (0*)
(=.](,00)) (0.) E (0.) Denote, V, 2Vo=0. St, (0; Iz; Fo Sr, (0; Fy 23, Fo-1)T 2 M Po..1 vi) (0; 1o-1)oV---- (0, Then show that,
We define
8 V 1 27 log dt)
P
Amax A (V.). v²2,(0))
t
P
Amax
VoLo, 7 s=1
1 Σ (0())
7 Σ
1
V 0=0. BlC, (1)
/ log(d!)
P
Amax
Vooo. t
Σ
(0) - Vo ((,0))))))))-)))))) >
8C 8C21V11 200 log(d))
CIV12
P
Amax
Σ
Y, f1, (0.; 2,
Fo-1)
C
V Aforyog((t log(dt)
V,
P
Amax
1 Σ
1
-2
(Y.
f1. (0.; x., Fー) ) Vo (i
2\
1
817 2 log (dt)
t
s=1
+ P
Amax
- Σ
1
V
>
2 V
8112 222 oogyyog(dt
t
s=1
(b)
81 N2e2 log(dl)
P
YY, - -1, (1, 00 TJ, F, Amax
11 By Fx >
/\
t
s=1
1
817² X90y (dt)
+ F
Amax (Va)
t s=1
2
t
(c)
2 Xfe7 log(dt)
(d)
1
2exp
S
(8)
4t
2tem
di
where, (a) follows from substituting the value of Vo_ (0) Vo_o. E[C, [[) from Lemma B.6, and (b) follows by triangle inequality, (c) follows by using two concentration inequalities stated below, and (d) follows by simplifying the equations.
Denote
Q.
-2 (r, 10, F, F+-1)) Amax (0. F+-1))
Also note that
17

---

Logits are All We Need to Adapt Closed Models
Amax
(0: s Y1 for all time using Assumption B.1.
(0.: 2 Y, 2Y,, - fI. (0,; IT, Amax (V fi. Fm 2c)= =P
- Σ Q. E9.2 2
P(
s=1
P 3 - Qo 2 =20)
(0)
S e [e -e-1Ei= 9o1
= = E - 0. [0--
( c
E E
e
xQr 1 [ B c ( 2: (
dxx 3a
(2A2171)))
E [ a 0 2.
(e- q.]
e de 2x-2
2/ (d) v exp
26 tnp x2
where (a) follows by Markov's inequality, (b) follows as Qs is conditionally independent given θz-1, (c) follows by unpacking the term for times and (d) follows by taking À = e/4t\/n? where A1 is defined in Assumption 5.1. Next we bound the second term of (8) below.
(a) =P (o))e))) 2 200e) - -drr [e (V.)) )V)
PO
mma 2))
PP
Amax (Va) 2 de
e X
E - Amax Amaax(V)) (V>)(
( M
Ammar (V.) 00 B. L E Amax (v,) (n)1 0
(0)
->CB
exp 2c x(n²² E e Eint Amer (V.) [
S e
-ee 20
( - (Vax) (V.)
= e
(d) e de 2cd 2-
2n
(e) S exp
2c tenp X
where (a) follows by Markov's inequality, (b) follows as Amax ( (V.) is conditionally independent given θ In the inequality (c) using the always valid upper bound of 2)1, we have that E[Anuxx (V.)l < 2X1. So the term in inequality (c) will become e 2dc X} +1Ads, Hence, we can upper bound the inequality (c) by a constant e >> 0 such that we have (lmse | 0.-1]= I] < 2x 2AxxA1 = exp(2X² 22n + 42A1) S exp(2cX>>,>>) The inequality (d) follows by unpacking the term for t times and (e) follows by taking À = e/ 2 and a1 defined in Assumption 5.1.
Lemma B.S. Let 0, 0. = =(V= -1 VV.((0.)
where O. is between Or and O.. Then we can show that
1/2 // 00. 0. 0 (.c,(0.) $ (0.) (v+2.(0))) -1 (V'C.
(2.) (0.) (+))))))))
18

---

Logits are All We Need to Adapt Closed Models
Proof. We begin with the definition of 0.-00 0. 1 (2C.(0.)
as follows:
10.-0 -9.1 0- (=C.(0.)
(a)
(0) 0.)
- T (0,, ((v:2:10))'v22
(o.) ((((0))(((²(((0)))) - (0.)
/ (V (V<C(0)) v2, (0.)
(0,)))))) 1/2 (122(0))"(100) (v)) 1/2 ' ||1² (04) (1:c10.)))
Vx (b) follows as 110, 10 = = -1 V..(O.), and (c) follows
where, (a) follows as | from Cauchy Schwarz inequality. The claim of the lemma follows.
Remark B.9. The proof of Theorem 1 consists of several steps. In the first step relate ViC,(0) to (O., for any 0 in a ball B around 0.. The ball B is assumed in Assumption B.1 to be a neighborhood where (0) satisfies a Lipschitz property. Assumption B.1 in Appendix B are standard and have also been made by Frostig et al. (2015); Chaudhuri et al. (2015); Mukherjee et al. (2022). Using Assumption 5.1 and Assumption B.1, we can show that for a large enough sequences of tokens t stated in Theorem 1 we have the following: (1) (O., lies between in the positive semidefinite order by scaled multiples of (OE, for any 0 E B, and (2) the empirical error minimizing 0, is Taylor series expansion around 0,
in the ball B with probability 1 1/t", which is the good event È. Then and using the fact that Vč, (@())) 0 along with the relation between V² and (O.), we can obtain an upper bound to ([@(t) 0. c,(0)) in terms of V (0.) )l(()=.....)- that can be shown to be decreasing with t. Further, (1(()) - / (o.) can also be used to obtain an upper bound to L. (0()) (0.) using a Taylor series expansion. ()(0)))(EE) where I(.) is the indicator. Since P(EE)= Σ 1/t", the second term can be bounded as maxøee (C) L. (0)) /t", while the first term simplifies to (1 + /t)
Finally we can bound E[C, (0)) B((C, (,(0)) B ((((((
Theorem 1. (Restatement of main theorem) Suppose e (1), (2(), .... (((0) : R + R are loss functions from a distribution that satisfies Assumptions 5.1, 5.2, and B.l. Define (.) = 1
1 E. Berrps_, [0) where,
may (a)(Cr(0)
i = argmingge E.z (0) If is large enough such that blog(d() S min
then for a
C/C2
C2
constant y 2 2, universal constants C2, Cz d, we can show that
(1 11-pl pt)
ot
Ct 27/2
of < (.(0,) - (0.) s(1+ pt)
max
o0
(c) (0.)
t
t
tt
where of = E 1 VL, (8.)
and P: = (CIC2 2n / log(dt)
Proof. Step 1: We first bound the
(2 - (0.) as follows
-22.
2 (22.(0) -
(4)
(0.) -
. + V.2. -.)
(6) C 0 0. 0.11=c..00))
1/2,, log(dt)
+V
(9)
1
where, (a) follows from triangle inequality, and (b) due to Assumption 1.1.3.d and Lemma B.7.
Step 2 (Approximation of L. (0.)) By choosing a sufficiently smaller ball B, of radius of min (1/ (10C1), diameter
(B)} ), the first term in (9) can be made small for 0 E B, Also, for sufficiently large f, the second term in (9) can be made
19

---

Logits are All We Need to Adapt Closed Models
arbitrarily small (smaller than 1/10), which occurs if logidt)
Hence for large and 0 B have
v2ž,(0) (O. 2 2,
(10)
2
Step 3 (Show 0, in Bi): Fix 0 between @ and 0. in BI. Apply Taylor's series approximation
(0) = Z. (0.) + (0.) (0- 0.) 2
(0- -0.) V2 0.)
We further reduce this follows:
()) - E. (0.) (a) (0.) T (0 0.) 10- 0.1.'
(b) 2 V (0.) T (0 - 0.) + 10 0. (.
2 - ||0 0. (.
V2 (0.)
4 (10- - 0, 0.1 (10 -10-0 0ollvec((0.)
(P22((0.))
(- V2 (0.) (
1 110 -0.11vc.10.
100 0 0.11
+ 4
(11)
T v2 v2 (0) (0 -.). and (b) follows as 0 is in between & and θ and then using (10). Note that in (11) if the right hand side is positive for some 0 B1, then f is not local minimum. Also, since v., (0.) + 0, for a sufficiently small value of vř, (e.) all points on the boundary of B1 will have values greater than that of 0.. Hence, we must have a local minimum of (0) that strictly inside B1 (for large enough). We can ensure this local minimum condition is achieved by choosing an large enough so that V
where, (a) follows as |10
0. 0.1 -.) 0.)
logidtt S e' min
diameter(8)) C2
C
using Lemma B.3 (and our bound on the diameter of B1 ). By convexity, we have that this is the global minimum, 0r, and so 0. E Bi for t large enough. We will assume now that is this large from here on.
): For the O(t) that minimizes the sum of squared errors, 0 (O,)). Again, using Taylor's theorem if 0g is an interior point, we have:
Step 4 (Bound 0. 0. @.llesc.te)))
0 = V v2 (0. 00.)
(12)
for some 0, between 0, and 0. Now observe that 0, is in B1 (since, for t large enough, O, E B, ). Thus it follows from (12) that,
0 = 0. -V=.(..)00).) -1 VŁ, (0.)
(13)
where the invertibility is guaranteed by (10) and the positive definiteness of V (0.) (by Assumption B.1 (3c)). We finally derive the upper bound to |0, - 0. (=C,(0.) as follows
| 0.
0.11
(4)
-
(LL 1/2 (?) (VP2 1/2 (2 (
(b)
cC2/ 7 log(dt)
(14)
where (a) follows from Lemma B.8, and (b) from Lemma B.3, (11), and is some universal constant.
Step 5 (Introducing 2): Fix between A. and 0, Apply Taylor's series
1 00 -0.-0.) VLL, (Z+) (Z)) T 0.)
(. L. (0.)
(15)
2
20

---

Logits are All We Need to Adapt Closed Models
Now note that both 0, and 2n are between 0, and 0., which implies 0, 0, and 2o 02 since 02 - 0.. By (9) and (14)
and applying the concentration inequalities give
V2C,(0)) - v2c, (0.) Pt
(16)
(2)) (0.) || S li 0.llveci(0.) S Pr
(17)
where = c + 221 log(d))
Step 6 (Define MI, and M2,)): It follows from the inequality (16) that
(.,) 3 (1+ = V' - (..) 3 pz L. (0.)
- (.CC))))CCC.)))) 3 1pl
( - v&c, (0.))l S Pt-
Then we can use the inequalities (16) and (17) to show that
(1 pb)) V L. (0.)
(2C
(1 p:) V L. (0.) 3 VLC, (Zx) (1+
Now we define the two quantities M1. and M2,: as follows:
M = (0.)) 1/2
(v22,10)) (V²L, ((00))))(( (0.)) 1/2
M2. = (V2 (0.)) -1/2
v'c, (Zi) (v'c, (0.)) -1/2
Step 7 (Lower bound (. - L. (0.)): Now for the lower bound it follows from Equation (15) that
(0.)) - L. (0.)
0 0.
r&h (2x) (0.-0.)
2
0 0. (2.00 -W.. (0: -0.)
2
(a) 1
u" M2
2
where, in (a) we define the vector u = (0)- (2L,(0.). Now observe from the definition of and then using the min-max theorem we can show that
(.) - C. (0.) N Amin (Mz.)) u² u1
1
00-00 0. 12
2 Amin (M2.)
c(((())) 1
(..) (0. -0.) 1
(TAC)(0) - "F=c,(0)
Amin (M2.)
2
1
Amin (Mi+)) Amin (Ma.r) (22.(0.) (0. -0.-0.) (V)c)e.))
2
(a) 1
(Amin (Mix))² 2 Amin (M2.)
V2, (0.) 2
2
where, in (a) we use the Equation (13).
21

---

Logits are All We Need to Adapt Closed Models
Step 8: Define (E) as the indicator that the desired previous events hold, which we can ensure with probability greater than
1 7
1 1
Then we can show that:
di
(0,) L. (0.) ZE
(2.(0)) - Li (0.) I(E)
1
(Amiiu (M...)) ?
Amin (M2.) VC, (0.) (( I(E)
2
1 dpr)
[1) V2, (0.)
2 (1
E
2
(1 mr) 1
_
V2. (0.) 1 (v)C)(0))
1 (1-1(noo.))
[1)0 (moote) e
@ (1 dp))
of 2
B
vï. V.. (0.)
V2 [ ((ooe))
2 (1 - cp) o - E
where, in (a) we have of == VL. (0.)
and c' is an universal constant.
(v=c,(0.))
Step 9: Define the random variable -= V2: (0.) (+=c()))))) With failure event probability of less than 2
1
di
for any 20, we have:
E z (2I(not {]] = E I EI) (z < <20) E I (011(zz2 (z²2))
(z² 2 20)
EToot E) + E n22 ( 722 2 2n)
< 20
722 22
20 2t
+R
E
20
20
E
[
26
20
E
[Z4]
27/2
where 20 17/2 VE
Step 10 (Upper Bound): For an upper bound we have that:
E [[,(0.) - (0.)
E (c.()) (1(0)))-o) L. (0.)) I(E) +E (L. I(not E)
maxace (Li(0) L. (0.))
s (c((0)) L. (0.)) 1(8)
17
22

---

Logits are All We Need to Adapt Closed Models
since the probability of not & is less than Now for an upper bound of the first term, observe that
La (0.) -c
I(8)
$ (Amuse (Mr..))
Amax (M2.))
V2, (0.) ||
(F=C I(E)
$ (1 + pi))
V [c. (v)c)(0))
[(8)
2
V2, V2, (0.) (1)0
$(+
o7 t
= =(1+dp))
where, d' is another universal constant.
C. Experimental Details
C.1. Dataset Statisties
We provide the processed data statisties in Table 6. We highlight that due the black-box assumption of the base model, the training set is used for ablation and qualitative analysis in Section 7.3 and Section 7.4.
Table Processed Dataset Statistics. Training only used for ablation and qualitative analysis due the black-bor box model assumption.
Dataset
Train
Validation
Test
E2E NLG
33,525 2,732 (filtered by categories)
4,299
4,693
Web NLG
844
720
CommonGen
1,476 (filtered for "man""
2,026
1,992
Adidas
745
100
C.2. Prompts
We now describe the prompts we used for the four datasets and three models.
E2E NLG Dataset
For the GPT2-M model, we use the prompt:
Given the following aspects of a restaurant, [attributes], a natural language sentence describing the restaurant is:
For the GPT2-XL model, the psompt
Imagine you are writing senteenee description for a restaurant, given the following aspects: [attributes), rumanbreedable natural language sentence describing the restaurant is:
For the LLaMA-3.1-8B model,
Please convert the following attributes into a coherent sentence. Do not provide an explanation.
Web NLG Dataset
For the GPT2-M model, we use the prompt:
23

---

Logits are All We Need to Adapt Closed Models
Convert the following facts into a coherent sentence:
Facts:
[ [facts]
Sentence:
For the GPT2-XL model, the prompt is:
You are given the following facts. Facts: sentence summarizing the facts is:
[facts] short, coherent
For the LLaMA-3.1-8B model, we use:
Do not provide an explanation or follow-up. Just convert the following facts of entity into coherent sentence. Facts:
[facts] Sentence:
CommonGen Dataset
For the GPT2-M and GPT2-XL models, the prompt:
One coherent sentence that uses all the following concepts:
[concepts],
is:
For the LLaMA-3.1-8B model, we use:
Please write a coherent sentence that uses all the following concepts.
Concepts:
[concepts] Sentence:
Adidas Dataset
For the GPT2-M and GPT2-XI. models, we use the same prompt:
Given the following attributes of product, write description.
Attributes:
[attributes] Description:
For the LLaMA-3.1-8B model, we use:
Please write a description of this product given the following attributes.
Actributes: [attributes] Description:
For in-context learning, we simply add sentence at the beginning of the prompt before adding the samples in the prompt: Below are
list of demonst inationsss ions:
For the qualitative analysis on the distribution shift in Section 7.4, we ask GPT-4o with the following prompt:
For Web NLG dataset:
Focus on all the samples, how much percentage is related to 'Person''?
For CommonGen dataset:
Focus on those samples whose target is related gender, how much percentage is related to ''woman' !
C.3. Metries
We report performance using seven standard metries often used in the natural language generation tasks. These are: (a) 2002) (measures n-gram overlap between the generated and reference texts, emphasizing precision). (b) ROUGE-1 (Lin, 2004) (computes unigram recall measure the overlap between generated and reference texts), (c) ROUGE-2 (Lin, 2004) (extends ROUGE-1 to bigrams, measuring the recall of two-word sequences), (d) ROUGE-L (Lin & Och, 2004) (uses the longest common subsequence to evaluate recall), (e) METEOR (Banerjee & Lavie, 2005) (combines unigram precision, recall, and semantic matching to assess similarity), (f) CIDEr (Vedantam et al., 2015) (measures consensus n-gram usage across multiple references, with tf-idf weighting), and (g) NIST (Doddington, 2002) (similar to BLEU but weights n-grams by their informativeness, favoring less frequent and meaningful phrases).
BLEU (Papineni
C.4. Performance and Efficiency Comparision with Parameter-Eflin Fine- Fine-Tuning
While our work focuses on black-box LLM adaptation where model weights are inaccessible, we include controlled comparison with Parameter- Fine-Tuning (PEFT) methods. Specifically, we implement LoRA (Hu al., 2021) with
24

---

Logits are All We Need to Adapt Closed Models
Table Comparison between Plugin and PEFT (LoRA, r= on four datasets using GPT2-XL and LLaMA-3.1 as base models. We
show mean and standard deviation of the metrics over five seeds.
Model
Method
BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
CIDEr
NIST
E2E NLG
GPT2-XL
Zeroshot
0.0562
0,4013
0.1636
0,2862
0,3697
0.0187
0.5338
GPT2-XL
LoRA (1=8)
0.2517_0.012
0.5712-0.000
0.307920.013
0.4317g0.011
0.5162:0.004
0.5225
1.217220..01
GPT2-XL
Plugin (Ours)
0.2470=a..0s
0.5536
20
0,4213=a.00s
0.5057
0.5455
1.2736_g0.05
LLaMA-3.1-88
Zeroshot
0.3226
0.6917
0.4050
0,5004
0.6041
0.9764
1.1310
LLaMA-3.1-8B
LoRA (r=8)
0.3702_a.016
0.7125
0.4236+0.014
0.5345=0.012
0.6413
1.1028
003
1.1827=0.085
1LLMMA331-8B
Plugin (Ours)
0.3691
0.71 0.711310.002
0.437410.004
0.5247 10.002
0.6392.0.000
1.1441 10.000
1.1749_0.034
Web NLG
GPT2-XL
Zeroshot
0.0317
0.2992
0.1321
0.2417
0.1969
0.0491
0.1826
GPT2-XL
LoRA (r=8)
0.1723_0.007
0,460420.010
0.261820:011
0.3628:a.015
0.401220.017
0,9018 028
0,2736_0.014
GPT2-XL
Plugin (Ours)
0.1673=4.004
0.461640.007
0.2527
0.3757
0,3895+0.007
0.8987 19887=0.013
0,264620.00z
LLaMA-331-8B
Zeroshot
0,1453
0,5278
0,3030
0.3982
0,4314
0.6991
0,2684
LLaMA-3.1-8B
LoRA (1=8)
0.2638_o.oo
0,6238
0.3927 009
0,472610.008
0.592710.013
1.6421
028
0.2379 1.0008
1.L.MA-3.--8
Plugin (Ours)
0.2542_0.004
0.6375go.005
0.3873
0.486920.00m
0.5724
1.5911 10011=00000 #0.040
0.2590
CommonGen
GPT2-XL
Zeroshot
0,0317
0.2992
0.1321
0.2417
0,1969
0.0491
0,1826
GPT2-XL
LoRA (r=8)
0.1826_a.o27
0.2137 0.0027=0.010
0,4447
0,47262o.009
0.7182
0.6725
GPT2-XL
Plugin (Ours)
0.4932,..00 0.1791_0.014
007
0.2288 004
0.4347
0.4702,00006
0.7283
0.6554_0.ooo 038
LLaMAA331-8B
Zeroshot
0.0643
0.2776
0.1181
0.2488
0.3857
0.3155
0.3347
I.LMMA-3I--BB
LoRA (r=8)
0.2736 _0001
0.5829 10.009
0.3206 1.0.0000000000
0.5026 10.012
0.5927 10.016
1.1121
0.7926_0.08a
LLaMA-3.1-8B
Plugin (Ours)
0.2665_0.010
0.5800
0.3139
0.5037
0.5829+0.008
1.0876+0.020
0.7031=0.007
Adidas
GPT2-XL
Zeroshot
0.0075
0.2309
0.0278
0.1438
0.1487
0,0184
0.4956
GPT2-XL
LoRA (1=8)
0.06292a.02z =0.028
0.28 0.2816+o.u8
0.0719#0.029
0.181620.0ss
0.2037
0.1231 126
0,6576_0.134
GPT2-XI.
Plugin (Ours)
0.0600_a.017
20.
0.0722
0.1725gn.orz
0.1995g0.01s
0.1195
0.6375_a.120
LLaMA-3.1-8B
Zeroshot
0,0120
0,2470
0.0318
0.1493
0.1526
0.0424
0,5285
I.LMA--.1-8B
LoRA (1=8)
0.0721 =0.02
0.2697 20.081
0.075640.028
0.1821 g0.020
0.2023
0. 0.1302
0.61 0.613720.172
LLaMA-3.1-8B
Plugin (Ours)
0.0611_a.00
0.271410.029
0.074210:020
0.1759
0,1990
0.1293 10.132
0.6361 16.11_10.34
ra matrices on the query and value projections of GPT2-XL and LLaMA-3.1-8B. and fine-tune the base models using the same task-specific data.
The performance results shown in Table 7. Consider GPT2-XL as reference example, Plugin adds 1-layer autoregressive Transformer with 30.72M parameters, while LoRA (r=8) introduces only 2.46M trainable parameters. However, Plugin requires no modification of the base model and can be deployed post hoc. Despite the access advantage of LoRA, the performance gap is minimal. As for computational efficiency, Plugin requires 196.2B FLOPs (up to 64 decoding steps), while LoRA uses 188.8B FLOPP-a difference of less than 5%. The gap narrows or inverts depending on model configuration. These results suggest that Plugin offers a competitive adaptation solution even under white-box conditions, while maintaining broader applicability in black-box settings.
C.5. Further Quantitative Analysis and Ablation
Following Section 7.3, we present the same ablation analysis using GPT2-M on the remaining three datasets. As shown in Figure the trends mirror those in Figure 2: the Plugin model consistently improves performance the base model becomes stronger with additional fine-tuning, underscoring the robustness and versatility of our approach. Similarly, Figure 6 confirms the pattern observed in Figure 3: single-layer reweighting model yields optimal performance, while deeper configurations tend to overfit and degrade quality. Across all datasets, initializing the reweighting model with a pretrained GPT2-Small consistently boosts effectiveness.
C.6. Influence of the architecture of the reweighting model in Plugin
We vary the choice of the reweighting model architecture. We find that causal transformer layer identical to those used in the base model performs best, as it can leverage the base model's 's logits and aggregate contextual information from prior tokens to better adapt the base model to the new data distribution. This conclusion is reinforced by Figure 7, where the transformer architecture consistently outperforms both the MLP (two layer with ReLU activation) and linear layers across
25

---

Logits are All We Need to Adapt Closed Models
Rouge-L
0.30
Base
(a) Web NLG
0.25
Plugin
9p0re
0.30
0.20
0.10
0.06
0.10
0.00
0.00
GPT2
GPT2 (seroshot)
(perowhog)
[1FT]
(AFT]
(9PT)
(IFT)
(2FT)
(FTT)
BLEU
Rouge-L
(6) Commonnen
0.25
Score
0.20
0.30
0.15
0.20
0.10
0.00
GPT2-M
GPT2-M
GPT2-M
GPT2-M
GPT2-M (peroshot)
GPT2-M Rouge-L
GPT2-M (2FT)
GPTZ-M (SFT)
(persehog)
(9PT)
0.30
(c Adidas
Sc9te
0.20
0.10
0.05
0.00
0.00
(eroohog
(1FT)
(AFT]
(IFT)
jeeroshot)
(IFT)
(IFT)
(SFT)
Figure Performance of applying single-layes reweighting model across increasingly fine-tuned GPT2-M models on the three datasets. Results demonstrate consistent improvements introduced by our method regardless of the strength of the base model.
BLEU
Rouge-L
(a) Web NLG
0.10
Score
0.30
0.10
0.06
0.10
0.00
0.00
96
6INM
OATS
1PM
OA12
Rouge-L
0.20
0.50
(b) Common
0.15
Gooe
0.10
0.00
0.00
CHOPP
Soud
4
12/gml
BLEU
Rouge-L
0.08
0.25
(c) Adidas
0.06
Score
0.15
0.04
0.02
0.05
0.00
Mo LWype
GPTS
GAT3
Figure 6. Performance of GPT2-M with varying reweighting model complexities the three datasets, measured by BLEU and Rouge-L. Results demonstrate a single reweighting layer achieves significant improvements, while increasing the number of layers beyond this leads performance degradation, likely due to overfitting. Using pretrained GPT2-Small the reweighting model largely boosts the performance, highlighting the benefits of leveraging pretrained models.
26

---

Logits are All We Need to Adapt Closed Models
all metries, indicated by higher means and narrower standard deviation bands. These results highlight the importance of leveraging the architectural capacity of transformers effectively adapt the logits of the base black-box model.
0.7
Linear
0.6
Transtormer
Mean Std
0.4
0.3
0.2
0.1
0.0
BLEU
Rouge-1
Rouge-2
Rouge-L
METEOR
CIDEr
NIST
Figure 7. Performance comparison of the weighting model architecture in Plugin. The transformer layer achieves the best performance with consistently higher means and narrower standard deviations. Shaded bands represent the standard deviation around the mcan.
C.7. Details for Adidas Qualitative Studies
Human Evaluation. We conduct human evaluation on 100 test passages from the Adidas product dataset, comparing outputs generated with and without applying the reweighting model, using LLaMA-3.1-88 human evaluators are presented with ground-truth Adidas product description and two randomly ordered descriptions: one generated with the reweighting layer and one without (i.e., we use the base model with ICL-3 as a much stronger baseline due to the low quality of the zero-shot). Evaluators are prompted to select the prediction closest to the ground truth. Results show that the output generated with the reweighting model is preferred on an average of 80.7 out of all 100 cases. The output descriptions from the base model without the reweighting are generally short and general. This demonstrates that our approach effectively adapts a closed model to the unique style of the given dataset.
the base model. Three
In this section, we display some details for the qualitative analysis on the Adidas product description dataset.
Details of Extracting Adidas Style Words. We discuss the details extracting the most frequent 50 words in the Adidas product description dataset as the "Adidas style" words. We argue that there does not exist gold-standard way to define the "style" words for a dataset. We extract these style words through minimal preprocessing pipeline: converting text to lowercase, removing special characters and numbers, and filtering out common English stopwords. We deliberately preserve the original word forms without lemmatization or stemming to maintain distinct style markers (e.g., keeping "comfortable" distinct from "comfort", "rumning" distinct from "run"). After tokenization using NLTK's word tokenizer, we count word frequencies across all product descriptions and select the top 50 most frequent words. This approach captures the exact vocabulary used in Adidas" product descriptions, including specific product features.
A statistics of the frequency of these top-50 words is shown in Figure 8.
Case Studies for Human Side-by-side Evaluation. Below we provide more case studies on the Adidas product description datasets. We display the given product attributes, base model's zero-shot outputs, base model with ICL-3's outputs, Plugin model's 's outputs, and the ground- truth references for each case.
Case
Given Product Attributes:
name [Cushioned Angle Stripe Low-Cut Socks 6 Pairs]
category [Accessories]
price [13]
color [Multicolor]
Base Model (Zero-shot): Cushioned Angle Stripe Low-Cut Socks Pairs, Multicolor.
Base Model (ICL-3): These adidas socks made with cushioned sole for extra comfort and support. The low-cut
27

---

Logits are All We Need to Adapt Closed Models
Word Frequency of the Top-50 Words in Adidas Dataset
835
made
642
shoes
2113
upper
318
Somfort
22/2
part
2251
boton
2114
materss 5
lokk
202
highperformancc
154
sares
173
wasla
Adidas Style Words
comfortable
and
167
day
157
153
sont
148
ambition
144
Toel
181
cushioning
130
fost
every
114
design
105
ightweight
2x
03
strpees
sustainable
feeling
go
good
84
aerorsacy
usod
Frequency
Figure 8. Word Frequency of the Top-50 Words Adidas Dataset.
design is perfect everyday wear, whether you'te running errands or hitting the gym.
Cushioned Angle Stripe Low-Cut Socks 6 Pairs the perfect blend of style and comfort. The low-cut design gives you sleek, modern look while still providing plenty of support for your feet. Made with high-quality materials, these socks will keep your feet feeling comfortable and looking chic all day long.
Plugin Model:
Ground-truth Reference: With lots of cushioning in the heel and toe and a soft, stretchy feel, these kids' socks are big on comfort. Whether they'r re bouncing around between classes, practice or after-school activities, their little feet will stay dry thanks to moisture-wicking yarn and a mesh upper. adidas Badge of Sport and 3-Stripes give socks classic adidas feel.
Case 2
Given Product Attributes:
name [Badge of Sport Tee]
category [Clothing]
price [21]
color [Grey]
Base Model (Zero-shot): A white t-shirt with a badge of the sport you love on
Base Model (ICL-3): The adidas Badge of Sport tee is classic. It's made with soft cotton and features the iconic 3-Stripes on the shoulders. The crew neck has ribbed hem for comfortable fit.
There's nothing more classic than white tee with your favorite team's 's logo. This adidas Badge of Sport Tee is an everyday essential that can be worn to the gym or on game days. It's made with recycled materials, so you're making sustainable choice. The soft cotton jersey lightweight, breathable, and comfortable.
Plugin Model:
Ground-truth Reference: Put your brand pride full display. simple yet versatile look, this long sleeve t-shirt
28

---

Logits are All We Need to Adapt Closed Models
keeps you comfortable while the move lounging home. A contrast adidas Badge of Sport stands out the chest. This product is made with Primegreen, series of high-performance recycled materials.
Case 3
Given Product Attributes:
name [Stretch Cotton Trunk Briefs Pairs]
category [Clothing]
price [21]
color [Grey]
Base Model (Zero-shot): Stretch Cotton Trunk Briefs Pairs, Grey.
Base Model (ICI-3)): These adidas men's trunks are made from soft cotton blend that feels great against the skin. The stretchy fabric moves with you, while the flatlock seams prevent chafing and irritation. A drawcord waistband allows for an adjustable fit.
Stretch Cotton Trunk Briefs Pairs. The stretch cotton fabric comfortable to These brief's made with classic design in mind, you feel confident and comfortable. They have been designed with a low waist, which gives them a flattering fit and makes them easy to get on.
Plugin Model:
Ground-truth Reference: Just look at these adidas boxer briefs. They'r re made with soft, quick-dry fabric that keeps you feeling fresh and a stay-put design that keeps you covered from your first rep to your last. Get comfortable and give your all. Our cotton products support sustainable cotton farming. This part our ambition to end plastic waste.
Case
Given Product Attributes:
name [adidas x Zoe Saldana Long Sleeve Tee]
category [Clothing]
price [28]
color [Black]
Base Model (Zero-shot): This is long sleeve with the adidas logo the front and Zoe Saldana's name the back. It has crew neck, short sleeves, and straight hem. The fabric is made of 100% cotton and is machine washable.
Base Model (ICL-3): This adidas x Zoe Saldana long sleeve tee is made with 100% recycled polyester. The soft, lightweight fabric has a smooth finish and features a crew neck and short sleeves. It's finished with an adidas logo on the chest.
Plugin Model: This not your average training t-shirt. modern, slim fit and the softest fabric make it one of our most comfortable tops date. The lightweight feel makes this perfect for all-day wear. Pair with pair of leggings joggers for that ultimate athleisure look you re always striving for ambition and comfort.
Ground-truth Reference: Your growth, your goals and the challenges you face along the way are all part of your journey. This adidas tee was created in collaboration with Zoe Saldana to inspire you to move with confidence, no matter where you are on your path. Smooth material lays softly on your body so you can feel comfortable in your own skin. A solid color offers styling versatility. This product made with recycled content as part of our ambition to end plastic waste.
29


Reviews:
[{"review_id": "Xg70iSnCeM", "reviewer": "Reviewer_keTv", "summary": "The key idea of the paper is to treat next-token prediction as a label noise correction problem, where discrepancies between the LLM’s broad training distribution and task-specific data are modeled through a transition matrix that reweights token probabilities during inference. The proposed Plugin model consists of a small reweighting network trained on limited task-specific data, which, when combined with the black-box LLM’s logits, effectively steers text generation towards the desired distribution. The authors provide theoretical guarantees showing that this probability reweighting approach converges to the target distribution with sufficient task data. Extensive experiments on multiple text generation benchmarks (E2E NLG, WebNLG, CommonGen, Adidas product descriptions) demonstrate that the Plugin model outperforms in-context learning and other adaptation methods, achieving better alignment with domain-specific content while requiring minimal computational resources compared to full fine-tuning. The results suggest that access to token logits could enable more powerful model customization, advocating for broader API-level exposure of logits in commercial LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and insightful review. \n\n**Regarding Comparison of Plugin with Parameter Efficient Fine-tuning (PEFT) methods like LoRA, Adapters:**\n>\"... The Plugin model is positioned as an alternative to fine-tuning, but it is not directly compared to parameter-efficient fine-tuning methods such as LoRA, Adapters, or QLoRA...\"\n\"... How many flops would the method save compared to vanilla LoRA?\"\n\nWe rely on only output logits of the black-box model and do not have access to its internal weights or architecture. Consequently, any form of fine-tuning—including parameter-efficient approaches like LoRA or Adapters—cannot be applied, which we clarify in the Introduction (lines 21–33, right column) and the Related Work (lines 261–273, left column). If we did have access to the model weights, parameter-efficient fine-tuning methods would indeed be the natural choice, as they use more information than just logits and are expected to yield better performance. Thus, we emphasize that the Plugin model is not an alternative to fine-tuning, but rather an approach that uniquely stands for adapting black-box LLMs which only provide logit access.\n\nNevertheless, to address the reviewer’s point, we conducted a comparison on the E2E NLG dataset by adding rank-$𝑟=8$ LoRA matrices in the $Q$ and $V$ attention layers of GPT2-XL. The results, which will be included in our final version, show that LoRA only slightly outperforms our Plugin in terms of task metrics. LoRA adds 2.46M parameters (rank-8, Q/V only), while Plugin adds 30.72M (one full layer). During inference (up to 64 tokens), LoRA requires 188.8B FLOPs while Plugin needs 196.2B FLOPs - a negligible difference in computational cost. Notably, the parameter and efficiency gap between LoRA and Plugin narrows when increasing LoRA's rank (r) while reducing Plugin's hidden dimensions - demonstrating how both approaches can be adaptively tuned to meet specific resource constraints while maintaining competitive performance. We would like to highlight that the fundamental distinction remains - LoRA requires full model access to modify internal layers, while Plugin enables post-hoc deployment without retraining.\n\n|E2E NLG|BLEU|Rouge-1|Rouge-2|Rouge-L|METEOR|CIDEr|NIST|\n|--|--|--|--|--|--|--|--|\n|Zeroshot       |0.0562|0.4013 |0.1636 |0.2862 |0.3697|0.0187|0.5338|\n|Plugin         |0.2470|0.5536 |0.3084 |0.4213 |0.5057|0.5455|1.2736|\n|PEFT (LoRA r=8)|0.2517|0.5712 |0.3079 |0.4317 |0.5162|0.5225|1.2172|\n\n**Regarding Missing References**\n> On the Duality between Gradient Transformations and Adapters. Lucas Torroba-Hennigen et al. 2025\nTuning Language Models by Proxy. Liu et al. 2024\nA Study on the Calibration of In-context Learning. Zhang et al. 2023\n\nThank you for pointing out these relevant works. We will add them to our final version. In particular, we note that Liu et al. (2024) describes the same method introduced in Liu et al. (2021), which we already cite in line 252. Below, we clarify how our approach differs from each reference:\n\nTorroba-Hennigen et al. (2025): They examine the equivalence between gradient transformations and adapters for efficient model adaptation, relying on full access to model weights and gradients. In contrast, our method requires no access to model internals; we adapt black-box LLMs solely by reweighting token-level logits.\n\nLiu et al. (2024): As noted, we already cite their earlier work (Liu et al. 2021), which presents the same core idea of combining logits. Our WeightedComb baseline (line 252, right column) is directly inspired by their approach.\n\nZhang et al. (2023): We will add this reference to our discussion on calibration. Their study focuses on aligning model confidence with predictive accuracy by adjusting confidence scores as one increases shots in the few-shot learning setting. Unlike them, we explicitly modify the token predictions themselves, rather than just calibrating confidence.\n\n**Regarding Domain Shifts, Extreme Domain Shifts, and Adversarial Domain Shifts**\n\nWe assume our black-box LLM already encodes extensive world knowledge. In this sense, any adaptation to a domain-specific dataset amounts to handling a distribution shift, as demonstrated in Section 7.1 with WEB NLG, E2E NLG, and CommonGen. Among these, the Adidas dataset represents a more extreme domain shift, given its specialized style of product descriptions. Additionally, the experiments in Section 7.3 can be seen as adversarial, since the Plugin is applied atop a model with known biases—for instance, the tendency to focus on infrastructure-related concepts in WEB NLG and male-related concepts in CommonGen. Although we did not explicitly label these settings “extreme” or “adversarial,” they do indeed meet those criteria to some extent, and we will clarify that in the final version.\n\nWe would also welcome any further examples or clarifications on what the reviewer would consider to be more extreme or adversarial distribution shifts in this context.", "claims_and_evidence": "The paper makes several claims regarding the effectiveness of the Plugin model for adapting black-box LLMs, and most of these claims are supported by theoretical justifications, empirical experiments, and comparative evaluations. The core claim—that token-level probability reweighting using logits alone is sufficient for effective task adaptation—is backed by a formal label noise correction framework, where the authors derive theoretical guarantees showing that the Plugin model can align token distributions with task-specific data under mild assumptions. Additionally, extensive experimental results on four datasets (E2E NLG, WebNLG, CommonGen, Adidas product descriptions) demonstrate that the Plugin model outperforms baselines such as zero-shot inference, in-context learning (ICL), and naive probability combination methods across multiple evaluation metrics (BLEU, ROUGE, METEOR, CIDEr, NIST). The ablation studies further support the claims by showing that model quality, reweighting complexity, and domain adaptation capabilities contribute to improved performance. However, some claims could benefit from stronger empirical validation. For instance, while the paper asserts that the Plugin model is effective under distribution shifts, the experiments focus on relatively constrained dataset modifications (e.g., filtering training data by entity types), and it remains unclear how well the approach generalizes to more extreme domain shifts or adversarial settings. Additionally, while the authors argue that their approach is computationally efficient, they do not provide a direct comparison of training or inference costs against alternative adaptation methods like parameter-efficient fine-tuning (LoRA, adapters), leaving room for further evidence on the trade-offs between performance gains and computational overhead. Nonetheless, the overall evidence presented is strong and well-aligned with the claims, making the Plugin model a compelling approach for logit-based adaptation of closed-source LLMs.", "methods_and_evaluation_criteria": "The proposed Plugin model and its evaluation criteria are well-aligned with the problem of adapting black-box LLMs without modifying model weights. The token-level probability reweighting framework is a reasonable method given the constraints of closed-source LLMs, and the formulation as a label noise correction problem provides a solid theoretical foundation. The authors evaluate the approach using four diverse text generation datasets—E2E NLG, WebNLG, CommonGen, and Adidas product descriptions—each representing different aspects of controlled text generation and domain adaptation.", "theoretical_claims": "I didn't check the orrectness of any proofs for theoretical claims.", "experimental_designs_or_analyses": "sound and well-structured, but there are a few areas where further validation or additional analysis could strengthen the claims. The authors conduct experiments across four datasets (E2E NLG, WebNLG, CommonGen, Adidas product descriptions) and compare their Plugin model to several strong baselines including zero-shot inference, in-context learning (ICL), and a weighted combination of model predictions. The use of seven standard NLG metrics (BLEU, ROUGE, METEOR, CIDEr, NIST) ensures a comprehensive evaluation of output quality. They also include a sound human evaluation. The Plugin model is benchmarked against ICL and naive probability reweighting, but not against LoRA, Adapters, or QLoRA, which might be feasible alternatives in cases where some access to model weights is possible.", "relation_to_broader_scientific_literature": "The paper situates itself within the broader literature on adapting large language models (LLMs) without access to model weights, drawing from areas such as prompt engineering, in-context learning (ICL), parameter-efficient fine-tuning (PEFT), label noise correction, and black-box model adaptation. The key contribution—reweighting token probabilities using logits as an alternative to full fine-tuning—builds upon prior work in label noise correction (Patrini et al., 2017) and autoregressive modeling, adapting these ideas to the language model decoding process. The formulation of next-token prediction as a noisy supervised classification problem is a novel connection that extends prior work on calibrating LLM outputs (Huang et al., 2024; Kapoor et al., 2024).", "essential_references_not_discussed": "Important references are missing: \nOn the Duality between Gradient Transformations and Adapters. Lucas Torroba-Hennigen, Hunter Lang, Han Guo, Yoon Kim.\n\nTuning Language Models by Proxy. Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, Noah A. Smith.\n\nA Study on the Calibration of In-context Learning.  Hanlin Zhang, Yi-Fan Zhang, Yaodong Yu, Dhruv Madeka, Dean Foster, Eric Xing, Hima Lakkaraju, Sham Kakade", "other_strengths_and_weaknesses": "The paper title is a bit bold and the methodology presented in the paper is conceptually interesting but complex, as it involves multiple layers of statistical modeling and adaptation that may not be immediately intuitive. The core idea—reweighting token probabilities using a transition model learned from task-specific data—relies on a label noise correction framework, which is commonly used in supervised classification problems but less so in language model decoding.", "other_comments_or_suggestions": "The Plugin model is positioned as an alternative to fine-tuning, but it is not directly compared to parameter-efficient fine-tuning methods such as LoRA, Adapters, or QLoRA. Including a discussion on when logit reweighting is preferable to PEFT would improve clarity.", "questions_for_authors": "How many flops would the method save compared to vanilla LoRA?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "STR2jHyVWy", "reviewer": "Reviewer_kses", "summary": "The paper tackles the issue of having to rely on prompt engineering while adapting closed-source LLMs. The proposed work formulates the problem as a supervised learning, where few task specific dataset is used to train the model. The closed source model is assumed to learn noisy labels of specific application, and by accessing logits of the tokens, the proposed loss function corrects the noisy label to adapt to a task. Extensive experiments are conducted on models, datasets, and with multiple evaluation criteria.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer’s positive remarks and valuable insights.\n\n**Regarding Inclusion of Closed Source Models**\nWe acknowledge that including more closed-source models could further strengthen the generality of our findings. However, most proprietary models currently do not expose their output logits. Hence, we could not experiment with them. Our approach specifically hinges on logit-level access—without revealing internal weights or architecture—which we believe is a comparatively straightforward adjustment for closed-source providers to implement, especially when compared to the complexities of releasing the full model.\n\nBy highlighting this limitation, we aim to encourage closed-source developers to consider offering logit-level access in the future, enabling more flexible and efficient adaptation methods for end-users.\n\n**Implications of making the transition matrix diagonal**\n\n**Benefits**\n\n1. *Reduced Complexity:* Learning a diagonal matrix involves only $|V|$ parameters, compared to $|V|\\times |V|$ parameters in a full matrix. This makes training computationally more tractable.\n\n2. *Straightforward Integration:* Because the transition matrix is treated like a single vector, standard autoregressive models (e.g., GPT-2, LLaMA) can be used directly, scaling easily with the dataset size.\n\n3. *Symmetric (Class-Independent) Noise:* A diagonal assumption naturally corresponds to label flips occurring with equal probability among all incorrect classes. This aligns neatly with widely studied symmetrical label-noise scenarios.\n\n4. *Stability in Estimation:* Each diagonal entry is an autoregressive parameter independently learnt without depending on other classes (vocabulary), reducing the risk of overfitting and simplifying parameter estimation compared to a fully dense matrix.\n\n**Limitation**\n\nAlthough adopting a diagonal transition matrix reduces complexity and simplifies parameter estimation, it also prevents the model from capturing any off-diagonal confusions—that is, instances where certain tokens are more likely to be misclassified as particular other tokens. Real-world data may involve non-symmetric noise patterns, domain-specific synonyms, and context-dependent misclassifications. A purely diagonal matrix may therefore oversimplify these nuances, leading to diminished expressive power and potentially missing important relationships within the noise structure.\n\nOverall, the diagonal constraint provides a practical, stable, and computationally efficient solution for autoregressive reweighing learning—one that is well-aligned with symmetrical label-noise setups—while sacrificing some flexibility in capturing sophisticated non-symmetric noise or domain-specific synonym patterns. \n\nWe plan to investigate non-diagonal structures to fully model these nuanced noise patterns and further improve adaptation in the future work.", "claims_and_evidence": "No issues found", "methods_and_evaluation_criteria": "No issues found", "theoretical_claims": "No issues found", "experimental_designs_or_analyses": "No issues found", "relation_to_broader_scientific_literature": "It is related to the adaptation of black-box LLM.", "essential_references_not_discussed": "No issues found", "other_strengths_and_weaknesses": "Strengths\n\n- The problem is well-motivated.\n\n- The approach to solving the adaption problem is novel. With this method, it can be used as a plugin to adapt a closed-source model.\n\n- Experiments are extensive to confirm the verification of the proposed method.\n\n- Theoretical analysis of the proposed method makes it guaranteed.\n\nWeakness\n\nInclusion of some other closed-source models would make the experiment more comprehensive.", "other_comments_or_suggestions": "Please refer to previous sections.", "questions_for_authors": "What are the implications of making the transition matrix diagonal?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8h0phrHQTu", "reviewer": "Reviewer_U4zS", "summary": "The paper proposes logit reweighting to adapt closed-source LLMs for task-specific generation without accessing model weights. By learning an autoregressive transition matrix from task data, it adjusts token probabilities during inference to align outputs with target domains. Experiments show improved style/keyword compliance over zero-shot and instruction tuning, advocating logit accessibility as a practical adaptation pathway. The method bridges theoretical label shift correction with efficient LLM customization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s encouraging words and thoughtful feedback.\n\n**Regarding human evaluation and limited generalization tests**\nWe already conducted a human evaluation (line 366, details in Appendix C.7) where three evaluators compared Plugin and ICL-3 on 100 Adidas samples, with Plugin preferred in 81% of cases—directly supporting output quality.\n\nWhile the Adidas dataset reflects a specialized domain shift, our study extends beyond a single brand. As shown in Section 7.1, we also evaluate on WEB NLG, E2E NLG, and CommonGen—each representing its own distribution shift relative to the black-box model’s pretraining data. In Section 7.3, we further explore adversarial shifts by testing Plugin on models with known biases (e.g., infrastructure in WEB NLG, male-related concepts in CommonGen), demonstrating Plugin’s broad applicability across diverse and challenging settings.\n\nFor most datasets, we follow standard practices from PEFT literature (Hu et al., 2021; 2023a) that rely on automated metrics comparing outputs to well-formed references to assess overall quality. This combination of human judgments (for Adidas) and standard metrics (for broader benchmarks and Adidas) provides both qualitative and quantitative evidence of the Plugin’s generalization capabilities.\n\n**Token Pruning**\nWe do not perform token pruning; instead, Plugin continuously reweights token probabilities at each decoding step without removing any tokens. This soft upweighing preserves vocabulary coverage and improves domain adaptation (see case study in line 408). For further clarification, we compare the total occurrences of the top-50 Adidas domain words in Plugin’s predictions versus the base model in the same case study. Across all samples, Plugin’s outputs include 25.6% occurrences of these words, while the baseline contains only 13.8%. We will add this in the paper.\n\n**Regarding FUDGE, Qiu et al.**\nWe do cite FUDGE in line 249 (right column). While FUDGE uses attribute-specific discriminators to control generation (e.g., formality), our method enables free-form domain adaptation via a single auxiliary model. We considered FUDGE as a baseline, but it requires one discriminator per predefined attribute—unsuitable for broad or evolving domain shifts, which are hard to define upfront.\n\nTempNet in Qiu et al. learns a single temperature per input and uniformly scales logits during generation. In contrast, Plugin reweights logits at each timestep, enabling finer, context-sensitive adjustments. Additionally, Qiu et al.'s use of DRO involves an inner maximization loop, making it more computationally intensive than our efficient empirical risk minimization (ERM). \n\nNonetheless, we include a comparison on the E2E NLG dataset, adapting TempNet to use ERM (instead of DRO) with GPT2-XL. As it applies global scaling per prompt, it underperforms in tasks requiring localized adjustments. These results will be added to the final paper.\n\n|Method|BLEU|Rouge-1|Rouge-2|Rouge-L|METEOR|CIDEr|NIST|\n|--|--|--|--|--|--|--|--|\n|TempNet |0.1325|0.4642|0.2516|0.3021 |0.4126|0.3627|0.8027|\n|**Plugin**|**0.2470** |**0.5536** |**0.3084** |**0.4213** |**0.5057**|**0.5455**|**1.2736**|\n\n**Regarding Theoretical Claims and Convergence Rate**\nTheorem 5.1 holds under mild, standard assumptions (5.1, 5.2, B.1) commonly used in convergence analyses (Frostig et al., 2015; Chaudhuri et al., 2015; Mukherjee et al., 2022). The noisy estimation of the transition matrix $T\\_t({\\theta}\\_{\\star};x_i,x_j,\\mathcal{F}^{t-1})$ can be understood in two ways: as direct estimation error in the matrix itself, \nor as error induced by the function $f\\_{I_t}({\\theta}\\_{\\star} ; x_i, x_j,\\mathcal{F}^{t-1})$ on which it depends (see Assumption 5.1). \n\nWe adopt the latter view, estimating $f_{I_t}(\\cdot)$ under a sequence of noisy autoregressive loss functions $\\ell_1(\\boldsymbol{\\theta}), \\ldots, \\ell_t(\\boldsymbol{\\theta}) : \\mathbb{R}^{|V|} \\rightarrow \\mathbb{R}$ (see Assumption 5.2). Under Assumption B.1—bounded gradients and Hessians of $f_{I_t}$—we show that the expected estimation error can be reduced and prove upper and lower bounds in terms of a problem-dependent quantity $\\sigma_t^2$, establishing a convergence rate of $\\Omega(\\sigma_t^2 / t)$. \n\nWe hope this clarifies. Our novelty lies in combining techniques from Frostig et al. (2015), Chaudhuri et al. (2015), Mukherjee et al. (2022), and Patrini et al. (2017), and presenting the **first finite-time convergence analysis** for transition matrix estimation in this autoregressive noisy loss setting. \n\nWe acknowledge that real-world settings may add noise in mapping $f_{I_t}$ to the transition matrix. While this could generalize our framework, incorporating it into our finite-time guarantee is left for future work. Regarding sensitivity analysis, our convergence rate depends on the variance-like term $\\sigma_t^2$; higher variance leads to slower convergence, naturally capturing sensitivity to estimation noise.", "claims_and_evidence": "The claims are partially supported: empirical results on style/keyword alignment validate performance gains, but theoretical guarantees (e.g., distribution alignment) rely on idealized assumptions (e.g., perfect transition matrix estimation) not fully verified in real-world noisy settings. Scalability claims for large vocabularies lack rigorous analysis of trade-offs with token pruning.", "methods_and_evaluation_criteria": "The methods (logit reweighting via transition matrices) align with the goal of adapting closed LLMs using limited access (logits only), and domain-specific metrics (style/keyword accuracy) suit tasks like product descriptions. However, human evaluation is notably absent for assessing output quality, and generalization tests are limited to narrow domains (e.g., one brand), leaving broader applicability under-explored.", "theoretical_claims": "The theoretical claim of distribution alignment (via logit reweighting under ideal transition matrices) is logically consistent given the assumptions, but the proof sketch (as described in the summary) assumes noiseless task-specific data and perfect estimation of the transition matrix—conditions unlikely in practice. No convergence rates or sensitivity analysis for estimation errors are provided, and empirical results do not explicitly validate the theoretical bound (e.g., measuring distribution divergence post-adaptation).", "experimental_designs_or_analyses": "The experimental design has validity in using domain-specific metrics (e.g., keyword accuracy for product descriptions), but human evaluation is missing, leaving output fluency/coherence unverified. Comparisons to baselines (zero-shot, instruction tuning) are reasonable, but scalability tests lack depth—token pruning’s impact on rare tokens is unstudied. Domain generalization is under-tested (e.g., single-brand data), raising concerns about broader applicability.", "relation_to_broader_scientific_literature": "The paper connects to label noise correction literature (e.g., noise transition matrices in supervised learning) by reframing LLM adaptation as correcting “noisy” general-purpose token distributions. It extends these ideas to autoregressive generation, differing from prior LLM adaptation (e.g., prompt tuning, soft prompts) by relying solely on logits, aligning with resource-efficient methods like light-weight finetuning but avoiding weight access. Theoretically, it bridges domain adaptation (e.g., label shift theory) with LLM customization, advancing closed-model adaptation paradigms.", "essential_references_not_discussed": "The paper does not cite controlled text generation methods (e.g., FUDGE Yang & Klein, 2021), which also modify logits using auxiliary models for task alignment. Additionally, Qiu et al. also investigated how to dynamically adjust logits by learning temperature parameters to adapt to different tasks.\n\n[1] Yang & Klein, FUDGE: Controlled Text Generation With Future Discriminators, In NAACL, 2021.\n[2] Qiu et al., To Cool or not to Cool? Temperature Network Meets Large Foundation Models via DRO. In ICML, 2024.", "other_strengths_and_weaknesses": "Strengths: The paper’s originality lies in creatively bridging label noise correction with autoregressive LLM adaptation, a novel conceptual link that unlocks practical utility for closed models. Its emphasis on logit accessibility addresses a critical industry need (adapting proprietary models without weight access), offering significant real-world relevance. The method’s lightweight design (task-specific matrix learning) is a pragmatic strength.\n\nWeaknesses: While innovative, the framing underplays overlaps with logit manipulation in controlled generation (e.g., FUDGE) and distillation. Broader claims about domain generalization lack empirical rigor (limited to narrow tasks/brands), and theoretical assumptions (perfect matrix estimation) are underexplored in practical noisy settings.", "other_comments_or_suggestions": "Suggestions:\n1. Human evaluation: Include user studies to validate output quality beyond automated metrics.\n2. Error analysis: Quantify how token pruning affects rare tokens or domain-specific terms.\n3. Comparison to logit-based methods: Explicitly contrast with FUDGE or distillation to clarify novelty.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwXo8i5QVa", "reviewer": "Reviewer_YK5M", "summary": "This paper studies the problem of adapting a black-box LLM to a downstream task, assuming access to the logits of output tokens. The authors propose a token-level probability reweighting algorithm that modifies token logits during inference. The core idea is to frame the adaptation problem as label noise correction in supervised classification and leverage an autoregressive probability reweighting model to estimate logits for the downstream task. Theoretical justifications are provided, and empirical studies on benchmark datasets show that the proposed approach outperforms several existing adaptation techniques.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, which has helped us strengthen our paper.\n\n**Regarding Logit access assumption**\nThe central goal of this paper is to encourage closed-source LLM providers to offer logit-level access as a practical middle ground when releasing full model weights is not feasible due to IP or privacy concerns. Our theoretical and empirical results show that even with this limited access, effective domain and task adaptation is possible. Unlike prior work assuming either full white-box access or fully opaque models, we demonstrate that logit access enables fine-grained control without exposing proprietary internals. With this work, we aim to motivate commercial providers to adopt this feasible and impactful compromise, filling a key gap in current adaptation methods.\n\n**Computational Cost vs. Simpler Methods**\nWhile prompt-based methods like prompt tuning or ICL may seem simpler, they often require extensive trial-and-error and show high variance, as reflected in our results. In contrast, our reweighting model offers consistent, theoretically grounded gains with lower overhead than exhaustive prompt engineering. It is also analogous to established methods like LoRA or Adapters in white-box settings, requiring a manageable training overhead. Ultimately, the stability, reliability, and performance improvements of our approach outweigh the modest compute required, especially when compared to the variability and manual tuning burden inherent in prompt-based techniques.\n\n**Baseline Comparisons: ICL Variants and API Access-based Methods**\nBased on the reviewer’s suggestion, we extended ICL baselines to include 5, 8, and 10 examples, and implemented Diao et al. ([2], Black-box Prompt Learning, 2023) with the recommended 75 API calls as mentioned in the paper. We did not include Sun et al. ([1], Black-Box Tuning, 2022), as Diao et al. [2] already demonstrated superior performance.\n\nImportantly, if logit access is available, Plugin can be layered on top of any prompt-based method using the best-found prompt. For instance, our Zeroshot prompt (noted in line 364, left column) is reused across all methods. We also apply Plugin on top of the best ICL variants (ICL-8/10) and Diao et al. [2]. The table below presents results across all datasets using GPT2-XL as the base model.\n\n|E2E NLG|BLEU|Rouge-1|Rouge-2|Rouge-L|METEOR|CIDEr|NIST|\n|--|--|--|--|--|--|--|--|\n|ICL-5|0.1226|0.4319 |0.2194 |0.3095 |0.4172|0.3162|0.7281|\n|ICL-8|0.1537|0.4432 |0.2439 |0.3180 |0.4268|0.3559|0.8253|\n|ICL-10|0.1582|0.4459 |0.2502 |0.3201 |0.4528|0.4125|0.9015|\n|Diao et al.|0.2287|0.5024 |0.2846 |0.3922 |0.4628|0.4216|0.8625|\n|**Plugin**|**0.2470**|**0.5536** |**0.3084** |**0.4213** |**0.5057**|**0.5455** |**1.2736**|\n|**ICL (best) + Plugin**|**0.3941**|**0.6713** |**0.4027** |**0.5379**|**0.5923**|**0.6172**|**1.5472**|\n|**Diao et al. + Plugin**|**0.4527**|**0.7126** |**0.5126** |**0.6027**|**0.6214**|**0.7002**|**2.0817**|\n\n|WEB NLG|BLEU|Rouge-1|Rouge-2|Rouge-L|METEOR|CIDEr|NIST|\n|--|--|--|--|--|--|--|--|\n|ICL-5|0.0826|0.3625|0.1725|0.2517|0.3261|0.1826|0.2614|\n|ICL-8|0.0943|0.3826|0.1926|0.2825|0.3425|0.2016|0.2611|\n|ICL-10|0.0813|0.3528|0.1718|0.2542|0.3321|0.1906|0.2425|\n|Diao et al.|0.1024|0.4016|0.2243|0.3017 |0.3527|0.4321|0.2631|\n|**Plugin**|**0.1673**|**0.4616**|**0.2527**|**0.3757** |**0.3895**|**0.8987**|**0.2646**|\n|**ICL (best) + Plugin**|**0.1926**|**0.5026**|**0.2735** |**0.3927**|**0.3872**|**0.9123**|**0.4267**|\n|**Diao et al. + Plugin**|**0.2137**|**0.6026** |**0.3021**|**0.5928**|**0.5766**|**1.0826**|**0.6142**|\n\n|Adidas|BLEU|Rouge-1|Rouge-2|Rouge-L|METEOR|CIDEr|NIST|\n|--|--|--|--|--|--|--|--|\n|ICL-5|0.0345|0.2654|0.0393|0.1601|0.1863|0.0338|0.6856|\n|ICL-8|0.0403|0.2527|0.0432|0.1628|0.1894|0.0615|0.6125|\n|ICL-10|0.0382|0.2537|0.0325|0.1528|0.1725|0.0452|0.5926|\n|Diao et al.|0.0417|0.2615|0.0671|0.1710|0.1826|0.0861|0.6034|\n|**Plugin**|**0.0600**|**0.2710**|**0.0722**|**0.1725** |**0.1995**|**0.1195**|**0.6375**|\n|**ICL (best) + Plugin**|**0.0591**|**0.2761**|**0.0754** |**0.1736**|**0.2047**|**0.1273**|**0.6415**|\n|**Diao et al. + Plugin**|**0.0623**|**0.2792**|**0.0773** |**0.1759**|**0.2148**|**0.1325**|**0.7024**|\n\nWe observed similar results on Common Gen and will include them in the second response due to space constraints in the rebuttal. \n\nAs shown, Plugin outperforms ICL even with 10 examples and surpasses Diao et al. (2023). While ICL’s performance plateaus with increasing examples—despite higher inference cost and variance—Plugin consistently offers greater accuracy and stability. Moreover, combining Plugin with the best ICL or Diao et al. setups yields further gains, highlighting the value of logit-level access in enhancing prompt-based methods.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper consider the adaptation of black-box LLM to downstream task, which is quite broader in scientific literature", "essential_references_not_discussed": "yes", "other_strengths_and_weaknesses": "Framing the problem as transition matrix estimation in the label noise correction framework and handling the challenge of an extremely large label space using an autoregressive reweighting model is new and insightful to me. The theoretical analysis effectively establishes key properties of the proposed algorithm.\n\nHowever, 1. the assumption of logit access significantly weakens the overall novelty and practical contribution of the work. The approach relies entirely on closed-source LLMs exposing logits, which is not currently supported by most commercial APIs, making the proposed learning setting somewhat artificial.\n\n2. The method requires training a separate reweighting model, introducing higher computational costs compared to simpler adaptation techniques like prompt tuning or in-context learning.\n\n3. The baseline comparisons (ICL-1, ICL-3) are relatively weak, as increasing the number of in-context demonstrations could likely improve their performance, making the reported advantage of the proposed method less conclusive. Meanwhile, other baseline adaptation method are not compared [1,2]\n\n[1] Black-Box Tuning for Language-Model-as-a-Service. ICML 2022\n[2] Black-box Prompt Learning for Pre-trained Language Models. TMLR", "other_comments_or_suggestions": "n/a", "questions_for_authors": "see weakness 3 above. Is there any comparison with other SOTA methods [1,2] for black-box LLM adaptation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a timely and well-motivated contribution to the adaptation of closed-source large language models (LLMs). The key insight is that access to token-level logits, if made available, would enable powerful post-hoc alignment mechanisms beyond prompt tuning. Framing next-token prediction as a supervised classification problem under label noise, the authors introduce a theoretically grounded reweighting framework, Plugin, which can steer black-box LLMs toward task-specific behavior using only logits and a small amount of labeled data.

The reviewers appreciated the novelty and practicality of the proposal, particularly its clean theoretical framing, empirical rigor, and relevance in the context of real-world LLM deployment constraints. Given its conceptual clarity, soundness, and strong empirical results, this paper makes a valuable contribution to the growing literature on black-box model alignment and LLM accessibility.
Decision: Accept (poster)