CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models
Authors: Aitor Ormazabal
Mikel Artetxe
Eneko Agirre
Venue: EMNLP 2023 Main (2023)
Keywords: black-box
language models
large language models
adaptation
domain adaptation
TL;DR: We propose a method for adaptation of large black-box models to new domains and tasks through learned combination with fine-tuned small models

Abstract: Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters.
However, this is incompatible with a recent trend in the field, where the highest quality models are only available as black-boxes through inference APIs. Even when the model weights are available, the computational cost of fine-tuning large LMs can be prohibitive for most practitioners.
In this work, we present a lightweight method for adapting large LMs to new domains and tasks, assuming no access to their weights or intermediate activations. Our approach fine-tunes a small white-box LM and combines it with the large black-box LM at the probability level through a small network, learned on a small validation set. We validate our approach by adapting a large LM (OPT-30B) to several domains and a downstream task (machine translation), observing improved performance in all cases, of up to 9%, while using a domain expert 23x smaller.

CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models
Aitor Ormazabal'
Mikel Artetxe²
Eneko Agirre'
HiTT Center, University of the Basque Country (UPV/EHU) Reka AI ai tor. www.zaaaleeee..ee..o.o eus
mikel@reka.ai
e a.agirre@ehu.eur
Abstract
Methods for adapting language models (LMs) to new tasks and domains have traditionally as- sumed white-box access to the model, and work by modifying its parameters. However, this is incompatible with a recent trend in the field, where the highest quality models are only avail- able as black-boxes through inference APIs. Even when the model weights are available, the computational cost of fine-tuning large LMs can be prohibitive for most practitioners. In this work, we present lightweight method for adapting large LMs to new domains and tasks, assuming no access to their weights or inter- mediate activations. Our approach fine-tunes a small white-box LM and combines it with the large black-box box LM at the probability level through a small network, learned on a small val- idation set. We validate our approach by adapt- ing large LM (OPT-30B) to several domains and downstream task (machine translation), observing improved performance in all cases, up to 9%, while using a domain expert 23x smaller.
?
ASEacc sold arDaa
Large bradkbce LR
carb did
P
Figure I: Illustration of our approach. We leverage a large black-box LM and a small white-box LM, fine- tuned domain-specific corpus. We combine both models' outputs at the probability level, through a com- bination function learned on a small fitting set, requiring very little compute. The resulting model adapts the large black-box to the target domain, performing better than either original ones.
are available, the computational cost of fine-tuning large models can be prohibitive. Consequently, do- main adaptation methods that cannot leverage the power of black-box LLMs are likely fall behind.
In this work, we propose simple and lightweight approach to adapt black-box LMs to new domains, without requiring access weights or intermediate activations. Our method consists of two main steps: (1) training a small, white-box model on the desired target domain, and (2) learn- ing function that combines the probability distri- butions from the large black-box LM and the small domain expert LM, producing a new probability distribution. The combination function is a small neural network that is trained small validation datasct.
1 Introduction
Natural language processing (NLP) has witnessed remarkable progress in recent years thanks to the development of increasingly powerful LMs (Brown et al., 2020; Andrew and Gao, 2007; Chowdhery et al., 2022; Touvron et al., 2023). Since these mod- els are usually generalists, it is often of interest to adapt them to new domains, underrepresented or not found the original training data. Typically, domain adaptation techniques assume white-box access to the model parameters, for example by fine-tuning on a particular target domain (Gururan- gan et al., 2020).
We evaluate our method by adapting black-box model to three distinct domains and downstream task-machine translation (MT). In all cases, we observe that the combined model outperforms both the large black-box model and the small domain expert. This shows that it is possible to adapt black- box LMs to new domains, opening an exciting line
However, this approach has become increas- ingly infeasible given the ongoing paradigm shift in the field-state-n state-of-thhearr models like GPT-4 and PaLM-2 are only accessible as black-boxes through inference APIs and, even when the model weights

---

of research.
g g(H(y)), H(y2)) E R is predicted and then the per-token combination is done as in consiant-vector.
2 Proposed method
Our approach works in two steps: (I) we train small domain expert LM, and (2) we learn a func- tion that combines the outputs of the domain expert LM and a large black-box LM at the probability level.
6. Full-scalar. A single À is predicted from full input distributions, À - ((),, and then the output is a linear combination as in the constant combination: f(y1. = Λy1 +
(1 X)y2. The function y is parametrized by a small neural network.
More concretely, an LM defines a probability distribution over the possible continuations of any given text. That is, given a sequence of tokens x (z1, 22, Fnn E V*, where V is the model vocabulary, an LM parametrizes PI. (Umeart) x), the probability that Unezt is the continuation of x in text. We let Ps denote our small domain expert LM, and PL denote the large black box generalist LM. Our combination function f de- fines a new combined probability distribution Po: Po(ynerd x) f(Ps(-(x)) f: RI
7. Full-vector. Token-wise version of the pre- vious combination, where vector À y2) E IRI is predicted, and the per- token combination is done as in constant- vector.
On one end of the spectrum, the mean and constant-scalar combinations have very low pacity, having zero and one Iearnable parameters, respectively. On the other end, the full combina- tions can represent rich combination functions, tak- ing advantage of the information in the full output distributions. The entropy combinations are moti- vated by the fact that we expect output distribution entropies to be informative to the combination func- tion; intuitively, knowing how certain each model is should be helpful when deciding which model to give more weight to. Additionally, token-wise ver- sions of each method further increase the capacity of the combination function. This setup allows us to study how important combination function ca- pacity is for the performance of the adapted model, as well as how this relates to the amount of data used for learning the combination.
(t.((x)) yoest Here R I V is vector-valued func-
(x),
tion that receives full probability distributions, and outputs new probability distribution.
To train the domain expert LM, we fine-tume a pre-trained model on a small domain-specific dataset. For the combination function, we consider several alternatives of varying capacity:
1. Mean. The arithmetic mean of the two distri- butions: y2) (y1 + y2)/2.
2. Constant-scalar. A linear combination of the two input distributions, with a constant combination factor A: f(y1,y2) = Ay1 + - 1--)y22
3. Constant-vector. A token-wise version of the previous combination, where A E R|V| is constant vector, and the combination fac- tor varies per-token: ((y1,y2) x X o y1 + (1 A) o y2, where o is the Hadamard (el- ementwise) product. Note the proportional- ity instead of equality in the definition, re-normalization is required when combining distributions pok-noken.
These combination functions can be learned without any access the LMs' weights internal states, and require only forward pass through the small set used train the combination network. We refer to the process of training the small net- work that parametrizes the combination function as fitting the combination function. Once the combi- nation function is fit, the combined model outputs valid probability distributions over continuations. and can be used as regular LMM
4. Entropy-scalar. A scalar is predicted from the entropies of each distribution,
Experimental setup
X = g(H(y1), H(y2)), and the output is linear combination as in constant-scalar: f(y1.)2) Ay1 + (1 A)y2- The function 9 is parametrized by small neural network.
3.1 Models
We use OPT-30B and OPT-1.3B (Zhang al., 2022) as our large black-box and small white-box
5. Entropy-vector. An token-wise version of the previous combination, where a vector À =
LMs, respectively. Our choice of OPT is motivated by the following reasons:

---

1. Both the small and large models must share the tokenizer in our current formulation.' Since we want to train the small domain ex- perts by fine-tuning an existing model, we need a model family that has both large and small models sharing the same tokenizer, which OPT provides.
512, with ReLU non-linearities, and one- dimensional output layer with sigmoid non- linearity, to force into (0,
Full-vector: Input layer same for full- scalar, 2 - followed by ID BatchNorm, two hidden layers of dimension 512, with ReLU non-linearities, and a |V|- dimensional output layer with a sigmoid non- lincarity, to force it into (0, 1)M1
2.
To rigorously determine what constitutes new domain for the models, we need to know what data they were trained on, which is not public for most proprietary models behind APIs.2
We train all combination networks using the Adam optimizer and a learning rate of 2 with the exception of constant-vector, for which we use a learning rate of 1e-2, 2, and a batch size of 1024. We run optimization for single epoch in all cases, found this to be enough in preliminary experi- ments.
We report results for the large model and the small fine-tuned model, which can be taken as the baselines, as well as their combination through our proposed method. For the parametrization of the combination functions, we use small neural networks, with the following architectures:
Note that the mean combination function has no learnable parameters. Finally, we also report max- prob oracle results as the upper-bound, which sim- ulates a perfect combination function that gives 100% of the weight to the best model for any given token.
Constant-sealar: A single neuron with imput, passed through sigmoid to force it into (0, 1).
Constant-vectors vector of neurons with no input, passed through a sigmoid to force into (0, IV
3.2 Evaluation
For evaluation, we adapt our model for three new domains and downstream task. The three new domains are defined by three datasets:
Ectaopy-scalar: Input layer two- dimensional, consisting of both entropies, followed by ID BatchNorm. two hidden layers of dimension 512, with ReLU non- linearities, and one-dimensional output layer with sigmoid non-linearity, to force it into (0, 1).
The Amazon Reviews dataset (McAuley al., 2015; He and McAuley, 2016), consist- ing of large collection of reviews and ratings entered by users on the Amazon website.
The Enron Emails dataset (Klimt and Yang. 2004), consisting of internal emails made pub- lic by the Federal Energy Regulatory Com- mission during the investigation of the Enron company.
Entropy-vector:
Input layer same as for entropy-scae scalar, followed by 1D BatchNorm, two hidden layers of dimension 512, with ReLU om-liinarities, and a /iiiiiaaaaaaaiiiiiiiiiiiiii -dddmenssinn output layer with sigmoid non-linearity, to force it into (0, I)
The FreeLaw subset of The Pile (Gao et al., 2021), consisting of large collection of court opinions from federal and state courts.
Input layer is -iimenmmesi,,ll,, consisting on the concatenated output dis- tributions for each model, followed by ID BatchNorm, two hidden layers of dimension
Full-scalar:
For each dataset, we extract two sets of 1000 1024-token sequences, which we call train-fit and test, respectively, and use the rest for the train set. The train-fit sets used to fit the combination functions, and we report perplexity on the test sets for evaluation. We use the train set fine-tune 1.T- using the Adam optimizer, 1024-token sequence length, a fixed learning rate of 4e-4, and a batch size of 1 1024 90 = 92160 tokens. the
'Although possible to either adapt LMs new vocab- ulary extend approach work with different tokenizers, that would add new dimension to our experiments, separate from the core research question that we want to study.
"While this is not problem for applying our method in practice, it does rule out proprictary black-box models for scientific study.

---

Amazon
Enron
Freelaw
en-de
en-cs
de-en
cs-en
avg
OPT-1.3B FT OPT-30B
17.00
3.30
4,98
OPT-1.3B FT OPT-30B
52.36
32.66
67.95
60.47
53,36
20.37
5.53
6.50
54.77
29.21
68.45
61.83
53.56
Mean
15.88
3.47
4.92
Mcan
57.62
35.34
69.84
63.62
56.61
Constant-scalar
15.80
3.27
4.84
Constant-scalar
57.73
35,08
69.70
63,70
56.56
Constant-vector
15.62
3.31
4.82
Constant-vector
57.71
34.69
69.60
63.64
56.41
Entropy-scalar
15.50
3.24
4.78
Entropy-scalar
57.87
35.18
69.59
63.88
56.63
Entropy-vector
15.41
3.24
4.76
Entropy-vector
58.11
35,41
69.44
64.06
56.76
Full-scalar
15,36
3.27
4.79
Full-scalar
57.98
35.06
69.57
63.59
56.55
Full-vector
15.43
3.27
4.79
Full-vectors
58.02
35.31
69.66
63.37
56.59
Max-prob (oracle)
12.59
2.89
4.12
Table MT results (BLEURT). The learned com- binations significantly outperforms both models in downstream task, often by a substantial margin.
Table 1: Domain adaptation results (perplexity). By combining a small domain expert and large general model, we achieve better perplexities than either of the original models.
Results
We next present our main results domain adap- tation ($4.1) and MT ($4.2).
case of Enron Emails we fine-tuned for single epoch, corresponding 3000k steps. For Amazon Reviews and FreelLaw performed 30k steps, and had to stop well before reaching the first epoch, due to compute constraints. Unless otherwise stated, the full train-fit sets are used to fit the combination functions.
4.1 Domain adaptation
We report domain adaptation results in Table 1. We observe that the combined models are able achieve substantially lower perplexities than either of the individual models. Even simple averaging works remarkably well, improving over both base- lines in Amazon Reviews and FreeLaw, but learned combinations perform best. The entropy-scalus combination works best across the board, achiev- ing relative improvement in perplexity of 9% in Amazon Reviews, 2% in Enron Emails and 1% in FreeLaw over the best single model. This supports our hypothesis that output distribution entropies are informative to the combination function. How- ever, higher capacity combination functions like full-scalar work better in some cases, as is the case for Amazon Reviews.
For downstream evaluation, experiment on English-Czech and English-German MT using the WMT21 dataset (Barrault al., 2020). We create a training set by verbalizing all the sentence pairs and concatenating them into a single corpus. De- tails of the verbalization templates can be found in Appendix A. We create a validation set follow- ing the same procedure (Akhbardeh et al., 2021), and extract train-fit set of 1000 1024-token sequences for fitting the com- bination functions, as we do in domain adaptation. Following the recommended practice in the area (Freitag et al., 2022), we use BLEURT (Sellam et al., 2020) on the WMT21 test set as our eval- uation metric, and report additional results with BLEU (Papineni et al., 2002) in Appendix B. We used 3-shot prompting for evaluation, longer sequence lenghts resulted in OOM issues in our hardware. We use the training set fine-tune OPT- 1.3B using the exact same settings described above. We train for 2k steps, corresponding to a total of around 2.5 million parallel sentences."
the WMT20 test set
Overall, results show that the adapted model is able leverage domain-specific knowledge in the small model, well as the knowledge in the large generalist model, order to improve over either of them. However, there is still a significant gap between the adapted models and the max-prot oracle, suggesting gains could still be made through better combination function.
4.2 Machine translation
Table 2 reports downstream results on MT. As for domain adaptation, all the Iearned combina- tions outperform both the small fine-tuned model and the large black-box model. This shows that our approach can work for adaptation to down- stream tasks, and is not limited to domain adapta- tion. Once again, the simple mean combination per-
'Although the full combined training set for English- German and English-Czech is bigger than 2.5M parallel sen- tences. we were interested in simulating the setting where limited translation data is available. Given enough parallel data, one can train a strong translation system from scratch, without having to adapt a generalist model.

---

forms very well, obtaining the second best results after entropy-vector. In any case, the combination function has a relatively small impact in MT, and even the worst performing approach brings large improvements over the baseline.
100
500
1000
OPT-L.3 FT OPT-30B
17.00
17.00
17.00
20.37
20.37
20.37
Mean
15.88
15.88
15.88
Constant-scalar
15,80
15.80
15.80
Constant-vector
15.80
15.66
15.62
Entropy-sealar
15.51
15.50
15.50
5 Analysis
Entropy-vector
15.52
15.45
15,41
Full-sealar 15.63
15.40
15.36
In this section, we study the following aspects of our approach:
Full-vector 15.71
15.49
15.43
Table 3: Perplexity on Amazon Reviews, using a differ-
How dependent the quality of the resulting model on the amount of data used to fit the combination function?
ent number of sequences to fit the combination function. Perplexity improves with the number sequences, but results already strong with only 100 sequences.
How dependent the quality of the resulting model on the amount data used to fine-tune the small LM?
2FF-13311 077-300
How much is general language modeling per- formance degraded by domain adaptation?
Is the learned combination interpretable?
5.1 Effect of the amount of data for fitting
L800
3000
2808
2530
order to study how the performance of the adapted model varies with respect to the amount of data used to fit the combination function, we fit each combination function three times, on vary- ing number of tokens. We report results for the Amazon Reviews dataset in Table 3, and additional results in Appendix B.
Figure 2: Perplexity on Amazon Reviews, varying the amount of fine-tuning steps.
avoid any loss in performance when combining with a poor domain expert. At the same time, it is also remarkable that the combination of vanilla OPT-1.3B and OPT-30B is not better than OPT- 30B alone. This can also be seen in Table 4, which compares using vanilla OPT-1.3B and fine-tuned OPT-1.3B as the small model. This shows that our reported improvements do not solely come from an ensembling effect, and our proposed approach effectively combines the power of the large LM and the domain expertise of the small LMM
As expected, performance improves with more training data. However, the difference varies across methods. For example, constant-scalar, which has a very low capacity, performs equally well when trained on 100 or 1000 sequences. On the other hand, the full-scalar and full-vector functions, that take the entire probability distribution as input, ben- efit from more training sequences. The entropy- scalar combination strikes a good balance, per- forming well across the board, and retaining strong performance when fit on as little as 100 sequences.
In addition, we observe that our combined LM substantially improves upon each individual LM as early as step 3000. fact, the gap between the small fine-tuned LM and our combined LM slightly narrows as training progresses. For in- stance, for entropy-scalar, the gap between the small LM and the combined LM is 2.18 perplexity points at step 3000 (12% relative improvement), which goes down to 1.5 for the fully fine-tuned model (9% relative improvement). This is intuitive, as the more data available in the target domain, the less useful will be integrating the general knowl- edge in the large LM.
5.2 Effect of fine-tuning steps
Figure 2 shows the performance of the adapted models, when fine-tuning the small model for a varying number of sequences. At step 0 (i.e., before fine-tuning begins), the small LM corresponds to vanilla OPT-1.3B, which performs considerably worse than OPT-30B on Amazon Reviews. Even in that case, entropy-scalar performs on par with OPT- 30B, while mean is slightly worse. This shows that learnable combination functions are able to

---

Orig
Amazon-fit
Mixin-fit
OPT-1.3B
26.03
17.00
Amazon
Pile
Amazon
Pile
OPT-30B
20.37
20.37
OPT-1.3B FT OPT-30B
17.00
19.78
17.00
19.78
Mean
21.12
15.88
20.37
6.82
20.37
6.82
Constant-scalar
20.28
15.80
Mean
15.88
7.72
15.88
7.72
venstant-vector
20.55
15.62
Constant-scalar
15.80
8.35
16.52
7.08
Entropy-scalar
20.37
15.51
Constant-vector
15.62
8.38
15.89
7.18
Entropy-vector
20.30
15.44
Estropy-scalar
15.50
7.35
15.80
6.94
Full-sealar
20.26
15.41
Entropy-vector
15.41
8.53
15.61
6,92
Full-vector 20.30
15.48
Full-scalar
15.36
9.31
15.45
6,85
Full-vector
15.43
10.07
15.48
6.91
Table Perplexity on Amazon Reviews, using either original OPT-1.3B fine-tuned OPT-1.3B the small LM. The combination methods barely improve upon OPT-30B when using the former, showing that our ap- proach does not only work due to an ensembling effect.
Table 5: Perplexity on Amazon Reviews and The Pile. using either the former to fit the combination function (amazon-fitt fit), or the concatenation of the two (mixin-fit).
Pile is almost entirely preserved, at the expense of slight degradation on Amazon Reviews. For example, for full-scalar, the combination fit on the mixin set achieves a perplexity of 15.45 on Amazon Reviews and 6.85 on The Pile, both within 0.1 of the best results for each dataset.
5.3 Effect on general language modeling
We are also interested in measuring how well the adapted models retain the general language mod- eling ability of the original large model. We use perplexity on The Pile (Gao et al., 2021) )) as proxy of general language modeling performance, as is a large collection of many datasets from different domains, often used to train generalist LMs (Black et al., 2022; Biderman et al., 2023). To this end, we also extract random train-fit and test subsets from The Pile. While some subsets of The Pile are also present in the training data for OPT, we do not measure performance on The Pile as a bench- mark for model quality, and are only interested in it as proxy for degradation in general language modeling ability of the adapted models.
Overall, these results show that large model can be adapted to particular domain while mitigat- ing degradation in the general domain by mixing in-domain and general text to train the combination function. Additionally, we find that different com- bination methods exhibit different behavior when it comes to general performance degradation, even when they exhibit similar in-domain performance.
5.4 Is the model combination interpretable?
We next analyze whether the weights given to each model by the combination function are in- terpretable. Figure 3 illustrates this over random sample from Amazon Reviews: we show which tokens are better predicted by each model, along with which model is given a higher weight for each token. Although we do not identify clear pat- tern for which tokens are better predicted by each model, we do observe that the coloring in the top and the bottom visualizations match quite closely. This means that the learned combination function is quite good at predicting when each model should be given higher weight."
We compare fitting the combination function the target domain train-fit, as done throughout the paper, as well as on the combination of the target domain and The Pile train-fit sets. Table reports results for Amazon Reviews, and full results can be found in Appendix B.
When fitting the combination function on Ama- zon Reviews, observe significant degradation The Pile. However, different combination meth- ods behave differently in this regard. For example, entropy-scalar and full- perform similarly in Amazon Reviews (15.50 vs 15.43), but the former performs much better on The Pile (7.35 vs 10.07). It is also remarkable that The Pile perplexity of the combined model remains far better than the small fine-tuned LM (e.g. 7.35 for entropy-scalar 19.78 for the small LM), while also performing better in-domain.
In order to quantitatively analyze this, we mea- sure the Spearman correlation between the weight given by the combination function, and the actual difference in log probabilities for each token. Re-
"A perfect combination function (corresponding to the max-prob oracle in Table 1) would always give 100% of the weight to the hest model for any given token, and both images would match up perfectly.
When fitting the combination function on the mixin set, we observe that performance on The

---

Domain
Pile
models through parameter-efficient adaptation tech- niques, such adapters (Houlsby al., 2019). soft-prompt tuning (Liu et al., 2022), or low-rank adaptation (Hu et al., 2022). However, all of these techniques require white-hox access to the origi- nal model and full backward passes, making them incompatible with black-box models.
Entropy-scalar
0.59
0.71
Amazon
Full-scalar
0.44
0.32
Entropy-scalar
0.49
0.75
Freelaw
Full-scalar
0.33
0.32
Entropy-scalar
0.54
0.75
Enron
Full-scalar
0.25
0.30
In contrast, discrete prompt tuning approaches allow for treating the large model as a black-box (Shin et al., 2020; Sun et al., 2022; Zhang et al., 2023; Cheng et al., 2023). However, these ap- proaches have only been proven in the limited set- ting of retrieving zero- or few-shot prompts that improve performance in set of NLP tasks that the base black-box is already capable of performing, opposed to general method of black-box model adaptation.
Table 6: Spearman correlation between the log- probability difference of the LMs and the weight given by combination function. Larger values mean that the learned combination is closer to the ideal oracle weighting. Rows represent adapted models on different domains and combination functions, fit on the in-domain train-fit.
sults are shown in Table 6. We limit our analysis to entropy-scalar and full-scalar, as they are the only ones that output a single combination factor that depends on the input. We observe significant corre- lations for all datasets, with entropy-scalar achiev- ing better correlation than full-scalar, especially The Pile. This is consistent with the results in Table 5, where full-scalar suffers bigger performance loss on The Pile. Somewhat surprisingly, correla- tion for entroyy-ccalar is better on The Pile than on the in-domain dataset, even though the com- bination function fit on the in-domain train-fit. One possible explanation is that The Pile better represents the training distribution of the large LM, making it better calibrated on it, which makes it easier for entropy-scalar to make predictions.
Concurrent to our work, Huang et al. (2023) pro- pose leveraging KNN retrieval from a data-store to augment an cxisting black-box LM. However, they only experiment with small GPT2 models the
and the adaptation depends find- ing adequate datastore, limiting application downstream tasks such as MT.
6.2 Domain experts for language modeling
Another line of research explores language mod- eling through a combination of separate domain experts. Li et al. (2022) achieve better performance than compute-matched single transformer models and highly parallel pre-training, by training inde- pendent domain experts, and combining them at the parameter level inference time. Gururangan al. (2023) extend this approach to automatically dis- covered domain clusters. Other approaches replace components of the transformer network with inde- pendent domain-dependent modules, as is the case of Gururangan et al. (2022) for metadata-defined domains, or Pfeiffer et al. (2022) for per-language modules. All of these pre-training approaches and seek to train better more efficient LMs, but cannot leverage existing powerful black-box mod- els. Our work, in contrast, seeks to adapt an exist- ing powerful black-box through leveraging a much smaller domain expert.
6 Related work
We present related work on domain adaptation of LMs (56.1), and language modeling through do- main experts (56.2).
6.1 Domain adaptation of LMs
Domain adaptation of LMs is an extensively stud- ied line of research. Traditional approaches include fine-tuning the model on domain- specific corpora, (Devlin et al., 2019; 9; Liu et al., 2019; Gururangan et al., 2020), data selection on the original general corpus (Aharoni and Goldberg, 2020; van der Wees et al., 2017), and adapting or extending the tok- enizer to achieve better performance on the target domain (Sachidananda et al., 2021).
Conclusions
Although effective, these full fine-tuning tech- niques are often infeasible at scale due to the ex- cessive compute required. Some approaches aim to reduce the resources required to fine-tune large
In this work, we present a method for adapting black-box LMs new domains and tasks, requir- ing access to probability-Level outputs only. We first fine-tune small domain expert white-box LM

---

have never gotten tired of that how ever many times listen just want listen to again.It looks like handkerchiel hem. but it is not. More straight with aillit on each side. Just was not what I was hoping for This has a very nice selection of cards to choose from and is very easy Works like a charm! charm!Everyboo Everybody loves Dumbo for all the right reasons great story with humor and pathos, wonderful music, and delightful animation. However, one seems to have noticed the underlying racial themes that fuel the plot. Dumbo's mom, and the other female elephants she lives and works with, are Indian elephants (small ears), Dumbo's i dad (Jumbo), from whom he must inherited his big ears, must have been African. Dumbo (and his/mother) were mocked and ultimately ostracized from decentielephant society because he was the product of mixed marriage. Only after he learns (with the holp of those zgol-Sur Sieedd lIve talkin' rross use his physical edofect! to excel at something (Iving) is accepted back into the circus, While "Dumbo" teaches us that we're all "special," " it also paints a rather darker picture of society being
love putting personalized names on my cards and this lets me do
intolrant of differences unless or until those differences can benelit that society
(u) Log- pre-babbaiilty difference between the small and large L.M. The small fine-tuned LM gave higher probabilitics to the green tokens, while the large black-box LM gave higher probability to the red ones.
have never gotten tired of that how ever many times I listen to it I Just want to listen to it again It looks like a handkerchief hem, but it not, More straight with slit on each side. Just was not what I was hoping for This has very nice selection of cards to choose from and is very thatWorks like a charm! Everybody loves Dumbo for all the right reasons great story with humor and pathos, wonderful music, and delightful animation. However, no one seems to have noticed the underlying racial themes that fuel the plot. Dumbo's s mom. and the other female elephants she lives and works with. are all Indian elephants (small ears). Dumbo's dad (Jumbo), from whom he must have inherited his big ears, must have been African, Dumbo (and his mother) were mocked and ultimately ostracized from decent elephant society because he was the product of a mixed marriage, Only after he learns (with the help of those zoot-suited, jive talkin' crows) to use his physical "defect" to excel at something (flying) is he accepted back into the circus. While "Dumbo" teaches us that we're re all "special," it also paints a rather darker picture of society being intolerant of differences unless until those differences can benefit that society
to use. I love putting personalized names on my cards and this lets me do
(b) Weight given to each model by entropy-sealar. Tokens for which a higher weight was assigned to the small fine-tuned LM are shown in green, while tokens for which the large black-box was given higher weight are shown in red.
Figure 3: Difference between the small fine-tuned LM and the large black-box LM according to pog-prbbybylity (a) and predicted weight (b). The closer the two match, the better the learned combination is at predicting which model will be "right" for given token. The text sample was chosen randomly from the Amazion Reviews testset.
on a domain-specific corpus, and then combine it with the large black-box through combination function learned on a small fitting set, yielding an adapted LM. Additionally, our method requires only access to probability level outputs, and thus allows to leverage powerful models optimized for inference or behind APIs, without the need for white-box access to the weights. We experiment on several datasets and a downstream task, as well performing extensive analysis of our method, reaching several conclusions:
is data available to fine-tune the domain ex- pert. In fact, the gains are higgest in this sce- nario, as the advantage of leveraging good black generalist decreases when a big in- domain corpus is available.
While adaptation to new domains incurs a loss of general language modeling ability, this varies per combination method, and seems to be largely mitigated by augmenting the small set on which the combination function is fit.
By combining a small domain expert and a large black-box model, the combined model outperforms either of the original ones in all cases, by as much 9% perplexity for do- main adaptation, and 6% BLEURT for MT, showing the effectiveness of our approach.
While our approach is effective, observed per- formance is still not close to the max prob oracle, which represents the idcal system where 100% of the weight is given to the best model at each time step. In future work, we would like to investigate the reasons for this gap, and potential ways of ad- dressing it.
While higher capacity combination functions can perform better when more data is used to learn the combination, lower capacity combi- nation methods remain competitive, and per- form better when learned on little data. In particular, the entropy based combinations, entropy-scalar and entropy-vector, perform well across the board, even when fit on as little as 100 sequences.
Limitations
While our method requires no access to the black- box model's weights and intermediate activations, it does assume access to the full output probability distribution, which might not be available for some models served through APIs The OpenAl API, for example, only returns the probabilities for the top 5 tokens. This is not an issue for the Constant combi- nations, and the Entropy methods could potentially
Our approach is effective even when little

---

also be adapted, by estimating the entropy from top K probabilities.
1-55, Online. Association for Computational Linguis- tics.
Additionally, even though we don't fine-tune the black-box at all, our method does require a forward pass of the large black-box through a fitting set, which might potentially be costly if done through APIs, depending on pricing and the size of the fitting set.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
Acknowledgements
Sid Black, Stella Biderman, Eric Hallahan, Quentin An- thony, Leo Gao, Laurence Golding. Horace He, Con- nor Leahy, Kyle McDonell, Jason Phang. Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20h: An open- source autoregressive language model.
Aitor and Eneko were partially supported by the Basque Government (LXA excellence research group IT-1805-22; IKER-GAITU project). Aitor was supported by a doctoral grant from the Spanish MECD.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
References
Roee Aharoni and Yoav Goldberg, 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7747- 7763. Online. Association for Computational Lin- guistics.
Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondfej Bojar, Rajen Chatter- jee. Vishrav Chaudhary. Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Ro- man Grundkiewiez. Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Ke Happosaa-KKaayire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Au- guste Tapo, Marco Turchi, Valentin Vydrin, and Mar- cos Zampieri. 2021. Findings the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference Machine Translation, pages 1-88, Online. Association for Computational Linguis- tics.
Daixuan Cheng, Shaohan Huang. Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang. Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang. 2023. Uprise: Universal prompt retrieval for improving zero-shot evaluation.
Äakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran. Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thamumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slay Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Galen Andrew and Jianfeng Gao. 2007. Scalable train- ing of L, -erguullriized log-lineau linear models. In Proceed- ings of the 24th International Conference on Machine Learning, pages 33-40.
Loie Barrault, Magdalena Biesialska, Ondiej Bo- jar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Had- dow, Matthias Huck, Eric Joanis, Tom Koemi, Philipp Kochn, Chi-kiu Lo. Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshi- aki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). Proceedings of the Fifth Conference on Machine Translation, pages
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing.

---

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo. Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Machine Learning: ECML 2004, pages 217-226.
Berlin, Heidelberg. Springer Berlin Heidelberg.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zeule- moyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 6 61-68, Dublin, Ireland. Association for Computational Lin- guistics.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He. Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.0005
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5557-5576, Seattle, United States. Association for Computational Linguisties.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-basee based recom- mendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 15,, page 43 52, New York, NY, USA. Asso- ciation for Computing Machinery.
Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettle- moyer. 2023. Scaling expert language models with unsupervised domain discovery.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311 311-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith, 2020. Don't it stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, 8360, Online. Association for Computational Linguisties.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Scbastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 3479-3495, Seattle, United States. Association for Computational Lin- guistics.
Ruining He and Julian McAuley. 2016. Ups and downs:
Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, WWW 16, page 507-517, Republic and Can- ton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021, Efficientt domain adaptation of language models via adaptive tokenization. In Proceedings the Second Workshop on Simple and Efficient Natural Language Processing, pages 155-165, Virtual. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, 2019, Parameter-efficcent transfer learning for nip.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Yangsibo Huang, Daogao Liu, Zexuan Zhong, Weijia Shi, and Yin Tat Lec. 2023. knn-adapter: Efficient domain adaptation for black-box language models.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Fric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the
Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In

---

2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222 4235, Online. Association for Computational Linguisties.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for -asaaag---mmoll-----seeeeee-eeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural ma- chine translation. In Proceedings of the 2017 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 1400-1410, Copenhagen, Denmark. Association for Computational Linguistics.
Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine transla- tion: A case study.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre- trained transformer language models.
A MT verbalizations
We verbalize the MT task by first adding prompt describing the task, and then adding several transla- tion examples. We chunk the translation examples in blocks of 5, that is, adding translation exam- ples per verbalization. We use two different task de- scriptiopns. shown in Table 7, and alternate evenly between both variations create the verbalized verbaliza-
training corpus. For inference, tion #1 and draw random translation pairs from the WMT21 development set to construct 3-shot prompt. We draw the random translation pairs once, and keep the 3-shot prompt fixed for all models.
B Full results
Full results for all combination methods, dataset sizes, and evaluation settings are shown in Table Table reports additional MT results using BLEU.

---

Verhalization #1
Verbalization #2
Translate the following sentences from $LI to SL2:
Given a sentence in $L1, translate it to $L2:
$LI: $S1
$LI: $S1
SL2: ST1
$L2: ST1
$L1: $S2
$LI: $S2
SL2: $T2
$L2: $T2
$L1: $S3
$L1: $S3
SL2: ST3
$L2: $T3
$L1: $S4
$1.1 $S4
$12: $T4
$1.2 ST4
$1.: $S5
$1.1: $55
$L2: ST5
$L2: STS
Table 7: Both verbalizations used for MT. $L1 SL2 represent the source and target languages, and $Si and STi represent the source and target sentences for the ith pair in the verbalization.

---

Pil.
7.56
8.90
8.37
8.33
8.05
9.45
9.27
15.55
6.82
5.75
Pil.
7.56
6.96
7.04
6.75
6.78
6.77
6.80
15.55
6.82
5.75
2000
Dom.
4.92
4.84
4.82
4.78
4,76
4.79
4.79
4.98
6.50
4.12
Dom.
4.92
5.15
4.96
4,92
4.85
4.85
4.87
4.98
6.50
4.12
PiL
7.56
9,90
8.05
9.12
8.05
9.44 9.48
9.30
15.55
6.82
5,75
Pil.
7.56
6.94
7.14
6.80
6.90
6.72
6.90
15.55
6.82
5.75
Freelaw
Freelaw
1000
Dom.
4.92
4.84
4.84
4.78
4.77
4.79
4.80
4,98
6.50
4.12
Dom.
4.92
5.16
4.95
4.88
4.83
4.90
4.89
4.98
6.50
4.12
Pil
7.56
8,98
7.68
7.84
8.05
8. 13
7,90
15.55
6,82
5.75
Pil.
7.56
6.92
7.37
6.76
6.92
6.78
7.05
15.55
6.82
5.75
100
200
COMBINATION FUNTION FIT ON MIX OF DOMAIN AND THE PILE tratin-fit fft
Dom.
4.92
4.84
4,90
4.78
4.80
4.82
4.85
4,98
6.50
4.12
Dom.
4.92
5.18
4.93
4.91
4.89
4.93
4.92
4.98
6.50
4.12
COMBINATION FUNTION FIT ON TARGET DOMAIN frain- fit
PIl
7.45
9.63
8.39
8.02
8.18 8.18
9.86
9.82
12.73
6.82
5.89
PIL
7.45
7.24
7.27
7.01
6.98
6.99
7.05
12.73
6.82
5.89
1000
2000
Dom.
3.47
3.27
3.31
3.24
3.24
3.27
3.27
3.30
5.53
2.89
Dom.
3.47
3.56
3.45
3.36
3.35
3.45
3.44
3. 9.30
5.53
2.89
Enron Emails
Pil
7.45
9.75
8.03
8. 10
8.53
10.11
10.54
12.73
6.82
5.89
Enron Emails
Pil.
7.45
7.24
7.32
7.02
7.04
7.06
7.17
12.73
6.82
5.89
500
1000
Dom.
3.47
3.27
3.34
3.24
3.24
3.27
3.27
3,30
5.53
2.89
Dom.
3.47
3.56
3.44
3.35
3.34
3.47
3.42
3.30
5.53
2.89
Pil
7.45
9.89
7.59
9.30
8.22
8.22
1.11
12.73
6.82
5.89
Pil.
7.45
7.22
744
6.98
7.12
7.10
7.37
12.73
6.82
5.89
100
200
Dom.
3.47
3.27
3.42
3.24
3.25
3.32
3.34
3.30
5.53
2.89
3.47
3.57
3.45
3.38
3.36
3.47
3.41
3.30
5.53
2.89
Pil
7.72
8.35
8.38
7.35
8.53
9.31
10.07
19.78
6.82
5.93
PIL
7.72
7.08
7.18
6.94
6.92
6.85
6.91
19.78
6.82
5.93
2000
Dom.
15.88
15.80
15.62
15.50
15.41
15.36
15.43
17.00
20.37
12.59
15.88
16.52
15.89
15.80
15.61
15.45
15.48
77.00
20.37
12.59
Amazon Reviews
Amazon Reviews
Pil.
7.72
8.35
8.12
7.51
8.20
9.28
9.62
19.78
6.82
5.93
Pil
7.72
7.10
7.29
6.98
6.96
6.90
6.98
19.78
6.82
5.93
500
1000
Dom.
5.88
15. 80
15.66
50
15.45
15.40
40
15. .9
17.0
20.37
12.59
Dom.
15. .888
16.47
15.85
15.71
15.68
15.42
15.53
17.00
20.37
12.59
PiL
7.72
8.35
7.82
7.30
8.10
7.97
7.90
19.78
6.82
5.93
Pil
7.72
7.06
7.53
6.92
7.07
6.91
7.16
19.78
6.82
5.93
200
Dom.
15.88
15.80
15.80
15.51
15.52
15.63
15.71
17.00
20.37
12.59
Dom.
15.88
16.56
15.85
15.87
15.69 3.69
15.55
15.63
17.00
20.37
12.59
OPT-301 30B Max-prob (oracle)
Max prob (oracle)
sequences
Constunt constunt-scalar scalaa
Constant-vees vector
Entropy satropy-scalls scalar
Entropy serooy--vectee vector
#Fit sequences
Constant constant-scalar scalar
Constant Constant-vector vector
Entropy scalar
Entropy vector
Eval domain
OPT- 1.3B FT
Eval domain
OPT- .33 FT
Full scalar
Full vell-vector
Full-scalar
Full-vector
OPT-30B 30B
Dataset
Dataset
Mean
Mean
Table Full results combination methods, when on different amount tokens, and on different domains.
Note that the #Fit sequences is doubled when fitting the combination function on mix of in-domain and Pile data,
since the same number of tokens drawn from each.

---

en-de
en-cs
de-en BLEURT
BLEURT
BLEU
BLEURT
BLEU
BLEU
BLEURT
BLEU
BLEURT
BLEU
Mean
57.62
14.39
35.34
5.76
69.84
26.72
63.62
21.57
56.61
17.11
Constant-scalar
57.73
13.76
35.08
5.66
69.70
26.68
63.75
21.32
56.56
16.86
Constant-vector
57.71
13.88
34.69
5.28
69.60
26.65
63.64
21.41
56.41
16.80
Ectropy-scalar
57.87
13.76
35.18
5.60
69.59
26.30
63.88
21.31
56.63
16.74
Entropy-vector
58.11
14.25
35.41
5,64
69.44
26.73
64.06
21.47
56.76
17.02
Full-scalar
$7.98
14.22
35.06
5.52
69.57
25.86
63.59
20.50
56.55
16.52
Full-vector
58.02
14.11
35.31
5.19
69.66
26.13
63.37
20.96
56.59
16.60
OPT-1.3B FT OРT-30B
52.36
15.05
32.66
5,48
67.95
25.27
60.47
19.13
53.36
16.23
54.77
9.64
29.21
3.13
68.45
24.08
61.83
18.49
53.56
13.84
Table 9: Full BLEU and BLBURT results for MT.


Reviews:
[{"review_id": "E4qQ3oCp0D", "reviewer": "Reviewer_ngzh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and their comments. Regarding the issue with APIs and requiring full access to the output distribution, we would like to point out that even when access to the full output distribution is not available, some combination functions like mean and constant-scalar can be readily used for scoring. In addition, a setting where we have access to the full output distribution of an LLM but cannot easily fine-tune it is not unrealistic: a research group or company might have an optimized inference solution for an LLM allowing to run inference over the model, but not enough resources to perform fine-tuning on it. This is realistic given that the memory and overall compute requirements of fine-tuning are significantly higher than those of inference. Our method would be readily applicable to this common use case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " This paper presents a lightweight method for adapting large black-box language models (LMs) to new domains and tasks, assuming no access to their weights or intermediate activations. The approach fine-tunes a small white-box LM and combines it with the large black-box LM at the probability level through a small network learned on a small validation set. The paper's main contributions are a method for adapting black-box LMs to new domains and tasks, requiring access to probability-level outputs only, and the validation of this approach by adapting a large LM to several domains and a downstream task, observing improved performance in all cases. This provides a new way to adapt powerful black-box models to new domains and tasks.\n\nThe authors comprehensively test their proposed methods against Domain Adaptation nuances and also Machine Translation for Amazon Reviews, Enron, and Freelaw (all three datasets have differences in linguistic characteristics - with a good combination of varied degree of language unstructuredness, which makes the results more reliable)", "reasons_to_reject": "Assuming access to output probability distribution is one of the major caveats of the methods presented by the authors, as models served through APIs,especially large language models cannot leverage the same framework for investigation. And given the recent interest in the investigation for LLM model, the proposed framework has limited applicability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors introduce a cost-effective method to adapt these large models without needing access to their internal structures or weights. The method involves fine-tuning a smaller, accessible model and integrating it with the larger model using a minor network. This technique was tested on a large language model, showing performance improvements of up to 9% using a domain-specific model that's 23 times smaller.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lxa9JkLIS6", "reviewer": "Reviewer_32aC", "summary": "", "questions": "How do you deal with the potential bias or ethical issues that might arise from adapting black-box LMs to new domains or tasks, especially when the source of the data is not transparent or trustworthy?\nHow do you plan to extend your method to other types of black-box models, such as image or speech models?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We comment on the questions and some of the concerns below:\n- Regarding the blind review violation, the paper was uploaded to arXiv on 22 May 2023. This is before the anonymity period, which began on 23 May 2023 (Important Dates section of https://2023.emnlp.org/ ). In fact, the existence of the online version was reported to EMNLP upon submission.\n- Regarding the tokenizer issue, while it is true that sharing the same tokenizer is a requirement, as long as the tokenizer of the black-box model is known, as is the case for example for the OpenAI API models, one can train the small expert model using this tokenizer, either from scratch or adapting an existing model with a new tokenizer, so our approach can be applied. \n- Regarding the lack of theoretical analysis, we find that the reviewer’s suggestions are vague and/or not applicable to our work. For instance, the reviewer suggests that we analyze the convergence of the combination functions, but this is not mathematically well-defined. For the convergence to be defined in the usual sense, one would need a sequence of elements belonging to a metric space. Here, we have a single combination function, not a sequence, and it is not clear how one would define a relevant distance metric to turn the space of combination functions into a metric space. It is similarly unclear how the terms “optimality” or “stability” could be defined for our combination functions so they can be analyzed theoretically. We would appreciate it if the reviewer could offer more concrete feedback.       \n- Regarding the comparison to products-of-experts models, while they share the aspect of using combination functions with our work, these are not adaptation, but pre-training approaches that seek to obtain better final models. They require substantially more compute to train from scratch, and  cannot leverage a pre-trained large model, making them significantly different in spirit to our approach. \n- Regarding the adaptation of our method to other modalities, such as image or speech models, as long as the model outputs a probability distribution over a vocabulary, our combination approach could be applied, no matter if this vocabulary represents textual tokens, visual codebook tokens, or any other kind. Thus we would expect our model to be easy to extend to other modalities that employ autoregressive generation over discrete tokens. \n- Regarding potential bias or ethical issues, our approach would face the same potential issues as any other other LLM or ML method, but we don’t expect it to be an issue specific to our approach. While the combined model could inherit the undesirable behaviors of either of the original models, there is no reason our approach would exacerbate such problems or create new ones. As such, we do not think that our work poses any new bias or ethical issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It proposes a novel and simple method for adapting black-box LMs to new domains and tasks, without requiring access to their weights or intermediate activations. This is useful for leveraging the power of large LMs that are only available as black-boxes through inference APIs, or when the computational cost of fine-tuning them is prohibitive. It explores different alternatives for the combination function that merges the probability distributions from the small domain expert LM and the large black-box LM, and studies how their capacity affects the performance of the adapted model. It also shows that output distribution entropies are informative for the combination function, and that entropy-based combinations perform well across different domains and tasks.", "reasons_to_reject": "It assumes that the large black-box LM and the small domain expert LM share the same tokenizer, which might not be the case for some models behind APIs. This could limit the applicability of the proposed method to some scenarios.\nIt does not provide any theoretical analysis or guarantees for the combination functions, such as their convergence, stability, or optimality. It also does not compare them with existing methods for combining probability distributions, such as mixture models or product-of-experts models.", "questions_to_authors": "How do you deal with the potential bias or ethical issues that might arise from adapting black-box LMs to new domains or tasks, especially when the source of the data is not transparent or trustworthy?\nHow do you plan to extend your method to other types of black-box models, such as image or speech models?", "paper_topic_and_main_contributions": "This paper is about a method for adapting large language models to new domains and tasks, without requiring access to their weights or intermediate activations. The main contributions of this paper are:\n\nIt proposes a simple and lightweight approach that combines a small white-box language model, fine-tuned on a domain-specific corpus, with a large black-box language model, through a combination function learned on a small validation set.\nIt shows that the adapted model outperforms both the small fine-tuned model and the large black-box model in several domains and a downstream task (machine translation), while using much less compute than full fine-tuning.\nIt analyzes the effect of the amount of data used to fit the combination function and to fine-tune the small model, as well as the effect on general language modeling performance and the interpretability of the learned combination.\n\nBut, it has been public on the arXiv (https://arxiv.org/pdf/2205.12213v2.pdf), I think this is not ok for the blind review.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Violation of blind review regulations, see arXiv (https://arxiv.org/pdf/2205.12213v2.pdf)."}, "extra_text": {}}, {"review_id": "LNXUc25m5K", "reviewer": "Reviewer_BRRy", "summary": "", "questions": "No questions, this work is really cool!", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work allows interpretable adaptation of black-box language models.\n\nThe proposed method thoroughly explores different combination functions, providing a great overview of of ways to externally adapt large language models.\n\nThis method is very useful and timely given the rise in black-box language models.\n\nThe paper is well written, easy to follow and reproducible.", "reasons_to_reject": "The work is only done in English. It would be nice to see how this method adapts large models to other languages or which combination works best in other languages.", "questions_to_authors": "No questions, this work is really cool!", "paper_topic_and_main_contributions": "This paper externally adapts black-box large language models with a white-box smaller language model to enable better performance to new tasks and domains. The larger model is adapted via the probability distribution, where the probability distribution of the black-box model is combined with the probability distribution of the white-box model. The paper explores a wide range of combinations of probability distributions with varying capacity.\n\nThe method is evaluated via model adaptation to new domains and downstream with machine translation. Entropy-based combination methods work well across the board, even when the model is fit with very few sentences.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Authors propose a method to efficiently adopt a small LM with the help of a larger LM by only having access to its probability outputs on a small development/validation set, without having need to the internal model weights. Results on English show the efficacy of the proposed method. From an efficiency perspective, the paper adds a lot of practical value to the literature while not solving all the issues in this space.
Decision: Accept-Main