Venue: EMNLP 2023

# CombLM: Adapting Black-Box Language Models

through Small Fine-Tuned Models

Aitor Ormazabal\({}^{1}\)   Mikel Artetxe\({}^{2}\)  Eneko Agirre\({}^{1}\)

\({}^{1}\)HiTZ Center, University of the Basque Country (UPV/EHU)   \({}^{2}\)Reka AI

aitor.ormazabalehu.eus mikel@reka.ai e.agirre@ehu.eus

###### Abstract

Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters. However, this is incompatible with a recent trend in the field, where the highest quality models are only available as black-boxes through inference APIs. Even when the model weights are available, the computational cost of fine-tuning large LMs can be prohibitive for most practitioners. In this work, we present a lightweight method for adapting large LMs to new domains and tasks, assuming no access to their weights or intermediate activations. Our approach fine-tunes a small white-box LM and combines it with the large black-box LM at the probability level through a small network, learned on a small validation set. We validate our approach by adapting a large LM (OPT-30B) to several domains and a downstream task (machine translation), observing improved performance in all cases, of up to 9%, while using a domain expert 23x smaller.

+
Footnote †: December 6-10, 2023 ©2023 Association for Computational Linguistics

## 1 Introduction

Natural language processing (NLP) has witnessed remarkable progress in recent years thanks to the development of increasingly powerful LMs Brown et al. (2020); Andrew and Gao (2007); Chowdhery et al. (2022); Touvron et al. (2023). Since these models are usually generalists, it is often of interest to adapt them to new domains, underrepresented or not found in the original training data. Typically, domain adaptation techniques assume white-box access to the model parameters, for example by fine-tuning on a particular target domain Gururangan et al. (2020).

However, this approach has become increasingly infeasible given the ongoing paradigm shift in the field--state-of-the-art models like GPT-4 and PaLM-2 are only accessible as black-boxes through inference APIs and, even when the model weights are available, the computational cost of fine-tuning large models can be prohibitive. Consequently, domain adaptation methods that cannot leverage the power of black-box LLMs are likely to fall behind.

In this work, we propose a simple and lightweight approach to adapt black-box LMs to new domains, without requiring access to weights or intermediate activations. Our method consists of two main steps: (1) training a small, white-box model on the desired target domain, and (2) learning a function that combines the probability distributions from the large black-box LM and the small domain expert LM, producing a new probability distribution. The combination function is a small neural network that is trained on a small validation dataset.

We evaluate our method by adapting a black-box model to three distinct domains and a downstream task--machine translation (MT). In all cases, we observe that the combined model outperforms both the large black-box model and the small domain expert. This shows that it is possible to adapt black-box LMs to new domains, opening an exciting line

Figure 1: **Illustration of our approach. We leverage a large black-box LM and a small white-box LM, fine-tuned on a domain-specific corpus. We combine both models’ outputs at the probability level, through a combination function learned on a small fitting set, requiring very little compute. The resulting model adapts the large black-box to the target domain, performing better than either of the original ones.**

of research.

## 2 Proposed method

Our approach works in two steps: (1) we train a small domain expert LM, and (2) we learn a function that combines the outputs of the domain expert LM and a large black-box LM at the probability level.

More concretely, an LM defines a probability distribution over the possible continuations of any given text. That is, given a sequence of tokens \({\bf x}=(x_{1},x_{2},...,x_{n})\in V^{*}\), where \(V\) is the model vocabulary, an LM parametrizes \(P_{LM}(y_{next}|{\bf x})\), the probability that \(y_{next}\) is the continuation of \({\bf x}\) in a text. We let \(P_{S}\) denote our small domain expert LM, and \(P_{L}\) denote the large black-box generalist LM. Our combination function \({\bf f}\) defines a new combined probability distribution \(P_{C}\): \(P_{C}(y_{next}|{\bf x})={\bf f}(P_{S}(\cdot|{\bf x}),P_{L}(\cdot|{\bf x}))_{ y_{next}}\). Here \({\bf f}:\mathbb{R}^{|V|}\times\mathbb{R}^{|V|}\rightarrow\mathbb{R}^{|V|}\) is a vector-valued function that receives full probability distributions, and outputs a new probability distribution.

To train the domain expert LM, we fine-tune a pre-trained model on a small domain-specific dataset. For the combination function, we consider several alternatives of varying capacity:

1. **Mean.** The arithmetic mean of the two distributions: \({\bf f}({\bf y_{1}},{\bf y_{2}})=({\bf y_{1}+y_{2}})/2\).
2. **Constant-scalar**. A linear combination of the two input distributions, with a constant combination factor \(\lambda\): \({\bf f}({\bf y_{1}},{\bf y_{2}})=\lambda{\bf y_{1}}+(1-\lambda){\bf y_{2}}\).
3. **Constant-vector**. A token-wise version of the previous combination, where \(\boldsymbol{\lambda}\in\mathbb{R}^{|V|}\) is a constant vector, and the combination factor varies per-token: \({\bf f}({\bf y_{1}},{\bf y_{2}})\propto\boldsymbol{\lambda}\circ{\bf y_{1}}+ (\boldsymbol{1}-\boldsymbol{\lambda})\circ{\bf y_{2}}\), where \(\circ\) is the Hadamard (elementwise) product. Note the proportionality instead of equality in the definition, as a re-normalization is required when combining distributions per-token.
4. **Entropy-scalar**. A scalar \(\lambda\) is predicted from the entropies of each distribution, \(\lambda=g(\mathrm{H}({\bf y_{1}}),\mathrm{H}({\bf y_{2}}))\), and the output is a linear combination as in _constant-scalar_: \({\bf f}({\bf y_{1}},{\bf y_{2}})=\lambda{\bf y_{1}}+(1-\lambda){\bf y_{2}}\). The function \(g\) is parametrized by a small neural network.
5. **Entropy-vector**. An token-wise version of the previous combination, where a vector \(\boldsymbol{\lambda}=\mathbf{g}(\mathrm{H}({\bf y_{1}}),\mathrm{H}({\bf y_{2}} ))\in\mathbb{R}^{|V|}\) is predicted , and then the per-token combination is done as in _constant-vector_.
6. **Full-scalar**. A single \(\lambda\) is predicted from full input distributions, \(\lambda=g({\bf y_{1}},{\bf y_{2}})\), and then the output is a linear combination as in the constant combination: \({\bf f}({\bf y_{1}},{\bf y_{2}})=\lambda{\bf y_{1}}+(1-\lambda){\bf y_{2}}\). The function \(g\) is parametrized by a small neural network.
7. **Full-vector**. Token-wise version of the previous combination, where a vector \(\boldsymbol{\lambda}=\mathbf{g}(\mathbf{y_{1}},{\bf y_{2}})\in\mathbb{R}^{|V|}\) is predicted , and the per-token combination is done as in _constant-vector_.

On one end of the spectrum, the _mean_ and _constant-scalar_ combinations have very low capacity, having zero and one learnable parameters, respectively. On the other end, the _full_ combinations can represent rich combination functions, taking advantage of the information in the full output distributions. The _entropy_ combinations are motivated by the fact that we expect output distribution entropies to be informative to the combination function; intuitively, knowing how certain each model is should be helpful when deciding which model to give more weight to. Additionally, token-wise versions of each method further increase the capacity of the combination function. This setup allows us to study how important combination function capacity is for the performance of the adapted model, as well as how this relates to the amount of data used for learning the combination.

These combination functions can be learned without any access to the LMs' weights or internal states, and require only a forward pass through the small set used to train the combination network. We refer to the process of training the small network that parametrizes the combination function as fitting the combination function. Once the combination function is fit, the combined model outputs valid probability distributions over continuations, and can be used as a regular LM.

## 3 Experimental setup

### Models

We use OPT-30B and OPT-1.3B Zhang et al. (2022) as our large black-box and small white-box LMs, respectively. Our choice of OPT is motivated by the following reasons 

1. Both the small and large models must share the tokenizer in our current formulation.1 Since we want to train the small domain experts by fine-tuning an existing model, we need a model family that has both large and small models sharing the same tokenizer, which OPT provides. Footnote 1: Although it is possible to either adapt LMs to a new vocabulary or extend our approach to work with different tokenizers, that would add a new dimension to our experiments, separate from the core research question that we want to study.
2. To rigorously determine what constitutes a new domain for the models, we need to know what data they were trained on, which is not public for most proprietary models behind APIs.2 Footnote 2: While this is not a problem for applying our method in practice, it does rule out proprietary black-box models for scientific study.

We report results for the large model and the small fine-tuned model, which can be taken as the **baselines**, as well as their combination through our proposed method. For the parametrization of the combination functions, we use small neural networks, with the following architectures:

* **Constant-scalar:** A single neuron with no input, passed through a sigmoid to force it into \((0,1)\).
* **Constant-vector:** A vector of neurons with no input, passed through a sigmoid to force it into \((0,1)^{|V|}\).
* **Entropy-scalar:** Input layer is two-dimensional, consisting of both entropies, followed by 1D BatchNorm, two hidden layers of dimension 512, with ReLU nonlinearities, and a one-dimensional output layer with a sigmoid non-linearity, to force it into \((0,1)\).
* **Entropy-vector:** Input layer is same as for _entropy-scalar_, followed by 1D BatchNorm, two hidden layers of dimension 512, with ReLU non-linearities, and a \(|V|\)-dimensional output layer with a sigmoid non-linearity, to force it into \((0,1)^{|V|}\).
* **Full-scalar:** Input layer is \(2|V|\)-dimensional, consisting on the concatenated output distributions for each model, followed by 1D BatchNorm, two hidden layers of dimension 512, with ReLU non-linearities, and a one-dimensional output layer with a sigmoid non-linearity, to force it into \((0,1)\).
* **Full-vector:** Input layer same as for _full-scalar_, \(2|V|\)-dimensional, followed by 1D BatchNorm, two hidden layers of dimension 512, with ReLU non-linearities, and a \(|V|\)-dimensional output layer with a sigmoid non-linearity, to force it into \((0,1)^{|V|}\).

We train all combination networks using the Adam optimizer and a learning rate of \(2\mathrm{e}{-3}\) with the exception of _constant-vector_, for which we use a learning rate of \(1\mathrm{e}{-2}\), and a batch size of \(1024\). We run optimization for a single epoch in all cases, as we found this to be enough in preliminary experiments.

Note that the **mean** combination function has no learnable parameters. Finally, we also report **maxprob oracle** results as the upper-bound, which simulates a perfect combination function that gives 100% of the weight to the best model for any given token.

### Evaluation

For evaluation, we adapt our model for three new domains and a downstream task. The three new **domains** are defined by three datasets:

* The **Amazon Reviews** dataset [9, 8], consisting of a large collection of reviews and ratings entered by users on the Amazon website.
* The **Enron Emails** dataset [76], consisting of internal emails made public by the Federal Energy Regulatory Commission during the investigation of the Enron company.
* The **FreeLaw** subset of The Pile [10], consisting of a large collection of court opinions from federal and state courts.

For each dataset, we extract two sets of 1000 1024-token sequences, which we call _train-fit_ and _test_, respectively, and use the rest for the train set. The _train-fit_ sets are used to fit the combination functions, and we report perplexity on the _test_ sets for evaluation. We use the train set to fine-tune OPT-1.3B using the Adam optimizer, a \(1024\)-token sequence length, a fixed learning rate of \(4\mathrm{e}{-4}\), and a batch size of \(1024*90=92160\) tokens. In the 

case of Enron Emails we fine-tuned for a single epoch, corresponding to 3000k steps. For Amazon Reviews and FreeLaw we performed 30k steps, and had to stop well before reaching the first epoch, due to compute constraints. Unless otherwise stated, the full _train-fit_ sets are used to fit the combination functions.

For **downstream evaluation**, we experiment on English-Czech and English-German MT using the WMT21 dataset Barrault et al. (2020). We create a training set by verbalizing all the sentence pairs and concatenating them into a single corpus. Details of the verbalization templates can be found in Appendix A. We create a validation set following the same procedure on the WMT20 test set Akhbardeh et al. (2021), and extract a _train-fit_ set of 1000 1024-token sequences for fitting the combination functions, as we do in domain adaptation. Following the recommended practice in the area Freitag et al. (2022), we use BLEURT Sellam et al. (2020) on the WMT21 test set as our evaluation metric, and report additional results with BLEU Papineni et al. (2002) in Appendix B. We used 3-shot prompting for evaluation, as longer sequence lenghts resulted in OOM issues in our hardware. We use the training set to fine-tune OPT-1.3B using the exact same settings described above. We train for 2k steps, corresponding to a total of around 2.5 million parallel sentences.3

Footnote 3: Although the full combined training set for English-German and English-Czech is bigger than 2.5M parallel sentences, we were interested in simulating the setting where limited translation data is available. Given enough parallel data, one can train a strong translation system from scratch, without having to adapt a generalist model.

## 4 Results

We next present our main results on domain adaptation (SS4.1) and MT (SS4.2).

### Domain adaptation

We report domain adaptation results in Table 1. We observe that the combined models are able to achieve substantially lower perplexities than either of the individual models. Even simple averaging works remarkably well, improving over both baselines in Amazon Reviews and FreeLaw, but learned combinations perform best. The _entropy-scalar_ combination works best across the board, achieving a relative improvement in perplexity of \(9\%\) in Amazon Reviews, \(2\%\) in Enron Emails and \(4\%\) in FreeLaw over the best single model. This supports our hypothesis that output distribution entropies are informative to the combination function. However, higher capacity combination functions like _full-scalar_ work better in some cases, as is the case for Amazon Reviews.

Overall, our results show that the adapted model is able to leverage domain-specific knowledge in the small model, as well as the knowledge in the large generalist model, in order to improve over either of them. However, there is still a significant gap between the adapted models and the max-prob oracle, suggesting gains could still be made through a better combination function.

### Machine translation

Table 2 reports downstream results on MT. As for domain adaptation, all the learned combinations outperform both the small fine-tuned model and the large black-box model. This shows that our approach can work for adaptation to downstream tasks, and is not limited to domain adaptation. Once again, the simple _mean_ combination per

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **Amazon** & **Enron** & **Freelaw** \\ \hline
**OPT-1.3B FT** & 17.00 & 3.30 & 4.98 \\
**OPT-30B** & 20.37 & 5.53 & 6.50 \\ \hline
**Mean** & 15.88 & 3.47 & 4.92 \\
**Constant-scalar** & 15.80 & 3.27 & 4.84 \\
**Constant-vector** & 15.62 & 3.31 & 4.82 \\
**Entropy-scalar** & 15.50 & **3.24** & 4.78 \\
**Entropy-vector** & 15.41 & 3.24 & **4.76** \\
**Full-scalar** & **15.36** & 3.27 & 4.79 \\
**Full-vector** & 15.43 & 3.27 & 4.79 \\ \hline
**Max-prob (oracle)** & 12.59 & 2.89 & 4.12 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Domain adaptation results (perplexity). By combining a small domain expert and large general model, we achieve better perplexities than either of the original models.**

\begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & **en-de** & **en-cs** & **de-en** & **es-cn** & **avg** \\ \hline
**OPT-1.3B FT** & 52.36 & 32.66 & 67.95 & 60.47 & 53.36 \\
**OPT-30B** & 54.77 & 29.21 & 68.45 & 61.83 & 53.56 \\ \hline
**Mean** & 57.62 & 35.34 & **69.84** & 63.62 & 56.61 \\
**Constant-scalar** & 57.73 & 35.08 & 69.70 & 63.70 & 56.56 \\
**Constant-vector** & 57.71 & 34.69 & 69.60 & 63.64 & 56.41 \\
**Entropy-scalar** & 57.87 & 35.18 & 69.59 & 63.88 & 56.63 \\
**Entropy-vector** & **58.11** & **35.41** & 69.44 & **64.06** & **56.76** \\
**Full-scalar** & 57.98 & 35.06 & 69.57 & 63.59 & 56.55 \\
**Full-vectors** & 58.02 & 35.31 & 69.66 & 63.37 & 56.59 \\ \hline \hline \end{tabular}
\end{table}
Table 2: **MT results (BLEURT). The learned combinations significantly outperforms both models in a downstream task, often by a substantial margin.** 

forms very well, obtaining the second best results after _entropy-vector_. In any case, the combination function has a relatively small impact in MT, and even the worst performing approach brings large improvements over the baseline.

## 5 Analysis

In this section, we study the following aspects of our approach:

* How dependent is the quality of the resulting model on the amount of data used to fit the combination function?
* How dependent is the quality of the resulting model on the amount of data used to fine-tune the small LM?
* How much is general language modeling performance degraded by domain adaptation?
* Is the learned combination interpretable?

### Effect of the amount of data for fitting

In order to study how the performance of the adapted model varies with respect to the amount of data used to fit the combination function, we fit each combination function three times, on a varying number of tokens. We report results for the Amazon Reviews dataset in Table 3, and additional results in Appendix B.

As expected, performance improves with more training data. However, the difference varies across methods. For example, _constant-scalar_, which has a very low capacity, performs equally well when trained on 100 or 1000 sequences. On the other hand, the _full-scalar_ and _full-vector_ functions, that take the entire probability distribution as input, benefit from more training sequences. The _entropy-scalar_ combination strikes a good balance, performing well across the board, and retaining strong performance when fit on as little as 100 sequences.

### Effect of fine-tuning steps

Figure 2 shows the performance of the adapted models, when fine-tuning the small model for a varying number of sequences. At step 0 (i.e., before fine-tuning begins), the small LM corresponds to vanilla OPT-1.3B, which performs considerably worse than OPT-30B on Amazon Reviews. Even in that case, _entropy-scalar_ performs on par with OPT-30B, while _mean_ is slightly worse. This shows that learnable combination functions are able to avoid any loss in performance when combining with a poor domain expert. At the same time, it is also remarkable that the combination of vanilla OPT-1.3B and OPT-30B is not better than OPT-30B alone. This can also be seen in Table 4, which compares using vanilla OPT-1.3B and fine-tuned OPT-1.3B as the small model. This shows that our reported improvements do not solely come from an ensembling effect, and our proposed approach effectively combines the power of the large LM and the domain expertise of the small LM.

In addition, we observe that our combined LM substantially improves upon each individual LM as early as step \(3000\). In fact, the gap between the small fine-tuned LM and our combined LM slightly narrows as training progresses. For instance, for _entropy-scalar_, the gap between the small LM and the combined LM is \(2.18\) perplexity points at step \(3000\) (\(12\%\) relative improvement), which goes down to \(1.5\) for the fully fine-tuned model (\(9\%\) relative improvement). This is intuitive, as the more data is available in the target domain, the less useful will be integrating the general knowledge in the large LM.

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **100** & **500** & **1000** \\ \hline
**OPT-1.3B FT** & 17.00 & 17.00 & 17.00 \\
**OPT-30B** & 20.37 & 20.37 & 20.37 \\ \hline
**Mean** & 15.88 & 15.88 & 15.88 \\
**Constant-scalar** & 15.80 & 15.80 & 15.80 \\
**Constant-vector** & 15.80 & 15.66 & 15.62 \\
**Entropy-scalar** & 15.51 & 15.50 & 15.50 \\
**Entropy-vector** & 15.52 & 15.45 & 15.41 \\
**Full-scalar** & 15.63 & 15.40 & 15.36 \\
**Full-vector** & 15.71 & 15.49 & 15.43 \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Perplexity on Amazon Reviews**, using a different number of sequences to fit the combination function. Perplexity improves with the number of sequences, but results are already strong with only 100 sequences.

Figure 2: **Perplexity on Amazon Reviews**, varying the amount of fine-tuning steps.

 



### Effect on general language modeling

We are also interested in measuring how well the adapted models retain the general language modeling ability of the original large model. We use perplexity on **The Pile**Gao et al. (2021) as a proxy of general language modeling performance, as it is a large collection of many datasets from different domains, often used to train generalist LMs Black et al. (2022); Biderman et al. (2023). To this end, we also extract random _train-fit_ and _test_ subsets from The Pile. While some subsets of The Pile are also present in the training data for OPT, we do not measure performance on The Pile as a benchmark for model quality, and are only interested in it as a proxy for degradation in general language modeling ability of the adapted models.

We compare fitting the combination function on the target domain _train-fit_, as done throughout the paper, as well as on the combination of the target domain and The Pile _train-fit_ sets. Table 5 reports results for Amazon Reviews, and full results can be found in Appendix B.

When fitting the combination function on Amazon Reviews, we observe a significant degradation on The Pile. However, different combination methods behave differently in this regard. For example, _entropy-scalar_ and _full-vector_ perform similarly in Amazon Reviews (\(15.50\) vs \(15.43\)), but the former performs much better on The Pile (\(7.35\) vs \(10.07\)). It is also remarkable that The Pile perplexity of the combined model remains far better than the small fine-tuned LM (e.g. \(7.35\) for _entropy-scalar_ vs \(19.78\) for the small LM), while also performing better in-domain.

When fitting the combination function on the mixin set, we observe that performance on The Pile is almost entirely preserved, at the expense of a slight degradation on Amazon Reviews. For example, for _full-scalar_, the combination fit on the mixin set achieves a perplexity of \(15.45\) on Amazon Reviews and \(6.85\) on The Pile, both within \(0.1\) of the best results for each dataset.

Overall, these results show that a large model can be adapted to a particular domain while mitigating degradation in the general domain by mixing in-domain and general text to train the combination function. Additionally, we find that different combination methods exhibit different behavior when it comes to general performance degradation, even when they exhibit similar in-domain performance.

### Is the model combination interpretable?

We next analyze whether the weights given to each model by the combination function are interpretable. Figure 3 illustrates this over a random sample from Amazon Reviews: we show which tokens are better predicted by each model, along with which model is given a higher weight for each token. Although we do not identify a clear pattern for which tokens are better predicted by each model, we do observe that the coloring in the top and the bottom visualizations match quite closely. This means that the learned combination function is quite good at predicting when each model should be given a higher weight.4

Footnote 4: A perfect combination function (corresponding to the max-prob oracle in Table 1) would always give 100% of the weight to the best model for any given token, and both images would match up perfectly.

In order to quantitatively analyze this, we measure the Spearman correlation between the weight given by the combination function, and the actual difference in log probabilities for each token. Re

\begin{table}
\begin{tabular}{l c c} \hline \hline  & **Orig** & **FT** \\ \hline
**OPT-1.3B** & 26.03 & 17.00 \\
**OPT-30B** & 20.37 & 20.37 \\ \hline
**Mean** & 21.12 & 15.88 \\
**Constant-scalar** & 20.28 & 15.80 \\
**Constant-vector** & 20.55 & 15.62 \\
**Entropy-scalar** & 20.37 & 15.51 \\
**Entropy-vector** & 20.30 & 15.44 \\
**Full-scalar** & 20.26 & 15.41 \\
**Full-vector** & 20.30 & 15.48 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Perplexity on Amazon Reviews**, using either original OPT-1.3B or fine-tuned OPT-1.3B as the small LM. The combination methods barely improve upon OPT-30B when using the former, showing that our approach does not only work due to an ensembling effect.

\begin{table}
\begin{tabular}{l l r r r} \hline \hline  & \multicolumn{2}{c}{**Amazon-fit**} & \multicolumn{2}{c}{**Mixin-fit**} \\ \cline{2-5}  & **Amazon** & **Pile** & **Amazon** & **Pile** \\ \hline
**OPT-1.3B FT** & 17.00 & 19.78 & 17.00 & 19.78 \\
**OPT-30B** & 20.37 & 6.82 & 20.37 & 6.82 \\ \hline
**Mean** & 15.88 & 7.72 & 15.88 & 7.72 \\
**Constant-scalar** & 15.80 & 8.35 & 16.52 & 7.08 \\
**Constant-vector** & 15.62 & 8.38 & 15.89 & 7.18 \\
**Entropy-scalar** & 15.50 & 7.35 & 15.80 & 6.94 \\
**Entropy-vector** & 15.41 & 8.53 & 15.61 & 6.92 \\
**Full-scalar** & 15.36 & 9.31 & 15.45 & 6.85 \\
**Full-vector** & 15.43 & 10.07 & 15.48 & 6.91 \\ \hline \hline \end{tabular}
\end{table}
Table 5: **Perplexity on Amazon Reviews and The Pile, using either the former to fit the combination function (amazon-fit), or the concatenation of the two (mixin-fit).



sults are shown in Table 6. We limit our analysis to _entropy-scalar_ and _full-scalar_, as they are the only ones that output a single combination factor that depends on the input. We observe significant correlations for all datasets, with _entropy-scalar_ achieving better correlation than _full-scalar_, especially on The Pile. This is consistent with the results in Table 5, where _full-scalar_ suffers a bigger performance loss on The Pile. Somewhat surprisingly, correlation for _entropy-scalar_ is better on The Pile than on the in-domain dataset, even though the combination function is fit on the in-domain _train-fit_. One possible explanation is that The Pile better represents the training distribution of the large LM, making it better calibrated on it, which makes it easier for _entropy-scalar_ to make predictions.

## 6 Related work

We present related work on domain adaptation of LMs (SS6.1), and language modeling through domain experts (SS6.2).

### Domain adaptation of LMs

Domain adaptation of LMs is an extensively studied line of research. Traditional approaches include fine-tuning the model on domain-specific corpora, Devlin et al. (2019); Liu et al. (2019); Gururangan et al. (2020), data selection on the original general corpus Aharoni and Goldberg (2020); van der Wees et al. (2017), and adapting or extending the tokenizer to achieve better performance on the target domain Sachidananda et al. (2021).

Although effective, these full fine-tuning techniques are often infeasible at scale due to the excessive compute required. Some approaches aim to reduce the resources required to fine-tune large models through parameter-efficient adaptation techniques, such as adapters Houlsby et al. (2019), soft-prompt tuning Liu et al. (2022), or low-rank adaptation Hu et al. (2022). However, all of these techniques require white-box access to the original model and full backward passes, making them incompatible with black-box models.

In contrast, discrete prompt tuning approaches allow for treating the large model as a black-box Shin et al. (2020); Sun et al. (2022); Zhang et al. (2023); Cheng et al. (2023). However, these approaches have only been proven in the limited setting of retrieving zero- or few-shot prompts that improve performance in a set of NLP tasks that the base black-box is already capable of performing, as opposed to a general method of black-box model adaptation.

Concurrent to our work, Huang et al. (2023) propose leveraging KNN retrieval from a data-store to augment an existing black-box LM. However, they only experiment with small GPT2 models as the black-box, and the adaptation depends on finding an adequate datastore, limiting application to downstream tasks such as MT.

### Domain experts for language modeling

Another line of research explores language modeling through a combination of separate domain experts. Li et al. (2022) achieve better performance than compute-matched single transformer models and highly parallel pre-training, by training independent domain experts, and combining them at the parameter level at inference time. Gururangan et al. (2023) extend this approach to automatically discovered domain clusters. Other approaches replace components of the transformer network with independent domain-dependent modules, as is the case of Gururangan et al. (2022) for metadata-defined domains, or Pfeiffer et al. (2022) for per-language modules. All of these are pre-training approaches and seek to train better or more efficient LMs, but cannot leverage existing powerful black-box models. Our work, in contrast, seeks to adapt an existing powerful black-box through leveraging a much smaller domain expert.

## 7 Conclusions

In this work, we present a method for adapting black-box LMs to new domains and tasks, requiring access to probability-level outputs only. We first fine-tune a small domain expert white-box LM

\begin{table}
\begin{tabular}{l l c c} \hline \hline  & & **Domain** & **Pile** \\ \hline \multirow{2}{*}{**Amazon**} & **Entropy-scalar** & 0.59 & 0.71 \\  & **Full-scalar** & 0.44 & 0.32 \\ \hline \multirow{2}{*}{**Freclaw**} & **Entropy-scalar** & 0.49 & 0.75 \\  & **Full-scalar** & 0.33 & 0.32 \\ \hline \multirow{2}{*}{**Enron**} & **Entropy-scalar** & 0.54 & 0.75 \\  & **Full-scalar** & 0.25 & 0.30 \\ \hline \hline \end{tabular}
\end{table}
Table 6: **Spearman correlation between the logprobability difference of the LMs and the weight given by combination function. Larger values mean that the learned combination is closer to the ideal oracle weighting. Rows represent adapted models on different domains and combination functions, fit on the in-domain _train-fit_.** 

on a domain-specific corpus, and then combine it with the large black-box through a combination function learned on a small fitting set, yielding an adapted LM. Additionally, our method requires only access to probability level outputs, and thus allows to leverage powerful models optimized for inference or behind APIs, without the need for white-box access to the weights. We experiment on several datasets and a downstream task, as well as performing extensive analysis of our method, reaching several conclusions:

* By combining a small domain expert and a large black-box model, the combined model outperforms either of the original ones in all cases, by as much as 9% perplexity for domain adaptation, and 6% BLEURT for MT, showing the effectiveness of our approach.
* While higher capacity combination functions can perform better when more data is used to learn the combination, lower capacity combination methods remain competitive, and perform better when learned on little data. In particular, the entropy based combinations, _entropy-scalar_ and _entropy-vector_, perform well across the board, even when fit on as little as 100 sequences.
* Our approach is effective even when little is data available to fine-tune the domain expert. In fact, the gains are biggest in this scenario, as the advantage of leveraging a good black-box generalist decreases when a big incoming corpus is available.
* While adaptation to new domains incurs a loss of general language modeling ability, this varies per combination method, and seems to be largely mitigated by augmenting the small set on which the combination function is fit.

While our approach is effective, observed performance is still not close to the max prob oracle, which represents the ideal system where 100% of the weight is given to the best model at each time step. In future work, we would like to investigate the reasons for this gap, and potential ways of addressing it.

### Limitations

While our method requires no access to the black-box model's weights and intermediate activations, it does assume access to the full output probability distribution, which might not be available for some models served through APIs. The OpenAI API, for example, only returns the probabilities for the top 5 tokens. This is not an issue for the Constant combinations, and the Entropy methods could potentially

Figure 3: **Difference between the small fine-tuned LM and the large black-box LM according to log-probability (a) and predicted weight (b). The closer the two match, the better the learned combination is at predicting which model will be “right” for a given token. The text sample was chosen randomly from the Amazon Reviews testset.**



also be adapted, by estimating the entropy from top K probabilities.

Additionally, even though we don't fine-tune the black-box at all, our method does require a forward pass of the large black-box through a fitting set, which might potentially be costly if done through APIs, depending on pricing and the size of the fitting set.

## Acknowledgements

Aitor and Eneko were partially supported by the Basque Government (IXA excellence research group IT-1805-22; IKER-GAITU project). Aitor was supported by a doctoral grant from the Spanish MECD.

## References

* Aharoni and Goldberg (2020) Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7747-7763, Online. Association for Computational Linguistics.
* Akhbardeh et al. (2020) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondrej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina Espana-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyrie, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Louirle, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In _Proceedings of the Sixth Conference on Machine Translation_, pages 1-88, Online. Association for Computational Linguistics.
* Andrew and Gao (2007) Galen Andrew and Jianfeng Gao. 2007. Scalable training of \(L_{1}\)-regularized log-linear models. In _Proceedings of the 24th International Conference on Machine Learning_, pages 33-40.
* Barrault et al. (2029) Loic Barrault, Magdalena Biesialska, Ondrej Bojar, Marta R. Costa-jussa, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philippo Kochn, Chi-kiu Lo, Nikola Ljubesic, Christof Monz, Makoto Morishita, Masasaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. 2020. Findings of the 2020 conference on machine translation (WMT20). In _Proceedings of the Fifth Conference on Machine Translation_, pages 1-55, Online. Association for Computational Linguistics.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Afhah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Auiya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling.
* Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shiranshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. Gpt-neox-20b: An open-source autoregressive language model.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pfrafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Grectchen Krueger, Tom Henighan, Rewon Child, Aditya Ramos, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
* Cheng et al. (2023) Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Denvy Deng, and Qi Zhang. 2023. Uprise: Universal prompt retrieval for improving zero-shot evaluation.
* Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyaschchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodokumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselim Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Heyontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thammalamayan Sankaranarayana Pilali, Marie Pellat, Aitor Lewkowycz, Eric Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuehi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Cathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding 

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and Andre F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
* Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021. The pile: An 800gb dataset of diverse text for language modeling. _CoRR_, abs/2101.00027.
* Gururangan et al. (2022) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix layers: Disentangling domains for modular language modeling. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5557-5576, Seattle, United States. Association for Computational Linguistics.
* Gururangan et al. (2023) Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2023. Scaling expert language models with unsupervised domain discovery.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8342-8360, Online. Association for Computational Linguistics.
* He and McAuley (2016) Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _Proceedings of the 25th International Conference on World Wide Web_, WWW '16, page 507-517, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp.
* Hu et al. (2022) Edward J Hu, velong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In _International Conference on Learning Representations_.
* Huang et al. (2023) Yangsibo Huang, Daogao Liu, Zexuan Zhong, Weijia Shi, and Yin Tat Lee. 2023. \(k\)nn-adapter: Efficient domain adaptation for black-box language models.
* Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004. The enron corpus: A new dataset for email classification research. In _Machine Learning: ECML 2004_, pages 217-226, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Li et al. (2022) Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. 2022. Branch-train-merge: Embarrassingly parallel training of expert language models.
* Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 61-68, Dublin, Ireland. Association for Computational Linguistics.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
* McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '15, page 43-52, New York, NY, USA. Association for Computing Machinery.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Pfeiffer et al. (2022) Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3479-3495, Seattle, United States. Association for Computational Linguistics.
* Sachidananda et al. (2021) Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In _Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing_, pages 155-165, Virtual. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892, Online. Association for Computational Linguistics.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In _Proceedings of the 

2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4222-4235, Online. Association for Computational Linguistics.
* Sun et al. (2022) Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xiepeng Qiu. 2022. Black-box tuning for language-model-as-a-service.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
* van der Wees et al. (2017) Marlies van der Wees, Arianna Bisazza, and Christof Monz. 2017. Dynamic data selection for neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1400-1410, Copenhagen, Denmark. Association for Computational Linguistics.
* Zhang et al. (2023) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihalov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koru, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. Opt: Open pre-trained transformer language models.

## Appendix A MT verbalizations

We verbalize the MT task by first adding a prompt describing the task, and then adding several translation examples. We chunk the translation examples in blocks of 5, that is, adding 5 translation examples per verbalization. We use two different task descriptipons, shown in Table 7, and alternate evenly between both variations to create the verbalized training corpus. For inference, we use verbalization #1 and draw 3 random translation pairs from the WMT21 development set to construct a 3-shot prompt. We draw the random translation pairs once, and keep the 3-shot prompt fixed for all models.

## Appendix B Full results

Full results for all combination methods, dataset sizes, and evaluation settings are shown in Table 8. Table 9 reports additional MT results using BLEU.

 



\begin{table}
\begin{tabular}{l|c} Verbalization \#1 & Verbalization \#2 \\ \hline Translate the following sentences from SL1 to §L2: & Given a sentence in §L1, translate it to §L2: \\
§L1: §S51 & §L1: SS1 \\ §L2: ST1 & §L2: ST1 \\ §L1: §S52 & §L1: SS2 \\ §L2: ST2 & §L2: ST2 \\ §L1: §SS3 & §L1: §SS3 \\ §L2: ST3 & §L2: ST3 \\ §L1: §SS4 & §L1: SS4 \\ §L2: ST4 & §L2: ST4 \\ §L1: §SS5 & §L1: §SS5 \\ §L2: ST5 & §L2: ST5 \\ \end{tabular}
\end{table}
Table 7: Both verbalizations used for MT. §L1 §L2 represent the source and target languages, and §Si and ¶1 represent the source and target sentences for the \(i\)th pair in the verbalization.





\begin{table}
\begin{tabular}{l l



\begin{table}
\begin{tabular}{l c r r r r r r r r r} \hline  & \multicolumn{3}{c}{**en-de**} & \multicolumn{2}{c}{**en-cs**} & \multicolumn{3}{c}{**de-en**} & \multicolumn{3}{c}{**cs-en**} & \multicolumn{3}{c}{**avg**} \\  & **BLEURT** & **BLEU** & **BLEURT** & **BLEU** & **BLEUrt** & **BLEURT** & **BLEU** & **BLEURT** & **BLEU** & **BLEURT** & **BLEU** \\ \hline
**Mean** & 57.62 & 14.39 & 35.34 & 5.76 & 69.84 & 26.72 & 63.62 & 21.57 & 56.61 & 17.11 \\
**Constant-scalar** & 57.73 & 13.76 & 35.08 & 5.66 & 69.70 & 26.68 & 63.75 & 21.32 & 56.56 & 16.86 \\
**Constant-vector** & 57.71 & 13.88 & 34.69 & 5.28 & 69.60 & 26.65 & 63.64 & 21.41 & 56.41 & 16.80 \\
**Entropy-scalar** & 57.87 & 13.76 & 35.18 & 5.60 & 69.59 & 26.30 & 63.88 & 21.31 & 56.63 & 16.74 \\
**Entropy-vector** & 58.11 & 14.25 & 35.41 & 5.64 & 69.44 & 26.73 & 64.06 & 21.47 & 56.76 & 17.02 \\
**Full-scalar** & 57.98 & 14.22 & 35.06 & 5.52 & 69.57 & 25.86 & 63.59 & 20.50 & 56.55 & 16.52 \\
**Full-vector** & 58.02 & 14.11 & 35.31 & 5.19 & 69.66 & 26.13 & 63.37 & 20.96 & 56.59 & 16.60 \\ \hline
**OPT-1.3B FT** & 52.36 & 15.05 & 32.66 & 5.48 & 67.95 & 25.27 & 60.47 & 19.13 & 53.36 & 16.23 \\
**OPT-30B** & 54.77 & 9.64 & 29.21 & 3.13 & 68.45 & 24.08 & 61.83 & 18.49 & 53.56 & 13.84 \\ \hline \end{tabular}
\end{table}
Table 9: Full BLEU and BLEURIT results for MT.