Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Authors: Albert Q. Jiang
Alicja Ziarko
Bartosz Piotrowski
Wenda Li
Mateja Jamnik
Piotr Miłoś
Venue: NeurIPS 2024 poster (2024)
Keywords: text embedding
embedding models
scaling laws
Primary area: natural_language_processing
TL;DR: Providing a compute-optimal recipe for efficient contrastive fine-tuning of pretrained language models into embedding models.

Abstract: Text embeddings are essential for tasks such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pretrained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and Low-Rank Adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively.

Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Albert Q. Jiang" University of Cambridge
Alicja Ziarko IDEAS NCBR University of Warsaw IMPAN
Bartosz Piotrowski IDEAS NCBR
Wenda Li University of Edinburgh
Mateja Jamnik University of Cambridge
Piotr Mitoš IDEAS NCBR
University of Warsaw IMPAN, deepsense.ai
Abstract
Text embeddings are essential for many tasks, such as document retrieval, clus- tering, and semantic similarity assessment. In this paper, study how to con- trastively train text embedding models in a compute-optimal fashion, given a suite of pre-trained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can used by practitioners to make informed design choices for their embedding models. Specifi- cally, our findings suggest that full fine-tum fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively.
Introduction
Optimal Loss vs Computational Budget
10
Text embeddings are vector representations of text sequences [Devlin et al., 2019]. desired property of text embeddings is that their dis- tribution in the embedding space reflects the semantic relations of the embedded texts. This property benefits multiple practical tasks [Muen- nighoff et al. 2023b), such as document retrieval (where documents are ranked based on the dis- tance between their representations and that of query), document clustering, sentiment analysis, or textual similarity measurement.
2 10
Loss
Optimal frontier
Full fine tuning
Blas tuning
LoRA
Black freezing
10k
1017
1016
Budget (FLOP)
In the era large language models (LLMs) that are pre-trained on vast textual data, it is nat- ural idea to capitalise on the language repre- sentations learned by these models to achieve high-quality embedding models. A common and effective approach has been to initialise embedding model from a pre-trained LLM and fine-tune it with contrastive loss (Neelakantan ctt al. 2022 Izacard et al. 2022]. This phase is
Figure 1: The optimal loss achieved using four dif- ferent fine-tuning methods (full fine-tuning, only tuning the bias, low-rank adaptation, and freezing transformer blocks) at given budgets. The hori- zontal axis is the computational budget in floating point operations (FLOP) and the vertical axis is the contrastive loss. The X marks are datapoints and dotted lines are fitted linear trends for different methods. The solid black line is the "optimal fron- tier," " i.e., the optimal loss achievable with fixed budget and the best method.
'Equal contribution.
'Equal advising contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of crucial importance, as the quality of the embeddings extracted directly from a pre-trained LLM (e.g., by averaging hidden states of the last layer) not sufficient for most tasks.
However, the best performing modern LLMs are usually massive in terms of the parameter counts (e.g., 175B parameters for GPT3 [Brown et al., 2020] and 340B for PaL.22 [Anil et al. (023) )) and are therefore difficult to train with limited resources. This motivates practical question, that to the best of our knowledge has not been addressed systematically, and thus we address it in this paper:
what is the best embedding model one can train from backbone decoder-only LLM with fixed training compute budget?
To answer this, we performed an extensive empirical study. We started by identifying design choices one can make when fine-tuning language models into embedding models, including: model size, data volume, parameter-efficier Then, we performed grid search over the pre-defined design choices, and found the best performing configuration under each computational budget. Using these findings, we established scaling laws for contrastive fine-tuning, which enabled us build an algorithm that produces general recipe for efficient fine-tuning of a decoder-only LLM that obtains a high-quality embedding model.
fine-tuning technique, and the hyperparameters of the chosen technique.
Contribution We comprehensively experimented with contrastively fine-tuning language models into text embedding models. We analysed how the choice of model sizes, data quantities, and fine-tuning methods affect the performance in the the findings into an algorithm that, given a fixed computational budget, predicts the optimal network architecture, data quantity, and parameter-efficient fine-tuning hyperparameters. We open-source the code to train and evaluate our models at: https: /tt::://ttttbb com/ Stt////DDD//tiiteeeee Embeddings
training regime. We compiled
2 Related work
Embedding models Training neural networks represent text continuous vectors was popu- larised by word2vec [Mikolov et al. 2013). which produced semantically meaningful embeddings of words. BERT (Devlin al. [019 and the further contrastively trained SimCSE [Gao et al., 2021c
quickly established encoder-only transformers as the go-to architecture for embedding text.
More recently, decoder-only transformers have become increasingly more powerful and efficient at the same time [Brown et al. 2020, Touvron et al., 2023 Diang et al. 2023]. Building an embedding model based on them utilises the knowledge from their pretraining, and thus it is a natural step. Neelakantan et al. [2022] set a successful precedent in this paradigm by initialising embedder training with decoder-only GPT models.
Notably, the current open state-of-the-art (the top open model on the MTEB MMuennighoff et al.. 2023b leaderboard, as of 18 May 2024). SFR-Embedding-- -MMMaarrall MMeng et al. 2024]. is also fine-tuned from the decoder-only Mistral 7B [Tiang et al. 2023).
Benchmarks for embedding models Embedding models are versatile in their applications. There- fore, broad and diverse benchmark to provide a robust measure of their performance is called for. The first such benchmark was BEIR introduced by Thakur et al. 2021]. It has nine different information retrieval tasks (e.g., duplicate-question retrieval or crration-predicctonn Recently. Muennighoff et al.] [2023b] introduced MTEB (Massive Text Embedding Benchmark), which is substantially larger than BEIR and, in addition to information retrieval, has seven other types of tasks (e.g.. clustering, summarisation, and pair classification) using 58 datasets and covering 112 languages. The authors also provide leaderboard that currently contains more than 220 entries.
on 18 datasets.
Scaling laws Scaling laws predict the performance of models at larger scale (e.g., more parameters, more data) from experiments at smaller scales. Kaplan et al. (2020) and Hoffmann et al. [2022] used empirically fitted laws to predict the performance of neural language models at different model sizes and data scales, making possible to calculate the optimal configuration before training. Muennighoff et al. [202aa] inspected model training in data-constrained, multi-epoel regime. Frantar et al. [2023] looked at scaling laws for language models in the context of weight sparsity. /Zhang et al.) [2024] investigates the scaling laws for model fine- minettnning including parameter efficient methods:
LoRA (Hu et al. 2022) and prompt tuning [Lester et al. [021]. They show that the scaling laws and the best fine-tuning method are task dependent. Biderman et al. (2024) investigates the properties of fine-tuning models with LoRA in detail, showing that LoRA underperforms full fine-tuning when
fine-tuning the models on mathematical or code datasets.

---

The single most related investigation is a concurrent work Fang et al. (2024), where the scaling laws for encoder models for retrieval are investigated. There are significant differences in the settings we consider. Our focus is on investigating the process of fine-tuning decoder-only models for good quality embeddings. Moreover, our main goal is to find which strategy for achieving good embeddings is optimal in a budget-restricted settings, while taking into account the popularity of applying parameter efficient methods for fine-ture fine-tuning, like LoRA or partial model freezing. In spirit, our work is similar to AutoML [He et al., [2021b) which aims to automatically find the optimal ML architecture. However, our method goes beyond and targets the data and the training method as well.
Parameter-efficient fine-tuning Modern language models have a lot of parameters, and fine-tuning them using sassuumer-rrde hardware is difficult in general. Notwithstanding the recent turn to in-context learning, fine-tuning is still necessary for most upplications. A range of techniques have been developed recently, to make fine-tuning more efficient (Houlsby et al. 2019) Li and Liang 2021 et al., 2022 He et al., [2021a]. These approaches are all applicable to embedding models. Muennighoff [022] explored simple parameter-effiienn approach to repurpose GPT models into embedding models where only the bias tensors of the transformer model are updated (Zaken et al., 2022]. Sun et al. 2024] developed a parameter-efficient method designed specifically for fine-tuning embedding models. However, systematic study of what parameter-efficient methods are optimal under what scenarios has not been performed, which is the aim of our work.
3 Preliminaries
3.1 Scaling laws and compute-optimal models
The constraint optimisation problem that we aim to solve is the following: given a fine-tuning corpus and a family of pre-trained decoder-only language models of different sizes, minimise the contrastive loss subject to fixed computational budget.
Following Hoffmann al. [2022] Section 3.2], we perform a hyperparameter search at different computational budget levels to obtain IsoFLOP profiles of models. We then find the optimal models each level, fit a loss-size scaling law, and extrapolate it to unobserved FLOP budgets. We only focus on contrastive fine-tuning since state-of-the-arr text embedder models are all trained in such
fashion [Gao et al. 2021e Neelakantan et al. 2022 Wang al., 2023].
We refer to the models with the lowest contrastive loss achievable with fixed computational costs as compute-optimal models. We focus on optimising three components specifically: model size, data quantity, and fine-tuning method. For each fine-tuning method, we find the optimal model configuration at each FLOP level and establish the relationship between the computational budget and optimal loss for that method. Then, for any unobserved computational budget, we use the scaling laws predict the loss, and pick the method that gives the lowest loss and its corresponding configuration.
3.2 Extracting representations from transformers
We start with a standard decoder-only transformer model architecture, such as GPT architectures RRad Ford et al. 2019 Brown et al. 2020), or Pythia [Biderman et al.) 2003). Given sequence of tokens (8) ....... S,). we pass them through the transformer model and average the last layer representa- tions as the embedding vector emb(S) Σ" h; E RT, where h; is the last layer hidden state of the token 8;.
S
Some works apply the last token pooling approach [Neelakantan al., 2022] instead of using mean pooling described above. Here, however, we default to mean pooling as in (Wang et al.) 2022 Li
et al. 2023 Günther al.] 2023]. It is more popular method overall, and moreover, find it to yield better performance, which we demonstrate in Appendix E.3
3.3
Contrastive fine-tuning
We use the contrastive loss as in (Neelakantan et al. 2022 which has been widely used in fine-tuning pre-trained language models [Wang et al. 2022 2023 Günther et al. 2023).
For batch of n examples, that is, n pairs of texts (11,01).-. (In, Vn). we only treat examples (z,, y as positive matches, and all the pairs (æ,, yj), where i A as negatives. Concretely, we calculate the batch contrastive loss in two steps. First, we calculate the logits:
logits [i,j] sim(emb(zs), emb(y,)) exp(r),
3

---

where sim is the cosine similarity function between two vectors, and T E R is the temperature. The total loss is the sum of the cross entropy losses on both the row and the column directions
labels [o, 1, n 1]
1_r cross_entrop"" (1ogssgsttooyyloiisss labels)
1_c cross_entron (logits. T,, labels)
loss (1_r + 1_c) / 2
Hard negatives in contrastive learning In our contrastive learning setting, we use in-batch examples negatives. Some datasets additionally include tailored negative examples (hard negatives) for each positive datapoint, which can be incorporated into the contrastive loss to promote learning more precise representations. Moreover, there are works that focus on approaches for mining hard negatives which result in better training data in the context of specific downstream tasks [Xiong et al.
2021 Zhan et al. 202T].
However, recent works aiming at training powerful, general-purpose purpose embedding models that do rely on datasets with hard negatives often arrive at embeddings by having two distinct training phases: the ex- pensive "unsupervised" phase one involving vast data from the internet, and then a "supervised" phase two, often targeted towards specific downstream task, where training data of lower quantity but con- taining high-quality hard negatives - is used [Li et al. 2023 Wang et al. [022 Xiao et al. 2003).. Since the first phase is more expensive, it will benefit more from scaling laws such as the ones derive.
Moreover, the usefulness of hard negatives highly depends on their quality, which may vary wildly between datasets. Therefore, conclusions reached with hard negatives are more closely tied to the datasets the models are trained on. Since we develop scaling laws agnostic to the dataset, we abstain from using hard negatives in our experiments.
3.4 Fine-tuning methods
The most basic and straightforward method of fine-tuning is full fine-tuning, where all the weights of the model are updated under the contrastive objective.
We also study parameter-efficien fine-tuning (PEFT) methods, which are popular, especially in academic settings, because they have lower memory requirements than full fine-tuning. Since the PEFT methods update fewer parameters in the network than there are in total, the relationship between their computational cost, model size, and data quantity is different from full fine-tuning. Hence, we individually study the IsoFLOP profiles for four fine-tuning methods, namely, full fine-tuning, block freezing. bias-only tuning, and Low-Rant Adaptation (LoRA). Each of the last three non-standard methods is briefly characterised below.
Block freezing In this approach, we freeze the LLM eoben-emb layer as well as the first transformer blocks so that their parameters stay fixed during fine-tuning, Only the latter part of the model is back- paooppropagated through and updates its parameters. To process the same amount of data, this is more computationally efficient than full fine-tuning. By varying the number of frozen blocks, one trades-off between the computational efficiency and flexibility of the model.
Low-rank adaptation (LoRA) This method was introduced by Hu et al. [2022) to significantly reduce the number of trainable parameters of a neural network while maintaining high performance. LoRA introduces a small number of weights into the model and only trains those. It can be applied to a subset of dense layers of a model. works by parametrising each dense layer W from a subset of linear layers of a model by two low-rank matrices A and B, and using W + AB instead of W, while training only A and B.
Bias-only tuning In this method, only bias parameters are fine-tuned, and the rest of the model remains frozen. This approach was proposed by Zaken al. [2022] parameter-efficien fine- tuning method for BERT encoders, and recently applied by Muennighoff [0022 to contrastively fine-tune GPT models.
3.5 Calculating computational cost
We denote the number of non-token-embedeee parameters used in the forward pass to process each to- ken as Ne, the number of www-ttkeeeeeeddiiie parameters in backward- wwoord-propagttionnnn as NB, the num-
We adopt the PyTorch notation here, where the cross entropy loss takes two arguments: the logits of the probability distribution, and the indices of the true categories.

---

ber of non-token- bonken-emeedding parameters updated during backward-pree propagation as Nu, the total num- ber of tokens processed as D. and the computational cost as floating point operations (FLOP). Fol- lowing the calculation by Kaplan et al. 2020). we derive the relationship between the variables to be
C= D + 2N D + 2Ng D.
In case of full fine-tuning, every parameter trainable, so Ny NB = Nu, and therefore
C = D.
This is the same as the formula for the computational cost used by Kaplan et al. 2020 for pre-training.
4 Experiments
We first specify the relevant details of our experimental setup (Section 4.1 the results of our experiments where we contrastively train a grid of models of different sizes, using different computational budgets, and apply different compute-optimal fine-tuning methods with varying hyperparameters (Section 4.2 Based on the collected data, we fit a mathematical formula describing the observed scaling laws (Section 4.3). Finally, we provide general observations and recommendations for efficient contrastive fine-tuning (Section 4.6). In these experiments we investigate the compute-constraing setup, but we present limited results on the data-constrained
Next, we present
setup in Appendix D
4.1 Experimental setup
We use eight decoder-only models from the Pythia suite [Biderman al. 2023 which have sizes 14M, 31M, 70M, 160M, 410M, IB, 1.4B, and 2.8B of parameters. All the models have been pre- trained on the Pile dataset [Gao et al. 2021a for 300 billion tokens. We consider six computational
budgets equally spaced on the log-scale: 1.5e15, 15, 6e15, 2.4e16, 6, 9.6e16, 3.8e17, and 1.5e18 FL.OP.
We fine-tune our models on the English partition of the BAAI BGE dataset [Xiao et al., 2023), which contains 200 million semantically related (query, value) pairs from various internet sources such as Wikipedia and Stack Exchange. Note that since the BAAI BGE dataset is massive, for all our experiments we fine-tune for less than one epoch on this corpus, which allows us to avoid the effect of diminishing returns subsequent training epochs Muunnighooff et al. 2023a].
We use the AdamW optimiser [Loshchilov and Hutter, 2019 and a cosine learning rate scheduler during training. The learning rate first goes through linear warm-up phase of 1/10 of the total steps to a peak that is 1/10 of the pre-training peak learning rate, that is. to learning rates between peak 1.2 x 10-h and 6 x 10-5 Then, it decays in cosine schedule to 1 1/10 of the maximum at the end of training. We employ gradient checkpointing (Chen et al.) 2016) and GradCache [Gao al. 202Th to limit the peak GPU RAM usage during training. We use the Transformers [Wolf et al. 2020 library for the training of the models and use the Accelerate [Gugger et al., [022] library to facilitate multi-GUU training. The batch size in our main line of experiments is 1024 sequences and the context length is 75 tokens. We give the full list of training hyperparameters in Appendix B
We also perform a less extensive grid of experiments to confirm that our findings are robust to change in hyperparameters. We find that our scaling law formula fitted to losses from models with batch size 1024 and context length of 75 tokens describes the loss for models trained with batch size 2048 and context length of 512 tokens very well. The detailed description and plots are shown in Appendix G
In addition to controlling the final training contrastive loss achieved by the models, we also measure downstream performance by evaluating the models mark MMuennighoff et al. [2023b]. The benchmark defines eight categories of tasks in total (e.g.. retrieval, semantic text similarity). We select one task from each category by determining which ones are the most correlated with the performance for the whole category (see Appendix B for details).
representative subset of the MTEB bench-
4.2 Experimental results for different methods
Full fine-tuning With full fine-tuning, the numbers of forward and backward parameters are the same, and the computational cost is straightforwardly C 6NN D. In Figure 2a] we present the final contrastive loss vs. the number of parameters in the network on log-log scale.
'The data collected in our experiments and their interactive visualisations are available in this Colab notebook:
5

---

Full fine-tuning loss vs model parameters
Block freezing, loss vs model parameters
3.0
3.0
Loss
1.0
1.0
500
0.3
3e407
1e+08
3e+088
1e+09
3e+09
3e407
1e+08
3e+08
1e409
3e409
Parameters
Parameters
FLOP 1.5e+15 6.0e+15
FLOP 1.5e+15 6.0e+15 2.4e+16 9.6e+16 3.8e+17
2.4e+16 9.6e+16 3.8e+17
1.5e+18
1.5e+18
(a)
(b)
Figure 2: (a) IsoFLOP profiles for full fine-tuning. The horizontal axis is the number of parameters in the model, and the vertical axis is the achieved loss. Both axes log-scale. The optimal model size tends to increase as the computational budget increases. (b) IsoFLOP profiles for block freezing. The axes are the same for full fine-tur one-ttuiigg Each data point denotes the optimal choice with respect to the fraction of active blocks during training. which noted above the points. The optimal model size tends to increase the computational budget increases, while the optimal active block fraction tends to slightly decrease as the model size gets larger.
As expected, larger computational budgets consistently improve the loss. The IsoFLOP profiles resemble the classical ones from Hoffmann et al. [2022]. However, they tend to be flatter when the model size and computational budget are both small both large. From the IsoFLOP profiles can see what is the optimal model size that minimises the loss at each computational budget.
Block freezing Since the Pythia models experimented with have 6, 12, 16, 24 32 blocks, freeze G Q...... B
5 of their blocks if their number is divisible by and 0 8: g....... g if is divisible by 8. In this setup, unlike for full fine-tuning, the token-embedding parameters always remain frozen even when all the transformer blocks are active. We have NB = NN < Ny, and hence cost C' = 2Np D.
In Figure 2b we present IsoFLOP profiles for optimal loss at given model sizes. The trend of the profiles is quite similar to that of full fine-tuning presented in Figure 2a
We plot the loss with varying model sizes, computational budgets, and fractions of frozen blocks in Figure 3 For smaller models (S 160M parameters), a lower fraction of frozen blocks leads to lower final loss across all computational budgets. However, for larger models, using more than 70% of transformer often only marginally improves the model. For instance, for the model with le9 parameters, it visible that freezing 50- 70% of the blocks gives the optimal results for all budgets except for 3.8c17 FLOP.
Bias-only tuning Here simply update only the bias parameters and not the weights of the model. Figure showws the final training loss across all the model sizes and computational budgets. Increasing the computational budget decreases the optimal loss that is achievable, although the absolute values of the losses are high compared to other fine-tuning methods (the lowest achievable loss is above 0.4 with bias-only tuning).
shows
Low-rank adaptation We use the LoRA implementation from the PEFT library [Mangrulkar et al. 2022 We attach the adapter to all the dense layers of the network, and vary the adapter rank from 8 up to 2048, since even lower ranks resulted in an unstable training.
We compare the loss obtained for each model size and computational budget for different LoRA rank values in Figure In almost all the combinations of the model size and computational budget, the rank of 32 or 128 (occasionally 512) is optimal. Wang et al. (2023) also use LoRA in the context of contrastive learning, but they fix the rank of for their experiments. Our findings suggest that this choice should be reconsidered being potentially suboptimal.

---

Parameters: 1.4e+07
Parameters: 3.1e+07 33.1
Parameters: 76+07
Parameters: 1.6e+08
3.0
5.0
3.0
3.0
Loss
1.0
0.3
0.3
0.3
0.3
0.50
0:73
0.30
0.75
block
Parameters: 4.1e+08
Parameters: 1e+09
Parameters: 1.4
Parameters: 2.8
3.0
3.0
3.0
30
1.0
1.0
0.3
0.3
0.3
Activo block
fraction
Acthve block
Active fraction
FLOP
1.5e+15
6.0e+15
2.4
9.6e+16
3.8e+17
1.5e+18
Figure 3: The effect of block freezing across all model sizes. Different colours signify different computational budgets. Unless the model large and the computational budget small, it is always better to update all the (non-embedding) weights of the model.
Bias tuning, loss vs model parameters
LoRA, loss vs model parameters
3.0
3.0
Loss
1.0
1.0
0.5
0.3
3e+07
1e+08
3e+0B
1e+09
3e+09
3e+07
1e+08
3e+08
1e+09
3e+09
Parameters
Parameters
FLOP
FLOP
1.5e+15
6.0e+15
2.4e+16
9.6e+16
3.8e+17
1.5e+18
1.5e+15
6.0e+15 2.4e+16 9.6e+16
3.8e+17
1.5e+18
(a)
(b)
Figure 4: (a) IsoFLOP profiles for bias-only tuning. The horizontal axis is the number of parameters in the model, and the vertical axis is the achieved loss. Both axes log-scale. The optimal model size increases as the computational budget increases, but the achievable loss is higher than for other fine-tuning methods. (b) IsoFLOP profiles for LoRA fine-tuning. The axes the same as for bias-only tuning. Each data point denotes the optimal choice of the rank of LoRA matrices given the size of the model and the computational budget.
we present IsoFLOP profiles for LoRA, where each data point represents the best LoRA rank setting for given model size and computational budget. We see that the relationship between the optimal loss and the LoRA rank is quite consistent across model sizes and computational budgets, meaning that the rank hyperparameter is not very sensitive with respect to the two variables. In practice, this means that setting the LoRA rank to 32 or 128 are likely to be good default choices.
In Figure 4b
Lookup table of optimal model sizes for different methods In Figure ba (for full fine-tunings Figure (b) LoRA), and Figure 13 (for freezing and bias tuning, Appendix AA). we visualise the relationship between the optimal model size and the computational budget. This makes looking up the optimal configuration for a given method at a given budget very convenient.
Downstream performance In Figures 8 10 and I1 (in Appendix A we present the IsoFLOP plots of the downstream performance for all the four fine-tuning methods on the subset of the MTEB benchmark (described in Appendix [B), Based on the performance results, we also present the optimal size choices per budget in Figure [5Tin Appendix A (which is performance analogue Figure [3] and the optimal performance plot in Figure 12 (a performance analogue of Figure T). These are mostly consistent with the loss results, albeit more noisy. In Appendix C we demonstrate the significant correlation between the final training loss and the downstream task performance.
7

---

Parameters: 1.4e+07
Parameters: 3.1e+07
Parameters: 7e+07
Parameters: 1.6e e08
5.0
1.0
0.5
Parameters:
Parameters:
Parameters:
2.0
3.0
3.0
1.0
1.0
0.3
LORA
FLOP 1.5e+15
6.0e+15
2.4
9.6e+16
3.8e+17
1.5
Figure 5: The effect of different LoRA ranks across all model sizes. Different colours signify different computational budgets. The inflected curves indicate that is less beneficial to use rank from either extremes of the spectrum (8 2048). The detrimental effect of the high rank of 2048 is stronger for lower computational budgets. Ranks of and 128 result in the lowest loss overall.
Full fine-tuning, parameters vs computational budget
LoRA, parameters vs computational budget
10.09
1c409
30
10400
1e+00
807
3e407
FLOP
FLOP
(a)
(b)
Figure Optimal model size vs. computational budget for (a) full fine-tuning, and (b) LoRA.
4.3 Scaling laws for embeddings
We aim to model the behaviour of the loss with an analytical formula as function of the number of parameters N and the number of training tokens D. Following Hoffmann et al. [2022] we start with the following formula:
A
B D
T(N, D) - C
No
where A, B, C, E R, are real-valued parameters to be fitted. We found that this formula reasonably describes the loss behaviour for each of our methods. However, the fraction of trainable parameters an important factor that is not considered the model above. Following the modelling Frantar et al. 2023). propose modified formula:
a. log ba + a, - S) Cs
D)S = C
No
D
where S is an additional variable representing the trainable fraction of parameters, and ad, by, Og, be.b..c. C, E are additional coefficients be fitted. We check that this new formula fits the data better than the original one, and conjecture that it can be used for larger models and larger computational budgets. More details about the derivation are presented in Appendix F
4.4 Compute-optimal frontier and recipe
In Figure I we plotted the best achievable losses against the computational budgets for the four fine- tuning methods considered. We then fitted linear trend for each method on log-log scale (equivalent to power law relationship normal scale), and highlighted the lowest loss achievable at given

---

Algorithm Recipe for compute-optimal embedding model
Input: compute budget C.
Output: fine-tuning method, model size, data quantity, and (optionally) the method's hyperparams. if B < 9.06e16 FLOP then
Use full fine-tuning.
Go to Figure 6a to find the optimal model parameters N given budget C. Calculate the data quantity D. return Full fine-tuning, N, D, 0.
else
Use LoRA.
Go to Figure 6b to find the optimal model parameters N given budget C. Go to Figure to find the LoRA rank R according to C and N.
return LoRA, N, D, (R).
budgets using the best corresponding fine-tuning method, We call the contour of this function that predicts the optimal loss across methods the compute-optimal frontier.
The equation describing the full fine-tuning and the LoRA linear fits are
In(loss) -0.21 In(budget) + 8.39, and In(loss) -0.2 In(budget) + 8.93,
respectively. The equations suggest that at budget lower than 9.06e16 FLOP, the optimal model is achieved with full fine-tuning, and at higher budget, the optimal model is achieved with LoRA.
With this, one becomes fully equipped deduce the optimal recipe for text embedding model training for a given budget, by carrying out the procedure detailed in Algorithm [ Note that although the block freezing approach is not predicted be optimal at any budget, performance quite close to optimal. It also has the advantage of being easy to implement and has lower memory requirements, hence might be good method to choose in practical use cases for larger budgets. Its optimal model size and fraction of active blocks hyperparameter can be worked out in the same way for the other methods, with the help of Figure and Figure 13 (left pane) in Appendix(A)
We briefly discuss the case of data-optimal frontier in Appendix D We do not emphasise this setup due it being less standard in language modelling research.
4.5 Generalisation
To assess the generality of our findings, we investigate whether our analytical formula for predicting loss-fitted to datapoints from the Pythia suite (from Section 4.3) can also describe the behaviour of
models from different family. We conduct experiments on two language models, namely Gemma
2B (Mesnard et al.) 2024) and Gemma 2 2B Team et al. [204]] We trained both models with the computational budgets of 1.5e15, 6e15, 2. 2.4e 16, 9.6e16, 3.8e17, and 1.5e18 FLOP.
Figure 7 presents results for Gemma 2B with both full fine-tuning and LoRA fine-tuning, while results for Gemma 2 2B are shown in Appendix A In both cases, the formula provides close approximation of the loss behavior. Although this study is limited to single model size, it offers an initial indication that our findings may generalize to new models.
4.6 Takeaways
We compile the takeaways from our extensive experiments by analysing each fine-tuning method individually and contrasting them afterwards.
Bias-only tuning not a good fine-tuning strategy for embedding model training, as is consistently worse than other strategies under IsoFLOP comparisons.
LoRA and block freezing are both effective approaches that improve the performance of the trained embedding models at higher budgets. For LoRA, the rank hyperparameter is not very sensitive to model size or computational budgets, with an optimum at around 128.
Finding the optimal recipe for fine-tuning embedding models requires non-trivial effort, and the resulting optimal configurations are sometimes counter-intuitive (e.g., not using the largest model size possible). Carefully tuning hyperparameters is crucial and should be practised systematically.

---

Full Fine-Tuning
LoRA Fine-Tuning
8.0
4.0
Actual Loss
Actual Loss
2.0-
1.0-
1.0
0.5-
Gemma 28 data points
Gemma 28 data points
Pythia data points
Pythia test data
0.25
Pythia train data points
0.25
Pythia train data points
0.25
0.s
1.0
2.0
4.0 8.0
0.25 a.s 1.0 2.0 4.
32.0
Predicted Loss
Predicted Loss
Figure 7: Actual loss vs. predicted loss for full fine-tuning and LoRA finetuning. Red points are the models from runs on Pythia suite of size smaller than 2.8B, blue points are from runs on Pythia 2.8B and green points are from runs on Gemma 2B. The formula is fitted using the red points.
5 Limitations and future work
Other families of models In this paper, we experimented with the Pythia [Biderman et al. 2023 suite pre-trained models, which are of different sizes but have all been trained on 300B tokens. This means that they have been over-trained with respect to the Chinchilla scaling law (Hoffmann et al. 2022] to different degrees. The relationship between the optimal model size and the computational budget might differ for suite of pre-trained models with different over-training ratios. We note that prior efforts [Dettmers et al. 20244 have found that fine tine-tuning conclusions reached with Pythia do generalise other model families such OPT [Zhang et al. (0222). LLaMA [ouvron et al. [2023) and BLOOM [Scao et al. 2022]. in the quantised setting. Indeed, in Section 4.5 we find that a model from different family (Gemma Mesnard et al. 2024 closely fits our loss predictions.
Averaging Due to constraints on our computational resources, we ran each experiment exactly once. Averaging over multiple random seeds for the experiments would reduce the variance involved in each, and potentially result in less noisy conclusions. This is straightforward extension that we leave to carry out in future work, should we have more resources. We also plan to evaluate the trained models on the entirety of the MTEB benchmark [Muennighoff et al., 2023b), instead of using the losses from the last steps of training, to better reflect the performances of embedding models.
Other forms of embedding readout We only experimented with mean pooling as a way of extracting embeddings from the transformer models. Other readout methods could be used, like the max pooling or extracting the hidden state corresponding to the -fe-uunne token, which might result in different scaling laws and optimal frontiers.
Inference cost analysis In this work, we only focus on the training cost and refrain from taking inference costs into account. However, in certain practical scenarios, the latter may also be relevant.
6 Conclusions
In this paper, we systematically investigated the influence of pre-trained model size, fine-tuning data quantity, and fine-tuning method in the final performance of embedding models repurposed from language models. As result, we devised an algorithm that takes in the computational budget and returns the optimal configuration for the embedding model to train. This enables researchers who wish to adapt language models to embed their own data, especially those with limited computational resources, to obtain optimal text embedding models with greater efficiency time and resources.
Acknowledgements
The authors would like to thank Denis Kuznedelev, Soroush Tabesh and Konrad Staniszewski for helpful discussions and feedback. AQJ is supported by a Peterhouse graduate studentship. PM was supported by National Science Center Poland under grant agreement /019//////T6//4444. We gratefully acknowledge Polish hiihhpperformacce computing infrastructure PLGrid (HPC Centers: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PLG/2023/016717.
10

---

References
R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, E. Chu, J. H. Clark, L. E. Shafey, Y. Huang. K. Meier-Hellsteem, G. Mishra, E. Moreira,
M. Omernick, K. Robinson, S. Ruder, Y. Tay, K. Xiao, Y. Xu, Y. Zhang, G. H. Ábrego, J. Ahn, Austin, P. Barham, J. A. Botha, J. Bradbury, S. Brahma, K. Brooks, M. Catasta, Y. Cheng, C. Cherry, C. A. Choquette-Choo, A. Chowdhery, C. Crepy, S. Dave, M. Dehghani, S. Dev, J. Devlin, M. Diaz, N. Du, E. Dyer, V. Feinberg, F. Feng. V. Fienber, M. Freitag, X. Garcia, S. Gehrmann, L. Gonzalez, and et al. Palm 2 technical report. CoRR, abs/2305. 10403, 2023. doi: URL http:: http///tioooogg.//////o/// 48550/ arXiv. 2305 10403
(0-48550/ARXIV.233..06000
D. Biderman, J. G. Ortiz, J. Portes, M. Paul, P. Greengard, C.
Jennings, D. King, S. Havens, V. Chiley, J. Frankle, C. Blakeney, and J. P. Cunningham. LoRA learns less and forgets less, 2024.
S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learming Research, pages 2397-2430. PMLR, 2023. (Rtpsttss////ooceediii mlr. https//222bbiieemmaan
html
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry. A. Askell, S. Agarwal, A. Herbert- Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler. M. Litwin, Gray, Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. CoRR, 1tt./2005.14166 2020. URL https: org/ hts//bb/2000 14165
T. Chen, B. Xu, C. Zhang, and C. Guestrin. Training deep nets with sublinear memory cost. CoRR, Abs/1604.0611 2016. URL pttttp://axxiv.org//aa 166400000b//0000077 06174
T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36, 2024.
Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171 Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1/N19 1423. URL https:://p::ot org/ 10. wwt.1/116//vn11// 1423
Y. Fang. J. Zhan, Q. Ai, J. Mao, W. Su, J. Chen, and Y. Liu. Scaling laws for dense retrieval, 2024. URL https: /tts:///rxxii org/ /
E. Frantar, C. Riquelme, N. Houlsby, D. Alistarh, and U. Evei. Scaling laws for sparsely-connected foundation models. CORR, abs/2309.0852 2023. doi: 10.4550/ARXXI22300008555 URL httts::/doi..rg:: 41111..880/1181111 arXiv 2309.03000 08520
L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800GB dataset of diverse text for language modeling. CoRR. abs/2101.00023
2021a. URL http::ts:///art http:///axvvvvv.m httg///b/22111.0000
L. Gao, Y. Zhang, J. Han, and J. Callan. Scaling deep contrastive learning batch size under memory limited setup. In A. Rogers, Calixto, I. Vulic, N. Saphra, N. Kassner, O. Cam- buru, T. Bansal, and V. Shwartz, editors, Proceedings of the 6th Workshop on Representation Learning for NLP, RepLANLP@ACL-UCCN ciation for Computational I Linguistics, 2021b. doi: htttps://doi http///oi.og//...//////
2021, Online, August 6, 2021, pages 316-321. Asso-
1-1.P---11311 1.31. URL
wwwp:/2pp...rrrpppp--ppp.-pp. 1.31
T. Gao, X. Yao, and D. Chen. SimCSE: Simple contrastive learning of sentence embeddings. In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 1894-6910 Association for Computational
11

---

Linguistics, 18653/v1/2021. www.lemmnlalal..ee maim.miin222.......... 552
URL
org/10
S. Gugger, L. Debut, Wolf, P. Schmid, Z. Mueller, S. Mangrulkar, M. Sun, and B. Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://
github. wow/huggingface aaaec///cceeeeateee. 2022.
M. Günther, L. Milliken, J. Geuter, G. Mastrapas, B. Wang, and H. Xiao. Jina embeddings: A novel set of high-performance sentence embedding models. CoRR, abs/2307.111222 2023, doi: /tggg...4550.00 48550/ arXiv 2307 11224
10:48550/ARXI..237...15
URL https:////o org/10.
J. He, C. Zhou, X. Ma. T. Berg-Kirkpatriee and G. Neubig. Towards unified view of parameter- efficient transfer learning. CoRR, ab./2100.043666, 2021a. UtL/istpp://rrvvv.oor//aaa 04366
X. He, K. Zhao, and X. Chu. AutoML: A survey of the state-of-the-art. Knowledge-Bacd Based Systems, 212:106622, 2021b.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optimal large language models. CoRR, abs/2203, 2022. doi: 10.48550/ 15556
ARXIV.2203. 15556. URL.ttps: htttt:////ooti org/ 10.r///11..8000 0..45505/ariiy
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for NLP. CoRR, abs/1902.00751,, 2019. URL http: /ttp:://axxvvv..o org/
00751
E. J. Hu, Shen, P. Wallis, Z. Allen-Zhu, Y. Li, Wang, L. Wang. and W. Chen. LoRA: Low- rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
https:// net/ orrmm?d--ZZeWeeaFY..
G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave. Unsupervised dense information retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022, 2022. URL
htttps://openreviee http://eeeeeeeeeeeeeeeeee htt//ooommmme/////// JKM
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel. G. Lample, L. Saulnier, L. R. Lavaud, M. Lachaux. P. Stock, T. L. Scao, T. Lavril, T. Wang. T. Lacroix, and W. E. Sayed. Mistral 7B. CoRR, abs/2310.06822. 2023. doi: 10.48550/ S0UUV.331000822.
bttps:s/s///o... org/ 10. 400.48850/arra arxiv 2310.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. CoRR, abs/2001.08866, 2020. URL https://arxize org/ abs/ 08361
B. Lester, R. AI-Rfou, and N. Constant.
The power of scale for parameter-efffcient prompt tuning. In M. Moens, X. Huang, L. Specia. and S. W. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045-3059. Association for Computational Linguistics, 2021. doi: 001118653//1/000 1/2.://12221..ELLPMMOOMMM WWW.LEENN.......M.. 243. URL httpssss/////...rr//// 18653 /0003/vv1//0 v122021. emnlp- cannp-mmin......mmmmmm.mmmmmmmmm 243
X. L. Li and P. Liang.
Prefix-tuning Optimizing continuous prompts for generation. CoRR,
0e//21111001900 2021. URL httphhtppp/// :/arxii///xxxx... org/ 00100
Z. Li, x. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2308.033 002281, 2023. doi: 10.48550/RRXI22308.0555 URL https:://doi. org/10 48550/ wtt.:///00/....5////aa......1
2308 03281
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. httpa//pp///pppeeeeeeeeeetttttts
12

---

S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul. and B. Bossan. PEFT: of arr parameter-efficient fine-tuning methods,
hotpa///...oouggggggg///////pppp 2022.
R. Meng, Y. Liu, S. R. Joty, C. Xiong, Y. Zhou, and S. Yavuz. SFR-Emmedding-Minn Mistral:: Enhance text retrieval with transfer learning. Salesforce AI Research Blog, 2024. (Rtht:a/:://bl
salesforccairesearch
sfr--embedddd-mmittrat
T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière M. S. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, Austin, J. Keeling, J. Labanowski, J. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, and et al. Gemma: Open models based on gemini research and technology. CoRR, www.22433.0299.... 2024. htt.hhtss://od. org 10. 48550/ arxiv 2403. 08295
doi: A00.55//RXX..2433..0
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In Y. Bengio and Y. LeCun, editors, 1st International Conference on Learning Representa- fions, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. URL http:/aaxvvvvovoraaaa/ attp//abb/111133933
arriv.
N. Muennighoff. SGPT: GPT sentence embeddings for semantic search. CoRR, abs/2202.08904, 2022. httpss//://aaaa org/
wttpa//wwww...rgaaa///........ wwwpa/////.cccco 08904
N. Muennighoff, A. M. Rush, Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and C. Raffel. Scaling data-constrained language models. CoRR, abs/2305. 16264, 2023a. doi: 10.44550/ARXXIV23006 16264. URL https:///ol
org/10.a 0/5//0048550 48550/ arXiv. 2305. 16264
N. Muennighoff, N. Tazi, Magne, and N. Reimers, MTEB: Massive text embedding benchmark. In A. Vlachos and I. Augenstein, editors, Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 2006-2029 Association for Computational Linguistics, 2023b. doi: 10.18653/ BRLLhhts:://dob crrooii.oggg/ie 10. 18653/v1/2023 eacl main.. 148
org/10.
A. Neelakantan, T. Xu, R. Puri, A. Radford, J. M. Han, J. Tworek, Q. Yuan, N. Tezak, W. Kim, C. Hallacy, J. Heidecke, P. Shyam, B. Power, T. E. Nekoul, G. Sastry, G. Krueger, D. Schnurr, F. P. Such, K. Hsu, M. Thompson, T. Khan, T. Sherbakov, J. Jang. P. Welinder, and L. Weng. Text and code embeddings by contrastive pre-training. CoRR, 0bs/2201.10005, 2022. URL https://arxix: wtw.x//v.rr//bb/////1000000
W. Qi Y. Ruan, Y. Zuo, and T. Li. Parameter-efffiiennt tuning on layer normalization for pre-trained language models. CORR, abs/2211 008682, 2022. doi: 10.48550/ARXIV.2211 1.08682. URL
https://doi.
org/10. 000.0.8850//880 arXiv. .211. 08682
A. Radford, J. Wa, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilie, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvo,, M. Gallé, J. Tow, A. M. Rush, S. Biderman, A. Webson, P. S. Ammanamanchi, T. Wang, B. Sagor, N. Muennighoff, A. V. del Moral, O. Ruwase, R. Bawden, S. Bekman, A. McMillan-Major, I. Beltagy. H. Nguyen, L. Saulnier, S. Tan, P. O. Suarez, V. Sanh, H. Laurençon, Y. Jernite, Launay, M. Mitchell, C. Raffel, A. Gokaslan, A. Simhi, A. Soroa, A. F. Aji. A. Alfassy, A. Rogers, A. K. Nitzav, C. Xu, C. Mou, C. Emezue, C. Klamm, C. Leong, D. van Strien, D. I. langlui
Adelani, and et al. BLOOM: 176b- 776b-parameter open-access multilingual language model. CORR. uttpstss://doo org/ 10../10.4888 48550/
abs/2211 1221110.5 1,05100, 2022. doi: 10448550/ARXXV.2211.055 arXiv. 21w..221..0110 05100
Sun, H. Zhang, Z. Liu, J. Bao, and D. Song. Llm-oriented retrieval tuner, 2024. URL https 7/ //arxiv. org/ abs/2403. 01999
G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ramé, J. Ferret, P. Liu, Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Stanczyk, S. Girgin, N. Momchev, M. Hoffman,
13

---

S. Thakoor, J.-B. Grill, B. Neyshabur, O. Bachem, A. Walton, A. Severyn, A. Parrish, A. Ahmad,
A. Hutchison, A. Abdagic, A. Carl, A. Shen, A. Brock, A. Coenen, A. Laforge. A. Paterson,
B. Bastian, B. Piot, B. Wu, B. Royal, C. Chen, C. Kumar, C. Perry, C. Welty, C. A. Choquette- Choo, D. Sinopalnikov, D. Weinherger, D. Vijaykumar, D. Rogozińska, D. Herbison, E. Bandy, E. Wang, E. Noland, E. Moreira, E. Senter, E. Eltyshev, F. Visin, G. Rasskin, G. Wei, G. Cameron, H. Batra, H. Dhand, I. Nardini, J. Mein, J. Zhou, J. Svensson, J. Stanway, J. Chan, J. P. Zhou, J. Carrasqueira, J. Iljazi, J. Becker, J. Fernandez, J. van Amersfoort, J. Gordon, J. Lipschultz, Newlan, J. yeong Ji, K. Mohamed, K. Badola, K. Black, K. Millican, K. McDonell, K. Nguyen, K. Sodhia, K. Greene, L. L. Sjoesund, L. Usui, L. Sifre, L. Heuermann, L. Lago. L. McNealus, L. B. Soares, L. Kilpatrick, L. Dixon, L. Martins, M. Reid, M. Singh, M. Iverson, M. Gömer, M. Velloso, M. Wirth, M. Davidow, M. Miller, M. Rahtz, M. Watson, M. Risdal, M. Kazemi, M. Moynihan, M. Zhang, M. Kahng, M. Park, M. Rahman, M. Khatwani, N. Dao, N. Bardoliwalla, N. Devanathan, N. Dumai, N. Chauhan O. Wahltinez, P. Botarda, P. Barnes, P. Barham, P. Michel, P. Jin, P. Georgiev, P. Culliton, P. Kuppala, R. Comanescu, R. Merhej, R. Jana, R. A. Rokni, R. Agarwal, R. Mullins, S. Saadat, S. M. Carthy, S. Cogan, S. Perrin, S. M. R. Arnold, S. Krause, S. Dai, S. Garg, S. Sheth, S. Ronstrom, S. Chan, T. Jordan, T. Yu, T. Eccles, T. Hennigan, T. Kocisky, T. Doshi, V. Jain, V. Yadav, V. Meshram, V. Dharmadhikari, W. Barkley, W. Wei, W. Ye, W. Han, W. Kwon, X. Xu, Z. Shen, Z. Gong, Kirk. Rao, M. Giang, L. Peran, T. Warkentin,
G. Martins, H. Hashemi, H. Klimczak-Pluciñan
Z. Wei, V. Cotruta,
Collins, Barral, Z. Ghahramani, R. Hadsell, D. Sculley, J. Banks, A. Dragan, S. Petrov, O. Vinyals, J. Dean, D. Hassabis, K. Kavukcuoglu, C. Farabet, E. Buchatskaya,
Borgeaud, N. Fiedel, A. Joulin, K. Kenealy, R. Dadashi, and A. Andreev. Gemma 2: Improving open language models at practical size, 2024. URL http::ttp::///ttt http/v/xxvvvv. wwwpiiiv.rgrbbb/b/eee wwwpg//gg//....////// 00118
N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych. BEIR: A heteroge- neous benchmark for zero-shot evaluation of information retrieval models. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URI. httt:::://attteess-enncaaaaass/s 2021/hasse
www.inuuri.... neurips.cc/papen ca/pa/er/ htmmll
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, F. Grave, and G. Lample. Llama: Open and efficient foundation language models. CORR, ahs/2302.13371 2023. doi: 10,48550ARXIV22302.1333 48550/8555a// arxiv. 2302. 13971
URL https: 11t..//////....5/////////./
L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang. R. Majumder, and F, Wei. Text embeddings by weakly-supervised contrastive pre-training. CoRR, abs/2212.03533, 2022. doi: 10.48550/
ARXIV.2212.0533.. Jtt. orgo 1tt.:e/.000......///a/a//... arxiv. 2212.03533
L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei. Improving text embeddings with large language models. CoRR, htss/2001.00668. 2023. URL https::/arrxiv.rrg// 00368
Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 38-45, Online, Oct. 2020, Association for Computational Linguistics. URL hhttpps:/wwirr acwweb. http://anttooogg//2.eeeeee
6
S. Xiao, Z. Liu, P. Zhang, and N. Muennighof. C-pack: Packaged resources to advance general chinese embedding. CoRR, abs/2309.07597, 2023, doi: 10.4850AARXIV.20000775 https://doi.
URL
org/10. httg//...855///e00
2309. 075977
L. Xiong, C. Xiong, Y. Li, K. Tang, J. Liu, P. N. Bennett, J. Ahmed, and A. Overwijk. Approxi- mate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htttttta//opeeeeeiieee
pee//forumiidzzefrfgyy/l
E. B. Zaken, Goldberg, and S. Ravfogel. Bitfit: transformer-based masked language-models. In S. Muresan, P. Nakov, and A. Villavicencio.
Simple parameter-efficien fine-tuning for
14

---

editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 2: Short Papers), ACL 2022. Dublin, Ireland, May 22-27, 2022, pages 1-9. Association URL https
for Computational Linguistics, 2022. doi: 10. 11866/////222A11-SSS00000
Zhan, J. Mao, Y. Liu, Guo, M. Zhang, and Ma. Optimizing dense retrieval model training with hard negatives. In F. Diaz, C. Shah, T. Suel, P. Castells, R. Jones, and T. Sakai, editors, SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1503-1512. ACM, 2021. https//ps http://o..orrgg// 10. 1145/3404835 .005/443448335.3628800000
doi: 0411145/3444335344263650
B. Zhang, Z. Liu, C. Cherry, and O. Firat. When sealing meets LLM finetuning: The effect of data, model and finetuning method. CoRR, abs/2402.17193, 2024. doi: 1044550/ARRXIV22402.17999
URL https: //tps://doi.. org/10, 48550/arXii 2002.
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, V. Lin, et OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01668 2022.
15

---

A
Additional results
In Figure (for full fine-tuning), Figure 9 (for block freezing), Figure T0 (for LoRA), and Figure 11 (for bias tuning), we present the IsoFL.PP profiles where downstream task performance is shown. The downstream performance is measured on subset of MTEB tasks described in Appendix B
Using the location of optimal loss given the computational budget for freezing and bias tuning methods illustrated in Figures 2b and 4a|- project optimal model sizes for each computational budget in Figure 13 Similar figures for tull fine-tuning and LoRA are presented in the main text in Figure 6a and Fi Figure respectively.
Using the location of optimal downstream task performance given the computational budget for all we project optimal model
the four fine-tuning methods illustrated in Figures 9 10 and 11
sizes for each computational budget in Figure 15
In Figure 14 we present how our formula fitted to models from the Pythia suite describes the loss behaviour of the Gemma 2 model.
B Downstream task evaluation details
For each category present in the MTEB benchmark [Muennighofi et al.) 2023b]. we take the data from the models evaluated in the original paper and select the task that has the highest correlation with the category average. This results in tasks from the MTEB benchmark used for the downstream task evaluation:
1. EmotionClassification (classification),
2. TventNeewsgouuusCluuserrnngggg (clustering),
3. SprintDuplicaeequussions (pair classification).
4. AskUbuntuDugQuestions (re-ranking),
5. SciFact (retrieval),
6. STS15 (semantic text similarity)
7. SummEval (summarization),
8. Tatoeba (bitext mining).
F Full fine-tuning, MTEB vs parameters
MTEB performance
0.54
0.51
0.48
0.45
3e+07
1e+08
3e+08
1e+09
3e+09
Parameters
FLOP
1.5e+15 6.0e+15 2.4e+16 9.6e+16 3.8e+17 1.5e+18
Figure IsoFLOP profiles for full fine-tuning, where performance on MTEB tasks observed instead of the loss. Both axes use log-scale. The performance remains approximately inversely correlated with the loss. However, interestingly, there are no corresponding plateaus on the IsoFLOP profiles.
16

---

Block freezing, MTEB vs parameters
0.56
MTEB performance
0.52
0.48
0.44
3e+07
1e+08
3e+08
1e+09
3e+09
Parameters
FLOP
1.5e+15 6.0e415
2.4e+16
9.6e+16
3.8e+17
1.5e+18
Figure 9: IsoFLOP profiles for freezing, where performance on MTEB tasks is observed instead of the loss. For given FLOP level, the model with the optimal active block fraction is depicted.
LORA, MTEB vs parameters
0.56
MTEB performance
0.52
0.48
0.44
3e+07
1e+08
3e+0B
1e+09
3e+09
Parameters
FLOP
1.5e+15 6.0e+15 2.4e+16
9.6e+16
3.8e+17
1.5e+18
Figure 10: IsoFLOP profiles for LoRA, where performance MTEB tasks is observed instead of the loss. For a given FLOP level, the model with the LoRA rank is depicted.
Bias tuning, MTEB vs parameters
MTEB performance
0.50
0.45
3e+07
1e+08
3e+08
1e409
Parameters
FLOP
1.5e+15 6.0e+15 2.4e+16
9.6e+16 3.8e+17 1.5e+18
Figure 11: IsoFLOP profiles for bias tuning, where performance on MTEB tasks is observed instead of the loss.
17

---

MTEB performance vs compute budget
MTEB performence
0.550
0.525
0.500
1.5e+15
6.0e+15
2.4e+16
9.6e+16
3.Be+17
1.5e+18
FLOP
Method + Blas luning
Full fine -uning + LoRA
Freezing
Figure 12: The optimal downstream task performance achieved using all four different fine-tuning methods at given budgets. This is a performance analogue of Figure (which shows final training loss instead).
Block freezing, parameters vs computational budget
Bias tuning, parameters vs computational budget
3e:409
Parsmettrs
30:08
3e+08
1e408
1c+00
30:07
18636
1e436
1e-18
Figure 13: Optimal choices of model size given the computational budget, with respect the final training loss, for block freezing and bias tuning.
Full Fine-Tuning
8.0
Gemma 2 28 data points
Pythia test data points
Pyenia train data points
4.0
Actual Loss
2.0
1.0
0.5
0.25
0.25
0.5
1.0
2.0
4.0 8.0
Predicted Loss
Figure 14: Actual loss predicted loss for full fine-tuning and LoRA finetuning. Red points the models from runs on Pythia suite of size smaller than 2.8B, blue points are from runs on Pythia 2.8B and green points from runs on Gemma 2B. The formula fitted using the red points.
18

---

Full fine-tuning, parameters vs computational budget
Block freezing, parameters vs computational budget
10409
Parameters
Perameters
Ic-10
Re+08
36408
1e:08
18+36
1e+17
1e:18
16:15
10:177
1e-18
FLOP
FLOP
LoRA, parameters vs computational budget
Bias tuning, parameters computational budget
1o+00
1e+09
Parameters
2e+000
36+08
1c:08
1e+00
1e+15
Ic+17
1e-18
18+15
le+17
1e-18
FLOP
FLOP
Figure 15: Optimal choices of model size given the computational budget, with respect to the performance on the downstream task evaluation.
For each of the task above, we specify its category in brackets. Since Tatoeba is not featured in the MTEB leaderboard we skip it when calculating our approximation of the MTEB performance of the embedding models.
In principle, one could evaluate all the checkpoints on all the MTEB tasks. However, we decided to evaluate representative subset in order to speed up evaluation (evaluating some MTEB task takes more than 24 hours, and we have close to 1000 checkpoints, so the overall cost is significant).
C Correlation between loss and MTEB performance
In our work, we use loss as the main metric used to formulate the conclusions in form of the compute- optimal recipe and the scaling laws. This is in line with other works investigating scaling laws in various contexts, like [Hoffmann al.) 2022] Muennighoff et al.) 2023a Zhang al. 2024). However, we additionally evaluated the trained models
subset of MTEB benchmark. We calculated the Spearman's 's rank correlation between the loss and the MTEB performance and is equal to -0.892, which is significant. Additionally, Figure T6 visually compares the loss and MTEB performance.
D Data-Constrained Setup
In our work, we mainly focus on finding the compute-optimal recipe. The training datapoints we share can however be utilised obtain conclusions about other setups. In particular, we can consider the data cata-constrained setup. Here, we compare models trained with the same amount of training data instead of the same amount of computation. In Figure 17 we present the plot the final contrastive loss vs. the amount of tokens seen by the network on log-log scale. The conclusion that can be drawn is that in a data-constrained setup, one should use the largest model available and fine-tune using full fine-tuning or LoRA with a rank on the bigger side.
E Hyperparameters for the training setup
Below, we list the hyperparameters that we used for training.
19

---

Loss vs MTEB performance
0.55
FLOP
MTEB performance
1.5e +18
3.8e+17
9.6e+16
0.45
2.4e+16
6.0e+15
1.5e +15
0.40
0.35
0.3
1.0
3.0
Loss
Figure 16: All the final checkpoints involved in suite of experiments, evaluated bath in terms of the final training loss and the performance on a representative subset of MTEB benchmark. The correlation between these metrics is significant.
Loss vs training data quantity
Method
bias
full
lora
3.0
suffix
Loss
Size
1.4e+07
1.0
3.1e+07
7.0e+07
1.6e
4.1e+08
1.0e+09
1e+06
1e+08
1e+10
2.8e+09
Tokens
Figure 17: Final training loss vs number of training tokens for all the models. It is visible that given the number of training tokens, the best performance is obtained by the largest model.
20

---

batch_size = 1024
length
AdamM. wwwght_ _dccyy = 0.1
tau 0.025 temperature parameter
E.1
Learning rate for block freezing and full fine-tuning
For all the experiments with block freezing and full fine-tuning, we base our peak learning rate on the peak learning rate from Pythia [Biderman et al. 2023). We calculate our learning rate by dividing the Pythia learning rate by ten. We use the cosine learning rate schedule with warmup, warmup period being fixed 10% the training steps. Learning rate starts at zero, goes up to the peak learning rate, then follows the cosine schedule back to zero.
E.2 Temperature and weight decay for partial tuning
We found the temperature affect the training performance rather significantly, while the weight decay was of a very slight importance. We run a grid search for partial tuning, varying temperature and weight decay and arrived at choice
= tū and weight decay 0.1. We compared the runs by calculating the results on a subset of MTEB benchmark. Grid results for temperature are detailed in Table] I with best scores for each the temperature value listed.
Table 1: A comparison of different temperature values
Temperature
Avg. score
50
0.5
40
0.51
35
0,5
30
0.5
10
0.47
1
0.3
E.3 Readout: last VS average
We found that using the averaging of embeddings as the pooling method results in better performance over selecting the last embedding. compare the results the best run for mean and last pooling on full MTEB without MSMarco subset. The scores are presented in Table 2
Table 2: Mean pooling works better than last pooling.
Pooling method
Avg. score
Mean pooling
0.47
Last pooling
0.36
E.4 LoRA hyperparameters
In order to be able to compare the loss from LoRA with partial block freezing, we leave most of the hyperparameters unchanged. We change the learning rate, we find the optimal learning rate by running grid search over hyperparameters. The result shows that for 3 different rank choices, the same learning rate works best. The exact values are presented in Table
E,5
Bias tuning hyperparameters
Similarly, as with LoRA, for bias tuning to be comparable with partial block freezing, we leave most of the hyperparameters the same. We vary the learning rate, we find the optimal learning rate by running a grid search over hyperparameters. The exact values can be seen in Table 4
21

---

Table 3: Learning rate for LoRA.
Learning rate
LoRA rank
Loss
le2
1.96
1c3
1.35
le4
2.22
1e2
1.96
1e3
16
1.34
le4
2.22
1e2
2.09
1e3
32
1.32
le4
2.19
Table 4: Learning rate for bias tuning.
Learning rate
Loss
le2
1.24
1c3
1.37
lc4
2.97
Scaling laws
We aim to express the formula describing the behaviour of the loss for our models function of the number of parameters and training tokens D. Following Hoffmann et al. [2022 we start with:
B
L(N, C
No
D
where A, B, C, 8 E IR are parameters to be fitted based on the data.
train using the contrastive objective instead of the next-token prediction task, and moreover we apply various compute-efficient methods making some fraction of the parameters of the model trainable. Despite that, can still observe certain linear trends that the loss follows. For all the methods, when the FLOP budget is not overly restricted, if fix the amount of both all the parameters well as the trainable parameters and vary the amount of FLOP, the loss behaves approximately linearly. We observe that for full fine-tuning, the points are roughly collinear, as in Figure 18. On the other hand, for LoRA and partial block freezing, we discover that while the lines for one model are parallel, they are translated, depending on the amount of trainable parameters, which can be observed in Figures [T 22
Our setting differs from that of Hoffmann al. 2022
B Following the Frantar et al. (2023] 's approach, we replace the term DA by (o=(1_))'* where is n2
an additional parameter indicating the fraction of the model's parameters that are trainable, and a, b, are real-values constants. Similarly, if we keep the forward token amount fixed and vary the amount of model parameters, the linear trend can be observed. Again, the lines are parallel and differ by an intercept, that seems to be linearly dependent on the logarithm of forward tokens, shown in Figure 20 Therefore, replace the term N by aa-log( +bu, where ag by are real-valued constants. The final formula describing the loss in our setting has the form:
D) = C+ ad log(D) + ba
+ a. (1 (1-S)) + Co
No
D
We split our dataset into train and test set, with test set being the points corresponding Pythia 2.8B which is the biggest model that we consider and train set the rest. Similarly to Hoffmann et al. [2022]. we use L-BGFS optimization algorithm and Huber loss. We find that 0.001 prevents over-fitting and decreasing it further does not bring more benefits. We then compare the base formula being fitted formula from Chinchilla to our modified formula and find that our works better, demonstrated in Both formulas are fitted after grid search the initial values of their coefficients. It can
Figure 19
22

---

Full fine fine-tuning, loss vs computational budget
3.0
Parameters
2.8e+09
1.0e+09
Loss
4.1e408
1.0
1.6e+08
7.0e+07
3,1e+07
1,4e+07
0.3
1e+16
1e417
1e+18
FLOP
Figure 18: Loss the axis and FLOP on the axis. Lines of the same colours correspond to the same amount of parameters in the model. Excluding points corresponding to severely constrained FLOP budget, the loss behaviour be approximated by straight lines.
42
(a) Full fine-tuning our formula
(b) Full fine-tuning, standard formula
(c) LoRA, our formula
(d) LoRA,
standard formula
(e) Block freezing, our formula
(1) Block freezing, standard formula
(g) Bias tuning, our formula
(h) Bias tuning, standard formula
Figure 19: The actual on the y/ axis and predicted loss on the x axis. Red points are the training set and blue the test set. Figures T9a 19c different fine-tuning methods, while Figures 19b
19e and 19g are the results of fitting our formula for
19d 19 and 19h are the result of fitting the standard formula from Hoffmann et al. 2022 It can be observed, that in contrast to the standard formula, our results in plots that are roughly linear, without additional trends visible.
be observed that we are able to accurately predict the loss for the bigger model for all the methods. Therefore, hypothetize that formula can be used for prediction the performance of models of an even larger scale.
G Robustness to increase in batch size and context length
Constrained by the computational budget, we performed the biggest line of experiments using batch size 1024 and context length 75. However, an increased context length as well as batch size might be desirable in many cases. Here, we demonstrate that the scaling laws established by us extend to setting with batch size 2048 and context length 512. For those experiments, we trained five pre-trained decoder-only models from the Pythia suite [Biderman et al. 2023]. with sizes 14M, 31 M1M, 70M, 160M, and 4 410M. We consider five computational budgets: 1.5e15. 6e15, 2.4e16. and 9.6e16 FLOP. We fit the formula following the method from Appendix with the change of using the
23

---

Loss Parameter Count
finetuning
freezing
16
12
LoRA
Bias finetuning
-0.5
14
18
20
22 14
18
20
22
Figure 20: Logarithm of loss on the y axis and logarithm of the parameter count on the x axis. The colors represent different quantiles of the amount of tokens used in the model fine-tuning. Linear dependency can be observed, with the trends being approximately parallel for different quantiles.
Parameters: 1.4e+07
Parameters: 3.1e+07
Parameters: 7e+07
Parameters: 1.6e408
3.0
3.0
1.0
1.0
1.0
0.3
03
Ie+10
1e417
Ie+38
Je+tb
Ie417
1e-18
Ie+0B
1e417
1e-18
Ie+1B
Ie417
1e-18
FLOP
FLOP
FLOP
FLOP
Parameters: 4.1e+08
Parameters: le+09
Parameters: 1.4e+09
Parameters: 2.8e+09
3.0
3.0
0.3
Ie+10
1n+17
Ic-38
1e+1B
Ie417
1e-18
Ic+00
16+17
1e-38
Ic+16
16417
1c-18
FLOP
FLOP
FLOP
FLOP
Active black fraction
0.25
0.50
0.75
1.00
Figure 21: Plots of losses for block freezing. Each subplot contains datapoints corresponding to one model size. Loss the y axis and FLOP on the axis. Lines of the same colours correspond to the same amount trainable parameters in the model. The loss behaviour can be approximated by straight lines.
24

---

Parameters: 1.4e407
Parameters: 3.1e+07 3..1
Parameters: 76+07
Parameters: 1.6e+08
1.0
100
10
0.3
0.3
1c-18
Parameters: 4.1e+08
Parameters: 1e409
Parameters: 1.9
Parameters: 2.
3.0
3.0
1.0
1.0
0.3
0.
0.3
0.3
FLOP
FLOP
Rank
32
128
512
2048
Figure 22: Plots of losses for LoRA. Each subplot contains datapoints corresponding to one model size. Loss on the y axis and FLOP on the axis. Lines of the same colours correspond to the same amount of trainable parameters in the model. The loss behaviour be approximated by straight lines.
(a) Full fine-tuning tuning
(b) LoRA
(c) Partial block freezing
(d) Bias tuning
Figure 23: Plots of predicted vs actual losses for the finetuning methods. The formula is fitted on the data coming from trainings with batch size 1024 and context length 75 (red points). It is then used to predict the losses for data points trained with batch size 2048 and context length 512 (green points).
number of training steps as the measure of data seen by the model instead of forward tokens. We then use to predict the loss for the changed hyperparameters values. The results are visible in Figure 23 We conclude that our findings are relevant for setups with a larger batch size and context length.
H Computational resources
For all the trainings, we used a cluster with A100 GPUs with 40GB VRAM or with 80GB VRAM. We used 4 CPUs and 128GB of RAM per GPU. We estimate that the results contained in this article were obtained by expanding somewhere in between 30k and 40k GPU hours.
I Dataset
We utilize the BAAI dataset available httttss//ddaa..b baai. ac. cn/detailss/BAAT--
has a custom licence that allows for conducting academic research.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are accurate reflection the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations extensively.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
26

---

Justification:
Our paper does not contain theoretical results, only empirical ones.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
We fully disclose the information needed to reproduce the results in the paper. Since the reproduction of the training results would rather costly, we also share the data we have got as the result of the training sweep, an intermediate step in reproduction of conclusions/results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will open-source the code used for training. We share the code supplementary material to the submission. We share the link for downloading the dataset in the zip file with the code. We moreover share the training results used for creating the plots here
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
p soteesssubbiiiiiooooooooooo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https / ctltsps.ccpubbicc/// gwwe//ggieeeeeeeee Salee//CooSuubiiseeee
wwwbbbmmsssiooooooiii.. for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all the training and test details. We moreover share the code that can be used to check those details without the necessity for consulting the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Running the experiments several times would be too costly. However, we would like to point out that it is known that language modelling generally results in stable trainings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the information on the computational resources needed for the experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/ cc. / htt.a/ipp....p.bbllececccccccaaa
Answer: [Yes]
Justification:
The research does conform the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: As this work aims at improving the efficiency for people to obtain embedding models, we only see positive social impacts. This was discussed in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cite the original paper that has produced the dataset we use. We provide the dataset webpage, which contains the licence in Appendix [] We cite the relevant packages that we used in our code section
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The main asset introduced the paper is the code, that is attached as a supplementary material and it does contain the documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "B7wBOdnyxx", "reviewer": "Reviewer_GCkf", "summary": "This paper focuses on the efficient contrastive training of text embedding models using pre-trained decoder-only language models. The main contribution is an algorithm that determines the best configuration of model sizes, data amounts, and fine-tuning methods for different computational budgets. Through extensive experimentation, the authors developed a guideline that helps practitioners choose the most suitable design options for their text embedding models. The study finds that full fine-tuning is best for lower computational budgets, while low-rank adaptation fine-tuning is optimal for higher budgets.", "questions": "Please refer to the previous sections.", "limitations": "This paper discusses the limitation in Section 5.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "The research question posed, “What is the best embedding model one can train from a backbone decoder-only LLM with a fixed budget?” is highly valuable and important. It addresses a critical need within the NLP and IR domains.\n\nThe paper boasts a robust experimental setup, and the conclusions drawn from these experiments are insightful. The analysis based on empirical results provides a useful reference for advancements in both NLP and IR fields. This thorough approach enhances the reliability and applicability of the research findings.", "weaknesses": "The paper does not introduce a new methodology. The techniques employed, such as scaling laws, extracting representations from transformers, and contrastive fine-tuning, are all well-established in the literature. \n\nThere are several existing studies that the paper fails to acknowledge, which could provide a richer context and enhance the literature review. Notable omissions include works like “LLM-Oriented Retrieval Tuner” and “Scaling Laws for Dense Retrieval.” Acknowledging and discussing these related studies could strengthen the paper by positioning it within the existing research landscape more accurately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their effort to assess our work! Below we respond to the two points of critique.\n\n**(1) The paper does not introduce a new methodology. The techniques employed, such as scaling laws, extracting representations from transformers, and contrastive fine-tuning, are all well-established in the literature.**\n\nWe agree with the Reviewer that our study does not introduce novel techniques – and this was not our intention in this project. We rather aimed to investigate and compare existing modern approaches in the practical, limited compute-budget setting. Before our work, no study had been devoted to establishing compute-efficient approaches to contrastive fine-tuning of decoder-only language models. Given the recent activity in this area, we found it to be an important research gap that needs to be addressed. We believe that our findings constitute a *concrete scientific contribution* to helping researchers and practitioners working with embedding models.\n\n**(2) There are several existing studies that the paper fails to acknowledge, which could provide a richer context and enhance the literature review. Notable omissions include works like “LLM-Oriented Retrieval Tuner” and “Scaling Laws for Dense Retrieval.”**\n\nThank you for pointing out these two recent works! We have extended the related work section to include both of them. Specifically, starting from line 95 (new text in bold):\n\n> These approaches are all applicable to embedding models. Muennighoff [1] explored a simple parameter-efficient approach to repurpose GPT models into embedding models where only the bias tensors of the transformer model are updated [2]. **Sun et al. [3] developed a parameter-efficient method designed specifically for fine-tuning embedding models.** However, a systematic study of what parameter-efficient methods are optimal under what scenarios has not been performed, which is the aim of our work.\n\nAlso, starting from line 84: \n\n> **The single most related investigation is a concurrent work [4], where the scaling laws for encoder models for retrieval are investigated. There are significant differences in the settings we consider. Our work focus is investigating the process of fine-tuning decoder-only models for good quality embeddings.** Moreover, our main goal is to find which strategy for achieving good embeddings is optimal in a budget-restricted setting, while taking into account the popularity of applying parameter efficient methods for fine-tuning, like LoRA or partial model freezing.\n\n### References\n\n[1] Muennighoff: SGPT: GPT sentence embeddings for semantic search. arXiv 2022\\\n[2] Zaken et al.: BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022\\\n[3] Sun et al.: LLM-Oriented Retrieval Tuner. arXiv 2024\\\n[4] Fang et al.: Scaling Laws For Dense Retrieval. SIGIR 2024\n\n---\n*We hope that we adequately addressed the Reviewer’s concerns. If that is the case, we kindly ask for a reconsideration of the paper score.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hbv0JObNK5", "reviewer": "Reviewer_4Bw9", "summary": "This paper investigates how to effectively train text embedding models from pre-trained decoder-only language models while considering computational budget constraints. The authors explore the influence of model sizes, fine-tuning methods, and computational budgets on the performance of the embedding models. The results demonstrate that full fine-tuning is optimal for lower computational budgets, whereas low-rank adaptation fine-tuning is more effective for higher computational budgets.", "questions": "Please see my comments in Weakness.", "limitations": "The paper has a proper Limitations and future work section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) This paper focuses on the important problem of finding the optimal training settings for embedding models within a fixed compute budget.\n\n(2) This paper conducts extensive experiments across 8 model sizes, 6 compute budgets, and 4 tuning methods.\n\n(3) Although the paper uses contrastive loss as the primary measure of model performance, the authors show a strong correlation between contrastive loss and downstream task performance.", "weaknesses": "(1) In L51-52, it is stated that \"given a fixed computational budget, predicts the optimal network architecture, data quantity, and parameter-efficient fine-tuning hyperparameters\". However, data quantity, a very crucial factor, is not investigated in the paper.\n\n(2) The conclusion that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively seems somewhat superficial.\n\n(3) I have concerns about the selection of the MTEB subset. Retrieval is a crucial task for which embedding models can be used. However, only SciFact is evaluated. Scifact is not a representative retrieval dataset as its corpus is relatively small (only with 5183 documents).\n\n(4) Figure 1 is somewhat hard to read since it is difficult to distinguish the lines representing full fine-tuning, LoRA, and block freezing.\n\n(5) The average pooling method to extract representation is not entirely reasonable, as the autoregressive characteristic would cause the outputs of the first tokens to lack information about the latter tokens.\n\n(6) L141-146 is not entirely accurate. Hard negative mining methods like [1] and [2] do not require this two-stage training procedure. This two-stage training procedure is more of a recent trend in training powerful general-purpose embedding models.\n\n[1] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, Arnold Overwijk. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ICLR 2021.\n\n[2] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma. Optimizing Dense Retrieval Model Training with Hard Negatives. SIGIR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for detailed and informative feedback. Below we address each of the concerns.\n\n**(1) [Despite you mention so in lines 51-52] data quantity, a very crucial factor, is not investigated in the paper.**\n\nIn this paper, we focused on investigating the compute-bounded setting – the data-quantity factor is present implicitly as it can be inferred from the FLOP limit and the model size. Following your comment, we recognise that the data-centric perspective is also very valuable. For this reason, we *attach Figure 2* to our 1-page pdf file with plots. A simple and practical conclusion can be inferred from it – if the data is restricted, one should choose the biggest model available and use full fine-tuning or LoRA with a rank on the bigger side. We have updated our paper to contain this plot and conclusion.\n\nWe note that these conclusions can be achieved using the raw experimental data that we provide – please, see the link at the bottom of page 5 of our paper.\n\n**(2) The conclusion that full fine-tuning and low-rank adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively seems somewhat superficial.**\n\nWe do agree with the Reviewer that our study did not reveal patterns that are unexpected or peculiar. Nevertheless, we argue that our findings constitute a solid empirical grounding and, thus a concrete scientific contribution to helping researchers and practitioners working with embedding models. It is the first study devoted to establishing compute-efficient approaches to contrastive fine-tuning of decoder-only language models. Given the recent activity in this area, we found it to be a serious research gap that needed to be addressed.\n\nAlso, please note that our study concerns not only the selection of the optimal method from amongst the four pre-selected, but we also go deeper and analyse important hyper-parameters of those methods (LoRA rank and active block fraction). This was not previously studied, with a notable exception of limited analysis in [2].\n\n**(3) [...] Retrieval is a crucial task for which embedding models can be used. However, only SciFact is evaluated. Scifact is not a representative retrieval dataset as its corpus is relatively small [...].**\n\nOur selection of SciFact was guided by its strong correlation (0.927) with the overall performance in the retrieval tasks category, as evidenced by the Table 11 of MTEB [8]. Nevertheless, we concur with the Reviewer's observation regarding the size of the SciFact benchmark.\n\nTherefore, we now run new experiments to investigate the *correlations* between the performance on SciFact and two large retrieval benchmarks: *Natural Questions* [6] (2.6M test data points), and *DBPedia* [7] (4.6M test data points) for our best models for each (FLOP, method) combination, which are *0.976* and *0.971* respectively (also see Figure 3 of the attachment). We hence conclude, that despite its humble size, SciFact is a satisfactory representative of the retrieval category. This clarification has been added to the paper.\n\n**(4) Figure 1 is somewhat hard to read since it is difficult to distinguish the lines representing full fine-tuning, LoRA, and block freezing.** \n\nWe thank the Reviewer for the suggestion. We have updated the figure to make it more legible in Figure 1 of the attachment.\n\n**(5) The average pooling method to extract representation is not entirely reasonable, as the autoregressive characteristic would cause the outputs of the first tokens to lack information about the latter tokens.**\n\nWhile using the average pooling method may indeed seem unintuitive, it works very well in practice. In *Appendix D, section 3*, we compare the two most popular methods for extracting embeddings – average and last pooling methods – for Pythia 410m, fine-tuned with the FLOP budget of 9.6e+16. We perform evaluation on the whole MTEB [8] without MSMARCO, and we find the difference in performance is 0.47 vs 0.36 in favour of average pooling.\n\nWe note that average pooling has been used in several previous works [1,2,3]. Moreover, theoretically, the average pooling method has the possibility to converge to the ‘last’ method by zeroing out all the previous tokens. \nHence, we argue the average pooling method is empirically and theoretically justified. \n\n**(6) L141-146 is not entirely accurate. Hard negative mining methods like [4] and [5] do not require this two-stage training procedure [which] is more of a recent trend in training powerful general-purpose embedding models.**\n\nWe indeed ignored the existing approaches for mining the hard negatives. We have rewritten the mentioned paragraph as shown below (changes in bold) – if something is still imprecise, we would appreciate feedback!\n\n> […] which can be incorporated into the contrastive loss to promote learning more precise representations. **Moreover, there are works that focus on approaches for mining hard negatives which result in better training data in the context of specific downstream tasks [4,5].**\n\n> However, **recent works aiming at training powerful, general-purpose embedding models** that do rely on datasets with hard negatives often arrive at embeddings by having two distinct training phases […]\n\n### References\n\n[1] Li et al.: Towards General Text Embeddings with Multi-stage Contrastive Learning. arXiv 2023\\\n[2] Wang et al.: Improving Text Embeddings with Large Language Models. arXiv 2024\\\n[3] Wang et al.: Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv 2022\\\n[4] Xiong et al.: Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. ICLR 2021\\\n[5] Zhan et al.: Optimizing Dense Retrieval Model Training with Hard Negatives. SIGIR 2021\\\n[6] Kwiatkowski et al.: Natural Questions: A Benchmark for Question Answering Research. TACL 2019\\\n[7] Hasibi et al.: DBpedia-Entity v2: A Test Collection for Entity Search. SIGIR 2017\\\n[8] Muenninghoff et al.: MTEB: Massive Text Embedding Benchmark. EACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1dbB2p5swn", "reviewer": "Reviewer_zJJS", "summary": "This paper proposes a new algorithm to automatically find optimal configurations of model sizes, data quantities, and fine-tuning methods for text embedding models at different computational budget levels by contrastive pretraining. Specifically, the paper analyzes the choices of different configurations and then designs an algorithm to predict optimal architecture based on budget. The paper also designs a new task to find optimal model configuration given limited budgets. The paper first introduces basics, including the scaling laws, the constrained optimization problem, the contrastive learning,  fine-tuning methods, and then computational cost formulas. The paper then experiments with different finetuning methods to find the optimal loss regarding parameters. Furthermore, the paper aims to find the relationship between loss functions regarding the number of parameters (trainable/total) and the number of training tokens. The paper then fit the plots to find the best equations.", "questions": "See weakness", "limitations": "The paper includes a limitation section in Section 5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a new computation optimization problem: given a fine-tuning corpus and a family of pre-trained decoder-only language models of different sizes, minimize the contrastive loss subject to a fixed computational budget. The newly proposed problem is important and has the potential to expand.\n2. The paper conducts a comprehensive analysis of existing fine-tuning techniques with different hyperparameters. The paper visualizes the results with clear figures and concludes with several useful results. The paper also predicts the optimal loss for different budgets. The paper evaluates the downstream results.\n3. The paper provides code, additional evaluation results and analysis in the Appendix, and detailed computational resources.", "weaknesses": "1.  The paper's generalization ability is limited. As mentioned in the limitation, the proposed computation law is only on the Pythia; additional language models might better reflect the generalization ability. The paper also only focuses on contrastive finetuning.\n2. The whole paper mainly focuses on the performance analysis. It would be better to include additional theoretical support for the phenomenon that appears in the paper. The paper can also be strengthened by testing the proposed performance law on some unobserved configurations. \n3. The paper claims to also analyze the relationships between the data quantities and the performance. However, I cannot find it in Section 4.2. The paper fails to follow the NeurIPS citation format, which uses numbers for references/citations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their effort to assess our work! Below we address each of the concerns.\n\n**(1a) The paper's generalization ability is limited. As mentioned in the limitation, the proposed computation law is only on the Pythia; additional language models might better reflect the generalization ability.**\n\nWe agree that having additional models would allow for a better reflection of the paper’s generalisation. During the rebuttal we ran experiments with a very recent model – Gemma 2b [1], using LoRA and full fine-tuning methods. \n\nImportantly, we found that the **the loss formula with parameters fitted to the Pythia data points, adequately describes the behaviour of Gemma 2b training for the ‘medium’ budgets**. \n\nAt the extremely low computational budget, Gemma performs better than the prediction, which intuitively is caused by the general higher model capability than those in the Pythia suite; also, scaling laws tend to be much less regular in smaller regimes – see Figure 4 in the pdf attachment.\n\nAs a result of being time-restricted, we did not perform the two highest budget trainings, but we believe that the results so far are a strong indication as to the generalisability of our findings. We will update our paper to contain those new findings once we have the complete results for Gemma.\n\n**(1b) The paper also only focuses on contrastive finetuning.**\n\nWe intentionally focus only on contrastive learning: this has become a standard approach to obtaining useful embedding models. We note that there has been no study before ours that helps to select the most compute-efficient variant of contrastive learning approaches in decoder-only models, rendering our study a worthwhile contribution.\n\n**(2) The whole paper mainly focuses on the performance analysis. It would be better to include additional theoretical support for the phenomenon that appears in the paper. The paper can also be strengthened by testing the proposed performance law on some unobserved configurations.**\n\nWe provide theoretical support by utilising a classical risk decomposition model (similar to that of the Compute-Optimal Scaling Laws paper [2]), and adapting it to account for the difference in forward and backward computation parameters in Section 4.3. With this theoretical framework we describe the loss behaviour in terms of the model parameters (both forward and backward), and the amount of data.\n\nWe also show that the scaling law generalises to the very recent Gemma 2B model (see above).\n\n**(3a) The paper claims to also analyse the relationships between the data quantities and the performance. However, I cannot find it in Section 4.2.**\n\nIn this paper, we focused on investigating the compute-bounded setting – the data-quantity factor is only present implicitly as it can be inferred from the FLOP limit and the model size. However, we now realize that the data-centric perspective is also very valuable and we should analyse it explicitly, especially in the light of the claim in line 157. For this reason, we *attach Figure 3* to our 1-page pdf attachment with plots. A simple and practical conclusion can be inferred from it – if the data is restricted, one should choose the biggest model available and use full fine-tuning or LoRA with a rank on the bigger side. We have updated our paper to contain this plot and conclusion.\n\nWe note that these conclusions can be achieved using the raw experimental data that we provide – please, see the link at the bottom of page 5 of our paper.\n\n**(3b) The paper fails to follow the NeurIPS citation format [...].**\n\nThank you for noticing it! This has already been fixed.\n\n### References\n\n[1] Mesnard et al.: Gemma: Open Models Based on Gemini Research and Technology. arXiv 2024\\\n[2] Hoffmann et al.: Training compute-optimal large language models. arXiv 2022\n\n---\n*We are grateful for the Reviewer’s feedback and hope the concerns are appropriately addressed. If that is the case, we kindly ask you to reconsider the paper score.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tg2bEczHgj", "reviewer": "Reviewer_xguk", "summary": "The paper fine-tunes Pythia LLMs with different sizes for information retrieval (IR) applications. The paper investigates the effect of different model sizes, FLOPs, fine-tuning methods such as LoRA or bias tuning, and some important hyperparameters (e.g., dimension in LoRA) on loss and downstream applications (measured by MTEB). Some takeaways are interesting and counterintuitive (e.g., the larger model does not necessarily achieve better loss).", "questions": "1. Although you show that the correlation between MTEB and loss is very high (−0.892) overall, some important trends seem to be different, which might affect your takeaway message. For example, the best score of MTEB always comes from the largest model in Figure 7-10, but the lowest loss often comes from smaller model size in Figure 2 and 4. The bias tuning methods are significantly worse in terms of loss, but it seems to achieve the best MTEB when FLOP is 5e+18 in Figure 11. When you write your takeaways, why not also include the results of MTEB?\n\n2. I don't understand l_r and l_c in line 134 and 135. Why don't we need to use labels.T for l_c?  If this is a standard loss, please cite. Otherwise, please provide more explanations/motivations for this loss.\n\n3. Why do you choose to use the English partition of the BAAI BGE dataset, which is designed for Chinese embedding? If it is not a common choice, you should explain more about this.\n\n4. Did you get the performance of full MTEB? Does the full MTEB give you different trends? In the figures, MTEB performances means the performances on the task subset, right? I think you should make it clear in those figures (MTEB performances -> MTEB subset performances).", "limitations": "The current limitation section is good and adding some discussion of inference cost could make it better.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper conducts very extensive experiments and analyses in the main paper and appendix. I appreciate all the efforts and believe that it would be very useful for practitioners who want to create their own embedding models from LLM.", "weaknesses": "Although I think the experiments in this paper are valuable, I think main goal of the analyses is not very practical. Personally, I think the term Compute-Optimal from Chinchilla (https://arxiv.org/pdf/2203.15556) and this paper is very misleading because it only considers the training cost and ignores the inference cost. In practice, the LLMs are often trained much longer than this \"optimal\" value. For example, Chinchilla-optimal for 8B LLM is just 200B tokens, but LLaMA 3 is trained using over 15T tokens (https://ai.meta.com/blog/meta-llama-3/). For the information retrieval (IR) models, the inference cost might be even more important compared to LLMs.\n\nIn addition to main complaint above, I think that some details could be explained more clearly (see the questions below). Finally, I think that the figures with FLOPs on the x-axis and different curves are for different model size are much easier to read than what the paper has now (the model size on the x-axis and different curves are for different FLOPs), but it might be my personally preference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their insightful feedback! Addressing your questions and concerns:\n\n**(W1) The term Compute-Optimal [...] is very misleading because it only considers the training cost and ignores the inference cost [which is important in practice].**\n\nWe do agree that our usage of the “compute-optimal” term is somewhat imprecise – we have updated the paper, pointing out that we mean the training cost only. We have also added an appropriate paragraph in the limitations section.\n\nWe believe that there is value in deriving the functional relationship between the model size, training method, data, and the final training loss (which is also done in many previous works on scaling laws). This relationship clearly characterises the ingredients needed to achieve particular performances. We refrain from taking inference costs into account, as they are hard to estimate. However, we stress that by having the functional form, the reader can analyse their particular use case, which may take into account the sum of the training and inference cost as a constraint, and find the optimal iso-total-cost models. \n\n**(W2) I think that the figures with FLOPs on the x-axis and different curves are for different model size are much easier to read than what the paper has now [...]**\n\nThank you for this suggestion – to increase the readability, we have added plots where the x-axis is the model size, not FLOPs. During experimentation we were looking at both versions and decided that it is better to present loss vs size plots as the IsoFLOP profiles in them directly answer our motivating question: given the FLOP budget, what is the optimal model size one should use?\n\n**(Q1) Although you show that the correlation between MTEB and loss is very high (−0.892) overall, some important trends seem to be different, which might affect your takeaway message. [...]**\n\nWe thank the Reviewer for raising this. Ultimately, the most suitable metric to use is the one most relevant to the reader. We believe people who have the need to fine-tune their own embedding models and the resources to do so should use their own benchmarks to derive the scaling laws most accurate for them. Without access to the relevant benchmark, we find the validation loss to be the smoothest metric to base scaling laws upon (again, this approach is typically assumed in the scaling laws works). The MTEB results have higher variances and do not necessarily align with the user’s requirement. \n\n**(Q2) I don't understand `l_r` and `l_c` in line 134 and 135. Why don't we need to use `labels.T` for `l_c`? If this is a standard loss, please cite. [...]**\n\nIt is a standard symmetric contrastive loss used throughout the contrastive learning literature for training textual embedding models [3,4,5,6] (and in other applications [7]). The specific formulation included in our paper originates from [3] (Section 2.2). We cite this work (please see line 126), but we will emphasise it more to avoid confusion – thank you for your suggestion!\n\nAs for the question about `labels.T` for `l_c`: it should remain `labels` – we just use PyTorch notation here; the first argument to the cross entropy function is a matrix, and the second argument is a vector indicating for each row what is the index of the “correct” element. (We link the documentation in the footnote on page 3.)\n\n**(Q3) Why do you choose to use the English partition of the BAAI BGE dataset, which is designed for Chinese embedding? [...]**\n\nThe reason for choosing BAAI BGE is that this dataset is a massive collection of data for training embeddings. Since we do not consider scaling laws under data constraints like [1], we needed a large enough dataset on which none of our experiments would exceed 1 epoch of training; BAAI BGE was the only published dataset satisfying our constraints.\n\nAs for why we used only the English partition of BAAI BGE: we want to explore a family of decoder-only models with different model parameter scales, and the Pythia family is the only available option at the time of experiments that offers lots of sub-billion-parameter models, which are common for embedding models. The Pythia family was pre-trained on The Pile [2], which is an English-only dataset, and hence it does not have Chinese knowledge and capabilities. Therefore, we only focused on the English partition of BAAI BGE. \n\nWe have updated the paper to include the full explanation provided here.\n\n**(Q4) Did you get the performance of full MTEB? Does the full MTEB give you different trends? [...] you should make it clear in those figures [that only a subset of MTEB was used].**\n\nThe Reviewer is correct here—for practical computational purposes, we did use a subset of MTEB instead of the whole benchmark. (As we explain in Appendix B, this subset was selected in a principled way to be representative of the whole MTEB.)\n\nDuring the rebuttal period, we checked that the results on the whole suite (except MSMARCO, which is the most expensive to run) are highly correlated with our selected subset. To this end, we compared the two evaluations for 8 checkpoints from the full fine-tuning experiment, and the Pearson correlation is 0.976. This information has been included in the revised version of the paper. We have also fixed the captions to “MTEB subset performance.”\n\n### References\n\n[1] Muennighoff et al.: Scaling data-constrained language models. NeurIPS 2023\\\n[2] Gao et al.: The Pile: An 800GB dataset of diverse text for language modeling. arXiv 2021\\\n[3] Neelakantan et al.: Text and Code Embeddings by Contrastive Pre-Training. arXiv 2022\\\n[4] Wang et al.: Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv 2022\\\n[5] Wang et al.: Improving Text Embeddings with Large Language Models. arXiv 2024\\\n[6] Günther et al.: Jina Embeddings: A Novel Set of High-Performance Sentence Embedding. arXiv 2023\\\n[7] Eysenbach et al.: Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference. arXiv 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the Reviewers for their effort and constructive feedback. We believe that it will lead to a substantially better version of our paper. 

We are pleased that the Reviewers found
* the research question our work addresses relevant and practical [GCkf, 4Bw9, zJJS, xguk],
* our experimental setup extensive and robust [GCkf, 4Bw9, xguk],
* the conclusions drawn insightful [GCkf, xguk].

We are also grateful for all the constructive concerns that have been raised and make a substantial effort to address them all. Below is a summary of the main points of our rebuttal, whereas detailed responses are provided below for each individual review.

---

## Summary of the rebuttal

* We provide responses – largely supported by new preliminary experimental results (see below) – concerning all the issues raised by the Reviewers, most importantly:
  * transferability of our conclusions to other models [zJJS],
  * omitting the explicit analysis of the data-constrained setting in our paper [4Bw9, zJJS],
  * the robustness of our evaluation [xguk, 4Bw9].
* We resolved to update our paper for the camera-ready version with new experiments, namely: 
  * the results from training the Gemma 2b model,
  * the evaluation results on a more realistic retrieval dataset.
* We improved the text of the paper in a few places:
  * we added new related work (kindly pointed out by Reviewer GCkf),
  * we made the discussion about the hard negatives more accurate (per suggestion of Reviewer 4Bw9),
  * we extended our limitations section with a paragraph admitting that our study remains inference-cost-agnostic (thanks to the comment by Reviewer xguk).

---

## New experimental results

During the limited rebuttal time, we have managed to run a number of experiments regarding the raised concerns: transferability of our conclusions to other models [zJJS] and our MTEB subset choice for evaluation [xguk, 4Bw9]. We give details in the answers to the reviewers indicated in the brackets. In a nutshell:
* We found that for a new model – Gemma 2B – for two tested methods (full fine-tuning and LoRA), the results follow the same pattern as for our Pythia experiments.
* We found that there is a very strong correlation between our selected subset and the whole MTEB evaluation.
* We also confirmed that regarding the retrieval task specifically, the results on the SciFact benchmark that we used are highly correlated with much larger retrieval benchmarks.

---

We believe that this makes our paper stronger, and again, we are grateful for the Reviewers’ suggestions!

Meta-review: This work investigates how to contrastively train text embedding models in a compute-optimal manner using a suite of pre-trained decoder-only language models. The motivation is solid, and the quality of the work is high. However, one limitation is the relatively small model sizes used in the study, with a maximum of 3B parameters. It would be more compelling to explore the scaling laws for larger models, especially considering that the leading embedding models today are typically around 7B parameters e.g., NV-Embed, bge-en-icl, SFR-Embedding.
https://huggingface.co/spaces/mteb/leaderboard
Decision: Accept (poster)