Effective post-training embedding compression via temperature control in contrastive training
Authors: Georgiana Dinu
Corey D Barrett
Yi Xiang
Miguel Romero Calvo
Anna Currey
Xing Niu
Venue: ICLR 2025 Spotlight (2025)
Keywords: representation learning
embeddings
text retrieval
nlp
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Fixed-size learned representations (dense representations, or embeddings) are widely used in many machine learning applications across language, vision or speech modalities. This paper investigates the role of the temperature parameter in contrastive training for text embeddings. We shed light on the impact this parameter has on the intrinsic dimensionality of the embedding spaces obtained, and show that lower intrinsic dimensionality is further correlated with effective compression of embeddings. We still observe a trade-off between absolute performance and effective compression and we propose temperature aggregation methods which reduce embedding size by an order of magnitude with minimal impact on quality.

Published as a conference paper at ICLR 2025
EFFECTIVE POST-TRAINING EMBEDDING COMPRES- SION VIA TEMPERATURE CONTROL IN CONTRASTIVE TRAINING
Georgiana Dinu, Corey Barretti; Yi Xiang, Miguel Romero Calvo, Anna Currey, Xing Niu
Amazon, USA
{gddinu, yxxan, miquelre, ancurrey, xingniujee amazon. com +Oracle, USA
corey Corry..aarettee www.bbbrrttttrrr.eccoo
ABSTRACT
Fixed-size learned representations (dense representations, embeddings) are widely used in many machine learning applications across language, vision or speech modalities. This paper investigates the role of the temperature parame- ter in contrastive training for text embeddings. We shed light on the impact this parameter has the intrinsic dimensionality of the embedding spaces obtained, and show that lower intrinsic dimensionality is further correlated with effective compression of embeddings. We still observe trade-off between absolute perfor- mance and effective compression and we propose temperature aggregation meth- ods which reduce embedding size by an order of magnitude with minimal impact quality.
INTRODUCTION
Fixed-sized learned representations (a.k.a,, dense representations, or embeddings) the backbone of many machine learning applications across language, vision, audio, and other modalities. Em- beddings play important role in retrieval applications, where given
input, vector similarity is used to retrieve relevant entries from previously indexed vector store. This in turn is an essential component of eerieval-auggeent crease the accuracy and capabilities of generative models. Beyond retrieval, embeddings also have applications in classification tasks, clustering, or parallel corpus mining, to name just few.
generation (RAG; Lewis al., 2020), which uses retrieval
Most embedding models are trained using form of contrastive loss, either in the entirety of train- ing or after an unsupervised pre-training stage. Contrastive loss functions minimize the distance between an anchor point and other similar data points, while maximizing the distance to set of negative examples. For training text embedding models, InfoNCE (van den Oord et al. 2018 is the standard contrastive loss used (see equation
The temperature parameter (r in equation D plays an important role in embedding model training, particularly as it modulates the model's sensitivity to difficult negatives. It has been shown that smaller values of lead to more uniform distribution in the embedding space (i.e., data points are further apart), while larger values lead to lower uniformity but better alignment (i.e., similar data points are closer together) (Wang & Liu 2020). In parallel, smaller r and high uniformity have been found to be correlated with better instance-wer wise discrimination, whereas lower uniformity and better alignment are correlated with good group-wise discrimination (Kukleva et al. 2023). In this paper, build on this work, performing an in-depth analysis of the effects of T on text embeddings as well as on post-training embedding compression.
Interaction between temperature and tasks perature on different text embedding tasks, where we specifically observe a trade-off between per- formance on retrieval and on clustering tasks a function of r. Diving deeper, we show that
We start off by investigating the impact of the tem-
Work done while Amazon.

---

Published as a conference paper at ICLR 2025
temperature directly impacts the intrinsic dimensionality of the embedding space and that lower temperatures lead to greater intrinsic dimensionality.
Impact of temperature on post-training compression These observations further raise the ques- tion of whether intrinsic dimension and uniformity properties of embedding spaces impact the abil- ity to efficiently apply post-training compression techniques. We test the hypothesis that smaller intrinsic dimensionality improves quality retention post-training embedding size reduction (or compression). Embedding compression is essential for reducing costs in practical applications; par- ticularly in retrieval, compression advantageous because it allows trained models to flexibly adapt to different storage requirements. We show that, as hypothesized, higher temperature does indeed correlate with higher quality retention after compression, and that the correlation holds for two dif- ferent methods of compression: random feature selection or truncation (Kusupati et al. 2024). and quantization via binarization (Bai et al. 202I).
Techniques for balancing temperature across tasks Finally, while these observations shed light on the impact of temperature in contrastive learning, they also confirm the difficulty of training an embedding model that is optimal across multiple tasks and requirements: absolute retrieval results are at odds with quality retention when compressing embeddings. To remedy this, we propose several methods for temperature aggregation and specialization that achieve the best of both worlds; we are able train models that match the performance of the best single model in retrieval, while showing 99% quality retention when the dimensions are quartered and when the embeddings is binarized.
2 PREREQUISITES AND RELATED WORK
Contrastive losses in representation learning Most embedding training algorithms employ form of contrastive learning, where an anchor point (a word token, sentence, image, etc.) is brought closer positive examples and sent further away from negative examples. Examples of positive data points are co-occurring words for word embeddings (Mikolov et al. 2013), similar sentences or neighboring sentences for sentence representations (Kiros et al., 2015| Devlin et al. 2019), or noisy versions of the anchor image in vision!
Several variations on contrastive losses have been proposed, most of them inspired by noise con- trastive estimation introduced by MMih & Kavukcuoglu (2013) and Gutmann & Hyvärinen (2012). Here, we use InfoNCE (van den Oord et al. 2018). the contrastive loss most commonly used in embeddings for text and image retrieval. InfoNCE is defined as:
(MW)), D1/T
LiufONCE 7) Be, (oppps
log {(( D/T + Ex- D)/T
(1)
where f is the embedding function to be learned, and x is an anchor point compared with a positive x and set of negatives x- via similarity function We choose to be cosine similarity, which has been repeatedly shown to outperform Euclidean or plain (un-normalized) scalar product for both language and vision applications. This is further scaled by the temperature parameter 1/7.
Many studies have found that the contrastive power of InfoNCE and similar losses increases with the number of negatives and the difficulty of negatives (Lazaridou et al. 2015 Tian et al. 2020; Khosla et al. (2020). In addition to this, it has been shown that InfoNCE loss itself, when used with normalized representations (i.e., using cosine similarity), performs hard negative mining implicitly (Khosla et al. (220).. More precisely, it upweights the negative samples that are further away from the anchor the embedding space. However, whether these are indeed difficult samples or false negatives depends on the supervision signal.
Impact of temperature The temperature parameter is commonly tuned in InfoNCE and related losses in order to obtain better representations. Several studies have shown that temperature further modulates the impact of difficult samples.
For full review of this work, please see surveys such as Kashyap et al. 2024
2

---

Published as a conference paper at ICLR 2025
Khosla al. (2020) and Wang & Liu (2020) showed that the difficult negatives have large gradient contributions in InfoNCE. They further investigated the relationship between InfoNCE and other losses when varying т. When + 0+, it becomes triple loss with margin of 0 using only one negative, the nearest negative:
(J. max
maa(s( ((=*)) s(fx),
where x is the negative that is maximally similar to the anchor point œ. When T + oc, InfoNCE converges the simple loss, which uses all negatives but does not apply a softmax transformation:
simple (J,z)= -)((= - (()))
In a similar vein, Robinson al. (2021a) show that one can upweight the difficult negatives by introducing a new concentration parameter 3 in the normalizer of the InfoNCE loss. Empirically, this acts the same fixing the temperature parameter used for positive samples and varying the temperature used for negatives, where 3 = 1/T. They show that large / leads to upweighting the contribution of the difficult samples, which corroborates the previous results.
One way to visualize this effect is to measure the impact that each negative sample has on the loss normalization factor. The plots in Figure[]show this impact in two settings: one with random vectors, which is similar to the beginning of training, and mid-training setting, in which the vectors are drawn from real batches when models are trained with different т values (see Section 3 for training details). We use 1024-dimensional embeddings and batch size of 100 negative samples.
X tau 0.400
tau 0.400
tau 0.300
0.200
0.200
tau 0.110
tau 0.100
lau 0.050
0.050
0.0085
0.0090
0. 0.0095
0.0100
0.0105
0.0110
0.000 0.025 0.050 0.075
0.125 0.150 0.175 Impact of individual negatives on the InfoNCE loss
0.100
Impact of individual negatives on the InfoNCE loss
Figure 1: The impact of individual negative samples, computed relative to all negatives in the batch, under different temperature settings. Left: using random vectors; Right: using vectors mid mid-training.
When the vectors random the effect is very clear: when 0.4, most data points contribute similarly to the loss, and as t decreases, some negatives become outliers, with their contribution being either enhanced diminished. When the vectors are drawn from real training batches mid- training, the distributions look similar for all т, with the exception of outliers: with smaller 7 we see that the difficult negatives have larger impact (note that the batch is identical for all models).
Choosing the optimal temperature parameter In computer vision, several works point to per- formance trade-offss when choosing small vs. large temperature values; despite difficult negatives being preferred in general, a small temperature is not universally superior. Waag & Liu (2020) highlight that while temperature modulates the impact of difficult negatives, there are other ways to do so, such as by explicitly selecting difficult negatives. As such, large T value, which weights all negatives the same way, can be coupled with difficult negatives, leading better performance overall. Also in vision, Kukleva et al. (2023) show that smaller temperatures are better at modeling tail classes while larger temperatures work best for the head of the distribution. They summarize this trade-off between instance- and group-wise discrimination, where smaller temperatures favor finer- finer-grained instance-level discrimination. As a solution, they propose varying the tempera- ture during training and show optimal results when large range is used, with from 0.07 to 1.0. Similarly, Robinson et al. (2021b) empirically show that larger temperatures promote the learning of easy features, while other features that could provide finer- merr-grained distinctions are ignored.
As way of explaining these results, several studies (Wang & Liu 2020 Wang & Isola 2020) relate their finding the observation that contrastive losses navigate trade-off between uniformity and
3

---

Published as a conference paper at ICLR 2025
alignment as defined in Wang & Isola (020). They show that when the number of negatives M converges to infinity, the normalized loss (-log M) converges to:
lim T(f,r) - log M = - _ B (fxx)² f(xt)] M-linf
Alignment
(4)
+ Exr-ratta loog
(e HTTT
Uniformity
Paata
Alignment measures the similarity of the anchor to the positive points, while uniformity measures the similarity to the negatives. The goal is to maximize distances between arbitrary points (have uniform space) while improving alignment.
TEMPERATURE IN CONTRASTIVE TRAINING OF TEXT EMBEDDINGS
Inspired by the performance trade-offs previously observed in computer vision tasks, we first inves- tigate the impact of temperature in training text embeddings.
3.1 EXPERIMENTAL SETUP
Text embeddings are typically trained to be able to solve a range of applications, from retrieval to classification and clustering. This is achieved by employing two or more training stages: in first stage, training is performed on large amounts of raw text using a reconstruction objective such as masked language modeling (MLM; Devlin et al. 2019). This is followed by stages where supervision consists of a mix of explicit contrastive data, examples including question answering data, semantically similar sentences or documents, documents and their summaries, and so on. We follow the same procedure here, employing two-stage training. We start from a fixed model pre-trained using the MLM objective. We then follow this with InfoNCE contrastive learning; this second stage is the only training stage that we vary in the following experiments.
more other
Model architecture We use the CodeSage architecture introduced in Zhang et al. (0024) Our models are trained with 356M parameters and 1024 embedding dimensions. Additional training parameters are given in Appendix
Data For the MLM training stage, we use 2T tokens of data consisting of 80% English data, with the remaining 20% spanning over 100 languages. We tokenize the text with tiktoken build 100k For the contrastive stage, we train on MS Marco (Bajaj et al. 2018 WWan al. 2023) NQ (Karpukhin et al. 2020; Gao & Callan 2021), NLI (Gao et al. 2022), ). HotpotQA (Yang et al. 2018), FEVER (Thorne et al., 2018), MIRACL (Zhang al.) (023), and Mr. TyDi (Zhang et al, 2021I, totaling approximately 2 million data points (see details in Appendix (A). We use the training splits of these datasets released by TTaakur et al. (021). We use in-hatch negatives and a batch size of 256 data points. Negatives are obtained by randomly sampling other data points from the same training set (homogeneous sampling). Sampling from the same dataset is common technique used to ensure that the in-batch negatives are not too easy. If a contrastive negative example is explicitly provided with the training data set, that data point it is added to the batch of negatives. Additionally, we assign weights to the negative samples based on similarity to the anchor, following Dong et al. (204).
vocabulary Fl
Evaluation We evaluate using the standard English MTEB benchmark (Muennighoff et al. 2023), which contains a total 56 datasets categorized into eight tasks tasks@olaasificatiinn (12 datasets), clustering (11), pair classification (3), re-ranking (4), retrieval (15), semantic text similarity (STS; 10) and summarization (I). We evaluate on all these tasks but focus on retrieval and clustering, which are evaluated using nDCG@ @ 10 and v-measure, respectively.
Note on MTEB instruction models is common in the literature to test embeddings using addi- tional information regarding the downstream task or the text to be embedded, in order to reduce the inherent ambiguity of modeling different tasks (Wang et al. 2024a). Common procedures include
using different prefixes for different tasks, or using free-form text (instructions; Su et al, 2023
Available hotpttp://// huatts/ss/aannnggge.co//////sssr
1clk base, available at ht http:ps//gtthhl...ooooo/////////
"A leaderboard in maintained hatnmmnkkkkkkkeeer///gggggiggggo

---

Published as a conference paper at ICLR 2025
Wang et al. 2024b; Muennighoff et al. 2024). However, when used in MTEB evaluations, it is often the case that: 1) instructions are individually tailored for each of the 56 MTEB test sets and 2) test set instructions this runs the risk of MTEB over-fitting and thus goes against the original goal of the MTEB bench- mark, which is robustness across different tasks. For the purposes of this study, do not instructions at train or inference time, similarly to models such as Neelakantan et al. (2022).
paired with the train ones, thus mapping train to test sets. We believe
3.2 PERFORMANCE WHEN VARYING TEMPERATURE
We start by measuring the impact of the т (temperature) parameter by varying t from 0.04 to 0.4 and measuring performance MTEB
Results for retrieval and clustering are shown in Figure 2 while results for the other MTEB tasks provided in Appendix B We separate retrieval and clustering from the other tasks due to the interesting trend we observe: retrieval performance monotonically decreases with 7 while clustering shows the opposite trend. This result seems to confirm the hypothesis that temperature affects group vs. instance-wise discrimination and that there is traded-off between the two.
50.0
retrieval
47.5
46.73
45.68
46.12
45.92
clustering
45.0
42.96
42.5
Performance
40.0
39.45
25
39.30
37.75
37.5
36.56
35.91
35.10
35.0
34.15
32.5
30.0
0.04
0.06
0.08
0.10
0.20
0.30
0.40
Temperature
Figure 2:
MTEB performance Retrieval and Clustering tasks, function of the temperature value used in contrastive learning.
Building on previous work, we attempt to characterize the distinction between embedding spaces obtained when using different temperature values. For this purpose form t-SNE projection of the embedding spaces by randomly sampling 5000 datapoints and plotting their 2-D representation (van der Maaten & Hinton 2008). We further implement uniformity met- ric similar to the definition in Equation | we modify it to use the normalized scalar product instead, order to match our InfoNCE objective which uses cosine. In both cases we sample datapoints from the RedditClustering test set in MTEB; however, results are consistent across other data sets.
techniques. We per-
1-SNE projections and uniformity values different temperatures are shown Figure 3 t-SNE projections indicate the appearance of better defined clusters as 7 increases, which is similar to ob- servations made in previous work for image embeddings. However, the uniformity metric increases when goes from 0.04 to 0.06, but shows slow downward trend after that. Wang & Liu (2020) also measure uniformity and note that uniformity decreases with large т; however, the trend does not hold when negatives selected to be more difficult. Our results also indicate that there is no monotonic relationship between temperature values and uniformity as defined this way. Indeed,
We were unable to train strong models with lower temperature values likely due to numerical instability, phenomenon also reported by Khosla al. 2020

---

Published as a conference paper at ICLR 2025
0.90
Unifarmity
0.94
0.92
0.88
0.86
0.00
0:00
0.3
04
Temperature: 0.06
Temperature: 0.2
Temperature: 0.4
Temperature
Figure 3: Embeddings projected into 2 dimensions using t-SNE and uniformity values across differ- ent temperatures.
uniformity measures mean cosine similarities: if the space becomes more clustered as t-SNE visual- izations indicate, this metric can increase can decrease.
Based on the t-SNE visualization and clustering performance, hypothesize that the temperature likely modulates the intrinsic dimensionality of the resulting embedding space, and vector space that more "clusterable" has a lower intrinsic dimensionality. We investigate this by adopting measure of intrinsic dimensionality based on PCA, which identifies principal components that explain the variance the data. We perform PCA on embedding spaces and plot the variance explained by each principal component. Furthermore, components required to explain a certain amount variance in the data. We use this number as a metric of intrinsic dimensionality and set the variance threshold to 95%.
compute the number of principal
Figure shows the explained variance of the first 300 principal components under different temper- ature values and plots the intrinsic dimensionality defined above. The results indeed show the intrinsic dimension decreasing when larger is employed.
0.030
700
Tamp 0.2
650
0.025
Rand
650
Biolained variance
0.020
600
0.015
509
430
0.000
350
Principal component
Temperature
Figure 4: Impact of temperature on the intrinsic dimensionality of the embedding spaces as com- puted by applying PCA to 5000 datapoints. Left: Explained variance for each principal component. Right: Intrinsic dimensionality calculated as the number of principal components needed to explain 95% of the variance in the data; this number is 896 for random vectors.
4 POST-TRAINING EMBEDDING COMPRESSION
The previous section showed that the temperature parameter impacts the intrinsic dimensionality of the resulting space, thus explaining why clustering results improve with large т. This result brings up a related question: can embeddings with lower intrinsic dimensionality compressed more efficiently?

---

Published as a conference paper at ICLR 2025
Compressing embeddings Compression is essential to reduce computational costs associated with storing embeddings for retrieval applications. Furthermore, flexible compression, meaning embeddings that perform optimally at full size well as when compressed, offers more ad- vantages. When training representations, the computational constraints for each downstream task are often unknown, and in such cases fixed-capacity representations may be complex too simple for the task at hand. This section focuses evaluation on retrieval datasets due to the significant costs associated with storing embeddings (Shakir al. 2024; ; other tasks are discussed in Appendix B
For flexible compression, simple, computationally efficient operations are preferable: dimensionality reduction via vector truncation (Kusupati et al. (2024) and quantization (Bai et al. [021 Zafrir al., 2019) are two such methods to compress embeddings. We investigate random feature selection and binarization (quantization to bit) using the sign function.
Random feature selection We expect embedding spaces with low intrinsic dimensionality to retain their quality when dimensionality reduction methods are applied: we hypothesize that this also holds for simple methods such as random feature selection due to the fact that low intrinsic dimensionality implies more redundancy in the feature space. In these experiments, we truncate vectors x to X[1:k] with & E {256, 512]. This reduces the size of the vectors by a factor of d/k where d is the original size of the embedding space (1024 in our experiments)
Binarization We perform post-training binary quantization using the commonly used sign function. This leads to representations that require only d bits for storage. This is more aggressive com- pression, reducing storage by 32x assuming 32-bit original precision. For binary embeddings. we replace cosine similarity with Hamming similarity (the number of positions where the two vectors are identical), as recommended by previous work.
102
256
46
45.58
45.68
15753
45.52
15'37
100
99.02
512
44.53
Retertion dm performance
90.55 99
58.05
44.28
98-40
58 19.42
43.18
97.54
Performree
96-
40-
91.34
36
0.04
6.05
0.08
0.30
0.20
0.30
0.40
0.04
0.06
0.08
0.10
0.20
0:30
0.40
lempar ature
Temperature
Figure 5: MTEB Retrieval performance when embeddings are truncated: Absolute performance (left) and quality retention computed relative to full-size performance (right). In terms of absolute performance, full dimensionality (1024). retrieval scores decrease as 7 increases. However, after truncation, optimal performance is obtained with larger 7 values of 0.08 and 0.1, due to the increase in retention as shown in the right figure.
Results for random feature selection are shown in Figure 5 We show both absolute performance as well as quality retention measured with respect to full size embeddings (1024 dimensions). Larger temperatures lead to increased quality retention when reducing the embedding size. The effect is more pronounced when truncating 256 dimensions, where the quality retention goes from 93% with - 0.04 to 99% for T = 0.4.
Figure shows a similar trend in the case of binarization, with larger 7 promoting better retention. This is surprising given that lower temperatures lead to embeddings that are sprea more widely. For example, Kukleva et al. (023) discretize an embedding space into 500 bins and look at the distribution of embeddings across these bins. Lower temperatures show more uniform distribution across these bins, which would have suggested better quantization properties. However, empirically we observe the opposite effect.
Forr the rest of the paper will continue the terms truncation and random feature selection interchange- ably they are identical in this work.
7

---

Published as a conference paper at ICLR 2025
Porformance 100 clmendirr
64:50
46.12
44.06
44.91
45.92
Retombon of pression performacee
06
93.00
90.27
03:08
07.79
97.85
04.59
95.06
97.75
95, 00
64
80
66,82
90.47
89.71
42-
87:10
833
$4,14
40
80
38-
Binary
512 aiary
23b-5iarr
0.04
0.05
0.08
0.30
0.20
0.30
0.04
0.06
0.08
0.10
0.20
0.30
0.40
Temparature
Terpeeetuure
Figure 6: MTEB Retrieval performance when embeddings binarized: Absolute performance (at 1024) and quality retention computed relative to full precision performance (at 256, 512 and 1024). Same as with truncation, peak absolute performance is obtained with the lowest temperature in the un-compressed case; however mid-range values are optimal after binarization due to the increase in quality retention associated with larger T (right-hand side figure).
Finally, when comparing the two methods of reducing embedding size, binarization is far superior: with т 0.1 it achieves 32x size reduction with 97.8% quality retention versus 4x with 96.4% quality retention when truncating to 256. When the two methods are combined, despite obtaining reduction of 128x in size, quality drops significantly with a retention of 87% for the same 0.1.
Matryoshka representation learning (MRL) The prior experiments tested if larger temperatures lead to embeddings that can be compressed better for free, meaning without training aware algo- rithm. We perform additional set of experiments, this time using MRL (Kusupati et al.) 2024), which is a simple method that has been shown to work well practice. MRI uses a loss function that additionally optimizes for smaller vectors obtained by truncating the original vectors:
(f,L(f,Z,T) >d Ae (f (fp.ad.)
(5)
where S[1.d.] truncates vectors to dimensions de S d, the original dimensionality of the space. We set & = and d, to [256, 512, 1024], and we use uniform A weights.
Results are shown in Table We first observe that MRL training improves over the best retrieval results even for full size embeddings. As expected it also improves both absolute performance and quality retention at 256; this effect is more pronounced for smaller т, but it is present across the entire temperature range. Interestingly, MRL training also improves binary performance, but only small values, taking quality retention from 95.3 96.1 for 0.04. Finally, it is interesting to observe that MRI. training lowers the intrinsic dimensionality of the space across all temperature values. This further strengthens the observed correlation between lower intrinsic dimensionality and improved truncated representations.
Overall, the main findings of this section can be summarized as:
Larger temperature values in InfoNCE lead to lower intrinsic dimensionality of the result- ing space; however, the mechanism through which this happens is still not clear. Larger temperatures lead to all negative samples contributing similarly to the loss. It may be the case that multiple (and heterogeneous) negatives lead larger number of smaller up- dates in the embedding space, subsequently resulting in redundant encoding of information across dimensions. In future work, we plan to investigate this hypothesis and experiment with different batch sizes and methods for negatives selection.
2. Lower intrinsic dimensionality correlated with better quality retention when embeddings are compressed. This holds both for random feature selection and for binarization.
3. MRL training. which promotes embeddings that can be truncated with minimal perfor- mance loss, further reduces the intrinsic dimensionality of the resulting space. This happens across different temperature values and leads to better quality retention when compressing with both truncation and binarization.
8

---

Published as a conference paper at ICLR 2025
Table 1: MTEB Retrieval performance across different values for InfoNCE (Baseline) and MRI. objectives. Full performance (first row) stands for performance at full dimensionality (1024) and at full precision. The rest of the results reduce embedding size by truncating 256 (while keeping full precision) and binarizing (while keeping 1024 dimensions).
Baseline Temperature
MRL
Temperature
.04
0.1
0.3
.04
0.1
0.3
Perf. full
46.7
45.9
38.6
47.2
45.4
38.8
Perf. at 256
43.6
44.2
38.2
45.2
44.4
38.5
Perf. at binary
44.6
44.9
37,9
45.3
44.3
38.1
Perf. 256/1024
93.2
96.4
98.8
95.8
97.8
99.0
Perf. binary/full prec.
95.3
97.8
98.1
96.1
97.7
98.0
Intrinsic dim.
654
623
430
626
616
396
Related work on embedding compression There is a large body of literature on dimensionality reduction and quantization for embeddings. Most recent work proposes training-aware methods, modified such that either 1) compressed embeddings full and compressed embeddings are predictive of the task at hand (Kusupati et al. 2024 Bai et al. 2021 Zatrir et al. 2019, Lin et al. 2017, Yamada et al., 2021, Izacard et al.) 2020; [Cenn et al. 2022 In the case of quantization, most studies focus on quantizing model weights, which also reduces inference time. It is difficult assess where our results fit within this body of work, due to differences in experimental settings, benchmarking, and goals.
where losses architectures
2) both
More relevant and proprietary models when vector truncation and binarization are used to reduce the size of em- beddings post-training. Specifically, the authors highlight the cost of storing large embeddings and test several models after binarization. They show that some models can achieve 92% performance retention with binarization, and 96% when binarization and reranking applied. These are lower than the retention scores we obtain, indicating that other differences, most likely in the way nega- tives are selected, further impact this property of the embedding spaces. When truncation applied, results vary depending on the model used and the compression ratio obtained. For example, the au- thors report 90% with compression for one model and 93.1 1% with 12x compression with another, which places our results in similar range: we obtain 94% retention with 4x reduction at 0.04.
experiments is the report by /hakir et al. (2024) which evaluates open source
5 MULTIPLE TEMPERATURES IN TRAINING
The previous section shows that larger temperatures lead to embeddings that can be more efficiently compressed. However, there still a trade-off between this desirable property and performance in absolute terms. In this section, investigate the of multiple temperature values during training in order to obtain a better trade-off between the two. We opt for aggregating loss terms which use individual temperature values, which is simpler than temperature schedules (employed for example in Kukleva et al. 2023).
Plain temperature aggregation For this setup, we replace the standard InfoNCE loss with sum over losses, each using individual temperature values.
TemmpAg (f,
(6)
where we use to stand for a vector of temperature values. In our experiments, we sum over T = temperature values [0.03, 0.06, 0.1], which cover the best range of values observed in previous sections. We set w; 1.0.
MRL temperature aggregation Previous experiments showed that MRL can further improve per- formance at reduced dimensions with no degradation at full dimensionality. We test loss that combines the two; it differs from the original MRL loss by allowing for temperature aggregation at all dimensions in training:
I TTempAggMMuUJ Ek De
TemppAge T ([1nd))))

---

Published as a conference paper at ICLR 2025
Table 2: MTEB Retrieval (Ret.) and Clustering (Clust.) performance across different models. "Full" stands for performance at full dimensionality (1024) and at full precision. The rest of the results report retrieval when: 1) truncating 256 while keeping full precision and 2) binarizing both with re-ranking (bin re-rnk and without, while keeping 1024 dimensions.
==
0.04
== 0.1
TempAgg
TempAggMRL
TempSpecMRL
Ret. (full)
46.7
45.9
46.4
46.6
46.0
Clust. (full)
34.1
37.7
36.4
36.9
37.4
Ret. 256
43.6
44.2
44.6
45.1
44.6
Ret. bin
44.6
44.9
45.2
45.2
44.8
Ret. bin re-rnk
45.6
45.6
45.9
46.1
45.5
Ret. 256/1024
93.2
96.4
96.2
96.9
97.1
Ret. bin/full prec
95.3
97.8
97.4
97.1
97.4
Ret. bin re-rnk /full pree
97.6
99.0
98.9
98.9
99.0
Intrinsic dim.
654
623
629
623
617
In this setup we use the same temperature range as above, which we apply 256, 512 and 1024 dimensions, using uniform X, = 1.0 weights.
Temperature specialization We implement third setup which uses the observation that lower temperatures are more beneficial for retrieval. Since reduced embeddings retrieval applications, we perform MRL training using a small r at 256 and a larger T at 1024. The small temperature will promote good retrieval performance at 256 while the larger value at 1024 will provide a temperature aggregation effect (note that a loss applied at 1024 will impact the first 256 dimensions, but viceversa):
mostly crucial in
(8) We set (256: 0.03, 512 0.06, 1024 0.1). In this last set of experiments, we also add re-ranking binary embeddings, which an effective method to improve performance in the bina- rization setup with minimal speed impact and vector storage impact. This consists of comparing binary passage vectors with full-precision query vectors (Yamada et al., (021I. Specifically, we retrieve the top 100 passages using binary representations and then re-rank them using the full- precision query.
TempSpee (f, PinntNC
Results shown in Table 2 We compare the aggregation methods with the baseline InfoNCE models using the best 7 for retrieval (0.04) as well as the best 7 across both retrieval at full size and compressed retrieval (0.1). Both TempAgg and TRM reach performance that is very close the best retrieval model while improving on clustering by a large margin. In parallel, the quality retention after compression also increases: from 95% to over 97% for binarization and from 93% to 96% for truncation to 256. Among these, TeMpSpecMRL is the best model w.r.t. quality retention. Best binarization results are obtained when adding the re-ranking stage: for several aggregation models, retention reaches an impressive 99% while reducing the size by factor of 32x.
Finally, the intrinsic dimensionality of the resulting spaces similar for all aggregation models and is lower than that of the = 0.04 models. This is in line with previous observations that intrinsic dimensionality correlates with quality retention when compressing.
FUTURE WORK
In future work, we plan to further explore the intrinsic dimensionality of embeddings spaces, and the subsequent relationship with compression under different training regimes. Additionally, an in- teresting question that arises from this work is why binarization (and presumably quantization in general) is a better method for flexible compression than dimensionality reduction. In our experi- ments, both methods shows similar quality retention across different settings, albeit with different compression ratios (32x and 4x for binarization and truncation, respectively). We will further plore whether this observation can be applied not only to obtain flexible embeddings, but also to improve performance overall as form of prramearmettiiooooonn in training.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Haoli Bai, Wei Zhang. Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. BinaryBERT: Pushing the limit BERT quantization. Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers). pp. 4334 4348. Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/v1/2021111 long.334. URL hetps: haclanthology.ooo
acl- aacllong..... 334
Payal Bajaj, Daniel Campos. Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018 URL https: /ttp:///axivv.i wwwpwwwwww.wo..../ooo/raaabaaaae 611 09268
Yankai Chen, Yifei Zhang, Huifeng Guo, Ruiming Tang, and Irwin King. An effective post- training embedding binarization approach for fast online top-k passage matching. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.). Proceedings of the 2nd Con- ference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the Natural Language Processing (Volume 2: Short Papers), pp. 102-108, Online only, November 2022. Association for Computational Linguistics. URL
12th International Joint Conference
setps::/aclanthologgy wttp:/yy.rgg//20//0 aacl- wacl-sshrrt... .4
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 1..18833/1//119111 1423. URL https: http:///aaaaathhcccc.c.o.m
N19- 1423
Hengkui Dong, Xianzhong Long, and Yun Li. Synthetic hard negative samples for contrastive learning. Neural Processing Letters, 56(1):33, 2024.
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval, 2021. URL https://arxiv: httpp:://arxiv.orr//ab//// wwt.:oogg//a/ooooooooooo 2104.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence wtt.:orgggaaaa/..aaa//////////.e 08821
embeddings, 2022.
Michael U. Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 131m111: February 2012. ISSN 1532-4435.
Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. A memory efficient baseline for open domain question answering. 2020. URL. https:
//axxiv.crg/abs/2011 /
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v 1/20000emmpp-maiin5550. URL
org. 2020 emnlp-main 550
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, and Soujanya Poria. comprehensive survey of sentence representations: From the bert epoch to the chatgpt and beyond, 2024. StLhtttps: httpp:s://aaxiii// /ttp///aww.vv..///////33.... 2200b//261.11111
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In
11

---

Published as a conference paper at ICLR 2025
Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL pttetpss//pproceedngg.s wwtpparrss.ccppppppee pwtppappr//222/haaii. dB
Ryan Kiros, Yukun Zhu, Russ Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015. URL httptt:p:s//prcceeddggee wttp://cceeeiiiggg..eee.eeeeeeee nwwrips. cctppperrrcppppppppppeppp.. files/
Pape77 Pappeee.
Anna Kukleva, Moritz Böhle, Bernt Schiele, Hilde Kuehne, and Christian Rupprecht. Temperature schedules for self-supervised contrastive methods on long-tail data. In The Eleventh International
Conference on Learning Representations, 2023, URthttps::/ wtpp/eeeeeeeeeeeeeeeeeeeeee / net. umr
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William Howard-Snydee Kaifeng Chen, Sham Kakade, Prateek Jain, and Ali Farhadi. Matryoshka representation learning, 2024. URL wtttts:://aartte org. abs 2205. 13147
Angeliki Lazaridou, Georgiana Dinu, and Marco Baroni. Hubness and pollution: Delving into cross- space mapping for zero-shot learning. In Chengqing Zong and Michael Strube (eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th Interna- tional Joint Conference on Natural Language Processing (Volume Long Papers). pp. 270-280, Beijing, China, July 2015, Association for Computational Linguistics. doi: 10.3 10.1115/1//15-100 1027. URL
oog.olo/oooooog11111o 10277
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Getrieval-augmenter genera- tion for knowledge-intensee nlp tasks. Advances in Neural Information Processing Systems, 33: 9459 9474, 2020.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems. NIPS' 17, pp. 344-352, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In Yoshua Bengio and Yann LeCun (eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013. htRt:t/::// http://axxvvv... org/ abs/ fabs/1001. 3781
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise- contrastive estimation. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceed- ings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 2013. URL
html
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark, 2023. URL
org. abs //t/0/222111111
abs
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela, Generative representational instruction tuning, 2024. URL
org/ /ogg/bbs/a0 2402. 09906
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter
12

---

Published as a conference paper at ICLR 2025
Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training, 2022. URL https://arxiv.ortt wttp://a///.....//oooo
Joshua Robinson, Ching-Yar Yao Chuang, Suvrit Sra, and Stefanie Jegelka, Contrastive learning with hard negative samples, 2021a. URL htttps:/aaxw...og///aaaaaaaaaaap
04592
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974 12 2021b.
Aamir Shakir, Tom Aarsen, and Sean Lee. Binary and scalar embedding quantization for signifi- cantly faster & cheaper retrieval. Hugging Face Blog, 2024. URL http::://uugiigggaee
ization
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction-fineture text embeddingss, 2023. URL https: http:///wwwww.......e.
09741
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021. URL
httts://arxiv org/ attpa/////.....2/ 08663
James Thore, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large- scale dataset for fact extraction and VERification. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi:
10.1865//1/N188-000 URL httt:ttts///claamtt httt:///AAA//ttppp///////aoooooo www.aaantoooooooooooooooooo wtwp:/ooyyorr//111171111
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, 2020. URL
https: /ttp//aavvvvv/. httgg//b//////
05849
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/ 007.03748 2018. utl//ttp://arxi..org// wwtp://vvv.rgaa///1/00/ 03748
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 9866):257--2655 2008. URL http::ttt///tt jm r. org wrggpapers/vv/
html
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2495-2504, 2020. URL
https: www.steeeecnooooi...eeeeeettttee htt.://ssscooaa..orrr/ooooppsaaa http://oopuul:::229909009
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Simlm: Pre-training with representation bottleneck for dense passage retrieval, 2023. httt:htts:///aa// wwtrxxiv..rgg/.. abs 2207 02578
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervisst contrastive pre-training, 2024a. URL https: wwwia. org/ abs www./22aa.0. 2212.2222222 03533
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Improv- ing text embeddings with large language models, 2024b. URL
www.aawwwww...aaaaaaoaoaoaoooore abs/
24C1.000 00368
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning. ICML'20 20.. orLR.org. 2020.
Ikuya Yamada, Akari Asai, and Hannanch Hajishirzi. Efficient passage retrieval with hashing for open-domain question answering. In Chengqing Zong Fei Xia. Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference Natural Language Processing (Volume 2: Short
13

---

Published as a conference paper at ICLR 2025
Papers), pp. 979-9999999999 799, Online, August 2021. Association for Computational Linguistics. doi: 10. 1atll./111------- 123. URL
123
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' un'ichi Tsujii (eds.). Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2369- 2380, Brussels, Belgium, doi: 10.1865/1//D1181222
2018. Association for Computational Linguistics. htt.ittt.ps//eeclaatttooo......
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: quantized 8bit BERT. CoRR, abss/9910006183 2019. URL httt:://aaxiii.//// org/
wwwpaas////o/oooooee/....o...aee http://bbs/11100/0 1910/ 06188
Dejiao Zhang. Wasi Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. Codesage: Code representation learning at scale. In The Twelfth International
Conference on Learning Representations, 2024. pttpttps:///peervvvii
net forum?
id=vÉzRR jumpX
Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual bench- mark for dense retrieval. In Duygu Ataman, Alexandra Birch, Alexis Conneau, Orhan Fi- rat, Sebastian Ruder, and Gozde Gul Sahin (eds.). Proceedings of the 1st Workshop on Mul- rilingual Representation Learning. pp. 127-137, Punta Cana. Dominican Republic, Novem- ber 2021. Association for Computational Linguisties. doi: 10.18653/v120021.mmm 1.rr1112..ml.112.... URL Buttss://cclantholoog
Xinyu Zhang. Nandan Thakur, Odumayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermee Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadch, and Jimmy Lin. MIRACL: A Multilingual Re- trieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computa-
tional Linguistics, 11:1114-11311 09 2023. ISSN 2307-387X. doi: 10. 162/tacl a 00595. URL. 00595
httpss///al..orr/00.11113.
14

---

Published as a conference paper at ICLR 2025
A FULL TRAINING SETTINGS
Figure 7 shows the training parameters used to train the models. Additionally, note that assign weights to the negative samples based on similarity to the anchor, following Dong et al. (2024)
max max_ mex__se__engtt_ 1024
max_steps 3000
-warmup_steps 58
--base_ wwt.ggooaaaaaahhheooo 4096
-weight_decay 0.1
-base_learningrrate 5e-06
1e-01
@pase_max_stedt 3000
lrr_scheduler__ype cosine
gradient_clip_val 1.0
-optimizer FusedAdam
Figure 7: Additional training parameters. We use in-batch negatives with batch size of 256 and homogenous sampling, meaning that the negative sample are drawn from the same training set. All models are tested after 2000 training steps.
Table lists the datasets used in the contrastive training stage.
Table 3: Datasets used in the contrastive training stage, MsMarco is downsampled to the size listed in the table. MIRACL and MrTydi are multilingual while the others En only. References given in the main text.
Dataset
Size (in k entries)
MsMarco
1,000k
NQ
150k
NLI
247k
HotPotQA
90k
FEVER
108k
MIRACL
600k
MrTyDi
40k
B RESULTS ON OTHER TASKS
The main results in section focus the effect of temperature retrieval and clustering tasks (see Figure 2). In this section we discuss the performance and compression rate of other MTEB tasks.
Figure shows performance (on the MTEB benchmark) as a function of temperature for the remain- ing MTEB tasks. Reranking performance is not affected by temperature. Performance other tasks shows a similar but much smaller effect to what we observed in retrieval, with higher temperature values leading to lower performance.
In Figure D we show retention of full-precision performance when truncating to 256 dimensions at different temperature values. Overall, we see similar pattern to what we observed in Figure 4 for retrieval, with higher temperature generally leading to higher performance retention, although this trend is particularly pronounced for clustering.
C RESULTS WITH ADDITIONAL TRAINING SETTINGS
In this section, vary the training settings whether results generalize different settings.
15

---

Published as a conference paper at ICLR 2025
Performance
reranking
classification
pair_ sair_cllasiffiattion
sts
summarization
0.04
0.06
0.08
0.10
0.20
0.30
0.40
Temperature
Figure 8: MTEB performance as a function of the temperature value used in the contrastive learning training stage.
102.5
100.0
Retention dim. performance
97.5
95.0
92.5
reranking
90.0
classification
pair_classee clw.sfissfiictionn
87.5
sts
summarization
85.0
dustering
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
Temperature
Figure 9: Retention of overall MTEB performance when truncating to 256 dimensions as a function of temperature value used in the contrastive learning training stage.
C.1 SAMPLING STRATEGY
By default in our experiments, we use homogeneous sampling (i.e., sampling negative examples from the same dataset the positives) to increase the difficulty of the negatives. Here, compare homogencous vs. heterogeneous sampling to see the effect of easier negatives on our experiments. Given that with lower temperatures, more difficult negatives receive higher weight, hypothesize that hetereogeneous sampling could have a similar effect to using higher temperatures.
Table compares homogeneous and heterogeneous sampling results. We find that heterogeneous sampling yields worse results than homogeneous sampling, particularly for retrieval. addition, heterogeneous sampling slightly increases retention of retrieval performance lower values of т.
16

---

Published as a conference paper at ICLR 2025
Table 4: MTEB retrieval (Ret.) and Clustering (Clust.) performance with homogeneous (default) and heterogeneous sampling. "Full" stands for performance at full (1024) dimensionality and full precision. The remaining results report retention of retrieval performance when: truncating 256 while keeping full precision and 2) binarizing without re-ranking while keeping 1024 dimensions
0.04
0.3
sampling
homogeneous
heterogeneous
homogeneous
heterogeneous
Clust. (full)
34.14
33.81
39.24
38.94
Ret. (full)
46.73
45.70
38.64
35.80
Ret. 256/T024
93.10
93.46
98.80
98.59
Ret. bin/full prec
95.34
95.53
98.06
97.20
Table 5: MTEB retrieval (Ret.) and Clustering (Clust.) performance with different numbers of training steps. "Full" stands for performance at full (1024) dimensionality and full precision. The remaining results report retention of retrieval performance when: 1) truncating to 256 while keeping full precision and 2) binarizing without re- re-ranking while keeping 1024 dimensions
T
0.04
0.3
steps
1k
2k
3k
4k
1k
2k
3k
4k
Clust. (full)
34.57
34.14
33.93
34.69
37.42
39.24
37.85
37.77
Ret. (full)
46.05
46.73
46.55
46.69
37.53
38.64
38.36
38.33
Ret. 256/1024
93.86
93.10
94.14
94.43
98.89
98.8
98.77
98.61
Ret. bin/full prec
95.62
95.32
94.96
95.79
97.90
98.14
97.70
97.88
However, given that the absolute performance is than with homogeneous sampling, this not useful in practice. Overall, the major patterns (improved clustering and compression at higher т, improved retrieval at lower 7) remain the same regardless of the sampling method.
C.2 NUMBER OF TRAINING STEPS
By default, we train our models with 2000 training steps. In Table 5 we report retrieval and clus- tering performance as well as retention of retrieval performance after truncating or binarizing when training for 1000, 2000, 3000, and 4000 steps. We see major differences in either full com- pressed performance for different number of steps, and the major findings of the paper remain the same regardless of the number of steps. This indicates that findings not an artifact of the stage of training.
17


Reviews:
[{"review_id": "hQuUolV4ic", "reviewer": "Reviewer_dzd1", "summary": "This paper researchers the effect of temperature in contrastive learning, specifically its effect on downstream performance as well as compressibility and intrinsic dimension. The idea of \"flexible compression\" is useful and novel as well. The experiments are intuitive and provide practical insight into why and how embeddings-training works under different temperatures.", "questions": "- Would learnable temperature improve results?\n- Why only optimize over three temperature values?\n- Do you expect these results to generalize to other domains, e.g. vision?\n- Is there any connection between your observations about the effect of temperature and the research into temperature values for the more general language modeling task?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Basic research into embedding spaces is important and underexplored. Temperature is a similarly crucial and \"taken-for-granted\" value for practitioners training models with contrastive loss.\n- The result that temperature decreases retrieval performance and increases clustering performance is interesting and useful, since many researchers simply average the performance of both tasks\n- Multi-temperature optimization is novel and appears useful (multi-temperature trained models improve over single-temperature ones)", "weaknesses": "- Findings are specific to training text embedding models for the zero-shot MTEB benchmark. Insights aren't general to contrastive loss (i.e. theoretical) or multimodal or vision models, even though those models also use the temperature parameter. \n- Multi-temperature optimization isn't super principled\n- No discussion of learnable temperature value (as used in the original CLIP paper, for example)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YarNUaYRf", "reviewer": "Reviewer_Gw7k", "summary": "The paper focus on analysis of the temperature impact on embeddings' properties when training using contrastive loss. Authors show the trade-off between retrieval and clustering tasks, they attribute to the intrinsic dimensionality. Low intrinsic dimensionality (associated with the large temperature) and high intrinsic dimensionality (corresponds to the low temperature) behaves differently under the compression. The former retains up to 99% of the performance when doing truncation or binarization, while for the latter retention is at most 97.5%.", "questions": "For authors: it was a pleasure to read your work. I have a few questions/suggestions.\n\n- When your report the retrieval/clustering results, are these average results across all tasks?\n- I am curious why there is no investigation on quantisation for clustering. Does that make any sense?\n- While retention % is higher with the higher temperature, the  absolute quality decreases significantly on retrieval. It took me a while to understand what is happening in Figure 5 and 6, so I suggest you either point it out clearly, or change visualisation method to avoid confusion", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well written; clear; easy to follow\n- Important analysis work on the embeddings that can benefit different areas of ML\n- Interesting connection to embeddings compression", "weaknesses": "- There is no comparison to other pre-trained models. So the studied effect can be the property of the model/data combination", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZAbinpoFZ", "reviewer": "Reviewer_11ep", "summary": "The paper explores the effect of temperature in contrastive learning for the efficient compression of embeddings. One of the main findings of the paper is that controlling the temperature impacts the intrinsic dimensionality of the embeddings. At the same time, they found that lower temperatures enhance retrieval performance while higher temperatures improve the clustering performance. \n\nThey explore how the temperature impacts compression performance by studying two methods of compression, namely truncation and quantization via binarization. The work suggests that we can find an optimal balance between retrieval and clustering performance by tuning the temperature.", "questions": "The weaknesses mentioned above.\n\nAs suggestions, the labels for Figure 3 are hard to read in the first 3 images, they should be made bigger.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper studies in-depth the influence of the temperature during contrastive training on the embedding compression and shows that this is a crucial ingredient to optimality of the resulting retrieval and clustering tasks. It clearly provides a guideline of the trade-off between memory efficiency and task-specific performance by tuning the temperature.\n\n2) This difference in absolute performance and quality retention across temperatures underlines the crucial role temperature plays in embedding quality. While higher temperatures retain compression better, low temperatures ensure higher retrieval accuracy, and this forms the practical basis for a variety of applications.\n\n3) Multi-temperature training strategies allow a good balance between retrieval accuracy and high compression retention, hence underlining the specialization of temperature as an effective way to optimize embeddings for multiple objectives in real-world tasks.", "weaknesses": "1) While the study indeed manages to adapt insights on temperature from the vision domain effectively, it is not introducing a fundamentally new theory and limits novelty in its contributions. The findings extend previous knowledge from vision papers without anything particularly new in terms of mechanisms that are specific to the case of text embeddings.\n\n2) It would be helpful if the paper is broadened by exploring other factors that influence embedding compression and performance, such as alternative sampling techniques for negative examples.\n\n3) Limited variation in the temperature values used in experiments. The range of temperature values in the experiments is quite narrow, with further gains/losses possibly missed by not considering a much wider range of configurations. Increasing this range or adaptive adjustment of temperature would reveal more about how best to trade off compression quality and retrieval performance across different tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j07dTAeKyz", "reviewer": "Reviewer_XBz5", "summary": "This paper investigates the role of the temperature parameter in contrastive training for text embeddings, focusing on its impact on the intrinsic dimensionality of the embedding spaces obtained. The study finds that lower intrinsic dimensionality is correlated with effective compression of embeddings, highlighting the importance of the temperature parameter in embedding compression. This research provides new insights into how the temperature parameter in contrastive learning affects embedding quality and compressibility.", "questions": "Here are some questions and suggestions for the authors:\n（1）Please provide detailed information about the computational costs and resource requirements of the proposed method, especially regarding the efficiency improvements compared to the uncompressed model.\n（2）Please clarify the innovations of the proposed method in relation to existing techniques, particularly regarding the impact of the temperature parameter on embedding compression.\n（3）It is advisable to revise the visual content and tables in the paper, such as Table 1 and Table 2.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper are summarized below：\n(1) The paper presents a novel investigation into the impact of temperature in contrastive training on embedding compression, exploring the relationship between intrinsic dimensionality and post-training compression techniques.\n(2)The research demonstrates high-quality experimental design, with comprehensive tests across various temperature values and compression methods, providing robust evidence for their hypotheses.", "weaknesses": "There is a substantive assessment of the potential weaknesses of the paper.\n(1) The paper may lack significant novelty, as it appears to be building upon existing work in contrastive learning and embedding compression techniques.\n(2) The paper's focus on post-training transformation and temperature aggregation methods may be too narrow, potentially overlooking other important factors in embedding quality and compression.\n(3) While the paper claims practical implications, it may not sufficiently demonstrate how its findings can be effectively implemented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)