Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddings
Authors: Rita González-Márquez
Philipp Berens
Dmitry Kobak
Venue: Accepted by TMLR ()

Abstract: Text embeddings, i.e. vector representations of entire texts, play an important role in many NLP applications, such as retrieval-augmented generation, clustering, or visualizing collections of texts for data exploration. Currently, top-performing embedding models are derived from pre-trained language models via supervised contrastive fine-tuning. This fine-tuning strategy relies on an external notion of similarity and annotated data for generation of positive pairs. Here we study self-supervised fine-tuning and systematically compare the two most well-known augmentation strategies used for fine-tuning text embeddings models. We assess embedding quality on MTEB and additional in-domain evaluations and show that cropping augmentation strongly outperforms the dropout-based approach. We find that on out-of-domain data, the quality of resulting embeddings is substantially below the supervised state-of-the-art models, but for in-domain data, self-supervised fine-tuning can produce high-quality text embeddings after very short fine-tuning. Finally, we show that representation quality increases towards the last transformer layers, which undergo the largest change during fine-tuning; and that fine-tuning only those last layers is sufficient to reach similar embedding quality.

Published in Transactions on Machine Learning Research (02/2026)
Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddings
Rita González-Máne Márquez
Hertie Instätute for AT in Brain Health University of Tübingen, Germeny
Philipp Berens
Hertie Institude for AI in Brain Health University of Tübingen, Germany
Dmitry Kobak
Hertie Institute for Al in Brain Health University of Tübingen, Germany
Reviewed on OpenReview: https: /ttps: www.//vee...e.eee.ooeee net/ forum? id- VRss Ih9w7h
Abstract
Text embeddings, i.e. vector representations of entire texts, play an important role in many NLP applications, such as retrieval-augmented generation, clustering, or visualizing collections of texts for data exploration. Currently, top-performing embedding models are derived from pre-trained language models via supervisod contrastive finc-tuning. This finc- luning stralegy relies on an external notion of similarity and annotated data for generation of positive pairs. Here we study self-supereised fine-tuning and systematically compare the two most well-known augmentation strategics used for fine-tuning text embeddings models. We assess embedding quality on MTEB and additional in-domain evaluations and show that cropping augmentation strongly outperforms the dropout-basel approach. We find that on out-of-domain data, the quality of resulting embeddings is substantially below the supervised state-of-the-art models, but for in-domain data, self-supervised fine-tuning can produce high- quality text embeddings after very short fine-tuning. Finally, we show that representation quality increases towards the last transformer layers, which undergo the largest change during fine-tuning, and that fine-tuning only those last layers is sufficient reach similar embedding quality.
1 Introduction
Representing texts as vectors is important in natural language processing for both supervised (e.g. spam detection, sentiment analysis, semantic matching) and unsupervised (e.g. clustering, visualization, retrieval) downstream tasks. With the advent of retrioval-angmented generation (RAG), has become increasingly important to produce high-quality text representations that enable retrieval of relevant texts that can be added to the prompt improve performance of generative large language models (LLMs) (Izacard et al., 2022b; Ram et al. 2023). Such represenlations (or text embeddings) can be obtained with a wide range of methods, from simple bag-of-words representations such TF-IDF (Jones) 1972) to transformer-basse large language models (sce Zhao et al. 2023 for a survey). Such language models are initially trained (pre-truined) with a token-level loss, and subsequent fine-tuning with text-level loss is needed to obtain useful text-level representations (Xu et al. 2023). We refer to models and representations fine-tuned for representing entire texts as sentence transformers and sentence or text embeddings, following Reimers & Gurevych (2019).
In recent benchmarks such as MTEB (Muennighoff al. 2023 sentence transformers relying on extensive supervised contrastive fine-tuning on large curated datasets have typically performed best, whereas models

---

Published in Transactions on Machine Learning Research (02/2026)
solely trained with self-supervised contrastive learning scored worse results. This is in stark contrast to puter vision, where sell-supervised learning (SST.) via data augmentations has been immensely successful in producing semantically meaningful image representations (Balestriero et al.) 2023). Various SSL approaches have been suggested for sentence-level fine-tuning, inchuding the dropout-basce based method employed by SimCSE
(aoo et al.) 202T) and the cropping-basel method employed by DeCLUTR (Giorgi et al. 2021). However, it remains unclear how their performance compares between each other and to the state-of-the-art (SOTA) embedding models, especially when evaluated on texts from some specific limited domain.
In this work we address these questions by systematically comparing the two most well-known data- augmentation strategics used for training text embeddings, and by comparing them to the state-of-the-art supervised models. By extensively assessing representation quality on MTEB tasks and additional in-domain evaluations, we show that:
using text crops as positive pairs for contrastive learning performs consistently better than the dropout-based augmentation used by SimCSE, contrary to the claims in Gao et al. (2021);
out-of-domain data, the quality of resulting embeddings is substantially below the supervised SOTA models, but for in-domain data, self-supervised fine-tuning produces high-quality sentence embeddings, comparable to those by the supervised SOTA models;
self-supervised finc-tuning
minimal amount of data (as few 10000 short input texts) already lead to large improvements in sentence embedding quality;
a large part of the improvement during SSL fine-tuning is due to the generic and domain- iimainnindependent adaptation that can be called sentence adaplalion;
representation quality increases towards the last transformer layers, which undergo the largest change during fine-tuning; and fine-tuning only those layers leads to similar embedding quality.
Our findings are noleworthy given that dropout augmentations of SimCSE (Gao et al.
2021 represent
one of the most well-known and well-cited SSL approaches in the literature on text embedding Dropout augmentations are still being used to fine-tune modern models (e.g. [BehnamGhader et al. 2024 and have also been adopted in other fields, e.g. for representation learning of single-cell RNA-sequencing data Yang
et al. 2022
We therefore believe that, even though modern text-embedding models are usually relying supervised fine-tuning, systematically comparing different augmentation approaches for self-supervised fine-tuning can provide valuable insights into which training strategies yield the best text representations.
2
Related work
2.1 Text embeddings from generative and non-generative language models
Early text embeddings were obtained with bag-of-words approaches, such as TF-IDE (Jones 1972 and BM25
[obertson et al. 1995 Kamphuis et al., 2020). Transformer-based (Vaswani et al., 2017) language models led a shift toward dense embeddings, which capture richer semantic information and context compared to the sparse bag-of-words representations. Initially, non-generative eneoder-only (i.e. BERT-like) models were used for generating text embeddings since they use bidirectional attention, which is more appropriate for embedding tasks. However, recent work has increasingly adopted generative decoder-only (i.e. GPT-like) models for embedding tasks to leverage powerful recent models with more extensive pre-training.
Encoder-only models These models receive a sequence of text tokens as input and produce a separate latent representation for each of the tokens as output, without any restrictions on the attention patterns, resulting in bidirectional self-attention. The BERT model (Devlin al. 2019) and its later variants such
I At the moment publication, Gao et al. (021) has 900 citations in Google Scholar, second only after Reimers & Gurevych 2019h with 20000 citations, in the literature on text/sentence embeddings.
2

---

Published in Transactions on Machine Learning Research (02/2026)
RoBERTa (Liu et al. 2019) and MPNet (Song et al. 2020) include an additional classification token [CLS] to serve a global representation of the full text in downst tasks. However, only a small fraction of typical BERT training is dedicated to sentence-level tasks, such that [CLS] representations do not usually perform well at encoding sentenco-level semantics (Thakur et al. 2021 Mucnnighoff et al. 2023 Simply averaging the embeddings across all tokens to obtain a sentence-leve representation does not perform well either (Muennighoff et al., 2023). Therefore, more sophisticated pooling (Wang Kuo 2020) and post- processing strategies (Li et al. 2020 Su et al. 2021) have been suggested to improve BERT-deriver sentence representations. Alternatively, model pre-trained the token level can be fine-tuned with sentence-level objective to improve the sentence-level representations (see below). Usually fine-tuning is performed on the average pooling of all tokens or the [CLS] token embedding.
Decoder-only models Decoder-only models are used for text generation and employ masked (as opposed to bidiroctional) attention, where each token can only attend to proceding tokens. Different strategies have been proposed to adapt generative models for texl-embedding lasks. One approach is append an instruction before the text and then use either the representation of the last [EOS] token (Wang et al. 2024 Li ct al. 2025) or the average representation across all tokens (excluding the instruction) (Behnam Ghader 2024 as the text embedding. This strategy was originally introduced in encoder-only models (Su ct al.
2023 but has increasingly been adopted for generative models. A generative model (with or without additional instructions) can also be fine-tuned with a sentence-level objective (see below). Prior to such fino-tuming, some works remove the causal attention masking, switching the model to bidiroctional attention [BehnamGhader et al., 2024 Lee al., 2025). Additionally some fine-tuned models use particular way of pooling such as position-weighted mean pooling (Muennihhoff 2022 or latent attention pooling [Lee et al. 2025).
et al.
Note that some tasks that are often used for benchmarking text embeddings can be handled by a generative model without any explicit text embeddings. For example, a classification task can be presented to a generative LLM with a prompt asking to predict the text's label from a given set of labels (Chen et al. 2025). This does not constitute a text embedding.
2.2 Contrastive fine-tuning of text embeddings
Whatever the exact model architecture and pre-training are, contrastive learning is commonly used for fine- tuning of text representations. Contrastive learning pulls together pairs of similar texts (positive pairs) and pushes apart pairs of non-similar texts (negative pairs) in the embodding space (Chen et al. 2020 Contrastive learning approaches differ in how similar texts are defined.
Supervised contrastive lcarning Here positive pairs are collected based on some explicit notion of similarity. Sentence-BERT (SBERT) (Reimers & Gurevych 2019) uses a curated dataset of paired texts such as question-answer pairs from Stack Exchange; their most recent model all-mpret-base-v2 (2021)
was trained on over one billion of such pairs. Similarly, BGE (Xiao et al., 2024). E5 (Wang et al.) 2022).
and GTE (Li et al.] 2023) all undergo contrastive fine-tuning using large datasets of text pairs curated from different sources, such as internet webpages (Litle-body pairs), scientific papers (title-abstract pairs). or community forums like StackExchange (question-answer pairs). For academic texts, SPECTER (Cohan et al. 2020) and SciNCL (Ostendorff et al. 2022) use citing and cited papers' abstracts to form positive pairs. Sentence-Te T5 (ST5) (Ni et al.) 2022) and Sentence-GPT (SGPT) (Muennighoff) 2022) are derived via contrastive fine-tuning of T5 (Raffel et al. 2020 and GPT (Radford et al. 2018) models on curated datasets of text pairs.
Self-supervised contrastive learning Here positive pairs are generated automatically from unpaired texts, similar to self-supervised learning in computer vision that relics on data augmentations (Chen et al. 2020). SimCSE (Gao et al., 2021) uses two different dropout patterns to form a posilive pair of embeddings. This approach has also been used by Liu et al. (2021) and Yan et al.] (202I) who additionally investigate other data-angmenntttin techniques such as randomly masking parts of the input text. The same dropout approach was also adopted by LLM2Vec (BehnamGhader et al.) (024) for fine-tune generative LLMs (after removing attention masking, above). Alternatively, outputs of two distinct networks can be used to
3

---

Published in Transactions on Machine Learning Research (02/2026)
generate positive pairs (Kim et al. 2021 Carlsson et al. 2021]. Further,
adjacent chunks of
text as positive pairs; this was applied to train RNNN Logeswaran & Lee, 201 8), GPT (Nelakantan et al. 2022), and BERT models (DeCLUTR, Giorgi et al. 2021 Contriever, Izacard et al. 2022a and CoCondenser,
Gao & Callan 2022). Recently, synthetic generation of positive pairs has been explored leveraging generative
LLMs (Zhang el al. 2023 Wang et al. 2024 but this approach comes with the additional compulations ional
cost of text generation.
2.3 Benchmarks
In benchmarks of sentence transformers, models trained a supervised way outperform the ones trained only with self-supervision (Thaur al.) 2021
superre
Muennighofi al. 2023). Among the models of BERT-base size, the BGE-base model (Xiao et al. 2024 with its further modifications and the GTE-base model (Li et al. 2023 are currently among the best performers, overtaking the SBERT's latest all-mpnet-base-v model. Larger models, such as BGE-large or commercial embedding models like text-embedding
from OpenAl and embed- embed-englistt english-v3 from Cohere, oulperform -EsT-base-sized models, in particular on some of the evaluation tasks.
3 Self-supervised contrastive fine- tuning
3.1 Augmentations and loss function
We out to investigate how effectively sentence representations from language model can be improved through self-supervised fine-tuning alone, and how different augmentation choices influence this improvement. For that, we leveraged contrastive learning approach analogous to SimCLR (Chen el al.) 2020 our
training strategy and compared various augmentation techniques for generating positive pairs, such as text
crops
Logeswaran & Lee 2018 Giorgi et al.] 2021 Neelakantan et al. 2022: Izacard et al. 2022a Gao &
Callan 2022), dropout-based based augmentation (Gao et al.) 2021), and variations of those (see Section A.1
Importantly, all augmentation approaches our comparison were constructed in a self-supervised manner, without relying on external notions of similarity or annotated data.
The cropping augmentation
set up follows: for each input text i in minibatch of size b, we cropped out all possible chunks of t = 2 consecutive sentences (discarding all sentences under 100 and over 250 characters long) and sampled two chunks, one as the anchor text and one as its positive partner pa. For example, if the abstract of our paper were in the training set, then one positive pair could look like this:
Text embeddings, ie. vector repre- sentations of entire texts, play an im- portant role in many NLP applica- tions, such as retrieval-uugmented gen- cration, clustering, or visualizing col- lections of texts for data exploration. Currently, top-performing embedding models derived from pre-trained language models via supervised con- trastive fine-tuning.
This fine-tuning strategy relies external notion of similarity and anno- tated data for generation of positive pairs. Here we study self-superoised fine-tuming and systematically com- pare the two most well well-known augmen- tation strategies used for fine-tuning embeddings models.
+
For the dropout-based augmentations, we used the approach SimCSE (Gao et al. 2021 We split each input text into groups of consecutive sentences in the same way as for the cropping a augmentation, have similar text lengths for both kinds of augmentations. Then, we sampled one single crop and passed it through the model twice, with two different random dropout patterns applied to it, yielding two different representations that we used anchor a and positive pair p..
As negative examples for text i we always used the positive partners of all other anchors within the same minibatch B. Unlike some other recent studies, we did not use any hard negalives (Cohan et al. 2020 Giorgi et al. 2021 Ostendorff et al. 2022).
During contrastive training, the cosine similarity between the representations of a, and Pi maximized, while minimizing the cosine similarities between representations of a; and P; for j # i within the same
4

---

Published in Transactions on Machine Learning Research (02/2026)
minibatch B. This can be achieved using the InfoNCE loss function (Oord et al. 2018 also known as the normalized temperature-scalel cross-entropy loss (NT-Xent) (Chen et al. 2020 For one sample i, the loss is given by:
exp (sim(as Pi)/T)
4 - log
(1)
Σ exp (sim(as, p;)/T) jEB
where sim(a, p) a (aTpp((aa)) lIpl) is the cosine similarity between a and p, the vector representations of texts a and p. We set the temperature to -= 0.05 and the batch size to = 64, the largest possible batch size given our GPU memory resourccs. We trained the network using the Adam optimizer (Kingma & Baj 2015) with learning rate fj = 10-5, with linear warm-up and linear decay. See Section A.1 for details on hyperparameter choices.
3.2
MTEB performance after SSL fine-tuning
3.2.1
Setup
As the first evaluation suite to assess the sentence representation quality, we used the Massive Text Em- bedding Benchmark (MTEB) (Muennighoff et al. 2023 Tt comprises older benchmarks, such as BEIR (Thakur et al. 2021 STS (Agirre et al. 2012); ; well wide range of downstream tasks from different modalities, such as clustering, classification, or retrieval.
We fine-tuned a base transformer (i.e. a model primarily pre-trained with a token-level loss) using the SST, training setup outlined in the previous section and compared two augmentation strategies: cropping and dropouts. We chose MPNet (Song et al. 20200] mpnet-base) as base model, following SBERT (Reimers & Gurevych 2019 model all-mpnet-
on HuggingFace). This model has bert-base architecture with 110 M parameters and uses 768 embedding dimensions. We used mcan pooling over all tokens to obtain single 768-dimensional output vector for each input text, but also compared other pooling strategies (Section
A.1 Supplementary Figure SIp. Furthermore, we also tested two other base models (BERT and RoBERTa: a: Devlin et al. 2019 Liu et al. 2019; Supplementary Table STp.
For fine-tuning the base model we used the ICLR datasel Goonzllez-Máqqueee & Kobak 2024), which consists of 24.347 scientific abstracts of all papers submitted the ICLR conference within the years 2017-2024, This recently assembled dataset is well-suited for our purposes because it is not part of the MTEB benchmark. Fine-tuning on this dataset therefore enables fair model comparison and avoids evaluation data leakage, which can potentially lead to inflated performance estimates.
For simplicity, we did not use all MTEB tasks for evaluation, but focused on subset of English tasks from five different modalities: clustering, reranking, retrieval, STS, and classification. Clustering tasks assess the K-means clustering results the embedding space; retrieval and reranking tasks assess the quality of the nearest neighbors in the embedding space; STS tasks measure how well the embedding represents not only small but also large ground-iruth pairwise distances; classification lasks use a logistic regression classifier to assess linear separability of ground-truth classes (see Section A.2 for details). These evaluation modalities quantify different aspects of sentence representation and cover the wide range of MTEB tasks. We used SentenceTransf ormers library that normalizes all text-embedding vectors to have unit norm.
3.2.2 Results
Out of the box, MPNet resulted in poor representations with block average across modalities of 38.9% (Table column 1). After fine-tuning the ICLR dataset for a single epoch, the quality of the embeddings markedly improved. Out of the two augmentation strategics, cropping worked much better than dropout: cropping-based fine-tuning out performeed dropout-based fine-tuning in 16/19 tasks (on average 51.8% vs 46.9%, Table [ columns 3 3-4), and in the other 3 the difference was small. Cropping- based fine-tuning block average score of 12.9 percentage points, but there were large differences between modalities: STS had the largest improvement (21.4 p.p.), followed by retrieval (20.0 p.p.), clustering (10.3 p.p.), reranking (10.1 p.p.), and classification (2.7 p.p).
yielded an improvement
5

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: MTEB tasks. Row blocks correspond clustering, reranking, retrieval, STS, and classification
tasks. All values in percent, higher is better. Models in columns 3-4 were finc-tuned the ICLR dataset.
(1)
(2)
(3)
(4)
(5)
Model
MPNet
SimCSE
MPNet Dropout
MPNet
SBERT
BGE-base
BGE-large
Augmentations
Crops
ArxiiChusteringP2P
27.8
35.4
33.3
38.3
48.1
48.7
48.6
BiorxirChusternggP2P
23.2
30.1
31.1
32.4
39.3
39.4
39.7
MedrxivClusteringPP2
22.5
28.0
29.3
30.8
35.6
33.2
32.6
ReddiichuuteriiggP22
37.4
44.7
49.5
55.9
56.6
62.7
64.7
StackExchangeCI..1
26.3
28.8
30.2
31.3
34.3
35.2
35.0
SciDocsRR
56.1
69.5
64.6
73.6
88.7
87.5
87.6
MindSmmallRerankingg
27.5
29.3
28.4
30.2
31.0
31.2
32.1
SCIDOCS
1.4
7.9
6.5
13.0
23.8
21.7
22.6
ArguAna
22.2
41.4
41.9
50.6
46.5
63.8
64.5
STS15
53.5
82.3
63.5
72.5
85.7
88.0
88.0
STS16
50.6
77.7
66.2
76.0
80.0
85.5
86.5
STSBenchmark
52.0
78.6
67.9
71.7
83.4
86.4
87.5
PolzanoolarityCaaa
66.5
72.0
63.3
62.5
67.1
93.4
92.4
Banking77Class.
57.4
74.4
63.3
67.7
81.7
87.0
87.8
ImdbClass.
61.8
66.5
66.2
64.9
71.2
90.8
92.8
Massivelntennt Class.
55.9
60.5
53.0
53.9
69.8
72.6
74.3
MassiveSoenarioClass
60.7
66.7
62.8
65.9
75.7
76.5
77.4
MTOPDoomainlaas
75.9
82.8
82.6
85.1
91.9
93.2
94.0
Tweet.ent..
52.8
52.2
51.6
49.7
55.0
59.4
59.9
Block average
38.9
51.0
46.9
51.8
58.8
62.9
63.5
We repeated the experiment using two further base models (BERT and RoBERTa) and observed the same results with MPNet, namely that fine-tuning using cropping augmentation yielded better represen- tations than fine-tuning using dropout (Supplementary Table [1]. Representations obtained with cropping- based fine-tuning were better than those obtained via dropout in 16/19 tasks for BERT and in 15/19 19 for RoBERTa. Compared to the out-of-the-box models, after cropping- based finc-tuning the representations improved by 6.8 percentage points for BERT and by 10.7 p.p. for RoBERTa. The exact ranking of MTEB modalities by improvement size varied between models, but retrieval always showed one of the most pro- nounced improvemenss, while in classification tasks there was near-zero change when using BERT and RoBERTa.
The off-the-shelf unsupervised SimCSE model (which is based on dropout fine-tuning) performed similarly to our dropout-based fine-tuned model on all tasks except for the STS and some of the classification tasks, where it was substantially better than all finettned base models (Table columns and 3; Table S1 columns 5 and 8), suggesting that the good performance of SimCSE: those benchmarks may be due to some other fino-tuning choices beyond the dropout augmentation. SimCSE yielded worse results than our cropping- based fine-tuning in the other three modalities, despite having being fine-tuned on two orders of magnitude more data (1 M Wikipedia sentences for SimCSE 24 k samples in experiments), demonstrating that, at least for most tasks, the choice of augmentation has a greater impact on performance than the amount of training data.
Cropping-bard based tunet MPNet was 7.0 percentage points below SBERT, which achieved 58.8% perfor- mance as block average. Fine-tuned MPNet performed closest SBERT in retricval tasks (3.3 p.p. below SBERT) and furthest in STS tasks (9.6 p.p. below SBERT). This demonstrates that cropping- based fine- tuning produced sentence-level model that showed substantial generalization despite very limited amount

---

Published in Transactions on Machine Learning Research (02/2026)
80%
60%
MTEB score
40%
MPNet base
Clustering
Reranking
Retrieval
20%
STS
Classification
Block
0%
100
200 Batch
300
Figure 1: MTEB score during fine-tuning, Improvement on block mean MTEB score and individual task modalities within the first fine-tuning epoch on ICLR dataset.
of fine-tuning (~24 k training samples, ICLR dataset only). Note also that some of the evaluation datasets, e.g. SCIDOCS, arXiv, and StackExchange, formed part of the training set of SBERT, possibly biasing SBERT performance estimales upwards.
Additionally, we compared the standard transformer models to two state-of-the-art models: BGE-base (109 parameters) and BaE-large (335 M parameters) (Xiao et al.] 2024) (see Table S2 for the list of all used models). BGF-base is among the best-performing models in public MTEB leaderboard within models of its size. BGrg-lrrg is the larger version of 3GE-base, and is among the top models overall, on par with -ext-embedding---1rrrg
from OpenAI. In the MTEB tasks we evaluated, both BGE models performed better than SBERT, especially on classification tasks (4.4 p.p. higher in block average) and substantially better than our cropping-based fine-tuned MPNet (11.4 p.p. difference). This suggests that additional training modifications, such as pre-training specifically targeted at sentence embeddings, instruction- based supervised fine-tuning, or hard negative samples, can bring embedding quality further.
3.2.3 Cropping-based fine-tuning very fast
We further analyzed the performance improvement within the single fine-tuning epoch. We found that the MTEB block average was improved by around 15 percentage points within the first 100 fine-tuning batches (6 400 positive pairs) (Figure D Afterwards, the MTEB score plateaued for all modalities and did not improve any further, and fine-tuning for more than epoch in the same datasct did not bring further improvements. As caveat, is possible that one would observe further slower gains performance with larger amount of training data and longer training duration.
These 100 batches of fino-tuning took only ~1 min of training time single GPU (NVIDIA RTX A6000). In comparison, the top-performing sentence transformer models are typically trained on large datasets, with substantial computational costs and training times. For example, the all-mpnet-base-ve SBERT model was trained in a supervised way using over one billion text pairs. Even though its performance is higher (58.8%), we could bring the same base model (mpnet-base) close to SBERT's performance on some of the tasks in few minutes of self-supervised training using five orders of magnitude less data.
3.3 Representation of a dataset for analysis and visualization
3.3.1 Setup
Beyond MTEB, we evaluated the models' ability generate mcaningful representations of given datasct for data analysis and visualization. This application of text emboddings is essential for understanding the structure of dataset and identifying outliers or potential data quality issues (Gonvález-Mārquezz et. al. 2024
7

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Representation quality of given dataset. kNN accuracy of the mean-pooling representation in percent (A: 10). We used 9:1 train/test split for the kNN classifier. Columns 1-2, 5-7: off-the- shelf models. Columns 3 4: MPNet fine-tuned on each dataset using cropping and dropout augmentations. Reported values should be interpreted with error of up to +1%, corresponding to the binomial standard deviation 100 100 p)/n for test set size n 2 2000 (smallest dataset) and accuracy 0.5.
(1)
(2)
(3)
(4)
(5)
(6)
Model
MPNet
SimCSE
MPNet
MPNet
SBERT
BGE-base
BGE-large
Augmentations
Dropout
Crops
ICLR
37.4
45.7
46.8
58.9
63.3
63.3
63.5
arXiv
37.8
40.0
39.9
44.2
46.2
46.0
46.0
bioRxiv
58.6
59.0
60.7
61.8
65.2
64.4
65.5
medRxiv
43.5
47.2
47.8
52.4
56.8
55.7
55.3
Reddit
62.6
59.9
57.8
72.0
75.0
79.2
80.0
StackExchange
39.3
40.7
41.6
45.6
50.6
51.4
51.5
Average
46.5
48.8
49.1
55.8
59.5
60.0
60.3
Anand et al. 2023 yet standard benchmarks often overlook it Given its practical importance, included this evaluation to provide more comprehensive assessment of sentence representation quality across tasks.
We assossed the dataset representation quality through k-ncarost-noighbor (kNN) classification accuracy in the high-dimensional (d = 768) embedding space. This metric is computed by classifying each point according to the majority class among its k nearest neighbors, and then comparing predicted class against the true class. We used 10 with Euclidean distance and non-normalized embeddings, but we obtained similar results using cosine distance (which is equivalent to using normalized embeddings; Table (S) It is measure of local coherence: it is high if each point's nearest neighbors belong to the same class as the point itself. This metric is particularly relevant for data exploration applications, e.g. visualization or clustering, as many unsupervised learning algorithms for these tasks rely on the &NN graph of the data. While this metric depends on the quality of the ANN graph similarly to retrieval or reranking tasks from MTEB, is simpler evaluate as requires only class labels rather than annotated samples or ranked neighbors.
To evaluate the self-supervised fine-tuning strategy, fine-tuned the same base model with the same augmentations and loss as in the previous section, but this time using as training data the same dataset that is being represented. That mcans, for cach datasct being evaluated, the basc model was fine-tuned separately on that dataset for one single epoch. Note that we purposefully used the entire dataset first for self-supervised training and later for supervised evaluation. As the self-supervised training does not have access to class labels, this setting does not present overfitting issues. Also, the augmentations operate within samples (c.g. two fragments of one sample text are encouraged be similar), and not across samples (i.e. two samples from the same class not. encouraged to be similar). We obtained similar results when conducting the same experiment with train/test split for both self-supervised training and supervised evaluation (Supplementary Table S4h.
We performed our fine-tuning experiments on six datasets: the arXiv, bioRxiv, medRxiv, Reddit, and StackExchange datasots from the P2P clustering tasks of the MTEB (Muennighoff et al. 2023 and the Kobak (024).. The datasets differed in the number of samples (18 733 thousand) and classes (26 610; Table (S5). Four of them comprised scientific abstracts from different disciplines, and the other two consisted of internet posts.
ICLR dataset Marquéee
3.3.2 Results
We found that, on average across datasets, the base MPNet produced representations with low accuracy (46.5% Table 2 and almost no semantic structure visible in 2D visualizations using t-SNE (van der Maaten
& Hinton 2008 (Figure 2 Fine-tuning MPNet for epoch for each dataset. increased the performance
8

---

Published in Transactions on Machine Learning Research (02/2026)
ICLR
bioRxiv
medRxiv
StackExchange
MPNet
37.4
58.6
43.5
39.3
Crops
58.9
61.8
52.4
45.6
SBER
63.3
65.2
56.8
50.6
Figure 2: Dataset visualizations. t-SNE visualizations of MPNet, SBERT, and cropping- fine-tuned MPNet embeddings of different datasets. Color corresponds to class labels. Numbers show kNN accuracy in 768D embedding space. We used TSEE with default parameters (Poliaar et al. 2024 model was always fine-tuned on the same dataset, as Table
The fine-tuned
55.8% on average, which was still below SBERT with 59.5%. For some datasets this improvement was particularly large; c.g. the reprosentation of the ICLR dataset improved by over 20 percentage points.
As in Section 3.3 gropping-based fine-tuning outperformed dropout-based fine-tuning in all datasets (on average 55.8% Vs 49.1%, Table 2 columns 4 5). The off-the-shelf SimCSE model produced similar represen- lations to our tropout-based fine-tuned model on all datasets.
The difference in performance between the cropping- fine-tumed MPNet and SBERT (3.7 p.p.) was less prominent in this task than in the retrieval and reranking MTEB tasks, and their 2D visualizations were qualitatively similar (Figure 2 This confirmed that cropping- based fine-tuning produced a sentence-level model that yielded high-quality representations, despite the limited amount of fine-tuning and the lack of supervision.
Furthermore, on three scientific datasets (ICLR, arXiv, medRxiv), the cropping-based fine-tuning matched the performance SciNCL and SPECTER, two off-the-shelf embedding models specifically designed and trained represent scientific abstracts (Table (66, columns 8-9), using scientific citations as positive pairs. On the non-scientific datasets (Reddit and StackExchange), cropping-basee fine-tuning unsurprisingly out- performed both SciNCL and SPECTER.

---

Published in Transactions on Machine Learning Research (02/2020)
60%
bo
ICLR KNN accuracy
bo
50%
As
Reddit
40%
Biandi
Anxiv
MPNet base
100
200
300
Batch
Figure 3: Sentence vs. domain adaptation. KNN accuracy on the ICLR dataset for MPNet fine-tuned scparately on four different datasets (arXiv, bioRxiv, Reddit, ICLR).
Additionally, evaluated the two state-of-the-art models also used in the previous section: BGE-base and BGE-large. In this task, their performance was closer to SBERT, only minimally surpassing it. The difference in performance between BGE and the cropping- fine-tuned MPNet was smaller here (~4 p.p.), showing that representation quality after self-supervised fine-tuning highly benefits from in-domain training data.
3.3.3 Sentence and domain adaptation during fine-tuning
When fine-tuning a base model pre-trained on the token level with a sentence-level contrastive loss specific dataset, two mechanisms can contribute to the performance improvement: the model adapting to represent sentences ("sentence adaptation") and the model adapting to the domain of the training data ("domain adaptation"). To disentangle the contributions of these two potential mechanisms, we performed self-supervised fine-tuning on one dataset and evaluated the model's performance on a dataset from a different domain. We used three MTEB clustering P2P datasets from the previous section (arXiv, bioRxiv, and Reddit) training datasets, and ICLR the evaluation datasct. For comparable training conditions, used dataset subsets equal to ICLR's size. We fine-tuned the base model separately on each dataset and measured the ANN accuracy on the ICLR dataset. We also fine-tuned and evaluated directly on ICLR for comparison with the setting when both adaptation mechanisms are present.
We found that the ICLR ANN accuracy training curve had a similar shape for all training datasets, including the ICLR dataset (Figure) 3). Most of the improvement happened within the first 100 batches, and after that the ANN accuracy increased only slightly. This agrees with what we observed previously using the MTEB score (Figure
Training on arXiv and bioRxiv yielded better ICLR performance than training Reddit, likely because scientific abstracts of other disciplines (arXiv and bioRxiv) have greater domain similarity with ICLR ab- stracts than internet posts (Reddit). This suggests that the performance of sentence embeddings trained with self-supervision decreases for out-of-domain data.
As the domains of Reddit and ICLR datasets are very different, the improvement in ICLR ANN accuracy obtained when training the Reddit dataset must be mostly due to sentence adaptation rather than domain adaptation. This improvement the difference in final performance between training on the Reddit and on the ICLR datasets (Ap, 5.3 p.p.). Thus, we conjecture that the majority of the improvement in MTEB score when trained on the ICLR dataset (Table D was due to generic sentence-level adaptation. This may also explain why the gap between our fine-tuned MPNet and SBERT was larger for MTEB than for the &NN accuracy evaluation, since the first scenario the model was always evaluated on out-of-domain data (compared the data used for contrastive fine-tuning..
14.7 p.p. in
experiment (As in Figure 3), which was larger than
10

---

Published in Transactions on Machine Learning Research (02/2020)
Random (Cmps]
MPNet
60%
60%
60%
KNN accuracy
KNN accuracy
Random
MPNet (Crops)
MTEB score
MPNet (Dropout)
SBERT
50%
50%
50%
40%
40%
40%
30%
30%
30%
2
4
8
10
12
0
6
8 10
12
0
2
6 Layer
8
10
12
Number of unfrozen layers
Layer
Figure Representation quality across layers. (a) ANN accuracy after fine-tuning MPNet with different number of initial layers frozen. The embedding layer was frozen in all settings. Zero unfrozen layers corresponds to no fine-tuning. (b) ANN accuracy after each layer for MPNet before and after fine-tuning all layers, for SBERT, for randomly initialized model with and without training with cropping augmentations. Layer O corresponds to the embedding layer. (c) MTEB block average score after each layer for MPNet before and after fine-tuning with both kinds of augmentations, and for SBERT. Layer 0 corresponds to the embedding layer. Here the text embeddings were not normalized for the evaluation, unlike in Section 3.2 therefore exact values are slightly different from Table 1
3.4 Self-supervised training without pre-training
To determine whether the token-level pre-training was necessary to achieve good sentence representations of given dataset, we performed cropping-basce contrastive training the bert-base architecture from scratch, without using pre-trained MPNet weights (Table SS column 10). Here, the performance did not saturate after epoch,
continued training for 10 epochs. On average across datasets, the resulting KNN accuracy only 2.6 percentage points below the one obtained from fine-tuning the pre-trained MPNet, and for some of the datasets there was no noticeable performance difference at all. However, when training the model on ICLR and evaluating on MTEB (setup from Section 3.2 but without classification tasks) the results were much worse compared to using the pre-trained MPNet for fine-tuning (18.9 p.p. difference).
As additional ablation, we performed the cropping-based contrastive training of bare, randomly initialized, embedding layer. This is direct token embedding model without any transformer architecture whatsoever. Training it for 10 epochs, we obtained dataset embeddings that average were 3.5 percentage points below the ones obtained from fine-tuning the pre-trained MPNet in ANN accuracy (Table S6 column 11). Again, when trained on ICLR and evaluated on MTEB (without classification tasks; non-normalized embeddings), the results were very poor, with a much worse block average compared to the full pre-trained MPNet with additional cropping fine-tuning (13.4 p.p. difference).
These results highlight that the simple token embedding layer achieve reasonable embeddings of given dataset when trained with cropping augmentations (dropout augmentation worked much worse; Table S6 However, the transformer archilecture and pre-training knowledge are clearly necessary when generalizing to other tasks and domains.
4 Representation quality across layers
To investigate whether fine-tuning the entire MPNet model was necessary to obtain high-quality sentence representations, we performed cropping- fine-tuning on the ICLR dataset. while freezing the embedding layer and various numbers of imitial layers. We observed that the performance rapidly improved with the number of unfrozen layers, and fino-tuning only the last 2 out of 12 layers for one epoch was sufficient reach almost the same value of kNN classification accuracy fine-tuning the full model (Figure 4b). Unfreezing additional layers led only to minor further improvements.
11

---

Published in Transactions on Machine Learning Research (02/2026)
When all layers were unfrozen, the last few layers underwent the largest change during fine-tuning, while the early layers barely changed, in agreement with previous findings in the supervised setting (Merchant et al.) 2020; Mosbach et al.] 2020). To quantify this, we measured the representation quality after each hidden layer before and after finc-tuning the full model (Figure --cc). The gap between them was close to zero for early layers and increased towards the last layers (for both kinds of augmentations: dropout and crops). We observed the same effect when fine-tuning MPNet on other datasets and using the ANN evaluation (Figure S2
Intriguingly, the representation quality across layers in our fine-tuned model as well as in oft-of-the-bar SBERT formed a U-shaped curve (Figure fp-)): before fine-tuning the embedding layer representation had the highest NNN accuracy and almost the highest MTEB score, and after fine-tuning it was surpassed by only the last layers. Across other datasets, the shape was different and not always U-shaped (Figure (S2), but fine-tuned models always exhibited steep rise in performance towards layer 12. The randomly initialized models did not exhibit this shape: after SSL training, the performance monotonically increased and plateaued half-way through the layers (Figure p))
It has been argued that misalignment between the training objective and the downstream task can lead to representation quality peaking in the middle layers (Bordes et al., 2023). This does not explain why the representation quality of non-fine-tuned MPNet sometimes was the highest: in the embedding layer, but prior work found similar results when evaluating BERT and some other language models on MTEB (Skean et al. 2025). However, none of these models had been fine-tuned for text embeddings, so evaluating their text- embeedibnd performance is not very informative. Our results extend those by SKean al. (2025) and show that, once fine-tuned for text representation, models' representation quality does increase towards the last layers.
A common practice in computer vision is to have several fully-connected layers (projection head) between the last layer and the contrastive loss (Chen et al. 2020). which are discarded after SSL training (guillotine regularization), to correct for the misalignment between training and evaluation (Bordes et al., 2023). Our results show that this was not necessary when using the cropping augmentation, as the representation quality peaked in the last layer. However, with dropout augmentation and MTEB evaluation, the representation quality peaked in the penultimate layer (Figure A Importantly, it was still lower than with the cropping augmentation. When analyzing MTEB task categories separalely, we saw a similar effect with cropping augmentation and some of the MTEB tasks (classification and STS), where representation quality peaked in the penultimate layer (Figure S3
5 Limitations
Our main results focus on two augmentation techniques cropping and dropout due to their popularity in the text-embedding literature. We explored additional strategies in pilot experiments (random masking of a fraction of the words, and a combination of that with cropping; Section [AI]). but these performed worse compared to cropping and we therefore excluded them from the full analysis. An interesting direction for future work would be to compare cropping augmentation ion to L.M-geenrratd contrastive pairs based on
rephrasing or reformulation of the anchor (Jiang et al. 2022 Wang & Dou 2023 Abaskohi et al. 2023
Another direction of future research is to investigate the effect of using hard negatives during training. Hard negatives are negative samples that are semantically or structurally similar to the anchor. They can be identified using external sources of information (such as citations; [ohan et al., 2020 Ostendorff et al., 2022), Le. in supervised manner. Recent works have also investigated self-supervised ways of mining hard negatives (Xiong et al. 2021 Liu et al. 2024 Li et al. 2024) and argued that this can lead to improvement performance.
Our comparison of the two augmentation techniques on MTEB lasks relies on models fine-tuned on the ICLR dataset (Section 3.2). : representing an out-of-domain evaluation scenario. While we studied performance differences across multiple training datasets in the in-domain sctting (kNN accuracy, Section 3.3), we did not extend this to the MTEB evaluation. The main reason was that we wanted to avoid any train-test overlap and chose the ICLR dataset for training as it is not part of the MTEB evaluations.
12

---

Published in Transactions on Machine Learning Research (02/2026)
6 Discussion
We showed that self-supervised fine-tuning is a powerful strategy for producing high-quality text embeddings with minimal training on in-domain data. To this end, we systematically compared different self-supervised augmentation tochniques under the exact same training setup and showed that cropping augmentations were much better than dropout augmentations in all evaluation modalities. This finding is noteworthy because dropout augmentations are one of the most well-cited SSL approaches in the literature on text embeddings (Gao et al. 2021 and continue to be sometimes used for training modern text embedding models (BehnamGhader el al. 2024
Despite self-supervised fine-tuning substantially improving text embedding quality, there was still a gap compared to the supervisod SOTA models. It unclear whether this gap stems from supervision itself or from the much larger and more diverse datasets on which SOTA supervised models are trained. Such models typically leverage diverse data spanning multiple domains, minimizing out-of-domain scenarios in contrast to our limited self-supervised training on single dataset. It remains open question whether this gap be bridged with extensive self-supervised training on diverse data or whether supervision is a key element to achieve better performance.
One challenge in model evaluation is the overlap between training and evaluation data across models and benchmarks. Supervisod models often construct fine-tuning pairs using the same external notion of similarity (class labels, citation relationships, etc.) that evaluation benchmarks use. For instance, the MTEB Scidoc- sRR task uses citation relationships to determine relevant texts, while SBERT used citation pairs from the same SCIDOCS dataset for fine-tuning, artificially inflating its performance on that lask. Although MTEB attempts to address this by removing certain tasks and introducing "zero-shot" scores, they do not give details on which datasets and tasks are problematic for which models. We believe that greater transparency is needed in training and evaluation procedures, to facilitate comparisons between existing models. Our work avoids this problem by comparing different. augmentation strategies (crops vs. dropouts) identical and controlled settings.
Acknowledgments
The authors would like thank Carsten Eickhoff for feedback and discussions. This work was funded by the Deutsche Forschunggeemeisccaatt (K06222/2-1) and by the Gemeinnützige Hertie-Stiftang. Philipp Berens and Dmitry Kobak are members of the Germany's Excellence cluster 2064 "Machine Learning New Perspectives for Science" (EXC 2064, ref 390727645). The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Rita González Márquez.
References
Amirhossein Abaskohi, Sascha Rothe, and Yadollah Yaghoobzadeh. LM-CPPF: Paraphrasing-guiidel data augmentation for contrastive prompt-based few-shot fine-tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 670-681, 2023.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agrre SemFval-2012 Task 6: A pilot on semantic textual similarity. In SEM 2012 The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sizth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385 393, 2012.
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin M Schmidt, Adam Treat, and Andriy Mulyar. GPT4A1I-J: An Apache-2 licensed assistant-style chatbot. Nomic AT technical reports, 2023.
Randall Balestricro, Mark Tbrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A cookbook of self-supervised learning. arXiv preprint Xio:22300 12210, 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2Vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024.
Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regular- ization: Why removing layers is needed to improve generalization in self-supervised learning. Transactions on Machine Learning Research, 2023.
Fredrik Carlsson, Evangelina Gogoulou, Erik Ylipää, Amaru Cuba Gyllensten, and Magnus Sahlgren. Seman- tic ro-tuning with contrastive tension. In The Ninth International Conference on Learning Representations, 2021.
Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, et al. Benchmarking large language modcls for biomedical natural language processing applications and recommendations. Nalure communicalions, 16(1):3280, 2025.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and GeofTrey Hinton. A simple framework for rastivee learning of visual representations. In Proceedings of the 37th International Conference on Machine Learn- ing, Procoedings Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. SPECTER: Document-level transformers. In Proceedings of the 58th Annual Meeting
representation learning using of the Association for Computational Linguistics, pp. 2270-2282, 2020.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelith International Conference on Learning Represenlations, 2024.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171-4186, 2019.
Luyu Gao and Jamic Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguislics (Volume 1: Long Papers), pp. 2843 2853,, 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894 6910, 2021.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learning for un- supervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp- 879 895, 2021.
Rita Goneález-Márquez and Dmitry Kobak. Learning representations of learning representations. In ICLR 2024 Workshop on Data-centric Machine Leurning Research (DMLR): Harnessing Momentum for Science, 2024.
Rita González-Márrueez Luca Schmidt, Benjamin M Schmidt, Philipp Berens, and Dmitry Kobak. The landscape of biomedical research. Patterns, pp. 100968, 2024.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856.
Gautier Izacard, Patrick Lewis, Maria. Lomeli, Lucas Hosscini, Fabio Petroni, Timo Schick, Jane Dwivodi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. arXiv preprint ar arXi::2208.0322) 1(2):4, 2022b.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, and Qi Zhang. PrompiBERT: Improving BERT sentence embeddings with prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8826 8837, 2022.
Karen Spärck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 1972.
Chris Kamphuis, Arjen P. de Vries, Leonid Boytsov, and Jimmy Lin. Which BM25 do you mean? a large- scale reproducibility study of scoring variants. In European Conference on Information Relrieval, pp. 28-34. Springer, 2020.
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. Self-guided contrastive learning for BERT sentence repre- sentations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2528-222 2540, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In The Third International Conference on Learning Represenlations, 2015.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-Embed: Improved techniques for training LLMs generalist embedding models. In The Thirteenth International Conference on Learning Representations, 2025.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9119 9130, 2020.
Chaofan Li, Minghao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Defu Lian, Yingxia Shao, and Zheng Liu. Making text embedders few-shot learners. In The Thirteenth International Conference on Learning Representations, 2025.
Shiyu Li, Yang Tang, Shizhe Chen, and Xi Chen. Conan-embedding: General text embedding with more and better negative samples. arXiv preprint ar Xix:24408. 15710, 2024.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint 2tff:2338.03288 2023.
Fangyu Liu, Ivan Vulié, Anna Korhonen, and Nigel Collier. Fast, effective, and supersid:: Transforming masked language models into universal lexical and sentence encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Longuage Processing, pp. 1442 1459, 2021.
Wenxiao Liu, Zihong Yang, Chaozhuo Li, Zijin Hong, Jianfeng Ma, Zhiquan Liu, Litian Zhang, and Feiran Huang. HNCSE: Advancing sentence embeddings via hybrid contrastive learning with hard negatives. arXiv preprint X11::2411..1156 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXie:1907.11699 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representations. In The Sirth International Conference on Learning Representations, 2018.
Amil Merchant, Flahe Rahimtoroghi, Ellie Pavlick, and Tan Tenney. What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackborNLP Workshop on Analyzing and Interpreting Newral Networks for NLP, pp. 33 44, 2020.
Marius Mosbach, Anna Khokhlova, Michael A Hedderich, and Dictrich Klakow. On the interplay between fine-tuning and sentence-level probing for linguistic knowledge in pre-trained transformers. In Findings of the Association for Computational Lânguisties: EMNLP 2020, pp. 2516, 2020.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904 2022.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text embedding bench- mark. In Proceedings of the 1 77hh Conference of the European Chapter of the Association for Computational Linguislics, pp. 2006 2029, 2023.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprinl ar 220: 10005, 2022.
Jianmo Ni, Gustavo Hernández Åbrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence-T5: Scalable sentence encoders from pre-trained text-to-text models. In Findingg of the association for computational linguistics: ACL 2022, pp. 1864 1874, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXim:807.0085 0374877, 2018.
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Bela Gipp, and Georg Rehm. Neighborhood con- trastive learning for scientific document representations with citation embeddings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 31670-11688 2022.
Pavlin G Poličar, Martin Stražar, and Blaž Zupan. openTSNF: A modular Python library for t-SNE dimensionality reduction and embedding. Journal of Statistical Software, 109:1 30, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training, 2018. URL
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Reseurch, 21(1):5485 5551, 2020.
Ori Ram, Yoav Levine, Itay Dahmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, Brown, and Yoav Shoham. In-context retrieval-angmeenod language models. Transactions of the Association for Computa- tional Linguistics, 11:1316 1331, 2023.
Nils Reimers and Iryna Gurevych. Centenco-BERT: Sentence embeddings using siamese BERT-networt Tn Proceedings of the 2019 Conference Fampérical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3982 3992, 2019.
Stephen E RRoerrtoo,, Steve Walker, Susan Jones, Micheline M Shancock-Beauliet Mike Gatford, et al. Okapi al. TREC-3. British Library Research and Development Department, 1995.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Proceedings of the 4 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2025.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic-Yan Liu. MPNet: Masked and permuted pre-training for language understanding. Advances in Neural Information Processing Systems, 33:16857 16867, 2020.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: wiatrructtooniinntune text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1102 1121, 2023.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. Whitening sentence representations for better semantics and faster retrieval. Xix preprint or 2rXi::2103.1599 15316, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A het- erogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirby-fifu Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(1)) 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Aduances in Neural Information Processing Systems, 30, 2017.
Bin Wang and C-C Jay Kuo. SBERT-WK: A sentence embedding method by dissecting bert-based word models. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2146 2157, 2020.
Hao Wang and Yong Dou. SNCSE: contrastive learning for unsupervised sentence embedding with soft negative samples. In International Conference on Intelligent Computing, pp. 419 4311.33.1000 Springer, 2023.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint ar 2r1iv:2212 233503333 2022.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pp. 11897-11916, 2024.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Nie. C-pack: Packed resources for general chinese embeddings. Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pp. 641-649, 2024.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In The Ninth International Conference on Learning Representations, 2021.
Lingling Xu, Haoran Xie, Zongxi Li, Fu Lee Wang, Weiming Wang, and Qing Li. Contrastive learning models for sentence representations. ACM Transactions on Intelligent Systems and Technology, 14(4):1-38 34, 2023.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Associalion for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1065-50755 2021.
Meng Yang, Yueyuxiao Yang, Chenxi Xie, Ming Ni, Jian Liu, Huanming Yang, Feng Mu, and Jian Wang. Contrastive learning enables rapid mapping to multimodal single-cell atlas of multimiliion scale. Nature Machine Intelligence, 4(8):696 709, 2022.
Junlei Zhang, Zhenzhong Lan, and Junxian He. Contrastive learning of sentence embeddings from scratch. In Proceedings of the 2023 Conference on Empirical Melhods in Nalural Language Processing, pp. 3916 3932, 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18222 2023.
17

---

Published in Transactions on Machine Learning Research (02/2026)
A
Appendix
A.1 Augmentation and hyperparameter choices for self-supervised fine-tuning
To select the optimal hyperparameters for our self-supervised fine-tuning, we performed a detailed hyperpa- rameter analysis using the ICLR dataset. We ran the fine-tuning of MPNet for one epoch, and assessed the final ANN classification accuracy (Figure S1 In each experiment, all other parameters were kept at their default values described in Section 3.1
Pooling We compared four different pooling strategies for forming the final sentence representation: age pooling, the classification token [CLS], the separation token [SEP] (appended at the end of each input text; similar to [EOS] token), and the seventh token (as an example of an arbitrary token). We obtained the best results using the average pooling and [SEP] token, with the other two options performing less well (Figure a).).
When evaluating off-the-shelf models, we always used the mean pooling. On the ICLR datasel, some off-the- shelf models showed slightly higher &NN accuracy in the [SEP] token representation than using the [CLS] token or average pooling, but the difference
small (Table S7 It has recently been shown in computer vision setting that additional tokens can be used by the translormer model 'registers' to store high-level features Darcet et al., 2024). Our results suggest that the same can happen with language models, since the [SEP] token often serves as a good sentence representation despite not being explicitly used for training.
We compared several values of temperature from 0.005 to 5.0, and found that the per- formance decreasod with increasing temperature, with r 0.005 and r = 0.05 yielding similar results (Figure STa). The value 0.5 used in SimCLR (Chen et al. 2020 performed less well.
Temperature
Cropping augmentation Our data augmentation consisted of 'cropping out' t consecutive sentences. We used periods (:) as sentence delimitators, and discarded sentences shorter than 100 characters and longer than 250 characters. We varied the number of consecutive sentences (decreasing the batch size accordingly, to make it fit into the GPU memory) and found that the performance generally decreased with t, with the optimal number being = 2 (Figure SIp). Note that in our sampling it was possible for the positive pair of text chunks overlap (but not coincide exactly).
Masking augmentation We also experimented with a masking augmentation that replaced a certain fraction of tokens in each imput chunk with the BERT's special [MASK] token. This was done top of the cropping augmentation. We found that masking led to deterioration of performance (Figure (S1p). Using masking augmentation without cropping (i.e. forming positive pairs by applying two different masking pallerns to the entire abstract) did not produce competitive results either.
Learning rate The performance increased with increasing the Adam's learning rate (Figure II until it became too large and the training diverged (n 2 5 10-4) For the bare embedding layer training, the optimal learning rate was 1) 5 10-1
A.2 Description of MTEB tasks
Here we provide a brief description of the MTEB tasks that we used for evaluation. We used the MTEB library (attps://github. com hammm/mbedddnngg-- hwteee/digg-ebbbnnenrrrreeeeeeee benchmark/mteb) for evaluation. Please see Muennighoff et al. 2023 and references therein for further details.
Clustering Each of the used datascts consists of texts and ground-truth class labels for cach text. The (exts are embedded using the model, and the embedding vectors are clustered using a mini-batch K-means algorithm with batch size b = 32 and K equal to the true number of classes. The evaluation score is the so called V-measure of agreement between cluster labels and class labels, which is invariant to the permutation of cluster labels. The whole procedure done separately on several non-overlapping batches and the results are averaged.
18

---

Published in Transactions on Machine Learning Research (02/2026)
Retrieval Each of the used datasets consists of a corpus of documents, queries, and mapping from each query to the relevant documents. The documents and queries are embedded using the model, and the aim is to find the relevant documents within the neighborhood of the query in the embedding space. Neighbors are determined using cosine similarity, and after ranking them, normalized discounted cumulative gain (NDCG) at nearest neighbors serves as the performance metric. NDCG is obtained by normalizing discounted cumulative gain (DCG), which is defined as:
rel;
DCG@K = Σ
logz (i + 1)²
where rel; is the relevance score of the item at position (which can be either binary or graded), and K is the number of top results considered. The Ideal DCG (IDCG) then calculated by sorting the results the optimal order (most relevant first). Finally, NDCG obtained by normalizing DCG with IDCG:
NDCG@K DCG@K IDCG@K 100%.
Reranking Each of the used datasets consists of query texts and list of relevant and irrelevant reference texts for cach query. They are all embedded with the model, and for each query, the text embeddings are ranked based on the cosine similarity to the query embedding. The resulting ranking is compared to the ground-truth ranking, scored for each query via average precision (AP) metric, and averaged across all queries (MAP). Average precision is defined as:
P(k) rel(k)
AP
number of relevant documents'
where P(k) is the precision rank k, rel(k) is the relevance of the item rank & (in this case only binary; if it is relevant and 0 otherwisc), and n is the number of retricved items. Precision at rank k is defined as:
P(k) Number of relevant items in top results"
with values going from to 1. MAP values also go from 0 to 1, with higher values being better.
STS Each of the used datasots consists of a set of sentence pairs, each pair with numerical score from to indicating similarity between the sentences (5 being most similar, and 0 most dissimilar). All sentences embedded with the model, and for each pair, the embedding similarity is computed using cosine similarity. These embedding similarities are then compared against ground-truth similarities using Spearman correlation.
Classification Each of the used datasets consists of a set of text and labels, which can be either topic labels or sentiment labels. The texts from the train and test sets are embedded with the model. The train set embeddings are used to train a logistic regression classifier with 100 maximum iterations, which scored the test set. The main metric is linear classification accuracy.
A.3 Software and Data
The analysis code is available at hutpp://ggthhh.ccm/rrrrrssaaaa/s
19

---

Published in Transactions on Machine Learning Research (02/2026)
B Supplementary tables and figures
Table $1: Different base models. This is an extension of Table Different base models (MPNet, BERT, RoBERTa) before and after fine-tuning with our main augmentations. All values in percent, higher better. Models in columns 2 2-3, 5 6, and 8-9 were fine-tuned the ICLR datasct.
(1)
(2)
(3)
(5)
(6)
(7)
(9)
Model
MPNet
MPNet
MPNet Crops
BERT
BERT Dropout
BERT
RoBERTa
RoBERTa Dropout
RoBERTa Crops
Augmentations
Dropout
Crops
Clustering
27.4
34.7
37.7
32.3
35.1
36.4
21.9
35.4
36.8
Reranking
41.8
51.9
46.7
47.4
51.5
40.3
47.1
48.8
Retrieval
24.2
31.8
15.6
29.4
7.2
24.0
25.0
STS
52.0
65.9
73.4
57.1
59.4
69.6
59.3
69.8
70.2
Classification
61.6
63.3
64.3
65.5
62.6
64.2
63.3
63.5
64.7
Block average
38.9
46.9
51.8
43.4
45.2
50.2
38.4
48.0
Table S2: Details of used models. Model name, Hugging Face URL, citation, and year.
Name
Hugging Face
Citation
Year
MPNet
microsoft/apnet-base
Song et al.
2020
2020
BERT
bert-base-uncased
(Devlin et al.
2019
2018
RoBERTa
roberta-base
Liu
2019
2019
SimCSE
Gao
202
2021
SciNCL
malteos/scincl
Ostendorff a.l
2022
2022
SPECTER
allenai/specter
Cohan et al.
2020
2020
SBERT
Reimers &
Gurevych
2019
2021
BGF-base
BAAT/bge-base-en--1..5
Xiao et al.
2024
2024
BGE-large
BAAT/bge-largo-nn-1.55
Xiao ot al.
2024
2024
Table S3: Effect of post-processing transformations. &NN accuracy on the ICLR dataset using different post-processing transformations of the MPNet mean pooling representation, obtained via the Euclidean and the cosine metrics for finding nearest. neighbors, before and after fine-tuning the model on the ICLR dataset.
Euclidean Cosine
Before fine-tuning
Raw
37.4%
39.6%
Centerod
37.0%
35.8%
Whitened
5.5%
18.5%
After fine-tuning
Raw
58.7%
59.3%
Centered
55.7%
54.7%
Whitened
36.8%
55.0%
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table S4: Representation of a given datasct; SSL with train/test split. Score is ENN accuracy of the mean pooling representation in percent. Unlike in Table 2 here self-supervised training was only done on the train set; then the classifier was trained on the train set and evaluated on the test set (we used a 9:1 train/test split).
Model
MPNet
MPNet
Pre-trained
yes Dropout
yes
Angmentations
Crops
ICLR
46.7
56.8
arXiv
38.9
44.1
bioRxiv
60.9
61.6
medRxiv
47.8
52.3
Reddit
59.6
71.8
StackExchange
41.3
45.1
Average
49.2
55.3
Table S5: Dataset statistics. Statistics of the datasets used in the experiments of Table 2 The Xiv. bioRxiv, medRxiv, Reddit, and StackExchange datasets are from the P2P elustering tasks of the Massive Text Embedding Benchmark (MTEB) [Muennighoff et al. 2023), and the ICLR dataset taken from Märquez & Kobak 2024). Length refers to the number of characters each sample text. For the arXiv dataset, we used secondary paper categories (e.g. "cs.Al") as labels.
Dataset
Samples
Classes
Mean length
Std length
TCLR
24347
46
1248
316
arXiv
732 723
180
1010
432
bioRxiv
53 787
26
1664
542
medRxiv
17 647
51
1985
843
Reddit
459 399
450
728
710
Stack Exchange
75.000
610
1091
809
Table S6: Representation of given dataset, additional models. This is an extension of Table 2 Columns 3-4: MPNet finc-tuned on cach dataset using dropout and cropping augmentations, identical to columns 4 from Table 2 Columns 9: off-the-shelf models. Columns 10-11: Full BERT model and embedding layer (Emb.) trained from scratch for 10 epochs using cropping augmentations.
(3)
(4)
(8)
(9)
(10)
(11)
Model
MPNet. yes Dropout
MPNet
SPECTER
SciNCL yes
BERT
Emb.
Pre-trained
yes Crops
yes
no
no
Augmentations
Crops
Crops
ICLR
46.8
58.9
56.8
57.0
57.1
57.3
arXiv
39.9
44.2
44.2
45.2
44.3
43.4
bioRxiv
60.7
61.8
64.8
66.4
60.6
60.7
medRxiv
47.8
52.4
52.6
52.8
44.9
49.1
Reddit
57.8
72.0
55.2
57.3
61.8
63.6
StackExchange
41.6
45.6
41.5
42.9
45.4
45.2
Average
49.1
55.8
52.5
53.6
52.4
53.2
21

---

Published in Transactions on Machine Learning Research (02/2026)
Table S7: Comparison of pooling strategies. kNN accuracy on the ICLR dataset of different of-the- shelf models using mean pooling, [CLS] token, and [SEP] token as sentence representations. DeCLUTR and SBERT were originally fine-tuned using mean pooling. SimCSF, SeiNCL, and SPECTER were originally fine-tuned using the [CLS] token. Best representation is in bold, best representation for each model is underlined.
Average
[CLS]
[SEP]
MPNet 37.4%
31.8%
36.3%
BERT
40.6%
28.2%
33.1%
SimCSE
45.7%
43.5%
46.4%
DeCLUTR
50.3%
45.0%
34.8%
SciNCL
57.0%
56.8%
57.8%
SPECTER
56.8%
54.1%
58.5%
SBERT
63.3%
56.8%
59.8%
60%
55%
50%
ANN
45%
Awerage
[CLS]
[SEP]
35%
0.005
0.05
0.5
5.0
10
Temperature
Number of sente nces
60%
55%
MNN sccurscy
50%
45%
40%
Avg. Grope
Aug Drepout
35%
0.0
0.2
0.4
0.5
0.8
f6-6
16-5
10-4
10-3
Fraction masked tokens
Learning rale
Figure S1: Hyperparameter tuning. kNN accuracies on the ICLR dataset used for self-supervieed training and evaluation, as function of different hyperparameter values. (a) Temperature т used to scale the similaritics the loss function. (b) Number of consecutive sentences used in the cropping augmentation. The minibatch size b was adapted depending on to make fit into our GPU memory: we used = 128 for 1; for 2,3, 4; b 32 for / = 5, 6, 7,8, 9; and b 16 for 10. (c) Fraction of masked tokens used in addition of the cropping augmentation. (d) Learning rate used by the Adam optimizer.
22

---

Published in Transactions on Machine Learning Research (02/2026)
medRxiv
b
Reddit
StackExchange
fine-luning
fineduninn
fine-luingg Alter tne-tuning
After time ine-tuning
After 1no suniig
50%
70%
KNN sccureo)
45%
85%
40%
40%
80%
55%
10 12
10 12
10 12
Layer
Layer
Layer
Representation quality across layers. ANN accuracy after each layer for MPNet before and after fine-tuming in the (a) medRxiv, (b) Reddit, and (c) StackExchange datasets. Evaluation is also done the respective dataset.
Figure S2:
MPNet crops
80%
70%
60%
50%
MTEB score
40%
30%
Clustering
20%
Reranking
Retrieval
10%
Classfiiationn
Black
0%
10
12
Layer
Figure S3: MTEB score across layers. MTEB score using the representation after cach layer for MPNet after cropping-based fine-tuning (as in Figure 4 split by task modality. Layer corresponds to the embedding layer. Asterisks highlight the highest performance in each task modality.
23


Reviews:
[{"review_id": "GJ3etLUqQr", "reviewer": "Reviewer_db5g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper compares two augmentation strategies for positive pair generation in\ncontrastive learning of text embedding — dropout and text cropping. Authors show that cropping consistently outperforms dropout on some MTEB tasks and several in-domain datasets, achieving results that are only\nmarginally below supervised SOTA. The pape ralso provides analysis on semantic structure of resulting embeddings, assessing dataset representation quality using KNN, and analysis on what causes the improvements in quality of text embeddings after fine-tuning (is it \"sentence adaptation\" or \"domain adaptation\"). Further, authors investigate which layers produce embeddings of the best quality, showing that fine-tuning only the last layers of a model lead to similar fine-tuning performance. \nOverall, the paper provides strong and consistent evidence that cropping-based fine-tuning outperforms dropout-based one across modalities in the setup of the paper, while being fast. This can be useful for practitioners who build self-supervised pipelines\n\nStrenghts:\n- Strong evidence that cropping outperforms dropout as an augmentation strategy for self-supervised fine-tuning of text embeddings in the given setup. That might be useful for practitioners who build self-supervised pipelines\n- The overall setup is clearly defined and seems to be reproducible (maybe except cropping algorithm parameters, e.g. overlap rate control, sentence boundary detection strategy, etc.). Release of the code would be beneficial (there is a phrase \"The analysis code is available at URL\" in the paper, but URL is invalid). Whole setup is controlled, effects of augmentations are isolated, which makes results trustable\n- Good analyses: 1) analysis of semantic structure of resulting embeddings 2) analysis on causes of improvements in quality of text embeddings after fine-tuning (is it \"sentence adaptation\" or \"domain adaptation\"), with evidence that SSL might underperform in out-of-domain settings 3)analysis on which layers produce embeddings of the best quality 4) evidence that fine-tuning only the last layers of a model lead to similar fine-tuning performance 5) analysis of speed (cropping-based fine-tuning is fast)\n\n\nWeaknesses:\n- Only one model (MPNet), few MTEB subtasks (clustering, reranking, retrieval, and STS), and two augmentation strategies (cropping and dropout) are considered. There is at least classification task that is different from mentioned in the paper, and, in my opinion, is worth testing on. Or a discussion on why these four subtasks are good indicators of overall performance would be beneficial. Regarding augmentation strategies, authors mentioned that they did not use hard negatives. I wonder if hard negatives could boost performance and influence results, as from my experience with contrastive learning in images this might be the case. \n- Abstract is hard to read: it begins with discussion of text embeddings, then jumps to images: \"This contrasts with computer vision, where self-supervised trainingbased on data augmentations has demonstrated remarkable success.\", and it is hard to understand if the next sentence is about text or visual domain. I would also think that SSL strategies for text embeddings are worth exploring regardless of their success in vision domain, as strategies used in these domains are not directly applicable  between domains, and, at least for me, this passage about images was confusing", "claims_and_evidence_yn": "Yes", "claims_explanation": "Paper is clearly written, experimental setup and analyses are also clear", "audience_yn": "Yes", "audience_explanation": "I think that the topic discussed in the paper (comparison of augmentation strategies for SSL for text embeddings) and results can be useful to practitioners doing SSL for text embeddings", "requested_changes": "My biggest concerns are that only one model is used (MPNet), and augmentation setup is highly limited (for example, usage of hard negatives is not discussed, though might have effect on performance). In my opinion, for the results of the paper to be clearly convincing to practitioners doing SSL, results should be obtained on multiple models and all the major parameters of augmentations (which is, in this case, usage of hard negatives ). However, I am not quite familiar with what models are typically used for SSL on text embeddings, and what are typical parameters of augmentations in this area, so I would be happy to discuss these points with the authors", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cmk4WLl6Ea", "reviewer": "Reviewer_Jwws", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows that cropping augmentation outperforms dropout (SimCSE) for self-supervised text embeddings. Cropping yields stronger results on MTEB and in-domain datasets, improving quality rapidly with minimal data and compute. Most gains arise from sentence-level adaptation, concentrated in the final transformer layers, allowing efficient partial fine-tuning. While cropping-based models approach supervised baselines like SBERT and SciNCL in-domain, they remain weaker on out-of-domain data. Overall, cropping is a simple and efficient augmentation that produces high-quality embeddings and challenges the dominance of dropout-based self-supervised methods. \n\nStrengths: \n1. The paper is well-structured and easy to follow. \n2. The problem investigated is interesting and important for training language models. \n3. The experimental results reveal some interesting findings, including (1) cropping is better than dropping, (2) the layer-wise adaptation is good for in-domain adaptation but not out-of-domain adaptation. \n\nWeaknesses: \n1. Despite cropping’s strong in-domain results, the method still underperforms supervised models like SBERT and BGE on out-of-domain tasks.\n2. The study focuses mainly on cropping vs. dropout, leaving other promising augmentations (e.g., paraphrasing with LLMs, semantic reformulations) underexplored.\n3. Some benchmarks overlap with data used in supervised models (e.g., SBERT with SCIDOCS), making absolute performance comparisons less conclusive and highlighting the need for more transparent evaluation settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper conducts a comprehensive evaluation to support the claim made in the paper.", "audience_yn": "Yes", "audience_explanation": "Pre-training language models is an important topic in the machine learning community. The paper investigates the tricks in training language models.", "requested_changes": "See weaknesses.", "broader_impact_concerns": "This paper shows that simple cropping augmentations can make self-supervised text embeddings more efficient and competitive, potentially reducing reliance on costly supervised datasets for building high-quality language representations", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OSWnBls0nC", "reviewer": "Reviewer_8WK6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the use of cropping to create positive pairs for document similarity learning, compared to dropout-based construction as well as current larger models.  Two settings are evaluated: (1) training on ICLR abstracts dataset and evaluating on a subset of MTEB tasks; (2) training on a few different arxiv abstract-based tasks and evaluating knn-classification on ICLR.  In both settings, simple cropping constructions with simple models (even a BOW embeddings-only model) provided performance better than dropout methods and sometimes approaching larger models' performance.", "claims_and_evidence_yn": "No", "claims_explanation": "While the results are suggestive and interesting points of evidence, they fall short of the more general claims that are made.  I would not conclude \"using text crops is consistently better than dropout\" (claim 1), or \"for out-of-domain data ... but for in-domain data ...\" (claim 2). These were shown they *can* be the case here, but the evaluation is too limited (only ICLR and small mteb subset) to show more than sometimes this simple method performs well.  Likewise, the claim that \"Cropping-based fine-tuning is very fast\" (sec 3.2.3) is pretty sweeping, but all that was shown is the initial adjustment from fine-tuning extends well to these small datasets.  Again, it can be a useful method, and starts to make the case that it can be used as a simple and potentially stronger baseline, but it's not clear whether the behavior seen here extends beyond this one dataset.", "audience_yn": "Yes", "audience_explanation": "I think the presentation of the findings could be adjusted and the investigation further developed, but overall this seem interesting to know about.  At the least it suggests a simple baseline system for text embeddings.", "requested_changes": "See claims section above.  Overall the application experiments are too limited to make broad statements.", "broader_impact_concerns": "none", "additional_comments": "The results for embeddings with no transformer, trained on crops, I think is interesting and could be further explored (Table S5 / sec 3.3.4 \"As an additional ablation...\") to get more clues as to what is important for these tasks.  In particular:  (1) how much does the vocabulary overlap between tasks, and how is out-of-vocab handled (ICLR data vs bioarxiv, for example)?  (2) Was this juts averaging word embeddings as the comparison model, or was there any small MLP etc on top?  It sounds like it's just averaging embeddings --- in which case it indicates a BOW model is almost as performant on these tasks as most of the more complex methods.  (3)  What are tasks where the BOW model fails, and does the fine-tuning explored in this paper work in these settings?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)