Length is a Curse and a Blessing for Document-level Semantics
Authors: Chenghao Xiao
Yizhi LI
G Thomas Hudson
Chenghua Lin
Noura Al Moubayed
Venue: EMNLP 2023 Main (2023)
Keywords: contrastive learning
sentence representation learning
document representation learning
semantics
document length
TL;DR: We present the overlooked length vulnerability of contrastive text encoders, and use length as a standalone solution to provide semantic signal for sentence and document-level representation learning.

Abstract: In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models.
In this work, we question the length generalizability of CL-based models, i.e., their vulnerability towards length-induced semantic shift. 
We verify not only that length vulnerability is a significant yet overlooked research gap, but we can devise unsupervised CL methods solely depending on the semantic signal provided by document length. 
We first derive the theoretical foundations underlying length attacks, showing that elongating a document would intensify the high intra-document similarity that is already brought by CL. Moreover, we found that isotropy promised by CL is highly dependent on the length range of text exposed in training.
Inspired by these findings, we introduce a simple yet universal document representation learning framework, 
**LA(SER)$^3$**: length-agnostic self-reference for semantically robust sentence representation learning, achieving state-of-the-art unsupervised performance on the standard information retrieval benchmark. [Our code is publicly available.](https://github.com/gowitheflow-1998/LA-SER-cubed)

Length is a Curse and Blessing for Document-level Semanties
Chenghao Xiao Yizhi Li G Thooms Hudson
Chenghua Lin Noura Al Moubayed Department of Computer Science, Durham University, UK Department of Computer Science, The University of Manchester, UK [chenghao xiao, g. hudson, noura. al al--moobayed yizhi. 1 wilehtttmmmi tmail. com
uk
chenghua wwtighha.1i@@manccciic n@manchester .ac.erraaccc.... uk
Abstract
Original and Elongated Sentence
In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models. In this work, we question length generalizability of CL- based models, i.c., their vulnerability towards length-induced semantic shift. We verify not only that length vulnerability is significant yet overlooked research gap, but can de- vise unsupervised CL methods solely depend- ing the semantic signal provided by doc- ument length. We first derive the theoretical foundations underlying length attacks, show- ing that elongating document would intensify the high intra-document similarity that is al- ready brought by CL. Moreover, we found that isotropy promised by CL is highly dependent on the length range of text exposed in training. Inspired by these findings, we introduce a sim- ple yet universal document representation learn- ing framework, LA(SER)3: angtt--ggnstic self-reference for semantically robust sentence representation learning, achieving state-of the art unsupervised performance on the standard information retrieval benchmark. Our code is publicly available.
What What is NLP? x20
Original and Elongated Sentence Sp
What is NCP7
What computer vision?
Text Encoder
What computer vision?
What computer vision? +50
What computer vision? x100
Sg x1
0.06
Similarity Matrix
Sg x20
0.01
0.23
Sg *50
0.29
0.34
9g
0.03
0.31
0.38
0.42
L*$S
02*
0gx'S
Figure 1: Demonstration of Elongation Attack Sen- tence Similarity. The similarity between sentence SA and Sp incorrectly increases along with elongation, i.c. cop--and-comcatenate the original sentence for multiple times, despite the semantics remain unaltered.
Due to the document length misalignment these two types of tasks, the potential gap in mod- els' capability to produce meaningful represen- tation at different length ranges has been rarely explored (Xiao et al., 2023b). Studies of docu- ment length appear have been stranded in the era where methods are strongly term frequency- based, because of the explicit reflection of docu- ment length to sparse embeddings. with little atten- tion given on dense encoders. Length preference for dense retrieval models is observed by Thakur et al. (2021), who show that models trained with dot-product and cosine similarity exhibit different length preferences. However, this phenomenon has not been attributed the distributional mis- alignment of length between training and inference domains/tasks, and it remains unknown what abil- ities of the model are enhanced and diminished when trained with a certain length range.
1 Introduction
In recent years, contrastive learning (CL) has be- come the go-to method train representation coder models (Chen al., 2020; He et al., 2020; Gao et al., 2021: Su et al., 2022). In the field of natural language processing (NLP), the effective- ness of the proposed unsupervised CL methods is typically evaluated on two suites of tasks, namely, semantic textual similarity (STS) (Cer et al., 2017) and information retrieval (IR) (e.g., Thakur et al. (2021)). Surprisingly. large number of works only validate the usefulness of the learned repre- sentations on STS tasks, indicating strong but widely-adopted assumption that methods optimal for STS could also provide natural transferability to retrieval tasks.

---

In this work, we provide an extensive analysis of length generalizability of standard contrastive learning methods. Our findings show that, with default contrastive learning, models' capability to encode document-level semantics largely comes from their coverage of length range in the training.
preferences. Xiao al. (2023b) further devised "ad- versarial length attacks" text encoders, demon-
strating that this vulnerability can easily fool text encoders, making them perceive a higher similarity between a text pair by only copying one of them n times and concatenating to itself.
We first depict through derivation the theoretical underpinnings of the models' vulnerability towards length attacks. Through attacking the documents elongation
In this section, we first formalize the problem of length attack, and then analyze the most important pattern (misaligned intra-document similarity) that gives rise to this vulnerability, and take an atten- tion mechanism perspective derive for the first time the reason why contrastive learning- based text encoders can be attacked.
by the simple operation, we show that the vulnerability comes from the further intensified high intra-document similarity that is already pronounced after con- trastive learning. This hinders a stable attention towards the semantic tokens in inference time. Further, we show that, the uniformity/isotropy promised by contrastive learning is heavily length- dependent. That is, models' encoded embeddings are only isotropic on the length range seen in the training, but remain anisotropic otherwise, hinder- ing the same strong expressiveness of the embed- dings in the unseen length range.
Simple Length Attack Given a sentence S with tokens {21, Inh, we artificially construct its elongated version by copying it times, and concatenating it to it- self. For instance, if = 2, this would give us S {=1,..., Dn T1, .... In Loosely speak- ing, we expect the elongation to be a "semantics- preserved" operation, as repeating a sentence m times does not change the semantics of a sentence in most cases. For instance, in the context of in- formation retrieval, repeating document d by m time should not make it more similar to a query q. In fact, using pure statistical representation such tf-idf (Sparek Jones, 1972), the original sentence and the elongated version yield exact same repre- sentations:
Problem Formulation:
In the quest to bridge these unideal properties, we propose simple yet universal framework, LA(SER)S: Length-Age Aangth-Agnosticc SElf Reference for SEmantically Robust SEntence Representation learning. By providing the simple signal that "the elongated version of myself 1) should still mean myself, and thus 2) should not become more less similar my pairs", this framework could not only act as an unsupervised contrastive learning method itself by conducting seff-refer,n but could also be combined with any contrastive learning- based text encoding training methods in a plug-and-pla fashion, providing strong robustness to length at- tacks and enhanced encoding ability.
5 a S(S, m) tf (f) - (f)
(1)
(2)
where f(-) denotes the elongation operator, and is random integer.
Therefore, no matter according the semantics- preserved assumption discussed previously, or reference from statistics-basee methods (Sparck Jones, 1972), one would hypothesize Transformer-bused models to behave the same. Formally, we expect, given Transformer-basee text encoder g(-) to map document into document embedding, we could also (ideally) get:
We show that, our method not only improves con- trastive text encoders' robustness to length attack without sacrificing their representational power, but also provides them with external semantic sig- nals, leading to state-of-the-an unsupervised perfor- mance on the standard information retrieval bench- mark.
2 Length-based Vulnerability of Contrastive Text Encoders
g(S) 9
Observation 1: Transformer-based text en- coders perceive diffferent semantics in original texts and elongation-attacked texts. The cen- tral problem is: given a comsformerrbaseed text encoder g(.), it is found empirically that:
Length preference of text encoders has been ob- served in the context of information retrieval (Thakur al., 2021), showing that contrastive learning-based text encoders trained with dot- product or cosine similarity display opposite length
g(S) + 9(5)
(4)

---

MinLLM fine tuned Short Question Pairs
Vonila NInLM
after elongation attacks, greatly increasing the per- ceived similarity, even for pairs that are not positive pairs. This phenomenon indicates a built-in vul- nerability in contrastive text encoders, hindering their robustness for document encoding. For refer- ence, we also plot out the same set of results on the vanilla MiniLM (Figure 2. right), demonstrating an opposite behavior, which will be further discussed in Proof of Concept Experiment 2.
van to tost
vandla tox: length allack
wrghh wtiakk 100
332
0-1 54
5,4 Fostive Poirs
10
32
64
of Positive Pairs
Figure 2: Distribution of positive pair cosine similar- ity. Left: MiniLM finetuned on only short document pairs with contrastive loss displays a favor towards attacked documents (longer documents). Right: the vanilla model displays an opposite behavior.
Observation 2: Intra-document token interac- tions experience a pattern shift after elongation attacks. Taking an intra-document similarity per- spective (Ethayarajh, 2019), we can observe that, tokens in the elongated version of same text, do not interact with one another as they did in the original text (see Proof of Concept Experiment 2). Formally, given tokens in S providing an intra-document sim- ilarity of sim, and tokens the elongated version S providing sim, we will show that sim + sim. This pattern severely presents in models that have been finetuned with a contrastive loss, while is not pronounced in their corresponding vanilla models (PoC Experiment 2, Figure 3).
We verify this phenomenon with Proof of Con- cept Experiment 1 (Figures 1, 2), showing that transformer-basee encoders perceive different se- mantics before and after elongation attacks.
Proof of Concept Experiment To validate Ob- servation 1, we fine-tune a vanilla MiniLM (Wang et al., 2020) with the standard infoNCE loss (Oord et al., 2018) with in-batch negatives, on the Quora duplicate question pair dataset (QQP). Notably, the dataset is composed of questions, and thus its length coverage is limited (average token length 13.9, with 98.5% under 30 tokens).
A significant increase on intra-document simi- larity of contrastive learning-basee models is ob- served by Xiao et al. (2023a), opposite to their vanilla pre-trained checkpoints (Ethayarajh,, 2019). It is further observed that, after contrastive learning, semantic tokens (such as topical words) become dominant in deciding the embedding of a sentence, while embeddings of functional tokens (such as stop- woodwwors) follow wherever these semantic tokens travel in the embedding space. This was formalized as the "entourage effect" (Xiao al., 2023a). Tak- ing into account this conclusion, we further derive from the perspective of attention mechanism, the reason why conducting elongation attacks would further intensify the observed high intra-document similarity.
With the fine-tume tunet model, we first construct two extreme cases: one with a false positive pair ("what is NLP?" v.s." "what is computer vision?"), one with a positive pair ("what is natural language processing?" v.s., "what is computational linguis- tics?"). We compute cosine similarity between mean-pooled embeddings of the original pairs, and between the embeddings attained after conducting an elongation attack with m = 100 (Eq. 1).
We found surprisingly that, while "whar is NLP?" and "what is computer vision?" have 0.06 cosine similarity, their attacked versions achieve 0.42 cosine similarity successfully attacked (cf. Figure 1). And the same between "what is natural language processing?" ?" and "what is computational linguistics?" goes from 0.50 to 0.63 similarity pattern augmented.
The attention that any token a; in the sentence S gives the dominant tokens can be expressed as:
e e Komsmant VaTe
On a larger scale, we then construct an infer- ence set with all the document pairs from Semantic Textual Similarity benchmark (STS-b) (Cer et al., 2017). We conduct clongation attack on all sen- tences with m 100 (Eq. 1). The distributions of document pair cosine similarity are plotted in Fig- ure 2. For the fine-tuned MiniL. (Figure 2, left), it is clearly shown that, the model perceives in gen- eral a higher cosine similarity between documents
Attention I + #dominant) iES
/vdz
(5)
where qi is the query vector produced by Wi. A. dominnntt is the transpose of the key vector pro- duced by Tdominant- and K. the transpose of the key vector produced by every token En. We omit the V matrix in the attention formula for simplicity.

---

After elongating the sentence m times with the copy-and-coneee operation, the attention distribu- tion across tokens shifts, taking into consideration that the default prefix [cls] token is not elongated. Therefore, inference time, [cls] tokens share less attention than in the original sentence.
Gr Attention(
D2 xir) iE$
Attention
3j
-> Fip)
EE
(8)
To simplity the following derivations, we further impose the assumption that positional embeddings contribute little to representations, which loosely hold empirically in the context of contrastive learn- ing (Yuksekgonul et al., 2023). In Section 6, we conduct an extra group of experiment to present the validity of this imposed assumption by showing the positional invariance of models after CL.
Let bominaal Vdx /dx be le, en / vdd be l,. e dv be Lus and e q /da// Von be lo we get:
Gd Attention
I4 iES
ra)
Attentionn
Fd)
ES
With this mind, after clongation, the same to- ken in different positions would get the same atten- tion, because they have the same token embedding without positional embeddings added. Therefore:
Ld
Ld
m
E4e
le
Enn
Σn
n
(c)
m
m
(9)
Similarly, we get:
Attention Ty -> Fddminmntt)
kT me
4 dominant dd
G. Enn 12
(10)
E In
TIL
10)
m Leak? vile (m
12
Also note that 4 > 4; that's why they are called "dominating tokens" in the first place (Xiao et al., 2023a). Therefore, prove that Ga Gr.
eackT Lominnnn /dk
Lends /onkk//Vdd /vdz
m-1 Maaa Vot
As result, with elongation operation, every ken is going assign even more attention the embeddings of the dominating tokens. And this ef- feet propagates throughout layers, intensifying the high intra-document similarity (" ("entourage effect") found in (Xiao et al., 2023a).
Attention 04 Tdominant) iES
Based on Eq. 6, we can see that attentions to- wards dominant tokens would increase after doc- ument elongation attack. However, we can also derive that the same applies to non-dominant to- kens:
Proof of Concept Experiment 2 With the deriva- tions, we conduct PoC Experiment 2, aiming demonstrate that intra-document similarity expe- riences a pattern shift after elongation attack, in- tensifying the "entourage effect", for contrastive fine-tuned models.
Attention x4 + Enon-dominant
ic3
Attention o
+ Tnon-dominant
iES
Taking the same fine-tuned MiniLM checkpoint from PoC Experiment 1, we compute the intra- document similarity of all the model outputs on STS-b. For each document, document embedding by mean-pooling, then com- pute the average cosine similarity between each token embedding and the document embedding."
In fact, every unique token except [cls] would experience attention gain. Therefore, have prove that, the attention gain Ga of dominant tokens (denoted as r.a) outweighs the attention gain G, of non-dominant (regular, denoted as r,)) tokens. To this end, we define:
first compute its
The results are shown in Figure 3. After elongation attacks, we can see an increase in the already high
Go A Attention 2; + xr)) Attention( x4 - xa) iEs
Notably, we further adjust these scores by the model's anisotropy estimation (average pair-wise similarity of random sampled tokens), because of the representation degeneration problem (Gao et al., 2019; Ethayarajh, 2019).
iES
(7)

---

In-document Token Interaction Pattern Shifts
Anisotrepy Pattem Shifts with Length Attacks
Intra-dox Similanty (adjusted)
Sentences
Length Attack 100
Length Attack 100
Anisocropy Estimation
C2)
CaR
64.
MiniLM finetuned on QQP Models
MiniLM vanilla
MiniLM finetuned on QQP Models
MiniLM vanilla
Figure 3: In-document Token Interactions experience pattern shift before and after contrastive fine-tuning: Using the vanilla model, tokens in the elongated version of a document become less like one another than the original un-attacked text; after contrastive fine-tuning, tokens the attacked text look more alike to one another. This empirically validates math derivation. Notably, measurements of both models have been adjusted by their anisotropy estimation (displayed value avg. intra similarity estimated anisotropy value).
Figure 4: Isotropy Pattern Shifts. Albeit contrastive learning has isotropy promise, question this by showing the model is only isotropic in its trained length range, remaining anisotropic otherwise (shown in- creased anisotropy after length attacks).
embeddings produced by the fine-tuned checkpoint start becoming anisotropic (Figure 4, left, pink). This indicates that, if a model has only been trained on short documents with contrastive loss, only the short length range is promised with isotropy.
intra-document similarity, meaning that all other tokens converge even further towards the tokens that dominate the document-level semantics.
On the other hand, elongation attacks seem be able to help vanilla pre-trained models to cape from anisotropy, interestingly (Figure 4, right, pink). However, the latter is not the key focus of this work.
When using the vanilla MiniLM checkpoint, the intra-document similarity pattern is again reversed. This opposite pattern is well-aligned with the find- ings of Ethayarajh (2019) and Xiao et al. (2023a): Because in vanilla language models, the intra- document similarity generally becomes lower in the last few layers, while after contrastive learning, models show a drastic increase of intra-document similarity in the last few layers. Also, our deriva- tions conclude that: the intra-document similarity shows an accumulated increase the last few lay- ers, this increase will be intensified after elongation; and less affected otherwise.
Method: LA(SER)3
After examining the two fundamental reasons un- derlying the built-in vulnerability brought by stan- dard contrastive learning, the formulation of our method emerges as an intuitive outcome. Nat- urally, we explore the possibility of using only length as the semantic signal to conduct con- trastive sentence representation learning, and pro- pose LA(SER) Length-Agnosti Self-Reference for Semantically Robust Sentence Representation Learning. LA(SER)S builds upon the semantics- preserved assumption that "the elongated version of myself 1) should still mean myself, and thus 2) should not become more or less similar to my pairs". LA(SER) leverages elongation augmenta- tion during the unsupervised constrastive learning improve 1) the robustness of in-document inter- action pattern in inference time; 2) the isotropy of larger length range. We propose two versions of reference methods, for different format availability of sentences in target training sets.
Complementing the intensified intra-document similarity, we also display an isotropy misalign- ment before and after elongation attacks in Fig- ure 4. With the well-known representation degener- ation or anisotropy problems in vanilla pre-trained models (Figure 4, right, green, Gao et al. (2019); Ethayarajh (2019)), it has been previously shown that after contrastive learning, a model's encoded embeddings will be promised with more isotropic geometry (Figure 4, left, green, Wang and Isola (2020); Gao al. (2021); Xiao et al. (2023a)). However, in this work, we question this general conclusion by showing that the promised isotropy is strongly
After elongation, the

---

Models
SimCSEf
ESimCSEf
DifICSEF
InfoCSEj
LA(SER)* (Ours)
Test Dataset L
base
large
base
large
base
large
base
large
base-64
base-128
trec-covid
0.2750
0.2264
0.2291
0.2829
0.2368
0.2291
0.3937
0.3166
0.2728
0,3463
nfcorpus
0.1048
0.1356
0.1149
0.1483
0.1204
0.1470
0.1358
0.1576
0.1652
0.1919
nq
0.1628
0.1671
0.0935
0.1705
0.1188
0.1556
0.2023
0.1790
0.1556
0.1354
fiqa
0.0985
0.0975
0.0731
0.1117
0.0924
0.1027
0.0991
0.1000
0.10
0.1090
arguana
0.2796
0.2078
0.3376
0.2604
0.2500
0.2572
0.3244
0.4133
0.4182
0.4227
zero-shot setting
webis-touche2020
0.1342
0.0878
0.0786
0.1057
0.0912
0.0781
0.0935
0.0920
0.1105
0.1
quora
0.7375
0.7511
0.7411
0.7615
0.7491
0.7471
0.8241
0.8268
0,7859
0.7741
cqadupstack
0.1349
0.1082
0.1276
0,1196
0.1197
0.1160
0.2097
0,1881
0,1687
0.1691
dbpedia-entity
0,1662
0.1495
0.1260
0.1650
0.1537
0.1571
0.2101
0,1838
0,1645
0.1663
scidocs
0.0611
0.0688
0.0657
0.0796
0.0673
0.0699
0.0837
0.0859
0.0764
0.0859
climate-fever
0.1420
0.1065
0.0796
0.1302
0.1019
0.1087
0.0937
0.0840
0.1311
0.1
scifact
0.2492
0.2541
0.3013
0,2875
0.2666
0.2811
0.3269
0.3801
0.3960
0,4317
hotpotqa
0.2382
0.1896
0.1213
0.1970
0.1730
0.2068
0.3177
0,2781
0.2827
0.2937
fever
0.2916
0.1776
0.0756
0.1689
0.1416
0.1849
0.1978
0.1252
0,2388
0.2691
average
0.2197
0.1948
0.1832
0.2135
0.1916
0.2030
0.2509
0.2436
0.2480
0.2594
Table I: Unsupervised BERT nDCG@ 10 performances on BEIR information retrieval benchmark. Results are from Wu et al. (2022a). A Unfair comparison. Notably, InfoCSE benefits from the pre-training of an auxiliary network, while the rest of the baselines and our method fully rely on unsupervised contrastive fine-tuning on the same training*iht setting described in $4. Note that with a batch size of 64, method already outperforms all baselines to a large margin except InfoCSE. Since we train with a max sequence length of 256 (all baselines are either 32 or 64), we find that training with a larger batch size (128) further stabilizes our training, achieving state-of- of-the-art results. Further, achieve state-of-the-an with only BERT base.
Self-reference In LA(SER)3 seli-ref setting, we take a sentence from the input as an anchor for each training input, and construct its positive pair by elongating the sentence to be m times longer.
(Nguyen et al., 2016), which is equivalent to in- domain-only setting of the BEIR information re- trieval benchmark (Thakur et al., 2021).
Evaluation datasets The trained models are mainly evaluated on the BEIR benchmark (Thakur et al., 2021), which comprises 18 datasets on tasks (fact checking, duplicate question re- trieval, argument retrieval, news retrieval, question- answering, tweet retrieval, bio-medical retrieval and entity retrieval). We evaluate on the 14 public zero-shot datasets from BEIR (BEIR- 14). And we use STS-b (Cer et al., 2017) only as the auxiliary experiment.
Intra-reference LA(SER) S intra-ref conducts intra- reference within the document. The two compo- nents of a positive pair are construeted from dif- ferent spans of the same document. Since we are only to validate effectiveness of LA(SER) intra-ref- we implement this in the simple mutually-excluded span setting. In other words, the LA(SER) intra-eff variant takes a sentence (either the first or ran- dom sentence) from the text as anchor, uses the rest of the text in the input as its positive pair, and elongates the anchor sentence m times as the augmented anchor.
-4)R.-14),
The reasons why we do not follow the de facto practice, which mainly focuses cherry-pickin the best training setting that provides optimal per- formance on STS-b are as follows: Firstly, perfor- mances on STS-b do not display strong correlations with downstream tasks (Reimers et al., 2016). fact, document-level encoders that provide strong representational abilities do not necessarily provide strong performance on STS-b (Wang et al., 2021). Furthermore, recent works have already attributed the inferior predictive power of STS-b performance on downstream task performances its narrow length range coverage (Abe et al., 2022). There- fore, we believe a strong sentence and document- level representation encoder should be evaluated
For both versions, we use the standard infoNCE loss (Oord et al., 2018) with in-batch negatives as the contrastive loss.
4 Experiments
Training datasets We conduct our experiments on two training dataset settings: 1) training"iks uses 1M sentences sampled from Wikipedia, in line with previous works on contrastive sentence rep- resentation learning (Gao et al., 2021; Wu et al., b); 2) training msmarco uses MSMARCO

---

Truined on wiki Self-reference
Trained on MSMARCO Self-reference
Trained on MSMARCO
Training Setting- >
Intra-reference
Perf. Gain
Perf.
COCO-DR (PT-unsup)
Perf.
Models -
Test dataset 1
SimCSE
LA(SER)"
SimCSE
LAISER)*
Gain
LA(SER)"
Gain
treesoovid
0,1473
0.2129
44.52%
0.1467
0.1646
12.22%
0.2597
0:2511
-3.3%
nncorpus
0.0764
0.1265
65.54% %
0.0796
0.0933
17.31%
0. 1853
0.1508
-18.62%
nq
0.0370
0.0836
125.88
0.0302
0.0391
29.55%
0.0268
0.0405
51.10%
fiqa
0,0288
0.0590
104.94% %
0.0260
0.0435
67.36%
0,0821
0.1030
25.48%
arguana
0.2277
0.3130
37.48%
0.2081
-5.74%
0.3834
11.42%
Zerr sooooothot setting
webis-touche2020
0,0289
0.0483
66.99%
0.0177
0,0296
67.71%
0.0736
0.0896
21.73%
quorn
0.6743
0.7095
5.22%
0.6527
0,6515
%
0.7976
0.7911
-0.82%
cqadupstack
0,0889
0.1279
43.90 %
0.0864
0.1105
27.95%
0.1380
0.1560
13.06%
abpedia-entity
0.0837
0.1138
36,04 %
0.0541
0.0558
3,03% %
0.0924
0.0825
-10.76%
scidocs
0.0259
0.0516
99.54% %
0.0178
0.0309
73.19%
0.0305
0.0492
61.56%
climate-fever
0.0127
0.0789
522.24%
0.0136
0.0198
45.11% %
0.0652
0.1108
69.84%
scifact
0.2174
0.3525
62.12%
0.2330
0,2276
-2.32%
0,4056
0.4076
0.49%
hotpotqa
0.0829
0.1646
98,56
0.0560
0.0750
34.07%
0.0383
0.0539
40.85
fever
0,0363
0.1001
175.88
0.0263
0,0340
29.41%
0.1421
0.2524
77.60%
average
0.1263
0.1816
43.78%
0.1177
0.1265
7.48%
0.1915
0.2087
8.97%
Table 2: Unsupervised Performance Trained with MiniLM-L6 Model. For self-reference settings, we compare with SimCSE (Gao et al., 2021). Notably, LA(SER)* self-rer can be viewed as plug and--aad module to SimCSE. as SimCSE takes an input itself as both the anchor and the positive pair, while LA(SER) ) self further elongates this positive pair. For mntra-referede setting, we compare with COCO-DA MRCO-DR&C (Yu et al., 2022). Notabaly, we only experiment with the unsupervised pre-training part of COCO-DR, as LA(SER) intra-ret can be viewed as plug-and- play module to this part. We believe combining with our method for a better unsupervised pretrained checkpoint, the follow-up supervised fine-tuning in COCO-DR can further achieve better results.
beyond semantic textual similarity tasks. However, for completeness, we also provide the results of STS-b in Appendix A.
et al., 2019) and M (Wang et al., 2020). To compare to previous works, we first train a ( self-ref on trainingMiki with BERT-base (uncased). We then conduct most of our in-depth experiments with vanilla MiniLM-L6 due to its low computational cost and established ofatc-of-the-art potential after contrastive tineituning."
Baselines We compare our methods in two settings. corresponding to the two versions of LA(SER)3: 1) Self-Reference Since we assume using the input itself as its positive pair in this set- ting, it is natural to compare LA(SER)3 self-ref the strong baseline SimCSE (Gao et al., 2021). In the trainingmiki setting, we further compare with E- SimCSE, DifICSE, and InfoCSE (Wu et al., 2022b; Chuang et al., 2022; Wu et al., 2022a). Notably, these four baselines all have public available check- points trained on training*iki
All experiments are run with 1 epoch, a learn- ing rate of 3c-5, temperature T of 0.05, max sequence length of 256, and batch size of 64 un- less stated otherwise. All experiments are run on Nvidia A 100 80G GPUs
Notably, previous works on contrastive sentence representation learning (Gao et al., 2021; Wu et al., 2022b; Chuang et al., 2022: Wu et al., 2022a) and some information retrieval works such as (Yu et al., 2022) mostly use max sequence length of 32 to 128. In order to study the effect of length, we set the max sequence length to 256, at the cost of constrained batch sizes and a bit of computational overhead. More detailed analyses on max sequence length are in ablation analysis ($5).
2) Intra-Reference. The baseline method in this case is: taking a sentence (random or first) from a document as anchor, then use the remaining content of the document as its positive pair. Notably, this baseline is similar to the unsupervised pretraining part of COCO-DR (Yu et al., 2022), except COCO- DR only takes two sentences from same docu- ment, instead of one sentence and the remaining part. Compared to the baseline, LA(SER)3 intra-rel further elongates the anchor sentence. In the re- sult table, we refer to baseline of this settings as COCO-DRpransup-
For the selection of the anchor sentence, we take the first sentence of each document in the main experiment (we will discuss taking a random sen-
"We use a 6-layer version by taking every second layer. www.g/nwwww..ooerreeeeee/aaaaaaa For instance, sentence- traaeeeeeeanntrrrmrllllllll- Mini----- M--1--v2 is SOTA sentence encoder finc-tuned with Min6L
Implementation Details We evaluate the ef- fectiveness of our method with BERT (Devlin

---

tence instead of the first sentence in the ablation analysis in $5.1). For LA(SER)) self-ref- we elongate the anchor sentence to serve as its positive pair; for LA(SER)* intra-ref we take the rest the document as its positive pair, but then elongate the anchor sen- tence the augmented anchor. For the selection of the elongation hyperparameter m, we sample random number for every input depending its length and the max length of 256. For instance, if a sentence has 10 tokens excluding [cls], we sample a random integer from [1,25], making sure it is not exceeding maximum length; while for a 50-token sentence, we sample from [1, 5]. We will discuss the effect of elongating to twice longer, instead of random-times longer in ablation $5.2.
ument pairs (see popular Sentence Transformers checkpoint all-MiniLN-LG-v2) its performance largely falls short under unsupervised training set- tings, which we speculate to be due to that the linguistic knowledge has been more unstable after every second layer of the model is taken (from 12 layers in MiniLM-L12 to 6 layers). Under such setting, LA(SER) intra-ref largely outperforms LA(SER) self-ref, by providing signals of more lex- ical differences in document pairs.
Ablation Analysis
In this section, we ablate two important configura- tions of LA(SER)3 Firstly, the usage of LA(SER)² involves deciding which sentence in the document to use as the anchor ($ 5.1). Secondly, how do we maximize the utility of self-referential elongation? Is it more important for the model to know "me" * m = me", or is it more important to cover a wider length range ($ 5.2)?
Results The main results are in Tables 1 and 2. Table shows that our method leads to state-of- the-art average results compared to previous public available methods and checkpoints, when training on the same training"iki with BERT.
Our method has the exact same setting (training a vanilla BERT on the same training*iki with the rest of the baselines except InfoCSE, which further benefits from the training of an auxiliary network. Note that with batch size of 64, our method already outperforms all the baselines to large margin except InfoCSE. Since we train with a max sequence length of 256 (all baselines are either 32 or 64), we find that training with a larger batch size (128) further stabilizes our training, achieving wate-of-the-an results. Moreover, we achieve state- of-the-art with only BERT hase-
5.1 Selecting the Anchor: first or random?
If a document consists of more than one sentence, LA(SER)3 requires deciding which sentence in the document
the anchor. We ablate this with both 3 self-ref and LA(SER) intra-rel on training"smaree because training wiki consists of mostly one-sentence inputs and thus is not able to do intra-ref or random sentence.
Anchor Sentence
Zero-shot Average
Performance Change
Method
In general, we find that our performance gain is more pronounced when the length range of the dataset is large. On BERT-base experiments, large n 10 performance gain is seen on NFCor- pus (doc. avg. length 232.26, SimCSE: 0.1048 >> L 0. 1919) Scifact (doc. avg. length 213.63, SimCSE: 0.2492 >> LA(SER)*: 0.4317), Arguana (doc. avg. length 166.80, SimCSE: 0.2796 LA(SER)3: 0.4227). On the other hand, our performance gain is limited when documents are shorter, such DBPedia (avg. length 49.68) and Quora (avg. length 11.44).
SimCSE
0.1177
First
17.48%
LA(SER) self-ref self-red
0.1265
SimCSE
0.1127
Random
110.05%
LA(SER) sdlfeef
0,1013
COCO- DR pi-ursup
0.1915
First
8.97%
LA(SER)
0.2087
COCO-DR
pi-unsup 0.1930
Random
5.33%
LA(SER) intra-eef
0.2033
Table 3: Taking First sentence Random sen- tence the anchor? ablated with MiniLM-L6 on trainingmmerco
Table 2 further analyzes the effect of datasets and LA(SER)3 variants with MiniLM-L6, showing a consistent improvement when used as plug-and- play module to previous SOTA methods.
The results are in Table 3. In general, we observe that taking random sentence anchor brings certain noise. This is most corroborated by the performance drop of LA(SER) self-ref random sentence, compared to its SimCSE baseline. How- ever, LA(SER)* intru-sim + random sentence seems to be able to act robustly against this noise.
We also found that, even though MiniLM-L6 shows great representational power if after super- vised contrastive learning with high-quality doc-

---

We hypothesize that as LA(SER)² provides aug- mented semantic signals contrastive learning, would be hurt by overly noisy in-batch inputs. By contrast, LA(SER)* intra-sim behaves robustly to this noise because the rest of the document apart from the anchor could serve as a stabilizer to the noise.
after contrastive learning on Sentence Lmmbed- ding Training Data"). Models after contrastive learning are Sentence Transformers (Reimers and Gurevych, 2019) models all 1an----me--as--eee and all-MiniLM-L12-v2.
We take the sentence pairs from STS-b test set as the inference set, and compute each model's perceived cosine similarity on the sentence pairs (distribution 1). We then randomly shuffle the word orders of all sentence Is in the sentence pairs, and compute each model's perceived cosine similarity with sentence 2s again (distribution 2).
5.2 Importance Seff-referential Elongation With the validated performance gain produced by the framework, we decompose the inner-workings by looking at the most important component, clon- gation. A natural question is: is the performance gain only brought by coverage of larger trained length range? Or does it mostly rely on the seman- tic signal that, "my-longer-sell still means myself?
The divergence of the two distributions for each model can serve proxy indicator of the model's sensitivity towards word order, and thus towards positional shift. The lower the divergence, the more insensitive that a model is about positions.
Elongation Mode
Max Seq. Length
Zero-shot Average
None
256
0.1263
We find that the Jenson Shannon divergence yielded by MiniLM has gone from 0.766 (vanilla) to 0.258 (after contrastive learning). And the same for mpnet goes from 0.819 (vanilla) to 0.302 (af- ter contrastive learning). This finding shows that contrastive learning has largely removed the con- tribution of positions towards document embed- dings, even in the most extreme case (with random shuffled word orders). This has made contrastively- learned models acting more like bag-of-words mod- els, aligning with what was previously found in vision- lisionnlanguage models (Yuksekgonul et al., 2023). Moreover, MiniLM uses absolute positional em- beddings while mpnet further applies relative po- sitional embeddings. We believe that the posi- tional insensitivity pattern holds for both models can partly make the pattern and LA(SER)* Ss util- ity more universal, especially when document en- coders are trained with backbone models that have different positional encoding methods.
Twice
256
0,1523
64
0.1778
Rundom
128
0.1764
256
0.1816
Table 4: 1) Elongdating to fixed-times longer or ran- dom time? 2) Do length range coverage matter? ab- lated with M-Li on trainingriki,
Table shows that, elongating to random-times longer outperforms elongating to a fixed two-times longer. We hypothesize that, fixed augmentation introduces certain overfitting, preventing the mod- els to extrapolate the semantic signal that "elon- gated me me". On the other hand, as long as they learn extrapolate this signal (by random times), increasing max sequence length provides decreasing marginal benefits.
6 Auxiliary Property Analysis
6.1 Positional Invariance
Recalling in Observation 2 and PoC experiment 2, we focused on analyzing the effect of elonga- tion attack on intra-sentence similarity, which is already high after CL (Xiao et al.. 2023a). There- fore, we have imposed the absence of positional embeddings with the aim to simplify the derivation in proving that, with elongation, dominant tokens receive higher attention gains than regular tokens. Here, present the validity of this assumption by showing models' greatly reduced sensitivity to- wards positions after contrastive learning.
Conclusion
In this work, we questioned the length generaliz- ability of contrastive learning- based text encoders. We observed that, despite their seemingly strong representational power, this ability is strongly vul- nerable to length-induced semantic shifts. we for- malized length attack, demystified it, and defended against it with LA(SER)3. We found that, teach- ing the models "my longer-self = myself" provides standalone semantic signal for more robust and powerful unsupervised representation learning.
We analyze the positional (in)sensitivity of models (MiniLM (Wang et al., 2020) and mp- net (Song et al., 2020) respectively before and
hittps:/hhuggingfaaee ct/atasets/// sentence
wrapiiinnn-rrriinn---eee data

---

Limitations
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
We position that the focus of our work lies more in analyzing theoretical properties and inner- workings, and thus mostly focus on unsupervised contrastive learning settings due to compute con- straints. However, we believe that with a better unsupervised checkpoint, further supervised fine- tuning will yield better results with robust patterns, We leave this line of exploration for future work. Further, we only focus on bi-encoder settings. In information retrieval, there are other methods in- volving using cross-encoders to conduct re-ram re-ranking, and sparse retrieval methods. Though we envision our method can be used as plug-and -la- mod- ule many of these methods, it is hard exhaust testing with every method. We thus experiment the and--and setting with a few representa- tive methods. We hope that future works could evaluate the effectiveness of our method combin- ing with other lines of baseline methods such as cross-encoder re-ranking methods.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP- (NNLP)) pages 55-65, Hong Kong, China, Association for Computational Linguistics.
Jun Gao, Di He. Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration prob- lem training natural language generation models. In International Conference on Learning Representa- tions.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings the 2021 Conference Empirical Methods in Natural Language Processing. pages 6894-6910.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738.
References
Kaori Abe, Sho Yokoi, Tomoyuki Kajiwara, and Ken- taro Inui. 2022. Why is sentence similarity bench- mark not predictive of application-oriinte formance? In Proceedings of the 3rd Workshop on Evaluation and Comparison of NI.P Systems, pages 70-87.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNLPP), pages 9119 9130.
task per-
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
Tri Nguyen, Mir Rosenberg. Xia Song, Jianfeng Gao. Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
and
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arxiv preprint arxiv 0107100770755
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. 2020. simple framework for contrastive learning of visual representations. In In- ternational conference machine learning, pages 1597-1607, PMLR.
Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similarity. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- lics: Technical Papers, pages 87-96.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang- Wen Li, Wen-Tau Yih, Yoon Kim, and James Glass. 2022. Diffese: Difference-baase based contrastive learning for sentence embeddings. In Annual Conference of the North American Chapter of the Association for Computational Linguistics.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLL
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
pages
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: masked and permuted pre-training for language understanding. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 16857-16867.

---

Karen Sparek Jones. 1972. statistical interpretation of term specificity and its application retrieval. Journal of documenatiin,, 28(1):11-21.
Chenghao Xiao, Zihuiwen Ye, G Thomas Hud- son, Zhongtian Sun, Phil Blunsom, and Noura Al Moubayed. 2023b. Can text encoders be deceived by length attack?
Hongjin Su, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. 2022. One embedder, any task: instruction-fineture text embeddings. arXiv preprint arXfv:2212.09741.
Yue Yu. Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. Coco-dr: Combating dis- tribution shifts in zero-shot dense retrieval with con- trastive and distributionally robust learning.
Mert Yuksekgonul. Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2023. When and why vision-language models behave like bags-of- words, and what do about it? In International Conference on Learning Representations.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for bet- ter semantics and faster retrieval. arXiv preprint
arXiv:2103.13116
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogeneous benchmark for zeoo evaluation of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Kexin Wang. Nils Reimers, and Iryna Gurevych. 2021. Tsdae: Using transformer-asedd sequential denoising auto-encoderfor unsupervised sentence embedding learning. In Findings of the Association for Compu- lational Linguistics: EMNLP 2021, pages 671 671-688.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference Machine Learning, pages 9929-9939.
PMLR.
Wenhui Wang, Furu Wei, Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression pre-trained transformers. Advances in Neural In- formation Processing Systems,
Xing Wu. Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022a. InfoCSE: aagrraatoon-aggeggatee contrastive learning of sen- tence embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3060-3070, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022b. ESim- CSE: Enhanced sample building method con- trastive learning of unsupervised sentence embedding. In Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, Republic of Korea. International Committce on Computational Linguistics.
Chenghao Xiao, Yang Long, and Noura Al Moubayed. 2023a. On isotropy, contextualization and learning dynamics of contrastive-hased sentence representa- tion learning. In Findings of the Association for Computational Linguisties: ACL 2023, pages 12266- 12283.

---

A Results of STS-b
this section, we present the results of STS-b test set (Table 5). As discussed in the main sec- tions, we position that STS-b is not correlated with downstream semantic tasks performance (Reimers et al., 2016; Wang et al., 2021), and effectiveness of document-level representation encoders should be evaluated beyond this task. The inferior predictabil- ity of STS-B on downstream task performances have been attributed to length ranges (Abe et al., 2022). We hypothesize that, training with a large max sequence length increases the uncertainty of elongation hyperparameter m of LA(SER)", result- ing in diverse length range, and less correspond- ing concrete examples at each length.
We show that, while out-performing SimCSE by a large margin on other downstream semantic tasks (Main Section, Table 1), our long sequence length poses a certain level of instability in converg- ing, showing a small performance drop on shorter sentences (STS-b). The converging instability is further confirmed by training an extra LA(SER)a 3 with [cls]-pooling, [cls]-pooling is faster in converging- as it involves only optimizing one token. Notably, SimCSE also uses [clsl-pooling Therefore, we roughly stay on-par with SimCSE on encoding shorter documents, while out-performing it by a large margin on other downstream tasks.
Method
Max Seq.
BERT-whitening
68. 19/71.433
BERT-flow
64
58.56/70.72
SimCSE
32
76.85
LA(SER)*-ment
256
75.61
LA(SER\--Ccls]
256
76.19
Table 5: STS-b test set results, compared with unsu- pervised sentence representation methods. SimCSE and LA(SER) are trained on the same training"lk; The two numbers of BERT-whitening and BERT-flow correspond optimizing NLI or target data (sts-b). Results are from the original works (Su et al., 2021; Li al., 2020; Gao et al., 2021).


Reviews:
[{"review_id": "snDaaz2XfC", "reviewer": "Reviewer_omiY", "summary": "", "questions": "The authors must discuss on how they handle the context length issue in encoders for very large elongated sequences. Do they employ linear attention mechanisms like reformer ? If not can authors report results using these encoders as they are a fairer evaluation for elongated sequences without truncation imposed to fit in memory.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments!\n\nP1 (Positional invariance): To address this concern, we have now conducted an extra group of experiments to present the sensitivity change of language models towards positions before and after contrastive learning.\nWe take 4 models (MiniLM and mpnet respectively before and after contrastive learning on [1]). We take sentence pairs from sts-b, compute their perceived cosine similarity (distribution 1), and then randomly shuffle the word orders of all sentence 1s in the sentence pairs, and compute their cosine similarity with sentence 2s again (distribution 2), and measure the divergence of these 2 distributions with Jenson Shannon Divergence.\nWe find that the JS divergence yielded by MiniLM has gone from 0.766 (vanilla) to 0.258 (after contrastive learning). And the same for Mpnet goes from 0.819 (vanilla) to 0.302 (after contrastive learning). Even though 0.258 and 0.302 are still not too close to 0, we see that the distributions almost fully overlap with each other in cases for both models  after contrastive learning. This finding has shown that contrastive learning has largely removed the contribution/sensitivity of positions, even in the most extreme case (with random shuffled word orders). This has made contrastively-learned models acting more like bag-of-words models, aligning with what was previously found in vision-language models as well [2].\nMoreover, MiniLM uses absolute positional embeddings while mpnet applies further relative positional embeddings. We believe that the positional insensitivity pattern holds for both models can also partly contribute to solving the reviewer’s concern on experimenting with all kinds of encoders (with different positional embedding methods).\n\nP2 (Catastrophic forgetting and overfitting): We partly discuss this issue in ablation analysis through the lens of elongation times m. We find that using a fixed-times elongation (say 2 times) might induce this kind of pattern overfitting, and probably does not teach the model to extrapolate meanings in unseen length ranges, but only teaches the model that 1=2. And using a random-times elongation is more extrapolative, as it’s essentially teaching the model to know that “Thanks for your valuable comments” means the same thing in position [1,5] just as what it means in position [5m+1,5m+5], for instance. We envision that the randomness in augmentation is robustness-preserving. Moreover, as the reviewer suggested, we aim to include some linguistic characteristics tests that could be done via probing the encoded embeddings in the final version given the chance.\n\nP3 (Discussions on results where margin of gain is low): In general, we find that our performance gain is more pronounced when the length range of the dataset is large. For instance, on BERT-base experiments, nfcorpus (doc. avg. length 232.26, nDCG@10 performance 0.1048 (SimCSE) -> 0.1919 (Ours)), scifact (doc. avg. length 213.63, nDCG@10 performance 0.2492 (SimCSE) -> 0.4317 (Ours)), Arguana (doc. avg. length 166.80, performance 0.2796 (SimCSE) -> 0.4227 (Ours)). On the other hand, our performance gain is low when documents are shorter, such DBPedia (avg. length 49.68) and Quora (avg. length 11.44). Results on MiniLM-L6 are a bit more nuanced, which we think is because the linguistic knowledge has been more unstable after every second layer of the model is taken (from 12 layers in MiniLM-L12 to 6 layers). We believe the ability to extrapolate meanings for longer text is beneficial in use cases where the domain and task-specific training set has limited length range but the model needs to face unexpected text range in inference time.\n\nP4 (Handle Context Length Issue for very large elongated sequences): We explain how we sample input-dependent elongation times in the paper (line 484 to line 490). In short, we do not make the elongated sequences too long. If our max sequence length is 256 in training, we randomly sample an elongation times from 1-25 times if an input is 10-token long excluding [cls], and 1-5 times if it’s 50-token long. This means that we do not elongate sequences that are originally longer than 128. This has ensured a diversity for our elongation times m, and teaches the model to extrapolate even to a longer length (with the learned pattern of 1= m). With this extrapolation, we do not need to train the models explicitly on say 1000-token inputs but the model will still be better than the baselines on dealing with 1000-token inputs. \nTherefore, we speculate that training with linear attention models that might be more beneficial when explicitly trained with longer texts, might not be as rewarding as the extrapolative ability itself. In line with our findings on BERT, we conjecture that it takes a larger batch size for LA(SER)$^3$ to show its advantages, when training on longer sequence lengths with linear attention models, and it’ll potentially be more beneficial to incorporate gradient caching for larger batch sizes, when training them. Due to compute constraints, we struggle to finish this part of experiments during rebuttal and will release the results in the final version for completeness.\n\n[1] https://huggingface.co/datasets/sentence-transformers/embedding-training-data\n\n[2] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D. and Zou, J., 2022, September. When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?. In The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper studies an important problem of length bias in dense retrievers. The proposed approach to tackle the same is intuitive and simple.\n\n2. the effect of length attacks exemplified by contrastive learning is explained well along with supporting experiments. The insight that the token interaction also change in a contrastive framework is interesting. the study of effects of length interplay with contrastive training on the geometry of the embedding space is interesting too.\n\n3. The experimental results seem convincing though the proposed approach does not give uniform gains on all datasets, overall it seems to perform well compared to existing approaches and relatively more robust to length based attacks.", "reasons_to_reject": "1. The proposed augmentation approaches based only on elongation are not convincing as depending on training sample size the model may overfit to this particular augmentation resulting in catastrophic forgetting. i believe a detailed study on whether the model fine tuned with LASER forgets other semantic characteristics learned is necessary. Also to evaluate robustness  different encoders must be evaluated.\n\n2. I'm not entirely convinced of the positional invariance argument. While set based applications require position invariance and self attention is by definition invariant unless added with positional embeddings other retrieval applications are position dependent. For example extractive summarization depends on position as terms appearing earlier in document are more relevant. And attention coupled with positional embeddings help surface this.  Authors must spend more effort in empirically showing this argument before making the assumption of positional embeddings having little impact.\n\n3. More detailed discussion of results would be appreciated. Also statistical significance tests would help determine the superiority of proposed approach in cases where the margin of gain is low.", "questions_to_authors": "The authors must discuss on how they handle the context length issue in encoders for very large elongated sequences. Do they employ linear attention mechanisms like reformer ? If not can authors report results using these encoders as they are a fairer evaluation for elongated sequences without truncation imposed to fit in memory.", "paper_topic_and_main_contributions": "The paper tackles an interesting and important problem of length bias in contrastive learning methods. The paper demonstrates interesting insight such as the effect of length based attacks on document similarity, further exemplified by contrastive learning. The work also shows that length also affects the geometry of the embedding space as the isotropy guarantees are not length agnostic. The authors also empirically demonstrate this and propose a length agnostic approach to increase robustness to length attacks through two augmentation approaches. The results in standard benchmarks seem encouraging in most cases.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0xuHRioMFG", "reviewer": "Reviewer_oY3S", "summary": "", "questions": "Please address the aforementioned weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback.\n\nW1 (The “What is” Word Overlap/Repetition): Our answers to this question are three-fold: \n1) Exactly, there exists a “what if” as a word overlap in both sentences in the example. However, given a robust semantic encoder, we would expect conducting the copy elongation wouldn’t strengthen this “semantic alignment” that much (from 0.06 to 0.42). Using tf-idf on the other hand, will make the copy-concat elongation provide the same similarity as before.  In real life, if we search “what is NLP”, we wouldn’t want the search engine to return an article about teaching the grammar of “what is”, for instance, where there will probably be more “what is” in it.\n\n2) Apart from this one example, we also conducted the same experiment on the corpus level (as shown in Figure 2 left), where there are a lot of sentence-pair examples that do not necessarily contain word overlapping, and the same trend holds.\n\n3) From a dominant token perspective, “what is” is likely given very little attention after contrastive learning already (as we explained in Observation 2). Therefore, it shouldn’t be dominantly augmenting the semantics even if we repeat it. Thus, the augmented similarity in the example largely comes from repeating NLP and CV - the dominant tokens (they might contain some same properties in some embedding dimensions that make the cosine similarity much more alike after augmenting).\nHope this discussion addresses the reviewer’s concern!\n\nW2 (Same loss): We use the standard InfoNCE loss as the contrastive loss, aligning with most papers in this research line (SimCSE, E-SimCSE, DiffCSE, InfoCSE, COCO-DR and more recent ones) and we do not benefit from an auxiliary network as in InfoCSE but still outperforms it in most IR tasks. We are one of the very few to systematically investigate the length generalization problems, and have combined the elongation training with our newly designed “intra-reference” method (Table 2). More importantly, it can be used as a plug-and-play module to any method that uses a contrastive loss.\n\nW3 (Evaluation): We would like to position that IR tasks provide the most effective way to test actually using the “frozen document embedding” to do inference only with similarity computation (Also importantly, IR provides a suite of tasks consisting of varied-length inputs). NLU tasks, on the other hand, require fully fine-tuning again and adding classification heads, which we position as not a “representation learning” problem. Were to solve NLU tasks, it would typically make more sense to finetune a vanilla model, instead of a model trained to encode representation. We also presented our sts-b results in Appendix A for completeness and also discussed why it is not good.\n\nW4 (Solve Length Vulnerability): Thanks for pointing this out! We will add a section to include more distribution plots in the camera-ready version given the chance. In short, yes, after our method, the misalignment in Figure 2 almost disappeared. Statistically, in an unsupervised setting trained on wiki with BERT-base-uncased, the Jenson Shannon Distance (JSD) between “Attacked v.s. Unattacked” goes from 0.39 (SimCSE) to 0.17 (LA(SER)$^3$, ours). In a supervised setting as in Figure 2, this number goes from around 0.6 to under 0.1 across all dataset settings we tried (e.g., QQP as in Figure 2, and MSMARCO, etc.). To compute the above values, we compute the cosine similarity distribution between unattacked sts-b sentence pairs (distribution 1), then compute the cosine similarity distribution  between original sentence 1s and sentence 2s attacked with random elongation (distribution 2), and compute the JS divergence between these 2 distributions. A lower number means that the semantics stay more stable with elongation attacks. \n\nW5 (elongation time m): The 100 times the reviewer saw is from the proof-of-concept experiment. In training, we explain the selection of m from line 484 to line 490, and also in ablation analysis. We sample an input-dependent m. If an input is 50-token long, we sample m from [1,5] to prevent it from exceeding 256. If it’s 10-token long, we sample m from [1,25], making sure it’s shorter than 256 tokens.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1: The authors introduce examples of length preference in dense retrieval models and assume that contrastive learning based models are also vulnerable to length. This is assumed to be affecting the actual model performance, and two experiments were designed to prove it. This motivation is one of the strengths of this paper.\n\nS2: The two experiments to demonstrate the length vulnerability are reasonably designed and I believe the results are actually sufficient to prove the length vulnerability issue of existing contrastive learning based models.\n\nS3: The self-reference and intra-reference ideas to overcome the length vulnerability problem are effective.", "reasons_to_reject": "W1: Motivation may come from sentences that are not actually used very often in real-life. Specifically, it is unlikely that the elongated sentences are commonly used in infromation retrieval or any other downstream task.  It is possible that the high similarity between the two elongated sentences is due to simple repetition. In the example in Figure 1, the (What is) token in the sentences S_a x 100 and S_b x100 is repeated, which may have led to the high similarity. In Figure 2, if S_a and S_b are subjected to the same m, it seems that the high similarity may be due to the many repetitions of the same token. I would like to hear the authors' opinions on this perspective.\n\nW2: The proposed method utilizes the same loss as InfoCSE, and the only difference with InfoCSE is that it uses elongated sentences as positive pairs.\n\nW3: As the authors mention as a limitation, they only experimented with BEIR. In Table 1, we can see that the experimental results are taken from InfoCSE's paper, and InfoCSE evaluated not only the IR task but also the NLU (GLUE Benchmark) performance. If there is an improvement in IR performance, but the performance in other tasks such as NLU decreases, it is difficult to see that the proposed method is meaningful. Therefore, experiments on various tasks are strongly required.\n\nW4: Does the proposed method actually solve the length vulnerability problem? For example, it would be nice to visualize the distribution obtained by the model applied with the proposed solution, just like Figure 2.\n\nW5: The authors set the default repeat size (m) for elongated sentences to 100, which seems too large to cover with a max sequence length of 256. How did you control this issue?", "questions_to_authors": "Please address the aforementioned weaknesses.", "paper_topic_and_main_contributions": "This paper studies the impact of sentence length on model performance. The authors find that in contrastive learning-based models, the meaning of the same sentence changes depending on the length of the sentence, suggesting that the existing contrastive frameworks suffer from the length vulnerability. The authors empirically found that models tend to rely on semantic signals from sentence length and that the isotropy of sentence embeddings is highly dependent on the length of the sentence used for training. Based on these two observations, the authors show that current contrastive frameworks are vulnerable to length and propose a length-agnostic contrastive learning framework to address this issue.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XNYgG2WyLG", "reviewer": "Reviewer_wNeu", "summary": "", "questions": "Please address the concerns in the \"Reasons to Reject\"", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable concerns.\n\nW1 (Evaluation): Our answers to this question is three-fold: \n1) As the main focus of our work on better dense retrievers, we would like to advocate the standardized evaluation of these models on tasks and real-life user cases that will benefit from the pipeline of “first pre-encode embeddings, then do inference with only similarity match”. Note that our evaluation is in line with some most established works in this field (e.g., Contriever [1], which is a widely-adopted unsupervised dense retrieval checkpoint; and also COCO-DR, which we compared to in our work as well). \n\n2) We would also like to discuss that: it would make less sense to use representation models like ours to do other downstream tasks, as other downstream tasks (say, classification tasks) do more one-time inferences, and it will make more sense to directly finetune a vanilla model with some extra task-specific heads, instead of using a representation model to pre-compute the embeddings for multiple-times inferences (especially with similarity match as in retrieval tasks).\n\n3) Apart from (document-level) dense retriever models, we also try to align with sentence-level representation models like SimCSE, InfoCSE, etc. As we discussed in the paper, the reason why we did not advocate evaluation on semantic textual similarity tasks is because previous works have found the performances on STS-b do not display strong correlations with downstream tasks [2,3] and this is because of its limited length range [4]. This becomes especially unsuitable because our work is on the problem of length, and needs evaluation from a wider length range. However, we also reported our sts-b performance in Appendix A for completeness, and discussed our performance.\nHopefully these discussions clear the reviewer’s concern on evaluation!\n\nW2 (Compare with previous works): As the reviewer noted, we identified two works that are relevant [5,6]. 1) The BEIR paper ([5]) identified that models trained with dot-product and cosine similarity exhibit different length preferences, but didn’t propose a method to address this vulnerability, but only suggested training with either dot-product or cosine similarity based on specific user case. 2) The only paper that suggests an approach to combat this issue is from [6]. Our paper differs in terms of a full mathematical explanation for this phenomenon; a systematic study of different elongation times (fixed-times longer or random-times longer) and its relationship with generalization; and a novel intra-reference approach, which is shown to perform better than the self-reference approach (see Table 2).\n\n[1] Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A. and Grave, E., 2022. Unsupervised Dense Information Retrieval with Contrastive Learning. Transactions on Machine Learning Research.\n\n[2] Reimers, N., Beyer, P. and Gurevych, I., 2016, December. Task-oriented intrinsic evaluation of semantic textual similarity. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 87-96).\n\n[3] Wang, K., Reimers, N. and Gurevych, I., 2021, November. TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 671-688).\n\n[4] Abe, K., Yokoi, S., Kajiwara, T. and Inui, K., 2022, November. Why is sentence similarity benchmark not predictive of application-oriented task performance?. In Proceedings of the 3rd Workshop on Evaluation and Comparison of NLP Systems (pp. 70-87).\n\n[5] Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. and Gurevych, I., 2021, August. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).\n\n[6] Xiao, C., Ye, Z., Hudson, G.T., Sun, Z., Blunsom, P. and Al Moubayed, N., 2023. Can Text Encoders be Deceived by Length Attack?.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides an extensive analysis of the length generalizability of standard contrastive learning methods, addressing a significant yet overlooked research gap. The authors derive the theoretical foundations underlying length attacks and demonstrate that elongating a document intensifies the high intra-document similarity already present in contrastive learning. This analysis sheds light on the vulnerability of contrastive text encoders to length-induced semantic shifts, highlighting the importance of considering length in representation learning.\n\n2. The paper proposes a simple yet universal framework called LA(SER) (Length-Agnostic Self-Reference for Semantically Robust Sentence Representation Learning) to bridge the unideal properties of contrastive learning. LA(SER) leverages the assumption that the elongated version of a sentence should still mean itself and should not become more or less similar to its pairs. By providing this simple signal, LA(SER) acts as an unsupervised contrastive learning method that enhances robustness to length attacks and improves encoding ability without sacrificing representational power. This framework offers a practical solution to address the length-based vulnerability of contrastive text encoders.", "reasons_to_reject": "1. One weakness of the paper is that the evaluation of the proposed LA(SER) framework is primarily focused on unsupervised performance on the standard information retrieval benchmark. While this evaluation provides valuable insights into the effectiveness of LA(SER)3 in improving contrastive text encoders' robustness to length attacks, it would be beneficial to also evaluate the framework on other downstream tasks or benchmarks to assess its generalizability and applicability in different NLP domains.\n\n2. The paper lacks a comprehensive comparison with existing methods that address the length vulnerability of contrastive learning. While the authors briefly mention related work on length preference of text encoders [4], there is no direct comparison with these methods or a discussion on how LA(SER)3 differs from or improves upon them. Including such a comparison would provide a clearer understanding of the unique contributions and advantages of LA(SER)3 in the context of existing approaches.", "questions_to_authors": "Please address the concerns in the \"Reasons to Reject\"", "paper_topic_and_main_contributions": "This paper introduces LA(SER), a length-agnostic self-reference framework, to improve the robustness of contrastive learning-based models to document length. Experimental results demonstrate the effectiveness of LA(SER) in enhancing the performance of contrastive text encoders. The method leverages length as a semantic signal and conducts contrastive learning with elongated sentences to improve the robustness and isotropy of representations.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "eZxrGkPsmi", "reviewer": "Reviewer_ZBQ8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful concerns.\n\nP1 (Writing and novelty): Thanks for pointing this out. In the final version, we will 1) add mathematical description of LA(SER)$^3$ self-reference and intra-reference in the method section to make it easier to follow. 2) improve the description of the novelty of this paper, including that we a) formally introduce the intra-document interaction change underlying elongation attacks through math derivation, b) conduct a systematic study of different elongation strategies (fixed-times longer or random-times longer) and their impacts on generalization, c) a novel intra-reference approach, which is shown to perform better than the self-reference approach.\n\nP2 (mathematical formulation): Indeed, there might be a set of dominant tokens that jointly decide the important aspects of the document-level semantics. Following the reviewer’s advice, we find that in order to introduce multiple dominant tokens, it might be most appropriate to introduce an “attention gain threshold” to decide whether a token belongs to “dominant tokens”. As our current math equations are defined to compute the attention gain of single tokens, if $X_{dominant}$ is now defined as multiple tokens, the new attention gain of dominant tokens will just be the sum of all the single dominant-token attention gains as defined in our current math equations. Notably, this adjustment does not change our derivation that elongation attacks make more attention lean towards dominant tokens than towards regular tokens.\n\nP3 (Analytical Experiments): Although we have presented two proof-of-concept experiments before proposing LA(SER)$^3$, we agree that we can benefit from more analytical experiments after the models have been trained with LA(SER)$^3$. In the final version, we will add the following experiments as also partly mentioned in replies to other reviewers: \n1) We will present that LA(SER)$^3$ does make the models more robust to length attacks by showing a) LA(SER)$^3$ (BERT-base-uncased trained on wiki with unsupervised contrastive learning) perceives a more stable cosine similarity distribution between attacked text pairs (only shifted from the cosine similarity distribution between unattacked text pairs with a Jenson Shannon divergence of 0.17, compared to 0.39 JS divergence when encoded with its SimCSE counterpart). b) showing some examples wrongly retrieved by models like SimCSE because they are long and contain many misleading lexical overlaps but not necessarily a right match, but avoided by LA(SER)$^3$. \n\n2) We will present a table to show the nDCG@10 per length range of documents retrieved by LA(SER)$^3$ models. We will show that LA(SER)$^3$ provides models with abilities to extrapolate accurate meanings in longer length ranges (by yielding a much higher nDCG@10 on longer texts), which the original datasets do not contain if we didn’t augment the data.\n\nWe believe adding such analytical experiments will add on to the interpretability of LA(SER)$^3$ performance gain. We are happy to add more experiments if the reviewer has other suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The computation of sentence representations is a crucial aspect of natural language processing (NLP) tasks, with potential applications across various downstream tasks. Enhancing the resilience of encoding models is thus a significant undertaking in NLP research.\n\nThis paper presents a mathematical formulation of the elongation attack for contrastive learning. It describes the variations in semantics between the original text and the text that has undergone an elongation attack, and the shift in patterns of intra-document tokens following such an attack. This could potentially offer a comprehensible reference for future studies.\n\nWith the aid of the generalized mathematical formulation, this paper suggests two approaches that can effectively counter the elongation attack.\n", "reasons_to_reject": "The writing and presentation of the paper require improvement, as the current version lacks distinctiveness as a novel contribution and may be challenging to differentiate from prior research [1].  The paper's introduction predominantly focuses on prior research. Similarly, Insufficient description of the proposed methods and the lack of formal representation of both methods make subsequent work difficult to fellow.\n\nThe proposed mathematical formulation lacks strict definition, as there exist various token forms that satisfy the $x_{dominat}$ definition. Consequently, this ambiguity could undermine the interpretability of the definition.\n\nWhile the paper includes comprehensive quantitative experiments, the absence of analytical experiments for the elongation attack may impede the ability to ascertain the actual effectiveness of the proposed method.\n\n[1]. Xiao C, Ye Z, Hudson G T, et al. Can Text Encoders be Deceived by Length Attack?[J]. 2023.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the vulnerability of contrastive learning (CL)-based models to length-induced semantic shift and proposes a length-agnostic self-reference framework for semantically robust sentence representation learning. The authors show that elongating a document intensifies the high intra-document similarity brought by CL and that isotropy promised by CL is highly dependent on the length range of text exposed in training. The proposed framework, LA(SER)^3, relies solely on the semantic signal provided by document length and achieves state-of-the-art unsupervised performance on a standard information retrieval benchmark.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this work the authors extensively analyze the effect of sentence/document length in contrastive training of sentence/document encoders. They show that contrastive training, recently ubiqitous in training sentence encoders, introduces a substantial length bias. The authors propose to reduce the length bias of the resulting embedding space by means of training data augmentation, i.e., synthetic elongation of training sentences. The effectiveness of the proposed approach is evaluated on the BEIR benchmark, which encompasses 14 different IR tasks with "documents" of varying lengths (sentences, tweets, news stories, ...).  

The reviewers are generally appreciative of the analysis of length biases in representations of encoders trained with contrastive objectives and the proposed augmentation-based remedy. Concerns raised w.r.t. shown positional invariance (i.e., that length matters more than the position of the content in the text) and downstream NLU evaluation have been, in my opinion, convincingly addressed by the authors, both by means of reporting additional experiments and acceptable argumentation (e.g., that one would fine-tune vanilla LMs rather than sentence/text encoders for supervised NLU tasks). Some concerns, however, have not been fully addressed: question of direct comparison against the approach of Xiao et al. (ICLR 23), and that of performance on STS (unsupervised, similarity-based). 

The work of Xiao et al in principle challenges the novelty of the contributions of this work -- as they also demonstrate that contrastive learning leads to overfitting to lengths observed in training data. I believe, however, that this work can be seen as concurrent/contemporary: it would still require that the authors compare against this work in the revised version of this work. The argument that authors offer against STS is that of limited length variance in the dataset, but I did not find this to be particularly convincing. STS results seem to show that LA(SER)^3, i.e., the augmentation strategy, seems to hurt the performance of short sentences. 

Overall, however, the contributions of this work outweigh the remaining concerns (which, however, I still believe should be remedied in the revised version of the paper)
Decision: Accept-Main