HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddings
Authors: Zhuofeng Wu
Chaowei Xiao
V.G.Vinod Vydiswaran
Venue: EMNLP 2023 Findings (2023)
Keywords: Contrastive Learning
Semantic Textual Similarity
Hierarchical Training
TL;DR: A new training paradigm for textual contrastive learning that can seamless integrate with previous frameworks and improve them.

Abstract: In this paper, we propose a hierarchical contrastive learning framework, HiCL, which considers local segment-level and global sequence-level relationships to improve training efficiency and effectiveness.  
Traditional methods typically encode a sequence in its entirety for contrast with others, often neglecting local representation learning, leading to challenges in generalizing to shorter texts. Conversely, HiCL improves its effectiveness by dividing the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence-level relationships. 
Further, considering the quadratic time complexity of transformers over input tokens, HiCL boosts training efficiency by first encoding short segments and then aggregating them to obtain the sequence representation.
Extensive experiments show that HiCL enhances the prior top-performing SNCSE model across seven extensively evaluated STS tasks, with an average increase of +0.2% observed on $BERT_{large}$ and +0.44% on $RoBERTa_{large}$.

HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence
Embeddings
Zhuofeng Wu University of Michigan Ann Arbor zhuofeng@umich. edu
Chaowei Xiao
VG Vinod Vydiswaran University of Michigan Ann Arbor vgvinodv@umich. edu
University of Wisconsin Madison cxiao34ewisc.edu
Abstract
To achieve better performance, existing methods mainly focus on designing better positive exam- ples (Hendrycks et al., 2020; Fang et al., 2020; Wu et al., 2020; Giorgi et al., 2021; Gao et al., 2021b) or investigating the role of the negative ex- amples (Robinson et al., 2020; Zhou et al., 2022; Wang et al., 2022).
In this paper. we propose a hierarchical contrastive learning framework, HICI, which considers local segment-level and global sequence-level relationships to improve train- ing efficiency and effectiveness. Traditional methods typically encode sequence in its tirety for contrast with others, often neglecting local representation learning, leading chal- lenges in generalizing to shorter texts. Con- versely, HiCL improves its effectiveness by di- viding the sequence into several segments and employing both local and global contrastive learning to model segment-level and sequence- level relationships. Further, considering the quadratic time complexity of transformers over input tokens. HiCL boosts training efficiency by first encoding short segments and then aggre- gating them to obtain the sequence representa- tion. Extensive experiments show that HiCL en- hances the prior top- performifg SNCSE model across seven extensively evaluated STS tasks, with average increase of 40.2% observed BERT large and +0.44% on RoBERTaaage
Despite the success, existing methods augment data at the level of the full sequence (Gao et al., 2021b; Wang et al., 2022). Such methods require calculating the entire sequence representation, lead- ing to high computational cost. Additionally, it also makes the task of distinguishing positive ex- amples from negative ones too easy, which doesn't lead to learning meaningful representations. Sim- ilarly, methods like CLEAR (Wu et al., 2020) demonstrated that pre-training with sequence-level naïve augmentation can cause the model to con- verge too quickly, resulting in poor generalization. In contrast, Zhang et al. (2019) considered mod- eling in-sequence (or local) relationships for lan- guage understanding. They divide the sequence into smaller segments to learn intrinsic and under- lying relationships within the sequence. Since this method is effective in modeling long sequences by not truncating the input and avoiding loss of infor- mation, it achieves promising results. Given this success, a natural question arises: is it possible to design an effective and efficient contrastive learn- ing framework by considering the local segment- level and global sequence-level relationships?
1 Introduction
Current machine learning systems benefit greatly from large amounts of labeled data. However, ob- taining such labeled data expensive through an- notation in supervised learning. To address this issue, self-supervised learning, where supervisory labels are defined from the data itself, has been pro- posed. Among them, contrastive learning (Chen et al., b.c, 2021; He et al., 2020; Grill et al.. 2020; Chen and He, 2021) has become of the most popular self-supervised learning methods due its impressive performance acToss various do- mains. The training target of contrastive learning is to learn a representation the data that maximizes the similarity between positive examples and min- imizes the similarity between negative examples.
To answer the question, in this paper, we pro- pose a hierarchical contrastive learning framework, HiCL, which not only considers global relation- ships but also values local relationships, as illus- trated in Figure 1. Specifically, given sequence (i.e., sentence), HiCL first divides it into smaller segments and encodes each segment calculate local segment representation respectively. It then aggregates the local segment representations be- longing to the same sequence to get the global
OOu code will he released at https: http:/gggttttttteeee hom....oommm....oo....o......... /Serxy/HiCL.

---

sequence representation. Having obtained local and global representations, HiCL deploys hierar- chical contrastive learning strategy involving both segment-level and sequence-level contrastive learn- ing to derive an enhanced representation. For lo- cal contrastive learning, each segment is fed into the model twice to form the positive pair, with segments from differing sequences serving as the negative examples. For global contrastive learning, HiCL aligns with mainstream baselines construct positive/negative pairs.
esimlhe
Cg = log (oimm( )/T +
Σ JEB
coim(hshg)/T
(1)
Benefiting from human-defined data augmenta- tions, it can generate numerous positive and neg- ative examples for training without the need for explicit supervision, which is arguably the key rea- son why sele-supervised learning can be effective.
We have carried out extensive experiments on seven STS tasks using well-representattve models BERT and RoBERTa as our backbones. We as- sess the method's generalization capability against three baselines: SimCSE, ESimCSE, and SNCSE. As a result, we improve the current state-of-the-an model SNCSE over seven STS tasks and achieve new state-of-the-an results. Multiple initializations and varied training corpora confirmed the robust- ness of our HiCL method.
Positive instance Designing effective data aug- mentations to generate positive examples is key challenge in contrastive learning. Various methods such as back-translation, span sampling, word dele- tion, reordering, and synonym substitution have been explored for language understanding tasks in prior works such as CERT (Fang et al., 2020), DeCLUTR (Giorgi et al., 2021), and CLEAR (Wu et al., 2020). Different from previous approaches that augment data at the discrete text level, Sim- CSE (Gao et al., 2021b) first applied dropout (Sri- vastava et al., 2014) twice to obtain two intermedi- ate representations for positive pair. Specifically, given Transformers model, Eg (parameterized by 0), (Vaswani et al., 2017) and a training instance 24, h2 = (3t) and h Eg. + (1:) are the pos- itive pair that can be used in Eq. 1, where p and are different dropout masks. This method has been shown to significantly improve sentence em- bedding performance on seven STS tasks, making it a standard comparison method in this field.
Our contributions are summarized below:
To the best of our knowledge, we are the first to explore the relationship between local and global representation for contrastive learning in NLP. We theoretically demonstrate that the encoding efficiency of our proposed method is much faster than prior contrastive training paradigms.
We empirically verify that the proposed training paradigm enhances the performance of current state-of-the-art methods for sentence embeddings on seven STS tasks.
Negative instance Negative instance selection is another important aspect of contrastive learn- ing, SimCLR simply uses all other examples in the same batch as negatives. However, in DCLR (Zhou et al., 2022), around half of in-batch negatives were similar to SimCSE's training corpus (with cosine similarity above 0.7). To address this is- sue, SNCSE (Wang et al., 2022) introduced the use of negated sentences as "soft" negative sam- ples (e.g., by adding not to the original sentence). Additionally, instead of using the [CLS) token't 's vector representation, SNCSE incorporates a man- ually designed prompt: "The sentence of z; means [MASK]" and takes the [MASK] token's 's vector to represent the full sentence 24. This approach has been shown improve performance compared using the [CLS] token's vector. this paper, we compare against SNCSE as another key bascline, not only because it is the current oate-of-the-ar
2 Preliminaries: Contrastive Learning
In this paper, we primarily follow SimCLR's frame- work (Chen et al., 2020a) our basic contrastive framework and describe it below. The general train- ing objective of contrastive learning (Oord al., 2018) is to distinguish similar pairs from dissimi- lar pairs, where similar pairs are constructed using pre-defined data augmentation techniques and dis- similar pairs are other examples in the same batch. Specifically, for an arbitrary example z; in batch B, the InfoNCE loss Ly brings the representation h closer positive instance representation hy and away from negative ones h m. If hi, h me are the representation vectors from the encoder, is a temperature scale factor (set to 0.05 following SimCSE), and sim(u, v) = //'u/|/al] ||||||||| denotes the cosine similarity, Cg is computed as:

---

positive
Global Contrastive
negative
negative
mean pooling
mean pooling
mean pooling
Local Contrastive
Encoder
Seg
Seg 2 ...
Seg
...
Seg
Seg
Seg
Seg 1
Seg 2
Seg L
Seq
Seq
Seq 72
Figure 1: The overview the HiCL framework with local contrastive and global contrastive objective.
evaluation tasks, but also because it effectively combines contrastive learning with techniques like prompt tuning (Gao et al., 2021a; Wu al., 2022c).
segments that can be sliced in seq; The slicing is performed using queue rule: every consecutive L tokens (with no overlap) group as one segment, and the remaining tokens with length no greater than form a separate segment. In other words, seg.,, = L, Vj E [1,4]; |segr,, E [1, LJ; and seq, concat 1. seg.,,
Momentum Contrast The momentum contrast framework differs from SimCLR by expanding the negative pool through the inclusion of recent in- stances, effectively increasing the batch size with- out causing out-of-memory issues. ESimCSE (Wu et al., 2022b) proposes a repetition operation to generate positive instances and utilizes momentum contrast to update the model. We include it as baseline for comparison to assess the ability of our model adapt to momentum contrast.
Unlike traditional contrastive learning, which en- codes the input sequence seq, directly, we encode each sub-sequence segi,j using the same encoder and obtain its representation: hij Eø(seg.,) where Eg is a Transformer (Vaswani al., 2017). parameterized by A. We aggregate the haj rep- resentations to obtain the whole sequence repre- sentation h; by weighted average pooling, where the weight of each segment segi,j is proportional to its length |seg.,): h = E,hi, Wijj where In Section 5.1, we explore other pooling methods, such as unweighted average pool- ing, and find that weighted pooling is the most effective. According to Table 2, most (99.696%) input instances can be divided into three or fewer segments. Therefore, we not add an extra trans- former layer to get the sequence representation from these segments, as they are relatively short.
3 Hierarchical Contrastive Learning
3.1 Overview
Figure shows an overview of HiCL. Our primary goal is to incorporate additional underlying (local) information in traditional, unsupervised text con- trastive learning. Two objectives are combined to achieve this goal.
|seg,,) W. Σx /seg..l
of sequences seq,) } in batch B, slice each seq; into segments {segr,1, segi,; seg,,,) of slicing length L, where n is the batch size, and 4 = 1 + [([seq. - 1)/L is the number of
Given
{
To use HiCL with SNCSE, we slice the input se-

---

quence in the same way, but add the prompt each segment instead of the entire sequence. We also apply the same method to the negated sentences.
standalone global contrastive loss is equally inad- equate, it omits local observation. We conduct an analysis in Section 5.4 to discuss the intricate relationship between two objectives.
3.2 Training Objectives
Local contrastive Previous studies have high- lighted the benefits of local contrastive learning for unsupervised models (Wu et al., 2020; Giorgi et al., 2021). By enabling the model to focus on short sen- tences, local contrastive learning allows the model better match the sentence length distribution, as longer sentences are less common. Building on the work of Gao et al. (2021b), we use dropout as min- imum data augmentation technique. We feed each segment seg,.j twice to the encoder using different dropout masks and pt. This results in positive pairs haj= = (s.g()) and h = ptg (seg..) for loss computation. As mentioned in Section 1, defining negatives for segments can be challenging. Using segments from the same sequence as neg- atives carries the risk of introducing correlations, but treating them as positive pairs not ideal ei- ther. We chose not to use segments from the same sequence as either positive or negative pairs and we will show that this approach is better than the other alternatives in Section 5.2. Hence, for segment segi.. we only consider negatives, segments from other sequences (sega,., k B {))). The local contrastive L is formalized as:
3.3 Encoding Time Complexity Analysis
According to our slicing rule, all front segments jeg in sequence seq, have length L and the last segment segi,,, has length |segi,1; E [1. L]. Hence, the encoding time complexity for HiCL is O(L (4, 1) |seg,, [2), while the conventional methods take:
O(sseq,1 O( x (4: - 1) 1)9)
> (4: (4- x O - + segi, 12)
which is (4--1 1) times more than that for HiCL. The longer the training corpus, the higher the benefit using HiCL. This suggests that our approach has variety of use cases, particularly in pre-training. due its efficient encoding process.
The practical training time is detailed in Ap- pendix A.1. In short, we are faster than baselines when maintaining the same sequence truncation size 512. For example, SimCSE-RoBERTai, takes 354.5 minutes for training, while our method only costs 152 minutes.
4 Experiment
sim(he tht e cim
4 log
4.1 Experimental Setup
+ Σ ham
Evaluation tasks Following prior works (Gao et al., 2021b; Wang et al., 2022), we mainly eval- uate on seven standard semantic textual similar- ity datasets: STS12-16 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK -eeK-KRRRlttdssss (Marelli et al., 2014). When evaluating using the SentEval tookit?, we adopt SimCSE's evaluation setting without any additional regressor and use Spearman's correla- tion as the evaluation matrix. Each method cal- culates the similarity, ranging from to 5, be- tween sentence pairs in the datasets. In our ablation study, we extend our evaluation to encompass two lengthy datasets (i.c., Yelp (Zhang et al., 2015) and IMDB (Maas et al., 2011)) and seven trans- fer tasks (Conneau and Kiela, 2018). Due space consideration, have provided detailed results for the seven transfer learning tasks in Appendix A.2.
Global contrastive The global contrastive ob- jective is the same as that used by most baselines, which tries to pull a sequence's representation hi closer to its positive h while keeping it away from in-batch negatives n s5 as defined by the global contrastive loss Lg in Eq.
Overall objective The overall objective is a combination of local and global contrastive loss, L = =aLt + (1 agg. where weight a C (0.01,0.33, 0.15] is tuned for backbone models. Our adoption of hybrid loss, with a lower weight- ing assigned to the local contrastive objective, is motivated by the potential influence of the hard truncation process applied to the sequences. This process can result in information loss and atypical sentence beginnings that may undermine the effec- tiveness of the local contrastive loss. Meanwhile, a
Pht http:////shthttccccc//cccccrrrrs

---

Table 1: Main results of various contrastive learning methods on seven semantic textual similarity (STS) datasets. Each method is evaluated full test sets by Spearman's correlation, "all" setting. Bold marks the best result among all competing methods under the same backbone model.
Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSE-BERT base
66.92
78.44
71.15
79.50
77.71
75.50
68.76
74.00
HiCL
69.04
80.68
72.71
80.39
78.68
76.96
70.35
75.541
ESSiCCSE-BERTT..
70.95
82.76
76.52
83.18
80.03
80.15
71.07
77.81
HiCL
70.05
83.18
76.51
82.97
80.31
80.02
72.53
77.941
SNCSE-BERT
70.15
84.36
76,86
83.21
80.16
81.09
75.04
78.70
HiCL
70.57
84.51
76.92
82.97
79.91
80.68
74.61
78.60
SimCSE-BERTHN lagee
70.17
83.27
72.82
82.53
77.05
78.25
66.01
75.73
HHCL
70.99
85.29
75.62
83.56
79.40
79.91
74.13
78.411
"ESinCSE-BERTT
71.43
83.91
75.88
83.84
78.86
79.74
73.59
78.18
HiCL
72.40
85.42
77.29
84.59
79.78
80.72
74.28
79.211
BICCSE-BERTEA large
72.03
86.80
78.48
85.27
80.65
82.20
74.40
79.98
+HiCL
72.40
86.78
78.50
85.52
80.85
82,09
75.13
80.181
SimCSE-RoBERTagge
69.47
82.12
73.96
82.48
81.11
81.06
69.45
77.09
HiCI.
69.36
81.77
73.75
82.58
81.03
81.06
69.58
77.02
ESimCSE- MEEISE---RRTaNN
69.03
80.73
73.35
81.26
81.25
80.26
68.11
76.28
HiCI.
69.98
81.31
74.39
82.87
81.43
80.74
69.09
77.121
SSNCSE-RoEERTaie Tabuwe
70.50
83.70
76.55
84.09
81.89
81.73
74.11
78.94
HHCL
71.01
83.70
76.08
84.06
81.89
82.07
73.54
78.91
SinCSS-RRBBRKTamn.
71.41
83.90
76.50
85.07
82. 10
82.75
71.17
78.99
HICL
72.36
84.08
76.35
85.07
82.49
82,95
71.40
79.241
SBiinSS--ROEETIH
70.92
83.25
75.62
81.91
80.16
81,73
72.03
77.95
HiCL
73.06
84.44
76.56
84.59
81.42
83.61
71.49
79.311
SNCSEE-RGBRRTAL
73.65
86.45
779.23
86.60
82.45
83.95
77.12
81.35
-HILL
73.74
86.19
79.76
86.70
83.20
84.52
78.45
81.791
Implementing competing methods We re-train three previous state-of-the-art unsupervised sen- tence embedding methods STS tasks Sim- CSE (Gao et al., 2021b), ESimCSE (Wu et al., 2022b), and SNCSE (Wang et al., 2022) with our novel training paradigm, HiCL. We employ the official implementations of these models and adhere to the unsupervised setting utilized by Sim- CSE. This involves training the models on a dataset comprising million English Wikipedia sentences for single epoch. More detailed information can be found in Appendix A.3. We also carefully tune the weight of local contrastive along with the learning rate in Appendix A.4.
and SNCSE baselines, respectively. (ii) HiCL with SNCSE-RoBERTTuees as the backbone has state-of-the-art score of 81.79. It is worth mentioning that SNCSE originally reported score of 81.77 based on full precision (fp32), but we achieved better result with half precision (fp16). (iii) HiCL enhances the performance across nine backbone models, yet marginally underper- forms (< 0.1) on three models employing the -base architecture.
achieved
All prior studies in the field have followed com- mon practice of employing the same random seed for a single run to ensure fair comparisons. We have rigorously adhered to this convention when presenting our findings in Table 1. However, to further assess the robustness of our method, we have extended our investigation by conducting mul- tiple runs with varying random seeds. As shown in Table 15, generally observe consistency be- tween the multi-run results and the one-run results. The nine backbone models on which HiCL demon- strated superior performance under the default ran- dom seed continue to be outperformed by HiCL.
4.2 Main Results
Table presents the results of adding HiCL to various baselines the seven STS tasks. We observe that (i) HiCL consistently improves the performance of baselines in the large model ting. Specifically, HICL improves the average scores by +0.25%, +1.36% and +0.44% on the RoBERTanage variants SimCSE, ESimCSE,

---

with three different versions: considering them as positive pairs, 2) treating them as negative pairs, and 3) categorizing them as neither positive nor negative pairs. The results in Table 4 of SimCSE and SNCSE indicate that the optimal approach is to treat them as neither positive nor negative conclusion that aligns with our expectations. An outlier to this trend is observed in the sults for ESimCSE. We postulate that this anomaly arises due to ESimCSE's use of token duplication to create positive examples, which increases the similarity of segments within the same sequence. Consequently, this outcome is not entirely unex- pected. Given that ESimCSE's case is special, we believe that, in general, it is most appropriate to label them as neither positive nor negative.
Input Length
[0, 32] 76.031%
(32, 64]
(64, 96]
Proportion
21.869%
1.796%
Input Length
(96. 128] 0.225%
(128, 256]
(256, 512]
Proportion
0.075%
0.004%
Table The input length distribution training corpus. Special tokens [CLS] and [SEP] counted.
5 Intrinsic Study and Discussion
5.1 Local Information Aggregation
Forming the representation for the entire sequence from segment representations is crucial to our task. We experiment with both weighted average pool- ing (weighted by the number of tokens in each seg- ment) and unweighted average pooling. Since most sequences are divided into three or fewer segments (as shown Table 2), we did not include an addi- tional layer (cither DNN or Transformer) to model relationships with < 3 inputs. Therefore, we opted to not consider aggregating through deep neural layer, even though this approach might work in scenarios where the training sequences are longer. Results in Table indicate that under three differ- ent backbones, weighted pooling is better strategy for extracting the global representation from local segments.
Backbone
Positive
Negutive
Neither
SinCsE-RoBERTa..
73.92
74.34
75.24
ESinCSF-RoBERTIE
76.96
74.35
74.37
SNCSE- RESSF-
77.33
79.33
80.86
Table 4: Development set results of SICK-R (Spear- man's correlation) for processing relationship of seg- ments from same sequence.
5.3 Optimal Slicing Length
To verify the impact of slicing length per- formance, we vary the slicing length from 16 to 40 for HiCL in SinCSE-ROBERTaung and SNCSE-ROBERTAaure settings (not counting prompts). We were unable to process longer lengths due to memory limitations. From the re- sults in Figure 2, we find that using short slicing length can negatively impact model performance. As we showed in Section 3.3, the longer the slicing length, the slower it is to encode the whole input quence, because longer slicing lengths mean fewer segments. Therefore, we recommend setting the default slicing length as 32, as provides good balance between performance and efficiency.
Backbone
Pooling
SICK-R
unweighted weighted
73.91
75.24
ESimCSE-RRBELRTanng
unweighted
73.13
weighted
74.37
SNCSE-RoBERTuung
unweighted
79.97
weighted 80.86
Table 3: Development set results of SICK-R (Spear- man's correlation) for different pooling matrices.
5.2 Relationships between Segments from Same Sequence
We acknowledge that using a non-fixed slicing strategy, such as truncation by punctuation, could potentially enhance performance. Nevertheless, in pursuit of maximizing encoding efficiency, we have opted for fixed-length truncation approach. leaving the investigation of alternative strategies future work.
When optimizing the local contrastive objective, an alternative approach is to follow the traditional training paradigm, which treats all other segments negatives. However, since different parts of sequence might describe the same thing, segments from the same sequence could be more similar com- pared random segment. As result, establish- ing the relationship between segments from the same sequence presents a challenge. We explore
5.4 How Hierarchical Training Helps?
One might argue that our proposed method could benefit from the truncated portions of the training

---

Table 5: Performance on seven STS tasks for methods trained on wiki-103 and 1 million English Wikipedia sentences. Each method is evaluated full test sets by Spearman's correlation, "all" setting.
Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SimCSE-RooBe RoBERT
71.72
84.33
76.45
84.16
80.36
81.90
72.27
78.74
HICL
72.45
84.69
76.98
84.79
81.64
82.43
71.54
79.22
We further verify this hypothesis by doing an abla- tion study on HiCL with various values a in the overall objective. Recall that our training objective is a combination of local contrastive and global contrastive loss. HiCL model with a = 0 is identi- cal to the baselines, except that it incorporates the omitted information. As shown in Table 7, train- ing with just global contrastive loss with complete information yields poorer results. Similarly, when a = the HICL model focuses solely on local contrastive loss, but also performs poorly, which indicates that global contrastive loss is an essential component in learning sentence representation.
3ICK-R (Dev
75
(
HCL-SNCSE-R RLLLSNNEERERRTTAUU lageRTa-
71
IICL- MammCC-RRLLRRR RoDERTs large
HCL-SimCSE RoBERTa
16
24
40
Slicing Length
Figure 2: Comparison between different slicing lengths over three backbone models by Spearman's 's correlation.
corpus the part exceeding the length limitations of baselines and thus, unprocessable by them. To address this concern, we reconstruct the training corpus in such a way that any sequence longer than the optimal slicing length (32) is divided and stored as several independent sequences, each of which fits within the length limit. This allows the baseline models to retain the same information as HiCL. The aforementioned models are indeed us- ing local-only loss, given they implement con- trastive loss on the segmented data. Table shows that HiCI. continues to exceed the performance of single segment-level contrastive learning models, indicating its superior performance is not solely reliant on the reduced data. The lower performance exhibited by these models effectively emphasizes the significance of incorporating hybrid loss.
0.0
0.01
0.05
0.15
SICK-R 75.10
75.19
75.24
74.07
0.3
0.5
0.95
1.0
SICK-R 73.44
74.68
74.87
74.39
Table 7: Ablation study over a SimCSE-Ro RoBERTa- large at SICK-R dev set (Spearman's correlation).
It's crucial to clarify that our approach isn't just a variant of the baseline with a larger batch. As Ta- ble indicates, in 99.9% of instances, the training data can be divided into four or fewer segments. Comparing SimCSE-BERT, base with a batch size quadruple that of our method (as shown in Ta- ble 10), it's evident that SimCSE at batch size of 256 trails our model's performance with batch size of 64 (74.00 vs. 76.35). Simply amplifying the batch size for baselines also leads to compu- tational issues. For example, SimCSE encounters an "Out of Memory" error with a batch size of 1024, a problem our model, with batch size of 256, avoids. Therefore, our approach is distinct and offers benefits beyond merely adjusting batch sizes.
Method
vanilla
+ HiCI.
full-
MImCSF-RoBERTaure
78.99
79.24
78.45
SNCSE-RoBERTaawy
81.35
81.79
80.41
Table Average performance baselines trained with full segments (abbreviated as full-) on seven STS tasks.
The intriguing insight from above observations is that the omitted data does not improve the perfor- mance of the baseline models; in fact, it hinders per- formance. We hypothesize that this is due a hard cut by length resulting in some segments beginning with unusual tokens, making it more difficult for the encoder to accurately represent their meaning.
5.5 HiCL on Longer Training Corpus
To further verify the effectiveness of HiCL, we add a longer corpus, WikiText-103, along with the original I million training data. WikiText-103 is a dataset that contains 103 million tokens from 28,47 475 articles. We adopt a smaller batch size of 64 to avoid out-of-mmmory issue. Other training

---

details followed the instructions in Section 4.1. As shown in Table 5, HiCL shows more improvement (+0.48%) compared to the version only trained on short corpus (+0.25%). This indicates that HiCL is more suitable for pre-training scenarios, particu- larly when the training corpus is relatively long.
another way of regarding them positive. As seen in the results in Section 5.2, treating segments from the same sequence as positive would negatively impact the performance SimCSE and SNCSE backhones. Thus, it is not surprising that HiCLv2 failed to show as much improvement as HiCL.
5.6 HiCL on Longer Test Datasets
SimCSE-BERTAN 75.73
HiCL
HiCLv2
The datasets that were widely evaluated, such as STS12-16, STS-B, and SICK-R, primarily con- sist of short examples. However, given our in- terest in understanding how HiCL performs on longer and more complex tasks, further con- duct evaluations the Yelp (Zhang al., 2015) and IMDB (Maas al., 201 1) datasets. Table provides an overview of these two datasets.
Avg
78.41
76.30
SNCSE-BERTI
HICL 80.18
HiCLv2
Avg
79.98
80.00
MAnCSF-RRBBRTaajie 78.99
HHCL 79.24
HiCLy2
Avg
79.08
SNCSE SNCSE-ROBERTIE RoBERTaag
HHCL
HiCLv2 81.49
Avg
81.35
81.79
Table 9; Average performance of HiCLv2 over seven STS datasets.
Specifically, we test with backbone model and follow the evaluation settings outlined in SimCSE, refraining from any further fine-tuning on Yelp and IMDB. The results are compelling, with our proposed method consistently outperforming SimCSE achieving a performance gain of +1.97% on Yelp and +2.27% on IMDB,
base
5.8 Baseline Reproduction
One might wonder why there are notable discrep- ancies between reproduced baselines and the numbers reported in the original papers. For in- stance, our SimCSE-BERT, base achieved score of 74.00, while the original paper reported 76.25. Indeed, this difference comes from the different hyperparameters we adopt.
Datasets
#Test
Length
SimCSE
+HiCI.
Yelp
38K
132.6
70.99
72.96
IMDB
25K
228.5
1 60.93
63.20
Table 8: Statistics and evaluation results on Yelp and
Different baselines adopt various configurations such batch size, training precision (fpl6 fp32), and other factors. Recognizing that these factors significantly influence the final results, our aim is to assess different baselines under consistent con- ditions. To clarify, it would be mislcading to eval- uate, say, SimCSE- BERCTE--ETTTAA hase with a batch size of 64 while assessing SNCSE-BERT base with batch size of 256. Such discrepancies could obscure the true reasons behind performance gaps. Therefore, we use a unified batch size of 256 for base models and 128 for large models.
IMDB datasets.
5.7 A Variant of HiCL
As we discussed in Section 5.2, the best approach to treat relationships between in-sequence segments is to consider them as neither positive nor nega- tive. However, this would result in losing informa- tion about them belonging to the same sequence. To overcome this, we consider modeling the rela- tionship between sequences and segments. Since each segment originates from a sequence, they in- herently contain an entailment relationship the sequence entails the segment. We refer to this vari- ant as HiCLv2. Additional details are provided in Appendix A.6.
To eliminate concerns about whether the proposed method can still work bascline's optimal hyperparameters,
reassess the SimCSE-BERThus model in Table 10. Regardless of whether we use SimCSE's optimal settings or our uniform configuration, our method consistently outperforms the baseline.
As shown in Table 9, explicitly modeling this sequence-segment relationship does not help the model. We think that it is probably because this ob- jective forces the representation of each sequence be closer to segments from the same sequence. When the representation of each sequence weighted average pooling of segments, pulls seg- ments from the same sequence closer, which is
Lastly, we want mention that some base- lines actually benefit from our standardized setup. For example, our reproduction of MInCSE-RoBERTaaa.. saw an increase, going from the originally reported 76.57 to 77.09.

---

Table 10: comparison between original SimCSE and reproduced SimCSE on seven semantic textual similarity (STS) datasets. Each method is evaluated full test sets by Spearman's correlation, "all" setting.
Method
batch
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
Original SimCSE-BEIE BERSEE
64
68.40
82.41
74.38
80.91
78.56
76.85
72.23
76.25
Reproduced SimCSE-BERT
256
66.92
78.44
71.15
79.50
77.71
75.50
68.76
74.00
+ HiCL
256
69.04
80.68
72.71
80.39
78.68
76,96
70.35
75.541
Reproduced SinCSE-BERThus
64
69.17
82.02
73.41
80.54
78.36
76.60
71.96
76.01
HiCL
64
69.44
82.10
74.48
81.62
78.77
77.75
70.26
76.351
6 Related Work
ers (Zhang et al., 2019; Yang et al., 2020). Both solutions divide the long sequence into smaller parts, making full use of the whole input for in- creased robustness compared to those using partial information. Additionally, hierarchical training is usually more time efficient. SBERT (Reimers and Gurevych, 2019) employs a similar idea of hier- archical training. Instead of following traditional fine-tuning methods that concatenate two sentences into one for encoding in sentence downstream tasks, SBERT found that separating the sentences and encoding them independently can drastically improve sentence embeddings. To our knowledge, the first to apply this hierarchical training technique textual contrastive learning.
Contrastive learning The recent ideas on con- trastive learning originate from computer vision, where data augmentation techniques such as AUG- MIX (Hendrycks et al., 2020) and mechanisms like end-to-end (Chen et al., 2020a), memory bank (Wu et al., 2018), 8), and momentum (He et al., 2020) have been tested for computing and memory effi- ciency. In NLP, since the success of SimCSE (Gao al., 2021b). considerable progress has been made towards unsupervised sentence representa- tion. This includes exploring text data augmen- tation techniques such as word repetition in ES- imCSE (Wu et al., 2022b) for positive pair gen- eration, randomly generated Gaussian noises in DCLR (Zhou et al., 2022), and negation of in- put in SNCSE (Wang al., 2022) for generating negatives. Other approaches design auxiliary net- works to assist contrastive objective (Chuang et al., 2022: Wu al., 2022a). Recently, a combination prompt tuning with contrastive learning has been developed in PromptBERT (Jiang et al., 2022) and SNCSE (Wang et al., 2022). With successful de- sign of negative sample generation and utilization of prompt, SNCSE resulted in the state-of-the-an performance the
Conclusion
We introduce HiCL, the first hierarchical con- trastive learning framework, highlighting the im- portance of incorporating local contrastive loss into the prior training paradigm. We delve into the op- timal methodology for navigating the relationship between segments from same sequence in the com- putation of local contrastive loss. Despite the extra time required for slicing sequences into segments, HiCL significiill accelerates the encoding time of traditional contrastive learning models, especially for long input sequences. Moreover, HiCL ex- cels in seamlessly integrating with various existing contrastive learning frameworks, enhancing their performance irrespective of their distinctive data augmentation techniques or foundational architec- tures. We employ SimCSE, ESimCSE, and SNCSE as case studies across seven STS tasks to demon- strate its scalability. Notably, our implementation with the SNCSE backbone model achieves the new state-of-the-art performance. This makes our hier- archical contrastive learning method promising approach for further research in this area.
seven STS tasks. Our novel training paradigm, HiCL, is com- patible with previous works and can be easily inte- grated with them. In our experiments, we re-train SimCSE, ESimCSE, and SNCSE with HiCL, show- ing a consistent improvement in all models.
Hierarchical training The concept of hierar- chical training has been proposed for long-text processing. Self-attention models like Trans- former (Vaswani et al., 2017) are limited by their input length, and truncation is their default method of processing text longer than the limit, which leads information loss. To combat this issue, researchers have either designed hierarchical Trans- formers (Liu and Lapata, 2019; Nawrot et al., 2022) or adapted long inputs to existing Transform-

---

Acknowledgment
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, Weiwei Guo. Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th Interna- tional Workshop Semantic Evaluation (SemEval 2014), pages 81-91, Dublin, Ireland. Association for Computational Linguistics.
The work is partly supported by the Endowment of Basic Sciences at the University of Michigan Med- ical School and by Dissertation Fellowships from the School of Information. Chaowei Xiao is sup- ported by the U.S. Department of Homeland Secu- rity under Grant Award Number, 17STQAC00001- 06-00.
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross- aross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) pages 497-511, San Diego, Califor- nia. Association for Computational Linguistics.
We thank Fei Sun, Tianyu Gao, Simone Conia, Pratyush Maini, Amrith Settur, Zhenhao Zhang, Ji- azhao Li, members of the University of Michigan's NLP4Health research group, and all anonymous reviewers for helpful discussion and valuable feed- back.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In #SEM 2012: The First Joins Conference on Lexical and Compu- tational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393, Montréal, Canada. Association for Computa- tional Linguistics.
Limitations
We acknowledge following limitations of our work: 1. Despite demonstrating the suitability of HICL for pre-training scenarios in Section 5.5, we were not able to fully pre-train our model from scratch. Further research is needed to verify the effective- ness of HiCL in this context.
2. Some baselines (e.g.. SimCSE) have shown that training a contrastive learning objective on human-labeled NIJ datasets can lead to improved performance. We did not investigate this super- vised setting.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013, *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume I: Proceedings of the Main Confer- ence and the Shared Task: Semantic Textual Similar- iry. pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Ethics Statement
Data All of the training data and evaluation benchmarks used in this paper are publicly avail- able and do not pose any privacy concerns.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation /SemEval-2017 pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
AI Writing Assistance We have utilized the ChatGPT to polish our original content, rather than generate new ideas or suggestions. Despite the fact that EMNLP 2023 has not clearly stated whether the use of generative language models needs to be disclosed, we believe it is best to be transparent and explicitly disclose this.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020a. simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119 Proceedings of Machine Learning Research, pages 1597-1607. PMLR
References
Ting Chen, Simon Kornblith, Kevin Swersky, Moham- mad Norouzi, and Geoffrey Hinton. 2020b. Big self- supervised models are strong semi-supervised learn- ers. arXiv preprint arXiv:2006.10029
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agiree, Weiwei Guo, lñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Jamyce Wiebe. 2015. SemEval-2015 task 2: Semantic tex- tual similarity, English. Spanish and pilot on inter- pretability. In Proceedings of the 9th International Workshop Semantic Evaluation (SemEval 2015). pages 252-263, Denver, Colorado. Association for Computational Linguistics.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020c. Improved baselines with momentum con- trastive learning. arXiv preprint arXiv:2003.04297,
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In 202] IEEE/CVE

---

Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 35745-15753
Neural Information Processing Systems, volume 33, Curran Associates, Inc.
pages
Xinlei Chen, Saining Xie, and Kaiming He. 2021. An empirical study of training self-supervised visual transformers, arXiv preprint arXiv:2104.00 020577
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic. Shang- Wen Li, Scott Yih, Yoon Kim, and James Glass. 2022. DifiCSE: Difference- based contrastive learning for sentence embeddings. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 4207-4218, Seattle, United States. Association for Computational Lin- guistics.
Dan Hendrycks. Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. 2020. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Represen- tations (ICLR).
Minqing Hu and Bing Liu. 2004, Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representa- tions. arxiv preprint 183iv:1803.05449.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrusing (IWP2005).
Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang. Fuzhen Zhuang. Furu Wei, Haizhen Huang. Liangjie Zhang, and Qi Zhang. 2022. Prompthert: Improving bert sentence embeddings with prompts. arXiv preprint 2201.00330
Hongchao Fang, Sicheng Wang. Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005. 12766.
Yang Liu and Mirella Lapata. 2019. Hierarchical trans- formers for multi-document summarization. In Pro- ceedings of the S7th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 5070- 5081, Florence, Italy. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch. and Danqi Chen. 2021a. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- val Language Processing (Volume I: Long Papers), pages 3816-3830, Online, Association for Computa- tional Linguistics.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 202 SimC Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zam- parelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (L)EC'14), pages 216-223, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for un- supervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1i International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 879-895, Online. Association for Computational Linguisties.
Piotr Nawrot, Szymon Tworkowski, Michat Tyrolski. Lukasz Kaiser, Yuhuai Wu. Christian Szegedy, and Henryk Michalewski. 2022. Hierarchical transform- more efficient language models. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1559-1571, Seattle, United States. Association for Computational Linguistics.
Jean- lcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. 2020. Bootstrap your own latent- new approach to self-supervised learning. In Advances in
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arxiv: 1000:007773744

---

Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACI,-04), pages 271-278, Barcelona, Spain.
Xing Wu. Chaochen Gao. Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022a. Infocse: Information-aggreaated contrastive learning of sen- tence embeddings. arXiv preprint arXiv:22000066
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022b. ESim- CSE: Enhanced sample building method for con- trastive learning of unsupervised sentence embed- ding. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 3898- 3907, Gyeongju. Republic of Korea, International Committee on Computational Linguistics.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu- lational Linguistics (ACI, '05), pages 11 115-124, Ann Arhor, Michigan. Association for Computational Lin- guistics.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natu- ral Language Processing (EMNLP-IJCNLP) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Rui Hou, Yux- iao Dong V.G. Vinod Vydiswaran, and Hao Ma. 2022c. IDPG: An instance-dependent prompt genera- tion method. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 5507-5521, Seattle, United States. Association for Computational Linguistics.
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020. Contrastive learn- ing with hard negative samples. arXiv preprint arXiv:2010.04592.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 163 1-312, Seattle, Washington, USA. Association for Computational Linguistics.
Zhuofeng Wu, Sinong Wang. Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012. 15466.
Liu Yang, Mingyang Zhang, Cheng Li, Michael Ben- dersky, and Mare Najork. 2020. Beyond 512 tokens: Siamese multi-depth waasformer-basse hierarchical encoder for long-form document matching. In Pro- ceedings of the 29th ACM International Conference Information & Knowledge Management, pages 1725-1734.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Îlya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Re- search, 15(56):1922) 1929-198888
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Xingxing Zhang, Furu Wei, and Ming Zhou, 2019. HI- BERT: Document level pre- pre-training of hierarchical bidirectional transformers for document summariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Compu- tational Linguistics.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR confer- ence on Research and development in information retrieval, pages 200-207.
Kun Zhou, Beichen Zhang, Xin Zhao, and Ji-Rong Wen. 2022. Debiased contrastive learning of unsupervised sentence representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 6120- 6130, Dublin, Ireland. Association for Computational Linguistics.
Hao Wang, Yangguang Li, Zhen Huang, Yong Dou, Lingpeng Kong, and Jing Shao. 2022. Sucse: Con- trastive learning for unsupervised sentence embed- ding with soft negative samples. arXiv preprint arXiv:2201.0597
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions language. Language resources and evaluation, 39(2): 165-210.

---

A Appendix
sure fair comparison, unify the training setup across all competing methods and strictly follow each baseline's ' hyperparameter tuning process to re-tune the optimal hyperparameters accordingly. Specifically, we employ a batch size of 256 for base models and 128 for large models. All base models trained using full precision (fp32), while large models are trained using half precision (fp16) to mitigate potential memory issues with certain models. worth noting that disparities in performance between our re-run models and the original baselines may arise due to our intentional parameter adjustments to facilitate a direct compar- ison with other baselines.
A.1 Training Time
The practical training time can be complex, as the actual encoding time does not strictly follow quadratic rule. However, our method demonstrates advantages in terms of efficiency when maintain- ing the same sequence truncation size. For exam- ple, while SinCSE-RRBERRTa-lre takes approx- imately 354.5 minutes for training, our method achieves the same task in just 152 minutes. The ac- celeration in time stems from two primary factors: I) Savings encoding time, as discussed Sec- tion 3.3, and 2) The capability of HiCL to handle significantly larger batch sizes, enabling parallel processing for further acceleration. Please note that these advantages in efficiency are discussed specifically in the context of training with identi- cal information, where the same truncation size is maintained. The original SimCSE, which adopts a truncation size of 32, is indeed faster than our approach as it does not need process truncated in- formation.
Following the procedures used by SimCSE and SNCSE, we evaluate the model every 125 training steps on the development set of STS-B and select the best checkpoint for the final evaluation on the test sets.
A.4 Hyperparameter tuning of HiCL
We first tune the weight of local contrastive a E [0.01, 0.05, 0.15) along with learning rate 5e-6 for large models and le-5 for base models. After fix- ing an optimized a, proceed tune the learning rate using the suggested values for each model, in- cluding 6 as an additional option. Specifically, we tune the learning rate to be one of the following values: [5e-6, 1e-5, 3e-5, 5e-5} } for SimCSE mod- els, (5e-6, le-5, 3e-5] for ESimCSE models, and {5e-6, le-5} for SNCSE models. The optmized hyperparameters of HiCL are listed in Table 133.
Backbone BERT'
BERT;
Roberta;
Roberta
SimCSE 87.5
344.5
92.4
354.5
HiCL
69.4
148.4
71.8
152.0
Table 12: Training minutes for models with 512 se- quence truncation size. for base models. for large models.
A.2 Transfer Learning
We also evaluate competing methods sev- eral transfer tasks: MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), MRPC (Dolan and Brockett, 2005).
Method
α
SimCSF-BER'
0.15
5e-5
SimCSE-BERT Iim-
0.I5
"ee-5
SimCSE-Ron COCCSEE-RBERTaus..
le-5
0.05
56-6
ESimCSF-BERT
0.01
3e-5
A comparison between SimCSE-BERT buse (us- ing its reported optimized hyperparameters and without adding MLM loss) and our model can be found in Table 11. While our approach may not demonstrate enhancements across all tasks, it does register improvements
ES BERT iare
0.01
5
ES Ro
0.01
16-5
DSSinCSS-RRBEERTaa..
0.01
5e-6
SNCSE- ANCSE-BERT
le-5
SNCSE- BERTINE
0.01
16-5
SNCSE RoBERTaaee
0.01
Te-3
out of the tasks, with each exceeding a 0.4% increase. The aggregate improvement stands at 0.39%.
SNCSE- DOSSS--RRBERTajnn
0.15
56-6
Table 13: Optimized hyperparameters of HiCL. Ir; learn ing rate.
A.3 Experimental setup
*e find that the results on the SICK-R --R development set were more consistent with the results on the seven STS test sets. Once an optimized checkpoint was identified through STS dev set, we fine-tuned the hyperparameters using the SICK-R development set.
Different baselines utilize varying training setups, including batch size, training precision (fp16 or fp32), and other factors. In order to maximally en-

---

Table 11: Transfer task results sentence embedding performance (evaluted as accuracy). Bold marks the best result among competing methods under the same backbone model.
Method
MR
CR
SUBJ
MPQA
SST2
TREC
MRPC
Avg
SimCSE-BERThan
80.34
85.35
94.65
89.21
84.68
88.00
73.10
85.05
+ HiCL
80.76
86.39
94.61
89.02
85.17
88.60
73.68
85.46
For HiCLv2, we use the optimized values in HiCL and only tune for (le-5, 3e-5, le-4, 3e-4). Optimized |8 is shown in Table 14.
segments from the same sequence. Intuitively, seg- ments from the same sequence likely be more similar each other than to a random segment. However, this is not always the case, as segments from the same sequence can have contrary mean- ing. Modeling this relationship is difficult because it has both high degree of correlation, yet no clear relationship between segments. To tackle this prob- lem, we instead focus on modeling the relationship between a sequence and its segments. This is more straightforward, as we know that segi,, comes from seq,, and therefore they naturally form an entail- ment relationship. By doing this, we also retain the information about whether two segments come from the same sequence. Figure 3 provides an overview of this variant framework.
Method
lr
BinCSF-BERTimn T
0.15
3e-4
5c-6
SNCSE-BERT 0.01
3e-4
55-6
SimCSE- SinCSE-RoBERTiane RoBERTauge
0.05
3c-5
5e-6
SNCSE-RoBERTaa.n 0.15
Te-5"
55-6
Table 14: Optimized hyperparameters of HiCLv2 for four models Section 5.7. Ir: learning rate.
A.5 Multiple Runs
In addition to the conventional practice of com- paring models under the same random seed, we test the generalizability of our proposed method by using different random seeds, presented in Table 15. An inherent challenge is the decision whether retune the hyperparameters, considering the optimized ones under one initialization may vary under a different one. With the aim of investi- gating whether the optimized hyperparameters can be effectively transferred across various random seeds, we have opted not to retune the hyperparam- eters."
We employ a third contrastive objective to model the entailment relationship. Specifically, a segment segi is entailed by sequence seq, but should not be entailed by seqk, Ak: + i. Therefore, we treat segijj and seq, a positive pair, and all other sequences in the batch negative pairs with seg; We optimize the following InfoNCE loss function:
sim(he Ax)/T
Le - llog gim((h.. Md/T + Σ cim((haghee/T
HiCL improves performance across the identical nine backbone models as shown in Table 1, thereby demonstrating its robust generalization capabili- ties. The average scores in the multi-run setting are uniformly lower than those in the one-run setting, possibly due to the lack of sufficient hyperparam- eter tuning. This lack of tuning may also result in reduced performance gap between HiCL and the baseline models.
(3)
Overall objective of HiCLv2 The overall ob- jective of HiCLv2 trastive, global contrastive, and entailment loss, given by L = aL, + BL. + (1 - - LB) where a and 3 are the weights.
combination of local con-
A.7 Computing Infrastructure
A.6 Variant HiCL
All models were trained on single 48GB memory NVIDIA A40 GPU for one epoch, using the same initialization, i.c.. the same random seed used by all baselines, for one run. The server has the following configuration: Intel(R) Xeon(R) Gold 6226R CPU @ 2.90GHz. x86-64 with CentOS 7 Linux operating system. PyTorch 1.7.1 is used as the programming framework.
Sequence-segmen entailment. The sequence- segment entailment objective is designed an al- ternative way to model the relationship between
"We note that both DEESSE--ERT--beee models exhibit a substantial standard deviation, complicating the assessment of the models' performance. As a consequence, we only make minor adjustments to the learning rate for these two models for both baseline and HiCL method.

---

Table 13: Results various contrastive learning methods on seven semantic textual similarity (STS) datasets. We report average results across 3 runs with different initialization. Each method is evaluated on full test sets by Spearman's s correlation, "all" setting. Bold marks the best result among all competing methods under the same backbone model.
Method
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SinCSB-BERTTas
=12
79.63-24
72.06
79.71
77.77
75,44
69.69-1.2
74,53
+ HiCL.
67.04_1.7
80.94
71.9310T
80.07
78.16103
75.55
69.57
74.7510r1
DISiICSE-BERT.H
70.07:19
81.95_1.8
75.35+1.a
88.85500
79,44400
79.66.07
70.93-03
77.1841.1
HICL
70.24
82.34_0.8
76.05+0.0
83.03+02
79.73403
79,47=00.
71.40=111
77.46+0.4
ENCSE-BERT
70.53
84.37_o.1
16.
83.59.00
80,28
81.28
74.99
78.86
HICL..
70.87.03
84,48
76.8410.2
83.21
79.6
80.87
74.85
78.67
SimCSS-BBERT.p
70.03-0.9
82.55-1.2
74.34
82.96:0.9
77,86+09
79.09
70.44
76.75409
HiCL
68.38
83.03 _22___________
14.04
83.10192
18.22
77.89 .20
73.99
76.95
ESSiMCSSEBBERT
22.25
84.77_us
2u.
84.38 24.
79.40.20
80.36
74.37.07
78.89207
HiCL
73.24
84.93
77.2A+0.1
84.73+01
80.08
80.79
74.23=0a
79.32+n11
SNNCS-BBER BESTS-
71.76
86.59-0.3
78.60
85.61
80.46+02
82.07
75.23-0.8
80.05
HICL
72.34
86.61=0.3
52.11
85.43
80.47+04
82.14
75.43=0.5
80.13
SinCCE--RoBKTTau
69.65
81.86=0.4
74.19
82,46+a.7
81,40
81.17
68.88
77.08
HICL
69.02.0.
81 2..44.0.4
74.05 +4.0
82.66 02.66=n.
81.62407
81 2002
68.87=07
77.05
ESSinSSE-RBBEE R
69,49103
8142_
73.7610.4
81.38
80,98 104
80.62
68. 18.7
76.70.04
HiCL
69.73109
81.48 0.2
14.3510-3
82.61
81.23104
80.67 1o.
68.69 68.69.0.
76.97
UNCCSF-ROBERRRINEE
70.47.012
83.27_0.4
76.15.0.4
84.13
81,48107
81.83
73.18
78.65 78.65104
HICL
70.16=0.9
=3,41
75.73+0.3
83,90sa2
81,32+0.
81,36
72.40=1.1
78.33
BtnCCS-RRBERRTiA
71.13108
83.33
75.6910.3
84.04.0.5
80.74
81,97.0.4
71.1310.3
78.291002
HICL
71 71.48_21.0
83.84_0.6
75. 75.90
84, 84.86.0.4
81,91
82.33
71.62-2011
78.852n.41
ESincSSE-RoBRRTuuse
72.70=1.6
83,49_0.2
76. 76.39.n.7
83.07
80.53
82,27
72. 72.66=0.6
78.73+07
HiCL
72.46.107
83.74_0.0
76. 2:
84.56:02
81,20
82,89.04
71,58-07
78.98+0.1.
73.18
86.15 0.5
79.08103
86.81 10.3
82.40 1o.1
83.48104
77.15
81.18102
HHCL
72.99.07
85.74_..8
79.07
86.45
82.77
83,70
77.84.0.
81.2210.7
Global Contrastive
Seq-Seg Entailment
mean pooling
mean pooling
mean pooling
Local Contrastive
Positive
Encoder
Negatives
Entail
Seg
Seg
Seg
Seg
Seg
Seg
Seg
Seg
Seg
Entail
Seq
Seq i
Seq n
Figure 3: The overview of HiCLv2 framework with local contrastive, global contrastive, and sequence-segment entailment objective.

---

Table 16: Results various contrastive learning methods on seven semantic textual similarity (STS) datasets. We report average results across 3 runs with different initialization. Each method is evaluated on full test sets by Spearman's 's correlation, "all" setting.
Method
loss
S1S12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg.
SinCSE-HHCCT
summation
71.48s10
83,84206
75.90_p.4
84.86.04
81.91
82.33,06
71.62102
78.85_oaf
SimCSE+HiCL
averaging
71.35
83,86an3
=0.
84.73.0.3
82.00
82, 12.10111
71.1100
28.
ASimCSE-HiiC
summation
72,46207
83.74
76.29-0.4
84 34.5620.0
81.20
82.8920
71.58207
78.96_0.1
ESmuCSE-HICL
averaging
71.81
83,53
75.77-m2
84,63-022
81.60494
82.45
71.32
78.73
"NCŠE-HiCL
summation
"f2.9
85.74ens
79.07=0.7
86 25.2
"82.77
83.70
77.84.0s
"81.22nn.ff"
SNCSEHHICL
averaging
72.79
86.05
79.12.0.
86.47.0.
83.03
83.53
77.10.09
81.16,03
A.8 Summation or Averaging in Contrasting Negatives
Eq. 2, we contrast a segment hij with seg- ments from different sequences hy,+, where , i, which is more like picking the correct pair out of a very big batch. We investigate an alternative way which includes weighted factor when comput- ing similarities between hisj and has,. Specifically, wk X (h..m.h)..)..)))) assumes contrasting over
a fixed small batch instead of using an extended big batch, where Wig = et is the number of segments that sequence k is divided into. In essence, this design would allocate varying weights to different segments in computation: segments derived from shorter sequences would receive higher weight, while those from longer sequences would be signed lower weight. A critical query here is: should we differentiate the importance of segments based on their originating sequence length? To an- swer this question, we compare this new design (named averaging) with our previous version in Eq. 2 (named summationn.
We benchmark the averaging loss against sum- mation using three different models: SimCSE, ES- imCSE, and SNCSE, all the RoBERTatarg back- bone. To control for initialization effects, we've 'e provided results from three-run evaluations on seven STS datascts in Table 16. It is clear that the summation loss consistently outperforms the averaging loss, which proves that varying the im- portance of segments based on source length may not yield beneficial outcomes.


Reviews:
[{"review_id": "OXWFMOjTPD", "reviewer": "Reviewer_69hi", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their encouraging feedback. We address the questions below:\n\n> *1. Too strong claims*\n\nThanks for the suggestion! We will revise the corresponding claims in the next version. Specifically:\n* in Line 22 \"achieving new state-of-the-art results\" -> \"gaining +0.2% on BERT_large and +0.44% on RoBERTa_large over prior best results\" \n*  in Line 23 \"we are the first to explore\" -> \"we explore\"\n\n> *2. Main Results do not seem to strongly support the claim*\n\nFirst, we'd like to emphasize that our method's **primary value** lies in its **seamless integration** with existing CL frameworks, irrespective of their augmentation methodologies or foundational architectures. This integration consistently drives improvements across two pivotal axes: encoding efficiency and overall performance.\n\nRegarding efficiency enhancements, both theoretical analysis (Section 3.3) and practical tests (Appendix A.1) are provided to show our model’s general improvement over baselines.\n\nIn terms of performance, let us recall the results in Table 1 on 12 backbones:\n* Our model (HiCL) closely aligns with four baselines, with a difference of less than 0.2%.\n* HiCL outpaces another four baselines, with gains between 0.2% to 1%.\n* Importantly, HiCL marks a notable uplift of over 1% for the last four baselines.\n\nThese outcomes are consistent across three separate initializations. While we acknowledge that our model shows similar performance to the baselines on the four backbones, **we wish to remind that performance enhancement is just one facet of our contribution.** Significantly, our proposed approach enhances the encoding efficiency of traditional CL methods considerably.\n\n> *3. Whether the improvement is statistically significant*\n\nWhat we have done: we executed the models on three separate random seeds and presented the average performance alongside the standard deviation in Table 12. A more thorough analysis can be found in Lines 345-358 and Appendix A.4. Briefly, there's a notable consistency between the multi-run results and the single-run outcomes. \n\nWe concur that the confidence interval provides a more meaningful measure of results' significance than the standard deviation. Acknowledging the importance of your input, we intend to delve deeper into the significance of improvements in the camera-ready version. We hope this could address the reviewer’s concern about whether the improvement is significant.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- First to propose a hierarchical contrastive learning framework (which is a bold claim and surprising if true)\n- Extensive ablation study and robustness checks\n- SOTA performance (at least in some settings)", "reasons_to_reject": "- Generally too strong claims (\"First\", \"SOTA\")\n- Main results do not seem to strongly support the authors' claim that HiCL generally improves the baseline models\n- Comparing just the average is a poor practice. I think the confidence intervals should be reported in order to judge whether improvement is statistically significant.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Main contributions: novel approaches to improve computational efficiency\n- first encoding local segments and then aggregating to obtain global sequence representation\n- this way, the authors claim to have improved training efficiency and also SOTA performance on seven STS tasks", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4lPfMbW9Jl", "reviewer": "Reviewer_2q7u", "summary": "", "questions": "see reason to reject (especially better motivation for the central method).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback. We address the comments below.\n\n> *1. Not well-motivated for local contrastive loss* and *the benefits of our method*\n\n> *1.1 Not well-motivated for local contrastive loss*\n\nIn general, the local contrastive is proposed for:\n* **Making the task of distinguishing positives and negatives more challenging** (stated between Line 54-61) for learning better representation. More specifically, encoding as a whole makes contrasting task too easy to learn meaningful representations, given that longer sequences naturally convey more information. As highlighted in Lines 58-61, CLEAR[1] has shown that naive sequence-level augmentation can lead to the failure of CL training. Similarly, SNCSE[2] concluded that introducing hard negatives increases the challenge of identifying positive pairs, ultimately leading to a more effective model. \n* **Processing longer sequences** (stated between Line 66-73) and **avoiding information loss** by not dropping the part exceeds length limitation.\n* **Addressing the expensive cost for encoding** the entire sequence directly (stated between Line 52-54).\n\nTherefore, to enhance encoding efficiency and intensify the challenges of the contrastive task, we opted for including local contrastive loss.\n\n> *1.2 The benefits of our method*\n\nTo address your concern regarding whether our proposed method simply serves as an alternative to the baseline with a larger batch size, we have undertaken further experiments. Given that in 99.9% of cases the training data can be segmented into four or fewer segments, we conducted a comparison between the baseline (based on SimCSE-BERT_base) with a batch size 4 times larger than our method. The results are as follows:\n\n|        |  batch  |  STS12 |   STS13    |STS14| STS15  | STS16 | STS-B |  SICK-R |   Avg. | \n|:-----:|:--------:|:----------:|:------------:|:--------:|:-----:|:--------:|:----------:|:------------:|:--------:|\n| SimCSE| 64     | 69.17   | 82.02  | 73.41  | 80.54 |78.36 |76.60  | 71.96   |76.01|  \n| SimCSE| 256   | 66.92  | 78.44  | 71.15  | 79.50 | 77.71 | 75.50 | 68.76  | 74.00 | \n|+HiCL     | 64     | 69.44 | 82.10    | 74.48 | 81.62 | 78.77 | 77.75   | 70.26   | 76.35↑|\n\nFrom above table:\n* **Clearly, SimCSE, when using a batch size of 256, lags behind our model that operates with a smaller batch size of 64 (74.00 vs. 76.35)**.\n* Increasing the batch size alone isn't a surefire way to boost performance, as corroborated by the SimCSE paper.\n\nFurther, **scaling up baselines by simply ramping up the batch size introduces computational challenges**. For instance, attempting a batch size of 1024 with SimCSE triggers an \"Out of Memory\" (OOM) error during training, an issue our method (with a batch size of 256) does not encounter conversely.\n\nIn light of these findings, we respectfully assert that **our methodology is not merely a \"trick\" analogous to employing baselines with larger batch sizes**.\n\n> *2. Very small improvement* and *mismatch performance for SimCSE-BERT model*\n\n> *2.1 Very small improvement*\n\nFirst, we *respectfully disagree* with the characterization that our performance improvements are very small.\n\nTo provide a fair assessment, out of 12 different backbones:\n* Our model closely matched four backbones (with <0.2% performance difference). \n* HiCL surpassed another four backbones by a margin of 0.2% to 1%. \n* Notably, HiCL achieved a substantial improvement, exceeding 1%, on the remaining four backbones.\n\nThe above results are verified on 3 different initializations and show good consistency. We mentioned between Line 900-907 that we do not re-tune the hyperparameters on different random seeds, which could explain why the improvement becomes smaller on 3-runs. We believe that by carefully fine-tuning the hyperparameters for each random seed, further enhancements can be achieved.\n\nWe'd like to underscore that our method's primary value lies in its **seamless integration** with existing CL frameworks, irrespective of their augmentation methodologies or foundational architectures. This integration consistently drives improvements across two pivotal axes: encoding efficiency (shown in Section 3.3 and Appendix A.1) and overall performance.\n\nWhile we acknowledge that our model shows similar performance to the baselines on the four backbones, we urge reviewers to **adopt a holistic perspective when evaluating our contributions, not solely based on performance**. Performance metrics are just one facet of our contribution.  It’s worth noting that our proposed methodology also improves traditional CL’s encoding efficiency by a large margin. \n\n> *2.2 Mismatch Performance for SimCSE model*\n\nThe discrepancy in performance for SimCSE models arises because we **unified the batch sizes to ensure a fair comparison** across baselines (using 256 for base models and 128 for large models). \n\nWe reassessed the SimCSE-BERT_base model using its optimal hyperparameters (batch size 64) and the results are displayed below. **Regardless of whether we use SimCSE’s optimal settings or our uniform configuration, our method consistently outperforms the baseline.**\n\n|        |  batch  |  STS12 |   STS13    |STS14| STS15  | STS16 | STS-B |  SICK-R|   Avg. | \n|:-----:|:--------:|:----------:|:------------:|:--------:|:-----:|:--------:|:----------:|:------------:|:--------:|\n|Original SimCSE      |  64    | 68.40  | 82.41  | 74.38  | 80.91 | 78.56 | 76.85 | 72.23  | 76.25 | \n|Reproduced SimCSE| 256   | 66.92  | 78.44  | 71.15  | 79.50 | 77.71 | 75.50 | 68.76  | 74.00 | \n| + HiCL |                   256     | 69.04   | 80.68  |72.71   |80.39  |78.68| 76.96  |70.35    |75.54↑| \n| Reproduced SimCSE| 64   | 69.17   | 82.02  | 73.41  | 80.54 |78.36 |76.60  | 71.96   |76.01|  \n|+HiCL | 64 | 69.44 | 82.10    | 74.48 | 81.62 | 78.77 | 77.75   | 70.26   | 76.35↑|\n\nAs outlined in Lines 858-874, baselines vary in configurations like batch size. Recognizing that these factors significantly influence the final results, our aim was to assess different baselines under consistent conditions. To clarify, it would be misleading to evaluate, say, SimCSE-BERT_base with a batch size of 64 while assessing SNCSE-BERT_base with a batch size of 256. Such discrepancies could obscure the true reasons behind performance gaps. \n\nLastly, we want to mention that some baselines benefitted from our standardized setup, displaying improved performance. For example, our reproduced SimCSE-RoBERTa_base jumped from an original 76.57 to 77.09. Likewise, SimCSE-RoBERTa_large saw an increase from 78.90 to 78.99.\n\n> *3. Generalizing to longer training corpus*\n\nWe'd like to draw the reviewers' attention to Section 5.5, where we have already evaluated HiCL's performance on the training corpora with the longer sequence length.\n\nSpecifically, we incorporated WikiText-103, which comprises 103 million tokens derived from 28,475 articles, with each article averaging 3617 tokens. Our findings indicate a more pronounced improvement for HiCL over the baseline when the training corpus is longer, with the enhancement increasing from +0.25% to +0.48%. We hope this answer could address your concern about train-test length mismatch setting (which is exactly train on long, test on short). \n\nWe would greatly appreciate it if the reviewers could re-assess our work, especially if our explanations have clarified any previous ambiguities. We will incorporate further motivation in our revised manuscript.\n\n---\n[1] Wu, Zhuofeng, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. \"Clear: Contrastive learning for sentence representation.\" arXiv preprint arXiv:2012.15466 (2020).\n\n[2] Wang, Hao, and Yong Dou. \"SNCSE: contrastive learning for unsupervised sentence embedding with soft negative samples.\" In International Conference on Intelligent Computing, pp. 419-431. Singapore: Springer Nature Singapore, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The \"intrinsic study and discussion\" section is thorough and sound. \n2. The encoding time advantage might be useful when generalizing to datasets with longer documents.", "reasons_to_reject": "1. The local contrastive loss is not well-motivated. It can be questioned that this is just beneficial/relavant to the optimization of the training. Empirically, this trick might be similar to baseline methods + a larger batch size.\n\n2. The performance improvement is very small. Also, in the main table, the reproduction of SimCSE BERT base is over 2 absolute points lower than reported in the original paper, and the reproduction of BERT large also falls largely behind perception from people with experience in the field (when you check Table 12 when the authors did 3 runs, this seems to be relatively fixed, but making the performance improvement even smaller).\n\n3. I will strongly suggest that the authors prove the effectiveness of this method on generalizing to longer documents, instead of competing on sts tasks. The authors first claim that baseline methods might fail to generalize to shorter sequence, but wikipedia-1M (train) and sts (test) are both short, making sts not a good suite to test out the superiority of the method (in terms of length generalization). In 3.2, the authors state that \"By enabling the model to focus on short sentences, local contrastive learning allows the model to better match the sentence length distribution, as longer sentences are less common.\" If the authors understand this correctly, it'll be more sound to prove the effectiveness of the method on train-test length mismatch settings (train on long, test on short; train on short, test on long).", "questions_to_authors": "see reason to reject (especially better motivation for the central method).", "paper_topic_and_main_contributions": "The paper proposes using hierarchical contrastive learning to improve unsupersvised sentence representation learning. The encoding efficiency of the method might be beneficial to the field.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p79ZcfTFcV", "reviewer": "Reviewer_etFb", "summary": "", "questions": "1. Does the author solely focus on linearly splitting sentences on average? The exploration of the hierarchical structure and semantics of the segments seems limited, which I believe is crucial for constructing positive and negative samples.\n2. Segment-level encoding only takes into account tokens within each segment. Could this potentially eliminate long-context dependencies, such as when two semantically related words in the same sentence are divided into separate segments?\n3. Does considering segments from two sentences solely as negatives introduce noise or even incorrect samples? If so, how can this be avoided? Providing examples in the paper would enhance understanding.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We will fix the typo pointed out. We address the questions in below:\n\n> *1. Motivation for local segments*\n\nIn general, the local contrastive is proposed for:\n\n* Making the task of distinguishing positives and negatives more challenging (stated between Line 54-61) for learning better representation. More specifically, encoding as a whole makes contrasting task too easy to learn meaningful representations, given that longer sequences naturally convey more information. As highlighted in Lines 58-61, CLEAR[1] has shown that naive sequence-level augmentation can lead to the failure of CL training. Similarly, SNCSE[2]  concluded that introducing hard negatives increases the challenge of identifying positive pairs, ultimately leading to a more effective model. \n* Processing longer sequences (stated between Line 66-73) and avoiding information loss by not dropping the part exceeds length limitation.\n* Addressing the expensive cost for encoding the entire sequence directly (stated between Line 52-54).\n\nTherefore, to enhance encoding efficiency and intensify the challenges of the contrastive task, we opted for including local contrastive loss. \n\n> *2. Potential noise introduced by fixed slicing length*\n\nThis is a good point, we actually have also observed the issue with fixed truncation size in Section 5.4 through the experiment that trains a solo local contrastive model. \nBetter segmentation could lead to a better performance, but will also come with other issues such as low encoding efficiency as it slices into more segments compared to fixed truncation strategy. \nFurther, traditional methods also introduce noise into training. For instance, “I like a banana but hate an apple” with truncation size 3 for illustration, traditional CL only contrasts using “I like a”, which can also mislead the model.\nOverall, even with our current fixed truncation strategy, we have already attained commendable results and plan to leave the exploration of dynamic truncation for future work. \n\n*Q1. Exploration on other hierarchical structure and semantics of the segments*\n\nWe acknowledge that other slicing ways (such as truncation by punctuation) might bring more improvements. However, as we mentioned between Line 419-425, due to time constraints and consideration for encoding efficiency, we have opted for a fixed-length truncation approach and leave the investigation of the approaches you mentioned to future work.\n\n*Q2. Drawback of segment-level encoding*\n\nWe agree that segment-level encoding could eliminate some long-text dependency. Yet, we compensate it somehow via aggregating all segment information to form the whole sequence-level representation. \nIt's worth noting that this isn't a challenge exclusive to our method. Traditional CL also meets with this issue when inputs surpass its length constraint, resorting to a break of long-text dependency.\n\n*Q3. The relationship of two segments from different sequences*\n\nAt present, we regard segments from disparate sequences as negative pairs. We recognize the potential pitfall of this approach, as it could give rise to false negatives—segments from distinct sequences could be semantic similar. \nNonetheless, this challenge isn't unique to our method. Traditional CL also involves false negatives when designating all other instances in a batch as negatives.\n\nWe noticed that one paper DCLR[3] partially addresses this issue by re-assigning the negative pairs that have a high semantic similarity (larger than a threshold). However, its computing of semantic similarity of each pair requires an external model, which makes it much more computationally expensive.\n\n-----\n[1] Wu, Zhuofeng, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. \"Clear: Contrastive learning for sentence representation.\" arXiv preprint arXiv:2012.15466 (2020).\n\n[2] Wang, Hao, and Yong Dou. \"SNCSE: contrastive learning for unsupervised sentence embedding with soft negative samples.\" In International Conference on Intelligent Computing, pp. 419-431. Singapore: Springer Nature Singapore, 2023.\n\n[3] Zhou, Kun, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. \"Debiased contrastive learning of unsupervised sentence representations.\" in ACL 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a pioneering approach by incorporating hierarchical relationships into contrastive learning within the field of NLP, as explicitly mentioned. While the application of hierarchical training in transformer-like architectures is well-documented in the second part of the Related Work section, the introduction of local contrastive objectives is a novel contribution.\n2. The paper has a comprehensive intrinsic analysis, and it is well-written and easy for readers to follow.", "reasons_to_reject": "The motivation provided in the paper to apply contrastive learning on shorter segments may not be sufficiently strong. The efficiency aspect mentioned on line 54, which highlights that hierarchical training is more time-saving due to encoding shorter contexts, seems unrelated to contrastive learning itself. In other words, there exist other efficient methods using BERT that can also reduce computational costs.\n\nIn terms of effectiveness (lines 54-61), the paper argues against solely relying on using entire sentences for contrastive learning. However, it is questionable whether using shorter segments alone is adequate, as focusing solely on sentence length may overlook other important factors. For example, shorter segments might introduce more noise and make it easier for two segments (from different sentences) to appear identical, resulting in them being mistakenly labeled as negative samples.\n\nThis suggests that a more comprehensive consideration of various factors beyond just segment length is necessary when exploring the effectiveness of contrastive learning.", "questions_to_authors": "1. Does the author solely focus on linearly splitting sentences on average? The exploration of the hierarchical structure and semantics of the segments seems limited, which I believe is crucial for constructing positive and negative samples.\n2. Segment-level encoding only takes into account tokens within each segment. Could this potentially eliminate long-context dependencies, such as when two semantically related words in the same sentence are divided into separate segments?\n3. Does considering segments from two sentences solely as negatives introduce noise or even incorrect samples? If so, how can this be avoided? Providing examples in the paper would enhance understanding.", "paper_topic_and_main_contributions": "The paper introduces a novel hierarchical contrastive learning framework (referred to as HiCL) that incorporates both local (segment-level) and global (sentence-level) relationships for unsupervised sentence embedding. The experimental findings demonstrate the efficacy of the proposed framework, consistently outperforming baseline models in the majority of cases. Furthermore, the framework showcases notable efficiency by significantly reducing training time.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L250 \"also not\" -> \"not ... either\"\n\nL569 \"same sequence\" -> \"the same sequence\""}, "extra_text": {}}, {"review_id": "Y0WUX3MFnC", "reviewer": "Reviewer_pM2M", "summary": "", "questions": "- In p.228, the authors mentioned that most input instances can be divided into three or fewer segments. This means that the sequence length is usually not long. If most of the inputs are long sequences, it makes sense to divide the sequence to obtain a segment-level representation, but in other cases, it is difficult to expect a significant effect of using segment-level representation.\n- This paper is assuming that the length of segments in a sequence is the same. However, the reviewer thinks that each segment length should be variable according to its locality. For example, given a sentence like “I like a banana but hate an apple”, if we assume that the number of segments is 3 as the segment length, the segments are <I like a>, <banana but hate>, and <an apple>. In this case, each segment does not contain its meaning well. In this example, is it correct that the segments are <I like a banana> and <but hate an apple>?\n- In Equation 2, is the term of the denominator about negative segments (i.e., sigma_{k<>i} sigma_{v} e^{sim(h_{i,j}, h_{k,v})/tau}) correct? I think that averaging e^{sim(h_{i,j}, h_{k,v})/tau} is better than the summation.\n- To see the robustness of the proposed model, it is recommended to apply it to other natural language processing tasks as well as the semantic textual similarity task.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. We address the comments below. \n\n> *1. Unclear figure*\n \nThanks for pointing out that our current figure makes some confusion to your interpretation. Here we make some clarifications:\n\n1. \"It is not clear how the encoder in Figure 1 is added on BERT and RoBERTa\"\n\n* The encoder itself is a BERT or RoBERTa. \n\n2. \"Is it correct to use the proposed hierarchical contrastive leaning as a loss function for BERT or RoBERTa model?\"\n\n* Yes, it is. The BERT and RoBERTa only serve as the basic backbone model to extract text features initially. SimCSE[1] also uses BERT and RoBERTa as the backbone, and computes contrastive loss based on the extracted sentence features. \n\n3. \"What does the arrows in Figure 1 mean?\"\n\n* In Figure 1, the green dashed arrows indicate that the destination vector serves as a negative instance in relation to the origin vector. Conversely, the blue arrows depict two passes of the same text with varying dropout, forming a positive pair. This is consistent with SimCSE's definition. \n\nWe wish to emphasize that our visual depiction of the training process is inspired by well-established conventions. A comparable illustration can be seen in Figure 1 of the SimCSE paper. However, we will refine the figure for clarity in our final version.\n\n> *2. Research goal*\n\nIn essence, we introduce a new contrastive learning paradigm that enhances existing CL frameworks, boosting both encoding efficiency and effectiveness to learn a good representation. That is to say. We are not aiming to design for the specific task such as semantic textual similarity but with the goal to learn a good representation. \nHowever, while the majority of preceding studies [1,2,3,4,5] focus their evaluation of CL objectives solely on the STS benchmark, SimCSE[1] emphasizes that for effective performance on other transfer NLP tasks, the CL objective must be complemented with additional losses. Given our mission to enhance the CL training paradigm, we've chosen to present our results on the STS benchmark, staying consistent with the prevalent norms in the CL community. \nIf you believe that prevailing conventions within this community have led to misunderstandings, we are open to refining our assertions in the revised version.\n\n> *3. Other technique contribution than segment contrastive learning*\n\nSegment-level contrastive learning is our center technique contribution. We made a lot of effort to explore what’s the most proper way to make it work, including but not limited to:\n1. studying how to aggregate segment representation to get sequence representation;\n2. figuring how to define the relationship between segments from the same sequence, shall we treat them as positive pairs or negative pairs or neither;\n3. finding optimal truncation length;\n4. studying the proper weight between two losses \n5. exploring a variant of HiCL that adds entailment judgment between segments and sequences. \n\nWe believe all these efforts together make our technique contribution strong, as also recognized by other reviewers:\n1. Reviewer etFB: \"The paper presents a **pioneering** approach\"\n2. Reviewer 69hi: \"First to propose a hierarchical contrastive learning framework (which is a **bold claim** and **surprising** if true)\"\n\n*Q1. Meaning of local contrastive when input is short*\n\nWe agree with you that our method becomes particularly compelling when handling a majority of long inputs. As mentioned in Section 5.4, HiCL shows more improvements on a long training corpus (WikiText-103 with 3617 tokens per article).\nPrior studies [1,2,3,4,5] uniformly trained on a short-input dataset, thus, to ensure a fair comparison, we opted to utilize the same data for our primary evaluation. Importantly, the impressive performance on this short dataset happens to demonstrate the superiority of our method, given its adaptability even with fewer segments.\n\n*Q2. Issue with fixed truncation*\n\nThis is a good point, we actually have also observed the issue with fixed truncation size in Section 5.4 through the experiment that trains a solo local contrastive model. \nBetter segmentation could lead to a better performance, but will also come with other issues such as low encoding efficiency as it slices into more segments compared to fixed truncation strategy. \nFurther, traditional methods also introduce noise into training. Take your example “I like a banana but hate an apple” with truncation size 3 for illustration, traditional CL only contrasts using “I like a”, which can also mislead the model.\nOverall, even with our current fixed truncation strategy, we have already attained commendable results and plan to leave the exploration of dynamic truncation as our future work. \n\n*Q3. Suggestion on revised loss*\n\nThanks for your suggestion. Our current version contrasts a segment ($h_{i,j}$) with every other segment from different sequences ($h_{k, v}$ where $k!= i$), which is more like picking the correct pair out of a big batch. \nWe interpret you were suggesting to add a weighted average factor in computing $\\sum_{k!=i}(w_k \\times \\sum_{v}e^{sim(h_{i,j}, h_{k,v})/\\tau}$), where the weight $w_k$ can be defined as the inverse of #segments that sequence k is divided into, i.e., $w_k = \\frac{1}{\\sum_{v} 1}$. In essence, this would allocate varying weights to different segments in computation: segments derived from shorter sequences would receive a higher weight, while those from longer sequences would be assigned a lower weight. \n\nA critical query here is: **why should we differentiate the importance of segments based on their originating sequence length?** Our approach treats each segment as an independent contrastive objective, and we don't think that varying their importance based on source length would be beneficial. \n\n*Q4. Recommendation on more evaluation tasks*\n\nOur primary objective is to refine existing CL techniques, focusing on both encoding efficiency and performance. To ensure fairness and comparability, we've adhered to evaluation standards set by prior research. Esteemed publications like [1,2,3,4], along with the recent miCSE [5] from ACL '23, consistently employ the benchmark we've chosen.\nThanks for your recommendation, we are committed to broadening our evaluation scope in the upcoming revision, incorporating transfer tasks: MR, CR, SUBJ, MPQA, SST-2, TREC, and MRPC. \n\n-----\n[1] Gao, Tianyu, Xingcheng Yao, and Danqi Chen. \"Simcse: Simple contrastive learning of sentence embeddings.\" in EMNLP 2021\n\n[2] Wu, Xing, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. \"Esimcse: Enhanced sample building method for contrastive learning of unsupervised sentence embedding.\" in COLING 2022\n\n[3] Wang, Hao, and Yong Dou. \"SNCSE: contrastive learning for unsupervised sentence embedding with soft negative samples.\" In International Conference on Intelligent Computing, pp. 419-431. Singapore: Springer Nature Singapore, 2023.\n\n[4] Zhou, Kun, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. \"Debiased contrastive learning of unsupervised sentence representations.\" in ACL 2022\n\n[5] Klein, Tassilo, and Moin Nabi. \"miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings.\" in ACL 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper propose a new hierarchical contrastive learning framework that considers local segment-level relationships, unlike existing methods, and experimented it with seven semantic textual similarity data sets, showing better performance than existing methods such as SimCSE, ESimCSE, and SNCSE.\n- This paper presents that the encoding process of the proposed method can reduce training time rather than prior contrastive learning models. Compared to the existing methods, the experimental results show that the encoding time of the proposed method is reduced, and the time complexity is theoretically analyzed.", "reasons_to_reject": "- Although the basic idea of this paper can be understood, the paper needs to be partially updated for readability. It is not clear how the encoder in Figure 1 is added on BERT and RoBERTa. Is it correct to use the proposed hierarchical contrastive leaning as a loss function for BERT or RoBERTa model? What does the arrows in Figure 1 mean? It is ambiguous whether it simply means a positive and negative relationship, or whether it affects finding negative samples.\n- This reviewer is not sure what the research goal is in the paper. Is the final goal to do sentence embedding or to solve a specific natural language processing problem such as semantic textual similarity through the proposed hierarchical contrastive learning?\n- Excepting considering the segment-level relationship, it is not sure what else the technical contribution is in the proposed method.", "questions_to_authors": "- In p.228, the authors mentioned that most input instances can be divided into three or fewer segments. This means that the sequence length is usually not long. If most of the inputs are long sequences, it makes sense to divide the sequence to obtain a segment-level representation, but in other cases, it is difficult to expect a significant effect of using segment-level representation.\n- This paper is assuming that the length of segments in a sequence is the same. However, the reviewer thinks that each segment length should be variable according to its locality. For example, given a sentence like “I like a banana but hate an apple”, if we assume that the number of segments is 3 as the segment length, the segments are <I like a>, <banana but hate>, and <an apple>. In this case, each segment does not contain its meaning well. In this example, is it correct that the segments are <I like a banana> and <but hate an apple>?\n- In Equation 2, is the term of the denominator about negative segments (i.e., sigma_{k<>i} sigma_{v} e^{sim(h_{i,j}, h_{k,v})/tau}) correct? I think that averaging e^{sim(h_{i,j}, h_{k,v})/tau} is better than the summation.\n- To see the robustness of the proposed model, it is recommended to apply it to other natural language processing tasks as well as the semantic textual similarity task.", "paper_topic_and_main_contributions": "The objective of the study is to design a hierarchical contrastive learning framework by considering the local segment-level and global sequence-level relationships. Given a sentence, the proposed model called HiCL first divides it into smaller segments and encodes each segment to calculate local segment representation. Then, it gets the global sequence representation by aggregating the local segment representations. In addition, HiCL can reduce the training time by first encoding short segments and then aggregating them to obtain the sequence representation. For the contrastive learning, they use the dropout of SimCSE to get a positive pair and use the negated sentences and prompt tuning of SNCSE as negative examples. The experimental results show that HiCL outperforms existing contrastive learning methods such as SimCSE, ESimCSE, and SNCSE.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "Justification for Ethical Concerns\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve unsupervised sentence representations and efficiency by proposing a hierarchical contrasting learning method. Reviewers praised the proposed methods, the thorough experiments, and clarity in the paper. Through detailed back and forth discussions some reviewer complaints were addressed and their scores raised. However, reviewers still found the method's motivation to be lacking and the performance gains to be slight.
Decision: Accept-Findings