On the Dimensionality of Sentence Embeddings
Authors: Hongwei Wang
Hongming Zhang
Dong Yu
Venue: EMNLP 2023 Findings (2023)
Keywords: Sentence embedding
dimension reduction
TL;DR: We conduct a comprehensive study on the dimensionality of sentence embeddings, and propose a two-step training method to improve the performance of low-dimensional sentence embeddings.

Abstract: Learning sentence embeddings is a fundamental problem in natural language processing.
While existing research primarily focuses on enhancing the quality of sentence embeddings, the exploration of sentence embedding dimensions is limited.
Here we present a comprehensive and empirical analysis of the dimensionality of sentence embeddings.
First, we demonstrate that the optimal dimension of sentence embeddings is usually smaller than the default value.
Subsequently, to compress the dimension of sentence embeddings with minimum performance degradation, we identify two components contributing to the overall performance loss: the encoder's performance loss and the pooler's performance loss.
Therefore, we propose a two-step training method for sentence representation learning models, wherein the encoder and the pooler are optimized separately to mitigate the overall performance loss in low-dimension scenarios.
Experimental results on seven STS tasks and seven sentence classification tasks demonstrate that our method significantly improves the performance of low-dimensional sentence embeddings.

On the Dimensionality of Sentence Embeddings
Hongwei Wang, Hongming Zhang, Dong Yu Tencent Al Lab Seattle
(hongweiw, hongmzhang, dyu)@global. sencent.com
Abstract
d-dim customicable)
O-dim (768 1024)
Learning sentence embeddings fundamen- tal problem in natural language processing. While existing research primarily focuses on enhancing the quality of sentence embeddings, the exploration of sentence embedding dimen- sions is limited. Here we present a comprehen- sive and empirical analysis of the dimensional- ity of sentence embeddings. First, we demon- strate that the optimal dimension of sentence embeddings usually smaller than the default value. Subsequently, to compress the dimen- sion of sentence embeddings with minimum performance degradation, we identify two com- ponents contributing to the overall performance loss: the encoder's performance loss and the pooler's performance loss. Therefore, we pro- pose two-step training method for sentence representation learning models, wherein the en- coder and the pooler are optimized separately mitigate the overall performance loss low- dimension scenarios. Experimental results seven STS tasks and seven sentence classifi- cation tasks demonstrate that method sig- nificantly improves the performance of low- dimensional sentence embeddings.
old pooler
Pooler
Ti
T2
Sentence Encoder
Exus
E
E2
Input
[CL.]
Tok
Tok 2
Tok N
sentence
Figure 1: The proposed architecture of a sentence repre- sentation learning model. The dimension of the pooler's fully connected layer changed from D to D d, where D is the hidden state dimension (768 for base models and 1,024 for large models), and d is the cus- tomizable sentence embedding dimension. The remain- ing part of the model (sentence encoder) is unchanged.
2022). However, the exploration sentence em- bedding dimensions remains limited. These sen- tence representation learning models typically em- ploy the default dimension of the model's 's hidden states as the dimension of sentence embeddings (e.g., 768 for BERT -like models and 1,024 for BaRTTagee- models). Nonetheless, the dimen- sion plays a critical role in sentence embeddings, and many research questions regarding its impact on sentence embeddings remain unanswered. For instance, does the default dimension yield the best performance? Can the dimension of sentence em- beddings be reduced to mitigate time and memory burdens in practical applications? Furthermore. how
1 Introduction
Learning sentence representation is fundamental problem in natural language processing. Sentence embeddings represent sentences as fixed-length vectors, which can be used in various downstream tasks, such as semantic textual similarity (STS) (Agirre et al., 2012, 2013; Marelli et al., 2014), in- formation retrieval (Mitra et al., 2017; Karpukhin et al., 2020; Thakur et al., 2021), and sentiment analysis (Pang and Lee, 2005; Hu and Liu, 2004; Pang and Lee, 2004).
maintain the performance of sentence embeddings when their dimension is reduced?
In this paper, we aim to answer the above ques- tions through comprehensive and empirical study of the dimensionality of sentence embeddings. Un- like conventional post-processing dimension reduc- tion methods, we propose direct modification of the output dimension the pooler in sentence rep- resentation learning models, as illustrated Figure
Existing work usually focuses on improving the quality of sentence embeddings by introduc- ing novel model architectures or training strategies (Reimers and Gurevych, 2019; Liu et al., 2021; Gao et al., 2021; Chuang et al., 2022; Su et al.,

---

1. This approach enables to generate sentence embeddings of any desired dimension while im- posing minimal computational overhead. Subse- quently, we evaluate sentence embeddings with var- ious dimensions across various downstream tasks. The findings indicate that the optimal dimension for sentence embeddings tends to be smaller than the default value used in the literature.
The key contributions of this paper are:
We propose customizing the dimension of sen- tence embeddings by directly modifying the output dimension of the pooler.
We demonstrate that default dimension of sen- tence embeddings commonly used in litera- ture is usually suboptimal.
Our findings also indicate a significant decline in the performance of sentence embeddings when the dimension is reduced beyond the optimal value. Therefore, we investigate whether the model's per- formance can be sustained in these low-dimension scenarios, This allows us to utilize sentence embed- dings with even smaller dimensions in practical ap- plications to reduce time and memory overhead fur- ther. Interestingly, we find that the model's perfor- mance deterioration in low-dimension scenarios is not solely attributed the decrease of the pooler's output dimension, but also to the degradation in the quality of the sentence encoder's output. As result, the performance loss can be divided into two components: the loss caused by the encoder and the loss caused by the pooler. We then propose a two- step training algorithm to mitigate the two aspects of the performance loss separately. First, the encoder side, we replace the current "pool-trained" encoder with "welll-rained" one. To achieve this, we train multiple models with different pooler's output dimensions and select the best encoder to replace the current one. Next, on the pooler side, since the pooler and the new encoder have not been trained together, we can fine-tune the pooler on top of the new encoder. This involves training the pooler from its current state while keeping the new encoder frozen, ensuring their compatibility and improving overall performance.
We discover that the performance loss of low- dimensional sentence embeddings can be di- vided into the encoder's performance loss and the pooler's performance loss.
We propose two-step training method to duce the two parts of the performance loss separately.
Sentence Embedding Compressor
Existing sentence representation learning models usually set the dimension of output sentence em- beddings the dimension of hidden states D, i.e., D 768 for base models and D 024 for large models. However, it worth noting that the default dimension may not always be optimal. Traditional dimension reduction methods, such as Principle Component Analysis (PCA) (Abdi and Williams, 2010), Isomap (Tenenbaum et al., 2000), and Lo- cally Linear Embedding (LLE) (Roweis and Saul, 2000), not suitable for our purpose here duc to the following reasons: (1) Our objective is conduct a comprehensive study on the impact of dimension, whereas these methods can only reduce dimension rather than increase it; (2) These meth- ods typically require access to the entire evaluation set before executing the algorithms, which may not be feasible in practical scenarios like online or streaming settings; (3) Utilizing these methods as post-processing step will introduce extra computa- tional overhead, which, to some extent, contradicts our initial goal of dimension reduction.
We conduct experiments seven STS tasks and seven classification tasks. Our proposed training method consistently outperforms all baseline meth- ods across all tasks, for instance, 1.50% to 4.92% improvement over the best baseline method on clas- sification tasks. Remarkably, our method reduces the dimension of sentence embeddings from 768 to 128 with almost no performance loss (from 76.57% 76.46% on STS tasks). In addition, we validate the effectiveness of the two steps in our proposed method by showing that their average improvement is 1.79% and 1. 1.17% respectively when trained with SimCSE, and 13. 1.16% and 0.83% respectively when trained with Sentence-BER on the STS-B dataset.
We propose a straightforward and efficient ap- proach to modify the dimension of sentence em- beddings. As illustrated in the left half of Figure I, sentence representation learning model, such as BERT (Devlin et al., 2018 8) or RoBERTa (Liu et al., 2019), usually includes a pooler top of the final hidden state of the [CLS] token. This pooler consists of a fully connected layer and a non-linear activation function. Initially, the pooler's purpose is to condense information from the input sentence into a fxxed-izee representation without changing

---

the embedding's dimension. However, we can alter the output dimension of the fully connected layer in the pooler from the default D customizable value of d. As a result, the pooler now serves as a compressor for sentence embeddings. Unlike conventional dimension reduction techniques, our method can generate sentence embeddings of any dimension. Furthermore, it does not require prior access the entire evaluation set and has minimal impact on computational overhead.
Acc Spearman corr.
TREC STS-8
204s 1024
768 si2 256 128
Sentence embedding dim
Figure 2: The results of accuracy / Spearman's 's correla- tion of SimCSI-RoBERTagee on four different datasets. The sentence embedding dimension is varied from 2,048 to 4 (the default value is D 768).
3 The Impact of the Dimension of Sentence Embeddings
We conduct study to examine the impact of the dimension of sentence embeddings on the perfor- mance of various downstream tasks. We select RoBERTauuse (Liu et al., 2019) as the sentence rep- resentation learning model and made its output di- mension configurable. We utilize the unsupervised SimCSE (Gao et al., 2021) as the training method, which takes input sentence and predicts itself in a contrastive objective with dropout used as noise. Similar to SimCSE, we train the model on one million randomly sampled sentences from English Wikipedia, then apply the model to the following downstream tasks: (1) TREC, question classifica- tion dataset containing 500 labeled questions in the test set with 6 class labels; (2) STS-B, semantic textual similarity dataset containing 1,379 sentence pairs in the test with 5 similarity grades: (3) CR. a binary sentiment classification dataset containing 3.773 sentences; (4) MRPC, binary paraphrase detection dataset containing 1,726 sentence pairs in the test set.
sion of sentence embeddings varies across differ- ent downstream tasks. Specifically, the optimal dimensions for TREC, STS-B, CR, and MRPC are 768, 256, 256, and 16, respectively. One possible explanation for this variation is that downstream tasks exhibit different levels of difficulty, requir- ing varying amounts of information to be stored in embeddings to achieve the best performance. This observation motivates further exploring the di- mensionality of sentence embeddings, particularly to enhance model performance in low-dimension scenarios.
4
The Proposed Approach
4.1 Performance Loss Decomposition
According to the result presented in Figure 2, the performance of sentence embeddings on most tasks declines their dimension decreases. The primary reason for the performance loss is that sentence embeddings become short retain sufficient information for downstream tasks. Nevertheless, given that the entire model trained end-to-end, is intriguing to examine whether the encoder component is affected when the output dimension of the pooler decreases. Therefore, we denote the final hidden state of [CLS] as the "output of the encoder" and utilize it the sentence embedding for downstream tasks.
The results of the accuracy Spearman's 's correla- tion of SimCSE-RoBERTAS are presented in Figure 2. The sentence embed- ding dimension d ranges from 2,048 to with the default value being D 768. As the sentence embedding dimension increases from 768, the per- formance consistently remains stable across all four datasets. However, when the dimension decreases from 768, we observe distinct patterns in the per- formance curves: The performance on TREC (the red curve) continuously decreases, and the perfor- mance on STS-B B and CR (the yellow and the blue curves) initially remains stable, then drops sharply. Conversely, the performance on MRPC (the green curve) remains consistently stable throughout. can be concluded that the optimal' dimen-
on the four datasets
The results of using the encoder's output and the pooler's 's output as sentence embeddings on the STS- B dataset are presented in Figure 3. Interestingly, when the pooler's output dimension d decreases. the encoder's performance consistently declines for all four models, though the dimension of the encoder's output remains unchanged. This finding
generally he understood as the dimension that maintains the hest performance while heing as small as possible.
Althouugh there is no strict definition for "optimal", it can

---

1:7
1.5
no pocier
80
80
60.55
11.05
800-5
79..5
77.1
6722
45.8
42
42
pooldf
78.6
88
00.1
19.7
80.44
79.0
77-11
677-1
66.1
poolersp
75
79
79.3
19.1
80.0
75.0
76.5
66.7
45.0
22
surui the recede
oupuit excode
poolerase
19.0
15.0
29.6
28.3
78.4
06.9
45.7
oupulio podter
oupulie
Pooler
poole
78.-7
18.2
29.2
78.
75.00
65.7
47.0
Poolar's cuspui dim
Poslar's cusput
pooler
17.4
17.9
76.3
77.7
74.1
(a) BERTuse
(b) BERTuage
74.2
74.8
73.0
70.8
-30
outpot the
curpet recdde ourpuc poerr
encodenas
uncoo
encoterp
vicodahre
encotets
552
Pooter's
Poster's cutoutumi
(c) RoBERTatse
(d) RoBERTaange
Encoder
Figure 3: The results of using the output of the encoder (red curves) and output the pooler (blue curves) as sentence embeddings the STS-B dataset. The train- ing method is SimCSE and the sentence encoders are BERT base. BERT RoBERTague and RoBERTaagge respectively. Figure 3c illustrates that the performance loss can be divided into the performance loss of the encoder and the performance loss of the pooler.
Figure 4: The results of Spearman's 's correlation of all possible combinations of encoders and poolers on the STS B dataset. See Section 4.2 for details.
BinCSE-RoBERTane using different pooler out- put dimensions (ranging from 768 to 4). This results in a model consisting of encodera and poolerg- We then combine each possible encoder; and pooler,, and utilize the new model encoder, + pooler, to generate sentence embeddings.
suggests that the performance loss is not solely attributed the decrease of the pooler's output dimension but also to a deterioration in the quality the encoder's output.
In Figure 4, each cell in the heatmap represents the Spearman's correlation
combined model on the STS-B dataset. Replacing the encoder with a superior one can usually substantially enhance the model's overall performance. For instance, the initial performance of the end-to-end training model with d 16 (encoder pooler1e) is 65.8. but it can be further elevated to 72.6 by replacing encoder with encoder64-
Figure 3c illustrates that the performance loss can thus be divided into two components: perfor- mance loss caused by the encoder and performance loss caused by the pooler. This decomposition of performance Joss enables an in-depth understand- ing of the model's s behavior in low-dimensional scenarios. Furthermore, it provides valuable in- sights into strategies that can improve the model's performance when working with smaller sentence embedding dimensions: By separately addressing the performance loss of the encoder and the pooler, can effectively enhance the performance of the entire model and subsequently combine the two modules achieve better outcomes.
We thus propose method to reduce the perfor- mance loss of the encoder, which is illustrated in Figure 5a. Given the target dimension d, we first train a sentence representation learning model with the pooler's output dimension being d, which con- sists of encodera and poolerg. Meanwhile, we train multiple models with other pooler's output dimen- sions (e.g., 512, 256, ...). From these models, we select the dimension opt that yields the optimal per- formance for encoder, replace the original encodera with encoder to improve the overall performance.
4.2 Reducing Performance Loss of the Encoder
a validation set. Lastly,
Figure indicates that the encoder's performance declines noticeably the pooler's output dimen- sion d decreases. It is worth noting that the en- coder's architecture remains unchanged regardless of d. As a result, we can easily replace "pool- trained" encoder with "well-trained" one eval- wate the model's overall performance can be en- hanced. We conduct end-to-end training of the
4.3 Reducing Performance Loss of the Pooler
Unlike the encoder, replacing poolers with a dif- ferent pooler, not feasible since the output di- mension the pooler must be exactly the target dimension d. It is important to note that pooler, is

---

d dim
opt dim
d-dim
Algorithm 1: Two-Step Training Approach
Pooler
Pooler Apo
Pooler
Input:
An unsupervised training corpus T, validation dataset E. the target dimension d. base sentence representation learning model M with customizable output dimension:
Train
Train
Encoder,
Encoderape
Encoder Ap
Replace
Output: A well-trained sentence representation learning model with output dimension
Train model with
Train model with pooler's cutput dim optimat
Replace ancodor,
// Step 1:
Reducing the encoder's s loss Determine the set of candidate dimensions D:
(a) Reducing performance loss of the encoder
3 Train M with out dim = d on I and obtain pooler,;
di didi
d-dims
for o' E D do
Pooler
Fine-tune
NEw pooler
Train M with with out _ddi d' T and obtain encoder,:;
Encodorgg
Freere
Encodor
Evaluate encodery on E;
6
Select the best encoder from {enonder he and denote it as encoderspti
1. Freeze emooder,g oy: and train pooler,
2. pantery
(b) Reducing performance loss of the pooler
// Step 2: Reducing the pooler's loss
Concatenate encoderops and poolers; Fine-tune pooler, on T with encoder.pr frozen, and obtain new-pnolera;
Figure 5: Illustration of reducing performance loss of the encoder and the pooler for sentence representation learning model.
return encoderopt new-poolera
trained jointly with encoder, rather than the current encoderopt. which implies that the parameters of poolerg may not be optimal for encoderopt- There- fore, as illustrated in Figure 5b, we freeze the pa- rameters of encoderopt and only fine-tune pooler,, until the model achieves the optimal performance. Here, we would like to emphasize the following points: (1) The parameters of encoderopt should remain unchanged, as encoderopt is already the op- timal encoder. If encoderopt is fine-tuned together with poolera. we would revert to the initial end- to-end training scenario, which has been shown to yield suboptimal performance. (2) Poolera should not be trained from scratch with randomly initial- ized parameters but rather fine-tuned starting from its current parameters, as it provides an excellent starting point. Our experimental results also vali- date that fine-tuning from the current parameters outperforms training from randomly initialized pa- rameters.
resulting in complexity of |D|C. The time com- plexity of the encoder evaluation is negligible com- pared to the training process. In step 2, the encoder is frozen, and only the pooler undergoes training, Since the pooler is merely a fully connected layer, while the encoder is typically much more complex than the pooler, the fine-tuning time for the pooler is negligible compared to C. Therefore, the overall time complexity of Algorithm 1 is |D|C.
Our algorithm generally requires more running time when the candidate dimension set D is larger. However, a larger pool of 'D will also increase the probability that encoder performs better, thereby improving the final performance.
Experiments
5.1 Experimental Setup
We evaluate our proposed two-step training algo- rithm on two types of datasets:
STS datasets. We include seven STS datasets in our experiments: STS 2012-2016 (Agirre al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017), and SICK- Relatedness (Marelli et al., 2014). Each dataset consists of sentence pairs and their corresponding ground-truth similarity scores. We use Spearman's 's correlation to evaluate the predicted results of our method and all base- line methods on the test set.
4.4 A Two-Step Training Algorithm
Our proposed two-step training approach is out- lined in Algorithm 1. The algorithm consists of two steps. In the first step, the primary objective is to acquire pooler, (line 2) and the optimal en- coder encoder (line 6). Subsequently, the sec- ond step involves fine-tuning pooler, while keeping encoderopt frozen (line 8).
The time complexity analysis of Algorithm is as follows. We use C' to denote the time required for training the entire model M once. In step 1, we train the model M for a total of [D) + 1 times,
Sentence classification datasets. These in- cludes MR (Pang and Lee, 2005), CR (Hu

---

Pooler's output
768
512
64
32
Encodera + poolera
79.62
79.22
79.11
78.17
77.72
74.02
65.82
51.72
(end-to-end training)
Encoderaps + poolera
80.08
79.26
79.11
78.66
77.94
74.78
69.62
60.27
(After step 1)
+0.46
+0.04
+0.00
+0.49
+0.22
+0.76
+3.80
+8.55
Encoderape new-pooler, (After step 2)
80.49
80.25
79.32
79.93
78.03
75.20
71.71
64.15
+0.41
+0.99
+0.21
+1.27
+0.09
+0.42
+2.09
+3.88
Table 1:
The results of Spearman's 's correlation (in %6 of our proposed algorithm on the STS-B dataset using the contrastive loss in SimCSE as the training objective. base model RoBERTabse- (1) The first block is the results of end-to-end training. (2) The second block results from step of our proposed algorithm. The numbers in the second line are the absolute improvement over the first block. (3) The third block results from step 2 of our proposed algorithm. The numbers in the second line are the absolute improvement over the second block. opt 256 for this experiment according to the first row of Figure
Pooler's output dim d
768
512
256
128
64
32
16
8
Encodera poolers
70.12
69.92
63.80
60.12
56.51
52.84
49.49
39.29
(end-to-end training)
Encoderaye poolera (After step 1)
73.50
69.92
73.34
73.53
72.50
71.46
68.36
64.78
+3.38
+0.00
+9.54
+13.41
+15.99
+18.62
+18.87
+25.49
Encoderoga + new-pooler,
73.61
70.14
73.95
73.81
73.12
72.88
70.58
65,92
(After step 2)
+0.11
+0.22
+0.61
+0.28
+0.62
+1.42
+2.22
+1.14
Encodery (poolery used only in training)
73.47
73.83
66.66
63.32
61.69
57.45
56.42
47.37
Table 2: The results of Spearman's correlation (in %) of our proposed algorithm on the STS-B dataset using the softmax classification loss Sentence-BEES
training objective. The first three blocks are similar as Table 1. The last block is the result of using encoder, pooler, for end-to-end training but only encoder, for inference. The base model is RoBERTa opl 512 for this experiment according to the last block.
and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe al., 2005), SST (Socher et al., 2013), TREC (Voorhees and Tice, 2000), and MRPC (Dolan and Brockett, 2005). A logistic regression classifier trained on top of (frozen) sentence embeddings. Each dataset consists of sentences and their class labels. Accuracy is used as the evaluation metric. We follow default configurations from SentEval2.
method to the direct end-to-end training method using SimCSE (Gao al., 2021).
5.2 Result of the Proposed Approach
The results of Spearman's correlation for our pro- posed method on the STS-B dataset are presented in Tables and 2. We select RoBERTa base model for both experiments. Table presents the result of using the contrastive loss in SimCSE as the training objective (see Section 3 for train- ing details). Table 2 presents the result of using the softmax classification loss in Sentence-BERT as the training objective. Specifically, following Sentence-BERT, we use SNLI and MNLI datasets as the training data, For pair of premise and hy- pothesis in SNLI/MNLI denoted as u and we first calculate their sentence embeddings and v, and then concatenate u, v. and v. followed by 3-way softmax classifier. The pooling function cls. The batch size is 64. Other hyperparame- ters are the same as reported in the Sentence-Bel BERT paper.
the
We use three traditional dimension reduction methods as baseline methods, including Princi- ple Component Analysis (PCA), Isomap (Tenen- baum et al., 2000), and Locally Linear Embedding (LLE) (Roweis and Saul, 2000). PCA is a linear dimension reduction method, while Isomap and LLE are nonlinear. We use the embeddings of the first 2,000 sentences from the unsupervised English Wikipedia (Gao et al., 2021) as training data for these models. In addition, we also compare our
Phttps: / http//ahhwu.mcmeeeeeoooeeeeeeeee

---

Methods
STS12
STS13
STS14 STS15
STS16 STS-B SICK-R Avg.
d 768 (w/o dimension reduction)
70.16
81.77
73.24
81.36
80.65
80.22
68.56
76.57
d 128
PCA
63.79
76.83
67.84
76.99
74.93
74.73
62.22
71.33
Isomap
51.55
56.97
45.52
53.17
56.01
49.26
51.36
51.98
LLE
38.54
45.41
34.76
42.42
45.69
40.22
42.15
41.31
MAnCSSE-RoBRRTaan w/ end-to-end training
69.10
79.69
70.80
78.70
79.66
78.17
68.32
74.92
SimCSE RoBERTaus w/ two-step training
70.15
80.66
72.38
81.74
80.62
79.93
69.77
76.46
d 64
PCA
65.94 66.94
75.81
72
75.97
73.78
73.08
60.46
70.25
Isomap LLE
49.69
54.85
43.25
49.90
53.99
46.82
49.32
49.69
33.54
42.57
32.38
38.78
40.24
36.40
37.04
37.28
w/ end-to-end training
66.29
78.76
70.55
80.18
78.33
77.72
67.85
74.24
BonCSEE-RBEKTaaee w/ two-step training
68.73
80.34
71.63
79.90
79.61
78.03
68.62
75.27
d
PCA
65.04
72.92
64.14
73.16
71.31
69.15
58.08
67.69
Isomap LLE
46.36
50.89
40.23
45.92
51.00
44.61
47.00
46.57
32,33
35.37
24.99
30.81
35.84
32.37
33.09
32.11
SimCSE--RBERTTau. w/ end to-end training
63.33
77.71
66.67
76.06
76.23
74.02
67.22
71.61
MemcsE-RRBERRIE w/ two-step training
67.89
77.80
69.77
77.66
77.38
75.20
68.26
73.42
PCA
62.75
67.67
$9.97
68.13
67.23
63.46
55.53
63.53
Isomap LLE
42.44
44.79
34.57
42.42
45.63
40,33
43.43
41.94
28.55
33,95
23.66
29.67
34.13
30.82
32.01
30.40
BRCCSE--BBBRTaa....... end-to-end training
54.16
67.31
55.85
63.92
70.70
65.82
63.64
63.06
SimCSF. RoBERTaaee w/ two-step training
64.82
75.16
65.32
74.87
74.25
71.71
66.15
70.33
PCA
53.31
56.66
51.23
$9.86
60.52
52.58
49.85
54.86
Isomap LLE
38.54
39.11
30.79
39.32
41.48
35.94
37.98
37.59
30.01
33.85
22.86
33.06
37.88
28
33.50
31.40
MunCSE-RoBERTaae end-to-end training
50.92
51.26
43.27
$9.03
51.72
53.30
LimCSE-RoBERIEE w/ two-step training
60.89
65.25
59.01
65.86
66.84
64.15
59.61
63.09
Table 3: The results of Spearman's 's correlation (in %) on seven STS datasets. Each black corresponds to specific dimension of sentence embeddings. The highest numbers across all methods are highlighted.
In Tables and 2. the first block shows the results of encoder, + pooler,, representing the end-to-end training approach. In the second block, we present the results of encoderopt + pooler,, corresponding step 1 of our proposed algorithm. The third block results from encoderopt + new-poolera, correspond- ing to step 2 of our proposed algorithm. In addition, in Table 2, we also present the result of Encoder, as the last block, in which encoderg + pooler used in end-to-end training but only encoder, is used in inference. Note that opt 256 in Table 1 while opt 512 in Table 2. The absolute improvement achieved by step and step 2 is also presented.
that of step 2 for two primary reasons, First, step 2 faces a greater challenge in improving the model since step has already substantially enhanced its performance. Second, the encoder is typically more complicated than the pooler, which offers greater potential for step to improve the perfor- mance. Moreover, the improvement is particularly pronounced when the dimension d is smaller, as the model has more room for improvement in low- dimensional scenarios.
5.3 Comparison with Baseline Methods
The results of comparing with baseline methods on STS tasks and classification tasks are presented in Tables and 4, respectively. Each block corre- sponds to specific dimension of sentence embed- dings. We do show the results of d 512 and 256 because their results are quite close 768. evident that our method consistently
We observe that step and step 2 of our method both yield significant enhancement to the model's performance. The average absolute improvement achieved by step 1 and step is 1.79% and 1.17% respectively using SimCSE, and 13.16% and 0.83% respectively using Centence-BERY Notably, the improvement brought about by step surpasses
achieves the best performance across almost all

---

Methods
MR
SUBJ
MPQA
TREC
MRPC
Avg.
d
768 (w/o dimension reduction)
MAnCSE-RoBEKTagg
81.04
87.74
93.28
86.94
86.60
84.60
73.68
84.84
d = 128
PCA
72.76
79.93
84.33
79.03
76,93
68.00
62.13
74.73
Isomap
58.82
65.46
72.20
68.03
57.39
31,20
67.13
60.03
LLE
58.07
65.41
69.64
68.99
57.66
28.00
66.49
59.18
SimCSF-ROBERRIE w/ end-to-end training
77.52
84,40
89.99
81.86
82.26
75,20
72.64
80,55
SimCSS-RoBERTTE RoBERTabee w/ two-step training
79.00
84.1
91.20
84.35
83.69
80.00
73.91
82.32
d 64
PCA
70.69
78.29
80.46
76.28
73.69
56.80
61.20
71.06
Isomap LLE
58.06
64.85
70.41
67.61
58.98
32.60
67.36
59.98
55.90
63.81
64.75
68.98
55.52
24.80
66.49
57.18
SinCSE-RoBEKTaate w/ end-to-end training
72.82
79.31
84.65
77.82
76.55
65.40
73.28
75.69
SimCSSE-RBBERTaae w/ two-step training
76.52
83.66
90.24
80.90
81.93
75.60
75.42
80.61
d
PCA
66.02
74.02
76.44
71.63
71.22
48.20
64.28
67.40
Isomap LLE
57.96
66.67
71.15
68
58.05
34.00
67.07
60.51
52.98
63.79
64.01
68.77
53.21
22.60
66.49
55.98
SimCSE-RoBERTE RoBERTauss w/ end-to-end training
72.15
78.67
79.53
77.10
74.42
58.80
72.32
73.28
CunCSE-RoBERIIE w/ two-step training
73.68
78.97
86.49
76.30
75.56
69.60
73.91
76.36
PCA
62.21
72.19
74.84
70.71
40.80
65.72
64.65
Isomap LLE
53.69
65.59
68.60
68.83
58.81
34.20
66.03
59.39
52.86
63.76
62.18
68.77
20.20
66.61
BennSEE-RBBRTaab... w/ end-to-end training
65.07
70.86
80.65
74.19
68.31
54.00
70.39
BinCSS-RRBBRRaann w/ twa-step training
69.21
74.68
84.24
74.09
73.37
59.20
73.33
72.59
PCA
58.85
67.82
67.98
66.61
63.70
36.60
64.16
60.82
Isomap LLE
55.17
67.16
68.96
61.29
27.00
66.84
50.16
63.76
60.98
68.77
18.80
66.49
54.16
CHnCSE-RoBERLan end-lo-end training
66.50
69.96
77.62
72.26
70.18
69.97
67.24
SimCSE-RRBERTauus two-step training
66.55
71.07
78.74
73.38
71.83
Table 4: The results of accuracy (in %) on seven sentence classification datasets. Each block corresponds to specific dimension of sentence embeddings. The highest numbers across all methods are highlighted.
cases. For example, when d = 32, our method out- performs the best traditional dimension reduction method by 5.73% and 8.96% on average for STS tasks and classification tasks, respectively. Sim- ilar to Table the improvement becomes more significant when the dimension decreases.
for sentence representation learning. For exam- plc, SBERT (Reimers and Gurevych, 2019) uses siamese and triplet network structures derive mantically meaningful sentence embeddings that can be compared using cosine-similarity: DPR (Karpukhin al., 2020) uses embeddings for in- formation retrieval, which are learned from a small number questions and passages by a simple dual- encoder framework. SimCSE (Gao et al., 2021) takes an input sentence and predicts itself in a contrastive objective with dropout used as noise. Building upon SimCSE, DiffCSE (Chuang et al., 2022) and ESimCSE (Wu al., 2021) further hance the method by improving the sampling ap- proach. InstructOR (Su al., 2022) embeds every text together with instructions explaining the use case, which can generate text embeddings for differ- ent downstream tasks and domains without further training. However, these works overlook the study of how the dimension of sentence embeddings im-
is exciting to observe that our method ex- hibits minimal performance degradation when decreases from 768 to 128 (from 76.57% to 76.46% on STS tasks), indicating that sentence embeddings can be effectively compressed to just 1/6 of the original size with almost loss in performance. We also observe that, despite being a linear di- mension duction method, PCA consistently outper- forms the other two nonlinear dimension reduction methods.
6 Related Work
Sentence Representation Learning
Researchers have proposed numerous methods

---

pacts the model's performance. In contrast, our work focuses on enhancing the performance of sen- tence embeddings in low-dimensional scenarios. Our proposed training algorithm can be employed in conjunction with any Transformers-based lan- guage models and the aforementioned sentence representation learning methods.
Limitations
In this paper, we aim to thoroughly comprehend the dimensionality of sentence embeddings, focusing primarily on empirical and experimental aspects. However, note that there remain unanswered ques- tions concerning the dimension of sentence em- beddings, especially from theoretical perspective, which we leave as future work.
Dimension Reduction
Dimension reduction is technique that reduces the number of features in a dataset while preserving the essential information. For instance, PCA (Abdi and Williams, 2010) is a linear dimensionality reduc- tion technique that finds a new set of uncorrelated variables (principal components) by projecting the data onto a lower-dimensional subspace while max- imizing the variance. Isomap (Tenenbaum et al., 2000) is nonlinear dimensionality reduction algo- rithm that preserves the geodesic distances between data points, creating a low-dimensional embedding that captures the intrinsic structure of the data man- ifold. LLE (Roweis and Saul, 2000) nonlin- ear dimensionality reduction method that seeks to preserve local relationships between neighboring data points, constructing a lower-dimensional rep- resentation based on linear combinations of these neighbors. However, as discussed earlier, these traditional dimension reduction methods are not suitable for our task they require access to the entire evaluation set in advance and they introduce additional computation cost. Another related work is (Yin and Shen, 2018), which theoretically studies the optimal dimension of word embeddings.
Firstly, Figure 3 illustrates that reducing the out- put dimension of the pooler leads to worse perfor- mance of the encoder. One possible explanation is that when the dimension too small, sentence em- beddings are unable to capture all the information in sentences, resulting in an inadequate represen- tation of sentences. Consequently, the quality of the back-propagated signal from the pooler dimin- ishes, which hinders the effective training of the encoder. However, theoretical understanding of this phenomenon is currently lacking.
Secondly, as depicted in Figure 4, replacing the current encoder encoder, with a "well-trained" encoderopt improves the performance of poolera's 's output. It should be noted that encoder and poolera are not trained jointly, which implies that the output embedding space of encoder and the input embedding space of poolera are not aligned. This suggests that a simple concatena- tion of encoder and pooler, might not produce embeddings with physical meaning. However, ex- perimental results demonstrate the effectiveness of this substitution strategy. The exact reason behind the improvement remains unknown.
Lastly, an intriguing relationship exists between PCA and the pooler of language models. While PCA applies a linear transformation to sentence embeddings, the pooler applies linear transforma- tion followed by a nonlinear function (tanh in our model). Notably, we also experiment with remov- ing the nonlinear function from the pooler, and find that the model's 's performance did not significantly change. Therefore, the pooler can be considered as a rough approximation of a PCA layer, and we
7 Conclusion
This paper presents a comprehensive and empirical study on the dimensionality of sentence embed- dings. First, we propose customizing the dimen- sion of sentence embeddings by directly modify- ing the pooler's output dimension. Subsequently, we demonstrate that the default dimension (768 or 1,024) of sentence embeddings commonly used in the literature are usually suboptimal. To enhance the performance of low-dimensional sentence em- beddings, we decompose the performance loss into the encoder's loss and the pooler's I loss. We then in- troduce two-step training method that separately addresses the two parts of the performance loss. Ex- perimental results demonstrate that our proposed training method consistently enhances the perfor- mance of sentence embeddings with low dimen- sions across all tasks.
indeed discover that PCA is the most effective di- mension reduction approach among the baseline methods. Given that the linear transformation in PCA aims project data onto oow-dinessional space while maximizing the variance, it intrigu- ing to investigate how the pooler projects sentence embeddings and whether a theoretical connection exists between the linear transformation in PCA and the pooler.

---

References
bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04800
Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statisticx, 2(4):433-459
Bill Dolan and Chris Brockett. 2005. Automati-
cally constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Eneko Agiree Carmen Banea, Claire Cardie, Daniel Cer. Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio. Montse Maritxalar, Rada Mi- halcea, et al. 2015. Semeval-2011 task 2: Semantic textual similarity, english, spanish and pilot on inter- pretability. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015).
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint wxXiv:2104.08822
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
pages 252-263.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez- Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval@ COLING, pages 91-91.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.0490
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016. Semeval- 2016 task I: Semantic textual similarity, monolin- gual and cross-lingual evaluation. In SemEval-2016. 10th International Workshop on Semantic Evalua- tion; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016. p. 497-511. ACL (Association for Computational Linguistics).
Fangyu Liu, Ivan Vulić, Anna Korhonen, and Nigel Collier. 2021. Fast, effective, and supersuuee Transforming masked language models into univer- sal lexical and sentence encoders. arXiv preprint 1eXiv:2104.0802.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907:1197.111999
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Aorzalez-Agirre. 2012. Semeval-2012 task 6: A pilot semantic textual similarity. In * SEM 2012: The First Joint Conference on Lexical and Compu- tational Semantics-Volle Volume 1: Proceedings of the main conference and the shared task, and Volume Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 385- 393.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. 2014. A sick cure for the evaluation of composi- tional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LRRCC 14), pages 216- 223.
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. sem 2013 shared task: Semantic textual similarity. In Second joint conference on lexical and computational semantics (* SEM), volume 1: proceedings of the Main confer- ence and the shared task: semantic textual similarity, pages 32 43.
Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017. Learning to match using local and distributed repre- sentations of text for web search. In Proceedings of the 26th international conference on world wide web, pages 1291-1299.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task Semantic textual similarity- and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. arXiv preprint cs/0506075.
Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang- Wen Li. Wen-tau Yih. Yoon Kim, and James Glass. 2022. Diffese: Difference-based contrastive learning for sentence embeddings. arXiv preprint 2xXiv:2004.10298
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese nert-networks. arXiv prepriat arXiv:1908 10084.
Sam T Roweis and Lawrence K Saul. 2000. Nonlinear dimensionality reduction by locally linear embedding. science, 29055500))2223222222
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018, Bert: Pre-training of deep

---

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Chrissopperr D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Hongjin Su, Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. 2022. One embedder, any task: instruction-fnetuned text embeddings. arXiv preprint 2rX:0:2212.09741.
Joshua B Tenenhaum, Vin de Silva, and John Langford. 2000. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):231922335
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. Beir: heterogenous benchmark for zero-shot evalua- tion of information retrieval models. arXiv preprint arXiv:2104.0860
Ellen M Voorhees and Dawn M Tice. 2000. Building question answering test collection. In Proceedings of the 23rd anmual international ACM SIGIR confer- ence Research and development in information retrieval, pages 200-207.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions language. Language resources and evaluation, 39: 165-210..
Xing Wu, Chaochen Gao, Liangjun Zang. Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021. Esimese: Enhanced sample building method for contrastive learning of unsupervised sentence embedding. arxiv preprint AXX::2109004380
7i Yin and Yuanyuan Shen. 2018. On the dimensionality of word embedding. Advances in neural information processing systems, 31.


Reviews:
[{"review_id": "8NEMHEdNYT", "reviewer": "Reviewer_wHEz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your valuable review. Our response to your comments is as follows.\n\n----\n\n**Q1**: *The evaluation experiments lack of strong baseline comparisons.*\n\n**A1**: We have conducted a comprehensive survey on the literature, but we are not aware of any method specifically working on reducing the dimension of sentence embeddings. As far as we know, this is the first paper focusing on sentence embedding dimension. Therefore, we can only include some traditional dimension reduction methods as the baseline methods. But note that PCA is actually a strong method according to the results in Table 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study offers a thorough empirical examination of sentence embedding dimensionality.\n- The method and experiments are clearly presented.\n- The empirical outcomes, especially those illustrating the efficacy of the two-step training model for sentence representations, are notably robust. These results underscore the method's ability to enhance the efficacy of reduced-dimension sentence embeddings across varied tasks.\n- The paper is well-written and easy to understand.\n- The experiments exhibit a wide scope, spanning multiple tasks and datasets.\n- The ablation studies provide clarity on the significance of various hyperparameters.", "reasons_to_reject": "- The evaluation experiments lack of strong baseline comparisons.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors proposed a method to tailor the dimensionality of sentence embeddings by directly altering the output dimension of the pooler. They posited that commonly accepted default dimensions for sentence embeddings, as referenced in much of the literature, are not always optimal. In efforts to reduce the dimensionality of sentence embeddings without significant performance drops, they discerned two primary contributors to performance deterioration: losses from the encoder and the pooler. Their findings indicate that the diminished performance of low-dimensional sentence embeddings can be bifurcated into these two distinct losses. As a resolution, they introduce a two-step training strategy for sentence representation models, wherein the encoder and pooler are optimized independently, ensuring minimal performance degradation in reduced-dimension contexts.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LqpueH4JJI", "reviewer": "Reviewer_85p1", "summary": "", "questions": "The paper mentions that the default choice of dimension size is suboptimal. It would be beneficial for the reader to understand the history behind this default dimension size. What is the reasoning (from prior work) for choosing those default values?\n\nThe experiments in this paper uses SimCSE with the contrastive sentence prediction objective for all observations and conclusions. Are there experiments with other objectives, that lead to different conclusions?\n\nAre there any differences in the hyperparameters tuned across the various experiments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your valuable review. Our response to your comments is as follows.\n\n----\n\n**Q1**: *What is the reason (from prior work) for choosing those default values of dimension size?*\n\n**A1**: The reason of choosing 768 or 1024 as the default dimension size for sentence embeddings is simply because the hidden dimension and the output dimension of BERT-base / BERT-large is 768 / 1024. Previous work usually uses BERT-like models as their base model, so they follow this default setting. We will add the explanation in future versions.\n\n----\n\n**Q2**: *The experiments use SimCSE with the contrastive sentence prediction objective for all observations and conclusions. Are there experiments with other objectives, that lead to different conclusions?*\n\n**A2**: We followed your comment and conducted more experiments using training objectives of Sentence-BERT. Specifically, following S-BERT, we use RoBERTa-base as the base model and SNLI/MNLI datasets as the training data. For a pair of premise and hypothesis in SNLI/MNLI denoted as u and v, we first calculate their sentence embeddings $u$ and $v$, and then concatenate $u$, $v$, and $u-v$, followed by a 3-way softmax classifier. The pooling function is $cls$. The batch size is 64. Other hyperparameters are the same as reported in the S-BERT paper. The result of the softmax training objective is as follows:\n\n| Pooler’s output dimension $d$ | 768 | 512 | 256 | 128 | 64 | 32 | 16 | 8 |\n| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| Encoder$_d$ + pooler$_d$ (end-to-end training) | 70.12 | 69.92 | 63.80 | 60.12 | 56.51 | 52.84 | 49.49 | 39.29 |\n| Encoder$_d$ only | 73.47 | 73.83 | 66.66 | 63.32 | 61.69 | 57.45 | 56.42 | 47.37 |\n| Encoder$_{opt}$ + pooler$_d$ (after step 1) | 73.50 | 69.92 | 73.34 | 73.53 | 72.50 | 71.46 | 68.36 | 64.78 |\n| Encoder$_{opt}$ + new-pooler$_d$ (after step 2) | 73.61 | 70.14 | 73.95 | 73.81 | 73.12 | 72.88 | 70.58 | 65.92 |\n\nThe results above are similar to Table 1. According to the second row (Encoder$_d$ only), $opt$ is set to 512 for this experiment. The results show that our two-step training method is effective for S-BERT with softmax training objective as well.\n\n----\n\n**Q3**: *Are there any differences in the hyperparameters tuned across the various experiments?*\n\n**A3**: Our proposed method in Algorithm 1 does not have hyperparameters. For the base model (BERT or RoBERTa) and the training objectives (SimCSE or S-BERT), the hyperparameters are the same as the default values in their public implementation codes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper raises some interesting questions about architectural choices in BERT models, and presents some valuable observations through well-crafted experiments.\n\nBased on the observations about dimensionality before and after the pooling stages, the paper is able to draw conclusions that inform architectural choices for researchers working with these models. By running these experiments across a range of tasks, the paper is able to demonstrate the general applicability of the main messages.\n\n", "reasons_to_reject": "The conclusions of this work are limited to BERT, with a specific pre-training process and objective. As such, it would be interesting to learn\nwhether the learnings and conclusions hold across models trained with other pretraining objectives and other variations in pre-training methodology.", "questions_to_authors": "The paper mentions that the default choice of dimension size is suboptimal. It would be beneficial for the reader to understand the history behind this default dimension size. What is the reasoning (from prior work) for choosing those default values?\n\nThe experiments in this paper uses SimCSE with the contrastive sentence prediction objective for all observations and conclusions. Are there experiments with other objectives, that lead to different conclusions?\n\nAre there any differences in the hyperparameters tuned across the various experiments?", "paper_topic_and_main_contributions": "This paper studies the dimension size used for sentence embeddings in typical NLP applications. The main premise of this research is that by adjusting the dimensionality of sentence embeddings generated from BERT-like encoder models at different stages of the encoding, one could potentially (a) obtain better semantic representations (sentence embeddings) from the model and (b) reduce the model size with negligible tradeoff in quality.\n\nIn this work, the authors use several benchmark tasks to conduct experiments on BERT models, varying the sentence dimension in the pooling layer, and in the layers before the pooling layer in the model. Through the set of experiments, the paper makes several interesting observations about dimension size and impact on task performance as well as on model size. Several learnings and conclusions are presented that would be interesting to the community.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2, title: \"Sentence Embedding Comperssor\" -> \"Sentence Embedding Compressor\""}, "extra_text": {}}, {"review_id": "h6bOziP7fj", "reviewer": "Reviewer_s3jK", "summary": "", "questions": "Do you think the improvement of the results with lower embeddings is due to the fact that the training of the encoder and pooler at the same time (in high dimension) performs bad in general? \nOr do you think, it is because lower dimensional embeddings allow to ``condense the information''  (or maybe both?)\nWould there be an experimental way of trying to answer this question?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your valuable review. Our response to your comments is as follows.\n\n----\n\n**Q1**: *The mention and references to similar existing related work is almost absent.*\n\n**A1**: To our best knowledge, we are not aware of such work specifically discussing the impact of output embedding dimension of BERT-like language models. We are still actively surveying the literature and will add the related work in the future.\n\n----\n\n**Q2**: *It would be interesting to see the results for various types of tasks, such as named entity recognition, or language translation.*\n\n**A2**:  We followed your comment and conducted experiments on the IR task (the NER and MT task you mentioned usually do not involve sentence embeddings). The IR task aims to retrieve relevant documents for a given query. This task is a perfect downstream application of sentence embeddings, because the query and documents can be encoded into embeddings, and thus the documents can be sorted according to their similarity with the query embedding. Specifically, we randomly select 200 questions from the NQ dataset (https://ai.google.com/research/NaturalQuestions). For each question, we retrieve top 25 paragraphs from Wikipedia dump (https://github.com/google-research-datasets/Attributed-QA)  that has the highest bm25 score with the question. So the entire document candidate pool has 5k paragraphs. Note that each question in NQ dataset has a ground-truth answer. Therefore, for a given question, any paragraph that contains the ground-truth answer to this question is treated as relevant. The result of top-1 accuracy on NQ dataset for d=128 is as follows:\n\n| Dataset | NQ |\n| --- | --- |\n| RoBERTa-base (d=768, w/o dim reduction) | 40.4 |\n| PCA (d=128) | 36.2 |\n| Isomap (d=128) | 29.1 |\n| LLE (d=128) | 25.4 |\n| RoBERTa-base w/ end-to-end training (d=128) | 38.5 |\n| RoBERTa-base w/ two-step training (d=128) | __40.1__ |\n\nThe experiment shows similar result with Tables 2 and 3, which demonstrates that our training method works well for the IR task.\n\n----\n\n**Q3**: *Do you think the improvement of the results with lower embeddings is due to the fact that the training of the encoder and pooler at the same time (in high dimension) performs bad in general? Or do you think, it is because lower dimensional embeddings allow to ``condense the information'' (or maybe both?)*\n\n**A3**: The reason that training the encoder and pooler at the same time in high dimension performs bad lies in that the dimension is too high and sentence embeddings are overfitting the training data. A reasonably low dimension embedding can restrict the capacity of the model and alleviate the overfitting issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The finding are interesting (lower dimensions are better for the tasks considered)\n\n- The presentation of the article (motivation, experiments, discussion) is clear\n\n- The two step training method (separate the encoder part from the pooler part) is simple to implement, and easy to reproduce.", "reasons_to_reject": "- The mention and references to similar existing related work (on the impact of dimension of the embeddings for popular language models, and not only dimension reduction in general) is almost absent. If there is few or none, then it should be made more clearly by the authors. \n\n- Although I find the main message of the article interesting, the types of experiments are limited (the ones conducted are thoroughly conducted, but there is no other task than STS and text classification), this limits the impact of the study. It would have been very interesting to see the results for various types of tasks, such as named entity recognition, or language translation. I do not expect the authors to perform the experiments for every NLP task, but at least one or two other task would help greatly to convince the reader.\n", "questions_to_authors": "Do you think the improvement of the results with lower embeddings is due to the fact that the training of the encoder and pooler at the same time (in high dimension) performs bad in general? \nOr do you think, it is because lower dimensional embeddings allow to ``condense the information''  (or maybe both?)\nWould there be an experimental way of trying to answer this question?", "paper_topic_and_main_contributions": "This article presents a study of the impact of dimension of word and sentence embeddings in popular language, including transformers. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Title of section 2: Comperssor -> Compressor\n"}, "extra_text": {}}, {"review_id": "eIsryUstik", "reviewer": "Reviewer_5JDc", "summary": "", "questions": "1. The SimCSE paper found the last second layer performs better in STS tasks when doing unsupervised training. If so, the pooling will not affect the sentence embedding dimensions. Do you have similar finding and how it affects your experiments?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks so much for your valuable review. Our response to your comments is as follows.\n\n----\n\n**Q1**: *The idea is not interesting. It is widely known that adjusting the dimensions is more efficient and the performance may not downgrade much when the compression is not too aggressive.*\n\n**A1**: Our contribution in this paper is three-fold: (1) the default dimension of sentence embedding is not optimal and can thus be reduced (this is what you mentioned); (2) the performance loss of low-dimensional sentence embeddings can be divided into two parts; (3) we propose a two-step training method to reduce the two parts of the performance loss separately. What you mentioned is only the Section 3 of this paper, while most of our contribution is presented in Sections 4 and 5.\n\n----\n\n**Q2**: *The proposed method is closer to training tricks. It shows some phenomena but there is no solid explanation of why it happens or presentation of the author's insights towards it.*\n\n**A2**: First, training tricks are actually important. RoBERTa is also some training tricks for BERT but it is undoubtedly a highly influential language model.  \nSecond, as mentioned in the section of limitation, it is true that we have no theoretical explanation for our findings, but we are the first to discover the phenomenon of performance loss decomposition, and we have accordingly proposed a realistic training method. We are not expected to solve all the (theoretical) problems regarding this phenomenon, and we are not supposed to be punished for this.  \nThird, we actually have the insight towards the training trick. The reason of proposing the two-step training method is exactly because we found that the performance loss can be decomposed into the encoder’s loss and the pooler’s loss. This finding is critical for improving the performance of language models in the scenario of low-dimension sentence embedding.  \nFourth, we actually have provided our theoretical guess in the section of limitation, which is the connection between the pooler layer and PCA. Further investigation on this problem is beyond the scope of this work and this conference (Empirical Methods in NLP).\n\n----\n\n**Q3**: *The author shows multiple variants of embedding dimensions and optimal dimensions. It would be more meaningful if I can know how to choose the optimal dimension besides trying all combinations.*\n\n**A3**: First, you can speed up the procedure of searching the optimum in Figure 4 by skipping columns beyond “encoder-32” and rows beyond “pooler-32”, since the performance has significantly dropped when the dimension reduces to 32.  \nSecond, searching the optimum in Figure 4 is actually not that time-consuming, because you only need to concatenate the encoder and the pooler and then evaluate this combined model. This does not involve training the model (so it is different from the traditional grid search on hyperparameters) and is thus time-efficient.\n\n----\n\n**Q4**: *The SimCSE paper found the last second layer performs better in STS tasks when doing unsupervised training. If so, the pooling will not affect the sentence embedding dimensions. Do you have similar finding and how it affects your experiments?*\n\n**A4**: We do have similar findings. You can observe from Figure 4 that, the row of “no pooler” is better than the row of “pooler-768” in most cases. But this finding does not affect our experiment, because in our model architecture, the pooler layer is used to adjust the embedding dimension from 768 to the target dimension d, so the pooler layer is mandatory in our model. This is different from SimCSE where the shape of pooler layer is 768*768, so the pooler is optional for SimCSE depending on whether it can improve the model performance.\n\n----\n\n**Q5**: *Typo: Line 250 - Figure 3c should be Figure 3.*\n\n**A5**: This is not a typo. Please note that in Figure 3c we explicitly annotate  the encoder loss and the pooler loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow.\n\n2. Reasonable performance after sentence embedding compression (by dimensions).", "reasons_to_reject": "1. The idea is not actually less interesting. It is widely known that adjusting the dimensions be more efficient and the performance may not downgrade much when the compression is not too aggressive.\n\n2. The proposed method is closer to training tricks. It shows some phenomena but there is no solid explanation of why it happens or presentation of the author's insights towards it.\n\n3. The author shows multiple variants of embedding dimensions and optimal dimensions. It would be more meaningful if I can know how to choose the optimal dimension besides trying all combinations.", "questions_to_authors": "1. The SimCSE paper found the last second layer performs better in STS tasks when doing unsupervised training. If so, the pooling will not affect the sentence embedding dimensions. Do you have similar finding and how it affects your experiments?", "paper_topic_and_main_contributions": "Study the dimensionality of sentence embedding and propose a way to get lower dimensional sentence embedding without much loss of performance on downstream tasks.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 250 - Figure 3c should be Figure 3.", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper systematically presents an interesting observation: when training with different dimensionality of a pooler, performance of using directly with the embedding out of transformer (without pooler) will change correspondingly. Thus performance loss could be decomposed into two part, and author proposed a way to select a low dimensionality with better performance. It's an interesting observation but the significance is not prominent. 

Pros:

1. Interesting observation.

2. Present it well and easy to follow

Cons:

1. After running the proposed 2-stage "multiple-train + plugin replacement" algorithm, the only benefit is for pooler to be from D to d, which is less than 1% or even less computation in the whole model. And the performance is not improved a lot either. This leads to the question why we want to study the problem.

2. Results are not consistent on all models. Authors agreed that for RoBERTa-base, the effect is not very prominent, but on other models it matters more.

3. The paper only focused on the "loss decomposition" part, but didn't explain why the performance could actually be "improved". For example, the Figure 3a, using a d=32 pooler to co-train and use encoder output give about 5% accuracy improvement, which is much larger than most improvement in the experiments. This effect is not clearly explained.
Decision: Accept-Findings