SlowGuess/NAACL_2022_OCR huggingface dataset:
whether a token is an inserted token or the original token. For deletion, we randomly delete  $15\%$  tokens in the sentence, and the task is to predict for each token whether a token preceding it has been deleted or not. The results are shown in Table 4. We can see that using either insertion or deletion achieves a slightly worse STS-B performance than using MLM replacement. For transfer tasks, their results are similar. Finally, we find that combining all three augmentations in the training process does not improve the MLM replacement strategy.
Pooler Choice In SimCSE, the authors use the pooler in BERT's original implementation (one linear layer with tanh activation function) as the final layer to extract features for computing contrastive loss. In our implementation (see details in Appendix A), we find that it is better to use a two-layer pooler with Batch Normalization (BatchNorm) (Ioffe and Szegedy, 2015), which is commonly used in contrastive learning framework in computer vision (Chen et al., 2020; Grill et al., 2020; Chen and He, 2021; Hua et al., 2021). We show the ablation results in Table 5. We can observe that adding BatchNorm is beneficial for either DiffCSE or SimCSE to get better performance on STS-B and transfer tasks.
Size of the Generator In our DiffCSE model, the generator can be in different model size from  $\mathrm{BERT}_{\text {large }}$ ,  $\mathrm{BERT}_{\text {base }}$  (Devlin et al., 2019), DistilBERT  $^{\text {base }}$  (Sanh et al., 2019),  $\mathrm{BERT}_{\text {medium }}$ ,  $\mathrm{BERT}_{\text {small }}$ ,  $\mathrm{BERT}_{\text {mini }}$ ,  $\mathrm{BERT}_{\text {tiny }}$  (Turc et al., 2019). Their exact sizes are shown in Table 6 (L: number of layers, H: hidden dimension). Notice that although  $\mathrm{DistilBERT}_{\text {base }}$  has only half the number of layers of BERT, it can retain  $97\%$  of
STS-B	Avg. transfer
SimCSE	81.47	83.91
DiffCSE w/ generator:
BERTlarge (L=24, H=1024)	82.93	85.88
BERTbase (L=12, H=768)	83.63	85.85
DistilBERTbase (L=6, H=768)	84.56	85.95
BERTmedium (L=8, H=512)	82.25	85.80
BERTsmall (L=4, H=512)	82.64	85.66
BERTmini (L=4, H=256)	82.12	85.90
BERTtiny (L=2, H=128)	81.40	85.23
Table 6: Development set results of STS-B and transfer tasks with different generators.
Ratio	15%	20%	25%	30%	40%	50%
STS-B	84.48	84.04	84.49	84.56	84.48	83.91
BERT's performance due to knowledge distillation.
We show our results in Table 6, we can see the performance of transfer tasks does not change much with different generators. However, the score of STS-B decreases as we switch from BERT-medium to BERT-tiny. This finding is not the same as ELECTRA, which works best with generators 1/4-1/2 the size of the discriminator. Because our discriminator is conditional on sentence vectors, it will be easier for the discriminator to perform the RTD task. As a result, using stronger generators (BERT<sub>base</sub>, DistilBERT<sub>base</sub>) to increase the difficulty of RTD would help the discriminator learn better. However, when using a large model like BERT<sub>large</sub>, it may be a too-challenging task for the discriminator. In our experiment, using DistilBERT<sub>base</sub>, which has the ability close to but slightly worse than BERT<sub>base</sub>, gives us the best performance.
Masking Ratio In our conditional ELECTRA task, we can mask the original sentence in different ratios for the generator to produce MLM-based augmentations. A higher masking ratio will make more perturbations to the sentence. Our empirical result in Table 7 shows that the difference between difference masking ratios is small (in  $15\% -40\%$ ), and a masking ratio of around  $30\%$  can give us the best performance.
Coefficient  $\lambda$  In Section 3, we use the  $\lambda$  coefficient to weight the ELECTRA loss and then add it with contrastive loss. Because the contrastive learning objective is a relatively easier task, the scale of contrastive loss will be 100 to 1000 smaller than
Table 7: Development set results of STS-B under different masking ratio for augmentations.
λ	0	0.0001	0.0005	0.001
STS-B	82.22	83.90	84.40	84.24
λ	0.005	0.01	0.05	0.1
STS-B	84.56	83.44	84.11	83.66
Table 8: Development set results of STS-B under different  $\lambda$  .
ELECTRA loss. As a result, we need a smaller  $\lambda$  to balance these two loss terms. In the Table 8 we show the STS-B result under different  $\lambda$  values. Note that when  $\lambda$  goes to zero, the model becomes a SimCSE model. We find that using  $\lambda = 0.005$  can give us the best performance.
6.1 Qualitative Study
A very common application for sentence embeddings is the retrieval task. Here we show some retrieval examples to qualitatively explain why DiffCSE can perform better than SimCSE. In this study, we use the 2758 sentences from STS-B testing set as the corpus, and then use sentence query to retrieve the nearest neighbors in the sentence embedding space by computing cosine similarities. We show the retrieved top-3 examples in Table 9. The first query sentence is "you can do it, too." The SimCSE model retrieves a very similar sentence but has a slightly different meaning ("you can use it, too.") as the rank-1 answer. In contrast, DiffCSE can distinguish the tiny difference, so it retrieves the ground truth answer as the rank-1 answer. The second query sentence is "this is not a problem". SimCSE retrieves a sentence with opposite meaning but very similar wording, while DiffCSE can retrieve the correct answer with less similar wording. We also provide a third example where both SimCSE and DiffCSE fail to retrieve the correct answer for a query sentence using double negation.
6.2 Retrieval Task
Besides the qualitative study, we also show the quantitative result of the retrieval task. Here we also use all the 2758 sentences in the testing set of STS-B as the corpus. There are 97 positive pairs in this corpus (with 5 out of 5 semantic similarity scores from human annotation). For each positive pair, we use one sentence to retrieve the other one, and see whether the other sentence is in the top-1/5/10 ranking. The recall@1/5/10 of the retrieval task are shown in Table 10. We can observe that DiffCSE can outperform SimCSE for
SimCSE-BERTbase	DiffCSE-BERTbase
Query: you can do it, too.
1) you can use it, too.	1) yes, you can do it.
2) can you do it?	2) you can use it, too.
3) yes, you can do it.	3) can you do it?
Query: this is not a problem.
1) this is a big problem.	1) i don 't see why this could be a problem.
2) you have a problem.	2) i don 't see why that should be a problem.
3) i don 't see why that should be a problem.	3) this is a big problem.
Query: i think that is not a bad idea.
1) i do not think it's a good idea.	1) i do not think it's a good idea.
2) it's not a good idea.	2) it is not a good idea.
3) it is not a good idea.	3) but it is not a good idea.
Table 9: Retrieved top-3 examples by SimCSE and DiffCSE from STS-B test set.
Model/Recall	@1	@5	@10
SimCSE-BERTbase	77.84	92.78	95.88
DiffCSE-BERTbase	78.87	95.36	97.42
Figure 2: The distribution of cosine similarities from SimCSE/DiffCSE for STS-B test set. Along the y-axis are 5 groups of data splits based on human ratings. The x-axis is the cosine similarity.
recall@1/5/10, showing the effectiveness of using DiffCSE for the retrieval task.
6.3 Distribution of Sentence Embeddings
To look into the representation space of DiffCSE, we plot the cosine similarity distribution of sentence pairs from STS-B test set for both SimCSE and DiffCSE in Figure 2. We observe that both SimCSE and DiffCSE can assign cosine similarities consistent with human ratings. However, we also observe that under the same human rating, DiffCSE assigns slightly higher cosine similarities compared with SimCSE. This phenomenon
Table 10: The retrieval results for SimCSE and DiffCSE.
Model	Alignment	Uniformity	STS
Avg. BERTbase	0.172	-1.468	56.70
SimCSE-BERTbase	0.177	-2.313	76.16
DiffCSE-BERTbase	0.097	-1.438	78.49
Table 11: Alignment and Uniformity (Wang and Isola, 2020) measured on STS-B test set for SimCSE and DiffCSE. The smaller the number is better. We also show the averaged STS score in the right-most column.
may be caused by the fact that ELECTRA and other Transformer-based pretrained LMs have the problem of squeezing the representation space, as mentioned by Meng et al. (2021). As we use the sentence embeddings as the input of ELECTRA to perform conditional ELECTRA training, the sentence embedding will be inevitably squeezed to fit the input distribution of ELECTRA. We follow prior studies (Wang and Isola, 2020; Gao et al., 2021) to use uniformity and alignment (details in Appendix C) to measure the quality of representation space for DiffCSE and SimCSE in Table 11. Compared to averaged BERT embeddings, SimCSE has similar alignment (0.177 v.s. 0.172) but better uniformity (-2.313). In contrast, DiffCSE has similar uniformity as Avg. BERT (-1.438 v.s. -1.468) but much better alignment (0.097). It indicates that SimCSE and DiffCSE are optimizing the representation space in two different directions. And the improvement of DiffCSE may come from its better alignment.
In this paper, we present DiffCSE, a new unsupervised sentence embedding framework that is aware of, but not invariant to, MLM-based word replacement. Empirical results on semantic textual similarity tasks and transfer tasks both show the effectiveness of DiffCSE compared to current state-of-the-art sentence embedding methods. We also conduct extensive ablation studies to demonstrate the different modeling choices in DiffCSE. Qualitative study and the retrieval results also show that DiffCSE can produce a better embedding space for sentence retrieval. One limitation of our work is that we do not explore the supervised setting that uses human-labeled NLI datasets to further boost the performance. We leave this topic for future work. We believe that our work can provide researchers in the NLP community a new way to utilize augmentations for natural language and thus produce better sentence embeddings.
In this research, Yung-Sung and Hongyin were partially supported by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Fund (InnoHK).
Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 497-511, San Diego, California. Association for Computational Linguistics.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder for english. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Xinlei Chen and Kaiming He. 2021. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljačić. 2021. Equivariant contrastive learning. arXiv preprint arXiv:2111.00899.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821.
John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. 2020. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. 2020. Bootstrap your own latent - a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284. Curran Associates, Inc.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. 2021. On feature decorrelation in self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598-9608.
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR.
Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for bert sentence representations. arXiv preprint arXiv:2106.07345.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2015. Skip-thought vectors. pages 3294-3302.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9119-9130, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Lrec, pages 216-223. Reykjavik.
Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. 2021. Coco-lm: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473.
Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. arXiv preprint cs/0409058.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 115-124.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962.
Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207.
Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. *Language resources and evaluation*, 39(2):165-210.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021. Consent: A contrastive framework for self-supervised sentence representation transfer. arXiv preprint arXiv:2105.11741.
Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2020. Universal sentence representation learning with conditional masked language model. arXiv preprint arXiv:2012.14388.
Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020. An unsupervised sentence embedding method by mutual information maximization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1601-1610.
We use a single NVIDIA 2080Ti GPU for each experiment. The averaged running time for DiffCSE is 3-6 hours. We use grid-search of batch size  $\in$  {64,128} learning rate  $\in$  {2e-6, 3e-6, 5e-6, 7e-6, 1e-5} and masking ratio  $\in$  {0.15, 0.20, 0.30, 0.40} and  $\lambda \in$  {0.1, 0.05, 0.01, 0.005, 0.001}. The temperature  $\tau$  in SimCSE is set to 0.05 for all the experiments. During the training process, we save the checkpoint with the highest score on the STS-B development set. And then we use STS-B development set to find the best hyperparameters (listed in Table 12) for STS task; we use the averaged score of the development sets of 7 transfer tasks to find the best hyperparameters (listed in Table 13) for transfer tasks. All numbers in Table 1 and Table 2 are from a single run.
hyperparam	BERTbase	RoBERTa_base
learning rate	7e-6	1e-5
masking ratio	0.30	0.20
λ	0.005	0.005
training epochs	2	2
batch size	64	64
Table 12: The main hyperparameters in STS tasks.
hyperparam	BERTbase	RoBERTa_base
learning rate	2e-6	3e-6
masking ratio	0.15	0.15
λ	0.05	0.05
training epochs	2	2
batch size	64	128
Table 13: The main hyperparameters in transfer tasks.
Method	BERTbase	RoBERTabase
SimCSE	110M	125M
DiffCSE (train)	220M	250M
DiffCSE (test)	110M	125M
During testing, we follow SimCSE to discard the MLP projector and only use the [CLS] output to extract the sentence embeddings.
The numbers of model parameters for  $\mathrm{BERT}_{\mathrm{base}}$  and  $\mathrm{RoBERTa}_{\mathrm{base}}$  are listed in Table 14. Note that in training time DiffCSE needs two BERT models to work together (sentence encoder + discriminator), but in testing time we only need the sentence
Table 14: The number of parameters used in our models.
Method	STS-B	Avg. transfer
SimCSE	81.47	83.91
+ Additional positives
MLM 15%	73.59	83.33
random insert 15%	80.39	83.92
random delete 15%	78.58	81.80
+ Additional negatives
MLM 15%	83.02	84.49
random insert 15%	55.65	79.86
random delete 15%	55.13	82.56
+ Equivariance (Ours)
MLM 15%	84.48	85.95
randomly insert 15%	82.20	85.96
randomly delete 15%	82.59	85.97
Table 15: Development set results of STS-B and transfer tasks for using three types of augmentations (replace, insert, delete) in different ways.
encoder, so the model size is the same as the SimCSE model.
Projector with BatchNorm In Section 5, we mention that we use a projector with BatchNorm as the final layer of our model. Here we provided the PyTorch code for its structure:
B Using Augmentations as Positive/Negative Examples
In Section 5, we try to use different augmentations (e.g. insertion, deletion, replacement) for learning equivariance. In Table 15 we provide the results of using these augmentations as additional positive or negative examples along with the SimCSE training paradigm. We can observe that using these augmentations as additional positives only decreases the performance. The only method that can improve the performance a little bit is to use MLM  $15\%$  replaced examples as additional negative examples. Overall, none of these results can perform
better than our proposed method, e.g. using these augmentations to learn equivariance.
C Uniformity and Alignment
Wang and Isola (2020) propose to use two properties, alignment and uniformity, to measure the quality of representations. Given a distribution of positive pairs  $p_{\mathrm{pos}}$  and the distribution of the whole dataset  $p_{\mathrm{data}}$ , alignment computes the expected distance between normalized embeddings of the paired sentences:
$$
\ell_ {\text {a l i g n}} \triangleq \underset {(x, x ^ {+}) \sim p _ {\text {p o s}}} {\mathbb {E}} \left\| f (x) - f \left(x ^ {+}\right) \right\| ^ {2}.
$$
Uniformity measures how well the embeddings are uniformly distributed in the representation space:
$$
\ell_{\text{uniform}}\triangleq \log \underset {x,y^{i.i.d.}\sim p_{\text{data}}} {\mathbb{E}}e^{-2\| f(x) - f(y)\|^{2}}.
$$
The smaller the values of uniformity and alignment, the better the quality of the representation space is indicated.
D Source Code
We build our model using the PyTorch implementation of SimCSE $^{7}$  Gao et al. (2021), which is based on the HuggingFace's Transformers package. $^{8}$  We also upload our code $^{9}$  and pretrained models (links in README.md). Please follow the instructions in README.md to reproduce the results.
E Potential Risks
On the risk side, insofar as our method utilizes pretrained language models, it may inherit and propagate some of the biases present in such models. Besides that, we do not see any other potential risks in our paper.
$^{7}$ https://github.com/princeton-nlp/ SimCSE  
 $^{8}$ https://github.com/huggingface/ transformers  
 $^{9}$ https://github.com/voidism/DiffCSE