SlowGuess/NAACL_2022_OCR huggingface dataset:
Yung-Sung Chuang† Rumen Dangovski† Hongyin Luo† Yang Zhang‡ Shiyu Chang*  
Marin Soljačić† Shang-Wen Li‡ Wen-tau Yih‡ Yoon Kim† James Glass†
Massachusetts Institute of Technology† Meta AI°
MIT-IBM Watson AI Lab† UC Santa Barbara*
yungsgung@mit.edu
We propose DiffCSE, an unsupervised contrastive learning framework for learning sentence embeddings. DiffCSE learns sentence embeddings that are sensitive to the difference between the original sentence and an edited sentence, where the edited sentence is obtained by stochastically masking out the original sentence and then sampling from a masked language model. We show that DiffSCE is an instance of equivariant contrastive learning (Dangovski et al., 2021), which generalizes contrastive learning and learns representations that are insensitive to certain types of augmentations and sensitive to other "harmful" types of augmentations. Our experiments show that DiffCSE achieves state-of-the-art results among unsupervised sentence representation learning methods, outperforming unsupervised SimCSE<sup>1</sup> by 2.3 absolute points on semantic textual similarity tasks.<sup>2</sup>
Learning "universal" sentence representations that capture rich semantic information and are at the same time performant across a wide range of downstream NLP tasks without task-specific finetuning is an important open issue in the field (Conneau et al., 2017; Cer et al., 2018; Kiros et al., 2015; Logeswaran and Lee, 2018; Giorgi et al., 2020; Yan et al., 2021; Gao et al., 2021). Recent work has shown that finetuning pretrained language models with contrastive learning makes it possible to learn good sentence embeddings without any labeled data (Giorgi et al., 2020; Yan et al., 2021; Gao et al., 2021). Contrastive learning uses multiple augmentations on a single datum to construct positive pairs whose representations are trained to
be more similar to one another than negative pairs. While different data augmentations (random cropping, color jitter, rotations, etc.) have been found to be crucial for pretraining vision models (Chen et al., 2020), such augmentations have generally been unsuccessful when applied to contrastive learning of sentence embeddings. Indeed, Gao et al. (2021) find that constructing positive pairs via a simple dropout-based augmentation works much better than more complex augmentations such as word deletions or replacements based on synonyms or masked language models. This is perhaps unsurprising in hindsight; while the training objective in contrastive learning encourages representations to be invariant to augmentation transformations, direct augmentations on the input (e.g., deletion, replacement) often change the meaning of the sentence. That is, ideal sentence embeddings should not be invariant to such transformations.
We propose to learn sentence representations that are aware of, but not necessarily invariant to, such direct surface-level augmentations. This is an instance of equivariant contrastive learning (Dangovski et al., 2021), which improves vision representation learning by using a contrastive loss on insensitive image transformations (e.g., grayscale) and a prediction loss on sensitive image transformations (e.g., rotations). We operationalize equivariant contrastive learning on sentences by using dropout-based augmentation as the insensitive transformation (as in SimCSE (Gao et al., 2021)) and MLM-based word replacement as the sensitive transformation. This results in an additional cross-entropy loss based on the difference between the original and the transformed sentence.
We conduct experiments on 7 semantic textual similarity tasks (STS) and 7 transfer tasks from SentEval (Conneau and Kiela, 2018) and find that this difference-based learning greatly improves over standard contrastive learning. Our DiffCSE approach can achieve around  $2.3\%$  absolute improve
<sup>1</sup>SimCSE has two settings: unsupervised and supervised. In this paper, we focus on the unsupervised setting. Unless otherwise stated, in this paper we use SimCSE to refer to unsupervised SimCSE.
2Pretrained models and code are available at https:// github.com/voidism/DiffCSE.
Human Language Technologies, pages 4207 - 4218
Figure 1: Illustration of DiffCSE. On the left-hand side is a standard SimCSE model trained with regular contrastive loss on dropout transformations. On the right hand side is a conditional difference prediction model which takes the sentence vector  $\mathbf{h}$  as input and predict the difference between  $x$  and  $x''$ . During testing we discard the discriminator and only use  $\mathbf{h}$  as the sentence embedding.
ment on STS datasets over SimCSE, the previous state-of-the-art model. We also conduct a set of ablation studies to justify our designed architecture. Qualitative study and analysis are also included to look into the embedding space of DiffCSE.
2.1 Learning Sentence Embeddings
Learning universal sentence embeddings has been studied extensively in prior work, including unsupervised approaches such as Skip-Thought (Kiros et al., 2015), Quick-Thought (Logeswaran and Lee, 2018) and FastSent (Hill et al., 2016), or supervised methods such as InferSent (Conneau et al., 2017), Universal Sentence Encoder (Cer et al., 2018) and Sentence-BERT (Reimers and Gurevych, 2019). Recently, researchers have focused on (unsupervised) contrastive learning approaches such as SimCLR (Chen et al., 2020) to learn sentence embeddings. SimCLR (Chen et al., 2020) learns image representations by creating semantically close augmentations for the same images and then pulling these representations to be closer than representations of random negative examples. The same framework can be adapted to learning sentence embeddings by designing good augmentation methods for natural language. ConSERT (Yan et al., 2021) uses a combination of four data augmentation strategies: adversarial attack, token shuffling, cut-off, and dropout. DeCLUTR (Giorgi et al., 2020) uses overlapped spans as positive ex
amples and distant spans as negative examples for learning contrastive span representations. Finally, SimCSE (Gao et al., 2021) proposes an extremely simple augmentation strategy by just switching dropout masks. While simple, sentence embeddings learned in this manner have been shown to be better than other more complicated augmentation methods.
2.2 Equivariant Contrastive Learning
DiffCSE is inspired by a recent generalization of contrastive learning in computer vision (CV) called equivariant contrastive learning (Dangovski et al., 2021). We now explain how this CV technique can be adapted to natural language.
Understanding the role of input transformations is crucial for successful contrastive learning. Past empirical studies have revealed useful transformations for contrastive learning, such as random resized cropping and color jitter for computer vision (Chen et al., 2020) and dropout for NLP (Gao et al., 2021). Contrastive learning encourages representations to be insensitive to these transformations, i.e. the encoder is trained to be invariant to a set of manually chosen transformations. The above studies in CV and NLP have also revealed transformations that are harmful for contrastive learning. For example, Chen et al. (2020) showed that making the representations insensitive to rotations decreases the ImageNet linear probe accuracy, and Gao et al. (2021) showed that using an MLM to replace  $15\%$  of the words drastically reduces performance on
STS-B. While previous works simply omit these transformations from contrastive pre-training, here we argue that we should still make use of these transformations by learning representations that are sensitive (but not necessarily invariant) to such transformations.
The notion of (in)sensitivity can be captured by the more general property of equivariance in mathematics. Let  $T$  be a transformation from a group  $G$  and let  $T(x)$  denote the transformation of a sentence  $x$ . Equivalence is the property that there is an induced group transformation  $T'$  on the output features (Dangovski et al., 2021):
$$
f (T (x)) = T ^ {\prime} (f (x)).
$$
In the special case of contrastive learning,  $T'$ 's target is the identity transformation, and we say that  $f$  is trained to be "invariant to  $T$ ." However, invariance is just a trivial case of equivariance, and we can design training objectives where  $T'$  is not the identity for some transformations (such as MLM), while it is the identity for others (such as dropout). Dangovski et al. (2021) show that generalizing contrastive learning to equivariance in this way improves the semantic quality of features in CV, and here we show that the complementary nature of invariance and equivariance extends to the NLP domain. The key observation is that the encoder should be equivariant to MLM-based augmentation instead of being invariant. We can operationalize this by using a conditional discriminator that combines the sentence representation with an edited sentence, and then predicts the difference between the original and edited sentences. This is essentially a conditional version of the ELECTRA model (Clark et al., 2020), which makes the encoder equivariant to MLM by using a binary discriminator which detects whether a token is from the original sentence or from a generator. We hypothesize that conditioning the ELECTRA model with the representation from our sentence encoder is a useful objective for encouraging  $f$  to be "equivariant to MLM."
To the best of our knowledge, we are the first to observe and highlight the above parallel between CV and NLP. In particular, we show that equivariant contrastive learning extends beyond CV, and that it works for transformations even without algebraic structures, such as diff operations on sentences. Further, insofar as the canonical set of useful transformations is less established in NLP than is in CV, DiffCSE can serve as a diagnostic
tool for NLP researchers to discover useful transformations.
3 Difference-based Contrastive Learning
Our approach is straightforward and can be seen as combining the standard contrastive learning objective from SimCSE (Figure 1, left) with a difference prediction objective which conditions on the sentence embedding (Figure 1, right).
Given an unlabeled input sentence  $x$ , SimCSE creates a positive example  $x^{+}$  for it by applying different dropout masks. By using the BERTbase encoder  $f$ , we can obtain the sentence embedding  $\mathbf{h} = f(x)$  for  $x$  (see section 4 for how  $\mathbf{h}$  is obtained). The training objective for SimCSE is:
$$
\mathcal {L} _ {\mathrm {c o n t r a s t}} = - \log \frac {e ^ {\mathrm {s i m} \left(\mathbf {h} _ {i} , \mathbf {h} _ {i} ^ {+}\right) / \tau}}{\sum_ {j = 1} ^ {N} e ^ {\mathrm {s i m} \left(\mathbf {h} _ {i} , \mathbf {h} _ {j} ^ {+}\right) / \tau}},
$$
where  $N$  is the batch size for the input batch  $\{x_{i}\}_{i = 1}^{N}$  as we are using in-batch negative examples,  $\mathrm{sim}(\cdot ,\cdot)$  is the cosine similarity function, and  $\tau$  is a temperature hyperparameter.
On the right-hand side of Figure 1 is a conditional version of the difference prediction objective used in ELECTRA (Clark et al., 2020), which contains a generator and a discriminator. Given a sentence of length  $T$ ,  $x = [x_{(1)}, x_{(2)}, \ldots, x_{(T)}]$ , we first apply a random mask  $m = [m_{(1)}, m_{(2)}, \ldots, m_{(T)}]$ ,  $m_{(t)} \in [0,1]$  on  $x$  to obtain  $x' = m \cdot x$ . We use another pretrained MLM as the generator  $G$  to perform masked language modeling to recover randomly masked tokens in  $x'$  to obtain the edited sentence  $x'' = G(x')$ . Then, we use a discriminator  $D$  to perform the Replaced Token Detection (RTD) task. For each token in the sentence, the model needs to predict whether it has been replaced or not. The cross-entropy loss for a single sentence  $x$  is:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {R T D}} ^ {x} = \sum_ {t = 1} ^ {T} \left(- \mathbb {1} \left(x _ {(t)} ^ {\prime \prime} = x _ {(t)}\right) \log D \left(x ^ {\prime \prime}, \mathbf {h}, t\right) \right. \\ \left. - \mathbb {1} \left(x _ {(t)} ^ {\prime \prime} \neq x _ {(t)}\right) \log \left(1 - D \left(x ^ {\prime \prime}, \mathbf {h}, t\right)\right)\right) \\ \end{array}
$$
And the training objective for a batch is  $\mathcal{L}_{\mathrm{RTD}} = \sum_{i=1}^{N} \mathcal{L}_{\mathrm{RTD}}^{x_i}$ . Finally we optimize these two losses together with a weighting coefficient  $\lambda$ :
$$
\mathcal {L} = \mathcal {L} _ {\text {c o n t r a s t}} + \lambda \cdot \mathcal {L} _ {\text {R T D}}
$$
The difference between our model and ELECTRA is that our discriminator  $D$  is conditional, so it can
use the information of  $x$  compressed in a fixed-dimension vector  $\mathbf{h} = f(x)$ . The gradient of  $D$  can be backward-propagated into  $f$  through  $\mathbf{h}$ . By doing so,  $f$  will be encouraged to make  $\mathbf{h}$  informative enough to cover the full meaning of  $x$ , so that  $D$  can distinguish the tiny difference between  $x$  and  $x''$ . This approach essentially makes the conditional discriminator perform a "diff operation", hence the name DiffCSE.
When we train our DiffCSE model, we fix the generator  $G$ , and only the sentence encoder  $f$  and the discriminator  $D$  are optimized. After training, we discard  $D$  and only use  $f$  (which remains fixed) to extract sentence embeddings to evaluate on the downstream tasks.
In our experiment, we follow the setting of unsupervised SimCSE (Gao et al., 2021) and build our model based on their PyTorch implementation. We also use the checkpoints of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as the initialization of our sentence encoder  $f$ . We add an MLP layer with Batch Normalization (Ioffe and Szegedy, 2015) (BatchNorm) on top of the [CLS] representation as the sentence embedding. We will compare the model with/without BatchNorm in section 5. For the discriminator  $D$ , we use the same model as the sentence encoder  $f$  (BERT/RoBERTa). For the generator  $G$ , we use the smaller DistilBERT and DistilRoBERTa (Sanh et al., 2019) for efficiency. Note that the generator is fixed during training unlike the ELECTRA paper (Clark et al., 2020). We will compare the results of using different size model for the generator in section 5. More training details are shown in Appendix A.
For unsupervised pretraining, we use the same  $10^{6}$  randomly sampled sentences from English Wikipedia that are provided by the source code of SimCSE. We evaluate our model on 7 semantic textual similarity (STS) and 7 transfer tasks in SentEval. STS tasks include STS 2012-2016 (Agirre et al., 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). All the STS experiments are fully unsupervised, which means no STS training datasets
are used and all embeddings are fixed once they are trained. The transfer tasks are various sentence classification tasks, including MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005). In these transfer tasks, we will use a logistic regression classifier trained on top of the frozen sentence embeddings, following the standard setup (Connseau and Kiela, 2018).
Baselines We compare our model with many strong unsupervised baselines including SimCSE (Gao et al., 2021), IS-BERT (Zhang et al., 2020), CMLM (Yang et al., 2020), DeCLUTR (Giorgi et al., 2020), CT-BERT (Carlsson et al., 2021), SG-OPT (Kim et al., 2021) and some post-processing methods like BERT-flow (Li et al., 2020) and BERT-whitening (Su et al., 2021) along with some naive baselines like averaged GloVe embeddings (Pennington et al., 2014) and averaged first and last layer BERT embeddings.
Semantic Textual Similarity (STS) We show the results of STS tasks in Table 1 including  $\mathrm{BERT}_{\mathrm{base}}$  (upper part) and  $\mathrm{RoBERTa}_{\mathrm{base}}$  (lower part). We also reproduce the previous state-of-the-art SimCSE (Gao et al., 2021). DiffCSE-BERT<sub>base</sub> can significantly outperform SimCSE-BERT<sub>base</sub> and raise the averaged Spearman's correlation from  $76.25\%$  to  $78.49\%$ . For the RoBERTa model, DiffCSE-RoBERTa<sub>base</sub> can also improve upon SimCSE-RoBERTa<sub>base</sub> from  $76.57\%$  to  $77.80\%$ .
Transfer Tasks We show the results of transfer tasks in Table 2. Compared with SimCSE-BERTbase, DiffCSE-BERTbase can improve the averaged scores from  $85.56\%$  to  $86.86\%$ . When applying it to the RoBERTa model, DiffCSE-RoBERTa also improves upon SimCSE-RoBERTa from  $84.84\%$  to  $87.04\%$ . Note that the CMLM-BERTbase (Yang et al., 2020) can achieve even better performance than DiffCSE. However, they use 1TB of the training data from Common Crawl dumps while our model only use 115MB of the Wikipedia data for pretraining. We put their scores in Table 2 for reference. In SimCSE, the authors propose to use MLM as an auxiliary task for the sentence encoder to further boost the performance of transfer tasks. Compared with
3https://github.com/princeton-nlp/SimCSE 4https://github.com/facebookresearch/SENTEval
Model	STS12	STS13	STS14	STS15	STS16	STS-B	SICK-R	Avg.
GloVe embeddings (avg.)	55.14	70.66	59.73	68.25	63.66	58.02	53.76	61.32
BERTbase(first-last avg.)	39.70	59.38	49.67	66.03	66.19	53.87	62.06	56.70
BERTbase-flow	58.40	67.10	60.85	75.16	71.22	68.66	64.47	66.55
BERTbase-whitening	57.83	66.90	60.90	75.08	71.31	68.24	63.73	66.28
IS-BERTbase	56.77	69.24	61.21	75.23	70.16	69.21	64.25	66.58
CMLM-BERTbase(1TB data)	58.20	61.07	61.67	73.32	74.88	76.60	64.80	67.22
CT-BERTbase	61.63	76.80	68.47	77.50	76.48	74.31	69.19	72.05
SG-OPT-BERTbase†	66.84	80.13	71.23	81.56	77.17	77.23	68.16	74.62
SimCSE-BERTbase	68.40	82.41	74.38	80.91	78.56	76.85	72.23	76.25
* SimCSE-BERTbase(reproduce)	70.82	82.24	73.25	81.38	77.06	77.24	71.16	76.16
* DiffCSE-BERTbase	72.28	84.43	76.47	83.90	80.54	80.59	71.23	78.49
RoBERTa(base(first-last avg.)	40.88	58.74	49.07	65.63	61.48	58.55	61.63	56.57
RoBERTa(base-whitening	46.99	63.24	57.23	71.36	68.99	61.36	62.91	61.73
DeCLUTR-RoBERTa(base)	52.41	75.19	65.52	77.12	78.63	72.41	68.62	69.99
SimCSE-RoBERTa(base)	70.16	81.77	73.24	81.36	80.65	80.22	68.56	76.57
* SimCSE-RoBERTa(base(reproduce)	68.60	81.36	73.16	81.61	80.76	80.58	68.83	76.41
* DiffCSE-RoBERTa(base	70.05	83.43	75.49	82.81	82.12	82.38	71.19	78.21
Table 1: The performance on STS tasks (Spearman's correlation) for different sentence embedding models.  $\clubsuit$ : results from Reimers and Gurevych (2019);  $\heartsuit$ : results from Zhang et al. (2020);  $\diamond$ : results from Gao et al. (2021);  $\spadesuit$ : results from Yang et al. (2020);  $\dagger$ : results from Kim et al. (2021);  $*$ : results from our experiments.
the results of SimCSE with MLM, DiffCSE still can have a little improvement around  $0.2\%$ .
In the following sections, we perform an extensive series of ablation studies that support our model design. We use BERT<sub>base</sub> model to evaluate on the development set of STS-B and transfer tasks.
Removing Contrastive Loss In our model, both the contrastive loss and the RTD loss are crucial because they maintain what should be sensitive and what should be insensitive respectively. If we remove the RTD loss, the model becomes a SimCSE model; if we remove the contrastive loss, the performance of STS-B drops significantly by  $30\%$ , while the average score of transfer tasks also drops by  $2\%$  (see Table 3). This result shows that it is important to have insensitive and sensitive attributes that exist together in the representation space.
Next Sentence vs. Same Sentence Some methods for unsupervised sentence embeddings like Quick-Thoughts (Logeswaran and Lee, 2018) and CMLM (Yang et al., 2020) predict the next sentence as the training objective. We also experiment with a variant of DiffCSE by conditioning the ELECTRA loss based on the next sentence. Note that this kind of model is not doing a "diff operation" between two similar sentences, and is not an instance of equivariant contrastive learning. As shown in Table 3 (use next sent. for  $x'$ ), the score of STS-B decreases significantly compared to DiffCSE while transfer performance remains
similar. We also tried using the same sentence and the next sentence at the same time for conditioning the ELECTRA objective (use same+next sent. for  $x^{\prime}$ ), and did not observe improvements.
Other Conditional Pretraining Tasks Instead of a conditional binary difference prediction loss, we can also consider other conditional pretraining tasks such as a conditional MLM objective proposed by Yang et al. (2020), or corrective language modeling,[5] proposed by COCO-LM (Meng et al., 2021). We experiment with these objectives instead of the difference prediction objective in Table 3. We observe that conditional MLM on the same sentence does not improve the performance either on STS-B or transfer tasks compared with DiffCSE. Conditional MLM on the next sentence performs even worse for STS-B, but slightly better than using the same sentence on transfer tasks. Using both the same and the next sentence also does not improve the performance compared with DiffCSE. For the corrective LM objective, the performance of STS-B decreases significantly compared with DiffCSE.
Augmentation Methods: Insert/Delete/Replace In DiffCSE, we use MLM token replacement as the equivariant augmentation. It is possible to use other methods like random insertion or deletion instead of replacement. For insertion, we choose to
5This task is similar to ELECTRA. However, instead of a binary classifier for replaced token detection, corrective LM uses a vocabulary-size classifier with the copy mechanism to recover the replaced tokens.
Edit distance operators include insert, delete and replace.
Model	MR	CR	SUBJ	MPQA	SST	TREC	MRPC	Avg.
GloVe embeddings (avg.)*	77.25	78.30	91.17	87.85	80.18	83.00	72.87	81.52
Skip-thought	76.50	80.10	93.60	87.10	82.00	92.20	73.00	83.50
Avg. BERT embeddings*	78.66	86.25	94.37	88.66	84.40	92.80	69.54	84.94
BERT- [CLS] embedding*	78.68	84.85	94.21	88.23	84.13	91.40	71.13	84.66
IS-BERTbase	81.09	87.18	94.96	88.75	85.96	88.64	74.24	85.83
SimCSE-BERTbase	81.18	86.46	94.45	88.88	85.50	89.80	74.43	85.81
w/ MLM	82.92	87.23	95.71	88.73	86.81	87.01	78.07	86.64
* DiffCSE-BERTbase	82.69	87.23	95.23	89.28	86.60	90.40	76.58	86.86
CMLM-BERTbase(1TB data)	83.60	89.90	96.20	89.30	88.50	91.00	69.70	86.89
SimCSE-RoBERTa(base)	81.04	87.74	93.28	86.94	86.60	84.60	73.68	84.84
w/ MLM	83.37	87.76	95.05	87.16	89.02	90.80	75.13	86.90
* DiffCSE-RoBERTa(base)	82.82	88.61	94.32	87.71	88.63	90.40	76.81	87.04
Table 2: Transfer task results of different sentence embedding models (measured as accuracy).  $\clubsuit$ : results from Reimers and Gurevych (2019);  $\heartsuit$ : results from Zhang et al. (2020);  $\diamond$ : results from Gao et al. (2021).
STS-B	Avg. transfer
SimCSE	81.47	83.91
DiffCSE	84.56	85.95
w/o contrastive loss	54.48	83.46
use next sent. for x'	82.91	85.83
use same+next sent. for x'	83.41	85.82
Conditional MLM
for same sent.	83.08	84.43
for next sent.	75.82	85.68
for same+next sent.	82.88	84.82
Conditional Corrective LM	79.79	85.30
Table 3: Development set results of STS-B and transfer tasks for DiffCSE model variants, where we vary the objective and the use of same or next sentence.
Augmentation	STS-B	Avg. transfer
MLM 15%	84.48	85.95
randomly insert 15%	82.20	85.96
randomly delete 15%	82.59	85.97
combining all	82.80	85.92
Table 4: Development set results of STS-B and transfer tasks with different augmentation methods for learning equivariance.
STS-B	Avg. transfer
DiffCSE
w/ BatchNorm	84.56	85.95
w/o BatchNorm	83.23	85.24
SimCSE
w/ BatchNorm	82.22	85.66
w/o BatchNorm	81.47	83.91
Table 5: Development set results of STS-B and transfer tasks for DiffCSE and SimCSE with and without BatchNorm.
randomly insert mask tokens to the sentence, and then use a generator to convert mask tokens into real tokens. The number of inserted masked tokens is  $15\%$  of the sentence length. The task is to predict