Towards Robust and Efficient Contrastive Textual Representation Learning
Authors: Liqun Chen
Yizhe Zhang
Dianqi Li
Chenyang Tao
Dong Wang
Lawrence Carin
Venue:  (2021)

Abstract: There has been growing interest in representation learning for text data, based on theoretical arguments and empirical evidence. One important direction involves leveraging contrastive learning to improve learned representations. We propose an application of contrastive learning for intermediate textual feature pairs, to explicitly encourage the model to learn more distinguishable representations. To overcome the learner's degeneracy due to vanishing contrasting signals, we impose Wasserstein constraints on the critic via spectral regularization.
    Finally, to moderate such an objective from overly regularized training and to enhance learning efficiency, with theoretical justification, we further leverage an active negative-sample-selection procedure to only use high-quality contrast examples.  We evaluate the proposed method over a wide range of natural language processing applications, from the perspectives of both supervised and unsupervised learning. Empirical results show consistent improvement over baselines. 

Under review as a conference paper at ICLR 2021
TOWARDS ROBUST AND EFFICIENT CONTRASTIVE TEXTUAL REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
There has been growing interest in representation learning for text data, based theoretical arguments and empirical evidence. One important direction in- volves leveraging contrastive learning to improve learned representations. We propose application of contrastive learning for intermediate textual feature pairs, to explicitly encourage the model to learn more distinguishable representations. To overcome the learner's 's degeneracy due to vanishing contrasting signals, we impose Wasserstein constraints on the critic via spectral regularization. Finally, moderate such objective from overly regularized training and to enhance learning efficiency, with theoretical justification, we further leverage an active regative-sample-seleee
procedure only use high-quality contrast examples. We evaluate the proposed method over a wide range of natural language processing applications, from the perspectives of both supervised and unsupervised learning. Empirical results show consistent improvement over baselines.
INTRODUCTION
Representation learning is one of the pivotal topics in natural language processing (NLP), in both supervised and unsupervised settings. It has been widely recognized that some forms of "general representation" exist beyond specific applications (Oord et al. 2018). To extract such generalizable features, unsupervised representation models are generally pretrained on large-scale text corpora (e.g., BERT (Devlin et al. 2018 Liu et al. 2019 Clark et al. 2020 Lagler et al.) 2013 to avoid data bias. supervised learning, models are typically built on top of these pre-trained representations and further fine-tuned on downstream tasks. Representation learning greatly expedites model deployment and meeln yields performance gains.
There has been growing interest in exploiting contrastive learning (CL) techniques to refine context representations NLP (Mikolov et al., [0)3a|b). These techniques aim to avoid representation collapse for downstream tasks, i.e., getting similar output sentences with different input in conditional generation tasks (Dai & Lin 2017). Intuitively, these methods carefully engineer features from crafted ("negative") examples to contrast against the features from real ("positive") examples. A feature encoder can then enhance its representation power by characterizing input texts at a finer granularity. Efforts have been made to empirically investigate and theoretically understand the effectiveness of CL in NL., including noise contrastive estimation (NCE) of word embeddings (Mikolov et al. (2033)) and probabilistic machine translation (Vaswani al. 2013) with theoretical developments (Gutmann & Hyvärinen 2010). More recently, InfoNCE (Oord et al. 2018) further links the CL to the optimization
of mutual information, which inspired a series of practical followup works (Tian al. 2020 Hjelm
et al. 2019a He et al. 2020 Chen et al., 2020).
Despite the significant empirical success of CL, there are still many open challenges its application to NLP, including (i) the propagation of stable contrastive signals. An unregularized critic function in CL can suffer from unstable training and gradient vanishing issues, especially in NI.P tasks due the discrete nature of text. The inherent differences between positive and negative textual features make those examples easily distinguished, resulting in a weak learning signal in contrastive schemes (Arora al. 2019). (ii) Empirical evidence (Wu et al. (017) shows that it is crucial to compare each positive example with adequate negative examples. However, recent works suggest using abundant negative examples, which are not akin to the positive examples, which result in sub-optimal results and unstable training with additional computational overhead (Ozair et al. 2019: McAllester & Stratos 2020).

---

Under review as a conference paper at ICLR 2021
In this paper, we propose two methods to mitigate the above issues. In order to stabilize the training and enhance the model's 's generalization ability, we propose to use the Wasserstein dependency measure (Ozair et al. 2019) as substitute for the Kullback-Leibler (KL) measure in the vanilla CL objective. We further actively select K' high-quality negative samples to contrast with each positive sample under the current learned representations. These supply the training procedure with necessarily large and non-trivial contrastive samples, encouraging the representation network to generate more distinguishable features. Notably, our approach also significantly alleviates the computational burden of massive features compared with previous works (Tian et al. 2020: Hjelm al. 2019b).
Contributions: (i) We propose Tiesserstein-regulaizee critic to stabilize training in generic CL framework for learning better textual representations. (ii) We further employ an active negative- sample selection method to find high-quality contrastive samples, thus reducing the gradient noise and mitigating the computation concerns. (iii) We empirically verify the effectiveness of our approach under various NLP tasks, including variational text generation (Bowman al. 2016), natural language understanding tasks on GLUE with supervised and semi-supervised setups (Wang et al. 2018), and image-text retrieval (Lee et al. 2018).
BACKGROUND
2.1 NOISE CONTRASTIVE ESTIMATION
Our formulation is inspired by Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010). which was originally introduced for unnormalized density estimation, where the partition functions intractable. To estimate a parametric distribution p. which refer as our targel distribution,
NCE leverages not only the observed samples A = (a1, ag .... an, (positive samples), but also the samples drawn from reference distribution q. denoted as B (b1. bg. ..., bn,) (negative samples). Instead of estimating directly, the density ratio p/g q is estimated by training critic between samples from A and B.
Specifically, let Z = (21, 2ng Zny+nn) + denote the union of A and B. A binary class label C₁ is assigned each 2,, where C, if 2, and C, otherwise.
The label probability is therefore
p(z) P(C = 11=) p(z) + 7z())²
79(2) 0|z) p(z) + 7g(2)²
(1)
where 7 22 is a balancing hyperparameter accounting for the difference in number of samples between A and B.
In practice, we do not know the analytic form of therefore, a classifier g = [0, 1] to estimate p(C = 1|2) trained. To get an estimation of the critic function g, NCE maximizes the log likelihood of the data for a binary classification task:
ng
L(A, B)
Σ loglg(a:)]
log(1 g(b:)]
(2)
2.2 CONTRASTIVE TEXTUAL REPRESENTATION LEARNING AND ITS CHALLENGES
ln Let (w be the observed text instances. We are interested in finding vector representation u of the text RD i.e., via an encoder u = Enc(w), that can be repurposed for downstream tasks. A positive pair refers to paired instances a: = (u,, 0;) associated with w,, where we are mostly interested in learning v is a feature at different representation level. In unsupervised scenarios, can be the feature representation at the layer next to the input text un, In supervised scenarios, 9i can be either the feature representation layer immediately after w, immediately before the label y, that corresponds to the input w, We will use x ((u, v) to denote the joint distribution of the positive pairs, with (u) and (v) for the respective marginals.
Contrastive learning follows the principle of "learning by comparison." Specifically, one designs negative sample distribution r(u', o²), and attempts to distinguish samples from (u, v) and r(u', o') with a critic function uu, v). The heuristic is that, using samples from 7 as references (i.e. to contrast against), the learner is advantaged capture important properties that could have been
2

---

Under review as a conference paper at ICLR 2021
otherwise missed (Hjelm et al., 2019a Oord et al. 2018). A popular choice of 7 is the product of marginals, i.e. T + To u' 2 Tu u T
where (u, v) are independent of each other, so that b, (u/, v/)) To. Inputting the new a, and b, 2
we obtain the new CL loss:
LNCE = Eu [logg(u, v)] + y/EE w'nxa [log(1 - g(u', v'))].
(3)
Note that when g is trained to optimality g (u, v) = p(C 1|u, v) under no. it establishes lower bound of mutual information (MI) between u and v for the positive distribution TTan et al. 2020
Neyshabur et al. 2018):
MI Tuz: mp) = KL(T ()) 2 Eu (opp:nloogg 9 (u, v)] + log
(4)
However, there are three concerns regarding why directly applying equation 3 might not be good in practice for learning the contrastive representation of the input text un.
Robustness. The first issue concerns the MI's strong sensitivity to small differences in data samples (Ozair et al. 2019 Tschannen et al. 2020). By definition in equation 4 mutual information is KL. divergence. It is well known that the KL divergence is metric-aware divergence measure, which implies a minor difference in representation can induce drastic changes in the mutual information, as special case of KL. Consequently, the learned g could be numerically unstable (Ozair et al. 2019), which makes the learned representations less robust and does not generalize well to downstream tasks, especially when features come from text (Chen et al., 2018).
Weak/vanishing contrastive signal. With poor initialization or poor choice of negative samples, the MI will vanish as the π (u, v) and TT (x, become far apart, delivering a faint and non- smooth gradient for training. In extreme case, the support for πt (u, v) and nu (u))).())) do not overlap, and the MI and the gradient will vanish to zero (Arjovsky et al. (017)).
selection strategy. Learning MI is generally considered sample inefficient. This point can be corroborated from several perspectives, ranging from theoretical arguments to practical considerations. To confidently estimate a lower bound to the MI, one would need a sample size exponential to the mutual information (i.e., N 2 exp(Ix+ (2,, v))) (Ozair et al. 2019 McAllester & Stratos) 2018). Also, both theoretical prediction and empirical evidence suggest large ratio 7 is needed for good performance (Tian al. 2020 Hjelm al., 2019a), imposing potential computational concerns for large training datasets. On the other hand, some studies report large y can instead deteriorate model performance (Tschannen al. 2020 Arora al. 2019). Such large y is also believed to be problematic especially when a strong association is expected between u and v. In that case, the majority of negative samples are so different from positive samples that the comparisons do not lend effective learning signals, but instead randomly drift the training (Gutmann Hyvärinen] 2010).
3
METHOD
3.1
MODEL OVERVIEW
We consider two remedies to mitigate the three issues mentioned in Section 2.2 (i) Regarding the robustness and the gradient-vanisting issues, we switch from the MI-based NCE to Wasserstein- based NCE, by imposing Wasserstein constraint to the critic function g. The Wasserstein distance yields continuous discrepancy measure over two distributions, even when they have no overlapping support, and suffers less from the issue of numerical instability (Neyshabur et al.] 2018). (ii) Regarding the issue of negative samples, we propose an active regative-sample-seleee to dynamically select the most challenging negative examples on-the-fly. This strategy smooths the learning signal (Wu et al., 2017), effectively improving the CL, meanwhile significantly reducing the computational overhead.
strategy,
To this end, we propose RECLAIM (RElaxed Contrastive Learning with ActIve Memory selection) robust and efficient CL framework. Our learning framework is illustrated in Figure] Details are explained in the following sections.
3.2 WASSERSTEIN CONSTRAINED CRITIC
In previous work, the critic is usually chosen to be a naive feed-forward neural network (Tian et al., 2020). However, discussed in Section 2.2 such a choice of critic function typically leads to based objective, which suffers from instability and vanishing-gradicn issues. Inspired by (Ozair
al. 2019), replace the KI. divergence in equation 4 with Wasserstein distance. Specifically,
3

---

Under review as a conference paper at ICLR 2021
Memory banks
ANS
negutive pairs {ul, v) {u, 2
Moneenturm Update
Wasserstein critic
Loss
Input
positive pair (u, v)
Text
Back- Packk-roopggtion
Encoder
Figure 1:
Illustration of our RECLAIM learning framework. An active negative-sample selection (ANS) module is applied for selecting challenging examples. We also relax the critic with the Wasserstein constraint.
we ensure the 1-Liischitzz constraint the critic function y (Arjovsky et al. said to be L-Lipschitz if ff() - -(()] & L|l yll. i.e., the difference in function outputs is controlled by the discrepancy in the inputs.
2017 A function
Instead of using a gradient penalty as in (Gulrajani et al. 2017: Ozair et al. 2019). we employ the spectral normalization (SN) (Miyato et al., 2018). We the SN because is efficient and also stable. It provides a more strict I-Lipschitz constraint than the gradient penalty. Specifically, SN controls the Lipschitz constant of the critic function, by constraining each layer şi in g. Formally, can be formulated as [ 1p = sup, (gV(g(()) where 6(-) is the spectral norm, é.e., the largest singular value of the input. For an affine transformation, such as linear function gı(x) W(ar, its spectral norm is || = 5( WW) When the activation function aj has Lipschitz norm equal to 1 (such ReLU (Jarrett et al. 2009) and Leaky ReLU (Maas al. 2013)), ), we have the following inequality: I19a 92 ip I 19 Lip aaplllllln Lpp.. With this inequality, we obtain the following bound:
a = || o an g2 ...... gal S | ||an || Misplll2llaa
I|ga|li.p
(5)
11 lod Lips 8(W)).
Applying the spectral normalization operation to each weight W, using WSN = W W enforces 5 5(W sN) = so that the right hand side 5 above-bounded by 1. This imposes the 1-Lipschitz constraint to the critic function y. thus stabilizing the learning signal for training. In practice, the spectral normalization can be estimated efficiently using power iteration.
3.3 ACTIVE NEGATIVE-SAMPE SELECTION (ANS)
Following the discussion in Section stable training requires adequate and high-quality negative samples. However, involving too many negative samples far apart from their positive counterparts does not yield effective training, and wastes computational resources. Inspired by the Triplet loss in deep metric learning [Wu et al. 2017 Tschannen et al. 2020), we propose to actively select a relatively small set of negative examples that most challenging to the discriminator at the current step, thus enabling the model to effectively extract features to distinguish the positives from the negatives.
Specifically, we use two memory banks B,, and B,, which store all previously extracted features u and e from seen training instances, respectively. When processing with each new training instances, we actively select the top K nearest neighbors Unn C B, and Vm C B, for u and v via cosine distance. With the QuickSelect algorithm (Hoare 196I), we are able to identify the top-K negative samples with time complexity O(KN). Under this setup, 3aan be written as:
LANS Ex(u. Dog(glu
Σ V'EVan
(log(1 (o(uvo)))
Σ
(log(1 g(u'vv)))) w)))
(6)
M M'EU..
When the dataset is large, this still cost much time in feature searching. Asymmetrie Locality Sensitivity Hashing (ALSH) (Shrivastava & Li| (2044) can be applied to hash the representations in mroximity-relevan manner. This helps os to reduce the time complexity of ANS sub-linear (Shrivas tava Li 2014). (Schroff et al. 2015) it was found empirically that relaxing the most challenging negative samples to semi-hard negative samples sometimes leads to better results supervised tasks like classification. This indicates that an approximate retrieval method like ALSH can still perform well. Our observations in experiments are consistent with these findings.
4

---

Under review as a conference paper at ICLR 2021
3.4 RECLAIM LEARNING PROCEDURE
Momentum update Before training, the two memory banks Bu, B, are initialized with Gaussian noise: {u' (0, E B.): (o' ~ (N( VV' E B.). Naively, one can directly replace old features in memory banks with new processed features corresponding to the same input data. However, the performance of such solution suboptimal model may change rapidly during the early training stages; such a practice reduces the consistency of the representations, and results in noisy learning signals (He et al. 2020; Wu et al. 2018).
practice. This is because the target
Therefore, we apply the momentum update mitigate such an issue. Specifically, assume a seen input instance æ reappears. There would be a snapshot feature pair previously stored in the Bu, B, of this æ, denoted as {ū, 8}. We further denote the newly computed (with the current encoder) feature be the momentum update parameter. We update the
pair correspond to x as fu, v), and let p E (0, 1] feature pairs in Bu, B,
u ύ= p)u + pi,
h p)v + pd
Note that the features in the memory bank are only updated in this way, and they are detached from the computational graph.
Joint Objective Optimization features for both u and v, denoted (u() and {e/) respectively. To get our CL loss, we apply the Wasserstein-distanee based NCE loss by simply imposing the I-Lipschitz constraint on critic in (6)
With our ANS module, we can obtain the top- K nearest negative
Assuming the task-specific loss is Lusk, the total loss therefore is formulated as
Liorll = Lusk AANSS
(8)
where À >> 0 is the hyper- parameter controlling the importance of the NCE loss. By minimizing equation 8 we can improve the training model performance. The detailed training procedure listed
in Algorithm
4 RELATED WORK
Connection to Mutual Information Estimation (MIE): MIE represents a family of MI-based representation learning methods. Early works on MIE, such as InfoMax (Linsker) 1988 Bell &
Sejnowski, 1995), advocate maximizing the MI between the input and output. MINE (Belghazi
al. 2018) uses bi-directional generative model to estimate the MI via the Donsker- Varadhan Varsdhn representation theorem. Deep InfoMax (Hjelm et al., 2019a) further developed MINE by removing one generator, seeking to maximize MI between local features and global features. Our method is more related to the InfoNCE and NCE objectives introduced in (Oord et al. 2018 Gutmann & Hyvärinen 2010), which are used to estimate the lower bound of the MI. In the proposed RECLAIM, Wasserstein distance is employed to estimate the distribution gap rather than KL-divergence, because it is more stable and robust in practice.
Contrastive Learning in NLP: Contrastive learning has been widely used in NLP (Smith & Eisner 20055 Collobert et al., 2011 Bordes et al.) [003] Hjelm et al.) [019a] Deng et al., 2020).
Broadly, CL methods attempt to differentiate observed data from artificial negative examples, to learn representations. In (Gutmann & Hyvärinen] 2010), the Noise Contrastive Estimation (NCE) metric is leveraged to differentiate the target sample from noise samples, learn word embeddings. Negative sampling proposed by Mikolov et al. (2013b) is a simplified variation of NCE loss, and it achieves great success in learning embeddings. Our CL approach is different from the above, since (i) we incorporate Wasserstein distance in NCE, and (éi) the proposed different from all other NCE-related works.
weweeeesallllllleeeeeeeeeeeeeeee strategy is
Connection to deep metric learning: Triplet loss is a classic approach in deep metric learning, and has already been widely used in retrieval models (Lee al. 2018) and recommendation sys- tems (Chechik et al.] 2010). It minimizes the distance between an anchor point to positive input, meanwhile maximizing its distance to a negative input. Motivated by the triplet loss, some works enforce constraints more than one negative examples. For example, PDDM (Huang et al. 2016) and Histogram Loss (Ustinova & Lempitsky 2016) use guadruplets. Moreover, the n-pair loss (Sohn 2016) and Lifted Structure (Oh Song et al. 2016) define constraints all images in batch. In these previous works, researchers have focused on enlarging the batch size, since they are only sampling the negative examples within the batch. Our approach incorporates the advantages of those works,
5

---

Under review as a conference paper at ICLR 2021
Model
CoLAA
SST-2
MRPC
MNLI
WNLI
STS-B
RTE
QQP
QNLI
BERT-base
55.6
91.2
89.47/85.22
83.60/84.15
50.7
89.2/88.8
60.0
90.2/87.0
90.2
BERT-RECLAIM
58.8
92.7
91.0/87.3
84,14/84.41
56.33
90.1/89.5
65.0
91. 0/.7887.
90.4
RoBERTa-base
61.3
94.1
91.1/87.7 93.0/9.2
87.4/87.2
53.6
90.5/90.3
70.8
91.3/88.4
92.4
KOBERTE-RECLIM
63.8
94.4
87.7/87.6
56.33
90.7/90.6
74.1
91.7/88.9
92.9
Table 1: Supervised results on GLUE dataset. Darker color indicates more improvement. Better in color.
and moves beyond them to allow active sampling of the most challenging negative pairs from all seen instances within the memory banks. Such global sampling strategy ensures the selected negative pairs for the same positive pair be more consistent during training, so that the training can be more stable.
5 EXPERIMENTS
Three experiments conducted to evaluate our approach: (1) supervised and semi-supervine natural language understanding (NLU) tasks on the GLUE dataset via BERT (Devlin et al.) 2018)) and RoBERTa (Liu et al. 2019); (it) image-text cross-domain retrieval task, and (122) text generation task with the VAE framework (Bowman et al. 2016 We set À = 0.1 in across all experiment, which are tested two NVIDIA TITAN
TITAIA X GPUs.
5.1 GLUE DATASET CLASSIFICATION
Dataset: The General Language Understanding Evaluation (GLUU) benchmark (Wang et al. 2018) is a collection of 9 datasets for evaluating natural language understanding models. Six of these tasks are either single sentence classification or paired sentence classification tasks.
Implementation details: We develop our approach on the open-sourced Hugging face Transformers codebase Wolf et al., 2 The hyper-parameters settings, i.e., learning rate, batch size, epoch numbers, etc., are all set to the default setup recommended by the official Transformer repository, for fair comparisons and reproducibility. We report results on the development sets after fine-tuning the pre-trained models (BERT (Devlin et al. 2018) and RoBERTa (Liu et al. 2019)) on each downstream task training data (no ensemble model or multi-task training is used). We utilize the 12-layer architecture (base model) for both BERT and RoBERTa, due limited computational resources. Parameters u, v are set to be the classification token embedding (CLS) and the embedded input text representation in BERT/RoBERTa, respectively.
We present comparisons on fully supervised tasks in Table I with K = 100 for RECLAIM. According to this table, RECLAIM consistently improves the performance on all GLUE datasets. For the semi-supervised experiments, we randomly draw (1%, 5%, 10%, 20%, 30%, 50% from each training dataset in GLUE, and results are provided in Table Note that in this case, we only use 10p-10 (K- = 10) negative examples, since the training dataset size is largely reduced. It can be seen from Table 2 that our approach generally achieves better results than just fine-tuning BERT on a small dataset.
We note that both fully-supervised and semi-supervised tasks on GLUE do not only serve as classic natural language inference (NLI) problems, but also serve a testbed for the generalization ability of a representation learner to new tasks. Presumably, results in Tables]] and suggest that the large-scale pre-trained BERT knowledge has been better transferred each task in GLUE in CL. approach, as improving the Wasserstein dependency measure (Ozir et al. 2019) can be seen as encouraging the model be "lossless" possible.
5.2 TEXT-IMAGE RETRIEVAL.
Dataset: We evaluate the proposed RECLAIM on both the Flickr30K (Plummer et al.) 2015) and MS-COCO (Lin al. 2014b) datasets. Flickr30K contains 31,000 images, and each photo has five captions. The Flickr30K dataset split following the same setup as (Karpathy & Fei-Fei 2015
version 2.5.1 using Pytorch 1.2.0 (Paskke
2017)

---

Under review as a conference paper at ICLR 2021
dataset
5%
10%
20%
30%
50%
CoLA-BERT
3.25
16.01
33.99
40.99
45.82
50.64
CoLA-RECLAIM
15.11
30.78
36.80
43.74
47.23
53.64
SST-BERT
68.53
89.15
89.80
90.32
91.25
91.80
SST-RECLAIM
83.28
89.25
90.41
91.79
91.92
STSB-BERT
56.57/43.87
56.66/ 56.66/52.27
22
74.64/74.47
83.20/83. 00//8336
85. 85.22/85.44
87. 74.34/87777
STSB-RECLAIM
56.78/53.42
/4.11/
79.34/79.36
03.48/83.25
85.33/85.60
03//85.00
97.73/87.95
MNLI-BET
48.40
73.67
76.46
79.56
80.56
81.99
MNLI-RECLAIM 54.22
73.89
77.07
79.63
80.65
82.59
Table 2: Semi- Semi-supervised results GLUE dataset (CoLA, SST-2, STS-B, MNLI). Results for QNLI, MRPC, and QQP in the Supplementary Material (SM). Darker color indicates more improvement. Better in color.
Faghri et al., 2018). We have a training dataset with 29,000 images, validation set with 1,000 images, and a test dataset with 1,000 images. MS-COCO contains 123,287 images, with human annotated descriptions per image. We use the same split as in (Faghri et al. 2018), í.e., the dataset is split into 113,287 training images, 5,000 validation images, and 5,000 test images.
Implementation details. For this image-text matching task, we use the Adam optimizer (Kingma Ba 2015) to train the models. Note that we developed our approach upon SCAN (Lee al. 2018 by substituting the triplet loss with our RECLAIM. In this experiment, u is the textual feature extracted from a GRU, and u is the image feature extracted from a ResNet. Training details are provided in the Appendix.
The performance of sentence retrieval with image query or image retrieval with sentence query is measured by recall at T (R@T)) (Karpathy& & Fei-Feil 2015), defined as the percentage of queries that retrieve the correct objects within those with top highest similarity scores as determined by the model. In this experiment, we can further improve the image-text retrieval results with the basic model design of SCAN. The improvement indicates that features extracted by our CL approach can better capture the common salient information between text and image pairs.
Sentence Retrieval
Image Retrieval
Method
R@I
R@5
R@ 10
R@D
R@S
R@ 10
Rsum
DPC (PC(ResNNel Zheng et al. 2017)
55.6
81.9
89.5
39.1
69.2
80.9
416.2
SCO (ResNet) Huang 2018
55.5
82.0
89.3
41.1
70.5
80.1
418.5
SCAN (Faster R-CNN, ResNet) Lee 2018
67.7
88.9
94.0
44.0
74.2
82.6
452.2
Vanilla CL
68.2
89.3
94.7
47.4
75.2
82.9
457.7
RECLAIM
69.5
91.2
95.5
48.6
77.1
84.2
466.1
DPC (ResNet) Zheng al. 2017
41.2
70.5
81.1
25.3
53.4
66.4
337.9
SCO (ResNet) Huang al. 2018)
42.8
72.3
83.0
33.1
62.9
75.5
369.6
SCAN (Faster R-CNN, ResNet ce et al. 2018
46.4
77.4
87.2
34.4
63.7
75.7
384.8
Vanilla CL.
47.1
79.6
88.1
35.5
64.2
76.9
391.4
RECLAIM
49.7
81.0
89.8
37.9
66.7
78.2
403.3
Table 3: Image-Text retrieval results with Recall@ K (R@ K). Upper panel: Flickr30K, lower panel: MSCOCO.
5.3 TEXT GENERATION
Dataset: We further evaluate our model on an unsupervised text generation task. Two commonly- used datasets are employed for this task, the Penn Treebank (PTB) (Mareus al.. 1993 Bowman et al. (016) and the Yelp corpora (Yang et al. 2017 He et al.] 2219). PTB is a relatively small dataset with sentences of varying lengths, whereas Yelp contains larger amounts of long sentences. Detailed statistics of these two datasets are summarized in the Appendix.
Implementation details: To ensure a fair comparison and reproducibility between models, we develop model based on existing codebase") The encoder and decoder are both 1-layer LSTMs, and the syper-parameter setup follows the instructions within the original codebase. In this task, the is the latent code in text VAE and is the word embedding vectors of the input text. The most commonly used metrics are applied to evaluate the learned language model, as listed in Table
According Table 4 by simply adding our proposed CL method, the base model can be further improved in terms of most of the automatic metrics. Lower negative ELBO indicates our approach
7

---

Under review as a conference paper at ICLR 2021
yields better language model. Larger KL divergence and larger Active Unit (AU) (Burda al. 2015) indicate that the latent space is more sufficiently made use of. We also observed that the posterior-collapsing problem is alleviated, with improved mutual information and KL (Fang et al.
2019); this is presumably due to the fact that we add additional CL. objective to the latent code and output improve the MI between them.
Methods
-EBBO
PPL
KL
AUT
Methods
ELBOL
PPL.
KL.1
MI
AUT
VAE
102.6
108.26
1.08
0.8
2
VAE
357.9
40.56
0.0
0.0
0
9(0.5): VAE
104.5
117.92
7.50
5
8(0.4)-VAE
358.2
40.69
2.0
SA-VAR
102.6
107.71
1.23
2
SA-VAE
355.9
39.73
2.8
1.7
Cyc-VAE
103.1
110.50
3.48
1.8
5
Cyc-VAE
355.9
39.73
3.8
2.4
11
iVAE
IVAE
348.2
36.70
4.6
IVAE-vanilaa CL
88.1
55.2
1.6
32
IVAE-vanila CL
350.1
37.1
3.7
IVAERECLAIM
85.6
48.7
5.88
4.43
32
IVAE-RECLAIM
346.6
36.5
4.2
4.4
32
Table Variational language modeling results on PTB (left) and Yelp dataset (right).
5.4 ABLATION STUDY
Choice of K Wc seek to further investigate how the negative sample size K influences the effective- ness of our model. To this end, we choose different K = {1. 10, 100, 300} in ANS for comparison. Besides testing different Ks with ANS, also test an alternative approach, where random sample 80% of features from the memory bank as negative samples instead of applying ANS. We denote this method simply the 80% Method. Also, the in-batch method denotes that only negative samples within the batch. The results can be found in Ta Note that those two tricks can be viewed two different ways for constructing vanilla contrastive learning (Vanilla CL) algorithm.
Model 80% w/o ANS
CoLA
SST-2
MRPC
MNLI
WNLI
STS-B
RTF
QQP
QNLI
57.8
92.1
89.7/85.7
83.84/84.21
56.33
90.1/89.5
63.1
91.0/7.88
90.2
In-batch
55.5
91.0
89.48/85.30
83.60/84.11
51.3
88.9/87.8
61.3
90.0/86.6
88.9
RECLAIM K=1
56.6
91.6
89.43/85.16
83.44/84.01
53.7
88.7/87.4
62.1
89.8786.3
89.7
RECLAIM K=10
57.8
92.1
90.2/86.3
83.71/84.15
55.42
89.4/89.2
64.5
90.4/87.3
90.3
RECLAIM K=100
58,8
92.7
91.0/87.1
84.14/84.41
56,33
90.1/89.5
65.0
91.0/87.8
90.4
RECLAIM K=300
58.4
92.7
91.1/87.5
84,20/84.52
55.63
90.1/89.5
64.6
91.2/87.9
90.5
Table 5: Results for different K choices on GLUE dataset. Th 80% method are also listed comparison.
From Table we observe that when K is small, the improvement is limited. In some tasks, such as MNLI, it is even worse than the BERT-base model. This finding consistent with arguments from previous works (Wu et al. 2017 Tschannen et al. 2020). For K = 100 and K = 300, comparable results are often observed, and either of them outperforms the others on certain tasks; K 300 seems to work better on tasks with larger datasets such as MNLI/QNLI. We hypothesize that this is because larger datasets contain more high-quality contrastive examples than smaller datasets, thus allow using large K without introducing much noise. Both of them show better results than the 80% Method without ANS.
Computational Efficiency MNLI, the biggest dataset in GLUE, is employed as a running-timm time benchmark to evaluate the computational efficiency among different approaches. We record the training time for the original BERT, = 100, and the 80% Method. Without any contrastive regularization, BERT takes approximately 45 minutes per epoch. For K 100, RECLÁIM needs 47 minutes per epoch. The 80% Method takes 81 minutes per epoch. The memory usage for BERT-base around 7.5GB for batchsize (per GPU) of TITAN X. K 100 takes additional 200MB, and the 80% Method takes full 12GB memory capacity. These empirical findings provide evidence that our method can be both efficient and effective. Due space limitations, other ablation studies. including the investigation of different À choices, are provided in the Appendix.
CONCLUSIONS
We have proposed a novel contrastive learning (CL) framework, RECLAIM, for natural language processing tasks. Our approach improves the "contrast" in the feature space, in an attempt make the features learned by the representation encoder to more distinguishable, representative and informative. We identified the challenges in CL training and proposed several remedies alleviate
8

---

Under review as a conference paper at ICLR 2021
these issues. Extensive experiments show that consistent improvements variety of NLP tasks, demonstrating the effectiveness of our approach.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In ICML, 2017.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. ICMI., 2019.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. ICML, 2018.
Anthony Bell and Terrence Sejnowski. An informatton-mxxiiizatioo approach to blind separation and blind deconvolution. Neural computation, 1995.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. NeurIPS, 2013.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowiez, and Samy Bengio. Generating sentences from a continuous space. CoNLL, 2016.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv:1509 509.00519, 2015.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 2010.
Liqun Chen et al. Adversarial text generation via feature-mover's 's distance. In NeurlPS, 2018.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 20Xx:20002.05009, 2020.
Kevin Clark, Minh- Than-Thang Luong, Quoc V. Le, and Christopher D. Manning, ELECTRA: Pre-training text encoders as discriminators rather than generators. ICLR, 2020.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 2011.
Bo Dai and Dahua Lin. Contrastive learning for image captioning. NeurIPS, 2017.
Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc' Aurelio Ranzato. Residual energy-based models for text generation. ICLR, 2020.
Jacob Devlin, Ming-Wes Wiig Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2018.
Fartash Faghri, David Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improved visual-semantic embeddings. In BMVC, 2018.
Le Fang, Chunyuan Li, Jianfeng Gao, Wen Dong, and Changyou Chen. Implicit deep latent variable models for text generation. EMNLP. 2019.
Yash Goyal, Tejas Khot, Douglas Summers-Stne Stayy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6904-6913, 2017.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of Wasserstein GANs. In NeurIPS, 2017.
Michael U Gutmann and Aapo Hyvärinen. conercoont estimation: A new estimation principle for unnormalized statistical models. AISTATS, 2010.

---

Under review as a conference paper at ICLR 2021
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Kinkg-iKikttkkkk Lagging inference networks and posterior collapse variational autoencoders. ICLR, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CVPR, 2020.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR 20/9. ICLR, 2019a.
Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchile Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019b.
Charles AR Hoare. Algorithm 65: find. Communications the ACM, 4(7):321-322, 1961.
Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-aware deep feature embedding. NeurIPS, 2016.
Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In CVPR, 2018.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In ICCV, 2009.
Andrej Karpathy and Li Fei-Fei. Deep sisual-semantic alignments for generating image descriptions. In CVPR, 2015.
Jin-Hwa Kim, Jaehyun Jun, and Byoung- Zhang. Bilinear attention networks. In NeurlPS, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
Klemens Lagler, Michael Schindelegger, Johannes Böhm, Hana Krásná, and Tobias Nilsson. Gpt2: Empirical slant delay model for radio space geodetic techniques. Geophysical research letters, 40 (6):1069-1073, 2013.
Kuang-Huei Lee et al. Stacked cross attention image-text matching. In ECCV, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014a.
Tsung- Yi Lin et al. Microsoft coco:
Common objects in context. In ECCV, 2014b.
Ralph Linsker. Self-organization in perceptual network. Computer, 1988.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. In arXiv preprint arXiv: /907. 1692, 2019.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. /CML, 2013.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. 1993.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. arXiv preprint arXiv: 1et.:1115..41111 2018.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. AISTATS, 2020.
Tomas Mikolov, Quoe V Le, and llya Sutskever. Exploiting similarities among languages for machine translation. arX9v:/309.4/66 2013a.
Tomas Mikolov, 1lya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NeurIPS, 2013b.
10

---

Under review as a conference paper at ICLR 2021
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally- normalized margin bounds for neural networks. ICLR, 2018.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. CVPR, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint wxXiv:807.00374: 2018.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. NeurlPS, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Bryan Plummer et al. Flickr30k entities: Collecting region-to-phrase correspondences for richer tesetecoe models. In ICCV, 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Anshumali Shrivastava and Ping Li. Asymmetric Ish (alsh) for sublinear time maximum inner product search (mips). In NeurIPS, 2014.
Noah A. Smith and Jason Eisner. Contrastive estimation: Training log-linear models on unlabeled data. In ACL, 2005.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. NeurlPS, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. ICLR, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. /CLR, 2020.
Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss. In Advances in Neural Information Processing Systems, 2016.
Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. Decoding with large-scale neural language models improves translation. In EMNLP, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In arXiv preprint 1804. 07461, 2018
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers:
State-of-the-an natural language processing. ArXiv, pp. arXiv-1910, 2019.
Chao-Yuan WU, Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In ICCV. 2017.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, 2018.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Kirg-Kirkpatrick. Improved variational autoencoders for text modeling using dilated convolutions. ICML, 2017.
Zhedong Zheng et al. Dual-path convolutional image-text embedding with instance loss. In arXiv, 2017.
11

---

Under review as a conference paper at ICLR 2021
dataset
1%
5%
10%
20%
30%
50%
QNLI-BERT
72.69
83.75
85.83
87.82
88.93
89.96
QNLI-ours
72.52
83.96
85.44
88.03
88.67
90.02
MRPC-BERT
31.62/0.00
68.38/8
388881.22 81.22
69. 40/80.07
71.85 71.85/80.41 /80.47
75.65/83.66
84.07 017888.55
MRPC-ours
31.62/0.00
68.38/81.88 /2.2222222
(0.59/80.38
72.88 /8080
19.82/86.46
11.92/87.65
QQP-BERT
75.43/70.95
82.98/78.16
84.65/79.09
86.76 $1.90
/83777 87. 87.82 87.82/83.77
89. 89.19/85.55
QQP-ours
73.56/70.33
82.57/78.45
84.91/80.14
87.23/ 87.23/83.22 /83.22
88.33/84 88449
89.59/85.94
Table 6: Semi Semi-supervised results GLUE dataset (QNLI, MRPC, and QQP).
A ALGORITHM
Here is the detailed algorithm for our RECLAIM framework.
Algorithm 1 RECLAIM training procedure.
1: Input: batch size dataset X. momentum parameter 1,, maximum number of iterations N.
2: for itr 1,... N do
Sample input œ X and initialize two memory bank Bu.
4: Get features: u, 8 = Network(ar)
5: Update B,, B, by using Equation /
6:
Using ANS algorithm to draw K negative features for u and v: {u.) K Bu. {w() B.
7: Stop gradient for {u, 11 { Bu, Bo
8: Calculate Equation [6 with Wasserstein constrained critic
Minimize the Ensemble loss: Luual Lask ALANS
10: end
B MORE RESULTS
B. MORE RESULTS FOR SEMI-SUPERVISED EXPERIMENT
Results for QQP, MRPC and QNLI are shown here in 6
B.2
MORE VAE RESULTS
Mothods
-EBBOL
PPL.
KL
MI"
AUT
Methods
-EBBO1
PPL.
KL1
MIT
AUT
102.6
VAE
357.9
40.56
0.0
0.0
VAE Reclaim
102,1
107.15
1.2
0.9
VAE Reclaim
352.1
40.2
2.1
1.7
10
Cyc-VAE
Cyc-VAE
355.9
39.73
3.8
2.4
Cyc-VAE with
Reclaim 98.8
105.7
4.18
2.2
10
Cye-VAB VAB Reclaim
350.2
39.73
4.5
3.3
25
Table Variational language modeling results on PTB (left) and Yelp dataset (right).
B.3 ABLATION STUDY
Importance of the Wasserstein regularizer
Model
CoLA
SST-2
MRPC
MNLI
WNLI
STS-B
RTE
QQP
QNLI
BERT-base Devlin et al. 2018
55.6
91.2
89.47/85.29
83.60/84.15
50.7
89.2/88.8 8
60.0
90.2/87.0
90.2
90.1.89.5
90.2
BERT-CL BERT-CC++assseee Wasserslein only
58.1
92.3
89.8/86.2
84.1/84.25
56.13
90.1/89.5
64.7
91.087.88
90.3
BERT RECLAIM
58.8
92.7
91.0/87.3
14.14/84
56.33
90.1/89.5
65.0
91.0887.8
90.4
Table 8: Results on GLUE dataset.
12

---

Under review as a conference paper at ICLR 2021
Model
CoLA
SST-2
MRPC
MNLI
WNLI
STS-B
RIE
QQP
QNLI
BERT-hase Devlin et al. 2018
55.6
91.2
89.47/85.29 29
83,6088,15
50.7
89.2/88.8 8
60.0
90.2/87.0
90.2
RECLAIM
0.01
55.8
91.5
89.66/85.38
83.64/84.18
52.7
89.4/89.0
63.0
90.6/87.3
90.2
RECLAIM
58.8
92.7
91.0/87.3
64.14784.41
56.33
90.1/89.5
65.0
.0878788
90.4
RECLAIM 0.5
57.1
92.2
90.2/86.8
83 33.81/84.31
55.7
89 8/89
63.0
90. 8/87.4
90.3
RECLAIM
55.5
89.33/85.25
83.39/83.38
51.7
89.2/88.1
62.1
90.0/86.6
90.0
Table 9: Results for different À choices GLUE dataset.
Model
CoLA
SST-2
MRPC
MNLI
WNLI
STS-B
RTE
QQP
QNLI
BERT-lagge
62.8
94.2
91. /19888.7
87.6/87.3
55.6
90.5/90.4 90.7/90.6
72.3
91.3/88.4
92.5
RECLAIM
63.8
94.4
93.0/90.2
87.7/87.6
56.33
74.1
91.7/88.9
92.9
Table 10: Supervised results on GLUE datasct with BERT-large model.
importance of À choices We also test the effect of choosing different À {0.01, 0.1, 1}. As shown in Table 9 We can see that when S 0.5, RECLAIM can consistently outperform the BERT-base model. may because we need to re-scale the contrastive loss the same numerical scale as the task-specific loss.
Experiment on BERT-Large We also GLUE experiment on BERT-large model, see whether our proposed algorithm can still be effective. Results can be found in Table I0
B.4
VQA TEST
We also tested our approach on Visual Question Answering (VQA) 2.0 task Goyal et al. (2017), which contains human-annotated QA pairs on COCO images (Lin et al. (2014)). For each image, an average 3 questions are collected, with 10 candidate answers per question. The most frequent answer from the annotators is selected as the correct answer.
Following previous work Kim et al, (2018), we take the answers that appear more than 9 times training set candidate answers, which results in 3129 candidates. Classification accuracy is used as the evaluation metric, defined min(1, humans provided ans.
In this setup, we choose u, v as question features and image features respectively. By applying our RECLAIM approach directly to BAN model, we in Table 2.
improvement over the VQA task as shown
Method
VQA-score
BAN Kim al. 2018
66.06
Ours
66.36
Table 11: VQA validation dataset results
C TRAINING DETAILS
Image-Texs Text Retrieval For the Flickr30K data, we train the model for 30 epochs. The initial learning rate is set to 0.0002, and decays by factor of 10 after 15 epochs. For MS-COCO data, we train the model for 20 epochs. The initial learning rate is set to 0.0005, and decays by 10 after 10 epochs. We set the hatch size 128, and threshold the maximum gradient norm to 2.0 for gradient clipping. We also the dimension of the GRU and joint embedding space to 1024, and the dimension of the word embedding to 300.
We choose batch size as 32 for all 9 GLUE tasks, and 2 x 10-5 is the starting learning rate. For each task, we only perform epochs, since some datasets, such as RTE is quite small, they can be easily got over-fitted.
GLUE
13

---

Under review as a conference paper at ICLR 2021
dimension of features and Note that CLAIM formulation do not require and have a matching dimension. Contrastive learning seeks to compare : uu, v) to т (u)TT()) not comparing directly with Though in practice, often map u and matching dimensions via MLP or RNN to balance their respective contribution to the loss.
Architecture for g: We use two different MLP layers first to map u. v into the same dimension, and then we feed both into a three layer MLP. Details will be included in our next revision. For instance, in BERT model, we will map word representations and hidden states to dim
choice of and In GLUE experiment: is chosen as word vectors (from one-hot tokens to real vectors), v is chosen as the BERT output features with dim=768.
In VAE setup: is chosen word vectors (from one-hot tokens to real vectors), is chosen the encoded latent variable z.
In Image-Texs retrieval setup: is chosen as word vectors (from one-hot tokens to real vectors), is chosen as the image features.
14


Reviews:
[{"review_id": "n1y1miqDIXU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-----------------------------------------\nSummary\n-----------------------------------------\nThis paper proposes an approach to improve (supervised and unsupervised) representation learning for text using constrastive learning. The proposed approach augments standard contrastive learning with: (1) Spectral-norm regularization of the critic to estimate the Wasserstein distance instead of the KL (as in the Wasserstein GAN-style approach), (2) Active negative sampling to select hard negative examples, and (3) momentum-based updates of intermediate features. The resulting contrastive learning objective can be combined with standard supervised and unsupervised objectives to improve downstream tasks.\n\nNone of these techniques is novel on its own to my knowledge, so the main contribution here is to show that combining these techniques for text representation learning yields strong empirical results. This combined approach, called RECLAIM, is applied to a suite of standard NLP tasks: transfer learning, image-text retrieval, and language modeling, where it is found to outperform standard baselines.\n\n-----------------------------------------\nStrengths\n-----------------------------------------\n- Sound improvements on top of existing finetuning methods for GLUE  (though I have questions about the semi-supervised setup; see questions).\n\n- Evaluation across a wide range of setups (though language modeling results aren't really valid in my opinion due to the iVAE perplexity numbers being not exact; see weaknesses).\n\n- Ablation studies across various augmentations, though it would have been interesting to also ablate on Wasserstein vs KL as well.\n\n-----------------------------------------\nWeaknesses\n-----------------------------------------\n- My biggest gripe is that the paper is presented as a \"representation learning\" work, yet much of the focus is on performance on underlying tasks. That is, the contrastive learning objective is simply used as an additional regularizer on top of existing objectives. Implicit in this is the idea that \"good representations should lead to good downstream performance\", but it would have been great to do closer analyses of the learned representations to see how this contrastive learning objective results in different representations (see some example analyses that could be conducted in the questions section).\n\n- Given that RECLAIM is really a source of regulariazation, I would have liked to seen it compared to existing regularization methods such as MixOut on GLUE (https://arxiv.org/abs/1909.11299).\n\n- Results on language modeling are not really valid since the iVAE codebase (on which this work is based) does not actually calculate perplexity (they only estimate perplexity, which is a drawback of the iVAE work). The language modeling results would be much more compelling if the authors showed improvements on top of works that calculate perplexity more properly, e.g. with a variational lower bound or an importance-sampled estimate (e.g. as done in works such as Cyc-VAE).\n\n- It seems like this objective can be combined with self-supervision (e.g. masked LM objective) to aid in better BERT-like models (which would have been a very interesting result), but this seems to have been left unexplored.\n\n-----------------------------------------\nQuestions/Comments\n-----------------------------------------\n- How do representations learned this way differ from the standard approach? There are many interesting analyses that could be conducted for the different tasks. For example: on GLUE, which layers of BERT are finetuned more (or less) with this method; on image-text retrieval, what are some mistakes made by the standard approach that this method is able to fix?; on language modeling, how do the learned latent codes differ compared to iVAE (i.e. is it more sensitive to nouns/verbs, etc?)\n\n- Does the semi-supervised experiments utilize the full unlabeled data? (i.e. do you train with the contrastive objective on unlabeled data, and the supervised + contrastive objective on the labeled data?)\n\n- Does the switch from KL to Wasserstein distance make the lower bound on MI (equation 4) invalid?\n\n- The results on GLUE would be much easier to read if the average results across tasks are also presented\n---------\nAfter rebuttal: Thank you for the response. I have decided to leave my score unchanged.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UYOux0zj8Su", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new contrastive learning methods based on intermediate textual feature pairs, which stabilizes training and overcomes vanishing contrasting signals by imposing Wasserstein constraints on the critic via spectral regularization, and leverages a fast and dynamic selection of high-quality negative samples. \n\nOverall, this is an interesting work on Contrastive learning for NLP, which\n-\tswitches from the MI-based NCE to a Wasserstein-based NCE for more robust and stable training\n-\tcreates a fast (with sub-linear Asymmetric Locality Sensitivity Hashing) and dynamically (with momentum update) select the most challenging negative examples on-the-fly.\n-\tseems to get solid performance gains on wide range of downstream applications.\n\nMy major concern is about the clarity of the paper and some detail information of model and experimental design.  It seems not very easy to reproduce corresponding codes and experiments only based on the current paper content. At least, there is quite some space to improve readability. For instance:\n-\tThe concrete u and v in each experiment should be more clearly explained, especially for u.\n-\tDetail design and Implementation of g\n-\t\"random sample 80% of features\" seems not clear enough for understanding\n-\tMore explanation for “Stop gradient for…” in Algorithm 1\n-\tOrganic integration of section 3.2/3.3/3.4 based on holistic and detail explanation of Figure 1 \n-\tMore clear illustrations on unsupervised/supervised/semi-supervised experimental designs and processes\n\nMinor issues:\n-\tSome notations are not self-contained. For example, “sup_x” in section 3.2 is not defined.\n-\tReference section should be more carefully checked. For instance, Hjelm et al. 2019a and 2019b are pointing to the same paper, and Lagler et al. 2013 seems to point to a wrong GPT2 paper which is not in the NLP domain.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8m58lHxobZD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to improve contrastive learning for text representation by tackling two challenges: (1) The conventional mutual-information (MI) based objective can results in unstable training due to the sensitivity of KL to changes of representation; (2) contrastive learning could require a large number of negative samples, which is inefficient. For (1), the paper replaces the KL loss with Wasserstain distance; while for (2), the paper selects top-K most difficult negative samples (defined as the nearest neiborhors to the current representations).\n\nThe contrastive learning loss is then used to augment the standard task-specific loss to enhance learning and enable semi-supervised training. Experiments on benchmarks for text classification (GLUE), retrieval, and language modeling show the proposed method improve over standard supervised/semi-supervised learning.\n\nThe proposed method is intuitive and simple, largely stitching together previous successful techniques (e.g., Wasserstein distance for robustness, hard negative samples, momentum update).\nSection 2.2 gives a nice summary of several core challenges of contrastive learning.\n\nQuestions / weaknesses:\n\n1) In section 2.2, is it necessary to introduce v? -- Constrative learning is usually seen as maximizing the MI between input w and representation u, i.e., MI(w, u). Why here MI(v, u), e.g., Eq.(4), is used?\n\n2) I'm a bit surprised that the experiments do not include any comparison with previous contrastive learning methods, but only compared with models without contrastive learning. \nSimilarly, the ablation study only compares results of negative sample sizes (K). It's necessary to include more comparsions to show how much improvement each of the two proposed techniques (Wasserstein distance, hard negative samples) reaches. \n\n3) The caption of Table.2 says \"Results for QNLI, MRPC, and QQP are in the Supplementary Material (SM). \". Yet those results are not included in the suppementary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-Q1s-tWMCDH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that there are some challenges in learning text representations with vanilla contrastive learning techniques, due to the metric-aware property of  KL divergence as well as the difficulties in selecting good negative examples. It then presents two remedies: employ a Wasserstein constraint to the critic function and a simple active sampling strategy for negative examples.\n\nI think the paper is not strong enough to get into ICLR. Firstly, the Wasserstein constraint has been proposed to improve general representation learning in Ozair et al. (2020). The novelty of this work is very limited compared to Ozair et al. (2020). The active sampling contribution is a simple heuristic. Secondly, the authors only compare the proposed method with the non-CL baseline. A vanilla CL baseline should be included in all the experiments. Finally, the improvement margins are also pretty small.\n\nThe writing can be much better if the authors can use examples for concrete tasks. For instance, I am not sure the distinction between u and v representations after reading sec. 2.2.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work brings improvement to contrastive learning method for text data by
combining a Wasserstein objective with a "memory bank" strategy
for getting (and updating) hard negative samples.
The approach leads to small but consistent improvements across a variety of representation learning tasks in both supervised and unsupervised settings. While the paper makes a useful contribution and evaluates with some success on downstream tasks, the reviewers would like to see some intrinsic, qualitative discussion of the representation learning itself, and in comparison to more powerful contrastive learning methods. Overall the work falls below the acceptance threshold in a very competitive venue, so I cannot recommend acceptance.

Even after discarding one uninformative review, the consensus remains borderline. The reviewers are, however, appreciative of the additional clarifying experiments provided by the authors. In the internal discussion, a concern was raised that the BERT baselines may not have a fair chance to compete, as they are not fine-tuned on the unsupervised data in the same way that the proposed method is, leading to possibly overestimating the improvement.  I encourage the authors to consider this.

Finally, while some concerns about clarity have been addressed, some remain in the current version. In particular, I spot at least three duplicate and inconsistent entries, for Hjelm et al, Lin et al, and McAllester and Stratos.)
Decision: Reject