Sentence embedding with contrastive multi-views learning
Authors: Antoine Simoulin
Venue:  (2020)
Keywords: contrastive
multi-views
linguistic
embedding

Abstract: In this work, we propose a self-supervised method to learn sentence representations with an injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be expressed out of compositional words operations. We aim to take advantage of this linguist diversity and learn to represent sentences by contrasting these diverse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim at building embeddings which better capture semantic and which are less sensitive to the sentence outward form.


Under review as a conference paper at ICLR 2020
SENTENCE EMBEDDING WITH CONTRASTIVE MULTI- VIEWS LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we propose self-supervised method to learn sentence representa- tions with injection of linguistic knowledge. Multiple linguistic frameworks propose diverse sentence structures from which semantic meaning might be ex- pressed out of compositional words operations. We aim take advantage of this linguist diversity and learn to represent sentences by contrasting these di- verse views. Formally, multiple views of the same sentence are mapped to close representations. On the contrary, views from other sentences are mapped further. By contrasting different linguistic views, we aim building embeddings which better capture semantic and are less sensitive to the sentence outward form.
INTRODUCTION
We propose to learn sentence embeddings by contrasting multiple linguistic representations. The motivation is to benefit from linguistic structures diversity to discard noises inherent to each repre- sentation. We aim at encoding high-level representations by aligning the underlying shared infor- mation from multiple views.
As illustrated in Figure [ we train our model with a contrastive framework which aims map- ping close input sentences to close representations while separating unrelated sentences. In Natural Language Processing (NLP), this framework has been widely used to learn word representations (Mikolov et al.) 2013a b) for example. This model relies the distributional hypothesis which conjectures that words within similar context share similar meaning. Such framework has also been extended to sentences with the similar hypothesis that the meaning can be inferred from the context sentences (Logeswaran & Lee 2018
Kiros et al.) 2015).
We propose extend this framework by assuming that different views of the same sentence should lead to close representation. We considered the dependency trees, a linguistic framework that de- scribes the compositional structure of sentence. As illustrated in Figure [ in this framework, the sentence is mathematically described as an oriented acyclic graph where the nodes are words and edges describe the relations between words. Such structure has benefited from an important attention in the NLP community and efficient parser tools for various languages are available, which makes possible to obtain such information almost freely in the sense does not require additional hand annotated data.
Tree representations are then mapped in shared embedding space using appropriate Tree LSTM networks introduced in Tai et al. (2015). Model parameters are learned using a discriminating ob- jective as proposed in Logeswaran & Lee (0018).
RELATED WORK
2.1 REPRESENTATION LEARNING
Representation learning has gained significant attention from the NLP community (Bengio et al. 2013 As illustrated in Figure it is structured in two-steps: (A) representation is learned using proxy objective usually very large corpora (B) the representation is then used to solve variety of downstream tasks. Literature comes with a variety of proxy tasks to learn representations. Proxy objectives usually fall in two categories: (i) predicting contextual information reconstructing

---

Under review as a conference paper at ICLR 2020
Sequential LSTM
A guitar is being played by the man.
ChildSum LSTM
Someone feeding animal.
Metria pairwise comparison
Softmax
the music filled with rage and fury.
bunch guys shore.
Multi-iiew
Contrastive
Contrastive multi-views framework. The model is trained to distinguish between different views of context sentences and negative examples. The two views are obtained using standard LSTM and a Tree L.STM networks on top of dependency tree structure. discriminative objective used to contrast between samples.
Figure 1:
an altered input or (ii) discriminating multiple representations of the same data among negative examples.
Proxy (i) has been declined into multiple variations in the literature. Words embeddings methods Mikolov al. (2013b) proposes that words with close distributions should lead to close represen- tations. For images, Noroozi & Favaro (2016) proposes to learn representations by solving jigsaw puzzles. With close taste to NLP tasks, Pathak et al. (2016) aims at learning representations by trying to fill the missing part of an image. van den Oord et al. (2018); Walker et al. (2016) methods aim at predicting the future to represent video or audio.
2.2 CONTRASTIVE LEARNING
The second proxy objective (üi) is motivated by the observation that end-to-end frameworks tend to learn semantic similarities among classes. For example Wu et al. (2018) reports that the hidden layers supervised neural networks assigns close representations to leopard and jaguar images. However an image from lifeboat and Icopard will be apart from each other. This motivates for contrastive learning methods where the network aims at distinguishing individual samples insure global consistency in the dataset.
Many contrastive learnings have been proved effective in variety of domains: image processing
TTin et al. 2019 Wu et al. 2018), audio input (van den Oord et al. 2018), sentence representation Logeswaran Lee, 2018) or word embedding (Mikolov et al. 2013a Mnih & Kavukeuoglu,
2013).
Besides nature of data, formalisms might differ on two main aspects: (i) How to measure proximity in both the original and the representation space and (ii) What training objective should be used to achieve well distinction between samples.
One possible answer how to select samples that should lead to close representation is consider different views of the same sample as proposed in Tian et al. (2019); Li et al. (2018) who propose to combine multi views of images with contrastive learning. In NLP, Logeswaran & Lee (011); van den Oord et al. (2018) propose contrastive frameworks for sentences but without multi-view. setting.
Multiple metrics to measure the proximity in the representation space are proposed in the literature: Tschannen al. (2019) enumerates multiple called critic functions. The most straight forward
2

---

Under review as a conference paper at ICLR 2020
also used in Logeswaran & Lee (2018) beeing the scalar product: ho(u, u² v. But other critics are considered such as the separable critic uu v) = 61(") (z()).
The objective function for contrastive learning has also been declined in multiple work: Many relate to Mutual Information (MI) (Hjelm et al.) 2019 Tschannen et al. 2019 Bachman et al. 2019) or
methods based on triplet loss max-margin (Chopra et al. 2005 Weinberger et al. 2005
METHOD
3.1 CONTRASTIVE LEARNING
Contrastive learning is self-supervised learning method which aims at learning semantic map- ping. Data is embedded with compact representations such that close samples are mapped to nearby points while unrelated ones affected to apart points. In practice different methods exist to learn such mapping- A straight forward method is to treat each data sample as distinct class and train classifier distinguish between individual instance classes. Such approach is used for word em- bedding where the goal is to predict a word given his context.
However this is computationally difficult and the outputs are not finite set in our case since an infinite number of correct sentences might be expressed. A method to approximate a full instance discrimination is use Negative sampling and Noise Contrastive estimation methods. For every data point r from a dataset D we have access to one paired sample x which should be close. We then draw negative samples zi 22 ... "x which should be apart. We learn a scoring function hg which assigns large scores to positive paired samples (I,, x+) while low values for negative pairs (xr, ..). In practice the function is trained using a classifier which aims at identifying the correct pair he(x "xx"x,")) among negatives pairs {ho(x* 2g ) (k==),,)) In our setup we use a softmax classifier
as follow:
exp(ho(x, ha (zx + Ex= ho(
p(1a 12,
,K ,
(I)
The classifier trained using the negative log likelihood loss:
L Σ logp (x [x. {2j L2 E,
(2)
RED
As in Logeswaran & Lee (2018) the used scoring function the inner product ho(u, v) = nf u in order avoid the situation where the model learns poor representations but compensates with an excellent classifier on the proxy task. However some authors report excellent results with other critic functions: for example Tai al. (2015) uses combination of absolute distance and angle measures. Given the scoring vectors for each pair representation we use a simple softmax classifier with log negative likelihood loss.
3.2 MULTI-VIEWS LEARNING
In our setup, the positive pair xx))))) is built from different views of the input data. From the orig- inal sentences, we construct multiple representations from different views of the object. For every sentence we consider two encoders. xg which is the representation obtained with bidirectional LSTM (Hochreiter & Schmidhuber (1977) who assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies. The final representation of the sentence the concatenation of both direction last state. 2a is obtained with the dependency tree representation combined with a ChildSum Tree LSTM (Tai et al.) (2015). The final representation is the state from the root node of the sentence graph. Negative examples are obtained using the dependency Tree LSTM. The dataset is therefore augmented with the positive examples and multi-views examples leading to D {I,, Fz, 2 21 2222"
Similarly to Logeswaran & Lee 2018), the model is trained to identify the sentence appearing in the context from the target sentence. The key difference and our contribution is to build different views
3

---

Under review as a conference paper at ICLR 2020
of the data. The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM. As described in the previous section, the objective is to maximize the log probability of the correct sentence to appear in the context.
4 EXPERIMENTAL SETUP
4.1 DATA
Models trained on the BookCorpus dataset Zhu et al. (2015). Since the dataset is no longer distributed, a similar dataset is generated using smashword open book data [ The obtained dataset contains 78M sentences from 17.000 books. We define 20k words vocabulary, containing the corpus most frequent tokens. Sentences are tokenized using Microsoft BlingFire tool Models are trained single epoch the entire corpus without any train-test split. The of Tree LSTM models supposes to parse sentences in dependency which is operated using the publicly available Spacy parser.
The contrastive learning method supposes to draw negative examples for each sentence pair. We fol- low the training procedure proposed in Logeswaaann Lee (2018) and build mini-batches respecting books sentences order. Therefore samples from the mini-batch are used as negative examples.
4.2 TRAINING PARAMETERS
Models are trained using the Adam optimizer Kingma Ba (2015) with a 5e-3 learning rate. Gra- dient is clipped with value of 5.0. The hidden layer for both Tree LSTM is fixed to 1000 and the embedding dimension to 300. The batch size is 100 and the model is trained on a 1080Ti Nvidia GPU. All model weights initialized with a xavier distribution Glorot & Bengio (2010) and biases to O. We observe biases initialization to have a significant impact on the training. Indeed leaf nodes initialized with zero vectors for state and hidden and initial biases with significant values tend to slow the training convergence.
SICK conwstream tasks performances
Spaany 1ee
Figure 2: Evolution of performances on the downstream SICK-E (top) and SICK-R (bottom) tasks. A point is computed every hour during the training phase. Despite the batch computation imple- mentation describes in Section A the Contrastive Tree L.TM is much slower to train. The training was stoped after 33 hours and training on 5.8% of the available training data. The evaluation setup for the SICK tasks is describe in Section 5
The Tree LSTM network was implemented using batch procedure described in Section A However the training was significantly slower than vanilla LSTM. The training phase was stopped after 33
'Corpus dataset was generated using http:///ghhhb.coo/osccc/oooorr/ http:////hthc.oo/////ooff/f/////
4

---

Under review as a conference paper at ICLR 2020
hours of training. The training phase was completed only 4.6M sentences among the 78M avail- able. We monitored the performance on downstream tasks during training as illustrated Figure 2
RESULTS AND ANALYSIS
As described in Figure 3 the process requires specific evaluation processes at each learning step. Although methods are developed to control the properties of the intermediate representation such as probing tasks for NLP [Cnneau et al. (2018), the intermediate representation is usually evaluated on his ability to solve the final task: downstream evaluation. To facilitate the comparison across multiple representations and limit the impact the downstream algorithm, the linear evaluation protocol (Kiros et al., 2015)) usually used. It proposes to solve the downstream task with minimal and simple model: a logistic regression. At test time, the concatenation of two encoders is used.
(8)
(A)
Target
Datasct
Dataset
Representation
Targots
Figure 3: (left) End-to-end framework where the algorithm learns to map directly the inputs to the target. The hidden sates of the neural network only used for this task. (right) In contrast to end-to-end approaches the algorithm proceeds two phases. First representation is learned with sulf-super proxy task. The same representation might then be used for different tasks.
5.1 DOWNSTREAM EVALUATION
Sentence representations evaluated on downstream tasks that require to capture the underlying sentence semantic. 7 classification tasks from the SentEval benchmark are applied: movie review sentiment (MR) (Pang & Lee 2005), product reviews (CR) (Hu & Liu 2004), subjectivity classi-
fication (SUBJ) (Pang & Lec| 2004). opinion polarity (MPQA) (Wiebe et al. (005). question type classification (TREC) (Voorhees) 2003). semantic relatedness and entailment (SICK-R, SICK-E)
Marelli et al. 2014) and paraphrase identification (MRPC) (Dolan et al., 2004).
The MR, CR, SUBJ, MPQA tasks are binary classification tasks with no pre-defined train-test split. A 10-fold cross validation is used in reporting test performance for these tasks. For other tasks we use the proposed train/dev/test splits. The dev is used for choosing the regularization parameter and results are reported the test set. We follow the linear evaluation protocol of Kiros et al. (205)) where a logistic regression classifier is trained on top of sentence representations with cross-validation procedure described earlier. This protocol facilitates the comparison across studies and avoids the case of a good classifier which compensates for bad sentence representations. The script from Conneau & Kiela (2018) is used for downstream evaluation.
The scores are not as good as the state the art results reported in Table I However the model was not trained the entire available training set used in Logeswaran & Lee (2018) The model shows encouraging properties and trends in Figure 2 suggest it might improve with larger exposition to the training data.
5

---

Under review as a conference paper at ICLR 2020
Model
MR
CR
SUBJ
MPQA
TREC
SICKR
SICKE
MRPC
Supervised training methods
GloVe Bow LSTM
78.1
80.4
91.9
87.8
85.2
76.4
81.1
85.3
BidirLSTM
Tree-LSTM
86.8
Infersent
81.1
86.3
92.4
90.2
88.2
88,4
86.1
76.2
Self-supervised training methods
Fastsent
71.8
78.4
88.7
81.5
80.3
Skip-Thought
76.5
80.1
93.6
87.1
92.2
85.8
82.0
Dissent
80.2
85.4
93.2
90.2
91.2
84.5
83.5
76.1
thougtts
82.4
86.0
94.8
90.2
92.4
87.4
84.0
Contrastive Tree
64.0
72.8
86.8
77.7
82.4
80.0
79.1
Table 1: Comparison of sentence representations on downstream tasks. For the SICK-R task, the pearson coefficient is indicated. For the MRPC task, this is the F1 score. The baseline is reported from (Logeswaran & Lee 2018) and is obtained using a bag-of-words representation. LSTM. BidirLSTM and Tree LSTM are reported from Tai et al. (2015). Fastsent is reported from (Lo- geswaran & Lee 2018), Dissent and Infersent from (Nie et al. 2019), Skip-thoughts from (Kiros 2015)) and Quick-thoughts from (Logeswaran & Lee 2018 The table is divided into different sections. The bold-face numbers indicate the best performance values among models the current and all previous sections. Best overall values in each column underlined.
5.2 PROBING TASKS
Conneau et al. (2018) argues that downstream tasks require complex form of inference, which makes difficult to assess the fine grained quality of a representation. Probing tasks are supposed to overcome this limitation and separately evaluate individual linguistic properties of representations. The probing benehmark includes surface information tasks: predicting the length of the sentence (SentLen) and the word content (WC). Syntactic tasks include predicting the depth of the sentence tree (TreeDepth), the root top constituents (TopConst) and the word order (BShift). Finally seman- tic tasks aim at predicting the verb tense (Tense), if the subject and complement singular plural NSubjNum, ObjNum) and semantic or coordination inversion (SOMO, CoordInv). The probing results presented in Table 2 for both Contrastive L.STM (CI.) and Contrastive Tree LSTM (CTL.).
Probing task
LSTM (CL)
Contrastive Tree LSTM (CTL)
SentLen
74.7
83.0
WC
66.4
23.1
TreeDepth
35.1
33.4
TopConst
81.2
77.3
Bshift
59.7
58.1
Tense
87.7
82.5
SubjNum
79.8
74.5
ObjNum
74.4
66.0
SOMO
52.3
50.5
Coordin
67.9
63.4
Table Probing task scores for the Contrastive Tree LSTM (CTL) and the Contrastive LSTM (CL). Bold-face numbers indicate the best performance values among the two models. The proposed train- test split is used for each task. Parameters are fixed on the dev set and results are presented on the test set. Results for the Contrastive LSTM (CL) are obtained using model we trained on version of the bookcorpus as described in Section
We observe the interaction between the standard Tree LSTM and the Tree LSTM is sensible to the sentence length which was also observed by TTai al. 2015). All other scores are bellow the results

---

Under review as a conference paper at ICLR 2020
obtained by the application of the sole LSTM, pointing that the introduction of linguistic knowledge does not compensate enough for the smaller training set.
5.3 QUALITATIVE ANALYSIS
Our training setup and Logeswaran & Lee (2018) share similar framework but with different models embed sentences. We retrieved the nearest neighbor from several query examples with the two methodologies. The sentences are extracted from the SICK test set. The closest neighbors are determined using the cosine similarity and presented Table Examples are selected to illustrate the ability of models to capture desired linguistic properties such passive form, concepts, counting faculties gender identification.
Top 5 sentences LSTM
Score
Top 5 sentences Tree LSTM
Score
The guitar is being played by the man
guitar is being played by the
0.046
guitar is being played by the
0.036
The man is playing an acoustic guitar The man playing the guitar
0.054
A guitar is being played by man
0.064
0.055
guitar is being passionately played
0.071
by a man
The man guitar player
0.057
The guitar player loves the man pas-
0.120
sionately
The guitar player loves the man pas- sionately
0.059
The guitar being played by the man who has the guitar case open for do- nations
0.123
woman is dancing on a stage
A woman is dancing
0.019
A young girl is dancing
0,083
woman is dancing gracefully
0.024
woman is washing a big pepper
0.092
A woman is dancing beautifully in
0.050
The woman is not seasoning the oil
0.096
cage
A woman is dancing and man is
0.058
A little girl is playing piano, which very big on stage
0.099
playing the keyboard
A woman is staging dance
0.073
A woman is dancing
0.102
and woman talking
man and a woman are speaking 0.048
man and a woman are speaking
0.014
A man and woman are sitting
0.077
A man is talking to woman
0.052
man is talking to woman
0.093
The man and woman are not walking
0.082
A woman is ignoring man
0.155
One man is talking to girl with internet camera
0.083
man is talking
0.156
man and a woman walking
0.086
A bunch of guys is the shore
group of people is near the ocean
0.134
bunch of guys on the shore
0.012
Two men are standing in the ocean
0.189
There is no one the shore person is rinsing steak with water
0.081
Four people are floating on a raft A crowd of people is near the water Two men are walking through river
0.202
0.092
0.213
A sea turtle is not hunting for fish sea turtle is hunting for fish
0.098
0.238
0.103
Table 3: Nearest neighbors retrieved by the Contrastive Tree LSTM (CTL) and the Contrastive LSTM (CL). For each sentence, the 5 closest neighbors from the test split of the SICK dataset are retrieved using the cosine similarity from their representations. Results presented in decreasing order of cosine distance.
Both models might be fooled with surface information and retrieve examples with similar words but different meanings. Sole LSTM encoder presents remarkable properties to identify passive form. The Tree LSTM presents interesting properties such retrieving sentences with similar principal

---

Under review as a conference paper at ICLR 2020
proposition but complement which do not alter the sense of the sentence. Tree LSTM also captures number properties and identifies concepts such as 4 people might be referred as a group.
CONCLUSION
We exploit the diversity of linguistic structures to build sentence representations. Out method shows promising results and does not require hand annotated data. More scalable implementations might be considered to explore more experimentation setups. Although results are below state of the art performances, model is trained only small proportion of the bookcorpus sentences stated in Figure 2 A larger exposition to the training data and an extended training time might benefit to the downstream and probing scores. Other linguistic structures might also be tested such as constituency tree associated with N-ary Tree LSTM or Tree LSTM improved with an attention mechanism.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information acToss views. CoRR, NDV11906009911 2019.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Patlern Anal. Mach. Intell., 35(8):1798-1888, 2013.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego, CA. USA, pp. 539-546, 2005.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018., 2018
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single \S&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2126-2136, 2018.
Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. COLING 2004, 20th International Conference on Computational Linguistics, Proceedings of the Conference, 23-27 August 2004, Geneva, Switzer- land, 2004.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pp. 249-256, 2010.
R. Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Karan Grewal, Philip Bachman, Adam
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 25, 2004, pp. 168-177, 2004.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7- 12, 2015, Montreal, Quebec, Canada, pp. 3294-3302, 2015.
Yingming Li, Ming Yang, and Zhongfei Zhang. A survey of multi-view representation learning. IEEE Transactions on Knowledge and Data Engineering, 1P:1-1, 09 2018.
Lajanugen Logeswaran and Honglak Lee. efficient framework for learning sentence representa- tions. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings, 2018..
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, LREC 2014, Reykjavik, Iceland, May 26-31, 2014., pp. 216-223. 2014.

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In 1tt International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Informa- fion Processing Systems 26: 27th Annual Conference Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, 3, Lake Tahoe, Nevada, United States., pp. 3111-311999 2013b.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., pp. 2265-2273, 2013.
Allen Nic, Erin Bennett, and Noah Goodman. Dissent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4497 4510, 2019.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jig- saw puzzles. Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part VI, pp. 69 69-84, 2016.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. 21-26 July, 2004, Barcelona, Spain., pp. 271-278, 2004.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect rating scales. ACL 2005, 43rd Annual Meeting of the Association for Compu- tational Linguistics, Proceedings of the Conference, 25-30 June 2005, University of Michigan, USA. pp. 115-124, 2005.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In 2016 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2536-2544, 2016.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pp. 1556-1566, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. abs/1900.05849, 2019.
Contrastive multiview coding. CoRR,
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. CoRR, abs/1907. 2019.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1807.00
2018.
Ellen M. Voorhees. Overview of the TREC 2003 question answering track. In Proceedings of The Twelfth Text REtrieval Conference, TREC 2003, Gaithersburg, Maryland, USA, November 18-21, 2003, pp. 54-68, 2003.
Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part VII, pp. 835-851, 2016.
10

---

Under review as a conference paper at ICLR 2020
Kilian Q. Weinberger, John Blitzer, and Lawrence K. Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in Neural Information Processing Systems 18 [Neural Information Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Canada), pp. 1473-1480, 2005.
Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3):165-22 210, 2005.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. CoRR, 0bss/180500997 2018.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 19-27, 2015.

---

Under review as a conference paper at ICLR 2020
A COMPUTING METHOD FOR TREE LSTM
We implemented batching procedure to fasten Tree LSTM computations. Group of nodes are com- puted sequentially insure all node children have already been computed. Nodes considered given their distance to the root node. First, Leaf nodes with highest depth are computed to progres- sively compute inner nodes. The Tree LSTM cell implementation is specifically designed treat simultaneously all nodes in each the step.
Step #3 (outputs of Step #2 are indicated in
Step #2 (outputs of Step #1 are indicated in
Step #1
Batch size
Figure 4: The batching procedure to optimize the graph computation. For each batch, the compu- tation is decomposed in steps which insure that every node dependent have already been computed. At each step, node with the same depth to the root are computed in single operation and the output fed to the next computational step.
12


Reviews:
[{"review_id": "HJlsoiwNtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new sentence embedding method. The novelty is to use dependency trees as examples in the self-supervised method based on contrastive learning. The idea to use linguistic knowledge in the design of sentence embeddings is attractive. The sentence representation is computed by a bi-LSTM and dependency tree representations are computed by Tree LSTM. The softmax classifier is trained using the negative log-likelihood loss.\n\nIn my opinion, the paper could not be accepted. As said before, the idea is attractive but the paper lacks motivations for the choice of dependency trees as additional linguistic knowledge. Indeed, the goal of the proposed algorithm should be made more precise. It is, in my opinion, very difficult to do better than existing sentence embedding methods and the proposed method should be used for specific downstream tasks where the structure of sentences is meaningful. Moreover, the proposed method do not scale well and empirical results on classical downstream tasks are not convincing. Last, in my opinion, the redaction of the paper should be improved and the bibliography should be updated. For instance, the best up-to-date sentence embedding methods are not cited (ELMO and BERT).\n\nDetailed comments.\n\n* Abstract and introduction. The description of the contribution is not precise enough. Please make precise what are \"multiple views\", \"different linguistic views\". Please explain why you choose dependency trees and explain why their use can improve sentence embeddings. \n* Related work. Please consider only word embeddings and sentence embeddings because the literature is sufficiently large in the last few years. Please update your related work with methods such as ELMO and BERT and subsequent work. Also recent papers study how BERT embeddings embed structural information and these should be discussed as you consider dependency trees in the construction of sentence embeddings.\n* The method does not scale well. The paper does not propose ideas to solve this problem. Why don't you consider the approach used in Logeswaran et al. \n* The qualitative analysis shows that similar sentences have a similar structure. This is not surprising because dependency trees are used for learning. But this should give ideas of downstream tasks for which the approach could be fruitful.\n* Many typos.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygW3uM6KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: This work proposes to learn sentence embeddings using both contrastive learning and multiple \"views\" of sentences.  This work largely builds off of [1], including using the same objective, but uses a multi-view approach to modeling.\n    - They apply the concept of multi-view models, specifically combining tree and linear LSTMs to learning sentence representations.\n    - They prepare a new, large-scale book dataset, which is useful because the previously commonly used book dataset was taken down for legal reason.\n    - They provide a fairly broad set of analyses on their model, both quantitative and qualitative, performance-driven and analysis-driven.\n- Review: The ideas and models presented in this paper are not new, while the supporting experiments are not very well done or convincing. Overall, I recommend rejecting this work.\n- The models are contrastively learned in that they are trained to embed \"similar\" sentences nearby in the embedding space, and \"dissimilar\" sentence far away, where \"similar\" sentences are defined as consecutive sentences. This method of learning textual representations is well-established in the NLP literature, mostly prominently in recent years with word embedding models like Skip-Gram and in sentence embedding models like in [1], [2], [3] (the next sentence prediction task), and several more.\n- In practice, the multiple views of each sentence that this paper considers boils down to encoding the sentence with a bidirectional LSTM and a TreeLSTM and concatenating the representations from each encoder. This idea again has been established in the literature ([4], [5], [6]).\n- The experiments don't seem setup to demonstrate that the multiple views are beneficial over a single view. In Table 1, there are rows for just an LSTM or just a TreeLSTM, but they seem to be trained with labeled data whereas the proposed method is trained self-supervised. A more informative comparison to demonstrate the value of using multiple views would be to train the LSTM and TreeLSTM with the same objective (and ideally model size). Overall, I don't think the claims in the paper are well-supported by the model proposed or the experiments.\n- I have a number of concerns about the experiments.\n    - \"Models are trained on a single epoch on the entire corpus without any train-test split\": so there is no early stopping? Why stop training after one epoch? Was there any indication you were overfitting the data?\n    - \"The training phase was stopped after 33 hours of training\": Why stop there? Computational constraints? Later comments suggest this is quite premature (\"training phase was completed on only 4.6M sentences among the 78M available\").\n    - The results seem to indicate that this method underperforming recent work significantly.\n\nAreas of improvement\n- Some of the language in the introduction and conclusion are a bit of a stretch. Using a linear and tree LSTM (based on dependency parses) doesn't really represent a \"diversity of linguistic structures\". \n- Related work: There's no mention of pretained language models, which could be seen as a form of representation learning for language, and have been hugely impactful in NLP.\n- Method\n    - Missing negative in the log likelihood\n    - Why do you use inner product if other works \"report excellent results\" with other scoring functions?\n    - \"assumes the underlying structure of the sentence to be a sequence, while allowing for long term dependencies\": If anything, the treeLSTM more easily allows for long-term dependencies than the linear LSTM.\n    - \"Negative examples are obtained using the dependency Tree LSTM\": I'm not totally sure how the negatives are obtained here.\n    - \"The target sequence is encoded using the sequential Tree LSTM, while the positive and negative samples are encoded using the ChildSum Tree LSTM\": why are the sentences not all encoded with the same encoder?\n    - It looks really odd that most of Table 1 is empty. Given your model, I imagine it can't have been that difficult to evaluate more baselines (BiLSTM and TreeLSTM) on the rest of the tasks.\n    - It'd be nice if you could clearly indicate in Table 1 which method is yours.\n- Results and Analysis\n    - The standard evaluation setting for sentence embeddings would be GLUE or SuperGLUE.\n    - A glaringly missing baseline is BERT (or any of its relatives), which is also self-supervised.\n    - The results are underwhelming, and as the author admits, somewhat premature as training didn't seem to finish.\n    - 5.2: what are the contrastive LSTM and Tree LSTM? Are those the learned encoders from the \"Contrastive Tree\" in Table 1, or are they trained from scratch?\n    - I don't think the analyses in Sections 5.2 and 5.3 or Figure 2 are particularly useful.\n- There are a noticeable number of typos. For example, in the abstract: \"this linguist[ic] diversity\" and \"better capture semantic[s]\". It'd be worthwhile to look over the paper closely for typos.\n\n\n[1] AN EFFICIENT FRAMEWORK FOR LEARNING SENTENCE REPRESENTATIONS. Lajanugen Logeswaran and Honglak Lee\n[2] Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. Yacine Jernite, Samuel R. Bowman, David Sontag\n[3] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova\n[4] Enhancing and Combining Sequential and Tree LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang.\n[5] Enhanced LSTM for Natural Language Inference. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, Diana Inkpen.\n[6] Improving Sentence Representations with Consensus Maximisation. Shuai Tang, Virginia R. de Sa.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJejdUk3qr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a self-supervised sentence embedding approach that incorporates a different view from plain text where some extent of linguistic knowledge is incorporated through the application of tree LSTM. The training procedure is standard contrastive framework where the model is encouraged to distinguish between context sentence (sentences appearing close to the target sentence) and negative samples. Evaluations are conducted on 1) downstream tasks, but with a simple logistic regression model on top of sentence embeddings; 2) probing tasks that more focus on surface information prediction, syntactic and semantic tasks; and 3) qualitative analysis with nearest 5 sentences.\n\nAlthough the experiments are thorough, I am in favor of rejecting this paper with the following reasons:\n\nFirst, the proposed model is trained with 4.6M sentences among 78M available for 33 hours. It is unclear why authors stop the training at this early stage but the results on all three evaluations seem to be inferior to the state-of-the-art by a big margin. I am happy to raise my score if authors can show the results of a well trained proposed model.\n\nSecond, the paper has some room for improvement in terms of clarity, to name a few:\n1) Authors can strengthen the motivation for multi-views learning in related work; \n2) Formula 1 for softmax is wrong;\n3) Contrastive LSTM and contrastive tree LSTM are not clearly defined in the paper, although the former should refer to quick-thoughts and the latter means the proposed method;\n4) In qualitative analysis, for the last example, there is exactly the same candidate with similarity score 0.012. According to cosine similarity, wouldn’t this be 0 and also show up in the baseline model regardless of the embeddings?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to learn sentence representations that incorporates linguistic knowledge in the form of dependency trees using contrastive learning. Experiments on SentEval and probing tasks show that the proposed method underperform baseline methods.

All reviewers agree that the results are not strong enough to support the claim of the paper and have some concerns about the scalability of the implementation. They also agree that the writing of the paper can be improved (details included in their reviews below). 

The authors acknowledged these concerns and mentioned that they will use them to improve the paper for future work, so I recommend rejecting this paper for ICLR.
Decision: Reject