Contrasting distinct structured views to learn sentence embeddings
Authors: Antoine Simoulin
Benoit Crabbé
Venue:  (2021)
Keywords: Sentence
Embeddings
Structure
Contrastive
Multi-views

Abstract:  We propose a self-supervised method that builds sentence embeddings from the combination of diverse explicit syntactic structures of a sentence. We assume structure is crucial to build consistent representations as we expect sentence meaning to be a function from both syntax and semantic aspects. In this perspective, we hypothesize that some linguistic representations might be better adapted given the considered task or sentence. We, therefore, propose to jointly learn individual representation functions for different syntactic frameworks. Again, by hypothesis, all such functions should encode similar semantic information differently and consequently, be complementary for building better sentential semantic embeddings. To assess such hypothesis, we propose an original contrastive multi-view framework that induces an explicit interaction between models during the training phase. We make experiments combining various structures such as dependency, constituency, or sequential schemes. We evaluate our method on standard sentence embedding benchmarks. Our results outperform comparable methods on several tasks.




Under review as a conference paper at ICLR 2021
CONTRASTING DISTINCT STRUCTURED VIEWS TO LEARN SENTENCE EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose self-supervised method that builds sentence embeddings from the combination of diverse explicit syntactic structures of sentence. We assume structure is crucial build consistent representations ing to be function from both syntax and semantic aspects. In this perspective, we hypothesize that some linguistic representations might be better adapted given the considered task or sentence. We, therefore, propose to jointly learn individual representation functions for different syntactic frameworks. Again, by hypothe- sis, all such functions should encode similar semantic information differently and consequently, be complementary for building better sentential semantic embed- dings. To assess such hypothesis, propose original contrastive multi-view framework that induces an explicit interaction between models during the training phase. We make experiments combining various structures such dependency, constituency, or sequential schemes. We evaluate our method on standard sen- tence embedding benchmarks. Our results outperform comparable methods several tasks.
expect sentence mean-
INTRODUCTION
We propose self-supervised method that builds sentence embeddings from the combination of di- verse explicit syntactic structures. Such method aims at improving the ability of models to perform compositional knowledge. In particular, we evaluate the embedding potential to solve downstream tasks.
Building generic sentence embeddings remains an open question. Many training methods have been explored: generating past and previous sentences (Kiros et al.) 2015 Hill et al., (016). discrimi- nating context sentences (Logeswaran & Lee| 2018), predicting specific relations between pairs of
sentences (Conneau et al. 2017 et al. 2019). While all these methods propose efficient training objectives, they all rely on a similar RNN as encoder architecture. Nonetheless, model architectures
have been subject to extensive work well (Tai al. 2015 Zhao al.) [015 Arora et al., 2017) Lin et al. 2017), and in supervised frameworks, many encoder structures outperform standard RNN networks.
We hypothesize structure a crucial element to perform compositional knowledge. In particular, the heterogeneity performances given models and tasks makes us assume that some structures may be better adapted for a given example or task. Therefore, combining diverse structures should be more robust for tasks requiring complex word composition derive their meaning. Hence, aim here to evaluate the potential benefit from interactions between pairs of encoders. In particular, we propose training method for which distinct encoders are learned jointly. We conjecture this association might improve our embeddings" power of generalization and propose an experimental setup to corroborate our hypothesis.
We take inspiration from multi-view learning, which is successfully applied in variety of domains. such framework, the model learns representations by aligning separate observations of the same object. Traditionally, views are issued from a complementary natural perception of the data. For example, a picture and a sound recording of dog. However, can be extended to any pair of samples that share similar semantic content, such the translation of the same sentence in two different languages. The definition can be extended to synthetic views, which are derived from the same unimodal data. In case, we derived multiple views from single sentence by pairing it with

---

Under review as a conference paper at ICLR 2021
distinct syntactic framework. We illustrated in Figure 2 two views derived from the same input sentence by applying respectively a constituent or dependency parser.
As proposed in image processing Tian et al. 2019 Bachman et al, 2019), we propose align the different views using a contrastive learning framework. Indeed, contrastive learning is broadly used in NLP Mikolov al. (2013b ax: Logeswaran & Lee (2018). We proposed enhance the sentence embedding framework proposed in Logeswaran & Lee Lee((2018) with a multi-view paradigm. As detailed in Section 2 composing multiple views has demonstrated its effectiveness in many NLP applications. However, as far as we are aware, combining distinct structured models to build standalone embeddings has not yet be explored. Nevertheless, this paradigm benefits from several structural advantages: as already mentioned, it pairs nicely with contrastive learning. It might thus be trained self-supervised manner that does not require data annotation. Moreover, contrary to models presented in Section 2 our method is not specific to a certain kind of encoder architecture. It does not require, for example, the use of attention layers or tree-structured models. More generally, it could be extended to any notion of view, even in other domains than language processing. Our setup could therefore be extended with any encoding function. Finally, our training method induces an interaction between models during inference and, paramountly, during the training phase.
Our paper is organized as follows: we detail our contrastive multi-view framework in Section In Section 4 we propose evaluation of our framework standard evaluation benchmarks and propose qualitative analysis from our embeddings.
2 RELATED WORK
Multi-view is effectively used in a broad variety of domains. For image processing, some methods aim to learn representations by filling the missing part an image or solving jigsaw puzzles. For video, Tian et al. (2019) propose build image tuples using video frames and flow. For audio, van den Oord et al. (2018) maximize the mutual information between the embedding of the signal different time steps.
Regarding NLP, combining different structural views has already been proven to be successful, Kong & Zhou T01T) provide heuristic to combine dependency and constituency analysis for coreference resolution. Zhou et al. (2016); Ahmed et al. (2019) combine Tree LSTM and standard sequential LSTM with a cross-attention method and observe improvement on semantic textual similarity task. Chen et al. (2017a) combine CNN and Tree LSTM using attention methods on a sentiment classification task, and CNN outperforms both Tree-LSTM and CNN separately. Finally, Chen al. (2017b) combine sequential LSTM and Tree LSTM for natural language inference tasks. However, to our knowledge, combining distinct structured models in a contrastive learning setup was not attempted to build sentence embeddings.
METHOD
Given a sentence s, the model aims at discriminating the sentences s in the neighborhood of from sentences s- outside of this neighborhood. This is contrastive learning (Section 3.1) The representation of each sentence acquired by using multiple views (Section 3.22p
3.1 CONTRASTIVE LEARNING
Contrastive learning is successfully applied in a variety of domains including audio van den Oord et al. (2018), image (Wu et al.] 2018 Tian et al., 2019). video or natural language processing for word embedding Mikolovv et al. (013b) or sentence embedding (Logeswaran & Lee 2018). Some mathematical foundations are detailed in Saunshi et al. (2019). The idea is to build a dataset such that each sample x is combined with another sample xt,, which is somehow close. For word or sentence embeddings, the close samples the words or the sentences appearing in the given textual context. For image processing, close samples might be two different parts of the same image. Systems are trained to bring close sentences together while dispersing negative examples.
In particular, a sentence embedding framework is proposed by Logeswaran & Lee (2018). The method takes inspiration from the distributional hypothesis successfully applied for word, but this
2

---

Under review as a conference paper at ICLR 2021
time, identify context sentences. The network is trained using contrastive method. Given a sentence s, a corresponding context sentence s* and a set of K negative samples $j ... "g* the training objective is maximize the probability of discriminate the correct sentence among negative samples: p(s+| 8..88. ..))))
The algorithm architecture used to estimate p close word2vee (Mikolov al. 2013b ap. Two
sentences encoders f and are defined and the conditional probability is estimated as follow:
(e)) gla
p(5 18,8 (.8k)) ef(s) g(st) + Σ e
At inference time, the sentence representation is obtained as the concatenation of the two encoders and g such + ( In Logeswaran & Lee (2018), f and g are chosen identical and consist two RNN. However, the authors observe that the encoders might learn redundant features. To limit this effect, they propose
distinct set of embeddings for each encoder.
We propose addressing this aspect by enhancing the method with a multi-view framework and using distinet structured model for the encodes and g. We hypothesize that some structures may be better adapted for given example or task. For example, Figure iilussraats that dependency parsing the verb "filled" the root node. Whereas in constituency parsing, subject and verb respec- tively, the right and left child from the root node. Therefore, the combination of different structures should be more robust for tasks requiring complex word composition and be less sensitive lexical variations. Consequently, we propose training procedure that allows the model to benefit from the interaction of various syntactic structures. The choice for the encoder architecture is detailed in the following section.
Agutuaris being played by
(()
the music is filedd withrage and tury.
s(x*)
A ounch guss son the shore
8 (5)
Someone feeding animel
g(xg)
Kmy af
Metric painwiee comperizon
Training loss
Mutt-view
Cantrastive
Figure 1: Contrastive training method. The objective is to reconstruct the storyline. Sentences are presented in their original order. Given an anchor sentence x, the algorithm should identify the context sentence + out of negative samples zj .T Sentences which are composed within pairwise distance matrix. A softmax classifier outputs the probability of each sentence pair to share the same context. In practice, consider two distance matrices given the view or g used to encode the anchor sentence and the target samples. We estimate the final probability the average of the two resulting probabilities.
encoded using separate views,
3.2 LANGUAGE VIEWS
Multi-view aims as learning representations from data represented by multiple independent sets of features. The method is not specific for any particular nature data and can be applied broad scale of domains, making it an efficient framework in self-supervised representation learning. As depicted in Scction [ we generalize the notion of view for a sentence as the application specific syntactic framework. For each view, representation of the sentence into an embedding space.
ad-hoc algorithm that maps the structured
For tree-based views, consider both phrase structure trees and dependency trees. The phrase structure of sentence is represented as nested multi-wond constituents. The dependency tree repre- sents the relationship between individual words. Although equivalences might be derived between
3

---

Under review as a conference paper at ICLR 2021
the two representations schemes, hypothesize that, in our context, the corresponding sequence of operations might allow capturing rather distinct linguistic properties. The various models may, therefore, be complementary and their combination allows for more fine-grained analysis. Together with the trees, we consider the following views of a sentence.
Bag Of Word (Bow) This setup does not assume any underlying structure. The sentence is mod- eled as an unordered set of words. The associated encoding method is simple commutative sum operation of the word emheddings. In our case, vectors are initialized using GloVe vectors (Penning- ton et al., 2014) publicly available_ We used 300-dimensional word vectors trained on the common crawl dataset (840B tokens) with vocabulary of 2.2M case sensitive words.
Vanilla LSTM (SEQ) assumes a sequential structure where each word depends on the previous words in the sentence. The framework is bidirectional sequential LSTTM (Hochreiter & Schmid- huber 1997). The concatenation the forward and backward last hidden state of the model is used as sequence embedding.
Sertence ambodding with Constituency parsing
Sentence embedding with Dependency
filed
Tree LSMM celiient child
Boot made and wrbbedding used the glonal
the
music
is
the
rage
filed
Concd arabadddi age
Word aenieddiiddddggg
and
fury
rege
Jury
Figure 2: (left) The sentence is parsed in constituency and the tree is binarized. The application of the N-Ary Tree LSTM on the obtained structure is represented. (right) The sentence is parsed in dependency and Child-Sum Tree LSTM model is recursively applied. This example illustrates the structural difference between these two views. Dependency parsing articulated around the verb "filled", which is the root node. In constituency, subject and verb are connected through the root node. The two architectures differ as the N-Ary Tree LSTM is structured as a binary tree and differentiate the left and right children while the Child-Sum Tree LSTM might have arbitrary number unordered nodes.
Dependency tree (DEP) In the dependency tree model, words connected through dependency edges. A word might have an arbitrary number dependants. As illustrated in Figure the sentence can be represented words are connected in the dependency tree. In our case, the dependency tree is obtained using the deep biaffine parser from Dozat & Manning 2017 For this view, compute sentence embeddings with the Child-Sum Tree LSTT model described in Tai et al.] (2015): Each node is assigned an embedding given its dependent with recursive function. The recursive node function is derived from standard LSTM formulations but adapted for tree inputs. In particular, the hidden state is computed as the sum of all children hidden states:
tree where nodes correspond to words and edges indicate whether not the
h
Σ hg RECU)
with C(j), the of children of node j. All equations are detailed in Tai et al. (2015). However, we slightly modify the computation h, using Equation As in Zhou al. (201)). we propose to
https:/ /
We use open- source implementation of the parser and replace the pos-tags features with features ob- tained with BERT. Therefore do not need pos-tags annotations to parse our corpus: httpa:://gtthii comm/yhhanccsbiiFFi parser
4

---

Under review as a conference paper at ICLR 2021
compute h, the weighted sum of children vectors in order to allow the model to filter semantically less relevant children.
h Σ
anyhe
(2)
kECU)
The parameters axj are attention weights computed using a soft attention layer. Given a node j. we consider it, .2..-.. h, the corresponding children hidden states. the soft attention layer produces weight og for each child's hidden state. We did not use any external query to compute the attention but instead use a projection from the current node embedding. The attention mechanism is detailed
in equations below:
Oj = w v. + ((9)
(3)
P& W h4 + 6(m)
(4)
0j p
0kj || 2 1lp l1
(5)
akj softmax (a1 anj)
The embedding at the root of the tree is used as the sentence embedding as the Tree LSTM model computes representations bottom up.
Constituency tree (CONST) Constituent analysis describes the sentence as a nested multi-word structure. In this framework, words are grouped recursively in constituents. In the resulting tree, only leaf nodes correspond to words, while internal nodes encode recursively word sequences. The structure obtained using the constituency neural parser from Kitaev & Klein (2018). The frame- work is associated with the N-Ary Tree LSTM, which is defined in Tai et al. 2015). Similarly to the original article, we binarize the trees to ensure that every node has exactly two dependents. The binarization is performed using left markovization and unary productions are collapsed in single node. Again the representation is computed bottom-up and the embedding of the tree root node is used sentence embedding. The equations detailed in Tai al. (2015) make the distinction be- tween right and left nodes. Therefore we do not propose to enhance the original architecture with a weighted sum the DEP view.
EXPERIMENTS
4.1 TRAINING CONFIGURATION
We train our model on the UMBC datase Han et al. 2013)). We filter 40M sentences from the tokenized corpus. We build batches from successive sentences. Given sentence in batch, other sentences not in the context are considered as negatives samples as presented in Section 3.1
For the vocabulary, we follow the setup proposed in Logeswaran & Lee (0018) and we train two models in each configuration. One initialized with pre-trained embedding vectors. The vectors are not updated during training and the vocabulary includes the top 2M cased words from the 300- dimensional GloVe embeddings Penniintonn et al. (2014). The other is limited 50K words initialized with a Xavier distribution and updated during training. For inference, the vocabulary is expanded to 2M words using the linear projection proposed in Logeswaran & Lee (2018); Kiros et al. 2015).
All models are trained using batch size of 400 and the Adam optimizer with 5e learning rate.
Regarding the infrastructure, we use Nvidia GTX 1080 Ti GPU. All model weights are initialized with a Xavier distribution and biases set to O. We do not apply any dropout. For the SEQ view, we use GRU cells instead of LSTM to match the implementation proposed in Logeswaran & Lee (2018) setup.
THypepparaeeters of the models such as the hidden size and the optimization procedure such as leamning rate are fixed given literature on comparable work. In particular Tai et al, (2015): Logeswaran & Lee 2018 and are detailed in Appendix.
Zhu et al. 2015 and traditionally used sentence em- Therefore, we prefer corpus freely available. https://ebs quity. umbe. edu/b edu/biogger umbe www.awebbbeee..e. corpus- of-3b- 1 11
'The bookcorpus introduced in
bedding is no longer distributed. UMBC dataset is available at:
The
ogger /20 /2013/0//3/// C17
5

---

Under review as a conference paper at ICLR 2021
4.2
EVALUATION ON DOWNSTREAM TASKS
MRPC
SICK-R
Model
Dim
Hrs
MR
CR
SUBJ MPQATREC
Acc
F1
MSE
Context sentences prediction
FastSent
500
70.8
78.4
80.6
76.8
72.2
80.3
FastSent + AE
S 500
2
71.8
76.7
88.8
81.5
80.4
71.2
79.1
Skipthought
4800
336
76.5
80.1
93.6
87.1
92.2
73.0
82.0
85,8
79.2
26.9
Skipthought + LN
4800
672
79.4
83.1
93.7
89.3
85.8
78.8
27.0
Quickthoughts
4800
80.4
85.2
93.9
89.4
92.8
84.0
86.8
80.1
25.6
Sentence relations prediction
InferSent
4096
81.1
86.3
92.4
90.2
88.2
76.2
83.1
88.4
DisSent Books 5
4096
80.2
85.4
93.2
90.2
91.2
76.
84.5
DisSent Books 8
4096
79.8
85.0
90.5
93.0
76.1
85.4
Pre-trained transformers
BERT-base [CLS]]
768
96
78.7
84.9
94.2
88.2
91.4
71.1
75.71
BERT-base [NLJ]
768
96
83.6
89.4
94.4
89.9
89.6
76.0
84.4*
Our models (GloVe Ve & Pretrained Embeddings)
SEQ. CONST
4800
41
79.8
82.9
94.6
88.5
90.4
76.4
83.7
86.1
78.9
26.3
DEP, SEQ
4800
27
79.7
82.2
94.4
88.6
91.0
77.9
84.4
86.6
79.8
25.5
DEP, CONST!
4800
39
80.7
83.6
94.9
89.2
92.6
76.8
83.6
87.0
80.3
24.8
Table SentEval Task Results Using Fixed Sentence Encoder. We divided the table into sections. The first range of models is directly comparable to our model as the training objective is to identify context sentences. The second section objective to identify the correct relationship between a pair of sentences. The third section reports pre-trained transformers based-models The last section reports the results from our models. FastSent is reported from Hill et al. (2016). Skipthoughts results from Kiros al. (2013) Skipthoughts LN which includes layer normalization method from Ba et al. (2016). We considered the Quickthoughts results Logeswaran & Lee (2018) with pre- training on the bookcorpus dataset. DisSent and Infersent are reported from Nie et (2019) and Conneau et al. (017)) respectively. Pre-trained transformers results are reported from Reimers & Gurevych| (2019). The Hrs column indicates indicative training time, the Dim column corresponds to the sentence embedding dimension. i indicates models that we had to re-train. Best results in each section are shown in bold, best results overall are underlined. Performance for SICK-R results are reported by convention as p and r 100.
As usual for models aiming to build generic sentence embeddings (Kiros et al. 2013 Hill et al., 2016 |Arora al. 2017] Conneau et al., 2017 Logeswaran & Lee 2018 Nie al. 2019)),)), we use the SentEval benchmark. SentEval is specifically designed to assess the quality of the embeddings themselves rather than the quality of model specifically targeting a downstream task, as the case for the GLUE and SuperGLuee benchmarks (Wang et al. (209b]a a). Indeed, the evaluation protocol prevents for üinetuning the model during inference and the architecture to tackle the downstream tasks is kept minimal. Moreover, the embedding kept identical for all tasks, thus assessing their properties generalization.
Therefore, classification tasks from the SentEval benchmark usually used for evaluation of sentence representation (Conneau Kiela 2018): the tasks include sentiment and subjectivity analysis (MR, CR, SUBJ. MPQA), question type classification (TREC), paraphrase identification (MRPC) and semantic relatedness (SICK-R). Contrasting the results our models on this set of tasks will help to better understand its properties.
We use either the pre-defined train/dev/test splits or perform a 10-fold cross-validation. We follow the linear evaluation protocol of Kiros al. (2015), where logistic regression or softmax classifier
SeatEval tool and data are freely available: Sent

---

Under review as a conference paper at ICLR 2021
Figure 3: Projection of the embeddings from the SUBJ task. (left) The DEP, CONST model is used (right) We train Quick-thought model using scripts from Logeswaran & Lee 2018 the UMBC dataset. Both dimension reductions are performed using the UMAP algorithm. In both cases, samples appear well separated given their labels.
is trained on top of sentence representations. The dev set is used for choosing the regularization parameter and results are reported on the test set.
4.3 EVALUATION ON DOWNSTREAM TASKS
We compare the properties of distinct views combination on downstream tasks and report the results with respect comparable state of the methods in Table The first set of methods (Context sentences prediction) relies on distributional hypothesis: models are trained to reconstruct books storyline.
The second set of models (Sentence relations prediction) is pre-trained on a supervised task. In- fersent[Connee al. (2017) is trained the SNLI dataset, which proposes predict the entailment relation between two sentences. DisSent Nie et al. (2019) proposes generalization of the method and builds a corpus of sentence pairs with more possible relations between them. Finally, include models relying on transformer architectures (Pre-trained transformers) for comparison. In particular, BERT-base model and BERT-model fine-tuned on the SNLI dataset Reimers Gurevych (2019). In Table I observe that our models expressing a combination of views such as (DEP, SEQ) or (DEP, CONST) give better results than the use model. It seems that the entanglement of views benefits the sentence embedding properties. In par- ticular, we obtain ofatt-of-the--oe art results for almost every metric from MRPC and SICK-R tasks, which focus paraphrase identification. For the MRPC task. we gain full point in accuracy and outperform BERT models. We hypothesize structure is important for achieving this task, especially the dataset is composed of rather long sentences. The SICK-R dataset is structurally designed to discriminate models that rely on compositional operations.
same view (SEQ, SEQ) used Quick-Thought
This also explains the score improvement this task. Tasks such MR, CR or MPQA consist in sentiment or subjectivity analysis. We hypothesize that our models are less relevant in this case: such tasks are less sensitive to structure and depend more on individual word lexical variation.
We finally observe our setup is competitive with models trained on broader datasets. Indeed, we use the publicly available UMBC corpus and limited the size to 40M sentences. comparison, the BookCorpus used in Kiros et al. (2015): Logeswaran & Lee 2018) consists in 74M sentences.
4.4 QUALTITATIVE RESULTS
We analyze the embeddings from qualitative perspective and explore the sentences from the SICK- R test set. We retrieved the closest neighbors using cosine distance. We compare the results with the tuick-thought model. We illustrated in Table 2 panel of examples presenting interesting linguistic properties. Models seem somehow robust to adjective expansions illustrated in the first examples. Indeed, the closest expression from "A black bird " is "A bird, which black". However the second

---

Under review as a conference paper at ICLR 2021
Encoder
Query and two closest sentences
Cosine distance
A black bird is sitting dead tree
bird, which is black, is sitting on a dead tree A dead bird is near black man sitting tree
0.118
DEP, CONST
0.139
A bird,, which is black, is sitting on dead tree
0.118
CONST, SEQ
The black bird is sitting in leafless tree
0.143
A bird.. which is black, is sitting on dead tree
0.172
Quickthoughts
A dcad bird is near black man sitting tree
0.172
Rugby is being played by some men
Rugby players are tackling each other
0.381
DEP, CONST
Some men are playing rugby
0.392
Guitar is being played by two men Rugby players are tackling each other
0,401
CONST, SEQ
0.403
Guitar is being played by two men
0,455
Quickthoughts
Rugby players tackling each other
0,462
A crowd of people is near the water
A crowd of people is far from the water
0.079
DEP, CONST
A group of people is near the occan
0.356
A crowd of people is far from the water
0.063
CONST, SEQ
A man is coming out of the water
0.313
A crowd of people is far from the water
0.067
Quickthoughts
Two people wading through the water
0.388
Table 2: A qualitative exploration of the sentence embedding space We embed the sentences from the SICK-R test set. Given query sentence, retrieve the closest two sentences from the dataset using cosine distance. We compare the results the semantic search using distinct views or single views combinations.
retrieved sentence semantically correct for the CONST, SEQ association only. Quick-thought and DEP, CONST present weakness toward word scrambling for this specific example. We investigate passive forms in the second example. The CONST, SEQ and Quickthougth models seem to attach to much weight to the sentence syntax rather than the semantic. This time the association of DEP and CONST views retrieve to corresponding active sentences. Finally, we observe how models behave when facing numeric information. Interestingly Quickthoughts and DEP, CONST able bring together" "crowd" and "group" notions.
From a graphic perspective, we projected in dimensions the sentences from the SUBJ task, for which we obtained state-of-the-art results. We use the UMAP algorithm for dimensionality reduc- tion and compare our multi-view setup with the Quick-thought model. The projection illustrated in Figure 3 While the Figure does not reveal any critical distinction between models, samples appear well separated in both cases.
CONCLUSION AND FUTURE WORK
Inspired from linguistic insights and work on supervised learning, we hypothesize that structure is central element to build sentence embedding. We propose in Section an original contrastive multi- view framework that aims to build embeddings from the interaction various structured models.
proposed to assess the quality of our embeddings and use them feature for the In Section 4
dedicated SentEval benchmark. We obtain state-of-the-art results on tasks which are expected, by hypothesis, to be more sensitive to sentence structure. Exploring our representation space from qualitative perspective, we observe the chosen pair of views might affect the reaction to linguistic perturbations such passive forms or adjective expansions. Some view pairs appear indeed more robust for some specific examples.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mahtab Ahmed, Muhammad Rifayat Samee, and Robert E. Mercer. Improving tree-lstm with tree attention. In 13th IEEE International Conference on Semantic Computing, ICSC 2019, Newport Beach, CA, USA, January 30 February 1, 2019, pp. 247-254, 2019.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough- bo-bett bascline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.064500, 2016.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximiz- ing mutual information across views. Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 15509-15519, 2019.
Liu Chen, Guangping Zeng, Qingchuan Zhang, and Xingyu Chen. Tree-Istm guided attention pool- ing of DCNN for semantic sentence modeling. In 5G for Future Wireless Networks First Inter- national Conference, 5GWN 2017, Beijing, China, April 21-23, 2017, Proceedings, pp. 52-59,
2017a. doi: (00.1007/978--3311-728233
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long doi: 10. 10.18653/v1/P17-1111
Papers, pp. 1657-1668, 2017b.
Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence repre- sentations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018, 2018
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 20/7, pp. 670-680, 2017.
Timothy Dozat and Christopher D. Manning. Deep biaffine attention for neural dependency parsing. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. 2017.
Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Jonathan Weese. Umbc_ebiquins core: Semantic textual similarity systems. In Proceedings of the Second Joins Conference Lexical and Computational Semantics, *SEM 2013, June 13-14, 2013, Atlanta, Georgia, USA, pp. 44 42,52,220 2013.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pp. 1367-1377, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997. doi: 20.1162/neco.1997.
Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Skip-thought vectors. In Advances Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7- 12, 2015, Montreal, Quebec, Canada, pp. 3294-3302, 2015.
Nikita Kitaev and Dan Klein. Constituency parsing with a self-at attentttentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018. Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 6676-2688 2018. doi: 10.18v53/v1/
P18-1249

---

Under review as a conference paper at ICLR 2021
Fang Kong and Guodong Zhou. Combining dependency and constituent based syntactic information for anaphoricity determination in coreference resolution. In Proceedings of the 25th Pacific Asia Conference on Language. Information and Computation, PACLIC 25, Singapore, December 16- 18, 2011, J, pp. 410-419, 2011.
Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings, 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Informa- fion Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013. Lake Tahoe Nevada, United States., pp. 3111-3119, 2013b.
Allen Nie, Erin Bennett, and Noah Goodman. Dissent: Learning sentence representations from explicit discourse relations. In Proceedings the 57th Conference of the Association for Compu- tational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 4497-4510, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1532-1543, 2014.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP. IJCNLP), November 2019.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 5628-5637, 2019.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pp. 1556-1566, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. abs/1906.05844, 2019.
Contrastive multiview coding. CoRR,
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1807 007.03748, 2018
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy. and Samuel R. Bowman. Superglue: stickier benchmark for general- purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 3261-3275, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019b.
Zhirong Wu, Yuanjun Xiong, Stella Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance-level discrimination. CoRR, abs/1805.01978, 2018.
Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In Pro-
ceedings of the Twenty-Fourle International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pp. 4069-4076, 2015.
Yao Zhou, Cong Liu, and Yan Pan. Modelling sentence pairs with tree-structured attentive encoder.
In COLING 2016, 26th International Conference Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan, pp. 2912-2922, 2016.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Com- puter Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 19-27, 2015. doi: 10. 101100/CCV.2015.11
11


Reviews:
[{"review_id": "ksZ0C8m5eNT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThank the authors for showing their effort in revising the paper.\nSome of my concerns have been solved thanks to the rebuttal.\n\n- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - \n\n#### Summary\n\n- This paper proposes a self-supervised (multi-view learning) method that constructs sentence embeddings by exploiting different types of sentence encoders, including tree encoders based on constituency and dependency trees. The intuition is that the integration of respective representations from disparate (linguistically-informed) encoders is effective since each linguistic formalism can provide different linguistic views. In experiments, the proposed method was tested on the SentEval benchmark and demonstrated its effectiveness compared to other baselines. The authors also conducted some qualitative analyses to reveal the inner working of their method.\n\n#### Pros (Reasons to Accept)\n\n- The introduction of (constituency and dependency) tree encoders to sentence embedding construction from the perspective of multi-view learning.\n\n#### Cons (Reasons to Reject)\n\n- Weak novelty: This work attempts to improve upon Logeswaran & Lee (2018) (so-called 'Quickthoughts') by applying tree (and some other) encoders in addition to sequential RNN encoders. However, it is hard to find a novel technical contribution that only appears in this paper, except that the authors demonstrated that the simple combination of two well-known concepts (i.e., Quickthoughts and tree encoders) is empirically effective for building sentence embedding. Even though the results presented in the paper is interesting, in my opinion, there should be a stronger contribution that leads to this paper to be unique and informative.\n- Doubts on the practical usefulness of the proposed sentence embeddings: Compared against typical frameworks that accept just plain sentences as input, the proposed framework requires input sentences to be parsed by external parsers, which might be problematic: Specifically, (1) the reliance on external parsers may cause unexpected error propagations from the parsers, considering that existing parsers are not perfect (in spite of recent improvements in developing good parsers). Moreover, reliable parsers are only available for a few resource-rich languages such as English. Therefore, some concerns about how to parse input sentences appropriately in different (or harsh) conditions should be (at least, briefly) covered in the paper. (2) As tree encoders are hard to be optimized with batch computations, there must be a decrease in inference speed if a model includes tree encoders. Considering this problem, the paper would be much persuasive when the inference speed of each compared model (in addition to its training time) is provided in experiments.\n\n#### Comments\n\n- I find myself being hard to believe that the comparison shown in Table 1 is fair enough, considering the following concerns. First, there are two factors that may impact a model's performance---(A) the architecture of the model and (B) the data used for training the model. Even though (A) is controlled in the presented experiments, (B) is not. In other words, readers cannot infer whether the performacne of a model is due to its architecture or its training data. I understand it's difficult and time-consuming to re-train all baselines. Nonetheless, the proposed model should be at least compared with the Quickthoughts model, which is the foundation of this paper, in the same condition for a fair comparison.\n- To make the paper more convincing, I recommend that the authors test the BERT model in the condition where the dimensionality of the BERT embedding is expanded to larger numbers, making it comparable to those of other models.  \nOne possible solution is to follow the method proposed in John Wieting & Douwe Kiela (2019).\nFor more details, please refer to John Wieting & Douwe Kiela, No Training Required: Exploring Random Encoders for Sentence Classification (ICLR 2019).\n\n#### Misc.\n\n- (Typo) Section 3.1. encodes -> encoders \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cbeO1Xh7MJE", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce a pretrianing paradigm based on contrastive learning between multiple syntactic views of the same sentence. The method maximizes representations between different setence encoders when given the same sentence, and minimize the similarity to all other sentence repre sentations. The results on the infersent benchmark show competitive performance of the approach when compared to non-syntactic pretraining methods.\n\nThere are a couple of concerns I have with the paper and some questions I will elaborate on in detail. In particular, this paper doesn't convince me that incorporating explicit structure is i) desirable and ii) necessary to achieve good results on the chosen benchmarks. I also have doubts about the comparison and the bechmark in general, especially given the fact that random encoders (wo/ explicit syntax) on pretrained word embeddings can do a pretty good job on those [2]. \n\nTherefore, at the current state of the paper I cannot recommend acceptance.\n\nDetailed comments:\n\n1) Are explicit, human designed syntax frameworks really needed? The authors argue that explicit syntax can help generalization, which I could maybe agree with, if trained parsers were perfect. Since they themselves are far from that and regularly fail on complicated sentence structures, I don't believe that this will assist in generalization. Rather the errors in parsing can lead to more systematic failures that the models might have a harder time to correct downstream. Prior work shows that Transformers trained on a (masked) LM objective learn about sentence structure implicitly [1]. In fact, although the authors \"hypothesize\" that explicit structure can help -- which they do not properly ablate in my opinion (see 2-3) --, the results in the paper show no clear advantages on their benchmarks when comparing to BERT. The increased complexity of the method (ie., relying on other trained models) should be warrented by showing a clear advantage in doing so.\n\n2) Fair comparisons: The models in the paper are trained on another pretraining corpus. Other baselines should have been trained on this corpus to have an apples to apples comparison. Furthermore, BERT models use much lower dimensional sentence embeddings which typically hurts performance. A way to make comparison fairer would be to apply a random projection of the 768 features to 4800 features. Note that this can have a dramatic impact even for BoW models on these benchmarks [2]. I also wonder how well a randomly initialized model would perform, that is, how much does the pretraining actually help? [2] shows that it might not be required at all. I think given the inductive bias coming from the parser and the use of TreeLSTMs, this effect might even be greater, because much of the information about the structure sentence is given apriori, which might even help random models to achieve better performance.\n\n3) What can we expect the pretraining to learn? For instance, wouldn't a satisfactory solution to the pretraining task for the 2 encoders to simply learn a bag-of-words representation? I believe that the large majority of the setences could easily be distinguished by that. Just memorize the exact words that occured in the sentence. I am not convinced from the paper that the model learns anything more semantic than that. More rigorous ablations are required to show the benefit of the pretraining method.\n\n\nMy personal opinion on introducing explicit syntax into neural nets (note that this will not be part of my decision): \n\nGiven all the evidence we have so far on pretraining LMs using generic model such as transformers, I just don't see any reason why we should still try to explicitly fit our potentially faulty and biased syntactic frameworks into neural networks when more generic models (such as transformers) can learn structure directly from the data. Our linguistic frameworks can still be super useful for understanding and systematically testing our models, but I think we should refrain from  introducing our potentially limited understanding of language into those models. A random thought: How would pretrained parsers help a model on twitter like text or child speech? \n\n\n[1] Jawahar et al., What Does BERT Learn about the Structure of Language? ACL, 2019.\n\n[2] Wieting et al., NO TRAINING REQUIRED: EXPLORING RANDOM ENCODERS FOR SENTENCE CLASSIFICATION. ICLR 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "BuJXY_ctPBz", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed an ensemble of sentence encoders (RNN, BoW, tree-based) that is trained with contrastive loss. The proposed approach is a simple extension of the work of Logeswaran & Lee (2018). Overall, I think the paper has several major weaknesses. \n\nFirst, the results on SentEval are not convincing in comparison to other general purpose methods (i.e., BERT or QuickThought) which do not require a parsed tree during training and inference. The results also do not justify the use of additional resources (dependency and constituency parsers) as it complicated the models and is not applicable for many other languages where parsers are not available or their performances are still far from usable. Note that, there are several pre-trained models after BERT (e.g., XLNet, XLM-R, ALBERT,...) which can be used to obtain sentence embeddings. I think the authors should also compare the results with those models.\n\nSecond, SentEval framework suffers from some drawbacks [1]. Using it alone is insufficient to draw any meaningful conclusion about sentence embeddings. I think that the authors could have used probing tasks [2] to evaluate their sentence embeddings with respect to linguistic properties.\n\nThird, while the paper claims posed a hypothesis that structure is crucial to build consistent representations, this is not shown in the paper. I think that the paper lacks some analysis/examples  to show in which cases modelling structure of the sentence explicitly is necessary. Other models that use Transformers or RNN exploit structure in language implicitly in one way or another. For example, Hewitt and Manning shows that syntax is embedded in deep models (i.e., BERT). Thus It’s not clear to me what is the advantage of the proposed approach in comparison to other models.\n\nComments for the authors:  I think that the comparison with previous work in table 1 is not meaningful since the authors used another dataset for training.\n\n\n\n[1] [Pitfalls in the Evaluation of Sentence Embeddings](https://www.aclweb.org/anthology/W19-4308/). Steffen Eger, Andreas Rücklé, Iryna Gurevych. RepL4NLP-2019\n\n[2] [What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties](https://www.aclweb.org/anthology/P18-1198/). Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, Marco Baroni. ACL 2018\n\n[3] [A Structural Probe for Finding Syntax in Word Representations](https://www.aclweb.org/anthology/N19-1419/). John Hewitt Christopher D. Manning. NAACL 2019\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors both for going the extra mile in doing further experiments for their response, and making the efforts to synthesize the main comments and concerns of the reviewers.

Overall, I'm pretty sympathetic to the idea that syntactic and semantic representations should be very helpful to learning sentence embeddings. They provide a form of scaffolding. But a reviewer notes and I think anyone will admit in 2020 that contextual language models like BERT also provide much of this scaffolding, and it falls to the paper author to provide convincing evidence that using external parsers is valuable and necessary in this quest. In this, the current paper seems to fall somewhat short.

Pros:
 - Clearly and honestly written paper
 - Good exploration of value of constituency & dependency parse representations
 - Exploits recent work in contrastive learning

Cons:
 - Insufficient novelty
 - Experimental comparisons not well controlled – too much apples and oranges. 
 - No comparisons of inference speed tradeoffs
 - Value of exploiting explicit syntax is too much assumed rather than explored
 - It's not established that use of explicit syntax really delivers versus alternatives such as contextual language models

Several of the reviewers felt that this paper was a fairly limited extension of L & L 2018, without any clearly novel contribution. The issue of comparability in results is complicated. There is a reason to move to a new standard corpus, rather than privileged people passing around archived copies of the old BooksCorpus, and I think your additional experiments show the results are "near enough" but there would still be much more archival value in a new paper having a set of comparable results on a new corpus. The big question of whether to do this or use BERT is better addressed in your additional experiments presenting a random projection of BERT to a comparable higher dimensional space. But unfortunately these results further weaken the clarity of the case for needing to head in the direction of this paper rather than just using a large pre-trained contextual LM.
Decision: Reject