Unsupervised Distillation of Syntactic Information from Contextualized Word Representations
Authors: Shauli Ravfogel
Yanai Elazar
Jacob Goldberger
Yoav Goldberg
Venue:  (2020)
Keywords: dismantlement
contextualized word representations
language models
representation learning

Abstract: Contextualized word representations, such as ELMo and BERT, were shown to perform well on a various of semantic and structural (syntactic) task. In this work, we tackle the task of unsupervised disentanglement between semantics and structure in neural language representations: we aim to learn a transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups of sentences which are structurally similar but semantically different, and use metric-learning approach to learn a transformation that emphasizes the structural component that is encoded in the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations by showing that they outperform the original contextualized representations in few-shot parsing setting.

Under review as a conference paper at ICLR 2020
UNSUPERVISED DISTILLATION OF SYNTACTIC INFOR- MATION FROM CONTEXTUALIZED WORD REPRESEN- TATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Contextualized word representations, such as ELMo and BERT, were shown to perform well on various semantic and structural (syntactic) task. In this work. we tackle the task of unsupervised disentanglement between semantics and struc- ture neural language representations: we aim to learn transformation of the contextualized vectors, that discards the lexical semantics, but keeps the structural information. To this end, we automatically generate groups sentences which are structurally similar but semantically different, and use metric-learning approach learn a transformation that emphasizes the structural component that is encoded the vectors. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics. Finally, we demonstrate the utility of our distilled representations showing that they outperform the original contextualized representations in a few-shot parsing setting.
INTRODUCTION
Human language is complex system, involving an intricate interplay between meaning (seman- tics) and structural rules between words and phrases (syntax). Self-supervised neural sequence mod- els for text trained with language modeling objective, such as ELMo (Peters et al. 2018), BERT (Devlin al. 2019), and RoBERTA (Liu et al.) 2019b), were shown to produce representations that excel in recovering both structure-relatee information (Gulordava et al. 2018 van Schijndel &
Linzen Wilcox al. 2018 Goldberg 2019) well as in semantic information Yang al. 2019
Joshi et al. 2019)).
In this work, study the problem of disentangling structure from semantics in neural language representations: we aim to extract representations that capture the structural function of words and sentences, but which are not sensitive to their content. For example, consider the sentences:
Neural networks are interesting.
3. I study neural networks.
2. Maple syrup is delicious.
4. John loves maple syrup.
While (1) and (2) are different in content, they share a similar structure, the corresponding words in them, while unrelated in meaning serve the same function. Similarly for sentences (3) and (4). In contrast, sentence (1) shares the phrase neural networks with sentence (3) and maple syrup is shared between (2) and B While the occurrences of each phrase share the meaning, they are used in different structural (syntactic) configurations, serving different roles within the sentence (appearing in subject vs object position) We seek a representation that will expose the similarity between "networks" in (1) and "syrup" in (2) while ignoring the similarity between "syrup" in (2)
and "syrup" in (4).
In this work we focus on English.
We focus on lexical semantics.
'There is syntactic distinction hetween the two, with "maple" being part of compound and "neural" heing an adjective. However, we focus in their similarity as noun modifiers in both phrases.
"These differences in syntactic position are also of relevance to language modeling, as different positions may pose different restrictions the words that can appear them.

---

Under review as a conference paper at ICLR 2020
We aim to learn a function from contextualized word representations to space that exposes these similarities. Crucially, we aim to do this in an unsupervised manner: we do not want to inform the process of the kind of structural information want to obtain. We do this by learning a transforma- tion that attempts to remove the lexical-semantic information in a sentence, while trying to preserve structural properties.
Disentangling syntax from lexical semantics in word representations is a desired property for sev- eral reasons. From a purely scientific perspective, once disentanglement achieved, one can better control for confounding factors and analyze the knowledge the model acquires, e.g. attributing the predictions of the model to one factor of variation while controlling for the other. In addi- tion to explaining model predictions, such disentanglement can be useful for the comparison of the representations the model acquires to linguistic knowledge. From a more practical perspective, disentanglement can be a first step toward controlled generation/paraphraaing that considers only aspects of the structure, akin to the style-transfer works in computer vision, i.e., rewriting a sentence while preserving its structural properties while ignoring its meaning, vice-versa. It can also in- form search-based application in which one can search for "similar" texts while controlling various aspects of the desired similarity.
To achieve this goal, we begin with the intuition that the structural component in the representation (capturing the form) should remain the same regardless of the lexical semanties of the sentence (the meaning). Rather than beginning with a parsed corpus, we automatically generate a large number of structurally-sime -iimilr sentences, without presupposing their formal structure ($3.1) This allows to pose the disentanglement problem as a metric-lcarning problem: we aim to learn a transformation of the contextualized representation, which is invariant to changes in the lexical semantics within each group of structralll-ssmilaa sentences ($3.3) We demonstrate the structural properties captured by the resulting representations in several experiments structurally-singg similar words and few-shot parsing.
among them automatic identification of
RELATED WORK
The problem of disentangling different sources of variation has long been studied in computer vision. and was recently applied to neural models (Bengio al. 2013 Mathieu al., 2016 Hadad al. 2018). Such disentanglement can assist in learning representations that are invariant to specific factors, such as pose-invariam invariant rice-recognition (Peng et 2017) or style-invariant digit recognition Narayanaswamy et al., 2017). From generative point of view, disentanglement can be used to modify one aspect of the input (e.g., "stylee),, while keeping the other factors (e.g.. "content") intact, as done in neural image style-transfer (Gatys 2017).
In the field of NLP, disentanglement much less researched. In controlled natural language genera- tion and style transfer, several works attempted to disentangle factors of variation such as sentiment or age of the writer, with the intention to control for those factors and generate new sentences with specific properties,
transfer existing sentences similar sentences that differ only in the those properties. Several works (Sohn et al.) 2015) Ficler & Goldberg 2017) have trained conditional generative models by explicitly conditioning a decoder network with vector of attributes. On train- ing, the attributes derive from the training sentence, while in testing the conditioning vector can be set to generate a text with the desired attributes. Other works Fu et al. (2018); Hu et al. (2017) aim to achieve style transfer (as opposed to generation) by explicitly training representations that are invariant to the controlled attributes (e.g. by co-training of generator and attribute discriminator). A decoder generates the transferred text from the disentangled representation and from an explicit attribute representation. Lample al. (0018) use a conditioned mack-translation approach to achieve similar goal. While these works try to disentangle sentence-level attributes, in this study we focus on disentangling between two components in the representations of individual words.
Several works examine the way semantic and syntactic information is distributed across the layers of neural models of text (Blevins et al. 22018] Tenney et al. (2019). They use diagnostic classifiers (Adi et al. 2016 Hupkes al. 2018) to predict syntactic properties and demonstrate that different parts of the model encode information in different levels of abstraction (e.g. POS information, dependency label and semantic role). Liu et al. (2019a) used diagnostic classifiers trained to predict various syntactic and semantic properties from ofate-ofthh--un LMs representations, and demonstrated that many syntactic and semantic distinctions encoded in the probed representations, Clark et al.
2

---

Under review as a conference paper at ICLR 2020
2019) probed the attention patterns of BERT, and showed that individual attention-heal focus in peliattiially-mmeminnn relations in the input.
Beyond the descriptive level, recent works have focused supervised extraction of syntax-related representations from neural representations. [Atette et al. (2018) used a linear transformation, in- spired from the notion of "similarity order" in classic word representation learning, to tailor uncon- textualized word representations to syntactic vs. semantic tasks. Hewitt & Manning (2019) demon- strated that it is possible to train a linear transformation, under which squared euclidean distance between transformed contextualized word vectors correspond to the distances between the respec- tive words in the syntax tree that represents the hierarchical structure of the sentence. Concurrent to this work, Li & Eisner (2019) have used a variational estimation method (Alemi et al. (016) of the information-bottleeedd principle (Tishby et al. 1999) to extract word embeddings that are useful to the end task of parsing.
While impressive, those works presuppose specific formal syntactic structure (e.g. annotated parse trees following specific linguistic annotation schema) and use this syntactic signal to learn struc- tural information in a supervised manner. In other words, these works assume a given structure, and use supervision to make the structural information more salient, mapping (or forcing) the neural representations to known linguistic properties. In contrast, we aim to expose the structural informa- tion encoded in the network in unsupervised manner, without pre-supp existing syntactic annotation scheme.
METHOD
Our goal is to learn a function f : IR" + R", which operates on contextualized word representations a and extracts vectors / (x) which make the structural information encoded in x more salient, while discarding as much lexical information possible. In the sentences "Maple syrup is delicious" and "Neural networks are interesting", we want to learn a f such that f(vg)p)) se f ((llewwrrss) where Wooed is the contextualized vector representation of the word in sentence i. We also want f (r ~ ft (e while keeping (e & f (Unewworks)
Moreover, we would like the relation between the words "maple" and "delicious" in the sec- ond sentence, to be similar to the relation between "neural" and "interesting" in the first sen- wairr""eeurll "interesting). ). Operativly, we represent pairs of words (x, y) by the difference between their transformation (xx fy)) and aim to learn I that preserves: (uiupl)) ((@eliciouu) x= ((iieura)) - (Uimeresting The choice to represent pairs this way was in- spired by several works that demonstrated that nontrivial semantic and syntactie relations between uncontextualized word representations can be approximated by simple vector arithmetic (Mikolov
tence: pair( maple v.2 (dclicious) z pair(": n
al. 2013a b Levy & Goldberg 2014).
To learn f. we start with groups of sentences that the sentences within each group are known to share their structure but differ in their lexical semantics. We call the sentences in each group structurally equivalent. Figure [ shows an example of two structurally equivalent sets. Acquiring such sets challenging, especially if we do not assume known syntactic formalism and cannot mine for sentences based on their observed tree structures. To this end, we automatically generate the sets starting with known sentences and sampling variants from language model ($B.I). Our sentence- set generation procedure ensures that words from the same set that share an index also share their structural function. We call such words corresponding.
When a train ticket is ourchased, contract established
Shapley participated in the debate with heber d
When a travel document is acquired, a settlement is declared
Khan joined In the sllent discussion "with
When a winning vehicle obta ned, a compelition is introduced
Parker figured in the "coming showdown "wit block leader
When winning bid is announced. winner is created
Moore engaged the modern struggle "with joseph israet
Two groups of seuucturally-qqiiallee sentences. In each group, the first sentence is origi- nal sentence from Wikipedia, and the sentences below were generated by the process of repeated BERT substitution. Some sets of corresponding words-that is, words that share the same structural function-are highlighted in the same color.
Figure 1:
3

---

Under review as a conference paper at ICLR 2020
We now proceed to learn function f to map contextualized vectors of corresponding words (and the relations between them, as described above) to neighbouring points in the space.
We train f such that the representation assigned to positive pairs pairs that share indices and come from the same equivalent set - is distinguished from the representations of negative pairs -challenging pairs that come from different sentences, and thus do not share the structure of the original pair, but can, potentially, share their lexical meaning. We do so using Triplet loss, which pushes the representations of pairs coming from the same group closer together ($3.3) Figure sketches the network.
Tripel VP.)
v* finetworks) Hinteresting)
VP. Risyrpp) cidul
V finetworks) ftfascinating)
notworks are interssinn
Maple syoup delicizus
They found networks be faseinntion
Anchor sentence
Pos sentence
Negative sentence
An illustration of triplet-loss calculation. Pairs of words are represented by the difference between their transformation f. which is identical for all words. The pairs of words in the anchor and positive sentences are lexically different, but structurally similar. The negative example presented here is especially challenging, it is lexically similar, but structurally different.
Figure 2:
3.1 GENERATING STRUCTURALLY SIMILAR SENTENCES
In order to generate sentences that approximately share their structure, we sequentially replace con- tent words in the sentence with other content words, while aiming to maintain the grammatically of the sentence, and keep its structure intact. Replacing words with words of the same POS, as done in Gulordava et al. (2018), does not answer to those requirements, as this method does not respect various restrictions that apply to words that share different function within the sentence. For exam- ple, verb argument structure dictates limitations on the arguments the predicate receives, and verbs differ in properties such whether or not they accept complement.
Replacing verbs with other verbs does not guarantee fulfilling these limitations. Since we do not want to rely on syntactic annotation (apart from the level of POS tags) when performing this placement, we opted to use pre- pre-trained language model BERT - under the assumption that strong neural language models do implicitly encode many of the syntactic restrictions that apply words in different grammatical functions (e.g., we assume that BERT would not predict transitive verb in the place of intransitive verb, verb that accepts a complement not accept a complement). While this assumption seems to hold with regard to basic distinctions such transitive vs. intransitive verbs, its validity is less clear in the more nuanced cases, in which small differences in the surface level can translate to substantial differences in the deep structure such as replacing a control verb with a raising verb. This limitation of the current approach, although we find that the average sentence we generate is grammatical and similar in structure to the original sentence, Moreover, as our goal is to expose the structural similarity encoded neu- ral language models, we find it reasonable to only capture the distinctions that are captured by state-of-ther ahe neural language model.
the place of a verb that does
Concretely, we rely on BERT masked LM model. We start each group with Wikipedia sentence, for which we generate k 6 equivalent sentences by iterating over the sentence from left to right sequentially, masking the word, and replacing with one of BERT's top-30 predictions. Crucially, to increase semantic variability, we perform the replacement in place (online), that is, after randomly choosing a guess w, we insert w to the sentence at index i, and continue guessing the i 1 word based the modified sentence We exclude closed set of a few dozens of words (mostly function words) and keep them unchanged in all & variations of a
Implementation
'We note that this process hears some similarity to Gibbs sampling from BERT conditioned L.M.
4

---

Under review as a conference paper at ICLR 2020
sentence. To the extent that BERT learns recover corrupted sentences by suggesting replacements that respect the probability distribution of actual natural language, the suggestions would be both semantically and structurally correct. We further maintain structural correctness by maintaining the POS, and encourage semantic diversity by the auto-regressive replacement process. We find that the average sentence is grammatical and maintain the structure of the original sentence, although some generated sentences violate these requirements. In the appendix !?? we show some additional generated groups, and highlight some recurring errors.
The sets in Figure were generated using this method.
3.2 WORD REPRESENTATION
We use the method generate = 150, 000 equivalent sets E, of structurally equivalent sentences, and collect the contextualized vector representations of words in these sets, resulting in 1,500,000 training pairs and 200,000 evaluation pairs for the training process of f. We experiment with both ELMo and BERT-based contextualized representations. In average, group of equivalent sentences. For ELMo, we represent each word in context as a concatenation of the last two ELMo layers (excluding the word embedding layer, which is not contextualized and therefore irrelevant for structure), resulting in representations concatenate the mean of the words' representation across all 23 layers of BERT-Large. with the representation of layer 16, which was found by Hewitt & Manning 2019) most indicative of syntax.
sample 11 pairs from each
dimension 2048. For BERT, we
3.3 TRIPLET LOss
We learn the mapping function f using triplet loss (Figure
Concretely, given group of equivalent sentences E,. we randomly choose two sentences to be the anchor sentence 5.A, and the positive sentence S and sample two different word indices {fi. Let 5 ii] be the contextualized representation of the i, word in sentence 5 The words S and [iz] from the anchor sentence would form a representation of a pair of words, which should be close to the pair S [ia].S'' from the positive sentence.
We represent pairs as their differences after transformation, resulting in the anchor pair VA and positive pair VP:
V f(SA - f(S
S E E,
V =((S (())((z))
S E E,
where / is the parameterized syntactic transformation we aim to learn. We also consider a negative pair:
V = -(s - -(S (12)
S g E,
coming from sentence S: which is not in the equivalent set.
As f has shared parameters for both words in the pair, it can thus be considered a part of a Siamese network, making our learning procedure an instance of triplet Siamese network [Schroff et al. 2015). We choose f to be a simple model: a single linear layer that maps from dimensionality 2048 to 75. The dimensional of the transformation was chosen according to development set performance.
We use triplet loss (Schroff et al., (2015) to move the representation the anchor vector V closer to the representation of the positive vector P and farther apart from the representation of the negative vector V Following Hoffer & Ailon 2015), we calculate the softmax version of the triplet loss:
ddist(VA.V ,V")
triplet vP VN)= edist(VA, edist(VA,VV))
(4)
where (ist((x,1) 3)
y is the cosine-distance between the vectors x and y. Note that triplet - 0 dist(vA_ [ dist(Va.VV - O, expected. The triplet objective is optimized end-to-end using
5

---

Under review as a conference paper at ICLR 2020
the Adam optimizer (Kingma & Ba 2015). We train for epochs with mini-batch of size 500 and take the last model as the final syntactic extractor. During training. the gradient backpropagates through the pair vectors to the parameters f of the Siamese model, to get representations of individ- ual words that are similar for corresponding words in equivalent sentences. We note that we do not the gradient the contextualized vectors: we keep them intact, and only adjust the learned transformation.
Hard negative sampling We obtain the negative vectors V using hard negative sampling. For each mini-batch B, we collect 500 {V²', V.") pairs, each pair taken from an equivalent set E, The negative instances V.A are obtained by searching the batch for vector that is closest to the anchor and comes from different set:
arg min (Vs VYA) V
(5)
where dist is again the cosine distance. In addition, we enforce a symmetry between the anchor and positive vectors, by adding pair (positive, anchor) for each pair (anchor, positive) in B.
That is, V.N is the "most misleading" word-pair vector: it comes from a sentence that has a different structure than the structure of V:² sentence, but is the closest to V.A in the mini-batch
EXPERIMENTS AND ANALYSIS
We have trained the syntactic transformation f in way that should encourage it to retain the struc- tural information encoded in contextualized vectors, but discard other information. We assess the representations our model acquired in an unsupervised manner, by evaluating the extant to which the local neighbors of each transformed contextualized vector f(x) share known structural proper- ties, such as grammatical function within the sentence. For the baseline, we expect the neighbors of each vector to share mix of semantic and syntactic properties. For the transformed vectors, we expect the neighbors to share mainly syntactic properties. Finally, we demonstrate that in a few- shot setting, representations outperform the original ELMO representation, indicating they indeed distilled from syntax, and discard other information that is encoded in ELMO vectors but is irrelevant for the extraction of the structure of sentence.
Corpus For training the transformation f. we rely on 150,000 sentences from Wikipedia, tokenized and POS-tagged by spaCy R The POS tags are used in the equivalent set generation to filter replace- ment words. Apart from POS tagging, we do not rely on any syntactic annotation during training- The evaluation sentences for the experiments mentioned below are sampled from a collection of 1,000,000 original and unmodified Wikipedia sentences (different from those used in the model training).
4.1 QUALITATIVE ANALYSIS
(-SNE Visualization Figure 3 shows a 2-dimensional t-SNE projection (Maaten & Hinton 2008) of 15,000 random content words. The leftt panel projects the original ELMo states, while the right panel is the syntactically transformed ones. The points are colored according to the dependency label (relation parent) of the corresponding word, assigned by the spacy parser.
As can be seen, in the original ELMo representation most states apart from those characterized by a specific pfrt-of-speech, such as amod (adjectives, in orange) or nummod (numbers, in light green) do not fit well into a single cluster. In contrast, the syntactically transformed vectors are more neatly clustered, with some clusters, such as direct objects (brown) and prepositionallohjeets (blue), that are relatively separated after, but not before, the transformation. Interestingly, some functions that used to be a single group in ELMo (like the adjectives in orange, or the noun-compounds in
A large enough mini- batch is necessary to find challenging negative examples.
We implicitly assume that any pair coming from different group of equivalent sentences is valid negative example- that is, does not share the structural relation that exists between the anchor pair's words. This is relatively mild assumption, as due to sparsity, in high probability two different sentences not share the very same structure
hhttps:/sspacy. ío/

---

Under review as a conference paper at ICLR 2020
ELMo
Transformed
Figure 3: t-SNE projection of ELMO states, colored by syntactic function, before (left) and after (right) the syntactic transformation.
green) now split into several clusters, corresponding to their use in different sentence positions, separating for examples adjectives that are used in subject positions from those in object position or within prepositional phrases. Additionally, as noun compounds ("maple" in "maple syrup") and adjectival modifiers ("tasty" in "tasty syrup") are relatively structurally similar (they appear between determiners and nouns within noun phrases, and can move with the noun phrase to different posi- tions), they split and grouped together in the representation (the green and orange clouds).
To quantify the difference, we run K-means clustering on the projected vectors, and calculate the average cluster purity score the relative proportion of the most common dependency label in each cluster. The higher this value is, the more the division to clusters reflect division to grammatical functions (dependency labels). We run the clustering with different K values: 10, 20, 40, 80. We find an increase in class purity following our transformation: from scores of 22.6%, 26.8%, 32.6% and 36.4% (respectively) for the original vectors, scores of 24.3%, 33.4%, 42.1% 1% and 48.0% (respectively) for the transformed vectors.
Examples Below (NT) the transformation. Note the high structural similarity of the entire sentence, as well as the function of the word within it (Q1: last word of subject NP in middle clause, Q2: possessed noun in sentence initial subject NP, Q3: head of relative clause of a direct object):
few query words (Q) and their closest neighbours before (N) and after
Q:in this way of thinking, impacting projectile goes into an ice-rich layer but no further. Nothey generally have a pre-engraved rifling band to engage the rifled launch tube, spin-stabilizing the projectile, hence the term rifle" NT: achieve a large explosive yield, a linear implosion weapon needs more material, about 13 kgs.
Q: the mint 's director at the time, nicolas peinado was also an architect and made the initial plans.
N: the director is angry at crazy loop and glares at him even trying to get a woman to kick crazy loop out of the show which goes unsuccessfully). ).
NT: jetley 's mother, kaushaliya rani, was the daughter of high court advocate shivram jhingan
Q: their first project is software that lets players connect the company 's controller their device
N: you could try use norton safe web, which lets you enter website and show whether there seems be anything bad in it.
NT: the city offers route-finding website that allows users to map personalized bike routes
4.2 QUANTITATIVE EVALUATION
We expect our transformed vectors to capture more structural and less lexical similarities than the source vectors. We expect each vectors' neighbors in space to share the structural function of the word over which the vector was collected, but not necessarily share its lexical meaning. We focus on the following structural properties:

---

Under review as a conference paper at ICLR 2020
Dep. edge
Hend's dep. odge
Tree path (complete)
Tree path
Tree path
Depth (correlation)
Lexical Match
(L=3)
(L=2)
Baseline (all)
0.580
0.473
0.166
0.353
0.566
0.448
0.736
Transformed (all)
0.699
0.603
0.253
0.523
0.735
0.561
0.284
Transformmd untrained (all)
0.142
0.528
0.407
Baseline (dilliult)
0,509
0.460
0,160
0.347
0.564
0.430
0.776
Transformed (difficultt
0.671
0.591
0.260
0.534
0.751
0.576
0.274
Table 1: Results in the closest-word queries, before and after the application of the syntactie trans- formation. "Basline" refers to unmodified ELMo vectors, "Transformed" refers to ELMo vectors after the learned syntactic transformation f. and trransformed-untraineee refers to ElMo vectors, after a transformation that was trained on randomely -initaliii ELMo. "Difficult" refers to eval- uation on the subset of POS tags which are most structurally diverse.
Dependency- edge of a given word (dep-edge), that represents its function (subject, object etc.)
The dependency edge of the word parent's (head's dep-edge) in the tree to represent higher level structure, such as a subject that resides within relative clause, as in the word 'man" in the phrase "the child that the man saw".
Depth in the dependency tree (distance from the root of the sentence tree).
Constituency-parse paths: Consider, for example, the sentence "They saw the moon with the telescope". The word "Telescope" is a part of a noun-phrase "The telescope", which resides inside a prepositional phrase "with the telescope", which part of the Verbal phrase ""Saw with the telescope". The complete constituency path for this word is therefore "NP- PP-VP", We calculate the complete tree path to the root seeee-att-cooplee as well as paths limited to lengths 2 and 3.
For this evaluation, we parse 400,000 random sentences taken from the 1-million-sentencee Wikipedia sample, run ELMo and BERT to collect the contextualized representations of the sen- tences' words, and randomly choose 400,000 query word vectors (excluding function words). We then retrieve, for each query vector æ, the value vector that is closest to æ in cosine-distance and record the percentage of closest-vector pairs (z, y) that share each of the structural properties listed above. For the tree depth property, we calculate the Pearson correlation between the depths of the queries and the retrieved values. We use Spacy parser for dependency-parsinng and the Berkeley Neural Parser (Kitaev & Klein| 2018) for constituency parsing. We exclude function words from the evaluation.
Easier and Harder cases The baseline models tend to retrieve words that are lexically similar. Since certain words tend to appear at above-chance probability in certain structural functions, this can make the baseline be "right for the wrong reason", as the success in the closest-word test reflects lexical similarity, rather than grammatical generalization of the model. To control for this confound- ing, we sort the different POS tags according to the entropy of their dependency- labells distribution, and repeat the evaluation only for words belonging to those POS tags having the highest entropy (those POS tags are the most structurally variant, and tend to appear in different structural func- tions). We find that the performance of the baselines (ELMo, BERT models) those words drops significantly, while the performance of our model are only mildly influenced, further indicating the superiority of our model in capturing structural rather than lexical information.
Results The results for ELMo are presented Table For BERT, we witnessed similar, but some- accuracy, 56.5% head's dependency-
what lower, accuracy: for example, 68. 1% dependency-cdee edge accuracy, and 22. 1% complete constituency-pee path accuracy. The results for BERT are available in the appendix $[ and for the reminder of the paper, we focus in ELMo. We observe significant improvement over the baseline for all tests. The correlation between the depth tree of the query and the value words, for examples, rises from 44.8% to 56.1 9% indicating that our model encourages the structural property of the depth of the word to be more saliently encoded in its representation compared with the baseline. The most notable relative improvement is recorded with regard to full constituency path to the root: from 16.6% before the structural transformation, to 25.3% after it an improvement of 52%. In addition to the increase in syntax-related properties, we observe sharp
8

---

Under review as a conference paper at ICLR 2020
drop- from 73.6% to 28.4% in the proportion of query-value pairs that are lexically identical (lexical match, Table 4). This indicates our transformation f removes much of the lexical informa- tion, which is irrelevant for structure. To assess to what extent the improvements stems from the information encoded in EL,Mo, rather than being an artifact of the triplet-loss training, we also eval- uate transformation f that was trained baseline (Conneau et al., 2018). We find this model performs substantially worse than the baseline (Table 4 TTranformedduntrraine (all)").
candomly-initiaiian ELMo, surprisingly strong
4.3 MINIMAL SUPERVISION FOR STRUCTURE DISTILLATION: FEW-SHOT PARSING
The absolute nearest-neighhour accuracy values may appear to be relatively low: for example, only 67.6% of the (query, value) pairs share the same dependency edge.
As the model acquires its representation without being exposed to human-mandatee mandated syntactic con- vention, some of the apparent discrepancies in nearest neighbours may be due to the fact the model acquires different kind of generalization, or learned a representation that emphasizes different kinds of similarities. Still, we expect the resulting (75 dimensional) representations to contain distilled structure information that is mappable to human notions of syntax. To test this, we compare seannddecy-ppaseee trained on our representation and on the source representation. If our repre- sentation indeed captures structural information, we expect to excel on a low data regime. To this end, we test our hypothesis with few-shot dependency parsing setup, where we train model to predict syntactic trees representation with only a few hundred labeled examples.
We use an off-the-shelff dependency parser DDozat & Manning 2016) and first swap the pre-trained Glove embeddings (Pennington et al., 2014) with ML contextualized embeddings (Peters et al.] 2018). In order to have fair comparison with our method, last layers of Elmo; we refer to this experiment as elmo. As our representation is much smaller than ELMo's (75 as opposed to 2048), a potential issue for low data regime is the high parameter number to optimize in the later case, therefore a lower dimension can achieve better results. We design two additional baseline experiments to remedy this potential issue: (1) Using PCA in order reduce the representation dimensionality. We randomly chose IM words from Wikipedia, calculated their representation with ELMo embeddings and performed PCA. This transformation applied during training on top of ELMo representation while keeping the 75 first components. This experiment referred to as elmo-pea. This representation should perform well if the most salient information in the ELMo representations are structural. We exepet it to not be the case. (2) Automatically
use the concatenation of the two
learning matrix that reduces the embedding dimension. This matrix learned during training and can potentially extract the relevant structural information from the representations. We refer to this experiment as elmo-reduced.
Lastly, we examine the performance of our representation, where we apply our structural extraction method on top of ELMo representation. We refer to this experiment as syntax.
We run the few-shot setup with multiple training size values: 50, 100, 200, 500. The results-for both labeled (L.AS) and unlabeled (UAS) attachment scores-are are presented in Figure and the numerical results are available in the appendix (B]
We notice that in the lower training size regime, we obtain the best performances compared to all baselines. The more training data is used, the gap between our representation and the baselines reduced, but the syniax representation still outperforms elmo. Reducing the dimensions with PCA (elmo-pca) works considerably worse than ELMo, indicating that the most salient information is indeed not structural, and the PCA loses important information. Reducing the dimensions with learned matrix (elmo-reduced) works substantially better than ELMo. and achieve the same UAS as our representation from 200 training sentences onward. However, our transformation was learned in an unsupervised fashion, without access to the syntactic trees. Finally, when considering the labeled attachment score, where the model is tasked at predicting not only the child-parent relation but also its label, our syntax representation outperforms elmo-reduced.
5 CONCLUSION
In this work, we propose an unsupervised method for the distillation of structural information from neural contextualized word representations. We used process of sequential BERT-based substitu-

---

Under review as a conference paper at ICLR 2020
Few shot parsing
Few shot parsing
0.85
0.85
0.80
0.80
0.70
0.65
0.65
0.60
reduced
0.60
reduced
eimo_pc
emo pca
0.50
0.0
100
200
300
400
500
100
200
300
400
500
Number training examples
Number of training examples
Figure 4: Results of the few shot parsing setup
tion to create a large number of sentences which are structurally similar, but semantically different. By controlling for one aspect structure while changing the other lexical choice, learn metric (via triplet loss) under which pairs of words that come from structurally-similar sentences are close in space. We demonstrated that the representations acquired by this method share structural properties with their neighbors in space, and show that with minimal supervision, those represen- tations outperform ELMo the task of few-shots parsing. The method presented here is a first step towards better disentanglement between various kinds of information that is represented in neural sequence models.
The method used to create the structurally equivalent sentences be useful by its for other goals, such as augmenting parse-tree banks (which are often scarce and require large resources to annotate). In future work, we aim extend this method to allow for more soft alignment between structurally-equivalen sentences.
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. CoRR, Abs/160 2016. URL
http://arxix wwtp:/oorggggbbbeeeee. 04207
Alexander Alemi, Ian Fischer, Joshua V. Dillon, and Murphy Murphy. Deep variational information bottleneck. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
Mikel Artetxe, Gorka Labaka, Inigo Lopez-Gazpio, and Eneko Agirre. Uncovering divergent lin- guistic information in word embeddings with lessons for intrinsic and extrinsic evaluation. arXiv preprint 02094, 2018.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, 2013.
Terra Blevins, Omer Levy, and I Luke Zettlemoyer. Deep RNNs encode soft hierarchical syntax. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, ACL., pp- 14-19, 2018.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv: (0X.0:1100.04341 2019.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pp. 2126 2136, 2018, doi: N0.18653/v1/P18 1198. URL https: wwww ac web. http://wwww......rr///aooooooooo
10

---

Under review as a conference paper at ICLR 2020
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HI.T, pp. 4161, 2019.
Timothy Dozat and Christopher D Manning. Deep biaffine attention for neural dependency parsing. arXiv preprint arXiv: 2016.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation. arXiv preprint arXiv:1707.02666 2017.
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. Style transfer in text: Explo- ration and evaluation. In Thirly-Second AAAI Conference on Artificial Intelligence, 2018.
Leon A. Gatys. Texture synthesis and style transfer using perceptual image representations from
convolutional neural networks. PhD thesis, University of Tübingen, Germany, 2017. URLhttpp
//d-nb.info/ 10t.1o//11111843522
Yoav Goldberg, Assessing BERT's syntactic abilities. CoRR, abs/ abs/1901.05287, 2019. URL http:
//aaxiv.rrg wwapr///a/////
1901. wwwpa///..........
05287
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. Colorless green recurrent networks dream bierarchically. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT, pp. 1195-1205, 2018.
Naama Hadad, Lior Wolf, and Moni Shahar. A two-step disentanglement method. In IEEE Confer- ence on Computer Vision and Pattern Recognition, (CVPR), pp- 772-780, 2018.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word represen- tations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pp. 4129 4138, 2019.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity-Bass Pattern Recognition Third International Workshop. SIMBAD, pp. 84-92, 2015.
Zhiting Hu, Zichao Yang. Xiaodan Liang, Ruslan Salakhutdinov, and Eric Xing. Toward con- trolled generation of text. In Proceedings of the 34th International Conference Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pp. 1587-1596, 2017. URL http://proceedinng mir. wwwpss/vo..... ht//hul17 hww.hhllehhhh. html
Dieuwke Hupkes, Sara Veldhoen, and Willem H. Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Intell. doi: 10. 1..1613/jair...11199 URL
Res., 61:907-926. 2018.
jair.1.11199
Mandar Joshi, Omer Levy, Daniel Weld, and Luke Zettlemoer. BERT for coreference resolution: Baselines and analysis. arXiv preprint arXiv: 09091, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, ICLR, 2015.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2018.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. Multiple-attribute text rewriting, 2018
Omer Levy and Yoav Goldherg. Linguistic regularities in sparse and explicit word representations. In Proceedings of the eighteenth conference on computational natural language learning, pp- 171-180, 2014.
Xiang Lisa Li and Jason Eisner. Specializing word embeddings (for parsing) by information bottle- neck. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNIP)). 2019.
11

---

Under review as a conference paper at ICLR 2020
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.0880 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: robustly optimized BERT pre- training approach. CoRR, abs/1907. 1177.11692.. 2019b.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579-2605, 2008.
Michaël Mathieu, Junbo Jake Zhao, Pablo Sprechmann, Aditya Ramesh, and Yann LeCun. Dis- entangling factors of variation in deep representation using adversarial training. In Advances in Neurul Information Processing Systems, pp. 5041 1-41-59 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen. Greg Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp- 3111-11133 3111-3119, 2013a.
Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 746-751,
2013b.
Siddharth Narayanaswamy, Brooks Paige, Jan-Wil Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank D. Wood, and Philip H. S. Torr. Learning disentangled rep- resentations with semi-supervised deep generative models. In Advances in Neural Information Processing Systems, pp. 5925-5935, 2017.
Xi Peng. Xiang Yu, Kihyuk Sohn, Dimitris N. Metaxas, and Manmohan Chandraker. Reconstructionn-aasee disentanglement for pose-invariant face recognition. In IEEE International Conference on Computer Visionn (ICCV), pp. 1632-1641, 2017.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532 1543, 2014.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pp. 2122-2227777 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, pp- 3483-3491, 2015.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the Conference of the Association for Computational Linguistics, ACL, pp. 4593- 4601, 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In Proc. of the Allerton Allerton Conference on Communication, Control and Computing, 1999.
Marten van Schijndel and Tal Linzen. In Tim Rogers, Marina Rau, Jerry Zhu, and Chuck Kalish (eds.), Proceedings of the 40th Annual Conference of the Cognitive Science Society, pp. 2600- 2605.
12

---

Under review as a conference paper at ICLR 2020
Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. What do RNN language models learn about filler-get gap dependencies? In Proceedings of the EMNLP Workshop BlackboxNLP: An- alyzing and Interpreting Neural Networks for NLP, pp. 211-221122 211-221. Association for Computational Linguisties, 2018.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with BERTserini. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics NAACL-HLT, pp. 72-77,2013 2019.
A APPENDIX
B COMPLETE PARSING RESULTS
Model
Number of sentences
50
100
200
500
ELMO
0.52
0.64
0.75
0.82
ELMO-reduced
0.55
0.65
0.75
0.82
ELMO-PCA
0.55
0.65
0.73
0.79
ELMO-syme -ynntxx (ours)
0.61
0.70
0.76
0.83
Table Labeled parsing scores (LAS)
Model
Number of sentences
50
100
200
500
ELMO
0.60
0.71
0.81
0.87
ELMO-reduced
0.68
0.76
0.83
0.88
ELMO-PCA
0.63
0.72
0.79
0.85
ELMO-syntax (ours)
0.69
0.78
0.82
0.87
Table 3: Unlabeled parsing scores (UAS)
C BERT CLOSEST- WORD RESULTS
Dep. edge
Head's
Tree
Tree path
Tree path
Depth (correlation)
Lexical Match
(complete)
(L=3)
(L=2)
Baseline (all)
0.549
0.432
0.146
0.310
0.436
0.829
Transformed (all)
0.681
0.565
0.221
0.471
0.697
0.597
0.319
Baseline (difficult)
0.478
0.429
0.143
0.310
0.521
0.428
0.820
Transforned (difficult)
0.652
0.565
0.225
0.482
0.714
0.601
0,300
Table Results the closest-wond queries, before and after the application of the syntactic trans- formation. "Basline" refers to unmodified vectors derived from BERT, and ""Transformed" refers to the vectors after the learned syntactic transformation f. "Difficult" refers to evaluation on the subset of POS tags which are most structurally diverse.
13


Reviews:
[{"review_id": "rJxPTT-IYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n=========\nThis paper aims to disentangle semantics and syntax in contextualized word representations. The main idea is to learn a transformation of the contexualized representations that will make two word representations to be more similar if they appear in the same syntactic context, but less similar if they appear in different syntactic contexts. The efficacy of this transformation is evaluated through cluster analysis, showing that words better organize in syntactic clusters after the transformation, and through low-resource dependency parsing. \n\nThe paper presents a simple approach to transform word representations and expose their syntactic information. The experiments are mostly convincing. I would like to see better motivation, more engagement with a wider range of related work, and more thorough quantitative evaluations. Another important question to address is also what kind of semantic/syntactic types of information are targeted, and how to handle the tradeoff between them, for instance for different purposes. \n\n\nMain comments:\n==============\n1. Motivation: I found the motivation for the problem understudied a bit lacking. The main motivation seems to be to disentangle semantic and syntactic information. But why should we care about that? Beyond reference to disentangling in computer vision, some more motivation would be good. The few-shot parsing is a good such motivation, although the results are a bit disappointing (see more on this below). Another possible motivation is potential applications of disentanglement in language generation. There is a line of work on style transfer also in language generation, and it seems plausible that the methodology could be applied to such tasks. \n2. The present work is well-differentiated from work on extracting syntactic information from word representations via supervised ways, as the current work does so in an unsupervised way. I don't quite get the terminological differentiation between \"mapping\" and \"extracting\" in the introduction, but the idea is clear. \n3. Have you considered alternative representations of word pairs besides the different of their transformations f(x)-f(y)? \n4. I found it interesting that the word representation from BERT is the concatenation of layer 16 with the mean of all the other layers. This is motivated by Hewitt and Manning's findings, and [5] found similar results. However, the different between layer 16 and others is not that large as to warrant emphasizing it so much. Perhaps a scalar mix with fine-tuning may work better, as in [5], or another method. Have you tried other word representations? I also wonder whether it makes sense to use different layers for different parts of the triplet loss, depending on whether to emphasize syntactic vs. semantic similarity. \n5. The introduction lays out connections to some related work, but leaves several relevant pieces missing. See examples below. \n6. The results in 3.3 are limited but useful. The comparison with a PCA-ed and reduced representation is well thought of, because of the risk with low-resource and high dimensionality. That said, I found the gap between the proposed syntax model and the ELMo-reduced disappointingly small. Even in the LAS, it seems like the difference is very small, ~0.5, although it's hard to tell from the figure. Providing the actual numbers and a measure of statistical significance would be helpful here. \n7. Some care should be taken to define what kind of semantics is targeted here. In several cases this is \"lexical semantics\", but then we have \"meaning\" in parentheses sometimes (end of intro). Obviously, there's much more to semantics and meaning that the lexical semantics, so a short discussion of how the work views other, say compositional semantics, would be good. \n\n\nOther comments:\n===============\n1. The introduction seeks a representation that will ignore the similarity between \"syrup\" in (2) and (4). I wonder if \"ignoring\" is too strong. One may not want to lose all lexical semantic information. Moreover, the proposed triplet loss does not guarantee that information is ignored (and justly so, in my opinion). \n2. In the example, \"maple\" and \"neural\" are said to be syntactically similar, although \"maple syrup\" is a noun compound while \"neural networks\" is an adjective-noun. Shouldn't they be treated differently then? Unless the notion of syntax is more narrow and just looks at unlabeled dependency arcs. \n3. Some experimental choices are left unexplained, such as k=6 (section 2.1) or mapping to 27 dims (section 2.3); these two seem potentially important. \n4. Section 2.3: do you also back-prop back into the BERT/ELMo model weights? \n5. The dataset statistics in section 3 do not match those in section 2.2. Please clarify. \n6. The qualitative cluster analysis via t-SNE (3.1) is compelling. It could be made stronger by reporting quantitative clustering statistics such as cluster purity before and after transformation. \n7. In the examples showin in 3.1, it would be good to give also the nearest neighbor before the transformation for comparison. \n8. The quantitative results in 3.2 convey the point convincingly. It's good to see also the lexical match measure going down. The random baseline is also a good sanity check to have. It would be good to provide full results with BERT, at least in the appendix and at least for section 3.2, maybe also for 3.3.\n9. More related work: \n+ Work that injects syntactic information into word representations in a supervised way, such as [1,2]\n+ Work that shows that word embeddings contain different kinds of information (syntactic/semantic), and propose simle linear transformations to uncover them. \n+ Engaging with the literature on style transfer in language generation would be good, as mentioned above for motivation, but also to situate this work w.r.t to related style transfer work. \n+ Another line of work that may be mentioned is the variety of papers trying to extract syntactic information from contextualized word representations, such as constructing trees from attention weights. There were a few such papers in BlackboxNLP 2018 and 2019. \n\nTypos, phrasing, formatting, etc.:\n============================\n- Abstract: a various of semantic... task -> various semantic... tasks; use metric-learning approach -> use a metric-learning approach; in few-shot parsing setting -> in a few-shot parsing setting\n- Wilcox et al. does not have a year\n- Introduction: few-shots parsing -> few-shot parsing\n- Method: extract vectors -> extracts vectors; Operativly -> Operatively \n- Section 3: should encourages -> should encourage; a few-shots settings -> a few-shot setting\n- 3.2: -- was not rendered properly\n- 3.3: matrix that reduce -> reduces \n\n\nReferences\n==========\n[1] Levy and Goldberg. 2014. Dependency-Based Word Embeddings\n[2] Bansal et al. 2014. Tailoring Continuous Word Representations for Dependency Parsing\n[3] Artetxe et al. 2018. Uncovering divergent linguistic information in word embeddings with lessons for intrinsic and extrinsic evaluation\n[4] Tenney et al. 2019. BERT Rediscovers the Classical NLP Pipeline\n[5] Liu et al. 2019. Linguistic Knowledge and Transferability of Contextual Representations", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SygRa5LTKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors state a clear hypothesis: it is possible to extract syntactic information from contextualized word vectors in an unsupervised manner. The method of creating syntactically equivalent (but semantically different) sentences is indeed interesting on its own. Experiments do support the main hypothesis -- the distilled embeddings are stronger in syntactic tasks than the default contextualized vectors. The authors provide the code for ease of reproducibility which is nice.\n\nThere is a short literature review, but I am wondering if something similar was done for static word embeddings. I understand that they are obsolete these days, but on the other hand, they are better researched, so were there any attempts to disentangle syntax and semantics in the classical static word vectors?\n\nOverall, I have no major concerns with the paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SklmVR-fqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CONTRIBUTIONS:\nTopic: Disentangling syntax from semantics in contextualized word representations\nC1. A method for generating ‘structurally equivalent’ sentences is proposed, based only on the assumption that maintaining function words, and replacing one content word of a source sentence with another to produce a new grammatical sentence, yields a target sentence that is equivalent to the source sentence. \nC2. The ‘structural relation’ between two words in a sentence is modeled as the difference between their vector embeddings.\nC3a. The structural relation between a pair of content words in one sentence is assumed to be the same as that between the corresponding pair in an equivalent sentence. \nC3b. The structural relation between any pair of content words in one sentence is assumed to be different from the structural relation between any pair of content words in an inequivalent sentence. \nC4. Given a selected word in a source sentence, to generate an alternative ‘corresponding’ content word for an equivalent target sentence, BERT is used to predict the source word when it is masked, given the remaining words in the source sentence. The alternative corresponding word is randomly selected from among the top (30) candidates predicted by BERT. Given a source sentence, the set of target sentences formed by cumulatively replacing content words one at a time in randomly selected positions defines an ‘equivalence set’ in which words in different sentences with the same left-to-right index are corresponding words. (To promote the formation of grammatical target sentences, a word is only replaced by another word with the same POS.) A pre-defined set of equivalence sets is used for training.\nC5. A metric learning paradigm with triplet loss is used to find a function f for mapping ELMo or BERT word embeddings to a new vector space of ‘transformed word representations’. Implementing C2 and C3a, given the indices i and i’ of two content words, the triplet loss rewards closeness of the difference D between the transformed embeddings of the pair of words with these indices in sentence S and the corresponding difference D’ for an equivalent sentence S’. Implementing C3b, the triplet loss penalizes closeness between D and D”, where D” is the difference between transformed word embeddings of a pair of content words in a sentence S” that is inequivalent to S. (Eq. 4).\nC6. (Implementing C5.) To form a mini-batch for minimizing the triplet loss, a set of (500) sentences S is selected, and for each a pair of indices of content words is chosen. Training will use the difference in the transformed embeddings of the words in S with these indices: call this D, and call the set of these (500) D vectors B. For each sentence S in B, a ‘positive pair’ (D, D’) is generated, where D’ is the corresponding difference for S’, a selected sentence in the equivalence set of S. Closeness of D and D’ is rewarded by the triplet loss, implementing C3a. To implement C3b, a ‘negative pair’ (D, D”), for which closeness is penalized by the loss, is formed as follows. D” is the closest vector in B to D that is derived from a sentence S” that is not equivalent to S. \nC7. 2-D t-SNE plots (seem to) show that relative to the original ELMo embeddings, the transformed embeddings cluster better by POS (Fig. 3). (No quantitative measure of this is provided, and the two plots are not easy to distinguish.)\nC8. Pairs of closest ELMo vectors share syntactic (dependency parse) properties to a greater degree after transformation than before (Table 1). To check that this goes beyond merely POS-based closeness, the syntactic relations that least determine POS are examined separately, and the result remains. Furthermore, the proportion of pairs of closest vectors that are embeddings of the same word (in different contexts) drops from 77.6% to 27.4%, showing that the transformation reduces the influence of lexical-semantic similarity. Similar results hold for BERT embeddings, but to a lesser degree, so the paper focusses on ELMo. \nC9. Few-shot parsing. Two dependency parsers are trained, one on ELMo embeddings, the other on their transformations (under the proposed method). In the small-data regime (less than 200 training examples), the transformed embeddings yield higher parser performance, even when the encoding size of the ELMo embeddings is reduced (from 2048 to 75) to match that of the transformed embeddings by either PCA or a learned linear mapping. (Fig. 4) \nRATING: Weak accept\nREASONS FOR RATING (SUMMARY). Using deep learning to create an encoding of syntactic structure with minimal supervision is an important goal and the paper proposes a clever way of doing this. The only ‘supervision’ here comes from (i) the function/content-word distinction (C1 above): two grammatical sentences are structurally equivalent if [but not only if] one can be derived from the other by replacing one content word with another; and (ii) filtering candidate replacement words to match the POS of the replaced word. BERT’s ability to guess a masked word is put to good use in providing suitable content word substitutions. The experimental results are rather convincing.\nREVIEW (beyond the summary above)\nC1. This assumption is famously not deemed to be true in linguistics, where the structural difference between ‘control’ and ‘raising’ verbs is basic Ling 101 material: see https://en.wikipedia.org/wiki/Control_(linguistics)#Control_vs._raising. This particular structural contrast illustrates how verbs can differ in their argument structure, without there being function words to signal the difference. So substituting *verbs* in particular may be non-ideal for the purposes of this work. Even the third example given by the authors in Sec. 3.1 illustrates a related  point, where function words do signal the contrast:  while the meaning of ‘let’ and ‘allow’ may be very similar, their argument structures differ, so that replacing ‘lets’ with ‘allows’ in the first sentence, or the reverse in the second sentence, produces ungrammatical results: \n*their first project is software that *allows* players connect the company ’s controller to their device\n*the city offers a route-finding website that *lets* users to map personalized bike routes\nTherefore, contrary to the paper, relative to linguistic syntactic structure, it is not a good result that ‘lets’ in the original version of the first sentence is the closest neighbor in transformed embedding space to ‘allows’ in the second. Rather, it is probably meaning, not structure, that makes ‘let’ and ‘allow’ similar.\nIt would improve the paper to make note of this general concern with C1 and to provide a response.\nOn another point, an important premise of the proposed method (C2 above) is that differences in vector space embeddings encode relations; this has been used by a number of previous authors since the famous Mikolov, Yih & Zweig NAACL2013, and that work should be cited and discussed.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byey-Ljz5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe authors proposed to disentangle syntactic information and semantic information from pre-trained contextualized word representations.\n\nThey use BERT to generate groups of sentences that are structurally similar (have the same POS tag for each word) but semantically different. Then they use a metric-learning approach to learn a linear transformation that encourages sentences from the same group to have closer distance. Specifically, they defined a triplet loss (Eq4) and uses negative sampling.\n\nThey use 150,000 sentences from Wikipedia to train the transformation. POS tags are obtained from spaCy. To evaluate the learned representations, they provided a tSNE visualization of the original and transformed representations (groups by dependency label); evaluate whether the nearest neighbor shares the same syntactic role; low-resource parsing.\n\nReasons of rejection:\n1. I don't agree with the authors' argument, \"we aim to extract the structural information encoded in the network in an unsupervised manner, without pre-supposing an existing syntactic annotation scheme\".  First, what do you mean by structural information without a clear definition? Also, in the method, the authors construct a dataset where each group of the sentence share similar syntactic structures (having the same POS tag). It seems there that the structural information just means POS tags.\n\n2. The author failed to convince me that the learned representation is more powerful than just combining POS tags with the original representations. Since POS tags are assumed to be available during training. I think a reasonable baseline in all experiments would be the performance based on POS tags. For example, in Figure 3, although the original EMLo representation does not correlates with the dependency label very much, the POS tags may do. In Figure 4, the authors should compare with delexicalized dependency parsing, which performs pretty well in los-resource setting.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to disentangle semantics and syntax inside of popular contextualized word embedding models. They use the model to generate sentences which are structurally similar but semantically different. 

This paper generated a lot of discussion. The reviewers do like the method for generating structurally similar sentences, and the triplet loss.  They felt the evaluation methods were clever.  However, one reviewer raised several issues.  First, they thought the idea of syntax had not been well defined. They also thought the evaluation did not support the claims.  The reviewer also argued very hard for the need to compare performance to SOTA models.  The authors argued that beating SOTA is not the goal of their work, rather it is to understand what SOTA models are doing.  The reviewers also argue that nearest neighbors is not a good method for evaluating the syntactic information in the representations.  

I hope all of the comments of the reviewers will help improve the paper as it is revised for a future submission.
Decision: Reject