Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction
Authors: Taeuk Kim
Jihun Choi
Daniel Edmiston
Sang-goo Lee
Venue:  (2020)

Abstract: With the recent success and popularity of pre-trained language models (LMs) in natural language processing, there has been a rise in efforts to understand their inner workings. 
In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. 
Our method provides an effective way of extracting constituency trees from the pre-trained LMs without training. 
In addition, we report intriguing findings in the induced trees, including the fact that pre-trained LMs outperform other approaches in correctly demarcating adverb phrases in sentences.

Published as a conference paper at ICLR 2020
ARE PRE-TRAINED LANGUAGE MODELS AWARE OF PHRASES? SIMPLE BUT STRONG BASELINES FOR GRAMMAR INDUCTION
Taeuk Kim', Jihun Choi', Daniel Edmiston" Sang-goo Lee' Dept. of Computer Science and Engineering, Seoul National University, Seoul, Korea Dept. of Linguistics, University of Chicago, Chicago, II., USA (taeuk, jhchoi,
sn. kr, daw.emmisttonuuchcccg.....o
ABSTRACT
With the recent success and popularity of pre-trained language models (I.Ms) in natural language processing, there has been rise in efforts understand their in- ner workings. In line with such interest, we propose a novel method that assists us in investigating the extent to which pre-trained LMs capture the syntactic notion of constituency. Our method provides an effective way of extracting constituncy trees from the pre-trained LMs without training. addition, we report intriguing findings in the induced trees, including the fact that some pre-trained LMs outper- form other approaches in correctly demarcating adverb phrases in sentences.
INTRODUCTION
Grammar induction, which is closely related to unsupervised parsing and latent tree learning, allows associate syntactic trees, i.e., constituency and dependency trees, with sentences. As gram- mar induction essentially assumes no supervision from gold-standard syntactic trees, the existing approaches for this task mainly rely on unsupervised objectives, such as language modeling (Shen) et al., 20188b 8b; 2019 Kim et al. 2019a b) and cloze-style word prediction (Drozdov et al.] 2019) to train their task-oriented models. On the other hand, there is a trend in the natural language pro- cessing (NLP) community of leveraging pre-trained language models (LMs), e.g., ELMo (Peters) al. 2018) and BERT (Devlin al. 2019), as means of acquiring contextualized word represen- tations. These representations have proven to be surprisingly effective, playing key roles in recent improvements in various models for diverse NLP tasks.
In this paper, inspired by the fact that the training objectives of both the approaches for grammar induction and for training LMs are identical, namely, (masked) language modeling, we investigate whether pre-trained LMs also be utilized for grammar inductionn/ussupervised parsing, espe- cially without training. Specifically, we focus on extracting constituency trees from pre-trained LMs without fine-tuning required in other cases where representations from pre-trained LMs are employed. This restriction provides us with some advantages: (1) it enables to derive strong baselines for grammar induc- tion with reduced time and space complexity, offering a chance to reexamine the current status of existing grammar induction methods, (2) it facilitates an analysis
introducing another task-specific module, least of which is usually
how much and what kind of syntactic information each pre-trained LM contains in its intermediate representations and atten- tion distributions terms of phrase-structur grammar, and (3) it allows us easily inject biases into our framework, for instance, to encourage the right-skewness of the induced trees, resulting in performance gains in English unsupervised parsing.
First, we briefly mention related work (2)) Then, we introduce the intuition behind our proposal in detail (33), which is motivated by our observation that we can cluster words sentence according to the similarity of their attention distributions over words in the sentence. Based on this intuition, we define a straightforward yet effective method (44 of drawing constituency trees directly from pre- trained LMs with fine-tuning addition of task- task-specific parts, instead resorting to the concept of Syntactic Distance (Shen et al.) b))8a]b). Then, we conduct experiments (55) on the induced constituency trees, discovering some intriguing phenomena. Moreover, we analyze the pre-trained

---

Published as a conference paper at ICLR 2020
LMs and constituency trees from various points of view, including looking into which layer(s) of the LMs is considered to be sensitive to phrase information ($
To summarize, contributions in this work are follows:
By investigating the attention distributions from Transformer-based pre-trained LMs, show that there is evidence to suggest that several attention heads of the LMs exhibit syn- tactic structure akin constituency grammar.
Inspired by the above observation, we propose a method that facilitates the derivation of constituency trees from pre-trained LMs without training. We also demonstrate that the induced trees can serve as a strong baseline for English grammar induction.
We inspect, in view of our framework, what type of syntactic knowledge the pre-trained LMs capture, discovering interesting facts, e.g., that some pre-trained LMs are more aware of adverb phrases than other approaches.
2 RELATED WORK
Grammar induction is a task whose goal is to infer from sequential data grammars which generalize, and are able to account for unseen data (Lari & Young (1990): Clark (200T); Klein Manning 2002|20004] to name few). Traditionally, this was done by learning explicit grammar rules (e.g., context free rewrite rules), though more recent methods employ neural networks to learn such rules implicitly, focusing more on the induced grammars' ability to generate parse sequences.
Specifically, Shen et al. (01 proposed Bersing-Reading-- Predict Network (PRPN) where the con- cept of Syntactic Distance is first introduced. They devised neural model for language modeling where the model is encouraged to recognize syntactic structure. The authors also probed the possi- bility of inducing constituency trees without access gold-standard trees by adopting algorithm that recursively splits a sequence of words into two parts, the split point being determined according to correlated syntactic distances; the point having the biggest distance becomes the first target of division. [Shen et al.] (2019) presented model called Ordered Neurons (ON), which is revised version of LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber (997) which reflects the hierarchical biases of natural language and can be used to compute syntactic distances. Shen et al. (018a) trained a supervised parser relying on the concept of syntactic distance.
Other studies include Drozdov et al. (2019), who trained deep inside recursive autoencoders (DIORA) to derive syntactic trees in an exhaustive way with the aid of the inside-outside algorithm, and Kim et al. (2019a) who proposed Compound Probabilistic Context-Free Grammars (compound PCFG), showing that neural PCFG models are capable of producing promising unsupervised parsing results. Li et al. (2019) proved that ensemble of unsupervised parsing models can be beneficial, while Shi et al. (2019) utilized additional training signals from pictures related with input text. Dyer et al. (2016) proposed Recurrent Neural Network Grammars (RNNG) for both language modeling and parsing, and Kim et al. (2019b) suggested an unsupervised variant of the RNNG. There also exists another line of research task-specific latent tree learning (Yogatama et al.) 2017 Choi al., 2018 Havrylov et al., 2019 Maillard et al.] 2019). The goal here is not to construct linguistically plausible trees, but induce trees fitted improving target performance. Naturally, the induced performance-based trees need not resemble linguistically plausible trees, and some studies (Williams) al. 2018a; Nangia & Bowman 2018) examined the apparent fact that performance-based and lingusitically plausible trees bear little resemblance to one another.
Concerning pre-trained language models (Peters et al. (2018); Devlin al. (2019): ): Radford et al. (2019); Yang et al. (2019); Liu et al. (2019b), inter alia) particularly those employing a Trans- former architecture (Vaswani et al. (017) these have proven to helpful for diverse NL. down- stream tasks. In spite of this, there is no vivid picture for explaining what particular factors contribute to performance gains, even though some recent work has attempted to shed light on this question. In detail, one group of studies (Raganato & Tiedemann (2018): Clark et al. (2019); [Ethayarajh] (2019); Hao et al. (201)); Voita et al. (2019), inter alia) has focused on dissecting the intermediate represen- tations and attention distributions of the pre-trained LMs, while the another group of publications (Mareček & Rosa (2018); Goldberg (2019); Hewitt & Manning (2019); Liu et al. (2019a); Rosa & Mareček (2019), to name few) delve into the question of the existence of syntactic knowledge in
2

---

Published as a conference paper at ICLR 2020
When
Noting others
the
0.35
price
plastics
took
0.25
0.30
esiimates
when
oT
0.4
pr
1987
0.20 Increases
Quantum
Chemical
0.15
be
Corp.
sustained
0.2
went
0.10
he
along
ramarks
ride
the
0.05
October
Some
say
When
the
of
plestics
took
1987 Quantii Chemical sat doong Tol the
Noting
others
ussimate
whan
price
Increasez
can
be
sustaimed
he
ramarts Some
soy October
Figure I: Self-attention heatmaps from two different pre-trained LMs. (Left) A heatmap for the average of attention distributions from the 7th layer of the XLNet-base (Yang et al.) 2019) model given the sample sentence. (Right) A heatmap for the average of attention distributions from the 9th layer of the BERT-base (Devlin et al.) (219)) model given another sample sentence. We can easily spot the chunks of words the two heatmaps that are correlated with the constituents of the input sentences, e.g., (Left) 'the price of plastics', 'took off in 1987', 'Quantum Chemical Corp.', (Right) 'when price increases be sustained', and 'he remarks'.
Transformer- basse models. Particularly, Mareček & Rosa (2019) proposed an algorithm for extract- ing constituency trees from Transformers trained for machine translation, which is similar to our approach.
MOTIVATION
As pioneers in the literature have pointed out, the multi-head self-attention mechanism (Vaswani al., (017) is a key component in Transformer-bastt language models, and seems this mecha- nism empowers the models to capture certain semantic and syntactic information existing in natural language. Among diverse of knowledge they may capture, this work we concentrate on phrase-structure grammar by seeking to extract constituency trees directly from their attention in- formation and intermediate weights.
In preliminary experiments, where we manually visualize and investigate the intermediate repre- sentations and attention distributions several pre-trained LMs given input, we have found some evidence which suggests that the pre-trained LMs exhibit syntactic structure akin to constituency grammar some degree. Specifically, we have noticed some patterns which are often displayed in self-attention heatmaps explicit horizontal lines, groups of rectangles of various sizes. As an attention distribution of a word in an input sentence corresponds to a row a heatmap matrix, these patterns indicates the existence of groups words where
can say that the appearance the attention distributions of the words in the same group are relatively similar. Interestingly, we have also discovered the fact that the groups of words we observed are fairly correlated with the constituents of the input sentence, as shown in Figure | (above) and Figure 3 (in Appendix A.I)
Even though we have identified some patterns which match with the constituents sentences, is not enough to conclude that the pre-trained LMs are aware of syntactic phrases as found in phrase- structure grammars. demonstrate the claim, we attempt to obtain constituency trees in a zero-shot learning fashion, relying only on the knowledge from the pre-trained LMs. To this end, we suggest the following, inspired from our finding: two words in a sentence are syntactically close to each other (i.e., the two words belong to the same constituent) if their attention distributions over words in the sentence are also close to each other. Note that this implicitly presumes that each word is more likely attend more the words in the same constituent enrich its representation in the pre- mre-trained LMs. Finally, we utilize the assumption to compute syntactic distances between each pair of adjacent words in sentence, from which the corresponding constituency tree be built.

---

Published as a conference paper at ICLR 2020
4 PROPOSED METHOD
4.1 SYNTACTIC DISTANCE AND TREE CONSTRUCTION
We leverage the concept of Syntactic Distance proposed by Shen et al. (2018a stituency trees from raw sentences in an intuitive way. Formally, given a sequence of words in
draw con-
sentence, W1, w2..... Wn, we compute = (d1,d2,... d2,...,,dd.................. .....] where d, corresponds to the syntac- tic distance between W; and W1+1- Each d, is defined as follows:
d, [g(()) ((w+4)))
(1)
where -(:-)) and g(-) are distance measure function and representation extractor function, respec- tively. The function I converts each word into the corresponding vector representation, while computes the syntactic distance between the two words given their representations. Once d is de- rived. it can be easily converted into the target constituency tree by a simple algorithm following Shen et al. 2018a For details of the algorithm, we refer the reader to Appendix A.2
Although previous studies attempted to explicitly train the functions f and g with supervision (with access to gold-standard trees, Shen et al. (2018a)) to obtain them as by-product of training particular models that are carefully designed to recognize syntactic information (Shen et al. 2018b 2019), in this work we stick to simple distance metric functions for and pre-trained LMs for g. for- going any training process. In other words, we focus on investigating the possibility of pre-trained LMS possessing constituency information in a form that can be readily extracted with straightfor- ward computations. If the trees induced by the syntactic distances derived from the pre-trained LMs are similar enough to gold-standard syntax trees, we can reasonably claim that the LMs resemble phrase-structure.
4.2 PRE-TRAINED LANGUAGE MODELS
We consider four types of recently proposed language models, These are: BERT (Devlin et al. 2019), GPT-2 (Radford et al., 2019), RoBERTa (Liu al. 2019b). and XLNet (Yang et al. 2019). They all have in common that they are based on the Transformer architecture and have been proven to be effective in natural language understanding (Wang et al. 2019) or generation. We handle two variants for each LM, varying in the number of layers, attention heads, and hidden dimensions, resulting in eight different cases in total. In particular, each LM has two variants. (1) base: consists of (=12 layers, 1=12 attention heads, and d=768 hidden dimensions, while (2) large: has 1=24 layers, a=16 attention heads, and d020 hidden dimensions ? LMs, unlike previous work which has mostly analyzed specific model, particularly BERT. For details about each LM, we refer readers to the respective original papers.
We deal with a wide range of pre-trained
In terms of our formulation, each LM instance provides two categories of representation extractor functions, G" and Gd. Specifically, G" refers to a set of functions Li = 1..... ........, .). each of which simply outputs the intermediate hidden representation of given word the jth layer of the LM. Likewise, Gdd is a set of functions ( D = 1,....,,45 1,kk 1,...,a + 1}, each of which outputs the attention distribution of an input word by the kth attention head on the jth layer of the LM. Even though our main motivation comes from the self-attention mechanism, we also deal with the intermediate hidden representations present in the pre-trained LMs by introducing G", considering that the hidden representations serve as storage collective information taken from the processing of the pre-trained LMs. Note that & ranges up to + 1, not a. implying that we consider the average of all attention distributions on the same layer in addition to the individual ones. This averaging function can be regarded different aspects of information, and we expect that this technique will provide a better option in some cases as reported in previous work (Li et al. 2019).
ensemble of other functions in the layer which are specialized for
Our parsing algorithm is an unbiased method in contrast to one (named as COO parser by Dyer et al. (2019) employed in most previous studies (Shen et al. [0018)] 2019 Htut et al., 2018 Li et al. 2019 Shi | 2019). This choice enables to investigate the exact extent which pre-trained LMs contribute their performance on unsupervised parsing, considering the fact revealed recently by Dyer al. 2019) that the COO parser has potential issues that prefers right-branching branching trees and does not cover all possible tree derivations. Furthermore, we can directly adjust the right-branching bias using our method in Section 4.4 needed. In case of GPT-2, 'GPT2' corresponds to the 'base' variant while 'GPT2-medium' to the 'large' one.
4

---

Published as a conference paper at ICLR 2020
One remaining issue is that all the pre-trained LMs regard each input sentence sequence of subword tokens, while our formulation assumes words cannot be further divided into smaller tokens. To resolve this difference, a complete word should be exploited to represent the word, and we have empirically found that the best result comes when each word is represented by an average of the representations of its subwords Therefore, we adopt the above heuristic in this work for cases where word is tokenized into more than two parts.
tested certain heuristics that guide how subword tokens for
4.3 DISTANCE MEASURE FUNCTIONS
For the distance measure function f. we prepare three options (P") for G" and two options (F'd) for Gdd. Formally, f E U*, where l" (COS, L1, fd (JSD, HEL]. COS, LI, L2, JSD, and HEL corresoond to Cosine, L1, and L2, Tansen-Shanoon and Hellinger distance respectively. The functions F" are only compatible with the elements of G and the same holds for F and Gd. The exact definition of each function is listed in Appendix A.3
4.4 INJECTING BIAS INTO SYNTACTIC DISTANCES
One of the main advantages we obtain by leveraging syntactic distances to derive parse trees is that we can easily inject inductive bias into our framework by simply modifying the values of the syn- tactic distances. Hence, we investigate whether the extracted trees from our method can be further refined with the aid of additional biases. To this end, we introduce a well-known bias for English constituency the right-skewness bias- simple linear form Namely, intention is to influence the induced trees such that they are moderately right-skewed following the nature of gold-standard parse trees in English.
Formally, compute d; by appending the following linear bias term to every d;:
d = AVG(() x -1- 11mmmmm---- 11mmmmmmmm)m) 1) x (i (i-11)) 1)),
where AVG(-) outputs an average all elements a vector, À is a hyperparameter, and ranges from to distances.
1. We write à = [dg de.....dm., in place of d to signify biased syntactic
The main purpose of introducing such a bias is examining what changes are made to the resulting tree structures rather than boosting quantitative performance per se, though is of note that it serves this purpose as well. We believe that this additional consideration is necessary based on two points. First, English is what known as head-initial language. That is, given a selector and argument, the selector has strong tendency to appear on the left, e.g., 'eat food', or 'to Canada". Head-initial languages therefore have an in-built preference for injected into syntactic distances derived from pre-trained LMs, structures. By adjusting the bias figure out whether the LMs are capable of inducing the gight-branchngg bias, which is one of the main properties of English syntax; if injecting the bias does not influence the performance of the LMs on unsupervised parsing, we can conjecture they are inherently capturing the bias to some extent. Second, as mentioned before, we have witnessed some previous work (Shen et al. 2018b 2019 Huut et al. 2018 Li al.) 2019 Shi et al. 2019) where the right-skewness bias is implicitly exploited, although it could be regarded as not ideal. What we intend focus on is the question about which benefits the bias provides for such parsing models, leading to overall performance improvements. In other words, we look for what the exact contribution of the bias when it is injected into grammar induction models, by explicitly controlling the bias using our framework.
We also tried other heuristies following previous work (Kitaey Klein, 2018 c.g., using the first or last subword of a word as representative, but this led to no performance gains. 'It is necessary carefully design biases for other languages as they have their own properties.
5

---

Published as a conference paper at ICLR 2020
5
EXPERIMENTS
5.1 GENERAL SETTINGS
5.1.1 DATASETS
In this section, we conduct unsupervised constituency parsing on two datasets. The first dataset is WSJ Penn Treebank (PTB, Marcus et al. (193)). in which human-annotated gold-standard trees are available. We use the standard split of the dataset- 2-21 for training. 22 for validation, and 23 for test. The second one is MNLI (Williams et al. 2018b), which is originally designed to test natural language inference but often utilized as a means of evaluating parsers. It contains constituency trees produced by an external parser (Klein & Manning 2003). We leverage the union of two different versions of the MNLI development set as test data following convention (Htut et al. 2018 Drozdov randomly sample 40K
al. 2019), and we call it the MNLI test set in this paper. Moreover, sentences from the training set of the MNLI to utilize them as a validation set. To preprocess the datasets, we follow the setting of Kim al. (2019a) with the minor exceptions that words are not lower-cased and number characters are preserved instead of being substituted by a special character.
5.1.2 IMPLEMENTATION DETAILS
For implementation, to compare pre-trained LMs in an unified manner, we resort to an integrated PyTorch codebase that supports all the models we consider For each LM, tune the best combi- nation of f and g functions using the validation set. Then, we derive a set of for sentences in the test set using the chosen functions, followed by the resulting constituency trees converted from each d by the tree construction algorithm in Section 4.1 In addition to sentence-level FI (S-FI) score, report label recall scores for six main categories: SBAR, NP, VP, PP, ADJP, and ADVP. We also present the results of utilizing d instead of d, empirically setting the bias hyperparameter À as 1.5. We do not fine-tune the LMs on domain-specift data, as we here focus on finding their universal characteristics.
We take four naïve baselines into account, random (averaged over 5 trials), balanced, left-branching, and right-branching binary trees. addition, we present two more baselines which are identical to models except that their g functions based than pre-trained ones. To be concrete, We provide 'Random XLNet-base (F" which applies the functions in F" on random hidden representations and 'Random XLNet-base (E'dy that utilizes the functions in Fd and random attention distributions, respectively. Considering the randomness of initialization and possible choices for f. the final score for each of the baselines is calculated as an average over 5 trials of each possible f, i.e., an average over 5 x 3 runs in case of F and 5 x 2 estimate the exact advantage we obtain by pre-training
randomly initialized XLNet-base rather
runs for ndd These baselines enable LMs, effectively removing additional unexpected gains that may exist. Furthermore, we compare our parse trees against ones from existing grammar induction models.
All scripts used in our experiments will be publicly available for reproduction and further analysis 5
5.2 EXPERIMENTAL RESULTS ON PTB
In T we report the results of the various models on the PTB test set. First of all, our method combined with pre-trained LMs shows competitive or comparable results in terms of S-F1 even without the right-skewness bias. This result implies that the extracted trees from our method can be regarded as a bascline for English grammar induction. Moreover, pre-trained LMs show substan- tial improvements over Random Transformers (XLNet-base), demonstrating that training language models on large corpora, in fact, enables the LMs to be more aware of syntactic information.
When the right-skewnee bias is applied to syntactic distances derived from pre-trained LMs, the S-F1 scores of the LMs increase by up to ten percentage points. This improvement indicates that the pre-trained LMs do not properly capture the largely right-branching nature of English syntax, at least when observed through the lens of our framework. By explicitly controlling the bias through our framework and observing the performance gap between our models with and without the bias,
https://githobs http://h....oooeeeeeeeeeeeeeeeen

---

Published as a conference paper at ICLR 2020
Table 1: Results the PTB test set. Bold numbers correspond the top 3 results for each column. L: laver number, A: attention head number (AVG: the average of all attentions). f: Results reported
by Kim et (2019)). I: Approaches which COO parser utilized.
Model
L
S-F1
SBAR
NP
VP
PP
ADJP
ADVP
Baselimes
Random
18%
Balanced Trees
18.5
75:
27%
8%:
18%
27%
25%
Branching
8%
Right Branching Trees
39.4
68%
24%
71%
42%
27%
38%
Random XLNet-base (P")
19.6
26%
12%
20%
23%
24%
Random Net-base (Fd)
19%8
Pre-trained LMs (w/o bias)
RFRRThaae
JSD
AVG
32.4
28%
42%
28%
31.
35%
63%
BERT-large
HEL
AVG
34.2
34%
.33%
27%
39%
37%
57%
GPT2
JSD
9
37.1
325
47%
27%
5.5%
27%
36%
GPT2-medium
1SD)
39.4
41%
51.%
21%
%
33%
RoBERTa-basc
JSD
33.8
40%
38%
33%
43%
42%
57%
RaBe
JSD
34.1
29%
46%
300%
37%
28%
40%
XL.Net base
HEL
AVG
40.1
35%
569%
26%
38%
47%
68%
XiNet-large
1,2
38.1
36%
51.
26%
41%
45%
69
Pre-trained LMs (w/ blas A=1.5)
BERT-base
HEL
AVG
42.3
45%
46%
49%
43%
41%
65%
BaRT-aarge
HEEL
AVG
44.4
55%
48%
48%
52%
41%
62%
GPT2
ISD
41.3
43%
.9%
38%
58 %
27%
4.5%
GPT2-medium
HEL
42.3
54%
50%
39%
56%
24%
41%
RoBERTa-base
JSD
51%%
55%
66%
RoBERTa- lurge
JSD
12
AVG
42.3
40%%
50%
43%
44%
48%
56%%
XLNet-base
HEL
AVG
48.3
62%
53%
50%
58 %
49%
74%
XLNet-large
HEL
AVG
46.7
5756
50%
54%
50%
57%
73%
Other models
PRPN(tune))*+
47.3
50%
59%
46%
57%
44%
32%
ON(uned))1
48.1
51%
64%
54%
38%
Neural PCFG!
71%
Compound PCFG
55.2
56%
74%
41%
68 %
40%
52%
we confirm that the main contribution of the bias comes from its capability to capture subordinate clauses (SBAR) and verb phrases (VP). This observation provides hint for what some previous work on unsupervised parsing desired to obtain by introducing the bias to their models. It is in- triguing to see that all of the existing grammar induction models are inferior the gighhbranching baseline in recognizing SBAR and VP (although some of them already utilized the right-skewness bias), implying that the same problem- mooell do not properly capture the branhhi-banningggg nature- may also exist in current grammar induction models. One possible assumption is that the models do not need the bias to perform well in language modeling, although future work should provide rigorous analysis about the phenomenon.
On the other hand, the existing models show exceptionally high recall scores on noun phrases (NP), even though pre- pre-trained LMs also have success to extent in capturing noun phrases com- pared to naive baselines. From this, we conjecture that neural models trained with a language modeling objective become largely equipped with the ability understand the concept of NP. In contrast, the pre-trained LMs record the best recall scores on adjective and adverb phrases (ADJP and ADVP), suggesting that the LMs and existing models capture disparate aspects of English syn- tax to differing degrees. To further explain why some pre-trained LMS are good at capturing ADJPs and ADVPs, manually investigated the attention heatmaps of the sentences that contain ADJPs or ADVPs. From the inspection, we empirically found that there are some keywords-inchee 'two', 'ago', 'too', and "fr''whichh have different patterns of attention distributions compared to those of their neighbors and that these keywords can be clue for our framework to recognize the existence of ADJPs or ADJPs. It is also worth mentioning that ADJPs and ADVPs consist of relatively smaller number of words than those of SBAR and VP, indicating that the LMs combined with our method have strength in correctly finding small chunks of words, i.c., low-level phrases.
Meanwhile, in comparison with other LM models, GPT-2 and XLNet based models demonstrate their effectiveness and robustness unsupervised parsing. Particularly, the XLNet-base model serves robust baseline achieving the top performance among LM candidates. One plausible explanation for this outcome is that the training objective of XLNet, which considers both autoen- coding (AE) and autoregressive (AR) features, might encourage the model to be better aware of

---

Published as a conference paper at ICLR 2020
40
40
36
36
32
32
Sentence
Sentence
24
24
Best hace god
beet Sarge gpe2-meddmm roburta ange xnab-arr
20
20
rotorta bosz xire
16
16
20 24
Layer
Layer
Figure 2: The best layer-wise S-FI scores of each LM instance on the PTB test set. (Left) The performance of the X-'base" models. (Right) The performance of the X- X-'large' models.
phrase structure than other LMs. Another possible hypothesis is that AR objective functions (e.g., typical language modeling) are more effective in training syntax-aware neural models than AE ob- jectives (c.g., masked language modeling). as both GPT-2 and XLNet are pre-trained on AR variants. However, it is hard to conclude what factors contribute to their high performance this stage.
Interestingly, there an obvious trend that the functions in the distance measure functions for attention distributions-lee lead most of the LM instances to the best parsing results, indicating that deriving parse trees from attention information can be more compact and efficient than extracting them from the LMs' intermediate representations, which should contain linguistic knowledge be- yond phrase structure. In addition, the results in Table [ show that large parameterizations of the LMs generally increase their parsing performance, but this improvement not always guaranteed. Meanwhile, as we expected in Section 4.2 and as seen in the 'A' (attention head number) column of [ the average of attention distributions the same layer often provides better results than individual attention distributions.
5.3 EXPERIMENTAL RESULTS ON MNLI
We present the results of various models on the MNLJ set in Table 3 Appendix A.5 We
observe trends in the results which mainly coincide with those of the PTB dataset. Particularly, (1) right-branchin trees are strong baselines for the task, especially showing their strengths in captur- ing SBAR and VP clauses/phrases, (2) our method resorting the LM instances is also comparable to the brahh-bbanching trees, demonstrating its superiority in recognizing different aspects of phrase categories including prepositional phrases (PP) and adverb phrases (ADVP), and (3) attention dis-
tributions seem more effective for distilling the phrase structures of sentences than intermediate representations.
However, there are some issues worth mentioning. First, the right-branching baseline seems to be even stronger in the case of MNLI, recording score of over 50 sentence-level F1. We conjecture that this result comes principally from two reasons: (I) the average length of sentences in MNLI is much shorter than in PTB, giving disproportionate advantage naïve baselines, and (2) our data preprocessing, which follows Kim et al. (2019)), removes all punctuation marks, unlike previous work (Htut al. 2018 Drozdov al.) 2019). leading to an unexpected advantage for the right- branching scheme. Moreover, it deserves to consider the fact that the gold-standard parse trees in MNLI are not hunan-andotated, rather automatically generated.
Second, in terms of consistency in identifying the best choice of f and g for each LM, we observe that most of the best combinations of f and g tuned for PTB do not correspond well to the best ones for MNLI. Does this observation imply that specific combination of these functions and the resulting performance do not generalize well across different data domains? To clarify, we manually investigated the performance of some combinations of and g. which are tuned PTB but tested on MNLI instead. As a result, we discover that particular combinations of and which are good PTB are also competitive MNLI, even though they fail to record the best scores MNLI.
8

---

Published as a conference paper at ICLR 2020
Table Results of training a pseudo-optimum fideal with PTB and XLNet-base model.
Model
S-F1
SBAR
NP
VP
PP
ADJPP
ADVP
Baselines (from Table 1)
Random Net-hhee (F*)
19.6
9%
26%
12%
20%
23%
24%
Random XLNNtt-ban Net-buse (F
20.1
11%
25%
14%
19%
22%
26%
J1.Net--he (A=0)
ISD HEL
AVG
40.1
35%
56%
26%
38%
47%
68%
XLNet-base (x115))
AVG
48.3
62%
53%
50%
58%
49%
74%
Trained models (w/ gold trees)
Random XLNeetbuus buse
fakei
41.2
28%
58%
29%
50%
35%
41%
XL.Net. base (warst
Fakal
58.0
47%
75%
56%S
71%
50%
61%
Xe.Net-base (best case)
fike
65.1
61%
82%
67%
78%
88%
73%
Concretely, the union (ISD) and 13) the best duo for the XLNet-base on PTB achieves 39.2 in sentence-level FI on MNLI, which is very close to the top performance (39.3) we can obtain when leveraging the XLNet-base. It is also worth noting that GPT-2 and XLNet are efficient in capturing PP and ADVP respectively, regardless of the data domain and the choice of f and g-
FURTHER ANALYSIS
6.1 PERFORMANCE COMPARISON BY LAYER
To take closer look at how different the layers of the pre-trained LMs are in terms of parsing performance, we retrieve the best sentence-leve F1 scores from the Ith layer of an LM from all combinations of and gi, with regard to the PTB and MNLI respectively. Then we plot the scores as graphs in Figure 2 for the PTB and Figure in Appendix A.4 for the MNLI. Each score is from the models which the bias is not applied.
From the graphs, we observe several patterns. First, XLNet-based models outperform other com- petitors across most of the layers. Second, the best outcomes largely shown in the middle layers of the LMs akin to the observation from Shen el al. (201)), except for some cases where the first lay- ers (especially in case MNLI) record the best. Interestingly, GPT-2 shows a decreasing trend in its output values as the layer becomes high, while other models generally exhibit the opposite pattern. Moreover, we discover from raw statistics that regardless of the choice of f and yt, the parsing per- formance reported as S-F1 is moderately correlated with the layer number 1. In other words, it seems that there are some particular layers the LMs which are more sensitive to syntactic information.
6.2 ESTIMATING THE UPPER LIMIT OF DISTANCE MEASURE FUNCTIONS
Although we have introduced effective candidates for f, we explore the potential of extracting more sophisticated trees from pre-trained LMs, supposing call it fidcal- To obtain fideal, we train a simple linear layer on each layer of the pre-trained LMs with supervision from the gold-standard trees of the PTB training set, while g remains unchanged-the pre-trained LMs are frozen during training. We choose the Xl.Net-base model as a representative for the pre-trained LMs. For more details about experimental settings, refer Appendix
equipped with pseudo-optimum f,
In Table 2, we present three new results using fideat- As baseline, we report the performance of fidcal with a randomly initialized XLNet-base. Then, we list the worst and best result of fideal according to when it is combined with the pre-trained LM. We here mention some findings from the experiment. First, comparing the results with the pre-trained LM against one with the random LM, we reconfirm that pre- pre-traine trainiinnng an LM apparently enables the model to capture some aspects of grammar. Specifically, our method is comparable to the linear model trained on the gold- standdadd trees. Second, we find that there is tendency for the performance of fitel relying on different LM layers to follow one we already observed Section (the best result comes from the middle layers of the LM while the worst from the first and last layer. Third, we identify that the LM has a potential to show improved performance on grammar induction by adopting a more sophisticated emphasize that method equipped with simple f without gold-standard trees
f. However, is remarkably reasonable in recognizing constituency grammar, being especially good at catching ADJP and ADVP.

---

Published as a conference paper at ICLR 2020
6.3 CONSTITUENCY TREE EXAMPLES
We visualize several gold-standard trees from PTB and the corresponding tree predictions for com- parison. For more details, we refer readers to Appendix|A. A.7
CONCLUSIONS AND FUTURE WORK
In this paper. we propose a simple but effective method of inducing constituency trees from pre- trained language models zero-shot learning fashion. Furthermore, we report set of intuitive findings observed from the extracted trees, demonstrating that the pre-trained LMs exhibit some properties similar to constituency grammar. addition, we show that our method can serve as strong baseline for English grammar induction when combined with (or even without) appropriate linguistic biases.
On the other hand, there are still remaining issues that can be good starting points for future work. First, although we analyzed our method based on two popular datasets, we focused only on English grammar induction. As each language has its properties (and correspondingly would need individualized biases), it is desirable to expand this work to other languages. Second, it would also be desirable investigate whether further improvements pre-trained LMs onto existing grammar induction models. Lastly, by verifying the usefulness of the knowledge from the pre-trained LMs and linguistic biases for grammar induction, we want point out that there is still much room for improvement in the existing grammar induction models.
be achieved by directly grafting the
ACKNOWLEDGMENTS
We would like to thank Reinald Kim Amplayo and the anonymous reviewers for their thoughtful and valuable comments. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRP201663C44775257,
REFERENCES
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd international confer- ence on Machine learning, pp. 99-96. ACM, 2005.
Jihun Choi, Kang Min Yoo, and Sang-goo Lee. Learning to compose task-specific tree structures. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Alexander Clark. Unsupervised induction of stochastic context-free grammars using distributional clustering. In Proceedings of the 2001 workshop on Computational Natural Language Learning- Volume 7, pp. 13, 2001.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNI.P Analyzing and Interpreting Neural Networks for NL.P, pp. 276-286, Florence, Italy, August 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4171-4186, Minneapolis, Minnesota, June 2019.
Andrew Drozdov, Patrick Verga, Mohit Yadav, Mohit lyyer, and Andrew McCallum. Unsupervised latent tree induction with deep inside-outside recursive auto-encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1129-112 1141, Minneapolis, Minnesota, June 2019.
10

---

Published as a conference paper at ICLR 2020
Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies, pp. 199-209, San Diego,
California, June 2016.
Chris Dyer, Gábor Melis, and Phil Blunsom. A critical analysis of biased parsers in unsupervised parsing. arXiv preprint arXiv: eetei.999999999 2019.
Kawin Ethayarajh. How contextual are contextualized word representations? comparing the ge- ometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55-65, Hong Kong, China, November 2019.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.05255 2019.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. arXiv preprint arXiv: 1908.000000 05662,, 2019.
Serhii Havrylov, Germán Kruszewski, and Armand Joulin. Cooperative learning of disjoint syntax and semantics. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1118-1128, Minneapolis, Minnesota, June 2019.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Pa- Minneapolis, Minnesota, June 2019.
pers), pp.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735- 1997.
Phu Mon Htut, Kyunghyun Cho, and Samuel Bowman. Grammar induction with neural language models: An unusual replication. In Proceedings of the 2018 EMNLP Workshop BlackboxNIPP. Analyzing and Interpreting Neural Networks for NLP, pp. 371-373, Brussels, Belgium, November 2018.
Yoon Kim, Chris Dyer, and Alexander Rush. Compound probabilistic context-free grammars for grammar induction. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 2369-2385, Florence, Italy, July 2019a.
Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Unsuper- vised recurrent neural network grammars. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pp. 1105-1117, Minncapolis, Minnesota, June 2019b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Nikita Kitaev and Dan Klein. Constituency parsing with self-attentive encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2676-2686 Melbourne, Australia, July 2018.
Dan Klein and Christopher Manning. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 478 485, Barcelona, Spain, July 2004.
Dan Klein and Christopher D. Manning. A generative constituent-conter model for improved gram- mar induction. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 128 135, Philadelphia, Pennsylvania USA, July 2002.
Dan Klein and Christopher D Manning. Accurate unlexicalized parsing. In Proceedings of the 41st Anmual Meeting on Association for Computational Linguistics- Volume 1, pp. 423 033-430. Associa- tion for Computational Linguisties, 2003.
11

---

Published as a conference paper at ICLR 2020
Karim Lari and Steve J Young. The estimation of stochastic context-free grammars using the inside- outside algorithm. Computer speech & language, 4(1):35 56, 1990.
Bowen Li, Lili Mou, and Frank Keller. An imitation learning approach to unsupervised parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- Florence, Italy, July 2019.
Nelson Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1073-1094, Minneapolis, Min- nesota, June 2019a.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692,, 2019b.
Jean Maillard, Stephen Clark, and Dani Yogatama. Jointly learning sentence embeddings and syntax with unsupervised tree-Istms. Natural Language Engineering, 25(4):433-449 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 1 19(2):313-330, 1993.
David Mareček and Rudolf Rosa. Extracting syntactic trees from transformer encoder self- attentions. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Inter- preting Neural Networks for NLP, pp. 347-349, Brussels, Belgium, November 2018.
David Mareček and Rudolf Rosa. From balustrades to pierre vinken: Looking for syntax in trans- former self-attentions. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 263-275, Florence, Italy, August 2019.
Nikita Nangia and Samuel Bowman. Listops: A diagnostic dataset for latent tree learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Student Research Workshop, pp. 92-99, 2018.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers). pp. 2227-2237, New Orleans, Louisiana, June 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Alessandro Raganato and Jörg Tiedemann. An analysis of encoder representations in transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Ana- lyzing and Interpreting Neural Networks for NLP. pp. 287-297, Brussels, Belgium, November 2018.
Rudolf Rosa and David Mareček. Inducing syntactic trees from bert representations. arXiv preprint arXiv: 11511, 2019.
Yikang Shen, Zhouhan Lin, Athul Paul Jacob, Alessandro Sordoni, Aaron Courville, and Yoshua Bengio. Straight to the tree: Constituency parsing with neural syntactic distance. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1171-1180, Melbourne, Australia, July 2018a.
Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. Neural language modeling by jointly learning syntax and lexicon. In International Conference on Learning Representations, 2018b.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Repre- sentations, 2019.
12

---

Published as a conference paper at ICLR 2020
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. Visually grounded neural syntax acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1842-1861, Florence, Italy, July 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 5998-6008, 2017.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, July 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations, 2019.
Adina Williams, Andrew Drozdov, and Samuel R. Bowman. Do latent tree learning models identify meaningful structure in sentences? Transactions of the Association for Computational Linguis- tics, 6:253-267, 2018a.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad -overace challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pp. 1112-1122, New Orleans, Louisiana, June 2018b.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems 32, pp. 7764,, 2019.
Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling, Learning to com- pose words into sentences with reinforcement learning. In International Conference Learning Representations, 2017
13

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 ATTENTION HEATMAP EXAMPLES
8.
0.40
Neventretss
moat
035
0.0
0.50
STATS
0.0
desresdorr
0.00
300137
Movetteoo
ent
Naccela
ras
Mered
2.35
Dote
2.35
malze
65
mest
mimum
64
at
recuring
The
tr
camh
morth
pricon
ore
nebl
-0.0
022
W012
sove
0027
rete
00.0
6.1
0.S
Abone
the
us
anceg
the
Uva
ko015
0.30
Niting
mh
Detrot
0.25
Ecbon
Co
Ibout
6655
mon
ins (00)
On
wat
WAY
Figure 3: Self-attention heatmaps for the average of all attention distributions from the 7th layer of the XLNet-base model, given a of input sentences.
14

---

Published as a conference paper at ICLR 2020
A.2 TREE CONSTRUCTION ALGORITHM WITH SYNTACTIC DISTANCES
Algorithm Syntactic Distances to Binary Constituency Tree (originally from Shen et al. (2018a)
1: [wi W2 W..: a sequence of words sentence length
2: [i, d2..... dn dn-1]: 1]:
a vector whose elements are the distances between every two adjacent words.
3: function TREE(S, d)
4: if then
node (())
else
7:
i max; (d)
8:
child, <- TREE(Se:. dei))
child, TR d><)
node + Node(child,, child,)
11:
end if
12:
return node
13:
end function
A.3 DISTANCE MEASURE FUNCTIONS
Table The definitions of distance measure functions for computing syntactic distances between two adjacent words in sentence. Note that g" (w:w), = g" (w;+t)) P (w()), and Q g" (w(,+)), respectively. hidden embedding size, n: the number of words (w) in a sentence (S).
Function (f) Definition
Functions for intermediate representations (F")
so)
(r ( -Z 8;3) 12/2
(i(r,s)
Σ1 2i I 5
(2
- (r. ( s1)
Functions for attention distributions (Fa)
JSD(PIIQ)
((DKI. MPI + DxL. Q M))/2)+
where M (P Q) /)
and O E Lwes A(w) log(A(w)/B(u))
HEL(P, Q)
EE (VPi 2)
A.4
PERFORMANCE COMPARISON BY LAYER ON MNLI
52
52
Sentence-leve Seeenc--veeel F1
44
Sentence-lever level
40
40
36
36
28
28
bert-large. apt2 medium robertu--aage strat-lange
gpt2
24
rolberta-base
24
abrict
20
20
1
2
3
6
7
10
11
12
10 12 14
16
18
20
22 24
Layer
Layer
Figure 4: The best layer-wise S-FI scores of each LM instance on the MNLI test set. (Left) The performance of the "b'ase' models. (Right) The performance of the X- 'large models.
15

---

Published as a conference paper at ICLR 2020
A.5 EXPERIMENTAL RESULTS ON MNLI
Table 4: Results on the MNLI test set. Bold numbers correspond to the top results for each column.
L: layer number. A: attention head number (AVG: the average of all attentions). f: Results reported
by Htut et al. (2018) 18) and Drozdov et al. (2019). $: Approaches in which COO parser is utilized. *:
These results are not strictly comparable ours, due the difference in data preprocessing.
Model
L
A
S-F1
SBAR
NP
VP
PP
ADJP
ADVP
Baselines
Random Trees
21.4
11%
25%
16%
22%
22%
27%
Balanced Trees
20.0
8%
29%
11%
20%
22%
32%
Left Branching Trees
8.4
Right Branching Trees
51.9
65%
28%
75%
47%
45%
309
Random XI N..Net-han (FR)
1.%
22%
XLNet-base (F
25%
Pre-trained LMs (wlo bins)
BERT-base
HEL
9
10
36.1
36%
37%
34%
45%
26%
42%
Bargelarge
JSD
17
10
37.0
38%
3218
344%)
50%
22%
39%
GPT2
JSD
10
44.0
4.3
53%
31%
60%
24%
40%
GPT2-medium
JSD
3
49.1
57%
32%
61%
44%
35%
37%
RaBERTa buse
ISD
10
9
36.2
26%
3.5%
34%
50%
2.3%
44%;
RoBERTa-lagge
JSD
20%
35%
28%
27%
XLNet-basc
HEL
39.0
25%
39%
28%
59%
35%
444%
XeNeetlarge
HEL
15
42.2
32%
49%
27%
62 %
325
49% %
Pre-trained LMs (w/ blas A=1.5)
BERT-base
HEL
2
12
$2.7
64%
35%
70%%
50%
46%
30%
BaRT-laage
30%
GPT2
HEL
10
52.2
57%
53%
49%
62%
32%
42%
GPT2-medium
HEL
2
53,9
53%
57%
50%
62%
299.
44%
RaBERTa bose
HEL
52.0
64%
31%
72%
49%
47%
30%
RoBERTa-large
L1
23
$2.7
55%
40%
65%
53%
43%
41%
XLNet-base
L2
57%
61%
55%
44%
57% %
XLNNt-large large
L22
12
53.5
54%
47%
59%
51.
48%
60% %
Other models
PRPN-UP!:
DIORA
51.2"
DIORA(-PP)*
$9.0"
A.6 EXPERIMENTAL DETAILS FOR TRAINING IDEAL DISTANCE MEASURE FUNCTION
In this part, we present the detailed specifications of the experiments introduced in Section 6.2 We assume /itea only compatible with the functions G as the functions Gd are not suitable for training the sizes of the representations provided by Gd are variable according to the length of an input sentence. To train the pscudo-optimat function fidcal- we minimize pair-wise learning-to-rant loss following previous work (Burges et al. 2005 Shen et al. 2018a):
rank dist
Σ [1 sign(doold b
(gold (d) drred 1+
(3)
where doold and dredd are computed from the gold tree and our predicted one, respectively. [x] is defined maz(0,x). We train the fidcal with the PTB training set for epochs. Each batch of the training set contains 16 sentences. We use an ADAM optimizer (Kingma & Ba 2014) with the learning rate 5e-4. We train the variations of fiteal differentiated by the choice of G and report
the best result in the Table 2 Each fideal is chosen based on its performance on the PTB validation set. Considering the randomness of training, every result for fideal averaged over 3 different trials.
A.7 CONSTITUENCY TREE EXAMPLES
We randomly select six sentences from PTB and visualize their trees, where the resulting group of trees for each sentence consists of gold constituency tree and two induced trees (one without the right-skewness bias and the other with the bias) from our best character in the induced trees indicates a dummy tag.
The 'T'
16

---

Published as a conference paper at ICLR 2020
NP-SBU
VP
These Includo
NP
CC
NP
childccae nitilive
NNS
NP
extens
NHNS
soon-oo exprr
broaks
NP
CG
RP
low-income housing
childicare Intiaaive
These indluce
and
for
research and cevelgmmerixxpprrdure
low housing
coan-to-xxpire tax breaks
estensions
inslude
and
extonsions of scon expire
breaks
low Income housing
research and developmemsssennduuse
Figure 5: Gold (top) and predicted trees (one withou the bias in the middle, the other with the bias the bottom) for the sentence "These include a child-care initiative and extensions of soon-to soon-to-expins to-epxire breaks for low-income housing and research develoomlp expenditures'.
17

---

Published as a conference paper at ICLR 2020
CC Bul
NP-SBU
VP
NP
NN
MD
VP
POS
ofter
would
VP
HÓFI
have VEN
NP
NP
NP
NNS
NP
Pp
NNP other shareholders
Ideal
RB
CD %
of
DT
NN
about
10
th combined company
But
Ideal
HOFI
given
about
would have
other shareholders
of
combined company
olier
But
HÓFI
Ideal
about
8
given
other shareholders
10 of
first olfer have
the
combined company
Figure 6: Gold (top) and predicted trees (one withou the bias in the middle, the other with the bias at the bottom) for the sentence 'But HOFI 's first offer would have given Ideal 's other shareholders about 10 % of the combined company'.
18

---

Published as a conference paper at ICLR 2020
CC
S
NP-SBJ
VP
and
NP-SBJ
VP
PRP
VBD
NP.PRD
DT
NN
VBD
NP-EXT
was
NÑP
DT NNP
stock plummeteeeeeeed
NNS
Friday the 13th
points
nearly
200
was Friday and
plummeted
the 13th the
nearly
stock market
200 points
and
was
Friday
the
plummeted
the 13th
stock market
nearly
200 points
Figure 7: Gold (top) and predicted trees (one without the bias in the middle, the other with the bias at the bottom) for the sentence 'It was Friday the I 13th and the stock market plummeted nearly 200 points'.
19

---

Published as a conference paper at ICLR 2020
PP- PP-TMP
NP-SBJ
VP
ADVP
PP-LOC
Vp
Until
RB
NNS
IN
NP
VBD
NP
and
VBD
NP
recently national governments
NNP
controlled
NP
Pp
allowed
ADJP
NN
Europe
JJS IN
Jj CC DT adventising
most of
DT
NN NN
litlle or no
time
Until recently national
controlled
govemments
time and allowed
advertising
Europe
most little
the no
recently
controlled
national govemments
Europe
most
and allowed litlle
the
or
time
advertising
Figure 8: Gold (top) and predicted trees (one without the bias in the middle, the other with the bias at the bottom) for the sentence 'Until recently national governments in Europe controlled most of the air time and allowed little advertising'.
20

---

Published as a conference paper at ICLR 2020
ADVP
NP-SBJ
RB
NNP
NNP
VBD
SBAR
Neverthleess
Ms. Garzareli said
NP-SBJ-1
VP
PRP
VID
VP
VBN
PP-CLR PP-TMP
PP-2
swamped
NP
NP
with over
from
NNS
NN NNS the
weekend
nervous shareholders
phone calls
Nevertheless
Ms. Garzarelli said she
from
weekend nervous shareholders
with phone calls over the
was swamped
Nevertheless
Garzarelli
she
swamped
phone calls
weekend
over the
nervous shareholders
Figure 9: Gold (top) and predicted trees (one without the bias in the middle, the other with the bias the bottom) for the sentence 'Nevertheless Ms. Garzarelli said she swamped with phone calls over the weekend from nervous shareholders'.
21

---

Published as a conference paper at ICLR 2020
NP-SBJ
ADVP
VP
NNS
CC
NNS
VBP
SBAR
Analysts and competitors however doubt
S
SPJSBJ
VP
DT
NNS
VBD ADJP-PRD
the numbers were
RB
JJ
that high
Analysts
and competitors however doubt
were that high
the numbers
Analysts
and competitors however doubt
the numbers were
that high
Figure 10: Gold (top) and predicted trees (one without the bias in the middle, the other with the bias at the bottom) for the sentence 'Analysts and competitors however doubt the numbers were that high'.
22


Reviews:
[{"review_id": "rkgaaBeRYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am satisfied with the author's response and I am increasing the score to 6 after the rebuttal.\n\n===\n\nThis paper introduces a new and simple method of probing whether syntax information under the form of constituency trees is present in recent pre-trained language models (e.g. BERT, RoBERTa, XLNet and GPT2) without any additional task-specific training. They use the previously proposed concept of \"syntactic distance\" [1] as a sufficient statistics for the constituency tree of a given sentence. They compute the distance between neighboring words in sentence using a distance function f(g(w_i), g(w_i+1)) where f is a divergence between distributions and g(w_i) is the self-attention distribution of w_i at a given layer: the intuition is that words that have similar self-attention distributions belong to the same constituent and thus their syntactic distance is small. In order to conform with right-skewness of English syntax, they propose an affine transformation of the distances that encourage right-skewed trees, and tune the hyper-parameter directly on the target F1 score wrt ground-truth trees. Results suggest that large, pretrained models capture constituency trees to some extent.\n\nThis is an empirical and analysis paper which can be considered as the “twin” of Hewitt et. al (2019, analyzes whether dependency structure can be probed from large pre-trained models). The paper is well-written and easy to understand. The experiments are in general well-organized even if they lack clarity at some point. The related work is comprehensive and covers most of the work in the domain of grammar induction and unsupervised parsing. In its current form however, I feel like the paper misses a \"second half\" therefore it isn’t quite above the acceptance bar. I'd be happy if the authors could kindly elaborate on the following major weaknesses: (i) unclear scientific motivation of using the right-skewness bias; (ii) lack of clarity in the experimental results; (iii) lack of depth and perspective.\n \n1) About right-skewness bias:\nI appreciate that the authors clearly guard against using explicit right-skewness biases. However, I am not sure why the authors use of Eq. (2) apart from boosting performance. The authors claim that \"the main purpose of explicitly injecting such a bias is examining what changes are made to the resulting tree structures\" but the purposed changes are not discussed at length in the text. Therefore:\n1.1) What's the scientific motivation (apart from boosting performance) of skewing the trees using Eq. (2) ?\n1.2) What can you infer from trees without bias and tree with bias ?\n\n2) About clarity in the experimental results:\n2.1) (Shen, 2018) use an \"implicit\" right-skewness bias. From the text, it seems that your results w/o bias use the same bias as (Shen, 2018) ? Therefore they actually have a bias ? This is a bit confusing. What about using the unbiased parsing algorithm as presented in \"Straight to the tree...\" ?\n2.2) This sentence is unclear: \"we select one derived from the best choice of f and g in terms of sentence-level F1 (S-F1)\nw.r.t. gold-standard trees as a representative for the LM\". Are you using S-F1 on *training* for tuning f and g ?\n2.3) This sentence is unclear: \"As LMs are not fine-tuned with training sets in our framework, we only use the test\nset of the PTB\", what do you mean by \"using the test set\" ? Do you mean evaluate or use it for tuning (cf 2.2) ?\n\n3) About perspective and depth:\n3.1) The fact that ADVP , ADJP performance is higher for the tested models is interesting. Do you have any hypothesis for why this is happening?\n3.2) Right-skewness seems to mainly help with VP. Can you formulate an hypothesis for why current models underperform on this label or do not show right bias?\n3.3) The results on Table 2 are interesting, but imho, they are basically showing that these models cannot do constituency parsing with just a linear probe on top of the representations and using the syntactic distance algorithm?\n3.4) Did you consider using other parsing algorithms e.g. chart parser, instead of just the syntactic distance ? That would make a stronger paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxacNTpFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n*Summary \n\nThis paper describes an effective method that induces constituency trees from pre-trained language models, which are attracting great attention recently. The authors demonstrated that the pre-trained language models have some properties that are similar to constituency grammar by showing some interesting features of the extracted trees. \nThis study is based on the motivation to unveil the reason why such pre-trained language models work and the extent to which pre-trained language models capture the syntactic notion of the constituency.\nThey also show that the method can become a reasonable baseline method for English grammar induction.\n\n*Decision and supporting arguments\n\nI'm leaning toward accepting the paper as a conference paper.\nAs the author describes, pre-training LMs are attracting attention, and many people want to understand their inner workings. The paper provides a systematic analysis of the pre-trained LMs from the viewpoint of grammar induction. Also, this paper is well-structured and offers a good literature review.  I think the paper has enough value to be published from the scientific viewpoint. \n\n\n*Additional feedback\n\nFigure 1 & 5-10 look rasterized. It's better to use vector images.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxE0JaLOB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[EDIT: Thank you very much for the thoughtful and extensive response. Based on the latest iteration of the paper, I am changing my score to 8]\n\nThis paper studies the representations learned by large pre-trained models trained on language modeling objective (or language modeling-like objective, in the case of masked models). In particular, the authors investigate whether constituency information is present in the hidden layers. In contrast to much existing work that probe for this information with (for example) linear models on top of hidden representations, the authors propose to directly extract binary trees from the model using \"syntactic-distance\" measure that is calculated in various ways (e.g. dot product of hidden representations, distance in attention distributions). Across various models, the authors find that it is indeed possible to induce linguistically meaningful trees, in particular outperforming a right-branching baseline that is strong for English\n\nI found this to be a creative alternative to the existing \"BERTology\" type papers that rely mostly on linear probes, and the experiments are done across a wide number of setups (e.g. across models/datasets).  However, I have several questions/issues with the paper if addressed, would make the paper much stronger:\n\n- The degrees of freedom afforded by the choice in number of layers, heads, similarity measure, etc. is quite high. For example, with 24 layers and 16 attention heads I count (24*(16+1)*2 + 24*3) = 888 different measures of distance that could be used to induce trees. I think the authors should make sure that they are not \"overfitting\" to the test set with the following set of tests:\n\n1. Only use the training set to select the layer/metric combination.\n2. See if the selected combination generalizes to other languages (from comparing Table 1 against Table 2, it seems like this doesn't even generalized within the same language across different domains? This is worrying).\n3. See what the performance is with random initialization. In particular, I would like to see two setups: (a) the network is randomly initialized, (b) the *representations* (i.e. attention/hidden states) are randomly initialized. The random initialization should have variance such that it's not just uniform distributions.\n\n- I found the right branching bias baseline experiment not very informative. As the authors allude to, performance on unsupervised parsing itself is not so interesting, and the point of the paper (in my view) is not to get the best unsupervised parsing performance.\n\n- Related to the above, it is known that the decoding algorithm of Shen et al. 2018 is itself heavily biased towards right branching trees. See in particular: https://arxiv.org/pdf/1909.09428.pdf. Given this, I am not sure if the supposedly good performance in parsing is due to the model actually learning constituency or just the bias.\n\n- It would have been interesting to see how the performance changes as:\n1. The model is fine-tuned on the PTB-training set.\n2. The model is trained from scratch on PTB (obviously with a much smaller model since PTB is smaller).\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents results of looking at the inside of pre-trained language models to capture and extract syntactic constituency. Reviewers initially had neutral to positive comments, and after the author rebuttal which addressed some of the major questions and concerns, their scores were raised to reflect their satisfaction with the response and the revised paper. Reviewer discussions followed in which they again expressed that they became more positive that the paper makes novel and interesting contributions.

I thank the authors for submitting this paper to ICLR and look forward to seeing it at the conference..
Decision: Accept (Poster)