Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution
Authors: Jaap Jumelet
Willem Zuidema
Venue: EMNLP 2023 Findings (2023)
Keywords: language models
perplexity
interpretability
PCFG
learning dynamics
synthetic data
TL;DR: Training Language Models on data generated from a transparent generative process allows for more controlled evaluation of their behaviour.

Abstract: We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.

Transparency at the Source: With Access to the True Distribution
Evaluating and Interpreting Language Models
Jaap Jumelet
Willem Zuidema
Institute for Logic, Language and Computation University of Amsterdam
{ fj.w. w. w. jumelet,
w. zuidemm)) e @uva. .n
Abstract
P (He Unddrr the building) -3.
(e Under building) - -2.21
We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data gen- erated using massive probabilistic grammar (based on state-split PCFGs), that is itself de- rived from a large natural language corpus, also provides complete control over the gen- erative process. We describe and release both grammar corpus, and test for natural- ness of our generated data. This approach allows us to define closed-form expressions efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives how closely they allow approximating lower bound perplexity. Our approach also allows us to directly compare learned repre- sentations symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access the underlying true source, our results show striking differences outcomes learning dynamics between different classes of words.!
log Pacza (w)
-10
lag Pum(w) GPT2 PCFG p: 98.4, R²: 97.0
Figure 1: Spearman correlation between the log proba- bilities of LM with GPT-2 architecture and the PCFG distribution. The LM has obtained distribution that aligns significantly with the true PCFG distribution.
or with the expected perplexity given a (compute- optimal) scaling law (Kaplan et al., 2020; Hoff- mann et al., 2022).
This approach has been enormously successful, but leaves number of interesting questions unan- swered. First, one consequence of the absence an explicit stochastic source is that it is impossible to exactly determine the optimal perplexity that a language model can obtain. Second, when design- ing interpretability methods assess whether LM has learned specific linguistic rules, lack gold standard. If an interpretability method fails to find evidence that the LM has learned specific linguistic construction we cannot be sure whether this is a failure of the LM, a failure of the inter- pretability method, or whether our assumption that knowledge of this linguistic construction as an es- sential component of English fluency is wrong.
1 Introduction
When we train Language Model on large nat- ural language corpora, we are effect estimat- ing probability distribution over possible next tokens or masked tokens. The true distribution is unknown, so we cannot directly quantitatively mea- sure how good our estimation is, or qualitatively assess whether our LM has discovered 'true' under- lying patterns in the data. The best measure perplexity
do is to sample from the same unknown distribution, and compare that perplexity to the perplexity we obtain with other model archi- tectures or training regimes (Brown et al., 1992),
One approach address these problems is move to artificially generated data, but such data is often of trivial complexity compared to the richness
All code, data, and checkpoints are available at: github. wwt.cccc/cccoooo /l clab//laalaaaooooo......... haabb/ccfglim

---

Optimal Perplexity
(M Iw,,)
Pross (w, Iw,,)
+ VP NP + NN VP Grammar
The seld that thare not kmow how... Ded you sop that
Evaluate
LM
PP
Treebank
Corpus
Model
Induce: Petrov (2006)
Generate
Causal / Masked LM
Probe
Interpret
Figure 2: Conceptual overview of our experimental pipeline. First, we induce massive probabilistic grammar from natural language treebank. From this grammar we generate a corpus, that is used to train various language models on. Then, with access to the true distribution of our grammar, we evaluate and interpret our models.
of natural language, in terms the vocabulary size or the number of grammatical rules.
LMs and Synthetic Data
Various approaches have controlled the structure of the training data to investigate particular prop- erties of neural models or the learning process it- self. A major line of work in this direction does this to investigate the impact of typological fea- tures of a language example, Cotterell et al. (2018) and Mielke et al. (2019) train LMs on parallel, translated corpora, which allows them to isolate the impact of typo- logical differences between languages. Ravfogel et al. (2019) investigate typological differences as well, but create synthetic differences by modify- ing English sentences typological features such as subject-object-vert order, an approach that has been pursued earlier by Wang and Eisner (2016). In all these papers the starting point of the training data is an (English) language corpus of which the underlying generative process is unknown.
In this paper, we explore an alternative approach. We also use generated data to train language mod- els on, but to make sure the data approximates natural language in complexity, probabilistic grammar, that is itself derived from a large natural language corpus. To obtain the gram- mar, we use an automatically parsed section from the The Pile corpus (Gao et al., 2020), and use the state-split framework (Petrov et al., 2006) one of the most successful statistical parsing frameworks from before the rise of deep learning- to obtain statistical grammar with more than 2 million rules. In $3.1 we describe the procedure for obtaining this grammar in detail.
massive
model performance. For
This setup allows us to compute the exact lower bound on perplexity, although that computation turns out to still be nontrivial. This is due to the computational complexity of the problem that makes naive approach infeasible, even on mod- hardware. One key contribution from this pa- per is a closed-form expression efficiently com- pute masked token probabilities for PCFGs, com- plementing the classic closed form for causal lan- guage modelling (Stolcke, 1995). Furthermore, our setup provides a gold standard to qualitatively as- sess results from interpretability methods against. We conduct a wide range of experiments, showing the naturalness of our generated data, determining the impact of model and data properties on lan- guage model performance, and interpreting model behaviour and learning dynamies using various in- terpretability tools. Figure 2 presents a schematic overview our methodology.
There is a small set of papers where all data is derived from artificial languages, that are under full control. Dankers et al. (2022) generate naturalistic data for MT evaluation, but for models that were trained on natural data. Papadimitriou and Jurafsky (2023) investigate the inductive biases of LMs by pretraining models on data derived from simple grammars that isolate abstract linguistic properties such as nested recursion and Zipfian power laws. White and Cotterell (2021) provide a setup using simple PCFGs, in which typological features can be controlled. Hopkins (2022) observe that the sim- plistic nature of the PCFGs used in their work may not be reflective enough of natural language, and proposes Pitman-Ya Yor processes obtain artificial language that are more natural language-like. The grammars in these papers, however, remain fairly simple, and vocabulary size, sentence length and

---

other quantities stay limited compared to those seen in natural languages. Our setup provides massive PCFG that derived from natural language, with an efficient procedure for comparing the grammar's distributions to those of trained LMs.
3.2 Perplexity Lower Bound
The performance of a language model is often eval- uated terms of its perplexity on a test set (Jelinek et al., 1977; Jurafsky and Martin, 2009). Perplexity is defined as the normalised inverse probability of test set 2);
3 PCFGs and Perplexity
In our work we make use of large-scale PCFGs to generate data which we train various language models. In this section give brief background on PCFGs and how to evaluate modelling perfor- mance on PCFG-generated data using perplexity.
PPL(D) P(D)
In P(D) D
= exp
Since the 'true' distribution over natural lan- guage is unknown, there is no way of determin- ing the optimal perplexity that is attainable by a model. In our setup, however, we do have ac- cess to the generative process behind our data. It therefore becomes possible to compute an exact lower bound for the perplexity, allowing to press exactly how accurately LM modelled the distribution set out by the PCFG.
3.1 State-split PCFGs
There is a rich history in NLP of modelling lan- guage using interpretable, hierarchical grammatical formalisms. One of the most common formalisms is the Probabilistic Context-Free Grammar (PCFG), which provided the backbone for various success- ful parsing methods (Klein and Manning, 2003; Collins, 2003; Charniak and Johnson, 2005). A PCFG consists of simple weighted rewriting rules and can be used for both parsing and generation, by sampling and expanding rules based on the dis- tribution described by the PCFG.
The perplexity lower bound depends on the lan- guage modelling objective of interest: for masked language modelling it will be different than for causal language modelling. For both objectives we need to define a way to extract the required proba- bility distributions out of the PCFG. We describe how this is possible in the next two sections.
The contextsfreeness assumption of PCFGs, however, makes it challenging to capture non- syntactic dependencies between phrases. One successful approach that addresses this is the state splitting procedure of Petrov et al. (2006), colloqui- ally known the 'Berkeley Parser'. The general idea of state splitting that we can split the non- terminal symbols in a PCFG into fine- grained sub- symbols, that are specialised towards specific lin- guistic phenomena using automatic learning. The splitting process is repeated multiple times, in an alternating split-and-mergh strategy: if the splitting of symbol does not result in considerable im- provement of the data likelihood the subsymbols are merged back. This results in optimal trade- off between fine-grained con-mnaminal categories that still generalise beyond the training data. Fur- thermore, by keeping track of the history of split states we can project the final, fine-grained gram- mar back to the simpler, coarser base grammar. This then allows for highly efficient coarse-to-fine parsing, making it possible to parse sentences with large-scale grammars which would be intractable with normal PCFG (Petrov and Klein, 2007).
3.2.1 Causal Language Modelling
When doing causal language modelling, we aim to model the distribution P(w:/ (w:s): the proba- bility of token w, conditioned on its prior context. The perplexity of a sentence can directly be decom- posed into a product of such conditional probabili- ties. by continuously expanding the joint distribu- tion over the entire string:
PPL(w) = (II, P(wx/wwe)) 1o]
Extracting a token distribution from a PCFG G, Po(ws)usss, is a topic that has been explored in detail since the 1990s (Jelinek and Lafferty, 1991; Hale, 2001). An efficient procedure is that of Stolcke (1995), who adapts the Earley pars- ing algorithm to compute the probabilities of pre- fix strings Wh+ The conditional token probabil- ity can then casily be recovered as the fraction of two subsequent prefix strings: Po(w:/we:)
"Shannon (1951) famously provides an approximation of the perplexity of the English language on character level, but this does not yield an exact lower bound.
Forr example, a simple PCFG may encode that dog is a noun and bark a verh, but from single S-NP VP rule it can not be deduced that dogs are more likely to bark than cats.
"Also known as autoregre ssive or left-to-right language modelling; the hest fittng term remains a topic of debate.

---

((Wre)))))) In our experiments make use of the implementation of Stolcke's proce- dure by Luong et al. (2013), called EarleyX.
We provide a detailed derivation in Appendix A. Our formulation allows for a highly efficient proce- dure of extracting these probabilities from a PCFG, since we can make use of the optimised coarse- to-fine parsing methodology of Petrov and Klein (2007) to obtain the in- and outside probabilities.
3.2.2 Masked Language Modelling
When masked language modelling, we aim to approximate the probability distribution of P(w; (),)). which is achieved by masking out the token at position i with special mask token. Un- like in causal language modelling, sentence's s probability not be decomposed into masked token probabilities. To aggregate the performance of masked LM, we therefore need make use the heeuo-loo-likellhoon (PLL) (Salazar et al., 2020). The pseudo-log soaaddo-oo--keeeeee log-likelioood expresses the log probability sentence as a sum of masked token probabilities:
PCFG Corpus
Our approach can roughly be divided into three parts (Figure 2): i) data generation using state- split PCFGs, ii) language modelling using Trans- former LMs, and iii) model evaluation, via probing-based interpretability tasks with focus on learning dynamics. In this section explain the data generation procedure, and do analysis of our generated corpus.
(L- - In ((wc)wr))
E;
4.1 Data Generation
We can then use the PLL to compute the pseudo- perplexity as follows:
Grammar To generate data we make use of the state-split PCFG procedure of Petrov et al. (2006) as explained in $3.1. The treebank we use for learn- ing PCFG is parsed version of the BookCorpus (Zhu et al., 2015; Conneau et al., 2018), which was parsed using the Stanford Parser (Manning et al., 2014), and is nowadays part of The Pile corpus under MIT License (Gao al., 2020). We learn the PCFG on sample of 500,000 sentences, which is the maximum corpus size we could learn on our system with 32GB RAM memory. Learning is done for 5 split-merge cycles, filtering rules with a probability below 10 Part of the learning proce- dure is transforming the base grammar to X-bar style grammar, which ensures that each rule has at most two children.
(WW w
(-PPL(w) = exp
To compute this quantity for a PCFG we need find a way to extract masked token probabili- ties from the grammar: Pa(wilun,) Since such a procedure does not yet exist, we propose novel method.' Fortunately, it turns out to be possible to do this efficiently by employing the inside-outside algorithm (Baker, 1979; Manning and Schütze, 2001).
The algorithm defines two prob- ability distributions. The inside probabilities define the probability of a substring Wppg given that the substring has non-terminal N as parent:
Inside-Outside
The resulting PCFG consists of 54,497 unique tokens and 2.5M rules: 2.22M terminal rules, 272k binary non-terminal rules, and 13k umary non- terminal rules. The original treebank contained 96 non-terminal symbols: after split-merging we ob- tain 718 split non-terminals. We plot fine-grained plot of the number of times each of the original 96 non-terminals has been split in Appendix
8 g) = P (o))
(1)
The outside probabilities define the joint proba- bility of generating the non-terminal NO that spans position p q. as well as all the words outside the substring Upgi
ar q) = Pa(w1(p-1) Nop 1m)+
(2)
Data Having learned the PCFG from treebank, we can sample from it to generate our training cor- pus data. For training our models we generate 7.5 million sentences (97 million tokens), and develop- ment/essteevaluation corpora of 50k/50k/10k sen- tences, respectively. Sentence lengths are capped
Using the in- and outside probabilities, we can directly express the masked token probability in closed-form as follows:
()(), E
(3)
'Concurrent to our work, Zhao et al. (2023) find similar formulation for expressing masked token PCFG prohabilities.
/https://gtthub.co com/s avpetrov/ berkeleyparser

---

normalsed frequency
Original Treebank
Rew PCFG Corpus
Spitt PCFG Corpus
Spiil PCFG
Trestank
Correlation
with Treebank
Raw PCFG
Treebank PCFG
Row PCPG
Spesimm
1=12
1.12
d-2 12 8-3.22
12
Log Irequency
Log frequency
Log frequency rank
Senfence Langin
M-gram
Figure 3: Relationship between voken's frequency rank and its frequency, which is near linear on a log- log plot. Rank and frequency are computed over two disjoint splits of the corpora, following the procedure of Piantadosi (2014).
Figure 4: Distribution over sentence length for three corpora, and the correlation of n-gram distributions with respect to the original Treebank data.
two independent corpora, and investigate the Zip- fian relationship based on that. We estimate the a and / parameters of Eq. using the MLE imple- mentation of Vogelmann (2020).
between 6 and 25, and duplicate sentences al- lowed (but with no overlapping sentences between the train/developmenttest corpora): if we would not allow this, the data distribution that the model has access to would not reflect the original PCFG distribution. We provide more detailed analysis of the final grammar and corpora in the following section, and sample of sentences in Appendix C.
We plot our results in Figure 3. It can be seen that all three corpora follow Zipf's law with al- most equal parameterisation a and B. Inter- estingly, the generated corpora yield considerably lower residuals compared to the original treebank data. This demonstrates that in a natural language for infrequent tokens there exists greater uncer- tainty between the relation of frequency rank and frequency. Nevertheless, it is encouraging to see that generated corpus follows a similar Zipfian distribution, likely an important property for the learnability of language (Kurumada et al., 2013; Hendrickson and Perfors, 2019).
4.2 Naturalness of PCFG Corpus
In this subsection we briefly investigate whether the corpus generated from grammar follows similar patterns as natural language. We compare the properties of our fine-gu fine-grained PCFG to both the original Treebank, and a simple count-based PCFG that is derived directly from the original Treebank distribution. We refer to these three sources as Split PCFG, Treebank, and Raw PCFG, respectively.
Sentence Length We investigate the distribution over sentence lengths in the corpora, and plot the histograms in Figure 4a. Both the PCFG corpora are skewed towards shorter sentences, whereas the original Treebank data is more uniformly spread out. The Split PCFG is more similar to the Tree- bank distribution than the Raw PCFG. We could opt to to subsample from our Split PCFG corpus to obtain a similar sentence length distribution as the Treebank corpus, but in that case the corpus distribution would no longer align directly with the PCFG itself.
Zipf's Law One important linguistic feature that has been studied in detail within quantitative lin- guistics is that the frequency of word is logarith- mically proportional
frequency rank. This feature referred to as Zipf's law, and has been shown to be a universal language feature (Zipf, 1936). Zipf's law, in the updated form of Mandel- brot (1953), states that the th most common word in corpus has frequency that is proportional to
f(r) & (r+
(4)
N-grams One aspect of natural language that state-split PCFGs aim to cover is that dependencies between words are often semantically correlated, as explained in $3.1. This phenomenon is known as selectional preference, and has been used by Hopkins (2022) investigate the naturalness of their training data. There are many ways that se- lectional preference can be investigated, but we focus on a simple heuristic here. For increasing n
We can investigate this feature for our generated data, and compare this against the frequency dis- tribution of the original treebank. We follow the procedure of Piantadosi (2014), who argues that frequency and rank should not be computed on the same corpus, since that way a token's rank always spuriously correlated to its frequency. In- stead, we can compute the rank and frequency on

---

we compute the Spearman correlation between corpus' frequency distribution over n-grams and the distribution of the original Treebank data. A high correlation then indicates that the corpus con- tains n-grams of a similar (natural) structure, and a low correlation indicates that the original n-gram dependencies are less present in the data. We com- pute this also for the Treebank itself, by splitting the corpus in half.
that are not part of the model tokenizer (i.e. those mapped to <unk>), since the language models did not have access to the true distribution of the
original token. Based on the log probabilities of the language model and those of the PCFG, we can also compute their correlation. This allows us to determine to what extent the distribution of the LM is aligned with the true data distribution of the PCFG. We report two metries: Spearman's p. parametric rank correlation, and the R coefficient, which measures the proportion of variation the PCFG probabilities that is predictable from the LM probabilities.
The results are shown in Figure 4b. For n = 1, the PCFG corpora both obtain a correlation that higher than the Treebank with itself. This may be related to the large residual that we saw in the Zipf distribution of Figure 3, which showed that the uni- gram distribution of the Treebank corpus contains more variance in the tail than the PCFG corpora. For n > 1, it can be seen that the Split PCFG obtains higher Spearman correlation than the Raw PCFG, which demonstrates it has improved on the modelling of selectional preference.
5.2 Model Architecture
To emulate the learning behaviour of well-known Transformer LMs, we train LMs with similar archi- tectures. For masked language modelling we train BERT, RoBERTa, and DeBERTa architectures (De- vlin al., 2019; Liu et al., 2019; He et al., 2021) of much smaller size, inspired by the BabyBERTa models of Huebner et al. (2021). For causal lan- guage modelling we train GPT-2 and OPT architec- tures (Radford et al., 2019; Zhang et al., 2022). In our experiments we use model architectures with the following configuration: 8 layers, 8 heads, 256- dimensional hidden states, batch size of 64, and learning rate of 5 10-4, Although more optimal configuration may be found through more elab- orate hyperparameter search, it already provides a compute-friendly setup for training highly adequate models, as we will see in subsequent experiments. For training the LMs we use the trans f orners library (Wolf et al., 2020), training the models for epoch with cosine learning rate scheduler. For tokenization
Other Frameworks It is encouraging see that our generated data generally follows similar pat- terns as the original natural data. However, we would like to stress that our usage of state-split PCFGs is certainly not the only option to obtain a generative process that models natural language well. Other frameworks that could be explored in future work include oo-parametric Bayesian meth- ods (Teh, 2006; O' O'Donnell, 2015), data-oriented parsing (Bod et al., 2003; van Cranenburgh al., 2016). or n-gram based models (Pauls and Klein, 2012).
5 Language Modelling
We now move on to our experiments on language modelling the PCFG corpus that has been defined in the previous section. The underlying PCFG of the data generation process allows to place the LM's performance against an optimal bascline.
whitespace tokenizer, with minimum token frequency of 5 in the training cor- pus (clse it is mapped an <unk> token), which results in vocabulary of 23,921 tokens. We use the same tokenizer for all models, to ensure a fair comparison across models.
5.1 Metrics
5.3 Language Model Performance
We compute the perplexity scores on the evaluation corpus for both the language models and the PCFG, using the methods of $3.2. We skip over tokens
We report the perplexity scores for our language models and PCFG in Table 1. We provide a re- gression plot of the results for the GPT-2 model in Figure 1. The considerable difference in PCFG perplexity between masked and causal language modelling shows that the amount of possible con- tinuations is, unsurprisingly, far greater for causal language modelling. Despite being a task with greater uncertainty, however, the causal LMs ac-
"Because the EarleyX algorithum has not been optimised for coarse-to-fine parsing, we were unfortunately forced to com- pute the causal PCFG probabilities on a smaller subset. The results reported are on a sample of 1 1100 token probabilities. Concurrent to our work, more efficient prefix probability algo- rithms have been introduced by Nowak and Cotterell (2023) and Opedal et al. (2023); we leave the incorporation of these procedures open for future work.

---

Architecture
Size
(v))PPL
R
P
1.
Causel Masked
PCFG
63.9
Relatve Portbuty
Masked
BERT
9.5M
71.9
92.7
96.9
2:0
RoBERTa
9.5M
71.8
92.7
96.9
DeBERTa
10.7M
71.1
93.0
97.0
Causal
PCFG
183.1
10
FLOPS (non-embeddin)
GPT-2
12.7M
192.8
97.0
98.4
OPT
15.6M
194.2
96.7
98.3
Figure 5: Language modelling performance expressed as function of training size (a) and model size (b). Rel- ative perplexity is computed with respect to the PCFG's perplexity lower bound, averaged over 3 seeds.
DeBERTa
41.1
GPT-2
115.0
Table Perplexity scores for various masked and causal LM architectures. R2 and Spearman's 's p computed with respect to the PCFG log probabilities. Size de- notes the total number of model parameters. TB denotes performance of LMs trained on the original treebank data.
sizes. Each model is trained for the same amount of steps (15,000), but for smaller corpus sizes we set the maximum number of epochs to 50, to reduce overfitting. In order to compare causal and masked LM performance directly, we report the relative perplexity, which is defined as the LM perplexity divided by the PCFG perplexity lower bound.
quire distributions that are closer aligned the true PCFG distribution than the masked LMs. can be seen that a higher Spearman's p and R² yields a lower perplexity for all models, which demon- strates that models improve their performance by aligning their distribution closer to the true distri- bution. As the DeBERTa and GPT-2 architectures obtain the best performance, will use these ar- chitectures in subsequent experiments evaluate and interpret their performance and behaviour.
For the impact of model size we train DeBERTa models with an increasing amount of layers, rang- ing from to 20. All models are trained on the maximum corpus size of 7.5 million sentences for I epoch. We report model performance as a func- tion of FLOPS across intermediate checkpoints, similar to how this is done for evaluating scaling laws of massive LMs (Kaplan et al., 2020; Hoff- mann et al., 2022). FLOPS approximated using the fvcore library.
Treebank We also trained masked and causal LMs on the sample of 500k sentences from the orig- inal treebank that had been used for inducing the state-split PCFG. This experiment serves as another baseline for the naturalness of the PCFG-generated data: the treebank data contains all possible linguis- tic cues that a LM could rely on, and worsened performance on synthetic data indicates that some of these cues have been lost during grammar induc- tion. We can see that this is the case in Table 1: the performance of both LMs is considerably better than the LMS trained on PCFG-generated data.
We report the results Figure 5. Inter- estingly, the causal LMs converge slower to the final performance than masked LMs, but ultimately reach a better relative perplexity. The corpus of 500,000 sentences appears to be the inflection point: the masked LMs appear to reach platcau at this point, whereas the causal LMs continue to improve with more training data.
Results
For the model size experiment it can be seen that performance improves as model size increases. A considerable gap between the optimal perplexity and model perplexity remains, however. The mod- els we consider here are still relatively small, and it appears that larger scale models may be bene- ficial for the task, without risk of overfitting yet. Nonetheless, it remains interesting see that our models follow a similar pattern as seen for large- scale LMs (Hoffmann et al., 2022), with a similar near- hear-linear decrease in perplexity in log space.
5.4 Corpus & Model Size
To investigate the impact of the amount of training data and the impact of model size, we conduct an additional experiment where
investigate how model performance shifts as these two aspects are varied. For the impact of corpus size test models on increasing training sets: ranging from 10.000 to 7.5 million sentences. The model architectures is the same as outlined $5.2, across all corpus

---

Learning Dynamics
Moan Divergence
1.00
Dattea
0.06
Probe Accurnty:
0.95
Pudde) PPorole)
Pu Pass/i
0.06
4.10
0.80
Metric
p
adecte
0.15
0.80
Prera
0.54
0.30
0.75
Pu
0.59
0.7
actictonn
Gursion Brodd
ponssationn promort
|P - Prorol
-0.51
PBotchos sean (X1003)
Figure 7: Left: POS Probing performance across layers (x-axis) and split out for multiple checkpoints through training. Right: Spearman correlation between POS probing probabilities of the true class and the token probabilities of the PCRG, LM, and probability diver- gence.
Figure 6:
The LM probability divergence from the PCFG distribution, aggregated by general part-of- speech classes, for the DeBERTa model.
6 Model Interpretability
2018). We conduct a probing experiment on the representations of the DeBERTa model, classify- ing the part-of-speech (or pre-terminal) node above each token in the syntax tree. We split the train and test sections based on the underlying token of POS tag: this ensures that the probe finds a robust signal that does not simply map a token's 's identity to its most common POS tag. but that generalises based on other tokens with that POS tag (Hewitt and Liang, 2019).
We investigate the behaviour and learning dynam- ics of the DeBERTa model from $5.3. Our setup provides an interesting test bed for interpretabil- ity purposes: since we have transparent access to the underlying structure of our data, we can make stronger assumptions about the cues the model must be relying on than would be possible in natu- ral language. We conduct two experiments, one on learning dynamics and another on Part-of-Speech probing.
Results We report the results in Figure 7. Per- formance reaches plateau at 40% of training, af- which it settles an accuracy of around 90%. We can see that POS information encoded con- sistently from around layer 3 in the model, and that this performance does not deteriorate in upper layers, unlike pretrained LMs like BERT (Tenney et al., 2019), where upper layers appear to encode sentence-level semantic features. This indicates that the PCFG nature of our data relies more syntactic cues, which likely the effect of a loss of naturalness, as explored $4.2.
6.1 Learning Dynamics
The LMs of $5.3 obtain a distribution that aligns closely to the true PCFG distribution. We investi- gate how this distribution forms during training by aggregating token probabilities to general POS class, mapping the 43 POS tags in the original data one of base classes. We report the divergence of the LM probabilities from the PCFG, expressed as the mean difference in log probabilities.
Results We report the results in Figure 6. We can see that different POS classes are learned at different stages during training. Punctuation and pronouns obtain a low divergence at the start of training already, which is maintained across check- points. A possible explanation for this might be that punctuation is less dependent of context, and more the position in the sentence: quotation marks occur mostly at the start and end of the sentence, and most sentences end with period symbol.
Additionally, we also investigate the correlation between the probabilities the POS probe and the data distribution. We would expect that if a model relies on the POS signal extracted by our probe for its token predictions, that in cases where these pre- dictions diverge from the PCFG distribution, the POS performance lower well. We compute the Spearman correlation for both the LM and PCFG distributions, as well as the divergence. Both LM and PCFG distributions obtain a positive correla- tion: this shows that the POS probe performs better on tokens with a higher probability. The divergence
6.2 POS Probing
Probing is an interpretability technique in which auxiliary classifiers are trained on top of the rep- resentations of LM, to find out whether abstract features are encoded (Adi et al., 2017; Hupkes et al.,
A NNote that our procedure right now does not take structural ambiguity into account, we leave the incorporation of the full POS distribution open for future work.

---

metric, however, yields a negative correlation: this shows that when the LM's probabilities diverge more from the PCFG distribution, the POS probing performance drops as well. We can conclude from this that representing POS information is a prereq- uisite for competent language model performance.
how evaluate the faithfulness of attribution and probing techniques. Prior research has attempted to define controlled environments that provide a certain degree certainty about how the model is solving a task. Examples include the training of models on small-scale artificial data to perfection (Hao, 2020; Jumelet and Zuidema, 2023), or by ex- plicitly providing shortcuts that the model must rely (Bastings al., 2022; Ebert et al., 2023). Scal- ing this up more realistic, natural data, however, remains an open problem. Our setup, with direct access to a natural-like distribution, now provides much stronger guarantees about what patterns in the data the model is relying on, and allows for gradual increase of the complexity of the data distribution.
7 Conclusions and Future Work
In this paper, we train language models on artificial data from an explicit, known source, but in setup where that data approximates natural language data in richness and complexity. This approach allowed us to compute an exact lower bound on perplexity, and evaluate how well different LM architectures can approach this optimum. We observed strik- ing differences in the learning dynamics between masked and causal language models, where the latter converge much more slowly, but ultimately approximate the optimum more closely. And observe large differences between word types in how well the trained LMs approximate the true probabilities from the underlying source grammar.
8 Limitations
Our work involves language-like artificial data. An fundamental limitation is the degree which stochastic grammars can indeed generate data that is language-like important respects. Although our analyses in $4.2 show that our data is much more language-like than the baseline we consid- ered, it is also clear that data still differs from nat- ural language distributions. More work is needed to assess which differences between artificial and natural language datasets affect the generalizability of findings on the former to the latter.
Our proposed methodology for evaluating and interpreting LMs thus involves inducing a rich prob- abilistic grammar, and using that grammar to gen- erate language-like training sets for neural lan- guage models. It did become clear, however, that our grammar induction has led to a sacrifice of var- ious linguistic cues, which is demonstrated by a lower selectional preference ($4.2) and higher per- plexity with respect to the original treebank ($5.3). We leave the exploration of which linguistic cues exactly are lost open for future work, for example by evaluating the LMs on the BLiMP benchmark for linguistic phenomena (Warstadt et al., 2020).
A more mundane limitation of our approach to generating data is that we rely on large auto- matically parsed corpus to induce our probabilistic grammar. It is clear that the automatic parses are far from error-free; it could be that the quality of our generated data would improve if the quality of the automatic parses is improved, by better automatic methods or manual correction.
We believe our methodology holds promises be- yond the specific modelling choices we have made in this paper. In the future we hope to see stud- ies using different grammar induction procedures and focusing on other interpretability techniques, including the ambitious efforts, under the label "mechanistic interpretability' to reverse engineer the behaviour of LLMs. Furthermore, we believe our work will be of interest to work on NLG eval- uation and uncertainty estimation, where having access to the true data distribution is important (Pi- mentel et al., 2023; Zhang et al., 2023; Giulianelli et al., 2023).
Our closed form equations for computing the lower bound on perplexity are exact, but when com- puting the lower bound for specific large dataset make use of some standard heuristics from the coarse-to-fine parsing literature where some ex- tremely low probability productions in long sen- tences might be pruned from the chart. We don't 't expect this to have noticable effects, but have not rigorously evaluated this assumption.
The implementation we use for computing the lower bound in the causal language modelling case, EarleyX, is not optimized for extremely large gram- mars, and does not offer coarse-to fine pruning. is therefore painfully slow, limiting the size of
Perhaps the most important relevance of our work, and such alternative approaches, in relation to a key open problem in interpretability research:

---

grammars which we could use. To scale up our ap- proach larger datasets, grammars and language models, a new, optimized implementation would be required. The recent approaches of Nowak and Cot- terell (2023) and Opedal et al. (2023) may provide a solution for this.
Michael Collins. 2003. Head-driven statistical models for natural language parsing. Comput. Linguistics. 29(4):589-637.
Alexis Conneau, German Kruszewski, Guillaume Lam- ple, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single S&.#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the As- sociation Computational Linguistics (Volume 1: Long Papers), pages 2126-21366 Melbourne, Aus- tralia. Association for Computational Linguistics.
In our interpretability analyses, where make use of knowledge of the exact rules of the source grammar, we use the gold standard derivation tree for each sentence, and - for computational reasons do not recompute the entire parse forest. effect, we thus ignore structural ambiguity in those analy- ses; we don't It expect this to have noticable effects, but have not rigorously evaluated this assumption.
Ryan Cotterell, Sabrina J. Mielke, Jason Eisner, and Brian Roark. 2018. Are all languages equally hard to language-model? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 536-541, New Orleans, Louisiana. Association for Computational Linguistics.
Acknowledgements
The authors are grateful for the useful feedback provided by Charlotte Pouw, Marianne de Heer Kloots, and the anonymous reviewers.
Verna Dankers, Elia Bruni, and Dieuwke Hupkes. 2022. The paradox the compositionality of natural lan- guage: A neural machine translation case study. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4154 4175. Association for Computa- tional Linguistics.
References
Yossi Adi, Einat Kermany. Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open- Review.net.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL-HLT 2019, Minneapolis, MN. USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
James K. Baker. 1979. Trainable grammars for speech recognition. Journal of the Acoustical Sociery of America, 65.
Jasmijn Bastings, Sebastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. 2022. "will you find these shortcuts?" a protocol for evaluating the faithfulness of input salience methods for text classification. In Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing, pages 976 776-991, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Sebastian Ebert, Alice Shoshana Jakobovits, and Katja Filippova. 2023. Understanding text classification data and models using aggregated input salience. R2HCAI: The AAAI 2023 Workshop on Representa- lion Learning for Responsible
Al.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy. 2020. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101 00007.
R. Bod, Scha. and K. Sima'an an. 2003. Data-oriented Parsing. Studies in computational linguistics. CSLI Publications.
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, Jennifer C Lai, and Robert L Mercer. 1992. An estimate of an upper bound for the entropy of english. Computational Linguistics, 18(1):31-4)
Mario Giulianelli, Joris Baan, Wilker Aziz, Raquel Fernández, and Barbara Plank. 2023. What comes next? evaluating uncertainty in neural text genera- tors against human production variability. CoRR, abs/2305.1 1777.
Eugene Charniak and Mark Johnson. 2005. Coarse- to-fine n-best parsing and MaxEnt discriminative reranking. In Proceedings of the 43rd Annual Meet- ing of the Association for Computational Linguistics (ACL 005), pages 173-180, Ann Arbor, Michigan. As- sociation for Computational Linguistics.
John Hale. 2001. A probabilistic Earley parser as a psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguistics.

---

Yiding Hao. 2020. Evaluating attribution methods us- ing white-bos box LSTMs. In Proceedings of the Third BlackboxNI.P P Workshop on Analyzing and Interpret- ing Neural Networks for NLP, pages 300-313, On- line. Association for Computational Linguistics.
In Findings of the Association for Computational Lin- guisties: ACL-IJCNLP 2023. Association for Com- putational Linguistics.
Dan Jurafkky and James H. Martin. 2009. Speech and language processing an introduction to natural language processing, computational linguistics, and speech recognition. Pearson Prentice Hall, Upper Saddle River, N.J.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- view.net.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. abs/2001.085 0.8361...1........
Andrew T. Hendrickson and Amy Perfors. 2019, Cross- situational learning zipfian environment. Cogni- tion, 189:11-22
CoRR.
Dan Klein and Christopher D. Manning. 2003, Accu- rate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Compu- tational Linguistics, pages 423-430, Sapporo, Japan. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and in- terpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 2733-2743, Association for Computational Linguistics.
Chigusa Kurumada, Stephan C. Meylan, and Michael C. Frank. 2013. Zipfian frequency distributions fa- cilitate word segmentation in context. Cognition, 127(3):439-453
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. An empirical analy- sis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 15:30016-30034
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/ 1ans/199..116 16692.
Minh-Thang Luong, Michael C. Frank, and Mark John- son. 2013. Parsing entire discourses as very long strings: Capturing topic continuity in grounded lan- guage learning. Trans. Assoc. Comput. Linguistics, 1:315-326.
Mark Hopkins. 2022. Towards more natural artificial languages. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 85 85-94, Abu Dhabi, United Arab Emirates (Hy- brid). Association for Computational Linguisties.
Benoît B. Mandelbrot. 1953, An informational theory of the statistical structure of languages. In Com- munication theory: papers read at Symposium on "Applications of Communication Theory" held at the Institution of Electrical Engineers, London, Septem- ber 22nd-26mn 26th 1952, pages 486-502.
Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more gram- mar with small-scale child-directed language. In Pro- ceedings of the 25th Conference Computational Natural Language Learning. pages 624-646, Online. Association for Computational Linguistics.
Christopher D. Manning and Hinrich Schütze. 2001. Foundations of statistical natural language process- ing. MIT Press.
Dieuwke Hupkes, Sara Veldhoen, and Willem H. Zuidema. 2018. Visualisation and 'diagnostic classi- fiers' reveal how recurrent and recursive neural net- works process hierarchical structure. J. Artif. Intell. Res., 61:907-926.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc- Closky. 2014. The stanford corenlp natural language processing toolkit. Proceedings of the 52nd An- nual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, System Demonstrations, pages 55-60. The Association for Computer Linguistics.
F. Jelinek, R. L. Mereer, L. R. Bahl, and J. K. Baker. 1977. Perplexity measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62:S63. Supplement 1.
Frederick Jelinek and John D. Lafferty. 1991. Computa- tion of the probability of initial substring generation by stochastic context-free free grammars. Comput. Lin- guistics, 113):315-3233
Sabrina J. Mielke, Ryan Cotterell, Kyle Gorman, Brian Roark, and Jason Eisner. 2019. What kind of lan- guage is hard to language-model? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 4975 4989, Florence, Italy. Association for Computational Linguistics.
Jaap Jumelet and Willem Zuidema. 2023. Feature inter- actions reveal linguistic structure in language models.

---

Franz Nowak and Ryan Cotterell. 2023. A fast algo- rithm for computing prefix probabilities. In Proceed- ings of the st Annual Meering of the Association for Computational Linguisties (Volume 2: Short Pa- pers), pages 57-69, Toronto, Canada. Association for Computational Linguistics.
the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3532-3542, Minneapolis, Min- nesota. Association for Computational Linguistics.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka- trin Kirchhoff. 2020. Masked language model scor- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguisties.
T.J. O' O'Donnell. 2015, Productivity and Reuse in Lan- guage: A Theory of Linguistic Computation and Stor- age. The MIT Press. MIT Press.
Andreas Opedal, Ran Zmigrod, Tim Vieira, Ryan Cot- terell, and Jason Eisner. 2023. Efficient semiring- weighted Earley parsing. In Proceedings of the s1 Annual Meeting the Association for Computational Linguistics (Volume 1: Long Pupers), pages 3687- 3713, Toronto, Canada. Association for Computa- tional Linguistics.
Claude Elwood Shannon. 1951. Prediction and entropy of printed english. Bell System Technical Journal, 30:50-64.
Andreas Stolcke, 1995. An efficient probabilistic context-free parsing algorithm that computes prefix probabilities. Comput. Linguisticx, 2152)1165-201,
Isabel Papadimitriou and Dan Jurafsky. 2023. Pre- train on just structure: Understanding linguistic inductive biases using transfer learning. CoRR, abs/2304. 13060.
Yee Whye Teh. 2006. A hierarchical Bayesian lan- guage model based on Pitman- processes. In Proceedings of the 21st International Conference Computational Linguistics and 44th Annual Meet- ing of the Association for Computational Linguistics, pages
Adam Pauls and Dan Klein. 2012. Large-scale syntactic language modeling with treelets. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 959-968, Jeju Island, Korea. Association for Computational Linguistics.
Sydney, Australia. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellic Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Associa- tion for Computational Linguisties, ACL 2019, Flo- rence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 4593-4601. Association for Computa- tional Linguistics.
Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein, 2006. Learning accurate, compact, and inter- pretable tree annotation. In Proceedings the s International Conference on Computational Linguis- tics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433-440, Sydney, Australia. Association for Computational Linguistics.
Andreas van Cranenburgh. Remko J. H. Scha, and Rens Bod. 2016. Data-oriented parsing with discontinuous constituents and function tags. Journal of Language Modeling, 4:57-111.
Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Human Language Tech- nology Conference of the North American Chapter of the Association of Computational Linguistics, Pro- ceedings, April 22-27, 2007, Rochester, New York, USA, pages 411. The Association for Computa- tional Linguistics.
Valentin Vogelmann. 2020. Statistical Methodology for Quantitative Linguistics: A Case Study of Learnabil- ity and Zipf's Law. Master Thesis. University of Amsterdam.
Steven T. Piantadosi. 2014. Zipf"s word frequency law in natural language: A critical review and future di- rections. Psychonomic Bulletin& & Review, 21:1112- 12- 1130.
Dingquan Wang and Jason Eisner. 2016. The Galac- tic Dependencies Treebanks: Getting More Data by Synthesizing New Languages. Transactions of the Association for Computational Linguistics, 4:491 505.
Tiago Pimentel, Clara Mcister, and Ryan Cotterell. 2023, On the usefulness of embeddings. clusters and strings for text generation evaluation. In The Eleventh In- ternational Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. Open- Review.net.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo- hananey. Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguis- tic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377- 392.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Jennifer C. White and Ryan Cotterell. 2021. Examining the inductive bias of neural language models with ar- tificial languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing, ACI/JCNLP 2021,
Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019, Studying the inductive biases of RNNS with synthetic variations of natural languages. In Proceedings of

---

(Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 454 463. Association for Computational Linguistics.
which can be rewritten
Σ Po(wi|NN war ( Po(N2) ux())
(6)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: Sfate-of tatte-of-theaaad art natural language processing. In Pro- ceedings of the 2020 Conference Empirical Meth- ods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 38-45. Association for Computa- tional Linguistics.
The first term is equal to P= due to the context-free nature of PCFGs the probability of to- ken w;; solely depends on its parent non-terminal token, and not on neighbouring tokens in the sen- tence. This term is therefore equal to the inside probability By i):
Pa(wi|NV w.y) Po(wulN2)
(7)
= i)
(8)
Shiyue Zhang. Shijie Wu, Ozan Irsoy, Steven Lu, Mohit Bansal, Mark Dredze, and David Rosenberg. 2023, MixCE: Training autoregressive language models by mixing forward and reverse cross-entropies. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), pages 9027-9050, Toronto, Canada. Association for Computational Linguistics.
The second term can be expressed in terms of outside probabilities. In the special case of a sub- string of length (wa), the words outside of the substring are equivalent to the masked token con- text w.y
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01068.
- Po wv))
(9)
We can expand this as
ay(t, = Pa(N Pa(u))
(10)
The first term, Pa (N r). is equal to the second term in Eq. 6. The second term, (a(w)) can be marginalised out over all non-terminals N i.e. over all potential non-terminal parents of token uy, By doing this we are able to express Pa (wy)) in terms of (normalised) outside probabilities:
Haoyu Zhao, Abhishek Panigrahi. Rong Ge, and San- jeev Arora. 2023. Do transformers parse while pre- dicting masked word? CoRR, abs/2303.08
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. ICCV, pages 19-27. IEEE Computer Society.
Pa(wa)) Σ Polw,,,
(11)
Eas(i
(12)
G.K. Zipf. 1936. The Psycho-Biology of Language: An Introduction to Dynamic Philology. Cognitive psychology. Routledge.
Po(Ng(uo,) = i)(i,
(13)
Po(w)p)
Masked Language Modelling in PCFGs
az(i,
(14)
In this section we demonstrate how the inside and outside probabilities can be used compute masked token probabilities.
Where liq. 13 is simple reordering of Eqq 10.
Plugging this all in Eq. 6 we obtain the masked token probability distribution expressed in terms of ousidee probabilities:
Firstly, the masked token probability for uy can be expressed as the marginalisation over all non- terminal symbols N that are the parent of w;:
ay(i,))
Pa(wi/wy)) Σ Po(we, No. Polw,,Ngluy Nz)lun)))
(5)
Po(w:|wy)) = E3
(15)
Σx (t,

---

B Number of Non-terminal Splits
We plot the number of times each non-terminal type has been split in our final grammar in Figure 8. The more often rule is split, the more fine-grained its distribution can become. It can be seen that in general open-class, pre-terminals lead to more state splits, although common non-terminals such as NP, VP, and have been split often as well.
Sample of Corpus Sentences
A sample of sentences taken from the evaluation corpus. We provide an example of the parse tree of the first sentence in Figure 9.
Her mouth wasn't very close. It met Calvin's 's lips and closed the wind.
The human ceased me to be average.
The Captain arrived and went apt of some not to think.
Liz spotted us through Bonjour and Monday, trav- eling around each of Ryland against the doorway.
After my injured procedure, you shrugged, back- stabbing into the wall.
Josh kept departing thirty hundred alibis no king
As separates or eventually killed the one top of giant length, Travis left.
A general car trespassed at tunnels on the im- age of the misplaced formation.
think that does find a perfect visit activity.

---

Non-termina
Pro-terminal
AD/P
APOSTROPHE
CC
CD
COLON
COMMA
CONjP
SCONjP
DoT
EX
FRAG
@rRAG
IN
@IN
LST
@LST
MD
@NcC
NNPS
NNS
SNP
POT
POS
aPP
PP
PAN
@PRN
PAT
@OP
gp
RR
GARC
RRC
as
SBARD
@SBARO
SWW
@SM
@UCP
UCP
UH
VBD
VIN
VaP
VBZ
gVP
WHADIP
@WHADVP
WHADYP
GIHHPP
WHIP
WRA
@X
10
# of split NTs
Figure 8: Number of splits per non-terminal. The '8' rules the result of the X-Bar binarization procedure.

---

So
@S3
DOT
NP
VP2
PRPS3
NN3
VP
ADJPs
Her
mouth
VBD2
RB17
RB21
JJ
was
n't
very
close
Figure 9: Example tree from the evaluation corpus. The @ non-terminals are the result of the X-Bar binarization procedure of Petrov et al. (2006).


Reviews:
[{"review_id": "3vsp28Xzzr", "reviewer": "Reviewer_ybVx", "summary": "", "questions": "- L240: I love the notation of $\\mathrm{\\psi LL}$ for **ps**eudo-log-likelihood.\n - L292: You used a machine with 32GB RAM. In modern computer architectures (e.g. an EC2 machine on AWS), one can easily obtain a compute instance with RAM >256GB. I wonder what the PCFG would look like if induced over a much larger corpus (e.g. 10M sentences instead of 500k).\n - L296: _X-bar style_ grammar ensuring each rule has at most 2 children: would the term _Chomsky normal form_ be a better description of the constraints you put on the grammar? _X-bar_ requires the structure of NP-bar, NP, and N, and it contains nuisances such as specifiers (e.g. SpecNP).\n - Table 1: For the masked LMs, is the PPL column actually $\\mathrm{\\psi}LL$?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the enthusiastic review. We will incorporate the suggested references in the paper. The work of Shin et al. is something we found out about after submitting, but is definitely highly relevant to our work: this goes to show that our approach is not just relevant to interpretability of language models, but other domains such as semantic parsing as well.\n\nWe completely agree, of course, that the induced PCFGs are far from perfect approximations of natural language distributions (we're not trying to reverse the neural language model revolution!). Our key point is that there are many interesting distributions \"in between\" the idealized, simple distributions used in prior formal work, and the natural, complex distributions used in current empirical work. Our paper involves a considerable amount of nontrivial, technical work to provide one such intermediate distribution.\n\nQ1: Inducing the grammar on a larger language sample would probably get rid of some outlier rules, but we do not think that the current treebank size is the main limiting factor for the quality of the generated corpus. Instead of increasing the sample size, improving the quality of the original treebank is probably a more useful endeavor. This is something we definitely aim to explore in future work (and we would love to hear suggestions for high quality treebanks; it turned out to be challenging to find large-scale treebanks that provide a broad reflection of natural language).\n\nQ2: The X-Bar transformation is a procedure stemming from the implementation of Petrov et al. (2006), and does include NP-bar-like non-terminals. Since our final grammar also includes non-terminal rules of the form A -> B, it is not fully in Chomsky Normal Form.\n\nQ3: The PPL column for MLM is indeed ψLL, we’ll update the table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A nice revival of the Berkeley parser and the construction of PCFGs using the state-split method.\n - A deep discussion on the use of a probabilistic grammar to evaluate neural sequence models.", "reasons_to_reject": "- A fundamental problem is how much an induced PCFG actually resembles the distribution of natural language sequences. The authors conceded that PCFGs are not the only way to obtain a generative model over sequences. A decoder-based Transformer itself is also a generative model over sequences. One could in theory evaluate PCFGs under the distribution from a Transformer. The choice of PCFGs as the base distribution is not convincing in my opinion.", "questions_to_authors": "- L240: I love the notation of $\\mathrm{\\psi LL}$ for **ps**eudo-log-likelihood.\n - L292: You used a machine with 32GB RAM. In modern computer architectures (e.g. an EC2 machine on AWS), one can easily obtain a compute instance with RAM >256GB. I wonder what the PCFG would look like if induced over a much larger corpus (e.g. 10M sentences instead of 500k).\n - L296: _X-bar style_ grammar ensuring each rule has at most 2 children: would the term _Chomsky normal form_ be a better description of the constraints you put on the grammar? _X-bar_ requires the structure of NP-bar, NP, and N, and it contains nuisances such as specifiers (e.g. SpecNP).\n - Table 1: For the masked LMs, is the PPL column actually $\\mathrm{\\psi}LL$?", "paper_topic_and_main_contributions": "This paper presents a setup to evaluate and interpret neural language models using artificial data drawn from a language model learned from a PCFG. The PCFG is constructed using the state-split method over a large corpus. This method allows the definition for a closed-form expression for obtainable perplexity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- L185: The correct citation of _perplexity_ is [Jelinek+77] listed below.\n - L215: This method of getting a probability distribution for tokens under a PCFG has been extensively used in [Shin+21].\n\n### References\n - [Jelinek+77]: F Jelinek, R L Mercer, L R Bahl, J K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. J Acoust Soc Am 62, S63.\n - [Shin+21]: Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained Language Models Yield Few-Shot Semantic Parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7699–7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics."}}, {"review_id": "JQDXNiXMTN", "reviewer": "Reviewer_45J7", "summary": "", "questions": "1. Would you have idea of how the relative ordering of models would change when training them on 7.5M natural sentences from The Pile / BookCorpus?\n2. Could you elaborate on how you would use your PCFG to execute an experiment like the one you mentioned in the introduction?\n3. Could you give some examples of natural linguistic phenomena that your PCFG couldn't possibly capture?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review; it is highly encouraging to hear they find our approach ‘a refreshing take on language modelling’!\n\nA full analysis of how this paper can in turn be a tool in interpretability research would easily fill up one or several new papers (future work!). The key point here is that research on the faithfulness of interpretability methods, implicitly or explicitly, relies on the assumption that their usefulness/faithfulness does not depend on the nature of the source distribution. To the extent that that assumption holds, our work can be used to more thoroughly assess existing methods. But it can also be used to investigate the validity of that assumption itself.\n\nQ1: That is a great question that we did explore during experimentation, but ultimately left out for space reasons. One finding is that a masked LM, using the DeBERTa architecture that we also used in experiments reported in the paper, resulted in a considerably lower perplexity when trained on the original treebank, than when trained on the PCFG data. This shows that there are various linguistic signals in the original data that are not tackled by the PCFG, but that are exploited by the LM. \n\nThis also relates strongly to Question 3. We suspect that most of the phenomena that the PCFG still fails to capture are related to semantic relations between phrases. While the state split procedure improves semantic relatedness with respect to the raw PCFG (as shown in Fig. 3b), the grammar is still not restrictive enough to only permit the generation of sentences that are semantically congruent across the entire sentence. In other words: our state-split PCFG overgenerates with respect to the original corpus, which increases the uncertainty over the generated corpus and results in a higher LM perplexity. \n\nWhile this issue could be improved with a better Treebank (see also our response to reviewer ybVx), there ultimately are limits to the expressiveness of a state split PCFG due to the finite number of non-terminal states. Simple example: to encode that a ‘dog’ is more likely to ‘bark’, and a ‘cat’ more likely to ‘meow’, we’d need separate S -> NP VP rules to carry over this information. The more fine-grained we want this semantic information to be encoded, the larger our PCFG would need to become.\n\nThanks for these questions. Your curiosity about these aspects of the paper is an argument for us to incorporate a broader discussion in the paper. In the revised version we will incorporate the original Treebank perplexities as well, since it is a useful way of placing the performance of our PCFG LMs in context.\n\nQ2: The PCFG provides explicit access to the underlying hierarchical structure of the data that an LM needs to model. This allows us to investigate and control the distribution of particular linguistic constructions, since we have access to the rules that underlie these constructions. An investigation of such nature would be similar to the work of White and Cotterell (2021), in which they modify various typological features of artificial languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents an innovative approach to examining the capabilities of language models, namely by using synthetically generated data at a much larger scale than previous work. In addition to contributing the resulting PCFG and dataset as a resource, they contribute practical techniques required to make that work, and that approach as a whole could be adopted by future work.\n- The paper presents a theoretical contribution to computing perplexity using a PCFG, assuming a masked language modelling setup. UPDATE AFTER REBUTTAL: as fellow reviewer ZP1G pointed out, there has been related work that made a similar contribution earlier this year.\n- The paper presents a thorough analysis of how models behave on this synthetic corpus, and presents results that could not previously be computed because the true underlying distribution of natural language is unknown. These results include indications of how close to the lower bound on perplexity the LMs can get and to what extent the probabilities assigned to sentences (or words of certain POS tags) correlate with the probabilities of the true distribution.\n\nOverall, I find this a refreshing take on language modelling: instead of approaching it as a black-box issue, it became fully transparent, without it being a toy setup.", "reasons_to_reject": "- Even though the authors comment on the fact that PCFGs do not suffice for capturing *natural* linguistic variation in the limitations section, they do not actively address this in the main paper when analysing models (they only address this based on the statistics regarding naturalness). I would expect: 1) some type of comparative analysis illustrating how perplexities on 7.5M sentences from the natural corpus compare to the perplexities reported for the artificial corpus (to understand whether the better models in this setup are also the better models in a natural scenario). After all, you don't need the lower bound to see that some models perform better than others. 2) a discussion of what natural linguistic phenomena cannot be captured using this type of state-split PCFG. (after all, natural language is not context free...)\n- I thoroughly enjoyed the work and recommend it for publication, but the reader is left wondering how this can be used in the future. The authors argue that this can be a tool for interpretability research, but in the paper itself that analysis remains somewhat shallow (the training dynamics of POS tags are much more simplistic than the example provided in the introduction regarding the prepositional dative). Given that the PCFG cannot really capture the intricacies of natural language, wouldn't we just be examining LMs' capabilities to learn extraordinarily large PCFGs, rather than *natural* language? Moreover, wouldn't the PCFG learnt itself be extremely hard to interpret?", "questions_to_authors": "1. Would you have idea of how the relative ordering of models would change when training them on 7.5M natural sentences from The Pile / BookCorpus?\n2. Could you elaborate on how you would use your PCFG to execute an experiment like the one you mentioned in the introduction?\n3. Could you give some examples of natural linguistic phenomena that your PCFG couldn't possibly capture?", "paper_topic_and_main_contributions": "When training language models, we evaluate them according to the perplexity measure. But because the true distribution underlying natural language is unknown, we do not have access to the lower bound on perplexity.\n\nThis paper, instead, trains LMs on synthetic data generated using a state-split PCFG obtained using automatically parsed data from The Pile. This allows computing an exact lower bound of the perplexity and provides better interpretability since the grammar indicates all that a model could/should learn, whereas in the case of natural language, we must simply assume the data captures all grammatical rules.\n\nThe contributions of this work are as follows:\n1. a theoretical contribution via the closed-form expression to compute a lower bound on perplexity for masked language modelling using the PCFG (based on pseudo-perplexity and the inside-outside algorithm).\n2. a resource contribution via the synthetically generated corpus. The corpus contains 7.5M sentences generated using a state-split PCFG, learnt based on 500k sentences from the BookCorpus (part of the Pile). The authors examine the naturalness of their corpus by inspecting whether Zipf’s law applies, inspecting sentence lengths and inspecting frequencies of n-grams. In the process, they show that the state-split PCFG indeed leads to more natural corpora compared to a raw PCFG.\n3. an evaluation of how models behave when trained on this corpus. 2 autoregressive models and 3 bidirectional models are trained on the data, their perplexity is compared to the lower bound, and correlations are reported comparing their probabilities to the PCFG’s probabilities. The key findings are: 1) causal language modelling has a higher perplexity than MLM, 2) lower perplexities correspond to higher correlations to the true distribution. Furthermore, the influence of corpus and model size are discussed.\n4. an analysis of how POS tags are learnt over the course of training, and over the course of different layers, where the LMs’ performance is contrasted with the true distribution again.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 11: \"us define\" --> \"us to define\"\n- line 78: Misses an article perhaps? the / an exact lower bound\n- line 86: \"both masked language modelling\" and... is something missing here? The usage of \"both\" suggests 2 things.\n- line 208: Looks like the 2nd w has a typo in the subscript? <i should both be low/small?\n- line 394: While the reader will understand what 3b means, the figure itself doesn't have a & b. Perhaps use the subcaption packages to actually have a & b in the figure itself, such that you can properly refer to them separately. This also applies to remaining figures and corresponding subfigures in the paper.\n- It appears that you use the wrong convention for indicating numbers >999, you should use a comma instead of a period in English. (This applies to numbers >999 throughout the paper,  which is why I’m not including a line number here)\n- line 588: & → and"}}, {"review_id": "hvqHIpYkSp", "reviewer": "Reviewer_ZP1G", "summary": "", "questions": "On line 542:\n> We investigate how this distribution forms during training by aggregating token probabilities to a general POS class, mapping the 43 tags in the original data to one of 7 base classes.\n\nWhat does this mean? It sounds like it means that each token is assigned to a fixed POS for each token in the vocabulary and you measure how well the learning LM matches the ground truth LM's distribution on that token type -- but POS tags are not fixed for the ground truth grammar? Why does the fact that the ground truth LM here is a PCFG matter? I think you could run this comparison on any ground truth generative model (like another causal LM)?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review, and the interesting criticism on our work. We hope our reply, and the comments from the other reviewers, might lead to a more positive assessment of the potential relevance of our work!\n\n> Overall, I think I'd say that I don't know what a practitioner is supposed to have learned from this paper.\n\nThe question here is who those practitioners are, in practice. For practitioners developing extremely large-scale LMs, our paper may not be of direct interest. Our target audience would rather be practitioners coming from an interpretability perspective and researchers with an interest in the linguistic properties of LMs. In interpretability research, the lack of faithfulness guarantees is a pressing issue, and a wide range of papers have resorted to using artificial language data to gain a more fundamental insight into LM behavior, as outlined in Section 2 of the paper.  We will try to sharpen these points further in the introduction, to make even clearer were we see the relevance of this work; we very much agree that the relevance does not come from PCFGs being particular good models of language, or from performance on PCFG-generated data being a good “sanity check”. Rather, the relevance of this work is, as we also wrote in response to Reviewer ybVx, in providing a distribution of intermediate complexity: not as language-like as those modeled by modern NLMs, but also not as idealized as in much prior work, and yet completely transparent.\n\n> Taking a step back, a PCFG and a (causal autoregressive) LM are both generative models of English, but PCFGs are not inherently more interesting to study. In fact, modern causal LMs are far probably more interesting because they are significantly better approximations of English.\n\nWe don't disagree! PCFGs are much worse approximations of NL; but the one aspect that still makes them interesting is their inherent interpretability and links to linguistic theory. While parts of our experimental setup could be done using an auxiliary language model as a source model (e.g. GPT-2), this would not resolve the inherent issue of having full control and transparency over the source data. \n\nWe were also surprised we couldn't find any prior work that had computed the masked LM probability for a PCFG, and are grateful for the referenced preprint which is indeed very relevant (for those authors, the closed form expression is the crucial step in proving theorem 1; in our work, it is crucial to adapt the Petrov et al. parser to compute the quantities we can compare with Transformer-based Masked LMs).\n\nQ1: Our data is generated following the PCFG rules. This means that a leaf node is generated from a rule of the form A -> w, and this final non-terminal A is what we’d call the POS tag of the token. If we were to use an auxiliary LM for generating the corpus, we would not have this knowledge, i.e. we wouldn’t know whether the auxiliary LM uses abstract POS classes for generating a token. This explicit linguistic structure is one of the reasons we want to use an interpretable PCFG instead of another black-box LM. We pose this is important, because numerous works on probing (e.g. Tenney et al., 2019a & 2019b) rest on the assumption that such linguistic abstractions are present in natural language, and that therefore it’s worth investigating whether LMs have picked up on these notions. Our setup provides explicit guarantees about the linguistic structure that a LM may pick up on.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper does have one interesting result: I do not know why the correlation of the trained LM probabilities to the PCFG probabilities is so much higher for the causal objective. I think there's some chance this is an artifact of the obvious fact that causal probabilities are lower in general but I'm not sure. I wonder if KL divergence would have been a more informative metric here.", "reasons_to_reject": "Unfortunately, I don't think the investigation in this paper is well motivated. Studying an LM architecture's ability to learn from artificial languages acts as a kind of sanity check -- can this model architecture even learn to approximate a particular class of models -- but that is not a contribution of this work? The contribution of this work is comparing learned LM perplexities to their lower bounds on a better -- but still not good -- PCFG approximation of English. This answers the question: how well can an LM architecture approximate a poor approximation of English? However, I do not think it says anything useful about what LM architecture does on real English, nor does it necessarily provide any more sanity checks than what already exist in the literature. \n\nTaking a step back, a PCFG and a (causal autoregressive) LM are both generative models of English, but PCFGs are not inherently more interesting to study. In fact, modern causal LMs are far probably more interesting because they are _significantly_ better approximations of English. Note that the authors never show the perplexity other their PCFG on a dataset and compare it to a causal LM. I assume, based on the amount of data it was trained on and the samples in Appendix C, that it is not nearly as good as modern LMs. The authors show that it more closely approximates the tree bank than worse PCFGs, but not that it comes anywhere close to e.g. GPT2 as a model of English. (Side note: I also think the wording on line 410-11 \"models natural language well\" is not appropriate, since the authors have not shown that their PCFG models natural language \"well.\")\n\nI think a PCFG is potentially more interesting for two reasons: \n1) both the causal and masked loss functions can be computed efficiently -- or at least approximated efficiently. Computing the masked loss function p(w _i | w_{\\I}) for a causal autoregressive LM is possible and not even that inefficient, though it would quite expensive to do for a whole corpus. \n2) A PCFG has certain properties (like POS tags) that seem worth \"interpreting.\" \n\nSacrificing LM quality for 1 seems pointless -- why compare your LM architecture's perplexity to a theoretical lower bound on a language that is not that much like English just because you can do so efficiently? \n\nI think the authors are more interested in 2), but again, what lessons should I take from this? For example, the authors state (line 534)\n\n> we can make stronger assumptions about the cues the model must be relying on than would be possible in natural language.\n\nIs that true? \"the model must be relying on\" is a statement about how the model works, which is independent of the distribution of the data. I think there's some kind of identifiability argument to be made that if you manage to achieve the optimal perplexity with model A on a synthetic dataset generated from a model B, then you must be able to recover the underlying properties of B from A, but I don't think that's true for any particular probing strategy. There could still be \"a failure of the interpretability method\" to use the authors words. And in any case, the model architectures here fail to achieve the optimal perplexity, so who knows what they \"must be relying on\" to make their decisions. \n\nThe authors claim that their algorithm for computing P(w_i | w_{\\I}) is a contribution. I think it is a fairly straightforward use of the probabilities from the inside-outside algorithm and can't be claimed as a contribution. For example, https://arxiv.org/abs/2303.08117 state it without much fanfare in equation (12). This was one of the top results when searching for \"inside outside probabilities masked language modeling\" on Google, though I understand this paper is from earlier this year and the authors may have missed it at time of writing, I would be surprised if this is the first time someone has computed this quantity. \n\nOverall, I think I'd say that I don't know what a practitioner is supposed to have learned from this paper.", "questions_to_authors": "On line 542:\n> We investigate how this distribution forms during training by aggregating token probabilities to a general POS class, mapping the 43 tags in the original data to one of 7 base classes.\n\nWhat does this mean? It sounds like it means that each token is assigned to a fixed POS for each token in the vocabulary and you measure how well the learning LM matches the ground truth LM's distribution on that token type -- but POS tags are not fixed for the ground truth grammar? Why does the fact that the ground truth LM here is a PCFG matter? I think you could run this comparison on any ground truth generative model (like another causal LM)?", "paper_topic_and_main_contributions": "This paper trains LMs (both causal and non-causal/masked) on synthetic data generated from a PCFG. Unlike past work, their PCFG is large, giving it the capacity to be a better approximation to true English. Since LMs in this paper are trained from a known distribution, one can compare perplexities of the LMs to (approximations of) the lower bound from the ground truth model. \n\nThe authors find \"striking\" differences causal and masked LM objectives in their correlation with the true probabilities from the PCFG. They also study learning dynamics of word classes during training and find differences in the learning rates of classes of words (for example, punctuation is \"learned\" more quickly than adjectives). \n\nRESPONSE to rebuttal:\n> [W]e very much agree that the relevance does not come from PCFGs being particular good models of language, or from performance on PCFG-generated data being a good “sanity check”. Rather, the relevance of this work is, as we also wrote in response to Reviewer ybVx, in providing a distribution of intermediate complexity: not as language-like as those modeled by modern NLMs, but also not as idealized as in much prior work, and yet completely transparent.\n\nThis is a contradictor: you start by saying that it does not matter if PCFGs are particularly good models of language -- but then say that the value of this work is that they are \"not as idealized\" as much as prior work -- which in this context means \"better models of language.\" This paper takes a formalism (PCFGs) with some nice interpretability characteristics and changes it on exactly one axis: how well it models real language. I do not see any justification for why \"better but still bad\" on this axis is useful.\n\nThese PCFGS are also very much *not* \"transparent\" -- the latent subcategories learned are just as opaque as the vectors in a modern LM, requiring their own probing to \"interpret.\" \n\n> We don't disagree! PCFGs are much worse approximations of NL; but the one aspect that still makes them interesting is their inherent interpretability.\n\nI agree that PCFGs are more interpretable than neural LMs; it does not follow that neural LMs trained on PCFG-generated data are more \"interpretable.\" I mentioned this conflation when I asked about this sentence:\n> we can make stronger assumptions about the cues the model must be relying on than would be possible in natural language.\n\nand do not see a response to it in your rebuttal. \n\nYour answer to my question about\n> We investigate how this distribution forms during training by aggregating token probabilities to a general POS class, mapping the 43 tags in the original data to one of 7 base classes.\ndid not answer what you did. I understand that the PCFG provides POS tags to generated sentences where other LMs do not. My question was how you compare the distribution of the LM -- which does not provide those POS tags -- with the PCFG that does. I can imagine several things you might have done but the text here is not clear. I should also note that the POS tags present during generation are not ground truth -- for each sentence, there are many trees (with many different POS tags) that could produce that sentence. It's probably the case the the POS tag posteriors given a sentence are fairly peaked, but I'm still", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/P12-1101.pdf (another approach to improving PCFGs)"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluates and interprets neural language models using artificial data drawn from a language model learned from a PCFG. The PCFG is constructed using the state-split method over a large corpus. This method allows the definition for a closed-form expression for obtainable perplexity. 

All reviewers bring up issues with using a PCFG to evaluate language models, questioning the claimed gains in interpretability and the degree of approximation of natural language. Despite a lively author-reviewer discussion period, there was no consensus among reviewers about the soundness and excitement of the work. The authors make some good arguments in their responses, and the manuscript can be strengthened by incorporating feedback from the author-reviewer discussion and expanding on the motivation for using a PCFG. On the positive side, reviewers praised the thorough experimental analysis and noted that the finding that causal and masked LMs show differences in their correlation with the true probabilities from the PCFG is quite interesting.
Decision: Accept-Findings