Detecting Syntactic Change with Pre-trained Transformer Models
Authors: Liwen Hou
David A. Smith
Venue: EMNLP 2023 Findings (2023)
Keywords: BERT
Transformers
language change
syntax
syntactic change

Abstract: We investigate the ability of Transformer-based language models to find syntactic differences between the English of the early 1800s and that of the late 1900s. First, we show that a fine-tuned BERT model can distinguish between text from these two periods using syntactic information only; to show this, we employ a strategy to hide semantic information from the text. Second, we make further use of fine-tuned BERT models to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. To do this, we employ an automatic part-of-speech (POS) tagger and use it to train corpora-specific taggers based only on BERT representations pretrained on different corpora. Notably, our methods of identifying specific candidates for syntactic change avoid using any automatic POS tagger on old text, where its performance may be unreliable; instead, our methods only use untagged old text together with tagged modern text. We examine samples and distributional properties of the model output to validate automatically identified cases of syntactic change. Finally, we use our techniques to confirm the historical rise of the progressive construction, a known example of syntactic change.

Detecting Syntactic Change with Pre-trained Transformer Models
Liwen Hou
David A. Smith Northeastern University dasmi thêccs. neu. edu
Harvard University liwenhouêfas. hww.ard. edu
Abstract
across centuries. We isolate all of the parlia- mentary debates from the 1830s and the 1990s.' These debates can be found on the U.K. Parliament Website's Hansard archive".
We investigate the ability of Transformer-base language models to find syntactic differences between the English of the early 1800s and that of the late 1900s. First, we show that fine- tuned BERT model can distinguish between text from these two periods using syntactic information only; to show this, we employ strategy hide semantic information from the text. Second, we make further use of fine-tuned BERT models to identify specific instances of syntactic change and specific words for which a new part of speech was introduced. To do this, we employ part-of-speech (POS) tag- ger and use it train corpora-specifie taggers based only on BERT representations pretrained on different corpora. Notably, our methods of identifying specific candidates for syntactic change avoid using any automatic POS tagger old text, where performance may unre- liable: instead, our methods only use untagged old text together with tagged modern text. We examine samples and distributional properties of the model output to validate automatically identified cases of syntactic change. Finally, techniques to confirm the historical rise of the progressive construction, known example syntactic change.
We fine-tune DERT-base-uncasee around 110 million parameters and pre-trained approximately 3.3 billion words. When training or testing BERT on a sentence, we first tokenize it using the original BERT-base-uncune tokenizer, as implemented by Hugging Face (Wolf et al., 2020). We focus two questions: First, to what ex- tent can BERT distinguish text from different time periods the basis of syntactic features alone? Second, can we use BERT help identify words sentences whose analysis reflects syntactic change in an unsupervised way?
which has
In $2, we find evidence that BERT can be used to detect syntactic differences between the English of the 1830s and that of the 1990s. Specifically, even after we control for confounders, a fine-tuned BERT model can still distinguish between these two periods even when the semantic information is hidden.
In $3, we describe how we train classifiers that can assign POS tags to words based a 1830s and 1990s understanding of English, respectively. Our main contribution comes in $4, where we use these classifiers to identify words whose analysis reflects syntactic change. Our method can auto- matically find words whose part of speech has changed. Moreover, our method avoids running POS tagger on old text, since automatic tagging may be unreliable in that scenario. We emphasize that
1 Introduction
Many researchers have taken advantage of the avail- ability of pre-trained language models to study se- mantic change (e.g., Giulianelli et al.. 2020; Hu et al., 2019; Fonteyn, 2020), but computational studies of syntactic change are more sporadic, re- lying on specially trained models (Merrill et al., 2019). In this paper, we investigate the extent to which BERT (Devlin al., 2019) is able to dis- tinguish between the syntax of English from the early 1800s and that of the late 1900s. To do so, we use the British Hansard corpus, which consists a digitized version of the U.K. Parliamentary debates starting in the early 1800s and spanning
deatgeddetection techniques work with manual annotation and identify examples of lan- guage change without any manual filtering; see Appendix B for list the top automatically de- tected words whose part of speech has changed.
'e chose these decades because they were earliest latest decade with large amounts data every year. Phttps: //ww. hansard- archive, parliament. uk/

---

To the best of our knowledge, such unsupervised syntactic change detection has not been previously done.
dicting whether a sentence comes from the 1830s or the 1990s. However, there are still several ways for such a model to use non-syntactic information. First, the model may use sentence lengths, since the distribution of the sentence lengths is different for the 1830s than for the 1990s. Second, the BERT model may simply be using the counts of the POS tags rather than real syntactic information.
Finally, we confirm a known change (an increase in the frequency of progressives) using our classi- fiers. We show that a BERT model fine-tuned on text from the 1830s displays a poorer understand- ing of the progressive construction compared to similar model trained on text from the 1990s.
To mitigate the length confounder, we place re- strictions on the lengths of our training sentences. We run experiments separately for the length bins 11-15 words, 21-25 words, 31-35 words, and 20 words. To check the extent to which BERT relies on syntactic information other than the counts of tags, we separately train a model on shuffled input data, in which the sequence of tags in the sentence appears out of order. Such shuffled data retains no information other than the tag counts; we then compare whether BERT trained on unshuffled data can outperform BERT trained on shuffled data. The results are summarized in Table 1.
2 Detecting Syntactic Change While Masking Semanties
In this section, we check the extent to which BERT is sensitive to changes in the syntax of English over time. This acts as a sanity check for later sections, in which we use BERT in a different way to identify specific instances of syntactic change.
We start by fine-tuning BERT-buse-uncasee on the classification task of predicting whether sen- tences come from the 1830s or the 1990s. To do so, we use the representation of the CLS token in the last layer of BERT and add linear layer to it; this model is then fine-tuned using labeled sentences from the 1830s and the 1990s.
We compare these results to a logistic regression model that takes in the frequencies for the 100 most common vocabulary items in the replaced corpus" (plus catch-all category for the remaining items) and the sentence length features. Such logistic regression model achieves an accuracy of 85 45,,,, which means it performs about as well as BERT finc-tuned on shuffled tags but loses to BERT when the latter is fine-tuned on unshuffled tags. Using sentence length as the only feature yields an accuracy of 60.33%.
After we fine-tune it on 100,000 sentences per time period, the BERT model achieves 98.6% accu- racy; however, it may use semantic information to distinguish between text from the two time periods, and we want to limit ourselves to syntactic infor- mation only in order to check the extent to which BERT can detect syntactic change over time.
To make BERT use syntactic information only, we mask semantic information by replacing all the open class words in each sentence with their POS tags, obtained using Stanford CoreNLP (Manning et al., 2014). In order to leverage BERT's pre- trained knowledge of English, we then replace each POS tag with a common English word that usually has that tag (in a similar vein to Gulordava et al., 2018); this way, the input to BERT looks more like "colorless green ideas" and not "JJ JJ NNS".
The conclusion of this experiment is that the fine-tuned BERT model must be using syntactic information tell apart sentences from the 1830s and the 1990s. Indeed, even when the length of the sentence is fixed, and even when the words have been replaced by their parts of speech, the model gains advantage from seeing the true order of the tags as they appeared in the sentence, instead of seeing shuffled tags. This advantage must funda- mentally come from differences in syntax that are present between the 1830s and 1990s sentences (and moreover, these differences must take a form that not simply tag counts).
After replacing words (other than forms of "to be" and "to have", prepositions, and determiners) with POS tags and after replacing each POS tag with representative word for that part of speech, we fine-tune BERT (with the added layer described above) the resulting data. The BERT model per- forms surprisingly well on this classification task, achieving over 90% accuracy for the task of pre-
Having established that BERT is sensitive to changes in the syntax of English, we next train BERT in a different way to identify specific in- stances of such change. Importantly, from here on determiners, and forms of "to be" and "to have".
'In addition to the typical words that represent different parts of speech, this replaced corpus also contains prepositions,
'The code for our experiments is available here: https: //gi thub. www/houl hwtp://teeeeeeeeeeeeeeeee http:////tttteiieneettiiii..iiat syntactic-chang

---

sent. length
shuffled
unshufiled
shuffied
unshufficd plain
Next, using POS tags obtained from Stanford CoreNLP for the 1990s portion of the British Hansard, we train a 2-layer perceptron (with one hidden layer of size 50)* $ to assign a Penn Treebank tag to each BERT representation of each word. An input to this 2-layer perceptron is a BERT embed- ding of word (extracted as described above). and the output is POS tag. This classification task ignores punctuation and has 21 classes: We merge the 'NN', 'NNP", 'NNS', and 'NNPS' tags into one single tag called 'Noun', after which we take the 20 most common tags ('Noun', 'IN', 'DT', 'JJ', 'VB', 'PRP', 'RB', 'VBN', 'VBZ', 'VBP", 'CC', 'MD', 'TO", 'VBD', 'CD', 'VBG', 'PRPS', 'WDT", 'WP' 'WRB') and then group the remaining POS tags into catch-all class called 'other'. We actually train two such perceptrons separately: one for the 1830s BERT embeddings (of words from the 1990s only) and one for the 1990s BERT embeddings (also only of words from the 1990s).
replaced
replaced
plain
11-15
83.37
87.74
94.70
96.69
21-25
86.44
90.54
97.39
98.63
31-35
89.37
93.12
98.20
98,90
20
85.20
89.30
96.60
97.40
Accuracy BERT fine-tuned on different types of data for predicting time period. All models were trained on 20,000 sentences and tested on 2,000 (half from the 1830s and half from the 1990s); the ex- ception was the last line, for which only 10,000 training sentences were used due to lack of data. Since test data contained tences, baseline accuracy is 50%. These numbers are based on a single run; however, note that an accuracy of p will have standard deviation around vp(1 - p)/2000, which ranges from 0.23% to 0.83% for the above, with higher accuracies having lower standard deviations.
Table I:
mixture of 1830s and 1990s sen-
out, we avoid using POS tagger on older English.
3 Training Classifiers
Note that the training and test data for this proce- dure consist only of sentences from the 1990s; this is done because the POS tags obtained from Stan- ford CoreNLP are likely to be more accurate on our modern data. Using the representations from the final layer of BERT (when the classifier has access to 50,000 training sentences, containing 815,578 words, and when the test set has 50,000 sentences, containing 816,428 words), the accuracy obtained by the 1990s model is 97.6%, and the accuracy of the 1830s model is 97.2%.
We use BERT to identify specific instances of change. To do so, we train two POS taggers, one based on text from the 1830s and the other based on text from the 1990s; in the subsequent sections, we will use these taggers and their errors to identify words or sentences that display syntactic change. In this section, we will focus on describing how these taggers are trained and their performance. Crucially, we will train these taggers without ever tagging text from the 1830s; all POS tags are only ever associated with text from the 1990s.
We repeat the above analysis using POS tags from Trankit (Nguyen et al., 2021) instead of Stan- ford CoreNLP because Trankit is a statc-of-the- art parser and tagger.? By using Trankit tags to train and test the above classifiers based on BERT representations, the accuracy obtained by the 1990s model was 98.4% (instead of 97.6% us- ing CoreNLP tags) and the accuracy obtained by the 1830s model was 98.0% (instead of 97.2% us- ing CoreNLP tags). The higher accuracy means that the task became easier when predicting Trankit tags instead of CoreNLP tags; since Trankit tags are also more accurate, it suggests that our pipeline is working as expected.
We start by fine-tuning two instances of BERT- base-uncased: one on text from the 1990s and the other on text from the 1830s. This is done via Masked--anguage Modeling in PyTorch (Paszke et al., 2019), where we process one sentence at a time with 15% of the tokens masked, and the model is trained to predict the masked tokens. We use each fine-tuned model to obtain BERT token representations for 1990s text and then average the token representations for all the WordPiece tokens within each word to obtain a representation of a word. We do this twice: once with the model fine- tuned on text from the 1830s and once with the model fine-tuned on text from the 1990s (yielding two difffrrnn embeddings of each word). However, even though we have models trained on two time periods, we use both models only embed words from the 1990s and not any words from the 1830s. At the end of this process, each word from the 1990s has two different BERT embeddings.
In $4 and later, we will use these pipelines (and the differences between their predictions) iden-
*We tried hidden layer sizes 100 and 200; this hyperparam- eter did not make a large difference. www.. wwwnn. edu/ wtw.naalllaaaa/////aaaaa//aalala tttgs//iaaggeeeaggeeeeeeeeeeeeee pdf "https://tranki readthedocs. wttpa/s..o///////////a/eee/ae//a ////t//// performance.htmr

---

tify specific instances of syntactic change between the 1830s and the 1990s.
by the total number of tag disagreements for that word, normalized by the frequency of the word in the 1990s data, with additional weight given to words which were common in the 1830s (to reduce the number of rare words in the final list). We run this experiment on 100,000 test sentences, using classifiers trained on 25,000 sentences each.
4 Identifying Words that Underwent Change
We now attempt to use the POS taggers trained above to identify words that underwent change. We employ two methods to do this: First, we find the instances for which the two trained models disagree on a POS tag (indicating that the model familiar with text from the 1 1830s would tag the word differ- ently compared to the one familiar with text from the 1990s). The second method involves mask- ing out the target word see if the 1830s model predicts the tag correctly on the masked word. If the masked word is tagged correctly but the un- masked word is tagged incorrectly, it suggests that the 1830s model can correctly parse the rest of the sentence but is specifically confused by the target word; when this happens, it indicates that the tar- get word may have undergone change. This latter strategy is depicted in Figure 1.
We then manually examine the top resulting words to check whether the identified words have truly undergone change; see $4.3. One example can be seen in the following sentence: "My right hon. Friend last met M. Chevenement to discuss co- operative weapons development among other top- ics on 4 May." In this example, the model trained on text from the 1990s tags the word "co-operative" as an adjective whereas the model trained on the 1830s believes that "co-operative" is a noun; the 1990s model is correct in this case.
4.2 Masking and Confusion
As before, we use both the 1830s model and the 1990s model to predict the POS tag of each word in held-out 1990s sentences. This time, we addi- tionally repeat this process with one word in the sentence masked out: we ask both the 1990s model and the 1830s model to predict the POS tag of the masked word (without knowing what it is). We mask out each word in each sentence, one a time; since the BERT embedding for single word de- pends on the entire sentence, this requires passing the same sentence repeatedly through BERT with a different word masked each time. To speed up this process, we skip applying the mask to words that are too common (within the one thousand most common words in the 1990s), since these constitute the majority of word occurrences and yet are less likely to exhibit syntactic change.
4.1 Tagging Disagreements
After training, we have two new POS taggers, one based on an 1830s understanding of English and one based on a 1990s understanding. We now run these taggers on test data (from the 1990s) to find words on which they disagree. In particular, we find words for which the model trained on text from the 1990s agrees with the Trankit tag but where the 1830s model disagrees. The cases where such disagreement occurs should be the cases in which an 1830s grammar leads the model to incorrectly parse the sentence (likely due to a change in En- glish grammar for the relevant word or perhaps one of its neighbors). Moreover, we use our classifiers to extract probabilistic predictions for the tags, and we search for word instances where probability as- signed to the correct tag by the 1830s model was least 50 percentage points lower than that assigned by the 1990s model.
After this process, each word in each sentence (except those we have deliberately skipped) will have four different contextual BERT embeddings: one by the 1830s model when the word was masked, one by the 1830s model when the word was unmasked, one by the 1990s model when the word was masked, and one by the 1990s model when the word was unmasked.
We do this for each word in each sentence of the 1990s test data. To help isolate interesting cases, we filter the words under consideration to those that occur at least 10 times in the 1830s text (indicating that they are not new words). We remove numbers, punctuation, and words written entirely in capital letters (likely acronyms). We also restrict to words for which such a tagging disagreement occurred at least twice. We then sort the resulting words
We now examine words for which the 1830s model gives better prediction of the POS tag when the word is masked compared to when it is unmasked. When this happens, it means that the 1830s model correctly identified which tag to expect in the position of the masked word, and yet

---

There example of sovereign countries which have given up their currencies. There no example of [MASK] countries which have given up their currencies.
BERT (fine-tuned 1830s 1990s)
2-layer perceptron
Trankit tag: JJ
1990s masked; JJ
1990s unmasked: JJ >90%
1830s masked: JJ 44%
POS'tag
1830s unmasked: Noun 80%
prediction
Figure 1: A depiction the models' predictive process. The 1830s model predicts the correct tag when the word "sovereign" masked (the tags assigned the highest likelihood after JJ were VBG with 21 % and DT with 20% probability) but predicts it incorrectly when it was unmasked; this is the opposite of the usual pattern, in which the prediction is worse in the masked scenario. This type of tagging error indicates that the 1830s model expects "sovereign" to be noun instead of adjective.
after the mask is lifted (and the word itself was revealed) the 1830s model "changed its mind" and assigned the wrong tag instead. In such cases, we say that the 1830s model was "confused" by the token. We restrict to cases where the 1990s model correctly tagged the word, masked; this helps remove Trankit tagging errors well typographical errors in the text.
ple sentences and tags provided by the models) is given in Appendix B. The words this table were automatically generated with no manual curation.
4.3 Evaluation
when it was
To evaluate the resulting words, we manually ana- lyzed the top 20 candidates produced by $4.1 and the top 20 candidates produced by $4.2. For each candidate word, examined 10 randomly sam- pled sentences containing that word from the 1990s and also 10 sentences from the 1830s, and we man- ually checked whether their syntax appeared have changed.
We then consider the remaining words for which the 1830s BERT model was confused more than restrict to words for which there were at least two separate occurrences where the 1990s model tagged the word correctly (with a confidence of at least 90%) and the 1830s model assigned a probability that is at least 25 percentage points higher to the correct POS tag when seeing the masked version of the sentence than it did when seeing the unmasked version. We run this experi- ment on 100,000 test sentences; train the 1990s and 1830s models, we 25,000 sentences each (just in the previous section). To help isolate interesting cases, we employ the same filters and the same sorting strategy as we did $4.1.
once; specifically,
Of the top 20 words from $4.1, seven (or 35%) were identified as having undergone syntactic change, with part of speech in the 1990s that is either the top 20 words from $4.2, 11 (or 55%) were iden- tified having undergone syntactic change.
greatly increased in frequency. Of
We note also that these two approaches identified different sets of words: Of the top 20 words on each list, there were only 4 words of overlap, while of the top 100 there were 15 words of overlap.
An example of word that found this way is "material", which was an adjective in the early 1800s but whose noun usage is modern. For in- stance, in the sentence "the inquiry will be free to publish whatever matcrial it wishes," Trankit cor- rectly tags "material" as a noun, and so does the 1990s pipeline, but the 1830s pipeline only gives the correct tag when the word "material" is masked out in that sentence; when it is unmasked, an adjec- tive is predicted.
We also attempt an automatic evaluation by us- ing Trankit to tag words from the 1830s and check- ing these tags changed. Of course, our identifica- tion methods purposely avoided using a tagger on old text, since it may be unreliable, and manual check reveals that the tagger made mistakes. Still, as sanity check, we use it to compare the two methods for identifying syntactic change.
For the purpose of automatic evaluation, we say that word has changed and gained a new POS if the tag is at least twice as likely to appear for
list of the top resulting words (with exam-

---

1990s usages of the word than for 1830s usages of the word. Under this criterion for change, and using the Trankit tags for both 1830s and 1990s data, we get that the top 100 candidates from the simpler identification method ($4.1) contained 20 changed words, while the top 100 candidates from our masking strategy ($4.2) had 35 changed words. Note that of the manually evaluated words, the au- tomatic evaluation agreed with 75% of the human judgements. These should be interpreted with cau- tion, but these results agree with our conclusion that the masking strategy is better at identifying changed words.
these probabilities get measure of the overall probability the model assigns to the correct tags of the sentence. We can repeat this for both the 1830s and 1990s classifiers, and we can also repeat this when a specific word is either masked or unmasked.
We say a model is "confused" if the product of probabilities assigned to the correct (Trankit) tags is higher when the word is masked than when it is unmasked. For instance, in the above example sentence "Opposition Members think that is how peace processes work," the 1830s model may as- sign wrong tag to the word "processes" when the word "work" is unmasked, but it might correctly parse the sentence if the word "work" is masked (in the latter case, it will assume the masked word an unknown verb, while in the former case it will try to parse the sentence with a noun in that posi- tion if it believes that "work" is a noun). In other words, if the 1830s model is confused by a word, it could be because this word underwent change from the 1830s to the 1990s. We create "confusion score" by taking the logarithm of the ratio of the two aforementioned products (i.e. the product of probabilities assigned to the other words when the selected word is masked, and the corresponding product when the selected word is unmasked).
5 Confusion Scores at the Sentence Level
In this next experiment, we try to discern the extent to which masking out word in a sentence makes the sentence more difficult for a fine-tuned BERT model parse. The idea is that, for a typical word, once the word is masked out the model has less information about the sentence, and its syntactic understanding should therefore decrease; on the other hand, for word that has an atypical or new usage, the model may end up being more confused after seeing the word than it is when the word is masked out. In the latter situation, would expect the model's '' syntactic understanding of the sentence improve if the word is masked out because, from the model's point of view, the word misleading. A similar idea was used in $4.2 at the word level; in this section we will look at the accuracy of the tags for the whole sentence before and after single word is masked.
We can get such a confusion score for any word in any sentence, and for either the 1830s model or the 1990s model. We focus on sentences from the 1990s that were not part of the training data. Since calculating confusion scores is resource-intensing we focus on a pre-selected set of word instances. such as those identified as likely having undergone syntactic change in $4.
For example, consider the following sentence: "Opposition Members think that how peace pro- cesses work." A model trained on English from the 1830s is likely to think that the word "work" is a noun rather than a verb. This could lead such a model to consider an incorrect way to parse this sentence, where the subject "peace" is doing the action of processing the object "work". In this case, if the word "work" is masked out, the sentence will be more likely to be parsed correctly.
5.1 Comparing Selected Words to Random Words
We select 10,000 word occurrences for which the tag is predicted differently by the 1830s model and the 1990s model (as discussed in $4.1). 1). For each word occurrence, we calculate confusion score. We then compare these scores to the confusion scores of 10,000 randomly chosen word instances. Our goal is to check whether the confusion scores of the 10, 10,000 selected words were higher than those of random words.
As proxy for the success of the parsing of the sentence, we feed in the sentence to the classifiers from $3 to get predicted tags for all the words in the sentence. We can use these classifiers to get probabilistic predictions, and in particular, we can get a probability assigned to the correct tag for each word in the sentence (using Trankit tags as ground truth). We then multiply together all
One possible confounder is that the selected words may be rarer than the random words. To adjust for this, we choose the random words more carefully: for each non-random word in the first set, we pick a random word conditional on its overall

---

sentences with word mistagged by 1830s model
for example, sentences that contain typographical errors, sentences that are in- correctly parsed by Trankit, sentences that were ungrammatical when originally spoken, or other special cases. By focusing on instances that the 1830s model found confusing but that the 1990s model did not find confusing, we increase the like- lihood of finding syntactic change.
than language change:
random sentences
1.2
relative frequenc
1.0
0.8
0.6
0.4
0.2
0.0
-20
-10
0
10
20
5.3 Manual Evaluation
log ratic masked unmasked probs given tags
We generate list of top 100 candidates for syn- tactic change based on these confusion scores. Of those 100 sentences, our manual evaluation deter- mined that 33 sentences were valid instances syntactic change. We also computed the sum of the reciprocal ranks of the 33 manually selected sen- tences that exhibited syntactic change. This sum was 2.802, while the maximum possible sum of re- ciprocal ranks would be $ + + +...++++ 33 4.089.
Figure 2: Distribution of confusion scores for selected words and random words.
frequency being close to that of the original word. We define close to be within 10 percentage points in the percentile of their frequencies. The resulting distribution of the confusion scores for the random words had lighter tails than the distribution of the selected words, especially on the right-hand side of the distribution. See Figure 2.
Detecting the Rise of the Progressive
As the figure shows, the selected word instances (for which the 1990s model and 1830s model dis- agree the tags) have higher proportion of high confusion scores than random words. Therefore, calculating confusion scores for the selected word occurrences a more resource-efficiet way find instances of high confusion scores.
In this section, we check the extent to which our techniques can identify syntactic change that was previously known to exist.
The frequency of the progressive has been in- creasing over time (e.g. Leech et al., 2009; Hou and Smith, 2021). An example of a progressive is "he is reading" (as opposed to "he reads"): the for- mer has a form of "to be" followed by verb that ends in "-ing". We identify progressive construc- tions using the dependency parse obtained from Trankit (Nguyen et al., 2021). We then mask out the present participle if it has the tag "VBG" and if the previous word is an inflection of the verb "to be". (This means that we mask out the tokens constituting the "-ing" word as well as the "-ing" suffix itself.)
5.2 Further Refinements
In addition calculating confusion scores for word in given sentence using the 1830s model, we also calculate confusion scores in similar way using the 1990s model. Recall that the confusion score tells us the extent to which the BERT-based tagger performs better when a word is masked out than when is not. Since we have two such taggers (one fine-tuned on parliamentary speeches from the 1830s and one fine-tuned on speeches from the 1990s), we can get two confusion scores for single word in single sentence.
Next, we use both the 1990s model and the I 1830s model to predict the POS tags of all the words in these masked sentences. We compare these pre- dictions each other and also the predictions that both models give for the unmasked versions of these sentences. Finally, we take randomly chosen words in randomly chosen sentences as a control group and repeat the process for this group.
In order to zoom in on syntactic change, we re- strict our attention to instances where the 1990s confusion score is less than or equal to zero. Since the confusion score is the logarithm of the ratio of masked to unmasked probabilities, this means are restricting to word instances where the 1990s model gave a better prediction of the tags when the word was unmasked than it did when the word was masked. Applying this restriction helps to re- move cases that are confusing for reasons other
We summarize the results in two figures. First, Figure 3 shows the confusion scores for the 1830s model on the progressive sentences compared random sentences. As the figure shows, the 1830s model is negatively confused when we mask out progressives, which means that it gets the predic-

---

1.2
Gupta al., 2021), which is consistent with our ob- servation that BERT fine-tuned on shuffled text can distinguish between two time periods with high ac- curacy; however, our experiments indicate BERT's accuracy still improves when it has access to word order, consistent with Papadimitriou et al. (2022), who argue that word order sometimes matters by considering non-prototypical cases where mean- ings and lexical expectations do not match.
sentences with progressives
1.0
random sentences
relative trequency
0.8
0.6
0.4
0.2
0.0
A sizable litcrature considers syntactic probes for BERT embeddings (e.g. Tenney et al., 2019; Jawahar et al., 2019; Hewitt and Manning, 2019; Pimentel et al., 2020), but to our knowledge none of the works that probe syntax consider Transformers fine-tuned different time periods.
log ratic of masked and unmasked probs given to true tags
Figure 3: A histogram for the confusion scores of the 1830s model on progressives and on random sentences.
tion right more often on the unmasked version of these sentences than it does on the masked versions. This makes sense, it corresponds to the insight that the 1830s model has a difficult time correctly tagging the masked progressives.
Syntactic change detection has been understud- ied in the computational linguistics and NLP lit- erature. One work on this topic is Merrill et al. (2019), in which an LSTM achieved higher accu- racy compared to model consisting of feedfor- ward networks on the task of predicting the year of composition of sentences. However, in addition to predicting the time period of text, we also identify specific instances of syntactic change. Computa- tional methods for the study of syntactic change also appear in Hou and Smith (2018); this work examined possible causes of the decline in the rate of passivization over time. Another related work is Lei and Wen (2020), which found dependency distance to be decreasing over time (between the years 1790 and 2017).
VIZ
sentences with progressives random sentences
0.6-
NNN
relative requency
0.5
0.3
0.2
0.1
15
10
5
5
10 15
log ratio of 1990s and 18305 probs given to true tags
Figure 4: A histogram for the logarithm of the ratios between the products of probabilities assigned to the true tags by the 1990s model and the 1830s model. We compare this for progressive sentences and random sen- tences. In all cases, both models were given the masked version of the sentences.
Many studies have used BERT to detect semantic change, including Giulianelli et al. (2020), Hu et al. (2019), and Fonteyn (2020). In addition, Beelen et al. (2021) fine-tune historical BERT models on corpus of 1enhtcy English books in order to tackle the task of time-sensitive Targeted Sense Disambiguation and find that "BERT works better when fine-tuned on data contemporaneous the target period"".
Figure 4 compares the predictions of the 1990s model to those of the 1830s model for the masked sentences. It shows that the 1990s model predicts the tags of the progressive sentences better than the 1830s model does, and the gap between their predictions is larger than it for random sentences. This is evidence that the 1830s model is unfamiliar with progressive constructions.
Conclusion
We investigated the extent to which fine-tuned BERT models can distinguish between the syntax of 1830s English and 1990s English. Our approach used masking techniques, and we repeatedly com- pared BERT models on masked and unmasked text.
7 Related Work
It has been claimed that the performance of Trans- formers on downstream tasks is insensitive to word order (Hessel and Schofield, 2021; Sinha et al., 2021a; Sinha et al., 2021b; Pham et al., 2021;
First, in $2, we hid semantic information by re- placing each word with its POS tag (and repre- senting that tag as a common English word that exemplifies that tag). We fine-tuned BERT on such

---

syntax-only data to see if it could tell apart text from the 1830s and text from the 1990s. We found evidence that BERT can be used to detect syntactic differences between these two time periods.
References
Mare
Alexander and Mark Davies. 2015.
The
Hansard Corpus 1803-2005.
http:
//www. hawharadaaraiiieeeeee.m par ament. uk/.
In the later sections, we masked out individual words to see whether a POS tagger based on BERT embeddings (fine-tuned on either 1830s or 1990s) still tag correctly when the word is masked. We filtered for specific type of error, where the 1830s model predicts the correct tag when the word is masked but not when it is unmasked, and where the 1990s model is correct even when the word is unmasked. This procedure identified candidates for syntactic change, examples of which we provide in Appendix B. We were also able to confirm the rise of the progressive construction.
Kaspar Beelen, Federico Nanni, Mariona Coll Ar- danuy, Kasra Hosseini, Giorgia Tolfo, and Barbara McGillivray. 2021. When time makes sense: A historically-aware approach to targeted sense dis- ambiguation. In Findings of ACL-IJCNLP, pages 2751 7751-2761.
Jacob Devlin. Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In MAACL, pages 4171-4186.
Lauren Fonteyn. 2020. What about grammar? Using BERT embeddings to explore functional shifts semi-lexical and grammatical constructions. Computational Humanities Research, pages 257- 268.
An important point to highlight is that our identi- fication of syntactic change relied on an automatic POS tagger only for the 1990s text; we avoided running any automatic tagging software on older English data. In fact, from $3 onwards, all we did with the older 1830s data was to fine-tune BERT on it in an unsupervised fashion. Despite this re- striction, we were able to automatically identify words whose part of speech had undergone change between the two time periods.
Mario Giulianelli. Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualised word representations. In ACL, pages 3960-3977
Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018, Colorless green recurrent networks dream hierarchically. In NAACL, pages 1205-
Ashim Gupta, Giorgi Kvernadzee and Vivek Srikumar. 2021. BERT & family eat word salad: Experiments with text understanding. In AAAI, pages 12946- 12954.
Limitations
There are several limitations to this work. First, we worked only with English and only with text from the I 1830s and 1990s. Our work also required the use of GPUs; see Appendix A for details of resources used. Although we were able to auto- matically identify specific instances of change (and we show automatically generated examples in Ap- pendix B), our evaluation was limited to the manual judgement of one author, and we were unable to de- termine the recall of our detection system (because the lack of objective criteria for what constitutes syntactic change). Additionally, our techniques are limited in the kinds of change we can detect: we focus on finding words whose typical part of speech has changed and on validating known syn- tactic changes such the rise of the progressive. We hope that similar techniques can be extended to other types of syntactic change in future work.
Jack Hessel and Alexandra Schofield. 2021. How effec- tive is BERT without word ordering? implications for language understanding and data privacy. In ACL- IJCNLP, pages 204-211.
John Hewitt and Christopher D. Manning, 2019. A structural probe for finding syntax in word represen- tations. In NAACL, pages 4129 1129-4138.
Liwen Hou. 2022. Detecting and Modeling Syntactic Change. Ph.D. thesis, Northeastern University.
Liwen Hou and David A Smith. 2018. Modeling the decline in English passivization. In SCiL,, pages 34- 43.
Liwen Hou and David A. Smith. 2021. Drivers of En- glish syntactic change in the Canadian parliament. In SCiL, pages 51-60.
Renfen Hu, Shen Li, and Shichen Liang. 2019, Di- achronic sense modeling with deep contextualized word embeddings: An ecological view. In ACL. pages 3899 3908.
A Acknowledgments
We thank Jacob Eisenstein and the anonymous re- viewers for their helpful feedback.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In ACL, pages 3651-3657.

---

Geoffrey Leech, Mariannn Hundt, Christian Mair, and Nicholas Smith. 2009. Change in Contemporary English: A Granmatical Study. Studies in English Language. Cambridge University Press.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush, 2020. Trans- formers: sfate-o--the-an art natural language processing. In EMNLP System Demonstrations, pages 38 45.
Lei Lei and Ju Wen. 2020. Is dependency distance ex- periencing a process of minimization? A diachronic study based on the State of the Union addresses. Lin- gua, 239:102762.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Me- Closky. 2014. The Stanford CoreNLP natural lan- guage processing toolkit. In ACL System Demonstra- tions, pages 55-60.
A Resources Used
In this work, we mentioned and used the follow- ing licensed resources: BERT (Devlin et al., 2019), which uses Apache 2.0 license; UK Parliament Hansard Corpus (Alexander and Davies, 2015), which uses Open Parliament Licence; Trankit large version 1.1.0 (Nguyen et al., 2021), which uses Apache 2.0 license; and Stanford CoreNLP version 4.2.2 (Manming et al., 2014), which uses the GNU General Public License v3.0. All of these licenses allow the type of research conducted in our work. Moreover, our work falls within the intended use of each of these resources.
William Merrill, Gigi Stark, and Robert Frank. 2019. Detecting syntactic change using neural part-of- speech tagger. International Workshop on Compu- lational Approaches to Historical Language Change, pages 167-174.
Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Vey- seh, and Thien Huu Nguyen. 2021. Trankit: light- weight transformer-based toolkit for multilingual nat- ural language processing. In FACL System Demon- strations, pages 80-90.
Isabel Papadimitriou. Richard Futrell, and Kyle Ma- howald. 2022. When classifying grammatical role, BERT doesn't it care about word order... except when it matters. In ACL, pages 636-643.
Our work required GPU computation. We used NVIDIA Tesla V100 and P100 GPUs over sev- eral months while working on the results of this paper. Running Trankit was the slowest step and took around 72 GPU hours; the total GPU time to reproduce our final results would be under a week.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high- per-oeerformncc deep learning library. In NeurIPS. pages 8024 8035.
B Automatically Identified Words
Table 2 shows the top of the list of automatically identified words whose part-of-speech usage may have changed; a longer list can be found in Ap- pendix A of the first author's thesis (Hou, 2022). For each word, we give an example sentence from the 1990s on which the 1830s model was "con- fused". For each example sentence, we give the Trankit tag and two sets of probabilistic tag pre- dictions given by the 1830s model; sponds to the tags predicted by the 1830s model for the highlighted word when the word was masked, and the other set corresponds to the tags predicted by the 1830s model for the highlighted word when the word was unmasked. (For brevity, we only list the tag predictions to which the model as- signs at least 30% probability.) An arrow such as 'JJ=Noonn' indicates that the correct tag in the 1990s was 'Noun' but that the unmasked 1830s model put a high weight on 'JJ' instead; this sug- gests the word changed from 'JJ' in the 1830s to 'Noun' in the I 1990s. (Recall that all POS tags for nouns were merged into a single 'Noun' category.)
Thang Pham, Trung Bui, Long Mai, and Anh Nguyen. 2021. Out of order: How important is the sequen- tial order of words in sentence in natural language understanding tasks? In Findings of ACL-IICNLP. pages 1145-11600
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020. coemmation-thcoretee probing for linguistic structure. In ACI, pages 4609-4622.
set corre-
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 202 la. Masked language modeling and the distributional hy- pothesis: Order word matters pre- pre-training for little. In FMNLP, pages 2888-2913
Koustuy Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. 2021b. UnNatural Language Inference. In ACL-IJCLLP, pages 7329-7346.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In ACI, pages 4593-4601.

---

opposite
"The mandatory life sentence does the opposite of what it sets out to achieve, and it should go."
JJ-Noun
Trankit: Noun masked tag: 84% Noun
unmasked tag: 93% JJ "I presume that that shows they will be on message until they die ."
presume
VB VBP
Trankit: VBP masked tag: 98% VBP unmasked tag: 38% VB, 32% VBP
corporal
"The school board wrote to me on 29 October advocating the reintroduction of corporal punishment in schools as a deterrent of last resort
Noun-+JJ
Trankit: JJ masked tag: 74% JJ unmasked tag: 95% Noun
intrepid
"The second category are the Royal Marine principal transport and landing ships, Fearless and Intrepid :"
JJ->Noun
Trankit: Noun masked tag: 64% Noun unmasked tag: 97% JJ
evil Noun +JJ
"The gunmen, in the evil atrocity that they committed, symbolised the past." !'
Trankit: JJ masked tag: 82% JJ
unmasked tag: 99% Noun
remembered
"It should be remembered that during the period 1974 79, when he was a senior Cabinet Minister, the steel industry inefficient and unprofitable
JJ-BN
Trankit: VBN masked tag: 86% VBN unmasked tag: 64% JJ, 36% VBN
shut
"My second charge is that the Prime Minister has consistently talked about the incentive society with open opportumities for all yet her policies have effectively
VBD+VBN
shut out a quarter of the nation."
Trankit: VBN masked tag: 97% VBN unmasked tag: 86% VBD
offending JJuNoun
"I can vouch for the fact that juvenile offending is major problem in my constituency." !"
Trankit: Noun masked tag: 98% Noun unmasked tag: 58% JJ
granting
"In most normal cases there will be the usual opportunity for consultation and granting of opinion from the local community
VBG-Noun
Trankit: Noun masked tag: 99% Noun unmasked tag: 98% VBG
disposed
"Mr. Streeter: To ask the Secretary of State for Defence what Ministry of Defence land and buildings in Plymouth he expects to be surplus to MOD requirement in the next five years and be disposed of the private sector "
Noun+VBBN
Trankit: VBN masked tag: 87% VBN unmasked tag: 48% Noun, 31% VBN "We should also encourage housing associations to feel the exciting and exhilarating wind of competition."
exciting
VBG- >JJ
Trankit: JJ masked tag: 99% JJ unmasked tag: 94% VBG
laying
"More significant orders especially those exercising the so called Henry VIII powers, to be subject to laying in draft and affirmative resolution." !
VBG->Noun
Trankit: Noun masked tag: 98% Noun unmasked lag: 98% VBG "Notwithstanding signal failures and operational difficulties, look forward to using that fast service." ."
notwithstanding
VBG-NN
Trankit: IN
masked tag: 84% IN
unmasked tag: 86% VBG
Table 2: Automatically identified words whose part-of-speech usage may have changed


Reviews:
[{"review_id": "kzM0D11LGD", "reviewer": "Reviewer_mmN7", "summary": "", "questions": "1. I assume that the words used to replace the POS in section 2 are always the same for both periods and there is no effect of selecting different such words - correct?\n2. I assume that the tags used in Trankit are not the same as those used in Stanford CoreNLP - but which ones?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for giving us feedback. The reviewer raised concerns about our approach, which we address here:\n\n>*the model pretends not to be supervised. However, a single tagger for modern English is used as a reference (line 270), which means that information generated in a supervised manner is included in the model*\n\nOur goal is to avoid supervision on the older English text. We allow supervision for text from the 1990s, and we never pretended otherwise. The reason to avoid using a POS tagger on English from the 1830s is that we do not have a trained tagger for this time period (nor do we have supervised training data for the 1830s).\n\n>*There is a kind of self-contradiction in the approach: the classification task requires the use of a POS tagger trained on standard contemporary English to historical text. I thought the authors wanted to highlight the problem of performing such an approach in line 071f.: \"our method avoids running a POS tagger on old text\" - and as admitted for Stanford CoreNLP by the authors in line 217f.; (I admit that from line 275 onwards the paper returns to its premise).*\n\nThis only applies to Section 2, which is a self-contained section with a separate approach and a separate goal. The rest of the paper (including the main model) does not use such a POS tagger on historical text and does not use any classifier from Section 2. The reviewer mentions \"line 275\", but Section 2 is from line 87 to line 175 (not 275).\n\n>*The model is not well described in all details: in particular, the part in lines 177-199 where it seems necessary to switch between the 1830- and 1990-related representations in order to transfer information from one \"period\" to the other is not understandable in the way it is described*\n\nIt is not necessary to switch between the 1830s and 1990s representations. We train two different instances of BERT for the two time periods, and we do not need to switch between the representations. We only compare the tagging predictions based on these BERT representations.\n\nWe respond to the reviewer's questions below:\n\n>*1. I assume that the words used to replace the POS in section 2 are always the same for both periods and there is no effect of selecting different such words - correct?*\n\nThe words are always the same for both periods. We did not test different word sets.\n\n>*2. I assume that the tags used in Trankit are not the same as those used in Stanford CoreNLP - but which ones?*\n\nTrankit and Stanford CoreNLP are both capable of outputting Penn Treebank Tags, so we deal with the same set of tags after switching taggers. Footnote #5 in our paper contains a link to the Penn Treebank.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper promises to avoid manual tagging of data to a larger degree, thus contributing to efficient modeling, especially in the area of historical texts where manual tagging is laborious and requires hard-to-find experts.\n2. The paper proposes a simple but nice classifier to separate semantic and syntactic information, at least superficially.\n3. The study of syntactic changes is an interesting area of research, especially from the point of view of using BERT and related approaches when semantic information is more or less ignored. The paper is an interesting proposal in this respect.", "reasons_to_reject": "1. The paper considers a particular type of morpho-syntactic change as an example of syntactic change, by studying POS tagging changes in context as an indicator of system-level syntactic change, using examples of words that are likely (or at least to an unknown degree) to be subject to conversion (https://en.wikipedia.org/wiki/Conversion_(word_formation)) - e.g., sovereign. It is not so clear to me why it should be too hard to detect such \"POS changes\", or changes in preferences for using the word in either POS (e.g. noun vs. adjective): wouldn't something like a simple regular expression approach, working on parse trees or something, and using available lexicon information (https://en.wiktionary.org/wiki/sovereign), do much of the job? At least this information could be used to test the algorithm of the paper. In any case, conversions and similar examples of POS changes are not, in my opinion, such a good example of syntactic changes. Also, the database is quite small: only two \"periods\" of two years are considered. So the results may not be very meaningful, since we don't know what happened in the time in between. And how much do the results depend on the very specific genre of the underlying text base: can we really derive generalizations about syntactic changes in English from this? There are at least some doubts. From my point of view, the paper is more about the applicability of different fine-tuned models to texts from periods on which they were not trained, rather than providing a valid model of syntactic change. However, the paper does not systematically examine alternative models; there is no comparative evaluation. The test data are only taken for sentences from the 1990 period: why not for the other period - the underlying tests do not seem to be comprehensive enough. In addition, the model pretends not to be supervised. However, a single tagger for modern English is used as a reference (line 270), which means that information generated in a supervised manner is included in the model. \n2. There is a kind of self-contradiction in the approach: the classification task requires the use of a POS tagger trained on standard contemporary English to historical text. I thought the authors wanted to highlight the problem of performing such an approach in line 071f.: \"our method avoids running a POS tagger on old text\" - and as admitted for Stanford CoreNLP by the authors in line 217f.; (I admit that from line 275 onwards the paper returns to its premise).\n3. The classifier for separating semantic and syntactic information is nice, but one cannot tell from its use what syntactic information is actually represented. BERT represents some syntactic information, but what exactly: one does not learn that much about it. In other words, the supposed syntactic classifier is a black box, which hides the actual syntactic information in use. Is it just n-grams of certain words of certain POS in a certain sentence window? Or something more interesting in terms of syntax?\n4. The model is not well described in all details: in particular, the part in lines 177-199 where it seems necessary to switch between the 1830- and 1990-related representations in order to transfer information from one \"period\" to the other is not understandable in the way it is described: I suspect that there is a simple algorithm behind this, but the description does not live up to this simplicity. Figure 1 is a big step forward in terms of comprehensibility, but the text itself falls short.", "questions_to_authors": "1. I assume that the words used to replace the POS in section 2 are always the same for both periods and there is no effect of selecting different such words - correct?\n2. I assume that the tags used in Trankit are not the same as those used in Stanford CoreNLP - but which ones?", "paper_topic_and_main_contributions": "The paper aims at an unsupervised approach to syntactic change detection based on BERT. It starts from a sentence-level classifier for distinguishing texts from different time periods based on syntactic information (while hiding semantic information) to identify words or phrases that manifest a relevant syntactic change. To this end, the paper focuses, among other things, on the detection of words that undergo POS changes. The main idea is to look for differences in two POS taggers specifically fine-tuned for the texts of the two \"periods\" under consideration, and to test whether they produce differences in the tagging of some test sentences when it comes to cases where a test word is masked once and then not: here, differences are taken as indicators of context-specific POS changes if the complementary model tends to not indicate such a difference.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"an automatic part-of-speech\" -> \"an part-of-speech\" (\"automatic tagger\" is an pleonasm)\n- \"BERT is aware of syntactic differences\" -> \"BERT can be used to detect syntactic differences\" (to speak of awareness is an anthropomorphism) - this should be corrected throughout thew paper\n- \"displays a poorer understanding of the progressive construction\" -> \"is a worse model for representing progressive construction\"\n- \"we want to restrict to syntactic information\" -> \"we want to limit ourselves to syntactic information\"\n- \"explosion.ai/blog/ud-benchmarks-v3-2\" -> \"https://trankit.readthedocs.io/en/latest/performance.html\"\n- \"the model trained on text from the 1990s believes the word “co-operative” is an adjective\" -> \"the model trained on text from the 1990s tags the word “co-operative” as an adjective\""}}, {"review_id": "wwKniteEo0", "reviewer": "Reviewer_wgbS", "summary": "", "questions": "A) Maybe you can say something about the tag-old-data-with-tagger-trained-on-new-data aspect?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n>*Maybe you can say something about the tag-old-data-with-tagger-trained-on-new-data aspect?*\n\nExcept in Section 2 (line 87 to line 175), we do not tag old data with a tagger trained on new data. For the main part of the paper, we only ever tag data from the 1990s. Our trick is to train a tagger that uses an older (1830s) understanding of English and have it tag 1990s data; we can then see where it makes mistakes. That is, we train BERT in an unsupervised fashion on the 1830s text. We then run this model to embed text from the 1990s and use these embeddings to train a tagger (trained only on 1990s data, but using the representations that were informed by an 1830s \"understanding\" of English). We identify instances of change by looking at tagging disagreements between this tagger and an analogous one based on 1990s data only; the tagging disagreements are checked for the 1990s data only. This way, all tags are only ever generated for the 1990s, and no tags are associated with text from the 1830s (outside of the separate analysis in Section 2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A good sequence of experiments building on top of each other that seem to be sound to me and which could be adapted to other questions.", "reasons_to_reject": "No real reasons", "questions_to_authors": "A) Maybe you can say something about the tag-old-data-with-tagger-trained-on-new-data aspect?", "paper_topic_and_main_contributions": "This submission proposes to use fine-tuned language models to detect syntactic change in language.  They use transcriptions from the UK parliament from the 1830s and 1990s, automatically annotate them with POS tags, train a language model to predict POS tags (one for each epoch) and then check whether the two models disagree on their predictions.  The second kind of test is to mask a word, predict the words POS and check whether the model gets the POS right with masking but wrong with the word present. (Plus some more aspects that I leave out here)\n\nIn their first experiment, the authors try to remove confounders by replacing words with other words of the same POS in the training and test data and show that they can still train a model that can distinguish between new and old text; meaning that the LLM can distinguish the different POS sequences and does not rely on word identity. By also training on shuffled data, they show that the model is not only learning bag-of-word information but takes into account the actual sequence.  This experiment is convincing.\n\nIn the second and main experiment (which I described in the first paragraph), two models are being trained. I am confused by the reporting of accuracy, given that the models are not trained on some manually labeled ground truth but the output of another part of speech tagger. Additionally, the (necessary) inclusion of a tagger to create the training material may introduce a bias, as this tagger is probably trained on modern data -- I am not sure how to think about a tagger (the BERT model) that was trained on old text, which in turn was annotated by a tagger trained on modern text.\n\nThe authors then go on and find words in the 1990 text where \n - the 1830 model predicts an incorrect tag and\n - the 1830 model predicts a correct tag when the target word is masked and\n - the 1990 model predicts a correct tag.\n\nThis approach seems sound to me for finding a change, but to better understand to what extend we are dealing with noise and not shift, it would probably also have been a good idea to cross-check the instances where the 1990 model is wrong on 1990 text but the 1830 model is right.\n\nI am not sure that all the experiments show that they extract syntactic shift. For example:\n - corporal (adj vs noun) might be due to a distribution shift of both meanings, exacerbated by the old adjective writing \"corporeal\"\n - inteprid occurs often as a noun because there is now a ship called \"HMS Inteprid\". This ship was under discussion due to ship renewals(?) in the 1990 and, as this ship did not exist in the 1830s, there was no proper noun usage back then. I do not think that people who have no interested in the British navy would now consider inteprid to be a noun.\n\nIn the third experiment, the authors take the same kind of test to the sentence level and look at the probability for all correct POS for a given sentence, again with the modes described above.  In the fourth experiment, they check whether sentences with progressives and show that the 1830 model assigns more probability to the correct tag sequence when the present participle is masked, which is not true (to the same extend) for random masking and also not true for the 1990 model.\n\nOverall, I found this to be a good paper with a clear sequence of questions and clear description of the methods and reasoning behind the methods.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ENzqO0nrLE", "reviewer": "Reviewer_kxVY", "summary": "", "questions": "- Will the code and data be released? I did not see any mention of it.\n- Section 3 is a bit confusing. The section starts with a paragraph saying that you train POS taggers, then later on we switch to classifiers. Should it be sequence taggers throughout the paper? Then we get into the training of MLM, and finally, the classification task using the learned representation. If the authors could make this section a bit clearer, we wouldn't need to read it over and over to wrap our head around what exactly is going on.\n- Section 5.3: The authors expose the results from this experiment but nothing is said. What do these results tell us?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback.\n\n>*Will the code and data be released?*\n\nThe data is publicly available from the UK government. We will release the code and the specific data splits that we used if the paper is accepted.\n\n>*The section starts with a paragraph saying that you train POS taggers, then later on we switch to classifiers. Should it be sequence taggers throughout the paper?*\n\nOur taggers are classifiers, as they take as input an embedding of a word and classify this embedding as one of 21 possible classes. Our taggers are not sequence-to-sequence taggers; instead, we use a BERT model to embed each word in a sentence, and we train a separate classifier to tag each word using only its embedding. (We do this twice, for two different BERT models, each fine-tuned on one decade.)\n\n>*Section 5.3: The authors expose the results from this experiment but nothing is said. What do these results tell us?*\n\nOf the top 100 candidate sentences that were automatically generated, our manual evaluation concluded that 33 of them were valid instances of syntactic change. We view this as a positive result, since a large fraction of the candidate changes were real, but one of our difficulties is that there is no literature on this task to which to compare our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is straightforward and pretty easy to follow.\n- The problem is well-motivated since they illustrate with clear experiments that Transformer-based models can identify syntactic change.\n- The experiment was conducted both at the word *and* sentence level and makes the paper really interesting from an anecdotal perspective\n- Section 6 is really interesting and I was left wanting more (see reasons to Reject)", "reasons_to_reject": "- While the experiments are clear, well conducted and all sound, I feel like it is missing a concrete NLP application (especially for a NLP application track). See \"questions for the authors\" section.\n- The manual evaluation lacks details and should be further expanded so that we can easily replicate the authors' experiment. Also, it has been mentioned in the limitations so not really impacting here, but it is at a really small scale.", "questions_to_authors": "- Will the code and data be released? I did not see any mention of it.\n- Section 3 is a bit confusing. The section starts with a paragraph saying that you train POS taggers, then later on we switch to classifiers. Should it be sequence taggers throughout the paper? Then we get into the training of MLM, and finally, the classification task using the learned representation. If the authors could make this section a bit clearer, we wouldn't need to read it over and over to wrap our head around what exactly is going on.\n- Section 5.3: The authors expose the results from this experiment but nothing is said. What do these results tell us?", "paper_topic_and_main_contributions": "This paper proposes a new way to detect syntactic changes between English texts from 1830 and the 1900s using the BERT model.\nThe paper has several contributions;\n- The authors show that a fine-tuned version of BERT can distinguish texts from different periods using only syntactic information by performing different transformations, which validates the need for the next contributions\n- They use a fine-tuned version of BERT to identify instances and words that are subject to syntactic change over time.\n- They validate the historical rise of progressive construction using their method\n\nThe authors conducted a series of automatic and manual evaluations which is highly welcomed.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 199: Is it from 1830 instead of the 1900s?\n- Section 6: Would it be possible to have an example of progressive vs non-progressive construction? I think it'll improve this section, which in my opinion should be further emphasized."}}]

Author rebuttal:

Meta-review: The authors check the syntactic change in the UK parliament data from 1830 to 1990. They fine-tune a PLM to detect the syntactic change and also train their own POS tagger, which they compare to the fine-tuned PLM. The main contribution of the paper uses a hybrid tagger, a tagger that uses embeddings that have been shown to understand old English.
Decision: Accept-Findings