An Application of Pseudo-log-likelihoods to Natural Language Scoring
Authors: Darren Abramson
Ali Emami
Venue: ICLR 2022 Submitted (2022)

Abstract: Language models built using semi-supervised machine learning on large corpora of natural language have very quickly enveloped the fields of natural language generation and understanding. In this paper we apply a zero-shot approach in- dependently developed by several researchers now gaining recognition as a significant alternative to fine-tuning for evaluation on common sense tasks. A language model with relatively few parameters and training steps (albert-xxlarge-v2) compared to a more recent language model (T5) can outperform it on a recent large data set (TimeDial), while displaying robustness in its performance across a similar class of language tasks. Surprisingly, this result is achieved by using a hyperparameter-free zero-shot method with the smaller model, compared to fine-tuning to the larger model. We argue that robustness of the smaller model ought to be understood in terms of compositionality, in a sense that we draw from re- cent literature on a class of similar models. We identify a practical cost for our method and model: high GPU-time for natural language evaluation. The zero-shot measurement technique that produces remarkable stability, both for ALBERT and other BERT variants, is an application of pseudo-log-likelihoods to masked language models for the relative measurement of probability for substitution alter- natives in forced choice language tasks such as the Winograd Schema Challenge, Winogrande, CommonsenseQA, and others. One contribution of this paper is to bring together a number of similar, but independent strands of research. We produce some absolute state-of-the-art (SOTA) results for common sense reasoning in binary choice tasks, performing better than any published result in the literature, including fine-tuned efforts. In others our results are SOTA relative to published methods similar to our own – in some cases by wide margins, but below SOTA absolute for fine-tuned alternatives. In addition, we show a remarkable consistency of the model’s performance under adversarial settings, which we argue is best explained by the model’s compositionality of representations.

Under review as a conference paper at ICLR 2022
AN APPLICATION OF PSEUDO-LOG-LIKELIHOOD TO NATURAL LANGUAGE SCORING
Anonymous authors Paper under double-blind review
ABSTRACT
Language models built using semi-supervised machine learning on large corpora of natural language have very quickly enveloped the fields of natural language generation and understanding. In this paper we apply a zero-shot approach in- dependently developed by number of researchers now gaining recognition as a significant alternative to fine-tuning for evaluation on common sense tasks. A lan- guage model with relatively few parameters and training steps (albert-xxlarge-v2 compared to more recent language model (TS) can outperform it on a recent large data set (TimeDial), while displaying robustness in its performance across similar class of language tasks. Surprisingly, this result is achieved by using hyperparameter-free zero-shot method with the smaller model, compared to fine- tuning the larger model. We argue that robustness of the smaller model ought to be understood in terms of compositionality. in a sense that we draw from re- cent literature on a class of similar models. We identify practical cost for our method and model: high GPU-time for natural language evaluation. The zero-shot measurement technique that produces remarkable stability, both for ALBERT and other BERT variants, is an application of sontd-loo-likelhhoooe to masked lan- guage models for the relative measurement of probability for substitution alter- natives in forced choice language tasks such as the Winograd Schema Challenge, Winogrande, CommonsenseQA. and others. One contribution of this paper is bring together number of similar, but independent strands of research. We pro- duce some absolute state-of-the-art (SOTA) results for common sense reasoning in binary choice tasks, performing better than any published result in the literature, including fine-tuned efforts. In others our results are SOTA relative to published methods similar to our absolute for fine-tuned alternatives. In addition we show a remarkable consis- tency of the model's performance under adversarial settings, which argue is best explained by the model's compositionality of representations.
in some cases by wide margins, but below SOTA
INTRODUCTION
Computational linguistics has made major strides in the adoption of machine learning techniques applied to unstructured corpora consisting of human generated natural language text. For exam- ple, some methods take advantage of the frequencies of words in natural human text to productive ends (Collobert al. 2011; Mikolov al., 2013a b] Peters et al. 2018). N-gram models providing frequencies of pairs, triples, etc. of words in natural text provided further gains on related tasks. However, very influential paper in 2018, signalling major shift in the application of machine learning to natural text, advocated for an architecture that has "a more structured memory for han- dling long-term dependencies in text, compared to alternatives like recurrent networks, resulting in robust transfer performance across diverse tasks." (Radford et al., 2018) This culminated in the application of the Transformer (Vaswani et al. 2017) to the creation of representations through lan- guage prediction tasks, motivated by the importance of long-term dependencies in natural language text for not only the choice of model, but also training data; as Radford et al. note, "Crucially, [BooksCorpus], common corpus for multitude of emerging transformer models, contains long stretches of contiguous text, which allows the generative model to learn to condition on long-range information."

---

Under review as a conference paper at ICLR 2022
Why might 'long stretches of contiguous text', via learning conditioned on that text, lead to success at diverse tasks like natural language inference, question answering, sentence similarity, and classifi- cation (Radford et al. 2018 Table 1)? After all, these tasks typically involve very short, independent sections of text.
Solving the Winograd Schema Challenge (WSC) Levesque al. (0122) seems to require vast amounts of common sense knowledge, and the job of learning long-term dependencies was sup- posed help replacing actual knowledge of the world with the proxy knowledge that human- generated text provides. Although language models do well at common sense benchmarks through fine-tuning, when we evaluate them using standard fine-tuning methods with a small, admittedly un- reliable qqick-probe', they do not generalize well to new samples that we offer. On the other hand, a recent zero-shot technique using an idiosyncratic model with several unique architectural and train- ing features shows remarkable consistency and absolute performance on our unreliable quick probe, but also on a family of challenging common sense problems.
1.1 SUMMARY OF CONTRIBUTIONS:
In this paper we investigate the properties of a language model with parameter sharing: a albert-xxxl a arge-v2, small in both parameter count and pre-training corpus relative to the field of language models generally. We find that soaddo-lo--ikelihoood (PLL) and ooken-normalied PLLs (NormPLL) methods for scoring natural language with this model performs at mixture of outright otate-of-the-art (SOTA) performance and robust, but SOTA performance just for zero-shot methods at series of recent binary common sense language tasks. The combination of model and method remarkable consistent, scoring around 75-80% under conditions both designed to be adversar- ial against language models. The approach is also robust against accidental processes that reduce zero-shot performance in language models generally, such as semantically and syntactically noisy data.
To our knowledge, our results are SOTA for any approach to the TimeDial (Qin et al. 2021 dataset; SOTA for any zero-shot approach to solving the train-xl split of Winogrande (Sakaguchi et al.) 2020]: SOTA for an average score on the perturbed Winograd set (Abdou et al. 2020): and, SOTA for any zero-shot approach to WSC, with the exception of reported result in which training and testing sets were mixed. In other cases, our approach is SOTA for zero-shot and competitive with fine-tuned approaches. We provide explanation for the results and their significance.
RELATED WORK
2.1 BIDIRECTIONAL VS. UNIDIRECTIONAL. MODELS
The two most recent GPT papers, 'Language Models are Unsupervised Multitask Learners' (Radford) al. 2019) and 'Language models are few-shot learners' (Brown al. (2200) identify their titles the nature or purpose of machine learning models for language with the purposes they put their GPT variants to in their paper. Emphatic titles aside, the most influential fine-tuning papers also advocate for few- and zero-shot results. A more important differentiator between GPT style and NormPLL-suitable models is the significant benefit of a bidirectional masked objective for success with PLL scoring methods over single-directional masked objectives, as Salazar et al. (2000) Zhou ct al. 2020), and Ma et al. (202I) show.
2.2 THE "QUICK-PROBE ASSUMPTION
In his discussion of Winograd schemas, Dennett defines what he calls the 'quick-probe assumption': success on a few Winograd schemas in a Turing test-style evaluation ought to indicate generaliz- ability of a computer's ability to make common sense judgements. not merely success the few examples like it, or examples like it in some superficial way only (Dennett 1984).
One of us, skeptical of fine-tuning for success at tasks like the Winograd Schema Challenge and similar problems, hand-made set of 20 sentence pair prior prior to collaboration on the present paper.
https://ancnymonnn 4wwe:/uwwuwoooooooo..... sci www.ciincceeeeeeeee /ttte///Noo/OPP// intetsSSoiinninnnn ng 4620/
wi ninorravversarial es json We have reproduced the dataset in its entirety in Appendix A.2.
2

---

Under review as a conference paper at ICLR 2022
The purpose of this set of Winograd-style pairs is to test whether fine-tuning can be attacked directly, as follows.
Suppose training set contains multiple complete pairs, such that reference is shifted every time a sentence has a twin that is different only in some modifier or short phrase. Then perhaps a pair in which reference isn't shifted will be scored poorly, if the model is spuriously using the modifier trick. This can be an exploited trick (at least in principle) if, for example, one member a Winograd schema pair is in the train set, and the other is in the test set
Here is an example from this small, hand-made data set:
1. This why people are supposed to take salt tablets when < mask > sweat a lot. Answers: people, salt tablets
2. This is why people are supposed to take salt tablets when mask > sweat little. An- swers: people, salt tablets
By substituting the answers for the mask above we get two pairs of sentences for model to score, or assess the relative likelihood of, resulting in two questions of the suitcase/trophy example above. The correct answer above for both examples is 'people", since salt tablets don't sweat.
Model
Fine-tuned
Zero-Shot
BERT
45%
53%
RoBERTa
50%
60%
ALBERT
55%
65%
DeBERTa
50%
55%
Table 1: Performance of various transformer models (large versions), Fine-tuning performed on Winogrande.
In Table we compare the performance of variety of models that have been fine-tuned on the Winogrande, a scaled WSC-variant debiased against RoBERTa (Sakaguchi et al.) 2020). We find that the BERT family of language models generally does poorly this data set when evaluating its fine-tuned discriminator on the data set. On the other hand, using a method of scoring sentences using language models in manner which is free of hyperparameters, in the second column, there is no training beyond the objective functions of the models during semi-supervised pre-training.
also
the models
Notice that a single model outperforms the others: a Albert-large The a a bert peart-xxx xx arre-v2 variant scores an impressive 80% on the Winogradversariall dataset we present. It is a well-defined question to ask whether this high value for that last variant is statistical fluke, or evidence of robust ability to score binary common sense sentence pairs at a rate of around 80%.
An anonymous reviewer points out that, on 20 coin flips, there is a greater than 25% chance of achieving more than 12 heads, or 60% accuracy. Therefore the results of Table 1 are not partic- ularly meaningful. We agree: these results are not particularly meaningful, by themselves. This paper argues on the basis of new results on very large binary choice and similar data sets that the 80% score achieved by the ilbert-xxlarge-....... -xxlaxgeeeee22 is due its compositionality of representation and corresponding systematicity of behaviour. We also cite independent research that supports our interpretation of our results.
2.3 HYPERPARAMETERS AND ZERO-SHOT
A broad survey of machine learning research concludes that demonstrating an ability to innovate in model construction dominates work done in data set collection and cleaning (Sambasivan al.) 2021). Resource constrained researchers are able to use platforms like huggingface to leverage pre- training with novel models contributed by other researchers. PLLS applied to language models for common sense tasks present both opportunities and challenges distinct from this standard approach
TThis is in fact turns out to be the case in the WNLI dataset which is part of the general natural understaning benchmark of SuperGI.UE Wang et al. (019)

---

Under review as a conference paper at ICLR 2022
to distributed work in NLP. Predictive language model scoring using pre-trained deep learning model has been used since at least (Linzen et al. 2016), although as we discuss below PLLs seem to display unique benefits for architectures with ichosyncratic features such parameter sharing and bidrectionality.
Despite its nascent application, scholarly literature has already recognized availability of GPU time for researchers as limiting factor in applying NormPLLs (our term) for large language models. Laban al. (2021) explicitly limit their investigation the smallest, 'base' models of BERT and RoBERTa in their published results. As we demonstrate below, ALBERT requires an order of mag- nitude more GPU time for NormPLL scoring than BERT, but we nevertheless provide results for number of important data sets using the 'xxlarge' variant of ALBERT.
In Appendix C we compare the approach we share here wide range of common sense tasks, including COPA (Roemmele et al. (011). The website associated with the COPA dataset contains an ethics injunction for its users with specific imperatives: researchers should not peek at the data set before they evaluating their model on it; and, researchers should only evaluate their method on COPA once[ zero-shot method scores an impressive 80% on COPA; as we argue below, sensi- tivity of the method to even extra spaces around punctuation marks necessitates a certain amount of familiarity with the data.
We critical views of fine-tuning in industry. A recent white paper eschews fine-tuning and even few-shot evaluation for assessing the representational quality of a natural language model because of their potential for spurious results Research parallelism as a consequence of the number of hypotheses tested It is beyond the scope of this paper, but there are number of methods, such as Bonferroni correction, that can be used in settings of multiple hypothesis testing. Regardless of one's priors for the compliance of fine-tuning of language models by the research community with statistical best practices, one may find zero-shot measurements of language models more reliable simply because of the fewer points of possible p-hacking, intentional or otherwise.
produce spurious results simply
METHODS AND RESULTS
3.1
METHODS
Here we describe three recent papers that all use some form of PLL NormPLL, recently published, that do not cite one another this speaks to a large community of focused and determined researchers wide variety of private and public settings. We employ the codebase of the first two approaches in preparation of our own results. For brevity we refer the reader to the papers mentioned below for an explanation of the algorithms.
3.1.1 MLM-SCORING
We first became aware of PLL scoring using language models via Salazar et al. (2020), and to our understanding their arxiv submission of that paper in late 2019 is the first treatment of the approach in the machine learning literature, although we acknowledge that the vast literature is growing ever more quickly. Much of our scoring is performed using the codebase associated with the paper One key advantage of this codebase is that its use of mxnet means that scoring of individual sentences is efficiently shared among multiple GPUs available. A minor disadvantage is that, in our experience on managed academic computing platform, is that package compatibility was harder to achieve.
Salazar et al. (2020) style scoring is reported with GPU time for evaluation on an academic comput- ing node with the following characteristics: 32 cores, RAM of 187G 192000M, Intel Silver 4216 Cascade Lake @ 2. 1GH x 480G SSD, and 4 GPUs, all NVIDIA V100 Volta (32G HBM2 memory).
3
*See wtt.eyoooooottteeee...eeee//o..o
5 For an casy to
scc
"See https://github.coo
4

---

Under review as a conference paper at ICLR 2022
Notably the Salazar et al. (2020) paper treats many topics of interest in machine learning related to language, but does not examine PtL-style scoring on any common sense data sets.
3.1.2 CATS SCORING
Zhou al. 22200) exclusively focuses on the application of NommPL--styl scoring common sense data sets. What we call the NormPLL algorithm is the Salazar et al. (2020) pseudo-log- likelihood scoring method, but dividing scores by the tokenized length of the expression. We had already completed a number of experiments when finding this codebase, and had already consid- ered the concept of normalization over tokenized length. When comparing Winograd-style pairs, substitutions are usually of similar length but they may not be.
advantage of this codebase ii that it can run in any environment that supports the huggingface and pytorch packages. A disadvantage of this approach is that there is no built-in parallelization across multiple GPUs available; however, because the NormPLL algorithm involves summing over multiple forward passes of language models, it is well-suited to standard MapReduce-styl parallelization.
3.1.3 ZERO-SHOP SHOT WITH HYPERPARAMETERS
Ma et al. (202)) present NormPLLs also under a scoring term (see their SMTM(()) definition) and then augment their performance by providing language models with additional mask-filling training on instances of common sense judgements with tags. It is interesting note that this results in the presentation of zero-shot results that are qualified with a '95% confidence interval'. Below we compare some of their results with NormPI.Ls using albert xxt a rge--v, with a fuller picture available in Appendix C.
3.2
STATE OF THE ART WITHOUT FINE-TUNING
3.2. PURELY WINOGRADVERSSRRIN DATASETS
We consider the performance of variety of models on data sets that, with no more than light prepro- cessing, provide pairs of sentences that are labeled according to super-majority human judgement of common sense, with exactly one right answer per pair.
In a forthcoming paper (anonymous) we demonstrate, using NormPLLs with a llbertxxxlaageeee.
6.5% average improvement over the best zero-shot scoring pre- sented in Abdou et al. (2020) over their 'perturbed' Winograd schemas. The perturbed schemas are explicitly designed to reveal the brittleness of language model performance on common sense tasks. We briefly report here pertinent results. Aug Acc- and absolute score for RoBERTa both improved significantly. The AugãAcc- for RoBERTa perturbed fell from worse than human better than human; absolute accuracy improved around 4%. ALBERT provided a higher average score than Abdou et al. (2020)'s 's best reported average score by 11%.
These surprising results prompted further investigation. The code made available with Zhou et al. (020) makes it trivial task (given GPU time) extend their implementation (see their SSore(()) defimition) of NormPLLs for the suite of tasks they provide. They test variations and sizes of GPT, BERT, XLNet, and RoBERTa. Table / reproduces the best scoring NormPLL and Human metrics from their table along with new results for a albert-xxlarge--t
Note our absolute improvement over their average score by substituting the computationally expen- sive per parameter ALBERT for RoBERTa. RoBERTa has approximately 50% more parameters and 10 times as much training data as ALBERT. These findings are out of step with the prevailing narrative that bigger is better for both language model size and pre-training corpuses.
Table contains results using PL.LS on variety of language models for the Winograd Schema Challenge data set (Levesque et al. 2012). In these data sets tokenized lengths tend to be similar across sentence pairs, and in these experiments we did not normalize scores when evaluating models. This data set is unusual in that every example contains the name of its author, researchers associated
"Se http:::///ithub..c/////aaaaaa
5

---

Under review as a conference paper at ICLR 2022
CA
WSC
SM
SMR
SWAG
HellaSwag
ARCTI
ARCT2
Average
roberta-large
0.962
0.694
0.792
0.512
0.769
0.5
0.606
0.599
0.679
albert-xxlare--2
0.972
0.798
0.733
0.571
0.789
0.553
0.493
0.554
0.701
HUMAN 0.993
0.920
0.997
0.975
0.880
0.945
0.909
0,909
0.945
Table 2: Comparison of albert pobeertxxxlargee v2 to best reported model in Zhou et al. Scored using the Zhou et al. (2000) method.
2020)
Zero-shot model
grad
grande
Model size
GPU time
xlm-mlm-17- gpt2-345m
55,44
52.03
3.41
1.IGB
04:36:56
57.19
56.40
0.79
1.4GB
03:24:50
bert-large-cased
-amm wwwm
65.97
57.32
8.65
1.2GB
02:55:28
roberta-large albert-xxlarge-vi 79.64
76.84
70.77
6.07
1.3GB
03:58:21
74.82
4.82
851MB
15:30:23
albert-xxlarge-v2
81.05
76.71
4.34
851MB
17:38:25
Table 3: PLL zero-shot performance on Winograd (Levesque al., 2012) and Winogrande (train- xl) (Sakaguchi et al. 2020) data sets for number of recent large language models. We have sorted by Winograd scores, ascending. Model size in Pytorch .bin from http:///uggingfcc.commmmm colmoooss. Scored using the Salazar et al. (2020) library.
with the authors. It also contains results for the train-xl split of the Winogrande data set (Sakaguchi al. 2020) containing over 44k crowdsourced Winograd schema-style examples.
The train-xl split contains 64,505 sentence comparisons. Each comparison involves scoring two sentences, and the model is scored correct if the higher scored sentence is labeled correct. This results in under 1 seconds of node time per row, or slightly under 0.5 seconds per sentence. This is slow by machine standards, but not slow by human standards. In each row we indicate the difference in score of given model for the two data sets.
We are not aware higher zero-shot score on this Winogrande split. Notice that the value reported in Appendix C for the "WG' column are reported for the much smaller development set from Wino- grande. We are aware of a higher zero-shot score for the Winograd Schema Challenge data set in Brown et al.] (2020)- 88.3*,, but that value is asterisked by the authors of that paper because they demonstrated that the web crawled pre-training corpus for GPT-3 is contaminated by portions the WSC dataset.
3.2.2 A RECENT (ALMOST) WINOGRADVERSARIAL DATASET: TIMEDIAL
Each row of the TimeDial dataset (Qin et al. 202 contains four substitutions into given sentence, two which are right and two which are wrong. The term '2-best accuracy' is defined such that given row is marked correct iff the scores for the correct substitutions than the highest scored incorrect substitution. In their paper, the authors describe the best fine-tuned performance for TimeDial on pre-trained model, achieved with T5 (first row of Table 4 as low as to question whether fine-tuning is viable approach to the reasoning over temporal dialogs. Note that our zero-shot approach improves absolute accuracy over their fine-tuning results with about one-third fewer parameters, two orders of magnitude less pre-training data, and no fine-tuning.
both scored higher
Table shows scores for a number models on the TimeDial data set that includes common sense judgements about the reasonability of judgements about time. Because the text data for these examples are so large, we artificially limit the pool to examples for which both scored passages are less than 450 tokens long once tokenized. This reduces the set by about 5%; in future work, methods like the ones used by Ma et al. (202 can be used to approxiate full NormPLLs for sections of text larger than can be scored on 32GB GPU; simple windowing is also a solution. Notice the significant increase in run-time for albert-xxlargee
due to its parameter sharing; at run- time, parameters are 'unrolled' across larger network than the size disk would suggest. Using NormPLL with a albert wwwettxxxaageeeeee produces a score on TimeDial that is, so far as we know, absolute SOTA -even when compared to the best fine-tuned model.

---

Under review as a conference paper at ICLR 2022
Model
2- 2-best Accuracy
Model size
GPU time
T5-large generation
0.748
2.75GB
unknown
laarr-aaseedwweeeeel whole-word kws albert-xxlarge-v2. kws albert-xxlarge v2, not kws
0.620
1.2GB
01:23:32
not kws
0.619
1.2GB
01:24:23
0.752
851MB
09:19:02
0.761
851MB
07:52:56
Table 4: PLL zero-shot performance TimeDial data set (Qin et al., 2021 for num- ber of recent large language models, with highest fine-tuned score from that paper in ital- ics. Scored using the [Salazar et al. (2020) library but with normalization by tokenized length. Dataset filtered to examples with tokenized length less than 450 tokens. Model fea- tures are reported from https///uggingffcc.co///// moddel with model size in Pytorch.bis 'kws' (short for 'keep weird spaces') indicates that the TimeDial dataset is used as original pre- sented https://zawi wtw.aawwwwihubbseecooooooo TimeDial wwa.ainnteesemeeee ison 'not kws' indicates the application of a string function to in- put that removes spaces before punctuation symbols.
com/ reweeeereeeeachh-aateeee
3.2.3 BRITTLENESS OF THE APPROACH
In Appendix C, Table 5, we provide full picture of results comparing zero-shot experiments on CSR benchmarks with large and extra large versions of ALBERT with the best performing model, our knowledge, reported in the literature corresponding to a RoBERTa-Large model trained on additional synthetic datasets drawn from a combination of knowledge bases including ATOMIC, ConceptNet, WordNet, and Wikidata from Ma et al. (202T). As can be seen, our zero-shot shot language model approach achieves best results for binary choice tasks, but peforms less well than their ap- proach, which augments language models with in-domain learning. For multiple choice questions with unique answer among more than two options. approach is inferior. Some data sets, such as COPA, can be rendered into two candidate sentence form: in this case, our performance is similar to binary choice problems.
Important findings that we highlight are that the while our experimentation demonstrates that with- out any additional data knowledge source (which in itself would have invite an opportunity for multiple experimentation, even in the zero-shot regime, i.c. multitake) ALBERT pre-trained only on its original pre-training corpora achieves SOTA number of the CSR benchmarks (e.g.. WSC, Winogrande, HellaSwag), performs competitively (but sightly worse) on others, and is yet outper- formed by large margin on few others, the most noticeable of which is on SIQA (-14.89%).
3.3 A TALE OF THREE WINOGRADS
Here we draw attention to three quantities that should, abstractly, be identical, but are instead differ- ent. The Winograd Schema Challenge is public dataset that is currently available in xml format on the open web 8 Visiting this site in modern browser such as Google Chrome results in a nicely formatted series of questions, reproduced in Appendix D. On the other hand, by 'viewing the source' of the rendered xml, a different representation can be seen making certain features more obvious of the dataset, also reproduced there.
The second representation makes more clear that there extra white space in the strings for some fields but not others; in some cases there is extra white space at the front, but not back of string. Also, there are initial capitalizations in the two answer fields that won't be appropriate when substi- tuted for the pronoun so to complete scoreable sentences.
Now consider the question: how does the a werbbeetxe wott-xxlareee xxxaarge-22 perform on the data set pre- sented in these figures? Consider table [ 0.798. In (anonymous), using Abdou et al. (2020)'s 's presentation, it is 0.796. Finally, according to table 3 it is 0.810. These scores are all suppos- edly produced using the same method on the same data set. The roberta-las large scores are, respectively, 0.694, 0.708, and 0.768.
*See htthh:ttt////t nyu. http://p....bb.aaaaaaa/appee///e http:///wwaaaaaaappeeee/////aa/a lec foo..m

---

Under review as a conference paper at ICLR 2022
Here is the source of the discrepancy. The highest scores in both cases, table correspond to PLL scoring on Winograd schema challenge data that we have provided a single python scrip for downloading from its public location the Web, and then provides some explicit cleaning and concatenating to produce two individual sentences. The other two scores are produced using pipelines from Zhou et al. (2020) for table and Abbdo al. (2200) for the perturbed results cite using our method.
Those pipelines included preprocessing of the xml into other formats that be inspected via the repositories for those papers 10 It is to the credit of the authors of both papers that their pipeline has been made public, including the parts. Thanks this transparency, we can report problems with both datasets.
The Zhou al. (2020) Winograd Schema Challenge data for table 2 contains what we call here 'weird spaces'. These are expressions such as care of john John In addition, it con- tains numerous odd concatenations, such as Xenophanesconveyaa Finally, it lower cases some proper names, likely in trying deal with leading Thes in answer fields, but not others. The Abdou al. (2020) data is entirely lower cased. The codebase does not provide the final sentence as used for model scoring, but inspecting the jsenl reveals many extra spaces before punctuation marks.
4 DISCUSSION
4.1 RECENT WORK ON COMPOSITIONALITY
Interest in the problem of compositionality has been reinvigorated in the context of the advancing capacities of neural networks for language. [Baroni (2020) and Russin al.) (202T) lay out exis- tence proofs. providing clear evidence of the learnability of compositional syntactic and semantic domains. Ontañón et al. (202T) go further, and for series of synthetic tasks that strongly benefit from compositionality, such arithmetic, they perform ablation experiments across number of features of modern Transformer architectures, notably for parameter sharing: an unusual feature of ALBERT.
Ontañón et al. 2021 conclude that weight sharing (sometimes called 'parameter sharing', as in that adopted by the Transformer- based model of ALBERT (Lan et al. (2019)) is, alone, design decision that "significantly boosts compositional generalization accuracy, and almost all models [with weight sharing] achieve a higher average accuracy across all datasets than their equivalent models [without weight sharingl... "oaigl...."(Otttiiiiii "OOntañón et al. 2021 6) 11 This use of 'compositionality has taken over the meaning of 'systematicitty' referring to behavioral consistency instead of representational form. Generalization accuracy across binary choice common sense natural language tasks as we have seen across a narrow range for -lbert-xxlarge-- v2 of about 76%-80% might be partly explained by this result for synthetic data sets.
How might bidirectional transformer encode generalizable language knowledge? Some recent probes of linguistic generalization measures whether BERT's layers can be interpreted as, to some degree, representing the knowledge expressed by Penn Treebank parse trees (Clark et al. 2019). (Hewitt & Manning 2019). Another approach offering a metric for assessing parse trees and local- izable activation in BERT claims in its title that the model 'Rediscovers the Classical NLP Pipeline'
Tenney et al. 2019).
These approaches have self-acknowledged limitations. Clark et al. (2019) and Tenney et al. (2019) both point out the poor performance of BERT coreference resolution. Hewitt & Manning (2019)
highlight that the evidence provided by syntactic probes are neither necessary nor sufficient for finding linguistic representations and their in downstream behavior. "For this reason, [they]
"emphasize the importance of combining structural analysis with behavioral studies... to provide a
wwwp::/uuuuuooe...iiiii..oo.sece A http://///aaaaoooo///sssssss///a www.aanniiinnnd.....p. com/ test/ and
wac/b1 bta.b//ebbrreeee/
aatta//aaase/eenhaane _acpcced_ds______ee sas__ssone wwwpnn/ttettttrgiiiiiee jsonl
"This paper was released after we had completed the vast majority of our experiments.
8

---

Under review as a conference paper at ICLR 2022
more complete picture of what information these models encode and how that information affects performance on downstream tasks." We find their motivating insight reminiscent of Dennett [191]). and endorse the need for behavioral studies that identify generalizable linguistic abilities in language models; structural probes are necessarily incomplete.
Winograd schemas notable in that many examples involve the production of sentence that alters parse trees on the basis of a semantic change in a single word. Consider the reference of "she' for the choice of good/bad in the following (Levesque al.) 2012): Although they ran at about the same speed, Sue beat Sally because she had such a good/bad start. These schemas, given robust human performance, suggest that a knowledge of syntax that is separate from world knowledge may be possible for human language, as suggested by [Miller (1999). Winograd schemas belong to class of coreference resolution problems. Through PI.L scoring with a lobert-xe peprrt-xx xx la rge-v2, we have presented a robust body of behavioural evidence that fewer parameters can produce more consistent coreference resolution behaviour than previously recognized.
4.2 ALBERT's UNIQUE ARCHITECTURAL FEATURES
One important consequence of the design decision to incorporate parameter sharing into trans- former architecture is that it trades parameter size for computation time. In other words, parameter sharing yields representations that are space efficient relative to other models, but time inefficient. At least some researchers have recently argued, though, that time is a resource that ought to be traded for accuracy benefits (Banino et al. 2021
In addition to a bidirectional masked language objective, like BERT descendants, ALBERT also has sentence order prediction task. This binary categorical objective function is more difficut than the original BERT sentence order prediction task which, as has been widely noted, reduces to topic good approximations. ALBERT's sentence
modeling, for which bag-of-word representations order prediction task corresponds to the problem of determining, for a pair of consecutive sentences, which comes first in the source text and which comes second. Consider the following pair of sentences chosen randomly from Wikipedia:
"Audrey Henshall was born in Oldham, Lancashire in 1927[2] and studied the Uni- versity of Edinburgh, graduating with an MA in 1949."
Sentence I:
Sentence 2: "From 1960 to 1971 she the Assistant Keeper of Archaeology the National Museum of Antiquities of Scotland."
One method to identify that Sentence 2 comes after Sentence I and not vice-versa is simply the presence of date information. The English language groups together many different causal relations and human experiences into physical metaphors, as has long been noted in the cognitive science literature (Thelen) 1995). The evidence suggests that ALBERT is an architecture that both excels the formation of compositional representations, but was also trained with an objective function that encourages learning of asymmetric relations, such as 'before"; furthermore, those relations are implicated across multiple domains of human activity.
5 CONCLUSION
The remarkable consistency of ALBERT's performance on Winogradversarial, TimeDial, WSC, and Winogrande datasets is a point of optimism for the generalizability of the performance of language models. A limitation of the current approach is that the robust performance seems be limited to cases in which common sense judgement can be expressed as the relative likelihood of two natural language alternatives, a promising avenue for future work.
We emphasize the improvement in both computational efficiency and accuracy that is effected for the TimeDial dataset by cleaning punctuation so as to more closely match normal human conventions. The difference between Grad and Grande (Table 3) across language models measured through PLLS provides early, incomplete evidence for the hypothesis that crowdsourcing common sense data sets produces measurable decline in data quality. The findings of this paper support the view that attention and care to data is as important as model innovations in machine learning generally, despite academic and industry practice not always matching this ideal (Sambasivan al. 2021).

---

Under review as a conference paper at ICLR 2022
6 REPRODUCIBILITY
We have prepared a public repository with the files that generated our results tables in .svv form, the .esv scored tables, and scripts read scores from .esv files.
The repository is available publicly at
REFERENCES
Mostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov, Desmond Elliott, and Anders Søgaard. The sensitivity of language models and humans to winograd schema perturbations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 7590-7604, 2020.
Andrea Banino, Jan Balaguer, and Charles Blundell. Pondernet: Learning to ponder. CoRR, abs/2107.05400/ 2021.
05407
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. Philosophical Transactions of the Royal Society B, 375(1791):20190307, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look an analysis of bert's attention. CoRR, abs/1906.04341, 2019. URL /RLLattp///axxiv.rrg
abs/ /0s//106... 04341
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ARTICLE):499- -2377, 2011.
Daniel C Dennett. Real patterns. The journal of Philosophy, 88(1):27-51, 1991.
DC Dennett. Can machines think? in (m. shafto, ed) how we know, 1984.
John Hewitt and Christopher Manning. A structural probe for finding syntax word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129 4138, 2019.
Philippe Laban, Luke Dai, Lucas Bandarkar, and Marti A. Hearst. Can transformer models mea- sure coherence in text: Re-thinking the shuffle test. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference Natural Language Processing, ACL/JCCNP 2021, (Volume 2: Short Pupers), Virtual Event, August 1-6, 2021, pp. 1058-1064. Association for Computational Linguisties, 2021. doi: 10.18653/v ach-srr 134. URL
http:::/oob..crr/0.118/////////2 www.::222.lllloo.....ao.. 134
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: /909. 1942, 2019.
Hector Levesque, Emest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- feenth International Conference the Principles Knowledge Representation and Reasoning. 2012.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the ability of Istms learn syntax- sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521 535, 2016.
10

---

Under review as a conference paper at ICLR 2022
Kaixin Ma, Filip llievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, and Alessandro Oltramari. Knowledge-dris wewledge-driven data construction for zero-shot evaluation in commonsense question answer- ing. In 35th AAAI Conference Artificial Intelligence, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint urXiv:1301.3P1 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 3111-11111111 2013b.
George A Miller. On knowing word. Annual review of psychology, 50(1):1-19, 1999.
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve compositional tasks. arXiv preprint wwrXiv2210..43337 2021.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HILT, pp. 2227-2237, 2018.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. Time- dial: Temporal commonsense reasoning in dialog. arXiv preprint (ee:210060457// 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series, 2011. 1.
Jacob Russin, Roland Fernandez, Hamid Palangi, Eric Rosen, Nebojsa Jojic, Paul Smolensky, and Jianfeng Gao. Compositional processing emerges in neural networks solving math problems. arXiv preprint arXiv:2105.088:: 08961, 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge scale. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 773--8740, 2020.
Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2699-2712, 2020.
Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. "everyone wants to do the model work, not the data work": Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-15, 2021.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceed- ings of the 57th Annual Meeting the Association for Computational Linguistics, pp. 4593-4601, 2019.
Esther Thelen. Time-scale dynamics and the development of an embodied cognition. Mind as motion: Explorations in the dynamics of cognition. pp. 69-100, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel R Bowman. Superglue: stickier benchmark for general-purooss language un- derstanding systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3266-3280, 2019.

---

("sentence": "Jordan wanted to appear nice to Jim so <mask> offered some breath mints", "option1": "Jordan", Under review as a conference paper at ICLR 2022
Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. Evaluating commonsense in pre-trained language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9733-9740, 2020.
A APPENDIX A WINOGRADVERSARIAL. AN ADJECTIVE FOR CRAFT DATASETS
A.1 WINOGRADVERSARIAL DATA FORMAT
Winogradversarial instance is defined formally as a tuple P= {S, C1, C2 K), where S is the sentence with masked token representing missing coreferring antecedent, C1 and C2 the candidates for the correct antecedent for the masked token and K indicates the correct antecedent. Note that both C1 and C2 appear in S. (,,C,, C2) are provided input for models, which must predict K (e.g., as the output of a binary classification over C1, C2). Additionally, for every sentence there is another sentence the set, S' that differs by by single word, also known as the switch word as in the original Winograd Schema Challenge. (i.e., S has some word w, while S' is identical S except w is replaced by another word, w' Crucially, while in the WSC, the modifying of by replacing word w by w' switches the correct label, in Winogradverserial, the correct label remains the same for both S and S". This may turn out to be adverserial to models that may be conditioned to switch their prediction decision simply based on the changing of the switch word.
Representative sentences and S' are the following, respectively, with w and w', underlined:
S =(Jordan) wanted appear nice to (Jim) so (mask) ate some breath mints.
2. S' =JJodann wanted appear nice {Jim} (mask) offered some breath mints.
Here, C Jordan, C2 Jim, K C2 Jordan, ate, and w/ offered
A.2 20 QUESTIONS
This is the complete 'winogradversarial dataset we used to generate the hypothesis for this paper. We have inserted line breaks here sentences to preserve readability.
sentence";
"Jordan wanted to appear nice to Jim so <mask> ate some breath mints", "option1": "Jordan",
"option2"; "Jim",
"answer"; "1";
"option2": "Jim",
"answer": "1";
("sentence"; "I get tons of spam at both locations but gmail incorrectly identifies <mask>.", "option1": "tons of spam", "option2": "both locations", "answer": "1":
("sentence"; get tons spam at both locations but gmail correctly identifies <mask>.", "option1": "tons of spam", "option2": "both locations", "answer": "1":
("sentence"; "Sydney isn't currently better than Jack but <mask> has the potential to be.", "option1": "Sydney", "option2": "Jack",
"answer"; "1")
12

---

Under review as a conference paper at ICLR 2022
("sentence"; "Sydney isn't eurrently worse than Jack but <mask> has the potential to be.", "option1": "Sydney", "option2": "Jack", "answer": "1")
("sentence"; "Homes should be prepared for children before you have <mask>.", "option1": "homes", "option2": "children",
"answer"; "2")
("sentence"; "Homes should be prepared for children after you have <mask>.", "optioni": "homes",
"option2": "children",
"answer": "2";
""sentence; "This is why people are supposed to take salt tablets when <mask> sweat a lot.", "option1": "people",
"option2": "salt tablets", "answer": "1")
("sentence"; "This is why people are supposed to take salt tablets when <mask> sweat a little.", "optionl": "people",
"option2": "salt tablets",
"answer": "1";
("sentence"; "Psychologists theorize that people are less comfortable when <mask> are given too few choices.", "option1": "Psychologists",
"option2": "people",
"answer": "2";
("sentence"; "Psychologists theorize that people are less comfortable when <mask> are given too many choices.", "option1": "Psychologists",
"option2": "people",
"answer": "2")
("sentence"; "The lemon cake tasted better than the banana muffin because <mask> was sweet.",
"option1": "lemon cake", "option2": "banana muffin", "answer"; "1";
("sentence"; "The lemon cake tasted better than the banana muffin because <mask> savoury." " "option1": "lemon cake", "option2": "banana muffin", "answer": "1":
("sentence"; "Mark won the competition over James because <mask> was too small.", "option1": "Mark",
"option2": "James",
"answer": "2")
("sentence"; "Mark won the competition James because <mask> was too large.",
"option1": "Mark",
"option2": "James",
"answer": "2") "2";
("sentence": "I prefer to purchase the purse over the shoe because <mask> is too cheap.", "optionl": "the purse", "option2": "the shoe".
"answer"; "2") ("sentence"; prefer to purchase the purse over the shoe because
13

---

Under review as a conference paper at ICLR 2022
<mask> is too expensive.", "option1": "the purse", "option2"; "the shoe", "answer": "2"
("sentence"; "The TV is so I decided to sell <mask>.", "option1": "the TV", "option2"; "the Ipad", "answer": "1";
valuable than the Ipad,
("sentence"; "The TV is more valuable than the Ipad, so I decided to buy <mask>.", "option1": "the TV", "option2": "the Ipad", "answer"; "1";
B APPENDIX B DEBERTA
Below we provide values for various Deberta models on our benchmark suite Winogradversarial datasets.
Note the warning message below from huggingface when using documented masked language model instantiation of Deberta vI models. v2 models threw an error when called from the
DebertaForMaske.o
method.
Some weights of the model checkpoint at microsoft/ceberta-llagg
were not used when initializing DebertaPorMaskedll::
www.eeewhediii........h..... 'config',
1m Inggesiiccioos....haa.d...eeeeee weightt,,
1m www.pp/ccios......aaaaa.eeeeeeea
1m wrr.lllctooos..ilaaaaaaaaeeeeea.
'1m_
This IS expected if you initializing DebertaForMaskedll from the checkpoint of a model trained on another task or with another
architecture (e.g. initializing a BertForsegerrcclasssfiiccaa model from BertPorPreTraining model).
This IS NOT expected if you are initializing DebertaForMaskedlMM from the checkpoint of a model that you expect to be exactly identical (initializing a BertFForsequencelllsss ficattttiiiiiii model from a tertforieuancellassiictctooo model).
Some weights of DeberttaForMaskedLM were not initialized from
the model checkpoint at mwwrosoftteeeeeetteeeeaaaa
and are newly initialized:
"cls. ort.appeeeeccio.....mmmooaaaaaae
"els h pialsppprrdiitiioobbiaa..""
"cls. deeeeee....iio..eeeeeeer..rrr weighttr
You should probably TRAIN this model on down-strean task to able to use for predictions and inference.
C APPENDIX C FULL COMPARISON WITH MA ET AL. (2021)
Our preliminary experimentation reveals that, as with the data sets discussed in the main text, the presentation format of the inputs in the datasets themselves results in significant variance of results
14

---

Under review as a conference paper at ICLR 2022
(e.g., punctuation details, lower-casing, perspicuity of wording) especially for the models that were not designed with parameter sharing. This latter point suggests two important conclusions; a) the robustness of ALBERT isn't is''t only demonstrateed on syntactic perturbations of single presentation of benchmark, but it robust to presentation changes altogether in a benchmark, while other trans- former models exhibit more brittleness and b) comparisons betwen ALBERT and such models as in Ma et al. (2021) should be treated as possibly incongruous, as the former may reflect a more universal estimation of the models' performance on common-senss reasoning question regardless of presentation or choice of external knowledge source, while the latter the upper-bounded performance of transformer models when under most favourable settings.
tasks
15

---

Under review as a conference paper at ICLR 2022
Table Zero-shot, single- take evaluation results of language models that have only been exposed to pre- tee-training corpora via original objective function (i.e., without selection process based multiple experimental configurations across various commonsense tasks. Human performance as well as the best lextitmultipee-eee taple-ttkee model Ma et al, (2021) are included for reference. In the case Ma et al. (2021). multiple takes corresponded selecting from variants of Roberta- peerta--lagge pre-trained on different Knowlede ledge Graphs, and reporting the best pererrrrrfrnnege variant in the zero vero-shot setting.
WGA
WinoBias
53.75
60.0
60.0
65.56
GAP
68.68
79.48
69.88
68.6
81.25
PDP
63.75
66.25
65.0
81.25
GAP
50.19
55.78
49.46
58.82
DPR
62.1
61.0
63.5
78.9
COPA
63.8
72.4
62.2
79.6
WG
51.14
56.35
55.25
52.48
62.11
54.7
60.9
94.1
SIQA
42.12
45.24
45.7
43.19
48.31
53.5
63.2
86.9
PIQA
59.19
67.9
61.15
61.8
70.02
67.5
72.4
94.9
CSQA
29.32
44.71
44.14
34.8
52.49
48.0
67.4
88.9
aNL]
54.96
65.01
55.74
66.84
59.2
70.5
85.6
Ma-Roberta-L (multitake*)
Ma-GPT2-L (multitake*)
bert- large- uncased
xInet-larg bagg-ccased
albert-xXlarge lxlaage--2
albert- albert-large-v2 large-v2
roberta- large
Human
Model
16

---

Under review as a conference paper at ICLR 2022
D APPENDIX D BROWSER RENDERS OF WINOGRAD
city councilmen refused the demonstrators permit because they feared violence.
Snipper: they feared violence
A. The city councilmen
B. The demonstrators
Correct Answer: A Source: (Winograd 1972)
Figure An xml entry from the Winograd Schema Challenge rendered in the Chrome browser.
<achema>
<eext>
<txt1>
city councilmen refused
pernit because<//xtl>>
11 <pron> they </pron>
<txt2> feared violence.
19 </ext2>
</text>
<quote>
<quotel></quotel>
<pron>they </pron>
18
<quote>>feared violence</quote2>
</quote>
<anawers>
<answer>The city councilmen </answer>
Tanswer>The demonstrators
</ \/anawere>
<corractAnswer> </carrectinsver>
<source> (Winograd 1972) </source>
</achema>
Figure 2: An xml entry from the Winograd Schema Challenge viewed through the browser's code editor.
17


Reviews:
[{"review_id": "z_c2KyP0vN8", "reviewer": "Reviewer_X2TK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\nPresents a new adversarial common-sense reasoning dataset which does not conform to the usual “flip a word and change the answer” paired sentence paradigm, and thus may be harder for models trained on such datasets.\n\nHighlights the shortcomings of fine-tuned language models on this adversarial common-sense reasoning dataset, and the relative strength of pretrained models.\n\nUnifies multiple independent research threads on the topic of using pre-trained models for language understanding.\n\nAchieves state-of-the-art performance on the TimeDial dataset using the albert-xxlarge-v2 model.\n\n**Weaknesses:**\n\nIntroduction presents a skewed history of the field, lumping together papers focusing on tagging using feedforward networks (Collobert et al., 2011), efficient learning of context-independent word-embeddings (Mikolov et al., 2013), and learning context-dependent word-embeddings using a recurrent architecture (Peters et al., 2018), all under the vague heading of “machine learning techniques applied to unstructured corpora providing frequencies of words”. It then points to the Transformer as heralding a new era in the field, but fails to cite “Attention Is All You Need” by Vaswani et al., which was the seminal paper that introduced the Transformer architecture.\n\nOne of the purported main contributions of the paper, the new Winogradversarial dataset, contains only 20 examples, and is therefore unlikely to be widely useful (see below). It purports to address a proposed pitfall of existing WSC datasets wherein the model can “cheat” if it has seen one instance of a paired example in the training set, and another in the test set, that it can know a priori that it should invert its response. However no direct evidence of this being an issue is provided for the WSC or WinoGrande datasets. If it was truly believed to be an issue there are multiple more direct ways of handling it, such as explicitly enforcing train/test disjointness, or even removing one of the two sentences from each pair altogether.\n\nPerformance on the Winogradversarial dataset itself, as shown in Table 1, is an unreliable indicator. For a model whose “true” accuracy is p, the number of questions it gets right on the Winogradversarial dataset can be modeled as a binomial random variable X ~ Binom(20, p). Setting p=0.5 (the average performance of the fine-tuned models), this implies that the probability of getting X≥12 (i.e. observing an accuracy of ≥60%) is >25%. This is a substantial fraction of the overall probability mass, and therefore unlikely to be meaningful. \n\nIt is difficult to pin down the overall purpose of the paper. Within the two introductory sections, topics discussed include:\n\n1. Strengths/weaknesses of word-frequency based approaches\n2. Importance of compositionality\n3. Concerns around fine-tuning approaches\n4. Concerns around existing WSC datasets\n5. The grounding problem\n6. The introduction of a new WSC dataset\n7. Concerns around current incentives and asymmetries in the field\n\nWhile I understand that these topics are conceptually overlapping, I would strongly encourage the authors to narrow their focus, so that a more targeted and convincing argument can be made.\n\nAs discussed in Section 3.1, the previous work of Salazar et al. (2020), Zhou et al. (2020), and Ma et al. (2021) overlaps heavily with the present work. In fact one of the major contributions of this work, namely showing the performantness of the pretrained albert-xxlarge-v2 on common-sense reasoning datasets, merely falls between the cracks of these other papers. In particular Zhou et al. evaluated common-sense reasoning with a variety of different pretrained Transformer models, just not with AlBERT specifically.\n\nIn the discussion section, the decreased performance on WinoGrande compared to Winograd is pointed to as evidence that the crowdsourcing of this dataset led to decreased quality. This is a questionable conclusion given that in the original Sakaguchi et al. (2020) paper introducing WinoGrande, fine-tuning on this dataset prior to evaluating on the WSC data by Levesque et al. yielded dramatic performance improvements, achieving 90.1% performance, only a few percent away from human-level, and much higher than the zero-shot performance numbers provided in this work.\n\nThe state-of-the-art performance achieved on the TimeDial dataset is not especially meaningful owing to the dataset’s extreme recency. The dataset has only been publicly available for 3-4 months at the time of writing, and as far as I can tell no fine-tuned model has ever been evaluated on it. This renders the claimed state-of-the-art status less impactful than it would have otherwise been.\n\nMinor Issues:\n\n* Top of Page 2: “We are concerned that **that** the fine-tuning”\n* Bottom of Page 2: Overly-verbose footnotes. Don’t need to specify where your example names came from, don’t need to point out that people with brain damage have diminished capacities\n* Middle of Page 2: “We take the long-standing response to this argument to be a rejection of premise that only classical logic can implement compositional representations.” - meaning is unclear, consider simpler language\n* Repeated: Failure to correctly capitalize “BERT”, “RoBERTa”, “AlBERT”\n* Bottom of Page 5: Overly-verbose footnotes. The availability or lack-thereof of academic computing resources is not relevant to the present paper. It is assumed that researchers are following the established principle of not peeking at the evaluation set, this does not need to be explicitly called-out.\n* Table 2: Inconsistent presence of zeros before the decimal point\n* Table 4: T5 accuracy formatted differently than other models\n* Bottom of Page 7: “Because **they** text data for these examples are so large”", "summary_of_the_paper": "The paper makes multiple independent contributions, including:\n\n1. The addition of a new adversarial common-sense reasoning dataset dubbed “Winogradversarial”\n2. The explicit gathering together of multiple research threads all exploring the use of pretrained language models for zero-shot prediction on language-understanding tasks\n3. The singling-out of the albert-xxlarge-v2 architecture as performing especially well on zero-shot common-sense reasoning tasks, achieving SOTA on the TimeDial dataset\n\nThe overall message of the paper is that the existing framework of fine-tuning models for common-sense reasoning tasks is flawed and prone to overfitting, and that more robust results can be achieved using pretrained models in a zero-shot manner. In particular, doing so with the albert-xxlarge-v2 model achieves state-of-the-art performance on some benchmarks.\n", "summary_of_the_review": "While the paper makes inroads in a few different interesting directions (presenting an alternative adversarial WSC-style question format, highlighting advantages of pretrained zero-shot models, achieving SOTA on a recently-released dataset), the connections between the ideas are tenuous, and the individual contributions are not sufficiently impressive to stand on their own. I encourage the authors to pick just one or two of these ideas, and flesh them out in a more thorough, convincing manner. In its present form, I believe this paper should be rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eatksiMD179", "reviewer": "Reviewer_zjM9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper investigates the performance of the Albert pretrained language model on Winograd-style tasks.\nThey find that it outperforms other models on some datasets and conclude that this is due to the Albert model reusing parameters.\n\nThe analysis of different language models in a zero-shot Winograd setting is an interesting area and the paper reports some strong results for the Albert model.\n\nThe paper appears to be more of an opinion piece and lacks a clear novel contribution. \nAn existing pre-trained language model (Albert) is evaluated on existing datasets (Winograd, Winogrande and Timedial) using well-established methods (PLL, NormPLL).\nThe finding that Albert outperforms more parameter-rich models like XLM or RoBERTa can be considered unintuitive, but the analysis in the paper is not sufficient to demonstrate the cause of this performance difference.\n\nThe paper argues that the better performance of Albert is due to the parameter reuse, which supposedly allows better modeling of language compositionality.\nHowever, evaluation is only performed by comparing to other well-known pre-trained language models. Each of these has different training data, different architecture, different number of parameters and a different training objective. Each of these could be responsible for one model outperforming another on a particular dataset.\nIn order to draw further conclusions, experiments should be performed by controlling these factors and only changing one component at a time.\n\nSection 2.1 seems to criticise previous papers (\"Language Models are Unsupervised Multitask Learners\" and \"Language models are few-shot learners\") for claiming that their titles define the purpose of language models. This criticism seems unnecessary - the papers are only showing that pre-trained language models can be used in such a way, not claiming that they cannot be used in any other way.\n\nThe experiments in 2.2, using only 20 sentence pairs, are not adding value to the paper. The dataset is far to small to draw any conclusions based on it.\n\nSection 3.2.1. describes results from a \"forthcoming publication\". By reproducing the whole table in the appendix, it seems to claim credit for the same contribution in multiple papers. I would recommend anonymously citing the forthcoming paper instead.\n\nDiscarding the higher GPT-3 results due to their larger training data does not seem properly motivated. The training data was not controlled for any of the other experiments and it was not ensured that the WSC sentences did not appear in the Albert training set.\n\nThe analysis in Section 3.2.3. seems to claim competitive results compared to Ma et al. (2021). However, Table 6 shows that Albert outperforms Ma et al only on 1 out of 5 datasets, which seems to contradict this.\n\nOverall, the paper is rather difficult to follow. The narrative jumps around quite a bit, while important components like datasets, evaluation metrics and baselines are not fully introduced.\n\n", "summary_of_the_paper": "The paper investigates the performance of the Albert pretrained language model on Winograd-style tasks.\nThey find that it outperforms other models on some datasets and conclude that this is due to the Albert model reusing parameters.", "summary_of_the_review": "The analysis of different language models in a zero-shot Winograd setting is an interesting area and the paper reports some strong results for the Albert model.\n\nThe paper appears to be more of an opinion piece and lacks a clear novel contribution. \nAn existing pre-trained language model (Albert) is evaluated on existing datasets (Winograd, Winogrande and Timedial) using well-established methods (PLL, NormPLL).\nThe finding that Albert outperforms more parameter-rich models like XLM or RoBERTa can be considered unintuitive, but the analysis in the paper is not sufficient to demonstrate the cause of this performance difference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HfzjntfCR60", "reviewer": "Reviewer_wa7Y", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper seems to introduce a new dataset that could be very useful to the community.  \n\nWeaknesses:\nThe paper is generally poorly written.  Even the main claims are hard to decipher, right from the abstract.  But the entire introduction provides no clear/coherent overview/findings or motivation.  The remainder of the paper continues the same way.  This is a pity, since it seems like the authors have some interesting results to present.  They are too unclear for me to be secure in restating them.\nA new dataset is introduced  (winogradversarial), over which the main evidence of the claims of the paper are obtained, without any description of its contents beyond a couple examples and an extremely obscure description of the purpose of its examples.", "summary_of_the_paper": "The paper demonstrates some evidence that albert-xxlarge-v2's NormPLL has compositional capacity.  The paper does this by introducing the Winogradversarial dataset and documenting performance.", "summary_of_the_review": "See weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "4t58U8YwEn", "reviewer": "Reviewer_T3k9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution seem to be state-of-the-art zero-shot results on commonsense tasks obtained using albert-xxlarge-v2. However, the authors talk about concepts such as compositionality and I am not sure how this relates the main experiments in the paper.\n\n*Strength*:\n- Best zero-shot results on commonsense datasets (as claimed by the authors, I only checked a couple of references to see if that claim holds true)\n\nThe main *weakness* of the paper is that the writing is confusing, which makes it difficult to see what the main point of the paper is. The authors seem to argue for multiple statements that show up once in the paper and are then dropped or forgotten. An example from the abstract: \"We provide evidence for the following conclusion: a language model with relatively few parameters, trained for relatively few steps, can perform robustly across language tasks in a manner that demonstrates compositionality, at the cost of GPU-time for language evaluation.\" It is unclear to me where compositionality is being demonstrated in the experiments. (Maybe in the experiments with the 20 sentence pairs? But I wouldn't consider those enough evidence.)\n\nAdditionally, some of the statements made by the authors seem wrong or at least misleading:\n- The authors argue against finetuning by proving a quote from Bender and Koller (2020) on page 3, but I don't see how the quote is related to finetuning as opposed to general language models.\n- The authors state in 3.1.1: \"We first became aware of PLL scoring using language models via Salazar et al. (2020), and to our understanding their arxiv submission of that paper in late 2019 is the first treatment of the approach in the machine learning literature\" However, I believe this approach has been around much longer as a way to score pairs (or sets) of sentences. For example, Linzen et al. (2016) use effectively the same approach with LSTM language models (https://arxiv.org/pdf/1611.01368.pdf).\nI would recommend to revise those statements for future versions of the paper.", "summary_of_the_paper": "I believe that the main contribution is that the paper shows that albert-xxlarge-v2 is the best zero-shot model on the commonsense datasets (out of multiple models available in huggingface that are being evaluated). However, the authors also seem to argue against finetuning as a general approach for commonsense reasoning, though I don't see how this is backed up by the paper (except for maybe the small experiment with 20 sentence pairs?). ", "summary_of_the_review": "While the authors present new state-of-the-art results, there aren't many insights besides the fact that albert-xxlarge-v2 is a good zero-shot model. The authors could design more experiments backing up their main claims (e.g., about compositionality).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper argues several loosely-related points about the evaluation of pretrained models on commonsense reasoning datasets in the Winograd style, and presents experiments with existing models on several datasets, including a novel 20-example benchmark. All four reviewers struggled to find a clear contribution or theme in this paper that is novel and thorough enough to meet the bar for publication at a selective general-ML venue.

I'd urge the authors to focus in on just one of these points and expand, and to consider submitting to a venue that more narrowly focuses on methods for commonsense reasoning in NLP.
Decision: Reject