Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper demonstrates some evidence that albert-xxlarge-v2's NormPLL has compositional capacity.  The paper does this by introducing the Winogradversarial dataset and documenting performance.
main_review: Strengths:
The paper seems to introduce a new dataset that could be very useful to the community.  

Weaknesses:
The paper is generally poorly written.  Even the main claims are hard to decipher, right from the abstract.  But the entire introduction provides no clear/coherent overview/findings or motivation.  The remainder of the paper continues the same way.  This is a pity, since it seems like the authors have some interesting results to present.  They are too unclear for me to be secure in restating them.
A new dataset is introduced  (winogradversarial), over which the main evidence of the claims of the paper are obtained, without any description of its contents beyond a couple examples and an extremely obscure description of the purpose of its examples.
summary_of_the_review: See weaknesses.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: I believe that the main contribution is that the paper shows that albert-xxlarge-v2 is the best zero-shot model on the commonsense datasets (out of multiple models available in huggingface that are being evaluated). However, the authors also seem to argue against finetuning as a general approach for commonsense reasoning, though I don't see how this is backed up by the paper (except for maybe the small experiment with 20 sentence pairs?). 
main_review: The main contribution seem to be state-of-the-art zero-shot results on commonsense tasks obtained using albert-xxlarge-v2. However, the authors talk about concepts such as compositionality and I am not sure how this relates the main experiments in the paper.

*Strength*:
- Best zero-shot results on commonsense datasets (as claimed by the authors, I only checked a couple of references to see if that claim holds true)

The main *weakness* of the paper is that the writing is confusing, which makes it difficult to see what the main point of the paper is. The authors seem to argue for multiple statements that show up once in the paper and are then dropped or forgotten. An example from the abstract: "We provide evidence for the following conclusion: a language model with relatively few parameters, trained for relatively few steps, can perform robustly across language tasks in a manner that demonstrates compositionality, at the cost of GPU-time for language evaluation." It is unclear to me where compositionality is being demonstrated in the experiments. (Maybe in the experiments with the 20 sentence pairs? But I wouldn't consider those enough evidence.)

Additionally, some of the statements made by the authors seem wrong or at least misleading:
- The authors argue against finetuning by proving a quote from Bender and Koller (2020) on page 3, but I don't see how the quote is related to finetuning as opposed to general language models.
- The authors state in 3.1.1: "We first became aware of PLL scoring using language models via Salazar et al. (2020), and to our understanding their arxiv submission of that paper in late 2019 is the first treatment of the approach in the machine learning literature" However, I believe this approach has been around much longer as a way to score pairs (or sets) of sentences. For example, Linzen et al. (2016) use effectively the same approach with LSTM language models (https://arxiv.org/pdf/1611.01368.pdf).
I would recommend to revise those statements for future versions of the paper.
summary_of_the_review: While the authors present new state-of-the-art results, there aren't many insights besides the fact that albert-xxlarge-v2 is a good zero-shot model. The authors could design more experiments backing up their main claims (e.g., about compositionality).
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates the performance of the Albert pretrained language model on Winograd-style tasks.
They find that it outperforms other models on some datasets and conclude that this is due to the Albert model reusing parameters.
main_review: The paper investigates the performance of the Albert pretrained language model on Winograd-style tasks.
They find that it outperforms other models on some datasets and conclude that this is due to the Albert model reusing parameters.

The analysis of different language models in a zero-shot Winograd setting is an interesting area and the paper reports some strong results for the Albert model.

The paper appears to be more of an opinion piece and lacks a clear novel contribution. 
An existing pre-trained language model (Albert) is evaluated on existing datasets (Winograd, Winogrande and Timedial) using well-established methods (PLL, NormPLL).
The finding that Albert outperforms more parameter-rich models like XLM or RoBERTa can be considered unintuitive, but the analysis in the paper is not sufficient to demonstrate the cause of this performance difference.

The paper argues that the better performance of Albert is due to the parameter reuse, which supposedly allows better modeling of language compositionality.
However, evaluation is only performed by comparing to other well-known pre-trained language models. Each of these has different training data, different architecture, different number of parameters and a different training objective. Each of these could be responsible for one model outperforming another on a particular dataset.
In order to draw further conclusions, experiments should be performed by controlling these factors and only changing one component at a time.

Section 2.1 seems to criticise previous papers ("Language Models are Unsupervised Multitask Learners" and "Language models are few-shot learners") for claiming that their titles define the purpose of language models. This criticism seems unnecessary - the papers are only showing that pre-trained language models can be used in such a way, not claiming that they cannot be used in any other way.

The experiments in 2.2, using only 20 sentence pairs, are not adding value to the paper. The dataset is far to small to draw any conclusions based on it.

Section 3.2.1. describes results from a "forthcoming publication". By reproducing the whole table in the appendix, it seems to claim credit for the same contribution in multiple papers. I would recommend anonymously citing the forthcoming paper instead.

Discarding the higher GPT-3 results due to their larger training data does not seem properly motivated. The training data was not controlled for any of the other experiments and it was not ensured that the WSC sentences did not appear in the Albert training set.

The analysis in Section 3.2.3. seems to claim competitive results compared to Ma et al. (2021). However, Table 6 shows that Albert outperforms Ma et al only on 1 out of 5 datasets, which seems to contradict this.

Overall, the paper is rather difficult to follow. The narrative jumps around quite a bit, while important components like datasets, evaluation metrics and baselines are not fully introduced.


summary_of_the_review: The analysis of different language models in a zero-shot Winograd setting is an interesting area and the paper reports some strong results for the Albert model.

The paper appears to be more of an opinion piece and lacks a clear novel contribution. 
An existing pre-trained language model (Albert) is evaluated on existing datasets (Winograd, Winogrande and Timedial) using well-established methods (PLL, NormPLL).
The finding that Albert outperforms more parameter-rich models like XLM or RoBERTa can be considered unintuitive, but the analysis in the paper is not sufficient to demonstrate the cause of this performance difference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper makes multiple independent contributions, including:

1. The addition of a new adversarial common-sense reasoning dataset dubbed “Winogradversarial”
2. The explicit gathering together of multiple research threads all exploring the use of pretrained language models for zero-shot prediction on language-understanding tasks
3. The singling-out of the albert-xxlarge-v2 architecture as performing especially well on zero-shot common-sense reasoning tasks, achieving SOTA on the TimeDial dataset

The overall message of the paper is that the existing framework of fine-tuning models for common-sense reasoning tasks is flawed and prone to overfitting, and that more robust results can be achieved using pretrained models in a zero-shot manner. In particular, doing so with the albert-xxlarge-v2 model achieves state-of-the-art performance on some benchmarks.

main_review: **Strengths:**

Presents a new adversarial common-sense reasoning dataset which does not conform to the usual “flip a word and change the answer” paired sentence paradigm, and thus may be harder for models trained on such datasets.

Highlights the shortcomings of fine-tuned language models on this adversarial common-sense reasoning dataset, and the relative strength of pretrained models.

Unifies multiple independent research threads on the topic of using pre-trained models for language understanding.

Achieves state-of-the-art performance on the TimeDial dataset using the albert-xxlarge-v2 model.

**Weaknesses:**

Introduction presents a skewed history of the field, lumping together papers focusing on tagging using feedforward networks (Collobert et al., 2011), efficient learning of context-independent word-embeddings (Mikolov et al., 2013), and learning context-dependent word-embeddings using a recurrent architecture (Peters et al., 2018), all under the vague heading of “machine learning techniques applied to unstructured corpora providing frequencies of words”. It then points to the Transformer as heralding a new era in the field, but fails to cite “Attention Is All You Need” by Vaswani et al., which was the seminal paper that introduced the Transformer architecture.

One of the purported main contributions of the paper, the new Winogradversarial dataset, contains only 20 examples, and is therefore unlikely to be widely useful (see below). It purports to address a proposed pitfall of existing WSC datasets wherein the model can “cheat” if it has seen one instance of a paired example in the training set, and another in the test set, that it can know a priori that it should invert its response. However no direct evidence of this being an issue is provided for the WSC or WinoGrande datasets. If it was truly believed to be an issue there are multiple more direct ways of handling it, such as explicitly enforcing train/test disjointness, or even removing one of the two sentences from each pair altogether.

Performance on the Winogradversarial dataset itself, as shown in Table 1, is an unreliable indicator. For a model whose “true” accuracy is p, the number of questions it gets right on the Winogradversarial dataset can be modeled as a binomial random variable X ~ Binom(20, p). Setting p=0.5 (the average performance of the fine-tuned models), this implies that the probability of getting X≥12 (i.e. observing an accuracy of ≥60%) is >25%. This is a substantial fraction of the overall probability mass, and therefore unlikely to be meaningful. 

It is difficult to pin down the overall purpose of the paper. Within the two introductory sections, topics discussed include:

1. Strengths/weaknesses of word-frequency based approaches
2. Importance of compositionality
3. Concerns around fine-tuning approaches
4. Concerns around existing WSC datasets
5. The grounding problem
6. The introduction of a new WSC dataset
7. Concerns around current incentives and asymmetries in the field

While I understand that these topics are conceptually overlapping, I would strongly encourage the authors to narrow their focus, so that a more targeted and convincing argument can be made.

As discussed in Section 3.1, the previous work of Salazar et al. (2020), Zhou et al. (2020), and Ma et al. (2021) overlaps heavily with the present work. In fact one of the major contributions of this work, namely showing the performantness of the pretrained albert-xxlarge-v2 on common-sense reasoning datasets, merely falls between the cracks of these other papers. In particular Zhou et al. evaluated common-sense reasoning with a variety of different pretrained Transformer models, just not with AlBERT specifically.

In the discussion section, the decreased performance on WinoGrande compared to Winograd is pointed to as evidence that the crowdsourcing of this dataset led to decreased quality. This is a questionable conclusion given that in the original Sakaguchi et al. (2020) paper introducing WinoGrande, fine-tuning on this dataset prior to evaluating on the WSC data by Levesque et al. yielded dramatic performance improvements, achieving 90.1% performance, only a few percent away from human-level, and much higher than the zero-shot performance numbers provided in this work.

The state-of-the-art performance achieved on the TimeDial dataset is not especially meaningful owing to the dataset’s extreme recency. The dataset has only been publicly available for 3-4 months at the time of writing, and as far as I can tell no fine-tuned model has ever been evaluated on it. This renders the claimed state-of-the-art status less impactful than it would have otherwise been.

Minor Issues:

* Top of Page 2: “We are concerned that **that** the fine-tuning”
* Bottom of Page 2: Overly-verbose footnotes. Don’t need to specify where your example names came from, don’t need to point out that people with brain damage have diminished capacities
* Middle of Page 2: “We take the long-standing response to this argument to be a rejection of premise that only classical logic can implement compositional representations.” - meaning is unclear, consider simpler language
* Repeated: Failure to correctly capitalize “BERT”, “RoBERTa”, “AlBERT”
* Bottom of Page 5: Overly-verbose footnotes. The availability or lack-thereof of academic computing resources is not relevant to the present paper. It is assumed that researchers are following the established principle of not peeking at the evaluation set, this does not need to be explicitly called-out.
* Table 2: Inconsistent presence of zeros before the decimal point
* Table 4: T5 accuracy formatted differently than other models
* Bottom of Page 7: “Because **they** text data for these examples are so large”
summary_of_the_review: While the paper makes inroads in a few different interesting directions (presenting an alternative adversarial WSC-style question format, highlighting advantages of pretrained zero-shot models, achieving SOTA on a recently-released dataset), the connections between the ideas are tenuous, and the individual contributions are not sufficiently impressive to stand on their own. I encourage the authors to pick just one or two of these ideas, and flesh them out in a more thorough, convincing manner. In its present form, I believe this paper should be rejected.
review_title: null

Meta-review: This paper argues several loosely-related points about the evaluation of pretrained models on commonsense reasoning datasets in the Winograd style, and presents experiments with existing models on several datasets, including a novel 20-example benchmark. All four reviewers struggled to find a clear contribution or theme in this paper that is novel and thorough enough to meet the bar for publication at a selective general-ML venue.

I'd urge the authors to focus in on just one of these points and expand, and to consider submitting to a venue that more narrowly focuses on methods for commonsense reasoning in NLP.
Decision: Reject