Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes two self-supervised pre-training tasks to further pre-train a pre-trained language model for commonsense reasoning. The first task is called concept-to-sentence generation, which reconstructs input sentences from noun/verb phrases extracted from the input. The second task is called concept order recovering, which predicts original sentences after shuffling the order of noun/verb phrases in input sentences. Experimental results show that the pre-trained language models fine-tuned with the two proposed tasks can lead to improvement on five commonsense reasoning benchmark datasets.

Strengths: 

+ The idea of teaching language models through self-supervised learning tasks is neat.

+ The performance of the proposed methods on few training examples looks great.

+ The results section is well structured. There are ablation studies on the training objectives of each proposed task as well as a comparison of generated sentences.

 
Concerns: 

- The key concern about the paper is the lack of rigorous experimentation to study the effectiveness of the two self-supervised learning tasks. First, the methods are only compared with T5-base related methods on the four commonsense classification tasks. The leaderboard of commonsense QA shows that more than 20 systems report an accuracy higher than 63.32, which is the best configuration of the proposed method. Second, the proposed tasks are applied only to T5. I am wondering if it is effective on the other pre-trained language models. Third, the performance improvement on the classification tasks appears marginal. Statistical tests are desirable to show if such improvements are significant.
 
- Noun and verb phrases extracted from sentences are not always concepts. Masking-out certain words in the input is similar to the idea of removing non-content words from input. A deeper analysis of the proposed method would have been nice to understand which part is effective in the new task, keeping content words or not using mask-out, what if only concepts from a knowledge base is kept instead of content words? 
 
-  The CALM model proposed in this work performs worse than the SOTA models in three out of four metrics on CommonGen, despite it uses less model parameters. What if the proposed tasks are applied on T5-Large?

Minor comments: 

* I am not convinced that the generated corrupted sentences are in general grammatically correct, as stated in Sec. 2.1, 

* I do not see a strong connection between completing COR and compositional reasoning, as stated in Sec. 2.1.

* The way of getting distractor sentences appears ad-hoc, may need further justification.

* Y in equation (5) and (6) needs explanation.

review_title: The paper suggests an interesting redirection for commonsense reasoning, but lacks of rigorous experimentation to justify the contributions.

final_score: rating: 7: Good paper, accept
review_content: This paper suggestsan intermediate training regime that can be used between pretraining and the end-task finetuning. The authors suggest that their method captures more commonsense knowledge by being focused on capturing knowledge about “concepts”. Four different denoising objectives—two generative and two discriminative—are proposed and described in detail, with various possible ways of optimizing for all four. Experimental results show improvements over both the base T5 model and the large T5 model. The proposed method achieves SoTA results on CommonGen with slightly more than half the parameters of the current SoTA model.  Ablations show the necessity of applying a 2-step intermediate training scheme with mixed training followed by joint training.  CALM shows better results with less data than the base model. 

Strengths:
- Unifying generative and contrastive training is an important and interesting goal.
- The objectives suggested are cheap to compute and seem to increase the signal available in the data.
- Extensive results show improvements over a base model and a larger model across a range of tasks.
- Performance with relatively little finetuning data are encouraging.

Weaknesses:
- Somewhat weaker results on some CommonGen metrics are disappointing.
- Using “concept” to stand in for verbs and nouns is somewhat confusing.

After reading other reviews and the authors’ responses to all of the reviewers, I recommend this paper by accepted—extensive results show that the CALM objectives offer more signal from data than current pretraining methods.

This paper suggests a number of cheap-to-compute corruptions of the input data that, when used to reconstruct the input, enrich the underlying model. These objectives certainly improve over the original T5 base _and_ larges models that are used as initializations, and especially outperform the base model in the low-data regime. The authors use objectives which capture both generative and discriminative information, which some have suggested contain mutually beneficial signal but have not been unified in a single training method.

Below are two paragraphs from my original review. The authors have done further experiments and show that there are still gains on these tasks when model sized is increased significantly. Furthermore, they have clarified that on the key metric of CommonGen they achieved SoTA with only slightly more than half the parameters of the current SoTA model. I therefore believe that these results show merit.
> However, in every task except CommonGEN the authors do not discuss any methods that are even close to the state of the art. For CSQA, the best number in this paper is 63.32 vs. 79.5 on the current leaderboard. Similar numbers are true for the rest of the tasks: 60.90 vs. 87 for OBQA, 71.01 vs. 90 for PIQA, and 63.20 vs.  89.70 for aNLI. I do not believe that SoTA results are necessary to write a good paper, and indeed the obsession our field has with SoTA is unhealthy. Yet, it is difficult for me to trust that the effects in this paper will generalize to better performing models without further evidence: what if the CALM intermediate objectives only help with mistakes that larger models do not make in the first place? 

> On the generative task CALM performs closer to SOTA, but it improves only slightly on T5. This is especially disappointing as the objectives introduced _directly_ match the task in CommonGEN, making this intermediate training a form of noisy training data rather than pretraining.

I still feel that the authors’ use of “concept” and “commonsense” is vague, when their method can be defined more clearly with more mundane terminology. In practice, the authors use nouns and verbs as their concepts, which is fine in terms of pretraining objectives, but surely does not capture the generality of concepts. The authors have somewhat clarified in this in their updated version.

Finally, the CALM intermediate objectives share many properties with all of the datasets tested on and are likely calibrating the model to the kind of correlations they should expect to predict in advance of finetuning. One way this can be seen is  that the slopes of the T5 and CALM lines are very similar after an initial “bump” which T5 likely needs to calibrate to the new distribution. This makes claims of learning “commonsense” hard to verify, though I do agree that _something_ relevant to solving these problems is clearly being learned.

Altogether, I think this paper makes an interesting contribution to the question of: How can we get the most pretraining signal from unstructured data using off-the-shelf tools? I recommend this paper for acceptance, though I encourage the authors to revise their paper to make this the focus of the story, rather than the vaguely defined notion of “concept”.
review_title: Interesting Proposed Objectives, Weak Results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
This paper proposes the “Concept Aware Language Model” (CALM) --- a pre-trained T5 Transformer is trained on self-supervised intermediate tasks to learn relational commonsense knowledge, before fine-tuning on downstream tasks. The intermediate tasks include (1) concept to sentence (c2s) generation - given a list of permuted concepts (verbs and nouns), generate the target sequence, (2) concept order recovery (cor) - given a sequence with the order of concepts (verbs and nouns) shuffled, generate the original sequence, (3) given a sequence and it’s perturbed version (concepts shuffled), generate the sequence (classification but as a generation task). Training is first done in a multi-task fashion, followed by a stage of training that uses generated outputs from (1) and (2) for (3).

The goal of the paper is to show that carefully designed objectives for self-supervised intermediate task training add relational commonsense knowledge which helps improve model performance on downstream commonsense reasoning tasks. 

Strengths:
- S1 - The idea of self-supervised intermediate task training is an exciting one especially in the form of adding reasoning capabilities that BERT, GPT, T5 etc. might not be acquiring during the large-scale pre-training phase.   
- S2 -  As shown by results in Table 1 on 5 commonsense reasoning tasks, the intermediate task training proposed in this work improves performance over and above the T5 model (and variants including with salient span masking for concepts).  
- S3 - The experiments involved averaging across 3 random seeds and the authors have reported confidence intervals.   

Weaknesses and questions:
- W1 - One missing baseline is T5 trained to unshuffle entire sequences - given an input with the tokens shuffled, generate the original sequence. This would show how much value c2s and cor are really adding. The current T5 baselines are all trained purely for infilling which seems a bit unfair compared to CALM which is generating entire sequences.  
- W2 - Given a T5 model that can score sequences (maybe after training on the autoencoding objective), would it score “apples grow on trees” higher than “trees grow on apples”? If yes, then the model seems to already exhibit this style of reasoning. Would it score “apples grow on trees” and “apples grow in the ground” similarly? The distinction here is between sequences that are non-grammatical or unlikely to ever appear versus sequences that may have appeared (eg: “potatoes grow in the ground”). Presently, (a) it’s unclear if the designed objectives are providing commonsense reasoning above something the model can know from autoregressive language model scoring, and (b) it appears that the objectives are not designed to add relational commonsense knowledge of the sort where we know apples don’t grow in the ground.   
- W3 - c2s is designed specifically to do well on CommonGen. Are the gains on this task smaller than what you might have expected? If yes, why isn’t it helping more?   
- W4 - Figure 4 needs error bars, the dev sets are really small and it’s hard to interpret which differences are significant.  



__UPDATE__

Thanks for the incredibly detailed response! I've raised my score to a 8.  
I do in general quite like the paper, and the responses here are thought-provoking. I'm not sure I'm totally convinced by the WSC results comparing CALM the classifier to T5 the sequence scorer. Not sure if it's an apples to apples comparison...but I'm not sure there's a straightforward setup for this, and perhaps it starts to get beyond the scope of what's being presented here. 
review_title: Interesting work

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: #### Summary
This paper addresses the issue of incorporating commonsense into large pretrained language models. They propose a pretraining method that does not leverage knowledge graphs but rather use approaches which involve corrupting the input sentence in various ways to recover the original sentence. This methodology is used to try to bake in commonsense into models. Results are shown on both discriminative and generative common sense datasets

#### Novelty and clarity
The training procedure of corrupting inputs to retrieve outputs is not new but the use on commonsense tasks does seem novel and also is an interesting approach. The paper was very clear to read and the technical aspects were well described.

#### Strengths
(1) The use of a self-supervised approach is great because it requires no annotation and the training procedure is simple. It is also described well
(2) The variety of baselines used is good and comparison against models larger than the proposed model is interesting to see.
(3) The use of generated sentences to improve language models on hard areas like commonsense and offensiveness is a great idea as it can help make the model more robust


#### Weaknesses
(1) There should be a more comprehensive set of results completed to see how much improvement this model has. Mainly on the CommonGen class there should be some manual evaluation done similar in the original paper to see if the outputted sentences make sense and that the improvement in automatic metrics is carried over into human evaluation. 
(2) A key aspect to look into is the robustness of this model. In the C2S approach the concepts were shuffled to generate the correct sentence. During inference time if the concepts were shuffled in a different manner would the model still be able to generate the correct sentences? There was three random seeds used but as was said "the performance is sensitive to different random seeds." which seems that the model isn't as robust to newly seen inputs

#### Detailed Comments
If spacy was also used for POS tagging along with tokenization this should be made clear. Also for every sentence was there 3 nouns/verbs extracted? 
One thing I'm unclear about is in Table 2 why is the "Our T5-Base" better than the "T5-Base" above? Is this T5 with additional epochs? I think this should be made clear. Additionally I wouldn't say and "is only slightly worse than KG-BART." It seems a lot worse especially on BLEU and CIDER. It is nice to see a smaller model is beating a larger model on some metrics
"The difference between CALM and CALM (Joint) is that the former is initialized by the CALM(Mix)." Did you mean to say latter instead of former? Also I don't see CALM (Join) in the table. I'm assuming this is CALM without Mix warmup


#### Questions for the Authors
1) How did you ensure shuffling the sentences still has grammatical correctness? A sentence like "Running I am" is not grammatically correct.
2) Instead of a POS tagger why did you not use an NER extractor? Also wouldn't swapping different fruits into sentences like replacing "Apples grow on trees" with "Watermelons grow on trees" help with robustness
3) And what point is the generative model good enough that it doesn't help to create distractor sentences



review_title: Nice approach to incoporate commonsense into large models but needs more evaluation

Meta-review: This paper presents two self-supervised learning objectives that can be used as intermediate pre-training tasks to refine the T5 sequence-to-sequence model between pre-training and task fine-tuning. It shows that, at small to moderate model sizes, adding this step significantly improves performance on commonsense-oriented target tasks.

Pros:
- This appears to be a fairly straightforward improvement in self-supervised learning in NLP, with fairly extensive experiments.

Cons:

- This model isn't trained at the same extremely large scales (10B+-words) as state-of-the-art models, and it performs significantly below the state of the art. It's not clear that the released model represents a useful model for any application as-is, and while it's likely, it's not proven that the ideas in this paper would still be useful at larger scales.
- Given that, it seems like the most likely audience for this work is other developers of pretrained models in NLP, which makes the fit to a general ML conference less clear.
- The framing around 'concepts' and, more importantly, the model name 'concept-aware LM', gives the unwarranted impression that the new model handles 'concepts' in a way that T5 doesn't. It is not reasonable to use the word 'concept' to refer to specific parts of speech in your title (even if you later explain that), and whether your model handles concepts in a categorically different way from T5 would take a substantial analysis to show, which doesn't seem to be present. I don't think this paper is up to ICLR's standards with the current name, and urge the authors to change it.

Decision: Accept (Poster)