Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Authors: Wangchunshu Zhou
Dong-Ho Lee
Ravi Kiran Selvam
Seyeon Lee
Xiang Ren
Venue: ICLR 2021 Poster (2021)
Keywords: Language Model Pre-training
Commonsense Reasoning
Self-supervised Learning

Abstract: Pretrained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks that require a syntactic and semantic understanding of the text. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational and compositional commonsense knowledge about everyday concepts, which is crucial to many downstream tasks requiring commonsense reasoning. To augment PTLMs with common sense, we propose generative and contrastive objectives as intermediate self-supervised pre-training tasks between general pre-training and downstream task-specific fine-tuning. We also propose a joint training framework to unify generative and contrastive objectives so that these objectives can be more effective.
Our proposed objectives can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge bases, yielding better performance on both NLU and NLG tasks. We apply our method on a pre-trained T5 model in an intermediate task transfer learning fashion to train a concept-aware language model (CALM) and experiment with five commonsense benchmarks (four NLU tasks and one NLG task). Experimental results show that CALM outperforms baseline methods by a consistent margin.

Published as a conference paper at ICLR 2021
PRE-TRAINING TEXT-TO-TEXT TRANSFORMERS FOR CONCEPT-CENTRIC COMMON SENSE
Wangchunshu Zhou'; Dong-Ho Lee²*, Ravi Kiran Selvam?, Seyeon Lee?, Bill Yuchen Lin', Xiang Ren2
Beihang University 2 University of Southern California
zhousangchunshuuuuaa. edu. en, (dongho. lee, xiangren] @usc.
ABSTRACT
Pre-trained language models (PTLM) have achieved impressive results in range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such masked token prediction (for BtRT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centri commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specife fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify gen- erative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept- -awere language model (CALM can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge graphs, yielding better performance on both NLU and NLG tasks. We show that while only incremenallly pre-trained on a relatively small corpus for a few steps, CALM out- performs bascline methods by consistent margin and even comparable with some larger PTLMs, which suggests that CALM can serve as a general, "plug-and-plas method for improving the commonsense reasoning ability of PTLM.
INTRODUCTION
Pre-trained language models (PLTMs) such as BERT (Devlin al. 2018) and T5 (Raffel al. 2019) have revolutionized the field of NLP, yielding impressive performance on various conventional natural language understanding (NLU) and generation (NLG) tasks. BERT and its novel variants such as RoBERTa (Liu et al. 2019) and ALBERT (Lan et al. 2019) capture syntactical and semantic knowledge mainly from the pre-training task of masked language modeling, while T5-style models such as BART (Lewis et al.) 2019) instead focus on masked span infilling tasks. Though yielding better performance on many downstream tasks, these pre-training objectives, however, do not explicitly guide the models to reason with concept-centric commonsense knowledge from language, including the relation and composition of daily concepts in our lives. This leaves room for equipping current PTLMs with richer commonsense reasoning ability.
For example, consider a multi-choice question "What do you fill with ink to write notes on a piece of copy paper? (A) fountain pen (B) pencil case (C) printer (D) notepad". The current of question answering model, UnifiedQA (Khashabi et al., (220)) which was fine-tuned on TS-large with multiple datasets, still predicts '(C) printer' its answer. The model may be overly sensitive to the co-occurrence between phrases in question sentence like 'ink' and 'copy paper' and the answer choice 'printer', but fails to reason with the concept-centric knowledge that 'fountain pen' is writing instrument that needs to be filled with 'ink'. Such mistake in commonsense reasoning becomes a bottleneck for current PTLMs (Davis & Marcus 2015). Towards augmenting PTLMS with more
Equal contribution. The work was done when Wangchunshu was visiting USC. "Code will be published at:
USC/UCCAAI

---

Published as a conference paper at ICLR 2021
knowledge, prior works mainly focus training larger models (Brown al. 2020), adding specific architectures to exploit external knowledge (Peters et al.) 2019). or incorporating knowledge bases for pre-training (Xiong et al. 2020 In this paper, instead look to explicitly teach pre- pre-trained models to write and reason with common concepts through novel pre-training strategies.
We present two kinds of self-supervised pre-training tasks: concept-to-sentence generation (C2S) and concept order recovering (COR). C2S trains the pre-trained model to compose ("write") sentences given set of concepts, and expects the generated sentences be fluent and plausible in terms of commonsense. COR aims to teach models to detect and revise corrupted sentence with incorrect ordering of concepts. As illustrated in Figure I both tasks require a pre-trained model to recall relevant commonsense facts about the concepts and to understand the underlying commonsense relations between them. Both of the proposed objectives can explicitly encourage the model to capture the relational concept-centric commonsense knowledge and perform compositional reasoning.
Specifically, we need a generative pre- mre-training objective to encourage models to capture this genera- tive commonsense reasoning ability, so that models can learn to generate sentences with common- sense knowledge for both C2S and COR. Also, to teach modes to distinguish truth sentences from less plausible ones, we need to teach models with discriminative commonsense through contrastive self-training. To unify both generative and contrastive objectives within a joint learning framework so that the model can learn both generative and discriminative commonsense knowledge at the same time, we propose to use the sentences generated by the model itself as the distractors and train the model to distinguish the generated sentences from real sentences. In this way, the model is forced to acquire new commonsense knowledge in order to distinguish the distractors generated by itself, which probably exploit the knowledge the model already possesses. Therefore, the model trained to iteratively improve upon itself in a self-play fashion. We share all the parameters between the generator (trained with the generative objective) and the discriminator (trained with the contrastive objective). then train multiple objectives with different prefixes. Compared to previous works (Peters) jet al., 2019 Li al. 2019: Xiong et al. 2020) that utilize external knowledge bases like Wikidata or NonceptNet, our approach can directly improve the generative and discriminative commonsense reasoning ability of PTLMs at the same time without relying on external knowledge bases.
To evaluate the effectiveness of our proposed method, apply our method in intermediate-tass transfer learning setting (Pruksachatkun et al. 2020) based on the pre-trained T5-base model to train Concept-Aware Language Model (CALM). While only continually pre- pre-trained on small dataset for a relatively fewer number of updates (compared to conventional pre-training). CALM consistently outperforms T5-base four summonsense-related NLU datasets (i.e., COMMONSENSEOA OPEN- BOOKQA, PIQA, and ANI.I) and COMMONGEN, a commonsense-relaan and careful ablation studies demonstrate the potential of our method to serve as "plug-and-play method for any pre-trained text-to-text transformer before fine-tuning on commonsense-related tasks. To the best of our knowledge, our work the first to investigate concept-centrie self-supervised objectives that improve both generative and discriminative commonsense reasoning ability of pre-trained language model.
NI. dataset. Our results
SELF-SUPERVISED OBJECTIVES FOR CONCEPT-CENTREE LEARNING
In this section, we first describe the proposed generative and contrastive objectives used for improving the commonsense reasoning ability of pre-trained text-to-text transformers. Then, we introduce the joint learning framework which unifies the proposed self-supervised objectives and learn unified text-to-text transformer based on pre-trained models such as T5.
2.1 GENERATIVE OBJECTIVES
Similar to many other pre-training tasks such as masked language modeling, we aim to teach models to recover original sentences from corrupted inputs, which is often regarded denoising process. We propose two generative self-supervised pre-training objectives: (C2S) and concept order recovering (COR).
generation
Concept Extraction. Given an input x = [x', I2...., LnI. we first conduct pfrt-of-speech tagging with Spacy for the sentence and extract Verb, Noun, and Proper Nouns from the sentence use as
2

---

Published as a conference paper at ICLR 2021
Concept-to-Sentenc
Concept Order Recovering
Input: <c2s> Generate a sentence with the concepts:
Imput: <cor> Correct the order the given sentence: Rahul stops him, fights his bar, and drives to local performance.
forward, 5 smppon, ignore, information, prosecutor
Text-to-Text
Text-to-Text Transformer
generative common sense
Transfomer
Output:
Output:
The information was forwarded to Simpson 's prosecutors, but was ignored.
Rahul fights him, stops his performance, and drives a local bar.
Figure 1:
Two self-supervised pre-training objectives that teach text-to-text transformers with generative common sense: (1) Concept-to-Sentence Generation (C2S) pre-trains the model to recover the original sentence with a shuffled concept set, e.g., (fowwrd, Simpson, ignore, information, prosecutor) + "The information
forwarded to Simpson's prosecutors, but it was ignored." (2) Concept Order Recovering (COR), similarly, teaches the model to correct the mispositioned concepts in the original sentence. For example, the concepts fights, bar, drives, performance), are randomly reordered the input, while the model should recover the original sentence.
concepts Next, we form concept-sets C [V1, 02.... Up, 71 72 .... nny where v, and n, denotes the i-th verb or noun/proper noun concept (token) in x. We denote C. and C, as the set of verb and noun/proper noun concepts respectively C. (i.e. C. [01, ....... Up) and C, [M1, m2,.... ngl.)
Concept-to-Sentence Generation (C2S). The concept-to-sentene generation (C2S) objective re- quires the text-to-text transformer recover the original sentence given only a few unordered keywords of the sentence. Specifically, given a sentence, we shuffle the extracted concept-set C to create the perturbed source sequence and train the model to generate the original sentence with prefix (denoted as <c2s>> as described in Fig. Formally, the C2S objective can be formulated as:
Le2s E
log plz PEMMUUUC))))]
where the PERMUTE() function randomly shuffle the concepts in the concept-set. This objective requires the model construct acceptable commonsense sentence by adhering and reasoning over the commonsense relations between the given concepts. Therefore, relational commonsense knowledge is implicitly injected into the parameters of the model. The C2S objective is motivated by the task proposed in Lin et al. (2020). Compared to their work, the concept-set used in C2S covers more concepts such as named entities, while the original task only includes the concepts appearing in ConceptNet. We apply the task in general domain and as a pre- pre-training objective, instead of merely serving as an evaluation task.
Concept Order Recovering (COR). As for the concept order recovering (COR) objective, we shuffle the order of concept sentence and train the model to recover the original sentence. As illustrated in Figure[] given input sentence "tree grows the apple,", the models would shuffle the concepts including "tree", " "grow", and "apple" to recover the original sentence "apple grows on the tree." The noise introduced by concept shuflling is different from that by traditional self-supervised objectives like mask language modeling and mask span prediction because the corrupted source sentences are in general complete (i.e., tokens spans are masked) and grammatically correct, while not acceptable in terms of commonsense because the order and relation between concepts are shuffled. By training the model to detect and correct the disorder of concepts in a sentence, the model is expected to acquire some relational commonsense knowledge like "apple generally grows on tree" instead of "tree grows on an apple."
Formally, the COR objective can be formulated as:
Lew E
C): CONCEETTPPPOEEPEPPPPCTEEE ())
We split the concepts with multiple tokens (under Spacy tokenization) into single token to ensure the concepts discussed afterwards all contain single token.
3

---

Published as a conference paper at ICLR 2021
where <cor> is the prefix for the COR objective illustrated in Figure 1 The function CONCEPT- PERMUTE() permutes the order between concepts in the same category (i.e. noun or verb) in the sentence, which can be formally defined as:
24
Dy $ C
CONCEPT-PERMUTE(X.CO [21, Z221 ................................ 2,,] where x
PERMUTE(C.)Li
Hi Uj
(3)
PERMUTE(C..D] I; nj
Our proposed objectives require the model to capture the relational commonsense knowledge between concepts and perform relational (COR) and compositional (C2S) commonsense reasoning in order to successfully reconstruct the original sentence. Therefore, the model is encouraged to acquire concept- centric commonsense knowledge more effectively. In contrast, conventional pre- pre-training objectives like masked language modeling and masked span infilling mainly focus on general token-level co-occurrence patterns and thus are less effective for learning commonsense knowledge.
2.2 CONTRASTIVE OBJECTIVE
The contrastive objective encourages the pre- trained model to distinguish the real sen- tence from a distractor sentence: sentence that is similar to the real sentence, gener- ally grammatically correct, but may not fol- low common sense. We expect it to im- prove the pre-trained model's discrimina- tive commonsense reasoning ability so that the model's s performance on commonsense- messonnng-discriming
Generative QA
Input: <cont> Which sentence is correct?: options:
1. The increased number of male visitors inspired by the article raised security concems
2. The increased article of male visitors raised by the number inspired security concerns
discriminative common sense
Text-to- Text Transformer
tasks, like Com- monsenseQA, can be improved. We formu- late the contrastive objective as Generative QA task: we take the concatenation a pre- fix <cont> (question context), the real sen- tence x (answer), and the distractor x' (dis- tractor) the input and train the model to output the real sentence x. Formally, we have the loss function of the contrastive objective defined as:
Output:
The increased number of male visitors inspired by the article raised security concerns
Figure 2: Overview of Contrastive self-supervised pre-training objectives. Generative QA style con- trastive objective requires the model to distinguish truth sentences from less plausible ones.
Leont = B log [0ogp(/]<cot333 PEw.:ccCooo,,..MMME::(:(::::
(4)
where the prefix <cont> is described in Figure The distractor x' is either constructed by concept shuffling as described previously (i.e. x'' = CONCEPT- PEA-P---PRMUUEEEE C)) when used independently, or generated by generator trained with the aforementioned generative objectives when used in the joint training framework, which will be described the next section.
JOINT TRAINING WITH GENERATIVE AND CONTRASTIVE OBJECTIVES
The aforementioned generative and contrastive self- supervised objectives can be applied independently or simply combined in a multi-task learning fashion. We argue that these two objectives can mutually re- inforce each other: generated sentences from the generative objective can help the contrastive mod- ule learn to distinguish commonsense sentences from less plausible ones.
Algorithm 1: Pre-training Concept-
Aware Language Model (CALM).
Input: Text-to- Transformer To, Text corpus X=[#ı, 22..., x,..
repeat
for each E X do
Extract the concept-set Ci; Construct the distractor sentence
= CONCEPTT-ERMUTTEC C.):
Therefore, we propose a joint training framework to unify generative objectives and contrastive objec- tives by using the generator to produce distractors for learning towards contrastive objective.
Update To with Eq. until maximum iterations reached, repeat
each i, E
Update To with Eq
Specifically, we have a generator Go (trained with the generative objectives) and discriminator D3
until maximum iterations reached;
4

---

Published as a conference paper at ICLR 2021
Oriihal Bentence x
Weight Sharing
She was the Arst woman hold the cossion
Discriminator (Text-to- Transformer)
Extracting Concept
Generator
Orginal Sertence
(Text-to Text Transformer)
CONCEPT-PERUUTBKCO
Distaator Benience
She was the Sirtt woman hold mhe poatiio
<cor> She was the Aist postion to hold the woman
COR
She was the first woman posiion the hold
P PERMUTE(O
Detinder Sentence
Woman holds the posntoo
Nold Woman Postion
Woman holds
Generative QA
She woman hold the position
Figure 3: Proposed Joint Training Framework. Given an input sentence ("She was the first woman to hold the position."), we extract concept-set C (woman, hold, position). Given æ and C, we produce corrupted source sequence x' either for C2S and COR. The generator trained with the corresponding objective recovers sentences as distractors x" to the discriminator, The discriminator is trained to distinguish truth sentences from randomly selected distractor among two objectives. Parameters between the generator and discriminator are shared.
(trained with the contrastive objective). Given an input sentence x, we first use the method for either C2S or COR to produce the corrupted source sequence x'. Then, use the generator Gg trained with the corresponding objective to generate the recovered sentence x" Go(x'). We then take x" as the distractor train the discriminator D& with the contrastive objective. The loss function of the proposed joint training framework consists of two parts: the first part is the loss of generative objectives, which identical to the loss described in Eq. and 2) and is used to update the generator Gg. The second part is the loss of the contrastive objective as described in Eq. (4) which can be formulated as:
Lcont_joint_ed IE( log (ely///oot>;rr/ Go(<c2>>; (<(<C222)))RM
(5)
Leont_joint_cor E log <conttt) Go (<cor>; CONCEPT COCCEPSCCCCCE
where Tecont joint c2s and I-cont_joint joint_iitttt cor is the contrastive loss with the distractor generated with either the C2S or the COR objective and y is the original sentence. We then have the overall objective for the joint training framework defined as
Ljoint
(Lees + Lecor + ((Laantjooint, c2s Lrcont_joint. soint_cor)
Lezs and Loor are defined in [ and Eq. ② respectively and is a hyperparameter controlling the relative weight between the generative and contrastive objectives. Note that since we would like inject both generative and discriminative commonsense reasoning ability into the parameters of a single text-to-text transformer, we share the parameters between the generator Ga and the discriminator Da..
Finally, we describe the overall procedure to apply the proposed self-supervised objectives and the joint training framework on a pre-trained text-to-text transformer. We apply two-stage training strat- egy. During the first stage, we apply our proposed generative and contrastive objectives individually on the model point for the second stage where the joint training framework is applied. We summarize the workflow of our method in Algorithm
multi-task learning fashion with different prefixes. This provides good starting
4 EXPERIMENTS
In this section, motivated by the observation of Pruksachatkun et al. (2020) that tasks requiring commonsense reasoning ability generally serve as good intermediate task, we test our method the intermediate task transfer setting. Specifically, we initialize our model with T5-base, text-to-text transformer model, and training the model with our proposed method as intermediate task before fine-tuning and target downstream tasks. Another reason for adopting this setting is because we expect our method to serve as "plug-and-pla" method that can be applied to any pre-trained text-to-text transformer by simply continually training for few steps.
treitrained
Details for Pre-training and Fine-tuning CALM is continually pre-trained with our proposed self-supervised objectives intermediate tasks based on the pre-trained T5-base model following
5

---

Published as a conference paper at ICLR 2021
the setting in Pruksachatkun al. (2020). We randomly sample 500K sentences from the English Wikipedia corpus" which is used for pre-training BERT and its variants, as the source dataset for our proposed self- sufersueeei objectives which serve as intermediate tasks. We then fine-tune the CALM on each downstream task individually and report the average performance of three runs with different random seeds for fine-tuning on each dataset since the performance sensitive different random seeds. Training details and hyperparameter settings are presented in Appendix A.I A.2
Datasets We consider five commonsense benchmark datasets as target tasks. We categorize these datasets into discriminative and generative tasks. Discriminative tasks are classification tasks while generative tasks are text generation tasks. We consider four datasets for discriminative task: Com- monsense (Talmor et al. 2018)) OpenbookQA (Mihaylov et al.) 2018). PIQA (Bisk et al. 2020). aNL.I (Bhagavatula et al. 2019) and one dataset for generative task: CommonGEN (Lin et al. 2020). Details on datasets are discussed in Appendix A.3
Compared Methods We compare our model with following models continually trained with different intermediate tasks based on the pre-trained T5-base model: (1) T5-base is the pre-trained TS-base model without continually training on any intermediate task. (2) T5-base w/ additional epochs is continually pre-trained using the original pre-training objective of T5 with additional training steps. The total number of additional training steps is equal to that of our final model. (3) T5-base + SSM is continual pre-trained with variant of the salient span masking objective (Guu al.)
2020 Roberts et al. 2020) objective that masks text spans of concepts extracted with POS tagging instead of named entities extracted by pre-trained NER model, which makes it more focused on concepts. (4) CALM(Generative--Oe
is continually pre-trained with the proposed generative objectives including concept-ro-senieen generation(C2S) and concept order recovering(COR) as intermediate tasks. (5) CALM(Contrastive-Onl
is continually pre-trained with the proposed contrastive objective as described in section 2cction222jume J.2jun juing the distractor generated by concept shuffling. (6) CALM(Mix-only) is continually pre-trained with both the generative objectives and the contrastive objective, combined with multi-task learning fashion with identical weights for each objective as the intermediate task. (7) CALM (w/o Mix warmup) is continually pre-trained with the joint training objective described in Eq (7) directly from the pre-trained b5-base model. (8) CALM is our main The difference between CALM and CALM (Joint) is that the former is initialized by the CALM(Mix) We also include the performance of the BERT-base model and two knowledge enhanced PTLMs that have similar architecture to BERT-base.
model trained as described in Algorithm| I
Evaluation Metrics For discriminative tasks, we choose accuracy as our metric following other conventional question answering tasks. For generative tasks, we report automated metrics including BLEU (Papineni et al., 2002), METFOR (Banerjee & Lavie 2005), CIDEr (Vedantam et al. 2015).
and SPICE (Anderson al. 2016) following the leaderboard of COMMONGEN (Lin et al. 2020).
Results for COMMONGEN are on the test set and others are on the official development set. We tune the hyperparameters based on the models' performance on in-house split dev set.
4.1 EXPERIMENTAL RESULTS
The result is presented in Table I First, we see that our CALM model consistently and significantly (with p-value < 0.01) outperforms the backbone TS-base model on all five datasets by margin range from 1.5 to 2.9 accuracy discriminative tasks and 1.5/0.6 BLEU/SPICE score on CommonGEN. This is an impressive result since we are only performing intermediate training on relatively small dataset for only around 20k updates. It demonstrates the potential of our method for serving as a "plug-and-play" method for packing more commonsense knowledge into a pre-trained text-to-text transformer. TableBalss shows that CALM performs comparably with several large-size PTLMs like BART, T5-large, and GPT-2 on the COMMONGEN dataset. The performance worse than KG-BART (Liu et al. 2220). the current state-of-the- on COMMONGEN, which is a contemporary work that exploits external knowledge bases as additional information, and is based on a larger backbone(i.e.. BART (Lewis et al.) 2019)).
In addition, we can observe that both the proposed generative and contrastive objective outperforms the backbone TS-base model, as well as its variants that continually pre-trained with the original masked span prediction objective and the concept-specine salient span masking scheme, when applied independently, Note that we find the variant of salient span masking that focuses on concept is not

---

Published as a conference paper at ICLR 2021
Methods
CSQA
OBQA
PIQA
aNLI
CommonGEN
Accuracy (official dov)
BLEU-
METEOR
CIDE
SPICE
BERT-base
53.08C+0. 16)
57.60( 8)
64.86(+0.52)
61888(+0.56)
ERNIE KnowBERT
54.06< 12)
58.900 38.90(+-0.9)
66.47(+0.58)
60.04(+0.46)
53,88C+00. 15)
66.61(#0.63)
63.18(+00.52
15-base
58.2001:1.0) 61.88(1:0.08)
68.14(21073)
61.10:1:0.30
24.90
31.20
12.99
32.40
T5 base cont. pretraining
58.10, 61.92(10.445)
61.15(10.555 100.77
25.10
31,00
13.12
32.40
15- + SSM
62.08(2:0:41)
58.30 1:088
68.27(40.71)
61.25((:::51)
25.20
31.20
13.28
32.40
CALM (Generative-Onll
62.26(+0.33)
58.90(20.4) =0:4)
68.91
60.95(11044)
25.80
31.20
32.60
CALM (Contrastive-Olly)
02.73(+:0.41)
59. 19.30( :0
70.67(#0.99 98)
61 (1.350(0006)
25.50
31.20
13.58
32.60
CALM (w/lo Mix warmup)
62.18(2:0.48)
20.57) 50.00
61.25)(0..55)
31.20
32.60
CALM (Mix-only)
63.02(+:0.4)
60 (0.400(:044
70.07(+0.99 98)
02. (2.75)+0.555)
26.00
31.20
13.82
32.80
CALM
63.32(20.35)
60.90(:0.4)
71.01(:0.61)
63.20(110522)
26.40
31.40
Table 1: Experimental results commonsense reasoning datasets. The first group of models are baselines. The models in the middle group and last group except the CALM model are trained with the proposed objectives independently and the final CALM model is trained by joint training. Best models are bold and second best ones are underlined within each metric.
Methods
CSQA
OBQA
PIQA
aNLI
CommonGENN
Accuracy (official dev)
BLEU4
METEOR
CIDE
SPICE
BERT-large
57.06(+0.12)
60.40((0.6)
67.08 (7.08(+0.61)
66.75(=0.6))
15-large
61.40(=1.0) 69.81(#11002)
72. 75.54( 1.09
28.60
30.10
14.96
31.60
CALM-large (Mix-only)
70.26(++023)
62.50(=1.0)
1..70
75.99(= 1..99=1..20
29.20
31.30
15.24
33.10
CALM-large
71.31(20.04)
66.00(=1.0)
75.11(+1165)
77.12(
29.50
31.90
15.61
33.20
RoBERTa-larg
71.21(100.25
63.90(10.8)
76.90X
82,35( L.0.4))
Table 2: Experimental results on large model. Comparison between large models of other PTLMs and CALM. Best models are bold and second best ones are underlined within each metric.
very effective. We suspect this is because the resulting training data would be somewhat similar to the original text infilling objective because concepts are very common the corpus and we only train for few steps. The combination of the generative and contrastive objectives (i.e., CALM(Mix--n/) yields further improvement upon the model trained independently with either generative or contrastive objectives. Also, we find that the CALM model consistently outperforms CALM(Mixx), demonstrating the effectiveness of the proposed joint training framework. Applying joint training directly on top of pre-trained model (i.e., CALM(w/o Mix warmup)) does not work very well, demonstrating the necessity of applying mixed training to initialize the model before starting joint training.
To further confirm the effectiveness of our approach, we also apply our method to continually pre-train T5- with the same data and number of training steps. We then compare the performance of the resulting model with that of the original large model in Table TO, We find that both the proposed training objectives and the joint training framework consistently and significantly (with p-value < 0.01) improve upon the original Ta-large, show- ing our approach is effective for models with different sizes. Our model also out- performs BERT-large by large margin. However, our model performs slightly worse compared to RoBERTa-large, We suspect this is because RoBERTa-large is optimized for more steps than T5-large and our CALM-large. This also observed in many other tasks and datasets.
CommonGEN
Methods
Parnms
BLEU-4
METEOR
CIDE:
SPICE
GPT-2
et al.
2019
25.90
UniLM
(Lonn Caal
2019
340M
27.70
29.70
14.85
30.20
BARE
Lows 24 al
2000
406M
26.30
30.90
13.92
30.00
T5 base
BRafee of
2019
220M
16.40
23.00
9.16
22.00
T3-larg
770M
14.96
31.00
KG-BAK
Liu
2021
406M
30.90
32,40
1688
32.70
T5-base (our implementation)
220M
24.90
31.20
12.99
32.40
CAT M-hose
220M
26.40
31.40
13.88
33.00
CALM-large
774M
33,20
Table 3: Comparison between PTLMs on CommonGEN, Above baselines are reported number in the leaderboard. T5 15-base(our implementation) uses different hyperparme- ter setting than that reported in the leaderboard.
4.2 PERFORMANCE ANALYSIS
Analysis on Generative objective To investigate the contribution of each generative objective, we conduct an ablation study by continually pre- ve-irrining three models from the same T5-base model with C2S, COR, and text infilling, which is the original objective for pre-training T5, the objective for the intermediate task. We continually pre-train these models for the same number of steps and then evaluate their performance by fine-tuning on different target tasks. The result is shown in Table 4
7

---

Published as a conference paper at ICLR 2021
Methods
CSQA
PIQA
CommonGEN
Methods
CSQA
PIQA
CommonGEN
Accuracy
BLEU-
METEOR
CIDEr
SPICE
Accuracy
BLEU-4
METEOR
CIDE
SPICE
T5- Text Indlling
61.92
68.00 19
25. 10
31.00
13:13
32.40
Multichunice QA
62:21
68.82
25.00
31.20
13.26
32.60
CALM-COS -OOR
62.36
68.77
25.70
$1.20
13:65
32.60
True/False
62:24
67.81
25. 10
31.30
13.11
32.80
CALM C25
62.24
68.75
25,90
31.40
1394
32,80
Generaziee QA
62.73
70.67
25,50
31,20
13.58
32,50
(a) Generative objectives
(b) Contrastive objectives
Table 4: Analysis on Contrastive and Generative objectives. Left table shows the performance on downstream tasks by pre-training with different generative objective (COR, C2S, and original objective for pre-training T5). Right table shows the performance on downstream tasks by pre-training with different task formats of contrastive objective.
We can see that both C2S and COR works better than the original masked span infilling objective on itself. This confirms the effectiveness of our proposed generative objectives on improving the commonsense reasoning ability of pre-trained text-to-text transformers.
Task Formulation of the Contrastive objectives For contrastive objectives, we test three different task formats: Multi-choice QA. Generative QA, and True/False. Multi-choice QA and Generative QA takes the concatenation of the real sentence and the distractor. Then, Multi-choice QA output the index of the real sentence following other conventional Multi-choice QA tasks, and Generative QA output the real sentence respectively. True/False takes either the real sentence or the distractor and train the model to perform a binary classification problem of whether the input sentence makes sense. The result is shown in Table 4 We could find that the format of Generative QA performs the best. We suspect this is because the Generative QA format is closer to the format used during the original pre-training stage of the T5 model and the format used for fine-tuning
Performance with fewer training examples To investigate the effec- tiveness of our objective in the low- resource setting, we explore the per- formance of our model and baselines fine-tuning with different fractions of the training data. From Figure 4 we can see that the performance improve- ment yielded by our models upon the T5-base model is more significant in the low-resource regime. This shows that CALM may already pack some commonsense knowledge in its parameters that it does not require much data for fine-tuning before obtaining a good performance. In contrast, the original T5 model requires much data for fine-tuning, which suggests it may fail to encode much commonsense knowledge and must fit the correlation patterns in the downstream datasets to get a good performance.
Commrrssnnse QA
Openbock QA
Accurryy
AccurscY
CALM Moccriy
CALM
25
CALY
30
20
30
60
200
20
40
60
170
Percent Training Data (%)
Percent Training Data (3S)
Figure 4: Performance of compared models fine-tuned with different fraction of the datasets.
Comparison of Generated Data Table 5 shows the comparison of generated examples for the COMMONGEN test set between T5-base and CALM. We can see that the sentences generated by CALM are generally more acceptable in terms of commonsense plausibility while TS-base sometimes generates sentences that do not make sense.
Concept-set
TS-hase
CAL.M-busee
Crass, Dog, Rall, Chase
dog chracal by the graes
dag chasing in the grses.
Net, Boot, Witer Hole: Phatt Dig Iaprodient, Add, Pus, Fry
Gohiig best casts the waier
fidrrmun casts into water from fishing bud.
smmn digs ingraliznts
tree plunt new trce digs the
mas digging hale to plant luee.
a pan
touch spiip to fry.
add the impreiliens a pan
Water. Hand, Wilk
A man
hand walking water. man bolding water.
mui bolding bottle water in
walks the stret.
Place. Mocll
man uses metul to muiic a puece meul.
woman usds mecal tool to make piece jowelry.
Table 5: Comparison of generated sentences with same concept-set.
Knowledge Probing To investigate how much concept-centric knowledge model pack, conducted two probing methods with our model: Language Model Analysis (LAMA) probe (Petroni al. 2019), Knowledge Intensive Language Task (KILT) (Petroni al. (020). We summarize the results on Table G|and Appendix A.4 We could find that our model outperforms the baseline.
5 RELATED WORK
Self-Supervised Language Representation Pre-Training. Motivated by the fact that words can have different meanings in different contexts, contextual language representation methods (McCann
8

---

Published as a conference paper at ICLR 2021
Methods
MRR
Precision@50
Precision 10
Precision I
Methods
FEVER
AY2
TS-Base
11.53
38.52
21.60
5.93
TS-hase
76.65
74.97
CALM (Mix-only)
11.77
38.93
21.92
6.10
CALM (Mix-only)
77.05
76.27
CALM
12.09
39,69
22.53
6,46
CALM
77.44
77.24
(a) LAMA probe
(b) KILT task
Table 6: Experimental results on Knowledge Probing. Left table shows the mean precision on LAMA probing task of ConceptNET. Right table shows the performance on Fact checking and Entity linking, which are from KILT task.
et al. 2017 Peters et al. 2018) have been developed and shown superior performance on downstream
tasks compared with static word embeddings Mikolov et al. (2013): Pennington et al. (2014). More
recently, large scale language models based on transformer architecture Vaswani et al. 2017) pre-
trained with either mask language modeling objective (Devlin al., 2018 Liu al. [2099] Lan et al.
2019) or mask span infilling objective (Lewis et al. 2019 Raffel et al. 2019) have been explored further advanced the state-of-the-an on multiple NLU and NLG tasks. Our method is based on these techniques and we focus on improving the commonsense reasoning ability of pre-trained text-to-text transformers. More recently, Clark et al. (2020) propose new seff-superviee pre-training objective called Replaced Token Detection (RTD). RTD uses mask language model like BERT to fill in the mask and train a discriminator to predict whether a token is generated or real. This pre-training paradigm is related to our proposed joint training framework. Some major differences include that (1) Our method employs sentence- distractors that are in general grammatically correct but not in line with commonsense, thus require the model to perform relational commonsense reasoning while RTD a token-level discrimination task and can often be solved with syntactic and shallow semantic knowledge (Rosset et al. 2020); (2) Our method unifies generative and contrastive objectives with one model, which can be applied to both NLU and NLG downstream tasks; and (3) The discriminator in framework is "contrastive", takes both the real sentence and the distractor as input simultaneously.
Knowledge-augmentee PTLMs. As standard pre-trained language models usually do not explicitly model knowledge, number of works have examined the problem of incorporating world knowledge with the PTLMs. Recent work Zhang et al.] (2019); Peters et al. (2019); Wang et al. (2020); Liu et al.) (2020) utilizes an external knowledge base to incorporate entity knowledge with PTLMs: however, these approaches require specialized resources like knowledge bases, which limits the domain they can be applied to. Xiong et al. (2020) proposes WikiLM that encodes world knowledge into the parameters of a BERT(Devlin et 20.8))01))likk pre-trained model with novel entity replacement detection objective that incorporates Wikipedia to form distractors. Their approach differs from ours because it requires an external knowledge base (i.e., Wikipedia) which limits the domain it can be applied, is limited to discriminative pre-training objectives and downstream tasks, and focuses on world knowledge instead of relational commonsense knowledge. More recently, (Rosset et al.) 2020) propose KALM, an entity-awart language model with more world knowledge packed into its parameters. Their method is restricted to the training of language models instead of masked language models or text-to-text transformers which can be used for more downstream tasks. Also, all the aforementioned work mainly focuses on world knowledge of named entities. In contrast, our work mainly focuses on commonsense knowledge about quotidian concepts.
CONCLUSION
We propose novel self-supervised strategies that encourage the model to focus on concept-centric information that is related to commonsense understanding and reasoning instead of simple word co- ocurrence patterns so that the commonsense learning capability pre-trained text-to-text transformers can be improved. Despite merely continually pre-trained on a small dataset with only around 20k steps, our CALM model consistently outperforms the T5-base model on all commonsense-rellan and even yields better performance compared with some larger size PTLMS on the COMMONGEN dataset. The performance gain is larger when we use fewer examples for fine-tuning on different downstream tasks, indicating that CALM effectively encodes more commonsense knowledge and rely less on fitting superficial patterns of datasets compared to traditional pre-trained language models. Our work suggests that text-to-text models can be pre- pre-trained with better parameter and sample efficiency by carefully designed sulf-supe objectives that focus more on the ability (c.g., commonsense reasoning ability) required by target tasks.
datasets,
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENT
This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600000 the DARPA MCS program under Contract No. N66001 1924033 with the United States Office Of Naval Research, the Defense Advanced Research Projects Agency with award W91 INF-19-20271, and NSF SMA 18-29268. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In ECCV, 2016.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Meusures for Machine Translation and/or Sunmarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https
7/ www.aaclwebt org anoto logy/005 0909
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. Abductive commonsense reasoning. arXiv preprint arXiv:1908.055: 05739, 2019.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fouth AAAI Conference on Artificial Intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003./0SS: 2020.
Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 58:92-103, 08 2015. doi: 10.1 00.1145/270143.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: we.Xiv:881...0 1840.04805 2018.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuetoo-Wennutee Hon. Unified language model pre- pre-training for natural language understanding and generation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Alhcé-BBunn Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 13063-13075. 2019.
WA Falcon. Pytorch lightning. GitHub. Note: http://gttaakkoooooyyyyylgggiiig 3, 2019.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval- augmented language model pre-training. arXiv preprint 2000:2000,.0909, 2020.
Johannes Hoffart, Mohamed Amir Yosef, Ilaiaa Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In Proceedings of the 2 Conference on Empirical Methods in Natural Language Processing, pp. 782-792, Edinburgh, Scotland, UK., July 2011 1, Association for Computational /at.//g/tttaaaooooee ant ho ogyy/111 1002
Linguistics. URL wnttt:t://// www. ac web org
D. Khashabi, S. Min T. Khot, A. Sabhwaral, O. Tafjord, P. Clark, and H. Hajishirii. Unifiedqa: Crossing format boundaries with single qa system. EMNLP findings, 2020.
10

---

Published as a conference paper at ICLR 2021
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert A lite bert for self-supervised learning of language representations. arXiv preprint 19Xi::900.111422, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.1444 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Ahdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequencceto--sequene pre-training for natural language generation, translation, and comprchension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, July 2020. Association for Computational Linguistics.
Shiyang Li, Jianshu Chen, and Dian Yu. Teaching pretrained models with commonsense reasoning: preliminary kb-based approach. arXiv preprint (rXXv:1909009744 2019.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: constrained text generation challenge for generative commonsense reasoning. Findings of EMNLP, 2020. to appear.
Ye Liu, Yao Wan, Lifang He, Hao Peng, and Philip S. Yu. Kg-bart: Knowledge augpentegeeted bart for generative commonsense reasoning. 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 2019.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pp. 6294- 6305, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 10XXvv:800.0221 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances neural information processing systems, pp. 3111-3119, 2013.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting the Association for Computational I Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguisties. doi: 10:3115/1073083110777 1073313.
URL httpt:t/////mm..... wwww///wwww./... ac web
Szg/anthology. /PO2-20022--20 1040
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint a 1802.05365, 2018.
Matthew E Peters, Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. arXiv preprint 17Xiv:1909.04164 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463-2473 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DD9 1250.
URL https: w/www a c. web. www.org//anthollgg////////

---

Published as a conference paper at ICLR 2021
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao. James Thome, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint wrXi:2200.00225,, 2020.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Huut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint ar arXiv:2005.0063 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910. 2019.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? arXiv preprint (xXi:20002.08900, 2020.
Corby Rosset, Chenyan Xiong, Minh Phan, Xia Song, Paul Bennett, and Saurabh Tiwary. Knowledge- aware language model pretraining, arXiv preprint arXiv:2007.0065: 2020.
Robyn Speer and Catherine Havasi. Representing general relational knowledge in ConceptNet 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC' 12), pp. 3679-3686. Istanbul, Turkey, May 2012. European Language Resources Associ- ation (ELRA). URL http:/ //w::///wwwwwww. 1072_ 1072_Paper... pdf
www. Lrec conf. www. wwwpp///oooeggg//eeeeeeeeeeeeeee Ltrp://ggg//ee2222/pp....
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega:: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937 2018.
James Thome,, Andreas Vlachos, Christos Christodoulopoulos. and Arpit Mittal. FEVER:
large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computatioonl Linguisties: Hu- man Language Technologies, Volume 1 (Long Papers), pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/ 1074. URL NN1-/1118111111
ht https: http://wwwwwww.b.bbr......
www www.waacwebb..... wrg.wwwwwww..........oo.go htt.a//www........aoooooooo/a/..
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL httph:hts///// //ww
/N0-00419
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6000 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus--base image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao. Daxin Jiang, Ming Zhou, et al. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint ar Xiv::2002.00 000.0/80888 808, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface 's transformers: State-of-the--ar natural language processing. ArXiv, abs/1910.03771, 2019.
Wenhan Xiong, Jingfei Du. William Yang Wang, and Veselin Stoyanov. Pretrained encyclopedia: Weakly supervised knowledge- pwweeege-pretraieedd language model. In International Conference on Learning Representations, 2020. URI. httphhttt//////eneet
zm64tDH
12

---

Published as a conference paper at ICLR 2021
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint arXiv:1905.07199, 2019.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 PRE-TRAINING DETAILS
The following details apply to both base architecture and joint-training architecture. We implement our pre-train models using Pytorch-lightning (Falcon 2019) and Hugginface's s Pytorch Transform- ers (Wolf et al. 2019). For pre-training phase, we use the Adam optimizer with maximum sequence length 256, train batch size 8, gradient accumulation 8, warmup steps 10000, weight decay 0.01 and adam epsilon le-6, We train the models with 8 V100 GPUs and FP32 precision for 17 hours. The model is pre-trate pre-trained for most 3 epochs prevent overfitting. We searched for the best learning rate for our model out of [le-4, 2e-5, 2e-6, 5e-7].
A.2
DETAILS
For fine-tuning, we use 4 V100 GPUs and use FP32. For all discriminative tasks, we use the Adam optimizer with maximum sequence length 256, batch size 4 and gradient accumulation 16. For generative task, we use the Adam optimizer with maximum source length 32, maximum target length 32, batch size 8, gradient accumulation 16. For all tasks, we use warmup fraction 0.01. Learning rates and train epochs are listed in Table 7
Hyperparameter
CommonsenseQQA
OpenbookQA
PIQA
aNLI
CommonGENN
Learning rate Train Epochs
[1c-4, 2c-4, 3o-4]
[5e-5, Ic-4, 2c-4, 3c-4]
[Ic-4, 2c-4, 3c-4]
[2o-5,
3e-5]
[2c-5]
20
20
20
10
20
Table 7: Fine-tuning hyperparameters.
A.3 DATASET PROPERTIES
Qommonsense(A (Tammor et al. 2018) is a multiple-choice question answering task, which picks the most appropriate answer general commonsense questions.
OpenbookQA (Mihaylov et al. 2018 is multiple-choice question answering task, which is modeled after open book exams on elementary-level core science questions. The task requires open book fact and additional commonsense which is not contained in the book. To test the commonsense reasoning ability, we do not use open book fact.
PIQA (Bisk et al., 2020) is multiple-choite question answering task, which chooses the most appropriate solution for physical commonsense questions.
aNLJ (Bhagavatula al. explanatory hypothesis given two observations from narrative contexts.
2019 is a bissry-classffication task, which picks the most plausible
CommonGEN (Lin et al. 2020) is constrained text generation task, which generates coherent sentence describing an everyday scenario using common concepts.
Dataset
Train
Development
Test
Source Example
Target Example
CommonieneeAA
9,741
1,221
1,140
context: What home entertainmett equipment requires cable?
options: I: radio vhack 2; nunstationn cobinet 4: television S: deak
OpenbookQA
4,957
500
contest You cue make triezcape with options.
confiic maiting
context:
When botling
It's ready,
PIQA
16,113
30084
options:
a plare Pour
sNLI
168,654
1,532
3,000
content was any hirbuaos. opioon:: 1: vus so exvited. 2: vur so mad.
got the party wus set up for my brenkee
CommaniGeeN
67,389
4.018
6,142
geserate sentence with these concepts. Apple Grow Tier
Apple grons on the toee
Table 8: Properties of Commonsense benchmark datasets.
14

---

Published as a conference paper at ICLR 2021
A.4 KNOWLEDGE PROBING
LAMA probe is consisting of a set of knowledge sources, each comprised of a set of fact. It defines that a pre-trained language model knows fact (subject, relation, object) such as (Bird, CapableOf, Fly) it can predict masked objects in cloze statement such as "Birds can [MASK]". For evaluation, we first filtered out examples that mask label is not in vocabulary list of T5. Then, we evaluate the model based on how highly it ranks the ground truth token against every other word in fixed vocabulary list of TS, and get mean precision at k to check whether the object is ranked among the top k results. We summarize the results of ConceptNet (Speer & Havasi 2012) in Table Unlike other language models which are optimised to masked word anywhere in a given sequence, T5 is trained with different denoising method. It might cause low performance on such slot filling task, but compared to T5, our model shows better performance compared to base model.
KILT task is benchmark for assessing models that need to access specific knowledge in defined snapshot of Wikipedia to solve tasks spanning five domains. The goal is to analyze the model whether it has task-agnostic representations of knowledge. We test our model on domain of fact checking, entity linking. Fact checking verifies textual claims against textual sources. For this task, we use FEVER (Thorne et al., 2018) which is a large dataset for claim veracity that requires evidence from multiple Wikipedia pages to determine whether the claim is supported or refuted. Entity Linking assigns Wikipedia page to entities mentioned in text. We use AIDA CoNLL-YAGO (AY2) (Hoffart eet al., 2011) which supplements the CoNLL 2003 (Tjong Kim Sang & De Meulder 2003 with Wikipedia URL annotations for all entities.
A.5 EXPERIMENTS WITH BART AS BACKBONE
To show that approach is versatile to different pre-trained models, conduct experiments with BART as the backbone model. We can see that our approach consistently and significantly (with p-value 0.01) improves BART-base on all datasets. This result shows that method is versatile to different pre-trained models.
CSQA
OBQA
PIQA
aNLI
CommonGEN
Methods
Accuracy (ofticial dev)
BLEU-4
METEOR
CIDEr
SPICE
Base-buse (Mix-only)
56.31(++0.28) 28) 58.22(++0.21)
67.53(+1.01 18.30(+1.1) 59.10(+100)
1.14))
25.10
29.50
13.16
30.20
CALM (BART-base)
69.40(=1))) 61.28(=0.30)
26.40
29.90
13.71
31.10
Table 9: Experimental results with BART backbone model. Best models are bold.
A.6 EXPERIMENTS WITH NOUN/VERB AS CONCEPTS
We also conducted ablation study about the choice of using either nouns verbs concepts. We can see that using either nouns-only or verbs-only as concepts for our approach leads to substantial performance drop. This supports choice about using both nouns and verbs as concepts.
CSQA
OBQA
PIQA
aNLI
CommonGEN
Methods
Accuracy (official dev)
BLEU-4
METEOR
CIDEr
SPICE
CALM
63.32(=0.355)
71.0((=0.61) 60.90(+0.4)
63.20(+0.522)
26.40
31.40
13.88
33,00
CALM--nuuns 62.45(-+0.42
99.40++..5)
69.05(++0700
61.55(+0.58)
25.70
31.20
13.17
32.60
CALM-verbs
62.51(=0.47)
59.10(+0.7)
69.24(=0.65)
61.40(+0.51)
25.60
31.20
13.24
32.60
Table 10: Experimental results with Noun/Verb as Concepts. Best models are bold.
A.7 HUMAN EVALUATION ON COMMONGEN GENERATIONS
We conducted a human evaluation of CommonGEN predictions between T5 and CALM. We asked three annotators to choose the most reasonable sentence between T5-base and CALM-base predictions. The evaluation was conducted on 50 test sentences in binary selection by majority voting. Cohen's Kappa score, which is a measurement of inter-annotator agreement, was 0.73. Annotators say that for 60% of test sentences, CALM-base generated better.
15


Reviews:
[{"review_id": "ePJMLI-Q-Bn", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThis paper addresses the issue of incorporating commonsense into large pretrained language models. They propose a pretraining method that does not leverage knowledge graphs but rather use approaches which involve corrupting the input sentence in various ways to recover the original sentence. This methodology is used to try to bake in commonsense into models. Results are shown on both discriminative and generative common sense datasets\n\n#### Novelty and clarity\nThe training procedure of corrupting inputs to retrieve outputs is not new but the use on commonsense tasks does seem novel and also is an interesting approach. The paper was very clear to read and the technical aspects were well described.\n\n#### Strengths\n(1) The use of a self-supervised approach is great because it requires no annotation and the training procedure is simple. It is also described well\n(2) The variety of baselines used is good and comparison against models larger than the proposed model is interesting to see.\n(3) The use of generated sentences to improve language models on hard areas like commonsense and offensiveness is a great idea as it can help make the model more robust\n\n\n#### Weaknesses\n(1) There should be a more comprehensive set of results completed to see how much improvement this model has. Mainly on the CommonGen class there should be some manual evaluation done similar in the original paper to see if the outputted sentences make sense and that the improvement in automatic metrics is carried over into human evaluation. \n(2) A key aspect to look into is the robustness of this model. In the C2S approach the concepts were shuffled to generate the correct sentence. During inference time if the concepts were shuffled in a different manner would the model still be able to generate the correct sentences? There was three random seeds used but as was said \"the performance is sensitive to different random seeds.\" which seems that the model isn't as robust to newly seen inputs\n\n#### Detailed Comments\nIf spacy was also used for POS tagging along with tokenization this should be made clear. Also for every sentence was there 3 nouns/verbs extracted? \nOne thing I'm unclear about is in Table 2 why is the \"Our T5-Base\" better than the \"T5-Base\" above? Is this T5 with additional epochs? I think this should be made clear. Additionally I wouldn't say and \"is only slightly worse than KG-BART.\" It seems a lot worse especially on BLEU and CIDER. It is nice to see a smaller model is beating a larger model on some metrics\n\"The difference between CALM and CALM (Joint) is that the former is initialized by the CALM(Mix).\" Did you mean to say latter instead of former? Also I don't see CALM (Join) in the table. I'm assuming this is CALM without Mix warmup\n\n\n#### Questions for the Authors\n1) How did you ensure shuffling the sentences still has grammatical correctness? A sentence like \"Running I am\" is not grammatically correct.\n2) Instead of a POS tagger why did you not use an NER extractor? Also wouldn't swapping different fruits into sentences like replacing \"Apples grow on trees\" with \"Watermelons grow on trees\" help with robustness\n3) And what point is the generative model good enough that it doesn't help to create distractor sentences\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HD2ykaLN-zh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper suggestsan intermediate training regime that can be used between pretraining and the end-task finetuning. The authors suggest that their method captures more commonsense knowledge by being focused on capturing knowledge about “concepts”. Four different denoising objectives—two generative and two discriminative—are proposed and described in detail, with various possible ways of optimizing for all four. Experimental results show improvements over both the base T5 model and the large T5 model. The proposed method achieves SoTA results on CommonGen with slightly more than half the parameters of the current SoTA model.  Ablations show the necessity of applying a 2-step intermediate training scheme with mixed training followed by joint training.  CALM shows better results with less data than the base model. \n\nStrengths:\n- Unifying generative and contrastive training is an important and interesting goal.\n- The objectives suggested are cheap to compute and seem to increase the signal available in the data.\n- Extensive results show improvements over a base model and a larger model across a range of tasks.\n- Performance with relatively little finetuning data are encouraging.\n\nWeaknesses:\n- Somewhat weaker results on some CommonGen metrics are disappointing.\n- Using “concept” to stand in for verbs and nouns is somewhat confusing.\n\nAfter reading other reviews and the authors’ responses to all of the reviewers, I recommend this paper by accepted—extensive results show that the CALM objectives offer more signal from data than current pretraining methods.\n\nThis paper suggests a number of cheap-to-compute corruptions of the input data that, when used to reconstruct the input, enrich the underlying model. These objectives certainly improve over the original T5 base _and_ larges models that are used as initializations, and especially outperform the base model in the low-data regime. The authors use objectives which capture both generative and discriminative information, which some have suggested contain mutually beneficial signal but have not been unified in a single training method.\n\nBelow are two paragraphs from my original review. The authors have done further experiments and show that there are still gains on these tasks when model sized is increased significantly. Furthermore, they have clarified that on the key metric of CommonGen they achieved SoTA with only slightly more than half the parameters of the current SoTA model. I therefore believe that these results show merit.\n> However, in every task except CommonGEN the authors do not discuss any methods that are even close to the state of the art. For CSQA, the best number in this paper is 63.32 vs. 79.5 on the current leaderboard. Similar numbers are true for the rest of the tasks: 60.90 vs. 87 for OBQA, 71.01 vs. 90 for PIQA, and 63.20 vs.  89.70 for aNLI. I do not believe that SoTA results are necessary to write a good paper, and indeed the obsession our field has with SoTA is unhealthy. Yet, it is difficult for me to trust that the effects in this paper will generalize to better performing models without further evidence: what if the CALM intermediate objectives only help with mistakes that larger models do not make in the first place? \n\n> On the generative task CALM performs closer to SOTA, but it improves only slightly on T5. This is especially disappointing as the objectives introduced _directly_ match the task in CommonGEN, making this intermediate training a form of noisy training data rather than pretraining.\n\nI still feel that the authors’ use of “concept” and “commonsense” is vague, when their method can be defined more clearly with more mundane terminology. In practice, the authors use nouns and verbs as their concepts, which is fine in terms of pretraining objectives, but surely does not capture the generality of concepts. The authors have somewhat clarified in this in their updated version.\n\nFinally, the CALM intermediate objectives share many properties with all of the datasets tested on and are likely calibrating the model to the kind of correlations they should expect to predict in advance of finetuning. One way this can be seen is  that the slopes of the T5 and CALM lines are very similar after an initial “bump” which T5 likely needs to calibrate to the new distribution. This makes claims of learning “commonsense” hard to verify, though I do agree that _something_ relevant to solving these problems is clearly being learned.\n\nAltogether, I think this paper makes an interesting contribution to the question of: How can we get the most pretraining signal from unstructured data using off-the-shelf tools? I recommend this paper for acceptance, though I encourage the authors to revise their paper to make this the focus of the story, rather than the vaguely defined notion of “concept”.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GFaBPjbgCWz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes the “Concept Aware Language Model” (CALM) --- a pre-trained T5 Transformer is trained on self-supervised intermediate tasks to learn relational commonsense knowledge, before fine-tuning on downstream tasks. The intermediate tasks include (1) concept to sentence (c2s) generation - given a list of permuted concepts (verbs and nouns), generate the target sequence, (2) concept order recovery (cor) - given a sequence with the order of concepts (verbs and nouns) shuffled, generate the original sequence, (3) given a sequence and it’s perturbed version (concepts shuffled), generate the sequence (classification but as a generation task). Training is first done in a multi-task fashion, followed by a stage of training that uses generated outputs from (1) and (2) for (3).\n\nThe goal of the paper is to show that carefully designed objectives for self-supervised intermediate task training add relational commonsense knowledge which helps improve model performance on downstream commonsense reasoning tasks. \n\nStrengths:\n- S1 - The idea of self-supervised intermediate task training is an exciting one especially in the form of adding reasoning capabilities that BERT, GPT, T5 etc. might not be acquiring during the large-scale pre-training phase.   \n- S2 -  As shown by results in Table 1 on 5 commonsense reasoning tasks, the intermediate task training proposed in this work improves performance over and above the T5 model (and variants including with salient span masking for concepts).  \n- S3 - The experiments involved averaging across 3 random seeds and the authors have reported confidence intervals.   \n\nWeaknesses and questions:\n- W1 - One missing baseline is T5 trained to unshuffle entire sequences - given an input with the tokens shuffled, generate the original sequence. This would show how much value c2s and cor are really adding. The current T5 baselines are all trained purely for infilling which seems a bit unfair compared to CALM which is generating entire sequences.  \n- W2 - Given a T5 model that can score sequences (maybe after training on the autoencoding objective), would it score “apples grow on trees” higher than “trees grow on apples”? If yes, then the model seems to already exhibit this style of reasoning. Would it score “apples grow on trees” and “apples grow in the ground” similarly? The distinction here is between sequences that are non-grammatical or unlikely to ever appear versus sequences that may have appeared (eg: “potatoes grow in the ground”). Presently, (a) it’s unclear if the designed objectives are providing commonsense reasoning above something the model can know from autoregressive language model scoring, and (b) it appears that the objectives are not designed to add relational commonsense knowledge of the sort where we know apples don’t grow in the ground.   \n- W3 - c2s is designed specifically to do well on CommonGen. Are the gains on this task smaller than what you might have expected? If yes, why isn’t it helping more?   \n- W4 - Figure 4 needs error bars, the dev sets are really small and it’s hard to interpret which differences are significant.  \n\n\n\n__UPDATE__\n\nThanks for the incredibly detailed response! I've raised my score to a 8.  \nI do in general quite like the paper, and the responses here are thought-provoking. I'm not sure I'm totally convinced by the WSC results comparing CALM the classifier to T5 the sequence scorer. Not sure if it's an apples to apples comparison...but I'm not sure there's a straightforward setup for this, and perhaps it starts to get beyond the scope of what's being presented here. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0AAGc4nty-t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two self-supervised pre-training tasks to further pre-train a pre-trained language model for commonsense reasoning. The first task is called concept-to-sentence generation, which reconstructs input sentences from noun/verb phrases extracted from the input. The second task is called concept order recovering, which predicts original sentences after shuffling the order of noun/verb phrases in input sentences. Experimental results show that the pre-trained language models fine-tuned with the two proposed tasks can lead to improvement on five commonsense reasoning benchmark datasets.\n\nStrengths: \n\n+ The idea of teaching language models through self-supervised learning tasks is neat.\n\n+ The performance of the proposed methods on few training examples looks great.\n\n+ The results section is well structured. There are ablation studies on the training objectives of each proposed task as well as a comparison of generated sentences.\n\n \nConcerns: \n\n- The key concern about the paper is the lack of rigorous experimentation to study the effectiveness of the two self-supervised learning tasks. First, the methods are only compared with T5-base related methods on the four commonsense classification tasks. The leaderboard of commonsense QA shows that more than 20 systems report an accuracy higher than 63.32, which is the best configuration of the proposed method. Second, the proposed tasks are applied only to T5. I am wondering if it is effective on the other pre-trained language models. Third, the performance improvement on the classification tasks appears marginal. Statistical tests are desirable to show if such improvements are significant.\n \n- Noun and verb phrases extracted from sentences are not always concepts. Masking-out certain words in the input is similar to the idea of removing non-content words from input. A deeper analysis of the proposed method would have been nice to understand which part is effective in the new task, keeping content words or not using mask-out, what if only concepts from a knowledge base is kept instead of content words? \n \n-  The CALM model proposed in this work performs worse than the SOTA models in three out of four metrics on CommonGen, despite it uses less model parameters. What if the proposed tasks are applied on T5-Large?\n\nMinor comments: \n\n* I am not convinced that the generated corrupted sentences are in general grammatically correct, as stated in Sec. 2.1, \n\n* I do not see a strong connection between completing COR and compositional reasoning, as stated in Sec. 2.1.\n\n* The way of getting distractor sentences appears ad-hoc, may need further justification.\n\n* Y in equation (5) and (6) needs explanation.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents two self-supervised learning objectives that can be used as intermediate pre-training tasks to refine the T5 sequence-to-sequence model between pre-training and task fine-tuning. It shows that, at small to moderate model sizes, adding this step significantly improves performance on commonsense-oriented target tasks.

Pros:
- This appears to be a fairly straightforward improvement in self-supervised learning in NLP, with fairly extensive experiments.

Cons:

- This model isn't trained at the same extremely large scales (10B+-words) as state-of-the-art models, and it performs significantly below the state of the art. It's not clear that the released model represents a useful model for any application as-is, and while it's likely, it's not proven that the ideas in this paper would still be useful at larger scales.
- Given that, it seems like the most likely audience for this work is other developers of pretrained models in NLP, which makes the fit to a general ML conference less clear.
- The framing around 'concepts' and, more importantly, the model name 'concept-aware LM', gives the unwarranted impression that the new model handles 'concepts' in a way that T5 doesn't. It is not reasonable to use the word 'concept' to refer to specific parts of speech in your title (even if you later explain that), and whether your model handles concepts in a categorically different way from T5 would take a substantial analysis to show, which doesn't seem to be present. I don't think this paper is up to ICLR's standards with the current name, and urge the authors to change it.

Decision: Accept (Poster)