Analyzing Commonsense Emergence in Few-shot Knowledge Models
Year: 2021

Abstract:
Recently, commonsense knowledge models — pretrained language models (LM) finetuned on knowledge graph (KG) tuples — showed that considerable amounts of commonsense knowledge can be encoded in the parameters of large language models [Bosselut et al., 2019]. However, as parallel studies show that LMs are poor hypothesizers of declarative commonsense relationships [Petroni et al., 2019] on their own, it remains unclear whether this knowledge is learned during pretraining or from fine-tuning on KG examples. To investigate this question, we train commonsense knowledge models in few-shot settings to study the emergence of their commonsense representation abilities. Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining. Importantly, our analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning provides novel insights into how this interface is learned.

text: tuned on knowledge graph (KG) tuples — showed that considerable amounts of commonsense knowledge can be encoded in the parameters of large language models [Bosselut et al., 2019]. However, as parallel studies show that LMs are poor hypothesizers of declarative commonsense relationships [Petroni et al., 2019] on their own, it remains unclear whether this knowledge is learned during pretraining or from fine-tuning on KG examples.
To investigate this question, we train commonsense knowledge models in few-shot settings to study the emergence of their commonsense representation abilities. Our results show that commonsense knowledge models can rapidly adapt from limited examples, indicating that KG fine-tuning serves to learn an interface to encoded knowledge learned during pretraining. Importantly, our analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning provides novel insights into how this interface is learned.

heading: 1. Introduction
text: Driven by advances in pretrained language models [Liu et al., 2019, Raffel et al., 2019], recent NLP systems have demonstrated considerable improvement [Clark et al., 2020, Lourie et al., 2021] on benchmarks that test the commonsense representation and reasoning abilities of NLP systems [Talmor et al., 2019, Sap et al., 2019b, Bhagavatula et al., 2020, Sakaguchi et al., 2020]. Despite this success, it remains unclear how much commonsense knowledge is actually directly encoded by these models. When prompted out-of-the-box to complete commonsense declarative relationships (as depicted in Figure 1), they exhibit limited ability to map their language modeling abilities to this task [Petroni et al., 2019, Zhou et al., 2020].
However, the zero-shot expression of declarative commonsense knowledge is but one use of language that these models are pretrained to manifest. Infilling evaluations that entangle this ability with general language expression may be narrow tests of true commonsense representation ability. Indeed, commonsense knowledge models [Bosselut et al., 2019] — which finetune pretrained LMs on examples from commonsense knowledge graphs [Speer et al., 2017, Sap et al., 2019a, Jiang et al., 2021] — learn to express declarative commonsense relationships much more effectively. This learning procedure allows these models to recalibrate to the actual task of hypothesizing declarative knowledge. Simultaneously, these systems are
evaluated in challenging train/test settings (where head entities tested on cannot be seen during training), indicating they may learn to transfer implicit representations of declarative knowledge that are learned during pretraining [Hwang et al., 2021].
However, current commonsense knowledge models are trained on a large number of examples from knowledge graphs. Consequently, this comprehensive finetuning obfuscates whether these systems exhibit knowledge that was encoded during language pretraining [Geva et al., 2020, Dai et al., 2021], or whether pretraining merely provides a favorable initialization to learn to generalize from examples in the KG. In this work, we explore this question by training commonsense knowledge models in a few-shot setting. We vary the training budgets of KG examples that they receive, and evaluate their capacity to hypothesize commonsense knowledge after training on each of these budgets. Using this framework, we discover whether the finetuning process causes the model to learn new knowledge from the KG, or whether the model learns an interface to existing knowledge it already encodes.
Our results demonstrate that few-shot commonsense knowledge models trained on knowledge graphs in few-shots — up to 10,000× fewer tuples than the full KG — exhibit strong performance relative to zero-shot LMs, and approach the performance of fully-trained knowledge models. Our analysis supports the hypothesis that few-shot finetuning allows the model to adapt knowledge it already encodes implicitly. We find that few-shot training mainly changes parameters in the attention heads of the decoder transformer blocks. The large feed-forward networks that serve as an implicit memory storage for information learned during pretraining [Geva et al., 2020] are minimally updated. Furthermore, we observe that larger knowledge models exhibit most of their parameter change across a more concentrated set of parameters, implying they learn knowledge during pretraining in a less entangled manner due to their capacity. Finally, we find that using natural language prompts to represent relation inputs accelerates commonsense emergence. Our code, few-shot training sets, and models are at https://github.com/allenai/few-shot-comet/.

heading: 2. Background
text: In this work, we investigate few-shot performance and parameter change of commonsense knowledge models to shed light on the dynamics of commonsense representation learning. Below, we describe background concepts to contextualize this analysis. Commonsense Knowledge Graphs are structured, relational representations of commonsense knowledge. In our study, we use Atomic2020 [Hwang et al., 2021],1 a commonsense KG with 1.33M inferential knowledge tuples about entities and events. It represents a large-scale commonsense repository of textual descriptions that encode social and physical aspects of common human experiences. Across its 1.33M tuples, it captures information about 23 relationship types. Example head entities and relations are shown in Table 1. Atomic2020 is split into training, development, and test subsets such that no head entities in one set appear in any other. This property allows models trained on this resource to be evaluated on their capacity to generalize commonsense relationships to new entities, events, and situations. Commonsense Knowledge Models represent facts by learning to encode a commonsense KG [Bosselut et al., 2019]. They are pre-trained as language models and fine-tuned on knowledge graph tuples to learn to hypothesize knowledge relationships through language generation. After training on a large collection of tuples from a KG, they learn the structure and relationships encoded by that KG. Furthermore, because they are seeded with pretrained language models, they learn to generalize the relationships to other entities about which the LM implicitly encodes knowledge [Petroni et al., 2019, Roberts et al., 2020]. Consequently, they can be used to produce precise knowledge on-demand for any entity that can be expressed through language.

heading: 3. Experimental Setup
text: In this section, we outline our experimental setting for training few-shot knowledge models.
Input Each tuple in the Atomic2020 knowledge graph is composed of a {head h, relation r, tail t} triplet. The head and tail entities in the triplet are natural language words or phrases.
1. Downloadable at https://allenai.org/data/atomic-2020
Commonsense knowledge models are trained by providing the tokens of h and r as inputs to the model and learning to generate the tokens of t. In this work, rather than initializing r with a new token and random embedding [Bosselut et al., 2019], we automatically format the input tuples into natural language prompts to represent the relation [Feldman et al., 2019, Jiang et al., 2020]. Table 2 shows examples of such prompts.
Training We source training tuples from the Atomic2020 training set [Hwang et al., 2021]. When constructing few-shot training sets, we set a target number n of examples to randomly sample from the knowledge graph for each relation (i.e., n = 3 =⇒ 3 examples × 23 relations = 69 total training examples). Once a training set of examples is produced, the model is trained on this subset to minimize the negative log-likelihood of the tokens of the tail entity for each tuple. We use the AdaFactor optimizer [Shazeer and Stern, 2018] with a constant learning rate of 0.001, a mini-batch size of 4, and train the model for 3 epochs. Unless stated otherwise (§5.3), we use T5-11B [Raffel et al., 2019] as a seed language model for all experiments. To pick hyperparameters during fine-tuning, we run 5 different sets of hyperparameters and use a validation set of equivalent size to the training set (n examples / relation) to evaluate performance [Perez et al., 2021].
Evaluation We evaluate the knowledge hypothesized by the trained few-shot models using human and automatic evaluations. For the human evaluation (Accept % in Table 3), we use the procedure described in [Hwang et al., 2021]. We ask annotators to label the plausibility generated tuples using a 4-point Likert scale: {always/often true (+2), sometimes/likely true (+1), false/untrue (−1), nonsensical (−2)}. We collect 3 annotations per relation, convert each annotation to an acceptability label (i.e., {+1,+2} → 3, {−1,−2} → 7) and use the majority label as the acceptability judgment. Evaluation agreement is measured using Fleiss’s κ = 0.49 for the acceptability judgment. We also evaluate hypothesized knowledge tuples using automatic metrics: BLEU-1 [Papineni et al., 2002], METEOR [Banerjee and Lavie, 2005], ROUGE-L [Lin, 2004], and CIDEr [Vedantam et al., 2015]. For few-shot experiments, we report average performance across 5 runs with different training sets.

heading: 4. Do few-shot knowledge models learn?
text: We evaluate the general few-shot commonsense interface learning capability of large-scale language models. We train COMeT models as described in §3 and set the number of training
examples per relation to n = 3. We report several zero-shot and few-shot augmentation baselines (e.g., GPT-{2 XL, 3}), which prepend n training examples (for the same relation) to the start of the input sequence as a prompt, and condition the generation on these additional examples. Few-shot augmentation baselines are not fine-tuned, but receive examples using the same prompt formats as the finetuning baselines (Tables 2, 8). Similar to fine-tuning, we report average performance across 5 different example sets used for prompting. As T5 models are not pretrained using a causal language modeling objective, reporting zero-shot evaluation performance for a multi-word generation task would be an unfair comparison, so we only report limited numbers in Table 6 of Appendix A. We also report the results of several fully supervised COMeT knowledge models.
Findings We find that both the few-shot augmentation and few-shot learning settings are able to produce high quality commonsense knowledge tuples, indicating that large LMs are also efficient few-shot learners (in addition to showcasing impressive few-shot prompting abilities). Using only n = 3 tuples per relation, both GPT-3 and COMeT (T5) produce highquality tuples that are accepted as plausible by human evaluators more than 70% of the time — 75.7% for COMeT (T5). While this performance falls short of the fully supervised knowledge model performance — 86.4% — it demonstrates that the commonsense representation abilities of language models extends much further than their zero-shot prompting abilities. Furthermore, these abilities can be measured by finetuning on limited examples, supporting the hypothesis that early-stage finetuning serves to learn an interface to knowledge learned during pretraining. In the following section, we support this hypothesis by investigating how parameters of large language models change during finetuning (across different training budgets n §5.2, different model size §5.3, and different prompting strategies §5.4).

heading: 5. How do knowledge models learn?
text: Despite strong empirical results showing that pretrained language models can rapidly adapt to become few-shot language learners [Schick and Schutze, 2021a], little work has focused on how language models learn to adapt knowledge learned from pretraining to perform well on downstream end tasks. We explore this problem in the context of few-shot knowledge modeling and investigate how the parameters of few-shot commonsense knowledge models
change during finetuning. We define three measures to investigate parameter change: absolute parameter change, angular parameter change, and distribution of parameter change.

heading: 5.1 Measuring parameter change
text: Notation To more easily discuss measures of parameter change during knowledge model fine-tuning, we review the structure of the transformer blocks in an encoder-decoder transformer LM [Vaswani et al., 2017]. Encoder blocks consist of a self-attention layer followed by a feedforward network. Each self-attention layer contains parameter matrices (referred to in the following sections as q, k, v) that project the query, key, and value inputs before computing a scaled dot-product attention between the key and value, and an output projection o of the attended value vectors. The trainable parameters in the feedforward network consist of two linear projection matrices wi, wo. Each decoder block consists of the same parameter matrices with an additional cross-attention layer (parameter matrices xq, key xk, value xv, and output xo) that attends to the outputs of the encoder blocks.
Absolute Parameter Change To measure a normalized quantity for average change in each parameter matrix, we compute a normalized `1 distance for each set of parameter matrices in the transformer blocks. Without loss of generality, for each of these parameter matrices, we define ΘPT as the matrix of parameters before finetuning, and ΘFT as the matrix of parameters post-finetuning. The normalized `1 distance between these two matrices for each parameter matrix type is then:
d`1 = 1
mn
∥∥ΘFT −ΘPT∥∥ 1 , (1)
where m and n correspond to the row and column dimensionality of the parameter matrix, and ‖·‖1 corresponds to the `1 norm. Intuitively, d`1 measures the degree of absolute per parameter shift between these two sets of parameters.
Angular Parameter Change However, measuring normalized `1 distance solely captures how the magnitude of the parameter matrix changes, but obfuscates relative drift between parameters in each matrix. For instance, a matrix with high average L1 change may in fact have small relative changes between its elements (i.e., the parameter change may mainly cause a scaling effect on the output vectors). Consequently, we measure the row-wise angular distance between ΘPT and ΘFT to capture the rotational drift of parameters during finetuning:
dang = 1
mπ m∑ k=1 cos−1 ( 〈ΘFTk ,ΘPTk 〉∥∥ΘFTk ∥∥2∥∥ΘPTk ∥∥2 ) , (2)
where k indexes a common row vector in both parameter matrices, 〈·〉 corresponds to an inner product, ‖·‖2 is the Euclidean (`2) norm, and cos−1 is the inverse cosine operation. Contrary to d`1 , which represents a per-parameter average distance, dang is computed between all corresponding rows of the parameter matrices, ΘPT and ΘFT , and averaged per row. We choose to average rows as opposed to columns as each row in the parameter matrix performs a dot product with the input vector to the transformation, and has a correspondence to an individual element in the output vector.
Distributional Parameter Change In additional to angular distance, we also report the distribution of parameter changes for a given matrix. More concretely, parameter changes may be concentrated across varying numbers of dimensions. We compute the change in each dimension by calculating W = |ΘPT −ΘFT |, sorting parameters by their absolute change, and plotting % cumulative parameter number against % cumulative mass as a function of a given parameter change wi ∈W :(
count(w ∈W,w ≤ wi) count(w ∈W ) ,
∑ w∈W,w≤wi w∑
w∈W w
) (3)
Figure 3 visualizes this relationship. Intuitively, a matrix with a small AUC has a skewed relative distribution of changes, signaling that a select number of dimensions are responsible for a large amount of the learning during fine-tuning. Likewise, a matrix with a large AUC learns a relatively even parameter change distribution.

heading: 5.2 How does few-shot learning change the parameters of the pretrained LM?
text: We first investigate how different example budgets affect parameter change. We train a few-shot COMeT (T5) model across different example budgets (n ∈ {3, 30, 300}). Table 4 shows that performance increases monotonically as we grow the example budget. To explore the differences between models, we investigate the parameter changes between each fine-tuned model in Fig. 4. We include additional heatmaps for angular change in the Appendix.
Findings In Figure 4, we observe that the `1 distance between ΘFT and ΘPT increases as we increase the training budget in the decoder. Interestingly, we note that most parameter changes occur in the decoder of the LM (and more strongly in its later layers), rather than the encoder, implying that much of the parameter shift may be due to learning how to express commonsense knowledge in a declarative form. Furthermore, when comparing the changes across parameter matrix types, the key and value matrices of the attention layers change more while the feedforward network parameters (wi, wo) remain relatively unchanged. These findings suggest the model learns how to process the structure of declarative commonsense knowledge expressions, but does not need to modify its stored representations of this knowledge learned during pretraining, hinting at a possible explanation for the success of adapters for limited-parameter finetuning of NLP models [Houlsby et al., 2019].

heading: 5.3 How does model size affect knowledge model learning?
text: We train a few-shot COMeT (T5) model (with n = 30) across different pretrained language model sizes, i.e. Small, Large, and 11B, with 60M, 770M, and 11B parameters, respectively.
Findings In Table 4, we observe that the importance of model size decreases as the training budget increases, reinforcing that model scale is critically important in few-shot learning settings [Brown et al., 2020]. When examining the parameter change heatmaps in Figure 5, we note the `1 distances between finetuned and original parameters increase as model size increases. As these distances are normalized by the number of parameters, this observation implies that when trained on the same examples, the smaller model experiences smaller magnitude gradient updates over time. This pattern indicates more destructive inference between parameter updates in the smaller models, an observation supported by the larger angular parameter changes in the encoders of the smaller models.
Interestingly, several parameter matrices display high angular change but low `1 change (e.g., in early wo layers of T5-11B), suggesting that a matrix with low average absolute parameter change may still contain critical dimensions which are used to learn the task. To investigate these differences, we look at the AUC heatmaps (Fig. 5, bottom row), and note that as the feedforward networks get larger with increasing model size, the relative number of parameters that change to encode new information drops drastically in both the decoder and encoder. Furthermore, the lower AUCs for the encoder parameter changes indicate that the encoder layers generally experience more concentrated changes as the model size increases.
These two patterns suggest that the increased capacity of larger language models allow them to encode knowledge in a less distributed manner, making their knowledge easier to access.

heading: 5.4 How do prompts influence knowledge model learning?
text: Recent work has shown that prompts can help models elicit knowledge from pretrained language representations [Feldman et al., 2019, Shin et al., 2020]. However, eliciting knowledge through zero-shot prompting has drawbacks. For example, the output is sensitive to subtle variations in the construction of the language prompt [Jiang et al., 2020]. Here, we explore whether prompts can accelerate few-shot learning by initializing natural language prompts for each relation (Table 2) training on these expressions of relations rather than initializing new relation token embeddings for each relation as in Bosselut et al. [2019]. As a control for whether the model understands the semantics of specific relation prompts, or merely benefits from arbitrary expressions of relations using language, we also train a model where we shuffle prompts among the relations (i.e. shuffling the Prompt column of Table 2).
Findings In Table 5, we see that knowledge models can efficiently learn from fewer examples when relations are represented using correct natural language prompts (∼10×
fewer examples), an observation made by contemporaneous works on other tasks [Scao and Rush, 2021]. Interestingly, however, we find that the absolute and angular distance heatmaps for prompt and embedding relation inputs are similar (Fig. 6; `1 heatmap in Appendix), implying that most of the parameter change during fine-tuning is not linked to the way the relation is represented. One notable difference is that models with embedding relation inputs have a significant angular change in layer 4 of the decoder wo matrix. Models that use prompts see a larger angular change in layer 2 of the decoder wo matrix. This discrepancy suggests that both models must learn to adapt their parameters to the new input format, but do so by adapting the same parameters at different layers. We note that there is no such concentrated angular change for the Shuffled setting, perhaps because the model must first unlearn language relationships between head entities and prompts, requiring more uniform parameter updates.

heading: 6. Related Work
text: Commonsense Knowledge Models Our work uses commonsense knowledge models, first proposed by Bosselut et al. [2019], to learn from commonsense knowledge graphs. Hwang et al. [2021] also trained commonsense models on Atomic2020, but focused on fully-supervised learning. Other works have developed commonsense knowledge models that are grounded to visual scenes [Park et al., 2020, Da et al., 2021], requiring multimodal commonsense inference generation. Recent works extend commonsense knowledge models beyond generating singlehop inferences and generate multi-branch [Bosselut et al., 2021] and multi-hop [Wang et al., 2020b] inferential structures. Commonsense knowledge base completion is also a closely related task to commonsense inference generation [Li et al., 2016, Saito et al., 2018]. Recent
works on this task combine language and graph structure representations for improved generalization [Malaviya et al., 2020, Wang et al., 2020a]. Our work differs from these prior studies by investigating commonsense knowledge models in few-shot settings, and analyzing the emergence of their capabilities as a function of parameter change.
Few-shot Learning with Prompting In recent years, the term few-shot has taken two meanings: the classical definition of training on limited examples [Fei-Fei et al., 2006, Fink, 2005], and a new in-context definition where models are given examples as a prepended augmentation of their context and process these examples as input to recognize the structure of a task [Brown et al., 2020]. Various contemporaneous works related to ours have studied the use of natural language and artificial prompts in language models [Schick and Schutze, 2021a, Gao et al., 2020, Tam et al., 2021, Schick and Schutze, 2021b] as accelerators of few-shot learning. Scao and Rush [2021] further empirically study the data efficiency advantage gained from using prompting to represent a task. Our work differs from these studies in that we do not use prompts to map a classification task to masked language modeling, but instead use prompts as priming text for a generation task. Most similar to our approach is perhaps Schick and Schütze [2020], which uses prompts for few-shot summarization, but their work does not focus on how few-shot training affects the learned representations of their model.

heading: 7. Conclusion
text: In this work, we propose few-shot commonsense knowledge modeling, where models are finetuned on a limited number of examples to evaluate how efficiently they can adapt their pretrained representations to the task of hypothesizing commonsense knowledge tuples. Our results demonstrate that large language models require few examples to adapt their learned representations to the task and our analysis explores how their parameters change to enable this rapid emergence of commonsense representation ability.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their feedback, and the Amazon Mechanical Turk community for help with annotation. The authors thank Vered Shwartz and Chris Manning for feedback on early drafts. We also gratefully acknowledge the support of DARPA under No. N660011924033 (MCS), JD.com, and the Allen Institute for AI. TPU machines for conducting experiments were provided by Google.

heading: Appendix A. Accuracy in zero-shot MLM setting
text: While little work has explored few-shot knowledge completion, recent works have investigated performance of zero-shot knowledge graphs [Petroni et al., 2019, Feldman et al., 2019]. Thus, we investigate the ability of T5-11B to complete commonsense knowledge in a zero-shot setting.
We use prompts to leverage the masking objective of the language model pretraining. Since the mask only predicts several tokens at a time, for relations with longer length tail entities, we allow the model to predict up to 7 mask tokens in succession, or until the model predicts an empty string for the mask. We suggest that this is still only a workaround, and masked models are poor predictors of longer length tail entities, as indicated by our results above.

heading: Appendix B. Additional Results on the Effect of Relation Input Format
text: In Table 7, we provide further experimental results on the effect of relation input format on few-shot performance. Namely, we extend the results from Table 5 with the case n = 300. These results confirm that knowledge models can efficiently learn from fewer examples when relations are represented using natural language prompts. We also show the results from using paraphrases of the main prompts for training (original and paraphrased prompts can be found in Tables 8 and 10, respectively). We find that the paraphrased prompts do cause a slight drop in performance, but that this drop is generally close to the margin of error, indicating that while prompt formulation is an important consideration [Jiang et al., 2020], fine-tuning on the prompts does make the model less sensitive to prompt variations.

heading: Appendix C. Additional Experiments on Parameter Change Measures
text: In addition, we provide extensive results of the `1 and angular distances, as well as the distributional parameter change metric (AUC), for both the encoder and decoder of various model sizes (Small, Large, and 11B) and different example budget (n ∈ {3, 30, 300}) (Figures 7- 36). When computing AUC diagrams, we round each weight change to the nearest 10−5.

Review summaries:
This paper analyzes the emergence of commonsense knowledge in few-shot knowledge models. In particular, it attempts to answer whether and when is this knowledge learned: during pretraining or from fine-tuning on KG examples. To investigate this, various commonsense knowledge models are trained in few-shot settings. The results and analysis in the paper shows that commonsense knowledge models can adapt from limited examples. Moreover, similar to a recent line of work, this work confirms that KG fine-tuning learns an interface to the commonsense knowledge learned during pretraining. Finally, the paper also presents an analysis of absolute, angular, and distributional parameter changes during few-shot fine-tuning. However, the insights here are less convincing to me.

This paper analyzes fine-tuning of pre-trained generative language models on a commonsense knowledge task, investigating how much the strong performance of the models is due to pre-training or to task-specific learning (via fine-tuning or prompting/in-context learning). The task is to predict natural text for the tail of (head, relation, tail) relationships from the Atomic_20^20 knowledge graph (KG), e.g. ("PersonX goes to the mall", xIntent) -> "to buy clothes". The paper shows that a T5-based model obtains a substantial fraction of the performance of fine-tuning on the full Atomic dataset, using few-shot learning with only a small number of examples (3) per relation. Few-shot learning also obtains a substantial improvement over zero-shot performance for this model. The paper also presents several measures of parameter differences between pre-trained and fine-tuned models and performs an analysis with them, showing that more parameters are updated in the models' decoders than in their encoders. Taken together with past work, the evaluation results and analysis support the paper's claim that most of the knowledge useful for the KG task was already present in the pre-trained models.