Can In-context Learners Learn a Reasoning Concept from Demonstrations?
Year: 2023

Abstract:
Large language models show an emergent ability to learn a new task from a small number of input-output demonstrations. However, recent work shows that in-context learners largely rely on their pre-trained knowledge, such as the sentiment of the labels, instead of finding new associations in the input. However, the commonly-used few-shot evaluation settings using a random selection of in-context demonstrations can not disentangle models’ ability to learn a new skill from demonstrations, as most of the randomly-selected demonstrations do not present relations informative for prediction beyond exposing the new task distribution. To disentangle models’ in-context learning ability independent of models’ memory, we introduce a Conceptual few-shot learning method selecting the demonstrations sharing a possiblyinformative concept with the predicted sample. We extract a set of such concepts from annotated explanations and measure how much can models benefit from presenting these concepts in few-shot demonstrations. We find that smaller models are more sensitive to the presented concepts. While some of the models are able to benefit from conceptpresenting demonstrations for each assessed concept, we find that none of the assessed incontext learners can benefit from all presented reasoning concepts consistently, leaving the incontext concept learning an open challenge.

heading: 1 Introduction
text: In-context learning (ICL) is the alternative to the conventional training of Large Language Models (LLMs) for specific task(s), where models are expected to learn a new task solely from the input text. In few-shot in-context learning that we focus on, the input text contains a set of demonstrations, i.e. the input-output examples of the task to be learned (Brown et al., 2020).
An ability to learn unseen tasks from natural instructions has practical and theoretical implications,
Does the following hypothesis ENTAIL or NOT ENTAIL the premise?
Premise: "Writing Java is not too different from programming with handcuffs." Hypothesis: "Writing Java is similar to programming with handcuffs." Label: ENTAIL
Premise: "The market is about to get harder, but not impossible to navigate." Hypothesis: "The market is not about to get harder, but impossible to navigate." Label: NOT ENTAIL
Premise: "If the scheme does not correspond, a negative impact on the results would be expected." Hypothesis: "If the scheme does not correspond, it would not be unexpected for it to negatively impact the results." Label:
Does the following hypothesis ENTAIL or NOT ENTAIL the premise?
Premise: "The cat sat on the mat" Hypothesis: "The cat did not sit on the mat" Label: NOT ENTAIL
Premise: "Some dogs like to scratch their ears." Hypothesis: "Some animals like to scratch their ears." Label: ENTAIL
Premise: "If the scheme does not correspond, a negative impact on the results would be expected." Hypothesis: "If the scheme does not correspond, it would not be unexpected for it to negatively impact the results." Label:
In-context Learner ENTAIL (correct)
NOT ENTAIL (incorrect)
Demonstrations sharing concept (double negation)
Random demonstrations
Pr ed
ic te
d sa
m pl
e D
em on
st ra
tio ns
In st
ru ct
io n
Figure 1: In this work, we assess In-context learners’ ability to improve when presented with demonstrations using a reasoning concept applicable in the prediction (§2). We extract these concepts from human explanations (§3.2) and assess models’ ability to learn to use these concepts, as reflected in improving their prediction quality.
both of which are of great significance; Understanding free-form user requests allow applying LLMs in applications of restricted, or limited data availability without over-specialization (Goodfellow et al., 2014). In-context learning can provide a handle of models’ behaviour, enabling the model to avoid specific erroneous predictions. In theory, a training process resulting in accurate new-task learner defines the sufficient conditions for the emergence of a specific level of generalization.
Recent LLMs trained on vast mixtures of tasks (Sanh et al., 2022a; Wang et al., 2022b; Chung et al., 2022) show a certain level of new-task ICL and gradually bring more attention and expectations in this direction. However, counterintuitively to the overall evaluations, in-context
107
learners (ICLs) also expose surprising behavioural artefacts; Liu et al. (2022) show ICLs’ sensitivity to the ordering of in-context demonstrations. Similarly, Lu et al. (2022) find surprising sensitivity of ICLs to the specific wording of the prompts. Min et al. (2022b) show that most of the model performance is persisted even when the contents of the demonstrations are randomly swapped. Contrary to the ability to learn from input, Wei et al. (2023) propose to attribute this to the over-reliance of incontext learners on semantics of the label tokens, especially in smaller models.
We find that the discrepancy in the perceived abilities of ICLs might be attributed to their limited evaluation, commonly performed with a random set of task demonstrations. However, for many open-ended tasks, such as Question Answering, or Translation, randomly-chosen demonstrations rarely present a reasoning pattern which can help with the prediction of new input (Figure 1; Right). We argue that the evaluation with mostly non-informative contexts also can not reflect on the ability of learning, as observed in humans1, as the gain of extrapolating associations presented in non-informative demonstrations can only bring little benefit to the practice.
We note that in the absolute numbers, the random-demonstrations evaluation also favours very large LLMs with a capacity to remember a wider variety of input distributions from pretraining; Conditioned by the capacity, very large LLMs can better modulate the behaviour based on demonstrations’ distribution, instead of learning new association(s) from the context.
Hence, in Section 2, we propose to evaluate models’ in-context learning ability primed with the demonstrations that exhibit a reasoning analogical to the one required for a robust prediction of the predicted sample (Fig. 1). We measure how well can the recent few-shot learners utilize identified concepts for more accurate predictions (§3) and find large discrepancies among the models and concepts.
Our main contributions are following: (i) We introduce a task of Conceptual Few-shot Learning, disentangling models’ ability to learn a new reasoning concept from other aspects of prediction quality. We show how such reasoning concepts can be extracted from human explanations. (ii) For a
1We restrain from discussing a concept of learning in the psychological scope, but we note that Concept learning fits well into a definition of Associative learning (Plotnik, 2012).
wide variety of recent in-context learners, we measure the ability to benefit from presented reasoning concepts. We show that while some models are better at learning concepts on average, this ability can not be attributed to the models’ size or training strategy.
Problem Definition Given a dataset D : {(x1 → Y1), .., (xi → Yi)} ∈ D containing pairs of input xj with associated label Yj , an in-context few-shot learner Θ(x) → y aims to predict a correct label yk+1 = Yk+1 given a sequence of k input-output demonstrations, and the predicted input xk+1:
Θ([x1 → Y1, .., xk → Yk], xk+1) → yk+1 (1)
We expect in-context few-shot learner Θ to model the relation of xi and yi by (i) identifying and (ii) extrapolating the relations of input and output presented in demonstrations. Each such relation is modelled by one or more latent concepts C:
∀ (xi, Yi) ∈ D : ∃ C : C(xi, Yi) = 1 (2)
We broadly define a concept C as any function C(x, y) → {0, 1}, constraining a space of valid outputs y to the ones where C(x, y) = 1. Thus, if Θ learns a concept C, it will never predict for x such y that C(x, y) = 0. In a composition {C}= {C1, .., Cj}, all Ci∈{C} must evaluate to 1.
Given that modelling of each C valid for the task of D restrain a set of possible predictions of Θ exclusively from incorrect predictions, extending a set of concepts learned in-context with complementary one(s) should never decrease the performance of the model Θ on D.

heading: 2 Conceptual Few-shot Learning
text: We reformulate in-context few-shot learning (1) to a conceptual few-shot learning, evaluating the ability of a few-shot learner Θ to identify and apply a user-chosen reasoning concept C shown in demonstrations. First, we classify evaluation samples such that the samples of the same category Xi require the concept Ci to map x to Y . Subsequently, in conceptual few-shot learning, we let the learner to infer a prediction for input xk+1 by presenting it with demonstrations (xj → Yj)1..k ∈ Xi, thus sharing the reasoning concept Ci with the predicted input xk+1:
Θ([x1 → Y1, .., xk → Yk], xk+1) where ∀(x1..k, Y1..k) ∈ Xi and xk+1 ∈ Xi (3)
We note that Θ can rely on other features than Ci, and such reliance is not easy to disentangle. Therefore, we propose to contextualize the results of Conceptual few-shot learning on a concept Ci a difference to the performance obtained in a random selection of demonstrations.
Additionally, to make the predictions based on two different sets of demonstrations mutually comparable without systematic bias (e.g. in samples’ complexity), we perform both random and conceptsharing evaluations with the same predicted samples xk+1, and only change the demonstrations2.
Informative Concepts Extraction Constructing a scaled evaluation with annotated reasoning concepts C is challenging since the annotations of such concepts in associated with the datasets are rare.
However, we find such reasoning inherently captured in human explanations of some datasets, where annotators are asked to collect answers to a question “why is [input] assigned [output]?” (Wiegreffe and Marasović, 2021).
The form of these explanations ranges from free-text explanations, including annotator-specific slang and stylistics, to semi-structured and structured explanations, cast to a pre-defined format, often consisting of a set of relations in a form “[subject1] [relation] [subject2]” that transitively maps [input] to [output] (Jansen et al., 2018). We focus on extracting the concepts from the subset of the semi-structured and structured explanations where the format consistency and non-ambiguity of the operands are reassured.

heading: 3 Evaluations
text: This section introduces few-shot learners that we evaluate for Conceptual few-shot learning and the datasets allowing us to extract reasoning concepts.

heading: 3.1 Few-shot Learners
text: T0 (Sanh et al., 2022b) introduce a set of incontext learning models fine-tuned from a T5 model (Raffel et al., 2020) on a variety of tasks in zero-shot settings, aiming to perform well on a task of previously-unseen categories. T0 is trained for seq2seq generation over a large set of diverse tasks cast to a unified input-output format provided by task-specific templates of Promptsource project (Bach et al., 2022).
2The implementation of Conceptual few-shot learning is available on https://github.com/MIR-MU/CoAT.
TK-INSTRUCT (Wang et al., 2022a) is a set of models trained for comprehension of annotatorlike instructions, consisting of a free-text task description and a set of input-output pairs, collected for more than 1,400 tasks of NATURALINSTRUCTIONS collection (Mishra et al., 2022). Note that, in contrary to T0, TK-INSTRUCT models can advance from being trained in the few-shot learning format, where the model was exposed to the format of a few input-output examples already in the fine-tuning.
FLAN (Chung et al., 2022) scales the approach of fine-tuning in a few-shot learning format to over 1,800 tasks of 146 categories including all resources of T0 and TK-INSTRUCT. Contrary to the former models, the training data mixture includes several datasets with chain-of-thought labels, where the model is trained to follow the annotated reasoning chain explicitly. We evaluate all publicly available T5-based FLAN models.
GPT3 (Brown et al., 2020) is a well-known causal language model that has first shown that in-context few-shot learning ability can emerge solely from vast amounts of unsupervised training data and parametrization, without fine-tuning. Alternatively to other approaches, INSTRUCTGPT (Ouyang et al., 2022) fine-tunes GPT3 to follow human instructions using obtained user feedback. We evaluate both these models through OpenAI APIs3.

heading: 3.2 Datasets
text: Following is a description of datasets that we use in Conceptual few-shot evaluation. Note that for each dataset, we highlight a single concept that we use in Conceptual few-shot evaluation as the C (§2). In the case of each model and dataset, we first evaluate all templates available in Promptsource and report the gain of utilising the chosen concept for the best-performing template.
WorldTree (Jansen et al., 2018; Xie et al., 2020) is a collection of 5,114 science exam questions with the explanations in the form of 9,216 shared facts supporting the assignment of the correct answer.
We use the shared facts as the concepts C and evaluate with the demonstrations of a maximal facts’ intersection with the predicted sample. Contrary to the other datasets, in WorldTree evaluation,
3https://beta.openai.com
we prepend the facts for all the demonstrations in the context before the demonstrations.
OpenBookQA (Mihaylov et al., 2018) is a collection of elementary-grade single-choice questions requiring common sense knowledge about the world. A set of 4,957 explanations take the form of a triple of (object, relation, object), such as “a stove generates heat” for a question “Which one of these can help a person cook their food? [four options]” and a correct option “a counter cooker appliance”.
To extract informative concepts C, we perform syntactic analysis of the explanation and extract
the relation, identified as a root of the sentence’s parse tree. Hence, in conceptual few-shot learning, we prime the aforementioned question with other question-options-answer pairs of the questions answerable by relating the input to output through the “generate” relation.
HotpotQA (Yang et al., 2018) is a QA dataset composed of questions requiring the QA model to jointly reason over multiple passages of multidocument contexts. Inoue et al. (2020) enrich the dataset with explanations from three human annotators. The explanations are structured in the form of triples (e1, r, e2), associating two entities (e1 and e2) through a relation r, such as (“Scott Derrickson”, “is”, “an American director”).
We extract the shared concepts C as pairs of (r, e2); Hence, Conceptual few-shot will prime the prediction with questions and contexts presenting the same entities in analogical relations to the ones the model should understand for correct prediction.
GLUE Diagnostic (Wang et al., 2018) contains approximately 1,100 diagnostic samples of Natural Language Inference intended to fool a simple statistical model. While the concepts are heuristically extracted in other cases, GLUE diagnostic directly annotates 30 distinct logical concepts needed in prediction, such as double negation, conjunction, or existential quantification. We directly use these logical concepts as the reasoning concepts C.

heading: 3.3 Baseline model (BASELINE-TK-QA-1B)
text: To contextualize the results of existing In-context learners, we additionally evaluate a simple newlycreated few-shot in-context learner trained on a single QA dataset. Similarly to TK-INSTRUCT, we construct the training examples of the metalearning task in the explicit few-shot learning format, as initially proposed by Min et al. (2022a), where the model is updated to predict correct labels with a set of randomly-selected demonstrations included in the input (Eq. (1)). This way, we fine-tune a T5-LARGE model (Raffel et al., 2020) on AdversarialQA dataset (Bartolo et al., 2021) until convergence on a validation split. We assess the resulting model on Conceptual few-shot learning together with other in-context learners, denoting its results as BASELINE-TK-QA-1B.

heading: 4 Results and Discussion
text: Figure 2 shows the change of models’ error between a random selection of demonstrations and Conceptual few-shot learning, i.e. with demonstrations sharing a selected concept (§2), ordered by models’ size.
For each of the assessed concepts, we observe statistically significant improvement for at least one of the models, which confirms our initial assumption on the informativeness of the extracted concepts in prediction.
However, we can see that the selection of demonstrations makes a large difference in many cases, and the difference also largely depends on the inspected concept. Following the results of specific models, we see many cases where the model is able to utilise one concept but fails to utilise, or even worsen the prediction then exposed to the other. The variance is larger for instruction-tuned TK-INSTRUCT models, excelling in utilising shared reasoning logic of GLUE, but to the contrary, degrading when being exposed to demonstrations supported by the shared facts in WorldTree. Contrary to these results is the case of InstructGPT that is agnostic to concepts except for GLUE.
Figure 3 shows the average of changes of Conceptual few-shot evaluation over the inspected four concepts. The aggregation uncovers that the gain from providing informative demonstrations largely varies among models, with T0 and smaller models (≤3B) benefiting from the presented concepts slightly more often; This could be caused by larger models’ increasing reliance on their memo-
rized knowledge. However, within the model-type groups, we also note that this trend is disputed by T0 and FLAN models.

heading: 5 Conclusion
text: This work introduces a task of conceptual few-shot learning that reflects on in-context learners’ ability to learn to apply a specific reasoning concept that can be informative for prediction. We assess a set of recent in-context learners for this ability over a set of concepts extracted from human explanations.
We find that none of the learners can benefit consistently from all concepts, even though at least one of the other models proves the concept to bear an informative value. Despite that, we still observe some interesting trends, such as the models of T0 are able to benefit from the concepts more often than others or that the concept-learning ability does not appear to relate to the model size.
We believe the future work can inspire in identifying possibly complex reasoning concepts in the explanations of human annotators and will scale the conceptual evaluation to a wider variety of concepts. We trust that an evaluation with a comprehensive selection of the concepts will allow us to more realistically assess the abilities of the newlydesigned language models in the fast-progressing development of new in-context learners.
Limitations
Concepts In this work, we extract the concepts from semi-structured explanations whose format reassures consistency and non-ambiguity of the exploited concept(s). The selection of datasets and corresponding concepts is primarily conditioned by data availability, as the semi-structured explanations are available merely for a small set of datasets.
We acknowledge that our selection of concepts is not representative for a vast variance of concepts that users might expect models to learn from context in interaction. Some important concepts’ features that we identify are following: (i) a number of premises or reasoning inference steps needed to map the input to output, (ii) the granularity of the reasoning steps, (iii) a type of the premises; For instance, whether the familiarity with a given concept requires a memorization of an entity property (such as “sun emits light”), or a reasoning mechanics such as analogical reasoning (“if animals can run and cat is an animal, then a cat can run”).
We invite future work to identify or propose a
taxonomy that would better reflect the wide variance of reasoning concepts that models are expected to comprehend in order to serve a wide scope of unseen tasks. Such taxonomy can motivate a more targeted collection of concepts from explanations, or annotation of new explanations demonstrating new concepts.
Models We acknowledge the limitation in a variance of evaluated models given by their availability and our computational possibilities. We evaluate only two models of the GPT family due to the usage limits of OpenAI API. Outside GPT models, we do not evaluate models over 20B parameters, given the infrastructure requirements of such settings. Nevertheless, we argue that the relevance of the models with constrained access, or resource requirements exceeding the limits of most organizations also remains a subject of open question.
Datasets One should note that the sizes of our evaluation datasets, for which we are able to extract concepts from explanations (Fig. 2), are too small to compare concept sensitivity between models. The sizes of our sensitivity evaluation datasets are the following: WorldTree: 2,204 samples, OpenBookQA: 792, GLUE Diagnostics: 282 samples, HotpotQA: 182 samples.
Ethical Considerations & Broader Impact
As outlined in Section 1, in-context learning recently presents a research direction of broad public interest, where the outstanding results on NLP benchmarks often do not meet the users’ expectations. It is understandable that the focus of development in in-context learning LLMs goes to measurable improvements on existing benchmarks, as ecologically-valid evaluations (de Vries et al., 2020) on end use-cases are timely and challenging to compare to related work.
Nevertheless, in this highly-exposed and fastpaced direction, we identify the necessity for the emergence of fast proxy measures that can shed light on the decision-making of the LLMs as expected by their end users.
The presented evaluation of models’ sensitivity to demonstrated reasoning concepts introduces a technical framework for quickly assessing models’ compliance with our expected functioning; However, a selection of a comprehensive set of concepts that we can agree our models should be able to learn remains a subject of open discussion.

heading: Acknowledgements
text: We acknowledge the Centre for Biomedical Image Analysis at Masaryk University, supported by MEYS CR (LM2023050 and CZ.02.1.01/0.0/0.0/18_046/0016045 CzechBioImaging) for their support in obtaining evaluations presented in this paper.

heading: A Details of Concept-aware Evaluations
text: Unless stated otherwise, we evaluate all models over all datasets and both demonstrations selection strategies consistently for ROUGE-L in default settings of Lin (2004), using a number of demonstrations k = 3 and contexts constructed in the following format:
“Input: x1 Prediction: Y1 Input: x2 Prediction: Y2 Input: x3 Prediction: Y3 Input: xpred”
Among both random and concept-sharing evaluations, we share the same xpred and only permute the demonstrations; We find cases where the filtering of predicted samples (xpred) to the ones sharing a concept with sufficient amount of (3) different samples needed for demonstrations makes the task systematically easier.
We diverge from the stated configuration only in the following cases:
• TK-INSTRUCT-11B and HotpotQA: we limit the evaluation contexts to at most 3.500 unique words, as we can not fit longer contexts into the memory. This might make the absolute results in this configuration overly optimistic, but still comparable within the Conceptual few-shot evaluation.
• GPT and HotpotQA: We completely exclude these evaluations given the fixed context window size of these models will exclude the xpred from prediction input in too many cases.
We choose evaluated GPT APIs based on OpenAI documentation4, picking for GPT and INSTRUCTGPT models marked as DAVINCI and TEXT-DAVINCI-003. Note that these identifiers might change in time, thus disallowing us to guarantee the reproducibility of their evaluations.

heading: B Computational Requirements
text: We run both training and evaluation experiments using single NVIDIA A100-SXM-80GB. The time and computational requirements of evaluation depend largely on the size of the evaluated model; We can evaluate the models up to 11B parameters on a single NVIDIA A100-SXM-80GB. The evaluation of Concept Few-shot learning on all our datasets, together with the Random reference evaluation takes approximately 2 hours for a 1B model.
4https://beta.openai.com/docs/ model-index-for-researchers

heading: Future Work
text: We believe the future work can inspire in identifying possibly complex reasoning concepts in the explanations of human annotators and will scale the conceptual evaluation to a wider variety of concepts. We trust that an evaluation with a comprehensive selection of the concepts will allow us to more realistically assess the abilities of the newlydesigned language models in the fast-progressing development of new in-context learners. Limitations
Concepts In this work, we extract the concepts from semi-structured explanations whose format reassures consistency and non-ambiguity of the exploited concept(s). The selection of datasets and corresponding concepts is primarily conditioned by data availability, as the semi-structured explanations are available merely for a small set of datasets. We acknowledge that our selection of concepts is not representative for a vast variance of concepts that users might expect models to learn from context in interaction. Some important concepts’ features that we identify are following: (i) a number of premises or reasoning inference steps needed to map the input to output, (ii) the granularity of the reasoning steps, (iii) a type of the premises; For instance, whether the familiarity with a given concept requires a memorization of an entity property (such as “sun emits light”), or a reasoning mechanics such as analogical reasoning (“if animals can run and cat is an animal, then a cat can run”). We invite future work to identify or propose a
taxonomy that would better reflect the wide variance of reasoning concepts that models are expected to comprehend in order to serve a wide scope of unseen tasks. Such taxonomy can motivate a more targeted collection of concepts from explanations, or annotation of new explanations demonstrating new concepts. Models We acknowledge the limitation in a variance of evaluated models given by their availability and our computational possibilities. We evaluate only two models of the GPT family due to the usage limits of OpenAI API. Outside GPT models, we do not evaluate models over 20B parameters, given the infrastructure requirements of such settings. Nevertheless, we argue that the relevance of the models with constrained access, or resource requirements exceeding the limits of most organizations also remains a subject of open question. Datasets One should note that the sizes of our evaluation datasets, for which we are able to extract concepts from explanations (Fig. 2), are too small to compare concept sensitivity between models. The sizes of our sensitivity evaluation datasets are the following: WorldTree: 2,204 samples, OpenBookQA: 792, GLUE Diagnostics: 282 samples, HotpotQA: 182 samples