Meta-in-context learning in large language models
Authors: Julian Coda-Forno
Marcel Binz
Zeynep Akata
Matthew Botvinick
Jane X Wang
Eric Schulz
Venue: NeurIPS 2023 poster (2023)
Keywords: Large language models
in-context learning
meta-learning
GPT-3

Abstract: Large language models have shown tremendous performance in a variety of tasks.  In-context learning  -- the ability to improve at a task after being provided with a number of demonstrations -- is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in-context learning. Looking at two idealized domains, a one-dimensional regression task and a two-armed bandit task, we show that meta-in-context learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we find that meta-in-context learning modifies the in-context learning strategies of such models. Finally, we broaden the scope of our investigation to encompass two diverse benchmarks: one focusing on real-world regression problems and the other encompassing multiple NLP tasks. In both cases, we observe competitive performance comparable to that of traditional learning algorithms. Taken together, our work improves our understanding of in-context learning and paves the way toward adapting large language models to the environment they are applied purely through meta-in-context learning rather than traditional finetuning.

Meta-in-context learning in large language models
Julian Coda-Forno 122, Marcel Binz'
Zeynep Akata²
Matthew Botvinick Jane X. Wang
Eric Schulz'
Max Planck Institute for Biological Cybernetics, University of Tübingen Tübingen, Germany; Google DeepMind- - London, United- Kingdom (julian.coda-fone fornoo tuebingen.mpg.ds.
Abstract
Large language models have shown tremendous performance in variety of tasks. In-context learning the ability improve at task after being provided with a number of demonstrations - is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in meta-in-comee in-context learning. Looking at two idealized domains, a one- one-dimensional regression task and a two-armed bandit task, we show that meta-in-come learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we find that meta-in-context learning modifies the in-context learning strategies of such models. Finally, we broaden the scope of our investigation to encompass two diverse benchmarks: one focusing real- world regression problems and the other encompassing multiple NLP tasks. both cases, we observe competitive performance comparable to that of traditional learning algorithms. Taken together, our work improves our understanding in-context learning and paves the way toward adapting large language models the environment they are applied purely through meta- meta-in-comt in-context learning rather than traditional finetuning.
Introduction
Large language models (LLMs) are taking not only machine learning research but also society by storm [1, 2, 3]. Part of what makes these models persuasive is that their abilities reach far beyond what we expected pure language models to do. They can, among other things, solve challenging reasoning problems, including university-level math questions [4] analogical reasoning tasks [5], out-of-the-box and without additional training.
Much of this power comes from what is known as in-context learning [6]. In-context learning (sometimes also called few-shot learning or few-shot prompting) refers to the ability of an LLM to improve at a given task after being provided with number of task-relevant demonstrations. This ability sets L.LMs apart from traditional models and led totally new paradigm one which eschews finetuning of weights task-specift data altogether and instead relies entirely on contextual information.
In the present paper, we ask whether the learning algorithm implemented through in-context learning be improved through in-context learning itself (without the need for any further finetuning of parameters). To study this question, we conducted several experiments where we presented an LLM with multiple learning tasks in a sequence. three distinct settings, we find evidence for the idea that the in-context learning abilities of an LLM can be recursively enhanced via in-context learning. thereby displaying form meta-in- in-centext learning. Figure provides high-level overview of our approach on an example supervised learning task.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Try predict y accurately:
to predict y accurately:
Try to predict accurately:
Task 1: x=
Task
Task 1:
Task 1:
Task 1:
Task 1:
Task
1:
Task
x=1, y=2 x=3, y=
x=1, y=2 x=3, y=6 x= y=
x=1, y=2
x=1, y=2
x=1, y=2
x=1, y=2
x=1. y=2
x=1, y=2
x=3, y=6
x=3, y=6
x=3, y=6
x=3, y=6
x=3, y=6
x=3, y=6
x=4, y=8
x=4, y=8
x=4, y=8
x=4, y=8
x=4 y=8
x=4,
Task 2:
Task 2: x=1, x=
Task 2:
Task 2:
Task 2:
Task 2: x=1, y=1
In-context learning
x=1,
x=1, y= x=2, y=3
x=1, y=1
x=1. y=1
x=2, y=3
x=2 y=3
x=2, y=3
x=5, y
x=5, y=9
x=5, y=9
x= y=9
[...]
[.--]
[...]
Task
Task N:
Task N:
x=0.
x=0. y=1
x=0, y=1
x=2, y
x=2, y=5
x=
Meta-in-context learning
Figure 1: High-level overview of our approach on example of multiple three-shot regression tasks. We present an LLM with N learning tasks in a row. Improvement within a task indicates that the model is capable of in-context learning. If in-context learning improves across multiple learning tasks, the model is also capable of meta-in-context learning.
More specifically, we first investigate meta-in-context learning on two artificial domains: supervised function learning task, and two-armed bandit task. For both of them, we find that sequentially presenting LLMs with multiple learning problems boosts their in-context learning abilities. We then use these idealized domains identify the drivers behind meta-in-conter learning. We find that meta-in-context learning adaptively modifies priors over latent variables, ultimately leading to priors that closely resemble the true statistics the environment. Furthermore, our analysis reveals that meta-in-context learning can not only be used to change prior expectations but is also capable of reshaping an LLM's learning strategies. Lastly, we apply our approach to two realistic domains. We first demonstrate that meta-in-contex: lcarning competitive with traditional algorithms on a benchmark of real-world regression problems. We then validated the transferability of our findings an NLP benchmark, confirming the effectiveness of our approach in diverse contexts.
be used to obtain a learning algorithm that is
2 Related work
In-context learning: Recent work has shown that LLMs can improve their performance after being shown a few task-relevant demonstrations - an ability referred in-context learning [6]. When and why in-context learning emerges is a matter of ongoing debate, with different theories being proposed. Chan al. [7] argued that properties of the training data such as burstiness and ssaationaaiity are key drivers behind in-context learning. Min et al. [8], on the other hand, found that ground truth demonstrations can be replaced with random labels while barely hurting performance, suggesting that the role of demonstrations is more prime the model for particular task. Finally, Xie et al. [9] suggested that LLMs internally need to infer latent variables to make better predictions about future word occurrences, thereby implementing form of Bayesian inference.
In-context learning can solve classical learning tasks: If LLMS do apply some form of Bayesian inference, then one would expect an LLM to also be able to solve classical online learning tasks, such as regression or classification, purely through in-context learning. Previous research suggests that this is indeed the case. Lovre [10], for instance, tested GPT-3 on a range of low-din low-dimensional classification and regression tasks and found that it was often on par with classical learning algorithms such as logistic regression. Likewise, Hegselmann et al. [11] tested an L.M''s few-shot classification abilities on tabular data. They found that their approach outperforms prior deep-learning-baal tabular classification methods on several benchmark datasets, and that performance further improves when

---

the model is provided with semantic information about the data.' LLMs are not only able to solve supervised learning problems but also simple reinforcement learning tasks. To provide one example. Binz & Schulz [12] evaluated GPT-3 on two-armed bandit task and found that its performance exceeded that of human participants who did the corresponding psychological experiment.
Meta-in-contexi learning stands in contrast to classical meta-learning schemes in which one adapts neural network to a distribution over learning problems by adjusting its weights [13, 14]. Historically, this approach has relied on recurrent networks [15, 16, 17. 18] but, more recently, researchers have also started use transformer- based architectures [19, 20, 21]. For example, Garg et al. showed "transformers can be trained from scratch to perform in-context learning of linear functions" and that the resulting models achieve "performance comparable to the optimal least squares estimator." In similar vein, von Oswald et al. argued that "transformers [can in principle] implement gradient descent in their forward pass" and provide empirical evidence that they indeed do so [19]. In contrast to these approaches, our approach adapts model to a series of learning problems entirely through the context itself opposed to updating weights.
Meta-in-context versus classical meta-learning schemes:
3 Experimental setup
We used the public OpenAl Python API [22] to run all our simulations. This API provides access to several LLMs from the Generative Pre-trained Transformer (GPT) family. We ran all our simulations on the TEXT-DAVINCI-O00 model, which is also known GPT-3. We set the temperature parameter to zero (leading to deterministic responses) unless otherwise noted and retained the default values for all other parameters. It is important to note that all experiments performed in this paper rely entirely on the in-context learning abilities of an L.M,, and do not involve any form of finetuning.
4 Learning one-dimensional functions
In our first set of experiments, we investigated GPT-3's ability for meta-in-context learning in simple one-dimensional supervised regression task. In this setting, we provided GPT-3 with list of input-target pairs from a given task and asked it to make accurate predictions for a new input value. This is one of the most fundamental machine learning problems, and its simplicity makes an ideal testbed for an initial analysis.
4.1 Methods
Function learning prompt
You observe 5 machines that produce output y for given input a. Each machine implements a different function.
Machine 1:
[1.
Machine 5:
52, y -209;
I 18, y =
-138;
60, y [insert]:
Every task n consisted of T = 5 input-target pairs (2t, fxx)), where t E [1, denotes the trial number. Each input-target pair was generated by a linear function the form f(x)) = a a + g(r) + Et, where (") and ((n) are task-specific parameters drawn from a probability distribution p(T). Inputs F, were sampled from ((0,, 100) and the trial-specific additive noise E, was sampled from (O, 1). For our meta-in-context learning simulations, we considered prompts that include data from up to five tasks, each corresponding to different underlying function. Following the schema
However, it is important to note that their approach does not entirely rely on in-context learning. but involves some additional finetuning.

---

A
B
C
GPT-3
(default)
GPT-3 after tasks
(oracle)
80
200
Regression coefficients
MSE
MSE
100
-20
40
0
2
4
5
2
4
5
Trial Task
Trial
Tasks
D
Task distribution
GPT-3 priors
Task
Task 2
Task 3
Task 4
Task 5
500
-500
100 100 100
100
100
X
X
X
X
Figure 2: Meta-in-context learning on the one-dimensional regression task (100 simulations). Errors bars represent 95% confidence intervals. A: MSE across trials for different models. B: GPT-3's MSE averaged over trials for each task. C: Effects of trial and task for estimating the MSE. D: GPT-3's prior expectations across tasks (blue) compared the true task distribution (orange).
outlined in Figure 1, we iteratively presented each data-point together with the history of previous observations, starting from the first trial in task one up to the last trial in task five. In the "function learning prompt" box above, you can see example prompt that provided to GPT-3.
4.2 Results
In preliminary simulations, we found that GPT-3 has strong bias toward increasing positive functions. To demonstrate the potential of our approach, we wanted to investigate whether it is possible to overwrite this bias via meta-in-context learning. specific functions with negative slope and intercept, i.e. (!)) (-22, 1) and b(n) ~-1100 1), and evaluated how observing multiple such tasks influences the model's behavior.
order to achieve this goal, we sampled task-
GPT-3 does in-context learning: We first established that GPT-3 can learn within single task without considering the effects meta-in-context learning. To do so, we only examined the first task while ignoring all the subsequent ones. We found that performance measured by the mean-squared error (MSE) improves with additional data-points as shown in Figure 2A (solid lines), GPT-3 matches (or even slightly outperforms) Bayesian linear regression (BLR) model with default standard normal priors, indicating that in-context learning is able to solve the given problem. This is especially noteworthy since GPT-3 never told that the underlying functions are linear, as opposed to BLR which has a built-in assumption of linearity.
GPT-3 does meta-in meta-in-conter context learning: Next, we investigated whether GPT-3 is capable of meta-in- context learning. For this, we inspected how performance changes across the five tasks. Figure demonstrates that iocttent learning is beneficial by comparing performance between the first

---

and the final task (solid vs dashed lines). For reference, we also plotted the performance of BLR model with access to the ground-truth data-generating distribution. Note that this model has access to privileged information and hence only serves as performance upper-bound. Figure 2B shows more detailed development of performance as we increase the number of tasks the prompt.
To test whether the effects of in-context learning and mncta-in-conted learning are statistically meaningful, we fitted linear regression model that included the trial and task number as independent variables on the MSE. We found statistically significant effects for both trial number (3 = -30 -30.614
3.08, 194,114, P < 0.001) and task number (3 = 13.22 + 3.08.
-8.455, p < 0.001),
confirming that GPT-3 is capable of both in-context and meta-in-conee conatett learning (see Figure 2C).
We speculated that GPT-3's perfor- mance improves during meta-in-context learning because it adapts its priors to true environmental statistics. In order to evaluate this hypothesis, collected GPT-3's prior expectations before each task by asking it to make sequential predictions for 20 evenly spaced input values (using the same prompt template above, but setting the temperature one and feeding back the model's own predictions as the training data). We omitted outlier predictions that had absolute values of 10, 000 or larger for this analysis. The estimated priors indicate that GPT-3 has initial bias toward increasing positive functions. However, already after two tasks it adapts its priors to decreasing negative func- tions, thereby closely matching the true environmental statistics (see Figure 2D). Furthermore, the prior for the intercept term is initially centered around zero but shifts toward the ground ground-truth value of 100 after three tasks.
-eea-in-context learning is driven by adaptation of priors:
Taken together, these simulations provide initial evidence that GPT-3 engages in meta-in-context learning, meaning that its in-context learning abilities can be recursively improved via in-context learning itself. We have suggested and empirically confirmed that GPT-3 accomplishes this by adapting its priors across multiple tasks.
Meta-in-contest learning is an emergent phenomenon: In order to gain a deeper understanding of the phenomenon's 's characteristics, we pursued examination into the progression of meta-in in-contextt learning proficiency in relation to both model complexity and dataset size. To this end, we undertook an analysis encompassing smaller GPT-3 models, specifically text-ada, text-cabbage, and text-curie [22]. Our observations (which can be found in Figure 5A, Appendix A. 1.).1) reveal a noteworthy trend, wherein solely text-davinci-002 seems exhibit meta-in-context learning capabilities for negative linear function.
To comprehensively explore meta-in-context learning, we extended our evaluation to encompass five distinct open-source models. This expansion allowed us to examine whether this phenomenon is also applicable to less opaque model architectures. Notably, while the majority of models (Falcon-400 [23] and Llama-2 [24] models) exhibited no indications of meta--i-conntex learning, intriguingly, MosaicML's mpt-30b [25] demonstrated an ability to manifest this phenomenon. We hypothesize that models trained in a regime with a longer context window more suitable to show the emergence of this phenomenon. Indeed, mpt-30b has a context-size of 8000 tokens as opposed to the I.LaMa-2 models which have been trained with context window of 4096 tokens. This is speculative and we leave the analysis of what factors influence the emergence of meta-in-context learning as a future research question. The results are included in Figure 5B, Appendix A.I.I.
Meta-in-context learning in open-source models:
Meta-in-contes learning with non-linear functions: We also aimed to determine whether GPT-3 could also adapt its meta-in-comt cn-context learning non-linear functions. In Figure 6, Appendix A.1. we present an analysis of quadratic function conducted within the same experimental framework. Our findings indicate that the model indeed exhibits the ability to meta-in-contex learn non- non-linear functions.
5 Experiments on two-armed bandit tasks
In next step, we wanted to investigate whether our results from the supervised setting also transfer to reinforcement learning paradigm. This setting adds an additional layer of complexity because requires the agent to learn from its own experiences instead of having access to ground-truth solutions of previous tasks. addition, allows us to investigate learning strategies and how they evolve during meta-in-context learning.
5

---

5.1 Methods
For our simulations, we considered simple two-armed bandit task, in which an agent repeatedly interacts with two slot machines. In each trial, the agent can select one of two machines and is rewarded based on probability distribution that associated with that machine. The agent's objective is to maximize the total amount of acquired points. We used cover story that involves a gambler visiting different casinos, which has been used in human experiments with similar tasks [26, 12], generate prompts:
Two-armed bandit task prompt
You are going to different casinos that own two slot machines. Choosing the same slot machine will not always give you the same points, but one slot machine is always better than the other. Within a casino, your goal is to choose the slot machine that will give you the most points over the course of 10 trials.
Each casino owns a different pair machine.
You have received the following points when playing in casino I:
[...]
You have received the following points when playing in casino 5: Machine delivered 4.2 points.
Machine F delivered -7.4 points.
Q: are now performing trial in casino Which machine do you choose be- tween machine J and machine F?
A: Machine [insert].
5.2 Results
For each task, we sampled independent mean rewards for each machine from NO 644.. The actually obtained reward generated using the mean reward that corresponds the chosen machine plus some additive Gaussian noise sampled from (O, /32). To stay consistent with the previous section, we considered prompts that include data from up to five different tasks for our meta-in-contee learning simulations (each of these tasks consisted of ten trials). We used letters indicate the different slot machines. For each task, we randomly sampled two different letters without replacement cancel out biases towards certain letters.?
We again first tested whether GPT-3 learns within a single task. Figure 3A confirms that this is indeed the case. Performance (measured in terms of regret) improves over the first four trials and plateaus afterward. However, when comparing GPT-3 to two baseline algorithms - a greedy policy and an upper confidence bound (UCB) algorithm we observed that lags in performance prior to meta-in-context learning.
GPT-3 does in-context learning:
GPT-3 does meta-in-context learning:
Like in previous experiment, found that GPT-3 improves during meta-in-conter lcarning: as it observes more tasks, it gets better at solving new two-armed bandit problems (see Figure 3B). GPT-3's performance matches that of greedy policy after having interacted with only four tasks and lags only slightly behind that of the UCB algorithm.
We quantified the effects of in in-context and meta-in-conters learning in a statistical analysis. To do so, we fitted linear regression model including the trial and task number as independent variables on the trial-wise regret. We found significant in-context learning (8 -0.221 + 0.012, -35.828, p < 0.001) and meta-in-context learning effects (3 -0.031 + 0.012, ( = 5.002, p < 0.001), thereby reproducing our results from the previous section (see Figure 3C).
Meta-in-context learning is driven by adaptation of priors: Following our earlier analysis, we speculated that part of this performance boost arises because GPT-3 adapts its priors toward the statistics of the tasks that were encountered during meta-in-context learning. To verify this, we
We left out the letters "I" and "U" as we have seen empirically that they induce biases. We also randomized the order of these letters in the questions at each trial to mitigate recency biases.

---

A
B
GPT-3
Greedy
GPT-3
0.00
GPT-3 after tasks
UCB
Greedy
-0.05
2.0
Regression coefficients
-0.10
Regret
Regret
1.5
-0.15
2
-0.20
1.0
2
6
8
10
2
3
5
TrialTask
Trials
Tasks
200
Reward priors
Regression coefficients
0.0
100
0
-0.5
1
2
3
4
5
True
RU
V/TU
TaskxV Taskx Taskx V TU
Tasks
Figure 3: Meta-in-context learning the two-armed bandit experiment (500 simulations). Errors bars represent 95% confidence intervals. A: Regrets across trials for different models. B: GPT-3's regrets averaged over trials for each task. C: Effects of trial and task for estimating the regret. D: GPT-3's prior expectation of rewards across games, E: Probit regression coefficients for different strategies and their interaction with task number.
probed its prior expectations before starting each task by asking "how rewarding do you expect machine X be?" We the temperature parameter 1 and repeated this question five times to reflect a sampling from a prior distribution. Figure 3D visualizes the change of priors across tasks. Before the initial task, GPT-3 expects rewards be distributed around larger positive values (M = 545.22 22, SD = 6359), However, at the end of meta-in in-cntext- learning, its priors match the true data-genn distribution closely (M 6.49, SD 4.41).
In addition to looking at the development of priors, the two-armed bandit setting also allows us to investigate whether any changes in strategies happen during meta-in-contex learning. For this, relied on an analysis originally proposed by Gershman [26]. The idea behind this analysis to define model that involves a parameterized combination of Boltzmann exploration, a UCB algorithm, and Thompson sampling, and then fit the parameters of this model to data generated by an agent (GPT-3 our case). is then possible to determine the extent to which the agent relied specific strategy by examining the resulting parameters.
We assume that the agent's beliefs over expected rewards at trial and action captured by the normal distribution p(ra.)) = N(Ha,, ) and define the following probit regression model based on the parameters of these distributions:
p(a: 0/w) + (wi V + W2RU, W3 W3V,/UT)
V, No,1 11,t
RU, = oo.1 - T1 TU, /0 + o
with denoting the cumulative distribution function of standard normal distribution. Equation 1 recovers a Boltzmann- exploration strategy for [w), W2, Wa] = [c. 0, 0), a variant of the UCB
Following Gershman [26]. we obtained these distributions by running Kalman filtering equations on the previously observed data. More details this analysis can be found in Appendix A.3.

---

algorithm for [W1, W2, W3] = [c, d, 0]. and Thompson sampling for [W1, W2, w3] = 0, 1] [27]. In our analysis, we furthermore included an interaction effect with task number for each factor to investigate how the applied strategies change during meta-in-context learning.
Meta-in-context learning reshapes learning strategies: We found positive main effect of the value difference V, (3 = 0.307 + 0.027, 21.759, < 0.001), indicating that GPT-3 engages in Boltzmann exploration. GPT-3 becomes more greedy during meta-in-context learning as shown by the positive interaction effect between task number and V, (3 = 0.128 + 0.025, 2 = 9.633, P < 0.001). Furthermore, we found negative main effects for both relative uncertainty RU, (B -0.160 = 0.010, 2 = -31.788, p < 0.001) and the uncertainty-scall 0.025, 2 = 5..393 p < 0.001), suggesting that GPT-3 avoids uncertain options by default. However, we also found a slight, but sigmificant, increase in UCB-based decisions during meta-in-contee learning (3 0.046 0.010,
value difference V ; (5 = =
9.279, p 0.001). Figure 3E shows visualization of all regression coefficients involved in this analysis.
The results presented in this section corroborate those obtained from the supervised setting. GPT-3 generally performed better in a two-armed bandit task after meta-in-context learning. We again observed that GPT-3 accomplishes this by adapting its priors across tasks. In addition, we investigated changes in strategies during meta-in-context learning and found that GPT-3 learned to perform better by exploiting more consistently.
6 Regression on real-world data
Next, we wanted to investigate whether our results obtained in the artificial domains scale up to real-world applications. To test this, we considered multi-dimensional regression benchmark which consists of 60 different real-worke datasets introduced in [28].
6.1 Methods
For each simulation, we randomly selected five different tasks from the benchmark and then sampled five points without replacement for each task. We used five features for all tasks. If a dataset contained less than five features, we omitted from our analysis, which yielded 42 remaining datasets. For tasks exceeding five features, used sub-sclection procedure that retained only the top five features based on their F-value with respect to the target variable evaluated in a univariate linear regression. To maintain consistent regression loss across all tasks, normalized both the feature and target spaces to the interval of [-1, 1]. The resulting prompts follow the general template outlined earlier:
Regression on real world data
You observe an input vector and have to predict the corresponding output y accurately as possible. You are given 5 different tasks.
Task 1:
[...]
Task 5:
[0081, -0116. -0.78, -0777, -0.45), -.77,-)
-0.34;
x -0.81, 0.63. 0..6..6..77. -0.75 -0883
-0.55, =
-0.68:
0.97, -0.92. 0097, -.97, -0.8),, y [insert]:
6.1.1 Results
GPT-3 does in-context and meta-in-context learning: Following the previous experiments, we investigated the learning curves of GPT-3 and GPT-3 after meta-in-contexy learning. We additionally included two baselines our analysis: BLR and random forest model. We measured performance by the root-mean- poot-men-ssquaan -quared eтror (RMSE). Figure 4A shows the learning curve for all these models. Like the previous cases, we found that GPT-3 improves with additional data-points, confirming that it is capable of in-context learning in this setting. In addition, meta-in-context learning further improved performance. GPT-3 after meta-in inca--nn-coneet learning matched the performance of BLR and was only slightly outperformed by the random forest model.

---

B
GPT-3
0.00
GPT-3
Roreo
GPT-3 after 5 tasks
BLR
GPT-3 after 5 tasks
0.8
30
Regression coefficients
-0.05
predictions (%)
RMSE
0.6
Extreme
20
-0.10
10
0.4
-0.15
1
2
4
5
1
2
3
4
Trial Task
similarity
Trials
Trials
Figure 4: Meta-in-context learning on the real-world regression benchmark (42 50 simulations). Errors bars represent 95% confidence intervals. A: RMSE across trials for different models. B: Percentage of predictions outside or equal to the extremes of the squashed target range. C: Effects of trial and task similarities for estimating the RMSE.
Meta-in-context learning constrains predictions: Most the improvement meta-in-context learning seems to come from zero-shot performance. We hypothesized that this is the case because GPT-3 acquires an understanding of potential target value ranges during meta-in-contex learning. To test this hypothesis, we plotted the proportion of GPT-3's outputs that fell the extreme values of the target space or outside of it in Figure 4B. We found that meta-in-contex learning substantially reduces the number of predictions that are outside of this range, indicating that meta-in meta-in-context learning helps to learn a better initial guess.
-ectt-incooneet learning works better when task similarity is high: However, learning good initial guess is not the full picture. We speculated that meta-in-context learning is more effective if there is higher similarity previously encountered tasks. To verify this, we fitted a linear regression model with trial and task similarity on the RMSE. We computed task similarity as the average similarity between the current data- mata-point and the data-points from all previously observed tasks (each individual similarity measure was obtained using a radial basis function kernel). We found significant effect of trial (3 -0.089 + 0.020, t -9.204, p 0.001) shown in Figure 4C, confirming that performance within a task improves with additional observations. Furthermore, found significant effect of task similarity (3 00.147 + 0.020, 115.223, p 0.001), suggesting that meta-in-context learning works best if similarity to previously encountered tasks high. In previous experiments, This experiment delves into the underlying reasons by proposing how the relatedness of different meta-in-context tasks affects their observed signal. Therefore, we expanded our analysis by also incorporating task similarity considerations to both Experiments I and 2 (see Figure 7, Appendix A.2). The analysis reveals the same observation: the more related the task, the more meta meta-in-conte in-context learning is beneficial.
showed that meta-in meta-in-contex learning improved performance.
The findings outlined in this section provide further evidence for meta-in-context learning in LLMs.* Notably, GPT-3 exhibited superior performance in a real-world multi-dimensional regression task following meta-in-context learning. We identified two reasons for this. First, meta-in-context learning constrained initial model predictions to the range of plausible values, and second, GPT-3 was able to leverage similarities previously encountered tasks in order improve its predictions.
7
Meta-in-context learning on natural language processing benchmarks
Finally, we examined whether vreta-in-contedd learning also improves upon in-context-learmin on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark [29].
'Note that while we were running our model simulations. OpenAl released new model GPT-4, We repeated the analysis from this section on this new model and deseribed our results in Appendix A.4.

---

7.1 Methods
We focus on the tasks from the STEM supercategory other supercategories together with the addition of meta-in-context learning cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with k E 1, examples from the same category before prompting on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from different categories. Examples of two STEM tasks and some extra analysis on open-source models on the entire MMLU can be found in Appendix A.5.
7.2 Results
Figure 9 in Appendix A.5 summarizes results. We found that meta-in-context learning was in general beneficial in terms performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached accuracy of 55.1% outperforming in-context by 22.4%. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they also transfer some knowledge from different categories.
8 Discussion
We have demonstrated that LLMs improve their in-context learning abilities via in-context learning itself, i.e., that they are capable of meta-in-context learning. Meta-in-context learning was only able overwrite an LLM's priors but also changed its learning strategies, as demonstrated in two artificial domains. Finally, we applied our approach to two benchmarks. First, a real-world benchmark of regression tasks where we found that meta-in-context learning leads to algorithms that are competitive with standard learning algorithms. Then, we verified the applicability of our results an NLP benchmark, providing further evidence of the versatility and effectiveness of our approach across diverse contexts.
Perhaps the most significant shortcoming of our model simulations is that they all relied learning tasks with just handful of observations. This limitation is mainly due to the practical constraint of finite context window coupled with meta-in veet--n-coontxt learning's rapid prompt length increase. To ensure that we remain within the allowed context length (and to keep the monetary costs for our simulations reasonable level), we had make this design choice. However, despite this restriction our simulations were sufficient to illustrate the potential of meta-in-context learning. We hope that future L.M iterations with longer context lengths and lower inference costs will allow us to extend our simulations to larger datasets.
believe that
In addition, the tasks we probed were rather simplistic in their nature. That being said, we think we have reasonably covered the space of fundamental learning paradigms, including a supervised problem, reinforcement learning problem, collection of real- world datasets and NLP benchmark. With the increasing availability of mudti-modal models, it will furthermore become feasible to apply approach other domains. In this context, two obvious candidates are classification tasks with visual stimuli or grid-based navigation tasks.
A promising direction for further research involves delving into the factors that contribute to the emergence of the meta-in-context learning phenomenon. One potential factor, as mentioned earlier, is the length of the context window. Another avenue to explore is to assess whether specialized models fine-tuned for in-context learning, like the framework proposed by Min et al. [30] "Meta-trainigg for In-Context Learning), yield optimal performance in the context of meta-in-contex learning.
In summary, our work has both near- and long-term consequences. In the near term, it indicates that it is not strictly necessary to engineer the perfect prompt for an LLM so that it can solve given learning problem [31]. Instead, LLMs are - to some degree - able to infer the required information from just handful of related-tasks examples. In the long term, it could point to a paradigm where we adapt these models to the environment they are applied in purely through meta-in-conter learning rather than finetuning them using traditional means.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Zeb Kurth-Nelas for the helpful discussions. Zeynep Akata acknowledges partial funding by the ERC (853489- DEXIM).

---

References
[1] Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arX1v:2303.10130,
2023.
[2] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences, 103:102274 2023.
[3] Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics, pages 1-4, 2023.
[4] Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang. Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.
Proceedings of the National Academy of Sciences, 11963)):e22333311 2022.
[5] Taylor Webb, Keith Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. arXiv preprint arXiv:2212.09 020222 2022.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[7] Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emer- gent in-context learning in transformers. Advances in Neural Information Processing Systems, 1118888-18991 2022.
[8] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12833 2022.
[9] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2021.
[10] Lovre, Who models the models that model models? an exploration of gpt-3's in-context model fitting ability. https: http:://wwwwwllgmemiiiii.e.. alignmentforum orgg/oosta/ccRaaaaarxtyyaa who-mone thoomodels--o whwolllstte-mmeelel thaeelss-hhttte
an- wnm-eeporatioonooo of.
[11] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models, In International Conference on Artificial Intelligence and Statistics, pages 5549-5581. PMLR. 2023.
[12] Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6) 120, 2023.
[13] Marcel Binz, Ishita Dasgupta, Akshay Jagadish, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-learned models of cognition. arXiv preprint (tti::233.0672999 2023.
[14] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, al. Meta-learning of sequential strategies. arXiv preprint arXiv: 905.03030. 2019.
[15] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
[16] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprini arXiv: 1.05663, 2016.
[17] Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1.02779, 2016.
12

---

[18] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-leaming with memory-augmentee neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[19] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174.
PMLR, 2023.
[20] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
[21] Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 2022.
[22] Openai api. https: : / /wppaform....mmmm/// openai com. Accessed: 2023-05-10.
[23] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099280 2023.
[25] MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. Accessed: 2023-06-22.
[26] Samuel Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34-
42, 2018.
[27] Marcel Binz and Eric Schulz. Modeling human exploration through resource-rational reinforce-
ment learning. In Advances in Neural Information Processing Systems, 2022.
[28] Jan Malte Lichtenberg and Özgür Simsek. Simple regression models. In IDM@ 2017.
[29] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint WwXiv:2009.03300 2020.
[30] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint 2rXiv:2110.15944 2021.
[31] Chenglei Si, Zhe Gan, Zhengyuan Yang. Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09 2022.
[32] Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
13


Reviews:
[{"review_id": "FxyDsbFcRA", "reviewer": "Reviewer_ShPz", "summary": "This paper demonstrates that large language models (LLMs) are capable of meta-in-context learning: updating their in-context-learning abilities when prompted with examples of several tasks\nThe authors empirically show this capability on several learning paradigms, including supervised learning (1D linear regression), reinforcement learning (2-armed bandits), and several real-world linear regression problems.\nThese experiments show that LLMs, and GPT-3/GPT-4 more particularly, can effectively adapt their in-context learning algorithm while also matching simple baselines like Bayesian linear regression (BLR), upper-confidence bound (UCB), and random forests.\nThey also show that meta-in-context can outperform standard in-context learning, which is especially relevant given the popularity of this technique with LLMs.", "questions": "- Do open-source models also exhibit meta-in-context learning? Why or why not?\n- How does meta-in-context learning depend on the size of the model / data it is trained on?\n- Can meta-in-context learning outperform in-context learning on standard NLP tasks like NLI, SQuAD, or GSM8K?\n- How is this work different from the Min et al.'s MetaICL paper?\n- Please fix your bibliography.", "limitations": "The limitations of the work are appropriately discussed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. I found the paper an enjoyable read. The research is relatively well motivated, the analysis are carefully detailed, and the method is clearly explained. In fact, I believe I could easily replicate the results from the paper because all experimental testbeds are precisely described and the exact prompts are provided (which also help illustrate the method; see blue panels). While more baselines could be included for completeness, the experiments clearly show the meta-in-context learning effect so I don't think they are necessary.\n2. Despite its simplicity, the presented idea is elegant and I could see it impacting research beyond large language models. For exampe it is not difficult to imagine extensions beyond text inputs (say, vision, audio, or robotics), or theory extensions (in the flavor of \"Transformers learn in-context by gradient descent\"), or even applying a similar strategy beyond in-context learning (eg, for prompt or prefix tuning). As such, I think it'll garner interest from the NeurIPS community.", "weaknesses": "1. Experimental design: given that there is no theory, I find the experimental section a bit light. First, the authors only include results with OpenAI models (GPT-3 and GPT-4) which are not open-source. So it's not clear if meta-in-context learning works as well (or at all?) with publicly available LLMs. Second, can *any* LLM meta-learn in-context? How does the meta-learning ability improve with model/data size? Third, meta-in-context is only tested on small scale and toy datasets -- even the \"real-world\" ones are only require simple linear regression models. It'd be much more compelling if the authors could show that meta-in-context learning also improves upon in-context-learning on standard NLP tasks. NLI-type tasks would be a promising start but I'd find question answering (eg, SQuAD) or even GSM8K much more interesting.\n2. Novelty: The proposed method is similar to \"MetaICL: Learning to Learn In Context\" by Min et al. and published at NAACL in 2022 -- yet, the authors don't even mention this work. As far as I understand, MetaICL pretrains the model such that the LLM weights are trained to adapt quickly to in-context prompts (hence they have learned to learn), while this work demonstrates that meta-learning emerges even with the standard LLM training objective. It'd be insightful to compare the two approach to quantify how much meta-training helps to adapt quickly. Min et al.'s work is especially relevant because it demonstrates benefits on real NLP tasks.\n3. Scholarship: Many references in the bibliography are broken. This might seem a detail but not when a third of the references don't even state a venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ShPz,\n\nWe appreciate that the reviewer found our paper well-motivated, clear enough to be replicated easily and the idea elegant and possibly impactful. We also thank the reviewer for their helpful comments and we have made a response to each of their comments along with suggested changes to the paper. \n\n>  First, the authors only include results with OpenAI models (GPT-3 and GPT-4) which are not open-source.  So it's not clear if meta-in-context learning works as well (or at all?) with publicly available LLMs\n\nWe appreciate the feedback and agree that some analysis on other LLMs (especially open-source models which are transparent on training and architectures) does improve the empirical analysis. Therefore, we conducted additional simulations of meta-in-context learning on the latest open-source models (MPT-30b, Falcon-40b, Llama-2-7b/13b/70b). We will add the following paragraph to our revised paper:\n\n\"**Meta-in-context learning in open-source models**: To comprehensively explore meta-in-context learning, we extended our evaluation to encompass five distinct open-source models. This expansion allowed us to examine whether this phenomenon is also applicable to less opaque model architectures. Notably, while the majority of models  (Falcon-40B \\cite{falcon40b} and Llama-2 \\cite{touvron2023llama2} models) exhibited no indications of meta-in-context learning, intriguingly, MosaicML's MPT-30B \\cite{MosaicML2023Introducing} demonstrated this ability. The results are included in the Supplementary material.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 6B.\n\n> How does the meta-learning ability improve with model/data size?\n\nWe thank the reviewer for the feedback as we concur with the significance of investigating whether meta-in-context learning emerges as a function of model or data size. In response, we will incorporate the subsequent section into our revised paper:\n\n\"**Meta-in-context learning is an emergent phenomenon**:\nIn order to gain a deeper understanding of the phenomenon's characteristics, we pursued an examination into the progression of meta-in-context learning proficiency in relation to both model complexity and dataset size. To this end, we undertook an analysis encompassing smaller GPT-3 models, specifically text-ada-001, text-babbage-001, and text-curie-001 \\cite{openaiAPI}. This analysis (which can be found in the Supplementary material) revealed a noteworthy trend, wherein solely text-davinci-002 seems to exhibit meta-in-context learning capabilities, thereby making it an emergent phenomenon.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 6A. \n\n> It'd be much more compelling if the authors could show that meta-in-context learning also improves upon in-context-learning on standard NLP tasks. \n\nWe appreciate this suggestion which was echoed by all other reviewers as well. We have therefore conducted additional simulations of meta-in-context learning on the Massive Multitask Language Understanding (MMLU) benchmark. We will add the following section to our revised paper:\n\n\"**Meta-in-context learning on natural language processing benchmarks**:\n\nFinally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark \\cite{hendrycks2020measuring}.\n\n**Methods**:\nWe focus on the tasks from the STEM supercategory as other supercategories -- together with the addition of meta-in-context learning -- cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with $k \\in {0, 1, 2}$ examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from different categories.\n\n**Results**:\nFigure 9 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of $55.1$ percent outperforming in-context by $22.4$ percent. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 9.\n\n> 2. Novelty: The proposed method is similar to \"MetaICL: Learning to Learn In Context\" by Min et al. and published at NAACL in 2022 -- yet, the authors don't even mention this work.\n\nThanks a lot for mentioning this paper – we had somehow missed it. We have added it to the related work section, where we also describe how it differs from our approach:\n\n\"Meta-in-context versus classical meta-learning schemes:\n[...] **Min et al. \\cite{min2021metaicl} proposed a method called \\emph{meta-training for in-context learning} following the same paradigm. Their work starts with a pretrained language model that is then trained (via an adjustment of its weights) on a large set of training tasks to do in-context learning, ultimately leading to a model with improved in-context learning capabilities.** In contrast to these approaches, our approach adapts a model to a series of learning problems entirely through the context itself as opposed to updating weights.\"\n\n> 3. Scholarship: Many references in the bibliography are broken. This might seem a detail but not when a third of the references don't even state a venue.\n\nThanks a lot for pointing out this mistake. We strive to take good scholarship seriously and therefore went through all references to fix cases that had missing venues or other incomplete information.\n"}, {"review_id": "3fGLmK5UWm", "reviewer": "Reviewer_xRJY", "summary": "The paper explores a phenomenon referred to as “meta in-context learning” in large language models, where the in-context learning abilities of these language models can be recursively enhanced through in-context learning itself. To demonstrate this, the researchers examine two idealized domains: a one-dimensional regression task and a two-armed bandit task. They show that meta-in-context learning (1) improves in-context learning performance (2) dynamically shapes the large language model's priors over expected tasks and (3) also modifies its in-context learning strategies.\n", "questions": "What are the main practical consequences or implications of your findings for specific real world scenarios in which in-context learning in LLMs is currently used? ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The high level question raised in this paper is interesting – can in-context learning be improved via the recursive operation of concatenating several different tasks in-context? \n* The experimental setup is clean and the obtained experimental results convey a clear message. They clearly show that for the examined problems meta in-context learning helps, and moreover they reveal that the improvement (also) comes from previous tasks in context shaping the LLM prior for the examined task.\n\n\n", "weaknesses": "* I found the setup and particularly the examined tasks to be too limited. Although the high level question is interesting I feel like much more could be done in order to shed light on it and connect it to actual scenarios in which in-context learning is used in practice. \nThe authors chose to focus on linear regression (synthetic and 60 real world tasks filtered down to 42) / synthetic bandit problems. The abstract and intro describe that in-context learning is a very important feature of LLMs which facilitated their success; while true it is not related to these types of tasks but rather to natural language ones. I think that the interesting high level question raised in this paper could easily have been studied for natural language tasks, and in particular relating it to papers showing multitask benefits which study which tasks are related and reinforce each other such as Sanh et al ICLR 2022, Aribandi et al, ICLR 2022 and others. \nI acknowledge that there is a growing literature of works theoretically analyzing and experimenting with in-context learning on synthetic / mathematical problems, the authors mention some of it. However, in this case I think that the merits of the synthetic / simple / mathematic setup were not fully utilized – for example, the authors could have studied in this simple setting how the relatedness of different tasks in the same context affects their observed signal (eg by defining a formal metric of distance / similarity between linear regression tasks). A small subset of the results in section 3.3 is dedicated to a statement about task similarity and I think it should be expanded for deeper insight. In the lack of such deeper investigations in the synthetic realm, and on the other hand no experimentation on language tasks, this paper misses out on real insight relevant for language related scenarios. For the scenarios studied in the paper, LLMs are not the go-to tool. I will note as an outlier to the above that the study of  in-context learning strategies in the bandit case is deep, novel, and leverages the clean mathematical structure of the synthetic experiment. However, even in this case, I see no conclusions applicable to real world scenarios in which LLM in-context learning is the go-to tool.\n\n* The paper is not segmented into “method”, “experimental setup”, “results” sections, but rather includes one section describing all of the above linearly according to the different tasks that were ran. This is not a problem a-priori, but I found the presentation hard to follow at times, and the authors may want to consider structuring the paper a bit more.\n\n* Minor remarks:\n** There are some some typos (eg, end of parentheses in line 77, named citations without a year)\n** The name “meta in-context learning” is used for a different method presented by Min et al. 2021 in this influential paper: “MetaICL: Learning to Learn In Context”. You may want to rename this paper. \n** I think that the authors overstate when they claim meta in-context learning to be related to psychological experiments. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xRJY,\n\nFirst, we thank the reviewer for finding the paper interesting with a clear message conveyed from the experiments. We also thank the reviewer for their helpful comments and we have made a response for each of their comments along with suggested changes to the paper:\n\n> I found the setup and particularly the examined tasks to be too limited. [...] I think that the interesting high level question raised in this paper could easily have been studied for natural language tasks.\n\nWe appreciate this suggestion which was echoed by all other reviewers as well. We have therefore conducted additional simulations of meta-in-context learning on the Massive Multitask Language Understanding (MMLU) benchmark. We will add the following section to our revised paper:\n\n\"**Meta-in-context learning on natural language processing benchmarks**:\nFinally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark \\cite{hendrycks2020measuring}.\n\n**Methods**:\nWe focus on the tasks from the STEM supercategory as other supercategories -- together with the addition of meta-in-context learning -- cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with $k \\in {0, 1, 2}$ examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from \\emph{different} categories.\n\n**Results**:\nFigure 9 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of $55.1$ percent outperforming in-context by $22.4$ percent. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 9.\n\n>  The authors could have studied in this simple setting how the relatedness of different tasks in the same context affects their observed signal (eg by defining a formal metric of distance /similarity between linear regression tasks).\n\nWe agree that beyond merely showcasing performance gains across tasks, our paper would benefit from delving into the underlying reasons for the improved performance of LLMs across tasks. In response, we have expanded our analysis by incorporating task similarity considerations, extending our approach initially adopted for Experiment 3 to both Experiment 1 and Experiment 2. We added task similarity as a regressor on Experiment 1’s and 2’s respective MSE/regret regression bar plots (Figure 2C and 3C).\n\nFor Experiment 1, we quantified task similarity using the average negative L2 norm of the underlying parameters (slope & intercept) with previous tasks. For Experiment 2, we quantified task similarity using the average difference of mean rewards with previous tasks. Our analysis shows a strong effect of Task similarity for each Experiment.\n\nFor a comprehensive visualization of these refined insights, we refer readers to the attached PDF which also includes the updated barplots in Figure 8.\n\n> I see no conclusions applicable to real world scenarios in which LLM in-context learning is the go-to tool.\n\nWe thank the reviewer for raising the point regarding conclusions applicable to real-world scenarios. We agree that adding more real-world scenarios would further strengthen our paper. To address this point, we have added experiments on a natural language benchmark (MMLU) as described in more detail in our earlier response above.\n\n> The paper is not segmented into “method”, “experimental setup”, “results” sections.\n\nWe thank the reviewer for this feedback, which we have used to restructure the sections of the paper. In particular, we have moved each experimental subsection one level higher (e.g., “3.1 Learning one-dimensional functions” is now “4 Learning one-dimensional functions”). Furthermore, each of these sections is now divided into a methods and results subsection. Finally, we have renamed “3 Experimental analyses” to “3 Experimental setup”. We hope that these changes make it easier for readers to follow our presentation.\n\n> Minor remarks: \n** There are some some typos\n** The name “meta in-context learning” is used for a different method presented by Min et al. 2021 in this influential paper: “MetaICL: Learning to Learn In Context”. You may want to rename this paper. \n** I think that the authors overstate when they claim meta in-context learning to be related to psychological experiments.\n\nWe addressed the typos and agree that the link to psychological experiments is not clear so we removed the paragraph from our paper.\n\nIn addition, we want to thank the reviewer for mentioning the MetaICL paper. We have added it to the related work section, where we also describe how it differs from our approach:\n\n\"Meta-in-context learning versus classical meta-learning schemes:\n[...] **Min et al. \\cite{min2021metaicl} proposed a method called \\emph{meta-training for in-context learning} following the same paradigm. Their work starts with a pretrained language model that is then trained (via an adjustment of its weights) on a large set of training tasks to do in-context learning, ultimately leading to a model with improved in-context learning capabilities.** In contrast to these approaches, our approach adapts a model to a series of learning problems entirely through the context itself as opposed to updating weights. \"\n\nNote that MetaICL stands for “meta-training for in-context learning” and not for “meta-in-context learning”, so we do not think that an entire renaming of our terminology is necessary. \n"}, {"review_id": "hSF6Kvpk5N", "reviewer": "Reviewer_3waD", "summary": "The authors undertake a study of \"meta in-context learning\" as a capability of Large Language Models, specifically focused on GPT-3 (with some initial experiments on GPT-4). The authors define meta in-context learning following a task-trial structure, in which the agent observes multiple tasks each consisting of multiple trials. In each trial, a given input-output pair is observed; on the final trial, the model needs to predict the output for an unpaired input. The (unobserved) function generating outputs from inputs differs between tasks. The authors show that GPT-3 is able to learn not only at the trial level (traditional in-context learning) but also across tasks; that is, it can get better both at modelling a specific function and at the general task of modelling functions (in a particular context). The authors further show that this holds across artificial simple regression, reinforcement learning and multiple regression.on real-world data.", "questions": "Is it possible to create a clear definition of what constitutes an in-context trial and what constitututes an across-context trial, i.e. where does in-context learning become meta in-context learning? Does an intial task trial always start with some degree of irreduceable uncertainty? \n\nSee the above questions about real-world examples or non-numerical examples of meta in-context learning.", "limitations": "No major negative societal implications are expected. The authors address a number of limitations, including low sample size and simplicity of the tasks, in the Discussion.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is well-written, and the results are thoroughly investigated and clearly conveyed. The authors seek to identify the specific causes of the behaviours they document and the extent to which they can be attributed to simpler processes, e.g. learning the output distribution for predicting on the first task of a trial. The paper includes a discussion of context window size-related limitations to the applicability of the results.\n\nOn originality, I don't know of any other works that address the question of meta-learning without weight updates. As the authors point out, the results could inspire other researchers and applied LLM users to persue multi-task setups rather than finetuning.", "weaknesses": "The paper is well-reasoned within the specific niche of tasks demonstrated, though the overall implications of the work are arguable. Because the examples are constrained to simple numerical problems, it's difficult to tell whether the capability can be usefully leveraged on other types of language tasks. In the discussion, where it's argued that meta in-context could replace finetuning, it would help to give a few examples of real-world tasks that could be reframed as meta in-context.\n\nWithin the artificial tasks, it would also be nice to see non-numerical demonstrations of meta in-context learning, both to show that it is possible on other types of tasks and because LLMs tend to struggle with numerical transformations. For instance, within-task could involve translating sentences in a simple made-up language into English (with words that are repeated across trials), while each task involves a different made-up language. This could provide a more convincing argument that meta in-context learning is a general phenomenon beyond math tasks.\n\nMore detail on the exploration probit regression model and investigated regression coefficients in the Supplementary would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3waD,\n\nWe thank the reviewer for finding the results thoroughly investigated and clearly conveyed. We also appreciate the comment on originality and belief in its potential impact for applying LLMs to multi-task setups rather than fine-tuning. We also thank the reviewer for their helpful comments and we have made a response for each of their comments along with suggested changes to the paper.\n\n> The paper is well-reasoned within the specific niche of tasks demonstrated, though the overall implications of the work are arguable. Because the examples are constrained to simple numerical problems, it's difficult to tell whether the capability can be usefully leveraged on other types of language tasks. [...]\nWithin the artificial tasks, it would also be nice to see non-numerical demonstrations of meta in-context learning, both to show that it is possible on other types of tasks and because LLMs tend to struggle with numerical transformations. \n\nWe appreciate this suggestion which was echoed by all other reviewers as well. We have therefore conducted additional simulations of meta-in-context learning on the Massive Multitask Language Understanding (MMLU) benchmark. We will add the following section to our revised paper:\n\n\"**Meta-in-context learning on natural language processing benchmarks**:\n\nFinally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark \\cite{hendrycks2020measuring}.\n\n**Methods**:\nWe focus on the tasks from the STEM supercategory as other supercategories -- together with the addition of meta-in-context learning -- cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with $k \\in {0, 1, 2}$ examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from different categories.\n\n**Results**:\nFigure 9 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of $55.1$ percent outperforming in-context by $22.4$ percent. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 9.\n\n> More detail on the exploration probit regression model and investigated regression coefficients in the Supplementary would be useful.\n\nWe agree with the reviewer that more details on this model would be helpful to the reader. We have therefore added a section to the Supplementary Materials called “Details on probit regression analysis” which describes the corresponding analysis in more detail:\n\n\"**Details on probit regression analysis**:\n\nWe have relied on a probit regression analysis proposed by Gershman \\cite{gershman2018deconstructing} to investigate which exploration strategies GPT-3 applies when interacting with two-armed bandit problems. This analysis assumes that the agent makes decisions based on three features: value difference $V_t$, relative uncertainty $RU_t$, and value difference divided by total uncertainty $V_t/TU_t$. Formally, these quantities are defined as follows:\n\nEQUATION\n\nwhere $\\mu_{a, t}$ and $\\sigma_{a, t}$ represent the agent’s beliefs about the mean reward and its corresponding uncertainty estimate for a given arm $a$ at trial $t$. We compute these values using via a sequential application of Bayesian inference, assuming normally-distributed priors and likelihoods (updates are only performed for the selected arm):\n\nEQUATION\n\nwhere $\\tau$ corresponds to the additive observation noise.\nThe resulting features are then entered into a probit regression model whose parameters are fit to agent choices via a maximum likelihood estimation using the statsmodels library \\cite{seabold2010statsmodels}. We additionally include an interaction effect with task number $k$ for each feature to investigate how exploration behavior changes across tasks. The final model thus contains six features:\n\nEQUATION\n\nThis probit model subsumes several well-known exploration strategies for specific settings of its parameters:\n\n1. Boltzmann-like exploration for $\\mathbf{w} = [\\mathbf{w}_{1}, 0, 0, 0, 0, 0]$. Note: Boltzmann exploration would use the logit function instead of the probit. However, the two can used to closely approximate each other: $\\sigma(a) \\simeq \\mathbf{\\Phi}(\\sqrt{\\frac{\\pi}{8}}a)$.\n2. a noisy version of the UCB algorithm for $\\mathbf{w} = [\\mathbf{w}_1, \\mathbf{w}_2, 0, 0, 0, 0]$.\n3. Thompson sampling for $\\mathbf{w} = [0, 0, 1, 0, 0, 0]$. For an exact derivation, see Gershman \\cite{gershman2018deconstructing}.\"\n\n> Is it possible to create a clear definition of what constitutes an in-context trial and what constitututes an across-context trial, i.e. where does in-context learning become meta in-context learning? Does an intial task trial always start with some degree of irreduceable uncertainty?\n\nThank you for this comment. We view the relationship between in-context learning and meta-in-context learning as similar to the relationship between Bayesian inference and hierarchical Bayesian inference. They are both based on the same algorithmic principles applied at different conceptual levels. In-context learning and Bayesian inference are used for within-task learning, while meta-in-context learning and hierarchical Bayesian inference are used to pool information across tasks.\n"}, {"review_id": "pPtibdtKMk", "reviewer": "Reviewer_NP3F", "summary": "The authors demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself.  \n\nThe paper misses the method section. I don't know the details and cannot tell the difference from previous work. \n\nI didn't get the novel part of the method. It seems an empirical study. \n\nThe authors had better do some experiments on benchmark datasets, such as the datasets from GPT-3 paper.", "questions": "Can you formulate meta-in-context learning? ", "limitations": "I didn't get the novel contribution of the work.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem explored is critical.\n2. The experimental analyses are interesing.", "weaknesses": "1. Missing model section. \n2. The method is more of an empirical study.\n3. The experiments are not solid. The authors focus on the GPT-3 model exploration but miss comparison to any benchmark dataset from GPT-3 paper. The shot number, context length, and language understanding in benchmark datasets are all critical issues to study. And GPT-3 is released quite a long time ago. Exploring recent LLMs, such as ChatGPT or GPT-4, would be better.\n4. The finding is similar to GPT-3 on domain adaptation, such as the work from \"Prompting GPT-3 To Be Reliable\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NP3F,\n\nWe thank the reviewer for their helpful comments and we have made a response for each of their comments along with suggested changes to the paper:\n\n> Strengths:\nThe problem explored is critical.\nThe experimental analyses are interesing.\n\nWe thank the reviewer for exposing the research problem as critical and the analyses as interesting.\n\n> Weaknesses:\nMissing model section.\nThe method is more of an empirical study.\n\nWe thank the reviewer for the feedback. However, we do not view this as a weakness. NeurIPS publishes hundreds of empirical studies every year that do not propose new models.\n\n> The experiments are not solid. The authors focus on the GPT-3 model exploration but miss comparison to any benchmark dataset from GPT-3 paper. The shot number, context length, and language understanding in benchmark datasets are all critical issues to study.”\n\nWe appreciate the criticism. In our revised paper, we have conducted additional simulations of meta-in-context learning on the Massive Multitask Language Understanding (MMLU) benchmark. We will add the following section to our revised paper:\n\n\"**Meta-in-context learning on natural language processing benchmarks**:\n\nFinally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark \\cite{hendrycks2020measuring}.\n\n**Methods**:\nWe focus on the tasks from the STEM supercategory as other supercategories -- together with the addition of meta-in-context learning -- cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with $k \\in {0, 1, 2}$ examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from \\emph{different} categories.\n\n**Results**:\nFigure 9 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of $55.1$ percent outperforming in-context by $22.4$ percent. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 9.\n\n> And GPT-3 is released quite a long time ago. Exploring recent LLMs, such as ChatGPT or GPT-4, would be better.\n\nWe appreciate the feedback and agree that more analysis on more recent LLMs (especially open-source models which are transparent on training and architectures) does improve the empirical analysis. Therefore, we conducted additional simulations of meta-in-context learning on the latest open-source models (MPT-30B, Falcon-40B, Llama-2-7B/13B/70B). We will add the following section to our revised paper:\n\n\"**Meta-in-context learning in open-source models**: \nTo comprehensively explore meta-in-context learning, we extended our evaluation to encompass five distinct open-source models. This expansion allowed us to examine whether this phenomenon is also applicable to less opaque model architectures. Notably, while the majority of models  (Falcon-40B \\cite{falcon40b} and Llama-2 \\cite{touvron2023llama2} models) exhibited no indications of meta-in-context learning, intriguingly, MosaicML's MPT-30B \\cite{MosaicML2023Introducing} demonstrated this ability. The results are included in the Supplementary material.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 6B.\nWe also want to point out to the reviewer that the GPT-4 model also was already included for the real-world regression in the Supplementary materials (see Figure 5).\n\n> The finding is similar to GPT-3 on domain adaptation, such as the work from \"Prompting GPT-3 To Be Reliable\".\n\nWe thank the reviewer for mentioning this paper. We have included a reference to it in our discussion. That being said, we do not think this paper is particularly relevant to our work (besides the fact that both are concerned with in-context learning). While Si et al. propose ways to make in-context learning more reliable using prompt engineering, we show that GPT-3 is capable of adapting its in-context learning abilities to previously encountered related tasks without the need for prompt engineering. In addition, \"Prompting GPT-3 to be reliable\" focuses on different metrics (4 aspects of reliability), whereas our paper focuses on a different ability (meta-learning). \n\n> Questions:\nCan you formulate meta-in-context learning?\n\nWe believe that Figure 1 provides a nice formulation of what we view as meta-in-context learning: \n\n“High-level overview of our approach on an example of multiple three-shot regression tasks. We present an LLM with $N$ learning tasks in a row. Improvement within a task indicates that the model is capable of in-context learning. If in-context learning improves across multiple learning tasks, the model is also capable of meta-in-context learning.”\n"}, {"review_id": "ZnWQJZuYIg", "reviewer": "Reviewer_6CRk", "summary": "The paper studies an ability of LLMs, called meta-in-context learning, which showcases that LLMs can recursively improve their in-context learning with demonstrations. The authors illustrate this capability with a regression task and a two-armed bandit task. The analysis demonstrates that LLMs are not only able to learn the task (underlying function) from the examples in this task but also can leverage the examples from other tasks (underlying functions).", "questions": "1. On line 118, it is mentioned that the GPT-3 is not told that underlying functions are linear, distinct from BLR. However, this raises the question of whether GPT-3 possesses the capability to handle non-linear functions. Additionally, if GPT-3 sees a few tasks with linear functions, is GPT-3 able to generalize to a non-linear function in a new task? The meta-in-context learning ability is more valuable if the tasks can be more different.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper highlights the meta-in-context learning capability of LLMs. The capability allows LLMs to be recursively improved via in-context learning with more tasks in context.\n2, The paper is clearly written and easy to follow.", "weaknesses": "1. The analysis in the paper demonstrates that LLMs can leverage the examples from other tasks, however, the definition of a task appears to be rather limited in this paper. In the linear regression experiment, for example, a task represents the parameters within a linear function. \n\n2. The authors introduce the concept of meta-in-context learning as a novel ability of LLMs, but it remains unclear how this differs from the traditional in-context learning ability. In this paper, each regression function is regarded as a task with a few examples following the function, and LLMs are found to be able to learn from other tasks. However, this ability seems to have been demonstrated in various other applications. For instance, LLMs can learn to generate dialog responses by observing example responses from other dialogs. If we apply the terminology used in this paper, each dialog can also be considered a task, with each individual utterance serving as an example.\n\nWhile I think the analysis in the regression and two-armed bandit tasks is commendable, it would be valuable to see such analysis on more realistic tasks, such as dialog generation (a dialog is a task),  and passage/image question answering (a passage or image is a task).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6CRk,\nWe thank the reviewer for their helpful comments and we have made a response to each of their comments along with suggested changes to the paper:\n\n> 1. The paper highlights the meta-in-context learning capability of LLMs. The capability allows LLMs to be recursively improved via in-context learning with more tasks in context. 2, The paper is clearly written and easy to follow.\n[...] I think the analysis in the regression and two-armed bandit tasks is commendable\n\nWe appreciate the reviewer’s comments on the clarity and presentation of the paper as well as exposing the two-armed bandit task as commendable.\n\n> 2. The authors introduce the concept of meta-in-context learning as a novel ability of LLMs, but it remains unclear how this differs from the traditional in-context learning ability. In this paper, each regression function is regarded as a task with a few examples following the function, and LLMs are found to be able to learn from other tasks. However, this ability seems to have been demonstrated in various other applications. For instance, LLMs can learn to generate dialog responses by observing example responses from other dialogs. If we apply the terminology used in this paper, each dialog can also be considered a task, with each individual utterance serving as an example.\n\nThank you for this comment. We view the relationship between in-context learning and meta-in-context learning as similar to the relationship between Bayesian inference and hierarchical Bayesian inference. They are both based on the same algorithmic principles applied at different conceptual levels. In-context learning and Bayesian inference are used for within-task learning, while meta-in-context learning and hierarchical Bayesian inference are used to pool information across tasks.  Formulating explicit tasks and episodes in the way that we do allows us to quantify the degree of meta-in-context learning, whereas simply having dialogue responses generated following from other dialogue is hard to quantify.\n\n> While I think the analysis in the regression and two-armed bandit tasks is commendable, it would be valuable to see such analysis on more realistic tasks, such as dialog generation (a dialog is a task), and passage/image question answering (a passage or image is a task).\n\nWe appreciate this suggestion which was echoed by all other reviewers as well. We have therefore conducted additional simulations of meta-in-context learning on the Massive Multitask Language Understanding (MMLU) benchmark. We will add the following section to our revised paper:\n\n\"**Meta-in-context learning on natural language processing benchmarks**:\n\nFinally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark \\cite{hendrycks2020measuring}.\n\n**Methods**:\nWe focus on the tasks from the STEM supercategory as other supercategories -- together with the addition of meta-in-context learning -- cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with $k \\in {0, 1, 2}$ examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from \\emph{different} categories.\n\n**Results**:\nFigure 9 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of $55.1$ percent outperforming in-context by $22.4$ percent. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.\"\n\nThe figure with the corresponding results can be found in the attached PDF under Figure 9.\n\n> On line 118, it is mentioned that the GPT-3 is not told that underlying functions are linear, distinct from BLR. However, this raises the question of whether GPT-3 possesses the capability to handle non-linear functions. Additionally, if GPT-3 sees a few tasks with linear functions, is GPT-3 able to generalize to a non-linear function in a new task? The meta-in-context learning ability is more valuable if the tasks can be more different.\n\nWe appreciate the feedback and agree that additional evaluation on non-linear functions does make the analysis more comprehensive. We have therefore added the following section to our revised paper:\n\n\"**Meta-in-context learning with non-linear functions**:\nWe also aimed to determine whether GPT-3 could use meta-in-context learning to adapt to non-linear functions. In the Supplementary material, we present an analysis using quadratic functions conducted within the same experimental framework. In summary, we find that GPT-3 indeed exhibits the ability to meta-in-context learn non-linear functions.\"\n\nFor a comprehensive visualization of these refined insights, we refer readers to Figure 7 in the attached PDF.\n"}]

Author rebuttal:
We would like to thank all reviewers for their valuable and thoughtful feedback.

* Reviewer ShPz found “the paper an enjoyable read” and stated that “the [our] research is relatively well motivated, the analysis are carefully detailed, and the method is clearly explained.”
* Reviewer xRJY said that the “experimental setup is clean and the obtained experimental results convey a clear message.”
* Reviewer 3waD said that our paper “is well-written, and the results are thoroughly investigated and clearly conveyed.”
* Reviewer 6CRk called the paper “clearly written and easy to follow.”

Furthermore, the paper scored an average of 3.0 on both soundness and presentation.

However, all reviewers also raised important points and provided helpful suggestions. We were able to incorporate all of these suggestions and believe that doing so has improved our paper significantly. To summarize, we have made the following modifications:

* We tested meta-in-context learning on a natural language processing benchmark (MMLU), where we found good performance (requested by all reviewers).
* We have included experiments with **eight** additional models to investigate:
 	1. whether meta-in-context learning is an emergent phenomenon that arises at scale (reviewer ShPz).
 	2. whether meta-in-context learning can also be found in open-source models (reviewer ShPz).
* We ran additional experiments with non-linear (quadratic functions), where we also found meta-in-context learning to be beneficial (reviewer 6CRk).
 * We performed additional analyses to investigate how task similarity affects meta-in-context learning performance (reviewers xRJY).
 * We incorporated references suggested by the reviewers (reviewers NP3F, xRJY, ShPz).
 * We added more details on the probit regression analysis from the two-armed bandit task (reviewers 3waD).
* We fixed reference issues, typos and improved the structure slightly.

We describe these changes in detail in our responses to the individual reviews below. We again want to thank the reviewers for their time and for actively taking part in the review process.


Meta-review: This paper documents meta-in-context learning, whereby including a set of tasks in a prompt can provide additional improvements compared with only examples from the intended task.  Personally, I don’t find this phenomenon surprising in light of in-context learning, but I agree that it is worth identifying and studying, and will be relevant to the NeurIPS community.  Reviewers seem satisfied with the experiments, presentation, and overall quality of the work, especially after the authors’ response and updates.  Furthermore, as highlighted in discussions with reviewers, meta-in-context learning could prove useful for practitioners in small data settings.  I recommend accepting the paper, and expect that the authors will ensure their updated manuscript includes all of the necessary changes.
Decision: Accept (poster)