Title: Meta-in-context learning in large language models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large language models

in-context learning

meta-learning

GPT-3
Abstract: Large language models have shown tremendous performance in a variety of tasks.  In-context learning  -- the ability to improve at a task after being provided with a number of demonstrations -- is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in-context learning. Looking at two idealized domains, a one-dimensional regression task and a two-armed bandit task, we show that meta-in-context learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we find that meta-in-context learning modifies the in-context learning strategies of such models. Finally, we broaden the scope of our investigation to encompass two diverse benchmarks: one focusing on real-world regression problems and the other encompassing multiple NLP tasks. In both cases, we observe competitive performance comparable to that of traditional learning algorithms. Taken together, our work improves our understanding of in-context learning and paves the way toward adapting large language models to the environment they are applied purely through meta-in-context learning rather than traditional finetuning.

Meta-in-context learning in large language models

Abstract

Large language models have shown tremendous performance in a variety of tasks. In-context learning - the ability to improve at a task after being provided with a number of demonstrations - is seen as one of the main contributors to their success. In the present paper, we demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself. We coin this phenomenon meta-in-context learning . Looking at two idealized domains, a one-dimensional regression task and a two-armed bandit task, we show that meta-in-context learning adaptively reshapes a large language model's priors over expected tasks. Furthermore, we fi nd that meta-in-context learning modi fi es the in-context learning strategies of such models. Finally, we broaden the scope of our investigation to encompass two diverse benchmarks: one focusing on realworld regression problems and the other encompassing multiple NLP tasks. In both cases, we observe competitive performance comparable to that of traditional learning algorithms. Taken together, our work improves our understanding of in-context learning and paves the way toward adapting large language models to the environment they are applied purely through meta-in-context learning rather than traditional fi netuning.

1 Introduction

Large language models (LLMs) are taking not only machine learning research but also society by storm [1, 2, 3]. Part of what makes these models so persuasive is that their abilities reach far beyond what we expected pure language models to do. They can, among other things, solve challenging reasoning problems, including university-level math questions [4] or analogical reasoning tasks [5], out-of-the-box and without additional training.

Much of this power comes from what is known as in-context learning [6]. In-context learning (sometimes also called few-shot learning or few-shot prompting) refers to the ability of an LLM to improve at a given task after being provided with a number of task-relevant demonstrations. This ability sets LLMs apart from traditional models and led to a totally new paradigm - one which eschews fi netuning of weights on task-speci fi c data altogether and instead relies entirely on contextual information.

In the present paper, we ask whether the learning algorithm implemented through in-context learning can be improved through in-context learning itself (without the need for any further fi netuning of parameters). To study this question, we conducted several experiments where we presented an LLM with multiple learning tasks in a sequence. In three distinct settings, we fi nd evidence for the idea that the in-context learning abilities of an LLM can be recursively enhanced via in-context learning, thereby displaying a form of meta-in-context learning . Figure 1 provides a high-level overview of our approach on an example supervised learning task.

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=1

x=2, y=3

x=5, y=9

[…]

Task N:

x=0, y=1

x=2, y=5

x=5, y=

Try to predict y accurately:

…

In-context learning

Meta-in-context learning

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=1

x=2, y=3

x=5, y=9

[…]

Task N:

x=0, y=1

x=2, y=

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=1

x=2, y=3

x=5, y=9

[…]

Task N:

x=0, y=

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=1

x=2, y=3

x=5, y=

Try to predict y accurately:

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=1

x=2, y=

Task 1:

x=1, y=2

x=3, y=6

x=4, y=8

Task 2:

x=1, y=

Task 1:

x=1, y=2

x=3, y=6

x=4, y=

Try to predict y accurately:

Task 1:

x=1, y=2

x=3, y=

Task 1:

x=1, y=

More speci fi cally, we fi rst investigate meta-in-context learning on two arti fi cial domains: a supervised function learning task, and a two-armed bandit task. For both of them, we fi nd that sequentially presenting LLMs with multiple learning problems boosts their in-context learning abilities. We then use these idealized domains to identify the drivers behind meta-in-context learning. We fi nd that meta-in-context learning adaptively modi fi es priors over latent variables, ultimately leading to priors that closely resemble the true statistics of the environment. Furthermore, our analysis reveals that meta-in-context learning can not only be used to change prior expectations but is also capable of reshaping an LLM's learning strategies. Lastly, we apply our approach to two realistic domains. We fi rst demonstrate that meta-in-context learning can be used to obtain a learning algorithm that is competitive with traditional algorithms on a benchmark of real-world regression problems. We then validated the transferability of our fi ndings to an NLP benchmark, con fi rming the effectiveness of our approach in diverse contexts.

2 Related work

In-context learning: Recent work has shown that LLMs can improve their performance after being shown a few task-relevant demonstrations - an ability referred to as in-context learning [6]. When and why in-context learning emerges is a matter of ongoing debate, with different theories being proposed. Chan et al. [7] argued that properties of the training data - such as burstiness and non-stationarity are key drivers behind in-context learning. Min et al. [8], on the other hand, found that ground truth demonstrations can be replaced with random labels while barely hurting performance, suggesting that the role of demonstrations is more to prime the model for a particular task. Finally, Xie et al. [9] suggested that LLMs internally need to infer latent variables to make better predictions about future word occurrences, thereby implementing a form of Bayesian inference.

In-context learning can solve classical learning tasks: If LLMs do apply some form of Bayesian inference, then one would expect an LLM to also be able to solve classical online learning tasks, such as regression or classi fi cation, purely through in-context learning. Previous research suggests that this is indeed the case. Lovre [10], for instance, tested GPT-3 on a range of low-dimensional classi fi cation and regression tasks and found that it was often on par with classical learning algorithms such as logistic regression. Likewise, Hegselmann et al. [11] tested an LLM's few-shot classi fi cation abilities on tabular data. They found that their approach outperforms prior deep-learning-based tabular classi fi cation methods on several benchmark datasets, and that performance further improves when

the model is provided with semantic information about the data. 1 LLMs are not only able to solve supervised learning problems but also simple reinforcement learning tasks. To provide one example, Binz & Schulz [12] evaluated GPT-3 on a two-armed bandit task and found that its performance exceeded that of human participants who did the corresponding psychological experiment.

Meta-in-context versus classical meta-learning schemes: Meta-in-context learning stands in contrast to classical meta-learning schemes in which one adapts a neural network to a distribution over learning problems by adjusting its weights [13, 14]. Historically, this approach has relied on recurrent networks [15, 16, 17, 18] but, more recently, researchers have also started to use transformerbased architectures [19, 20, 21]. For example, Garg et al. showed 'transformers can be trained from scratch to perform in-context learning of linear functions' and that the resulting models achieve 'performance comparable to the optimal least squares estimator.' In a similar vein, von Oswald et al. argued that 'transformers [can in principle] implement gradient descent in their forward pass' and provide empirical evidence that they indeed do so [19]. In contrast to these approaches, our approach adapts a model to a series of learning problems entirely through the context itself as opposed to updating weights.

3 Experimental setup

We used the public OpenAI Python API [22] to run all our simulations. This API provides access to several LLMs from the Generative Pre-trained Transformer (GPT) family. We ran all our simulations on the TEXT-DAVINCI-002 model, which is also known as GPT-3. We set the temperature parameter to zero (leading to deterministic responses) unless otherwise noted and retained the default values for all other parameters. It is important to note that all experiments performed in this paper rely entirely on the in-context learning abilities of an LLM, and do not involve any form of fi netuning.

4 Learning one-dimensional functions

In our fi rst set of experiments, we investigated GPT-3's ability for meta-in-context learning in a simple one-dimensional supervised regression task. In this setting, we provided GPT-3 with a list of input-target pairs from a given task and asked it to make accurate predictions for a new input value. This is one of the most fundamental machine learning problems, and its simplicity makes it an ideal testbed for an initial analysis.

4.1 Methods

Function learning prompt

You observe 5 machines that produce an output y for a given input x . Each machine implements a different function.

Every task n consisted of T = 5 input-target pairs ( x t , f ( x t )) , where t ∈ [1 , T ] denotes the trial number. Each input-target pair was generated by a linear function of the form f ( x t ) = a ( n ) x t + b ( n ) + ε t , where a ( n ) and b ( n ) are task-speci fi c parameters drawn from a probability distribution p ( T ) . Inputs x t were sampled from U (0 , 100) and the trial-speci fi c additive noise ε t was sampled from N (0 , 1) . For our meta-in-context learning simulations, we considered prompts that include data from up to fi ve tasks, each corresponding to a different underlying function. Following the schema

A

B

C

1

2

3

4

5

Trial

0

100

200

MSE

GPT-3

GPT-3 after 5 tasks

BLR (default)

BLR (oracle)

1

2

3

4

5

Tasks

40

60

80

MSE

Trial Task

-

20

0

Regression

coe

ffi

cients

0

100

x

-

500

0

500

f(x)

Task 1

0

100

x

Task 2

0

100

x

Task 3

0

100

x

Task 4

0

100

x

Task 5

Task distribution

GPT-3 priors

D

outlined in Figure 1, we iteratively presented each data-point together with the history of previous observations, starting from the fi rst trial in task one up to the last trial in task fi ve. In the 'function learning prompt' box above, you can see an example prompt that we provided to GPT-3.

4.2 Results

In preliminary simulations, we found that GPT-3 has a strong bias toward increasing positive functions. To demonstrate the potential of our approach, we wanted to investigate whether it is possible to overwrite this bias via meta-in-context learning. In order to achieve this goal, we sampled taskspeci fi c functions with a negative slope and intercept, i.e. a ( n ) ∼ N ( -2 , 1) and b ( n ) ∼ N ( -100 , 1) , and evaluated how observing multiple such tasks in fl uences the model's behavior.

GPT-3 does in-context learning: We fi rst established that GPT-3 can learn within a single task without considering the effects of meta-in-context learning. To do so, we only examined the fi rst task while ignoring all the subsequent ones. We found that performance as measured by the mean-squared error (MSE) improves with additional data-points as shown in Figure 2A (solid lines). GPT-3 matches (or even slightly outperforms) a Bayesian linear regression (BLR) model with default standard normal priors, indicating that in-context learning is able to solve the given problem. This is especially noteworthy since GPT-3 was never told that the underlying functions are linear, as opposed to BLR which has a built-in assumption of linearity.

GPT-3 does meta-in-context learning: Next, we investigated whether GPT-3 is capable of meta-incontext learning. For this, we inspected how performance changes across the fi ve tasks. Figure 2A demonstrates that meta-in-context learning is bene fi cial by comparing performance between the fi rst

and the fi nal task (solid vs dashed lines). For reference, we also plotted the performance of a BLR model with access to the ground-truth data-generating distribution. Note that this model has access to privileged information and hence only serves as a performance upper-bound. Figure 2B shows a more detailed development of performance as we increase the number of tasks in the prompt.

To test whether the effects of in-context learning and meta-in-context learning are statistically meaningful, we fi tted a linear regression model that included the trial and task number as independent variables on the MSE. We found statistically signi fi cant effects for both trial number ( β = -30 . 614 ± 3 . 08 , t = -19 . 514 , p < 0 . 001 ) and task number ( β = -13 . 26 ± 3 . 08 , t = -8 . 455 , p < 0 . 001 ), con fi rming that GPT-3 is capable of both in-context and meta-in-context learning (see Figure 2C).

Meta-in-context learning is driven by adaptation of priors: We speculated that GPT-3's performance improves during meta-in-context learning because it adapts its priors to true environmental statistics. In order to evaluate this hypothesis, we collected GPT-3's prior expectations before each task by asking it to make sequential predictions for 20 evenly spaced input values (using the same prompt template as above, but setting the temperature to one and feeding back the model's own predictions as the training data). We omitted outlier predictions that had absolute values of 10 , 000 or larger for this analysis. The estimated priors indicate that GPT-3 has an initial bias toward increasing positive functions. However, already after two tasks it adapts its priors to decreasing negative functions, thereby closely matching the true environmental statistics (see Figure 2D). Furthermore, the prior for the intercept term is initially centered around zero but shifts toward the ground-truth value of -100 after three tasks.

Taken together, these simulations provide initial evidence that GPT-3 engages in meta-in-context learning, meaning that its in-context learning abilities can be recursively improved via in-context learning itself. We have suggested and empirically con fi rmed that GPT-3 accomplishes this by adapting its priors across multiple tasks.

Meta-in-context learning is an emergent phenomenon: In order to gain a deeper understanding of the phenomenon's characteristics, we pursued an examination into the progression of meta-in-context learning pro fi ciency in relation to both model complexity and dataset size. To this end, we undertook an analysis encompassing smaller GPT-3 models, speci fi cally text-ada, text-cabbage, and text-curie [22]. Our observations (which can be found in Figure 5A, Appendix A.1.1) reveal a noteworthy trend, wherein solely text-davinci-002 seems to exhibit meta-in-context learning capabilities for a negative linear function.

Meta-in-context learning in open-source models: To comprehensively explore meta-in-context learning, we extended our evaluation to encompass fi ve distinct open-source models. This expansion allowed us to examine whether this phenomenon is also applicable to less opaque model architectures. Notably, while the majority of models (Falcon-40b [23] and Llama-2 [24] models) exhibited no indications of meta-in-context learning, intriguingly, MosaicML's mpt-30b [25] demonstrated an ability to manifest this phenomenon. We hypothesize that models trained in a regime with a longer context window are more suitable to show the emergence of this phenomenon. Indeed, mpt-30b has a context-size of 8000 tokens as opposed to the LLaMa-2 models which have been trained with a context window of 4096 tokens. This is speculative and we leave the analysis of what factors in fl uence the emergence of meta-in-context learning as a future research question. The results are included in Figure 5B, Appendix A.1.1.

Meta-in-context learning with non-linear functions: We also aimed to determine whether GPT-3 could also adapt its meta-in-context learning to non-linear functions. In Figure 6, Appendix A.1.2, we present an analysis of a quadratic function conducted within the same experimental framework. Our fi ndings indicate that the model indeed exhibits the ability to meta-in-context learn non-linear functions.

5 Experiments on two-armed bandit tasks

In a next step, we wanted to investigate whether our results from the supervised setting also transfer to a reinforcement learning paradigm. This setting adds an additional layer of complexity because it requires the agent to learn from its own experiences instead of having access to ground-truth solutions of previous tasks. In addition, it allows us to investigate learning strategies and how they evolve during meta-in-context learning.

5.1 Methods

For our simulations, we considered a simple two-armed bandit task, in which an agent repeatedly interacts with two slot machines. In each trial, the agent can select one of two machines and is rewarded based on a probability distribution that is associated with that machine. The agent's objective is to maximize the total amount of acquired points. We used a cover story that involves a gambler visiting different casinos, which has been used in human experiments with similar tasks [26, 12], to generate our prompts:

Two-armed bandit task prompt

You are going to different casinos that own two slot machines.

Choosing the same slot machine will not always give you the same points, but one slot machine is always better than the other. Within a casino, your goal is to choose the slot machine that will give you the most points over the course of 10 trials. Each casino owns a different pair of machine.

You have received the following points when playing in casino 1:

You have received the following points when playing in casino 5:

Machine F delivered -7.4 points.

Machine J delivered 4.2 points.

Q: We are now performing trial 3 in casino 5. Which machine do you choose between machine J and machine F?

A: Machine [insert].

5.2 Results

For each task, we sampled independent mean rewards for each machine from N (0 , √ 64) . The actually obtained reward is generated using the mean reward that corresponds to the chosen machine plus some additive Gaussian noise sampled from N (0 , √ 32) . To stay consistent with the previous section, we considered prompts that include data from up to fi ve different tasks for our meta-in-context learning simulations (each of these tasks consisted of ten trials). We used letters to indicate the different slot machines. For each task, we randomly sampled two different letters without replacement to cancel out biases towards certain letters. 2

GPT-3 does in-context learning: We again fi rst tested whether GPT-3 learns within a single task. Figure 3A con fi rms that this is indeed the case. Performance (measured in terms of regret) improves over the fi rst four trials and plateaus afterward. However, when comparing GPT-3 to two baseline algorithms - a greedy policy and an upper con fi dence bound (UCB) algorithm - we observed that it lags in performance prior to meta-in-context learning.

GPT-3 does meta-in-context learning: Like in our previous experiment, we found that GPT-3 improves during meta-in-context learning: as it observes more tasks, it gets better at solving new two-armed bandit problems (see Figure 3B). GPT-3's performance matches that of a greedy policy after having interacted with only four tasks and lags only slightly behind that of the UCB algorithm.

We quanti fi ed the effects of in-context and meta-in-context learning in a statistical analysis. To do so, we fi tted a linear regression model including the trial and task number as independent variables on the trial-wise regret. We found signi fi cant in-context learning ( β = -0 . 221 ± 0 . 012 , t = -35 . 828 , p < 0 . 001 ) and meta-in-context learning effects ( β = -0 . 031 ± 0 . 012 , t = -5 . 002 , p < 0 . 001 ), thereby reproducing our results from the previous section (see Figure 3C).

Meta-in-context learning is driven by adaptation of priors: Following our earlier analysis, we speculated that part of this performance boost arises because GPT-3 adapts its priors toward the statistics of the tasks that were encountered during meta-in-context learning. To verify this, we

A

B

C

2

4

6

8

10

Trials

0

2

4

Regret

GPT-3

GPT-3 after 5 tasks

Greedy

UCB

1

2

3

4

5

Tasks

1

.

0

1

.

5

2

.

0

Regret

GPT-3

Greedy

TrialTask

-

0

.

20

-

0

.

15

-

0

.

10

-

0

.

05

0

.

00

Regression

coe

ffi

cients

D

E

1

2

3

4

5

True

Tasks

0

100

200

Reward priors

V

RU

V/TU

Task

×

V

Task

×

RU

Task

×

V/TU

-

0

.

5

0

.

0

Regression

coe

ffi

cients

probed its prior expectations before starting each task by asking 'how rewarding do you expect machine X to be?' We set the temperature parameter = 1 and repeated this question fi ve times to re fl ect a sampling from a prior distribution. Figure 3D visualizes the change of priors across tasks. Before the initial task, GPT-3 expects rewards to be distributed around larger positive values ( M = 545 . 22 , SD = 6359 ). However, at the end of meta-in-context learning, its priors match the true data-generating distribution closely ( M = 6 . 49 , SD = 4 . 41 ).

In addition to looking at the development of priors, the two-armed bandit setting also allows us to investigate whether any changes in strategies happen during meta-in-context learning. For this, we relied on an analysis originally proposed by Gershman [26]. The idea behind this analysis is to de fi ne a model that involves a parameterized combination of Boltzmann exploration, a UCB algorithm, and Thompson sampling, and then fi t the parameters of this model to data generated by an agent (GPT-3 in our case). It is then possible to determine the extent to which the agent relied on a speci fi c strategy by examining the resulting parameters.

We assume that the agent's beliefs over expected rewards at trial t and action a are captured by the normal distribution p ( r a,t ) = N ( µ a,t , σ a,t ) 3 and de fi ne the following probit regression model based on the parameters of these distributions:

with Φ denoting the cumulative distribution function of a standard normal distribution. Equation 1 recovers a Boltzmann-like exploration strategy for [ w 1 , w 2 , w 3 ] = [ c, 0 , 0] , a variant of the UCB

algorithm for [ w 1 , w 2 , w 3 ] = [ c, d, 0] , and Thompson sampling for [ w 1 , w 2 , w 3 ] = [0 , 0 , 1] [27]. In our analysis, we furthermore included an interaction effect with task number for each factor to investigate how the applied strategies change during meta-in-context learning.

Meta-in-context learning reshapes learning strategies: We found a positive main effect of the value difference V t ( β = 0 . 307 ± 0 . 027 , z = 21 . 759 , p < 0 . 001 ), indicating that GPT-3 engages in Boltzmann exploration. GPT-3 becomes more greedy during meta-in-context learning as shown by the positive interaction effect between task number and V t ( β = 0 . 128 ± 0 . 025 , z = 9 . 633 , p < 0 . 001 ). Furthermore, we found negative main effects for both relative uncertainty RU t ( β = -0 . 160 ± 0 . 010 , z = -31 . 788 , p < 0 . 001 ) and the uncertainty-scaled value difference V t / TU t ( β = -0 . 400 ± 0 . 025 , z = -5 . 393 , p < 0 . 001 ), suggesting that GPT-3 avoids uncertain options by default. However, we also found a slight, but signi fi cant, increase in UCB-based decisions during meta-in-context learning ( β = 0 . 046 ± 0 . 010 , z = 9 . 279 , p < 0 . 001 ). Figure 3E shows a visualization of all regression coef fi cients involved in this analysis.

The results presented in this section corroborate those obtained from the supervised setting. GPT-3 generally performed better in a two-armed bandit task after meta-in-context learning. We again observed that GPT-3 accomplishes this by adapting its priors across tasks. In addition, we investigated changes in strategies during meta-in-context learning and found that GPT-3 learned to perform better by exploiting more consistently.

6 Regression on real-world data

Next, we wanted to investigate whether our results obtained in the arti fi cial domains scale up to real-world applications. To test this, we considered a multi-dimensional regression benchmark which consists of 60 different real-world datasets introduced in [28].

6.1 Methods

For each simulation, we randomly selected fi ve different tasks from the benchmark and then sampled fi ve points without replacement for each task. We used fi ve features for all tasks. If a dataset contained less than fi ve features, we omitted it from our analysis, which yielded 42 remaining datasets. For tasks exceeding fi ve features, we used a sub-selection procedure that retained only the top fi ve features based on their F-value with respect to the target variable evaluated in a univariate linear regression. To maintain a consistent regression loss across all tasks, we normalized both the feature and target spaces to the interval of [ -1 , 1] . The resulting prompts follow the general template outlined earlier:

Regression on real world data

You observe an input vector x and have to predict the corresponding output y as accurately as possible. You are given 5 different tasks.

6.1.1 Results

GPT-3 does in-context and meta-in-context learning: Following the previous experiments, we investigated the learning curves of GPT-3 and GPT-3 after meta-in-context learning. We additionally included two baselines in our analysis: BLR and a random forest model. We measured performance by the root-mean-squared error (RMSE). Figure 4A shows the learning curve for all these models. Like in the previous cases, we found that GPT-3 improves with additional data-points, con fi rming that it is capable of in-context learning in this setting. In addition, meta-in-context learning further improved performance. GPT-3 after meta-in-context learning matched the performance of BLR and was only slightly outperformed by the random forest model.

A

B

C

1

2

3

4

5

Trials

0

.

4

0

.

6

0

.

8

RMSE

GPT-3

GPT-3 after 5 tasks

RandomForest

BLR

1

2

3

4

5

Trials

0

10

20

30

Extreme

predictions (

%

)

GPT-3

GPT-3 after 5 tasks

Trial

Task

similarity

-

0

.

15

-

0

.

10

-

0

.

05

0

.

00

Regression

coe

ffi

cients

Meta-in-context learning constrains predictions: Most of the improvement of meta-in-context learning seems to come from zero-shot performance. We hypothesized that this is the case because GPT-3 acquires an understanding of potential target value ranges during meta-in-context learning. To test this hypothesis, we plotted the proportion of GPT-3's outputs that fell at the extreme values of the target space or outside of it in Figure 4B. We found that meta-in-context learning substantially reduces the number of predictions that are outside of this range, indicating that meta-in-context learning helps to learn a better initial guess.

Meta-in-context learning works better when task similarity is high: However, learning a good initial guess is not the full picture. We speculated that meta-in-context learning is more effective if there is a higher similarity to previously encountered tasks. To verify this, we fi tted a linear regression model with trial and task similarity on the RMSE. We computed task similarity as the average similarity between the current data-point and the data-points from all previously observed tasks (each individual similarity measure was obtained using a radial basis function kernel). We found a signi fi cant effect of trial ( β = -0 . 089 ± 0 . 020 , t = -9 . 204 , p < 0 . 001 ) as shown in Figure 4C, con fi rming that performance within a task improves with additional observations. Furthermore, we found a signi fi cant effect of task similarity ( β = -0 . 147 ± 0 . 020 , t = -15 . 223 , p < 0 . 001 ), suggesting that meta-in-context learning works best if similarity to previously encountered tasks is high. In previous experiments, we showed that meta-in-context learning improved performance. This experiment delves into the underlying reasons by proposing how the relatedness of different meta-in-context tasks affects their observed signal. Therefore, we expanded our analysis by also incorporating task similarity considerations to both Experiments 1 and 2 (see Figure 7, Appendix A.2). The analysis reveals the same observation: the more related the task, the more meta-in-context learning is bene fi cial.

The fi ndings outlined in this section provide further evidence for meta-in-context learning in LLMs. 4 Notably, GPT-3 exhibited superior performance in a real-world multi-dimensional regression task following meta-in-context learning. We identi fi ed two reasons for this. First, meta-in-context learning constrained initial model predictions to the range of plausible values, and second, GPT-3 was able to leverage similarities to previously encountered tasks in order to improve its predictions.

7 Meta-in-context learning on natural language processing benchmarks

Finally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark [29].

7.1 Methods

We focus on the tasks from the STEM supercategory as other supercategories - together with the addition of meta-in-context learning - cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with k ∈ { 0 , 1 , 2 } examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from different categories. Examples of two STEM tasks and some extra analysis on open-source models on the entire MMLU can be found in Appendix A.5.

7.2 Results

Figure 9 in Appendix A.5 summarizes our results. We found that meta-in-context learning was in general bene fi cial in terms of performance. The biggest bene fi t was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of 55 . 1 %outperforming in-context by 22 . 4 %. This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.

8 Discussion

We have demonstrated that LLMs can improve their in-context learning abilities via in-context learning itself, i.e., that they are capable of meta-in-context learning . Meta-in-context learning was not only able to overwrite an LLM's priors but also changed its learning strategies, as demonstrated in two arti fi cial domains. Finally, we applied our approach to two benchmarks. First, a real-world benchmark of regression tasks where we found that meta-in-context learning leads to algorithms that are competitive with standard learning algorithms. Then, we veri fi ed the applicability of our results in an NLP benchmark, providing further evidence of the versatility and effectiveness of our approach across diverse contexts.

Perhaps the most signi fi cant shortcoming of our model simulations is that they all relied on learning tasks with just a handful of observations. This limitation is mainly due to the practical constraint of a fi nite context window coupled with meta-in-context learning's rapid prompt length increase. To ensure that we remain within the allowed context length (and to keep the monetary costs for our simulations at a reasonable level), we had to make this design choice. However, we believe that despite this restriction - our simulations were suf fi cient to illustrate the potential of meta-in-context learning. We hope that future LLM iterations with longer context lengths and lower inference costs will allow us to extend our simulations to larger datasets.

In addition, the tasks we probed were rather simplistic in their nature. That being said, we think we have reasonably covered the space of fundamental learning paradigms, including a supervised problem, a reinforcement learning problem, a collection of real-world datasets and an NLP benchmark. With the increasing availability of multi-modal models, it will furthermore become feasible to apply our approach to other domains. In this context, two obvious candidates are classi fi cation tasks with visual stimuli or grid-based navigation tasks.

A promising direction for further research involves delving into the factors that contribute to the emergence of the meta-in-context learning phenomenon. One potential factor, as mentioned earlier, is the length of the context window. Another avenue to explore is to assess whether specialized models fi ne-tuned for in-context learning, like the framework proposed by Min et al. [30] (Meta-training for In-Context Learning), yield optimal performance in the context of meta-in-context learning.

In summary, our work has both near- and long-term consequences. In the near term, it indicates that it is not strictly necessary to engineer the perfect prompt for an LLM so that it can solve a given learning problem [31]. Instead, LLMs are - to some degree - able to infer the required information from just a handful of related-tasks examples. In the long term, it could point to a paradigm where we adapt these models to the environment they are applied in purely through meta-in-context learning rather than fi netuning them using traditional means.

Acknowledgments and Disclosure of Funding

We would like to thank Zeb Kurth-Nelson for the helpful discussions. Zeynep Akata acknowledges partial funding by the ERC (853489 - DEXIM).

References

Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130 , 2023.

Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and Individual Differences , 103:102274, 2023.

Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. Science in the age of large language models. Nature Reviews Physics , pages 1-4, 2023.

Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, et al. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences , 119(32):e2123433119, 2022.

Taylor Webb, Keith J Holyoak, and Hongjing Lu. Emergent analogical reasoning in large language models. arXiv preprint arXiv:2212.09196 , 2022.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems , 35:18878-18891, 2022.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 , 2022.

Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations , 2021.

Lovre. Who models the models that model models? an exploration of gpt-3's in-context model fi tting ability. https://www.alignmentforum.org/posts/c2RzFadrxkzyRAFXa/ who-models-the-models-that-model-models-an-exploration-of .

Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classi fi cation of tabular data with large language models. In International Conference on Arti fi cial Intelligence and Statistics , pages 5549-5581. PMLR, 2023.

Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023.

Marcel Binz, Ishita Dasgupta, Akshay Jagadish, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-learned models of cognition. arXiv preprint arXiv:2304.06729 , 2023.

Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv:1905.03030 , 2019.

Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Arti fi cial Neural Networks , pages 87-94. Springer, 2001.

Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 , 2016.

Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl 2 : Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779 , 2016.

Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pages 1842-1850. PMLR, 2016.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583-30598, 2022.

Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661 , 2022.

Openai api. https://platform.openai.com . Accessed: 2023-05-10.

Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Gof fi net, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fi ne-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. Accessed: 2023-06-22.

Samuel J Gershman. Deconstructing the human algorithms for exploration. Cognition , 173:3442, 2018.

Marcel Binz and Eric Schulz. Modeling human exploration through resource-rational reinforcement learning. In Advances in Neural Information Processing Systems , 2022.

Jan Malte Lichtenberg and Özgür Simsek. Simple regression models. In IDM@NIPS , 2017.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943 , 2021.

Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150 , 2022.

Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an ability of LLMs, called meta-in-context learning, which showcases that LLMs can recursively improve their in-context learning with demonstrations. The authors illustrate this capability with a regression task and a two-armed bandit task. The analysis demonstrates that LLMs are not only able to learn the task (underlying function) from the examples in this task but also can leverage the examples from other tasks (underlying functions).\n\nSTRENGTHS:\n1. The paper highlights the meta-in-context learning capability of LLMs. The capability allows LLMs to be recursively improved via in-context learning with more tasks in context.\n2, The paper is clearly written and easy to follow.\n\nWEAKNESSES:\n1. The analysis in the paper demonstrates that LLMs can leverage the examples from other tasks, however, the definition of a task appears to be rather limited in this paper. In the linear regression experiment, for example, a task represents the parameters within a linear function. \n\n2. The authors introduce the concept of meta-in-context learning as a novel ability of LLMs, but it remains unclear how this differs from the traditional in-context learning ability. In this paper, each regression function is regarded as a task with a few examples following the function, and LLMs are found to be able to learn from other tasks. However, this ability seems to have been demonstrated in various other applications. For instance, LLMs can learn to generate dialog responses by observing example responses from other dialogs. If we apply the terminology used in this paper, each dialog can also be considered a task, with each individual utterance serving as an example.\n\nWhile I think the analysis in the regression and two-armed bandit tasks is commendable, it would be valuable to see such analysis on more realistic tasks, such as dialog generation (a dialog is a task),  and passage/image question answering (a passage or image is a task).\n\nQUESTIONS:\n1. On line 118, it is mentioned that the GPT-3 is not told that underlying functions are linear, distinct from BLR. However, this raises the question of whether GPT-3 possesses the capability to handle non-linear functions. Additionally, if GPT-3 sees a few tasks with linear functions, is GPT-3 able to generalize to a non-linear function in a new task? The meta-in-context learning ability is more valuable if the tasks can be more different.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The authors demonstrate that the in-context learning abilities of large language models can be recursively improved via in-context learning itself.  

The paper misses the method section. I don't know the details and cannot tell the difference from previous work. 

I didn't get the novel part of the method. It seems an empirical study. 

The authors had better do some experiments on benchmark datasets, such as the datasets from GPT-3 paper.

STRENGTHS:
1. The problem explored is critical.
2. The experimental analyses are interesing.

WEAKNESSES:
1. Missing model section. 
2. The method is more of an empirical study.
3. The experiments are not solid. The authors focus on the GPT-3 model exploration but miss comparison to any benchmark dataset from GPT-3 paper. The shot number, context length, and language understanding in benchmark datasets are all critical issues to study. And GPT-3 is released quite a long time ago. Exploring recent LLMs, such as ChatGPT or GPT-4, would be better.
4. The finding is similar to GPT-3 on domain adaptation, such as the work from "Prompting GPT-3 To Be Reliable".

QUESTIONS:
Can you formulate meta-in-context learning?

LIMITATIONS:
I didn't get the novel contribution of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors undertake a study of \"meta in-context learning\" as a capability of Large Language Models, specifically focused on GPT-3 (with some initial experiments on GPT-4). The authors define meta in-context learning following a task-trial structure, in which the agent observes multiple tasks each consisting of multiple trials. In each trial, a given input-output pair is observed; on the final trial, the model needs to predict the output for an unpaired input. The (unobserved) function generating outputs from inputs differs between tasks. The authors show that GPT-3 is able to learn not only at the trial level (traditional in-context learning) but also across tasks; that is, it can get better both at modelling a specific function and at the general task of modelling functions (in a particular context). The authors further show that this holds across artificial simple regression, reinforcement learning and multiple regression.on real-world data.\n\nSTRENGTHS:\nThe paper is well-written, and the results are thoroughly investigated and clearly conveyed. The authors seek to identify the specific causes of the behaviours they document and the extent to which they can be attributed to simpler processes, e.g. learning the output distribution for predicting on the first task of a trial. The paper includes a discussion of context window size-related limitations to the applicability of the results.\n\nOn originality, I don't know of any other works that address the question of meta-learning without weight updates. As the authors point out, the results could inspire other researchers and applied LLM users to persue multi-task setups rather than finetuning.\n\nWEAKNESSES:\nThe paper is well-reasoned within the specific niche of tasks demonstrated, though the overall implications of the work are arguable. Because the examples are constrained to simple numerical problems, it's difficult to tell whether the capability can be usefully leveraged on other types of language tasks. In the discussion, where it's argued that meta in-context could replace finetuning, it would help to give a few examples of real-world tasks that could be reframed as meta in-context.\n\nWithin the artificial tasks, it would also be nice to see non-numerical demonstrations of meta in-context learning, both to show that it is possible on other types of tasks and because LLMs tend to struggle with numerical transformations. For instance, within-task could involve translating sentences in a simple made-up language into English (with words that are repeated across trials), while each task involves a different made-up language. This could provide a more convincing argument that meta in-context learning is a general phenomenon beyond math tasks.\n\nMore detail on the exploration probit regression model and investigated regression coefficients in the Supplementary would be useful.\n\nQUESTIONS:\nIs it possible to create a clear definition of what constitutes an in-context trial and what constitututes an across-context trial, i.e. where does in-context learning become meta in-context learning? Does an intial task trial always start with some degree of irreduceable uncertainty? \n\nSee the above questions about real-world examples or non-numerical examples of meta in-context learning.\n\nLIMITATIONS:\nNo major negative societal implications are expected. The authors address a number of limitations, including low sample size and simplicity of the tasks, in the Discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores a phenomenon referred to as “meta in-context learning” in large language models, where the in-context learning abilities of these language models can be recursively enhanced through in-context learning itself. To demonstrate this, the researchers examine two idealized domains: a one-dimensional regression task and a two-armed bandit task. They show that meta-in-context learning (1) improves in-context learning performance (2) dynamically shapes the large language model's priors over expected tasks and (3) also modifies its in-context learning strategies.\n\nSTRENGTHS:\n* The high level question raised in this paper is interesting – can in-context learning be improved via the recursive operation of concatenating several different tasks in-context? \n* The experimental setup is clean and the obtained experimental results convey a clear message. They clearly show that for the examined problems meta in-context learning helps, and moreover they reveal that the improvement (also) comes from previous tasks in context shaping the LLM prior for the examined task.\n\nWEAKNESSES:\n* I found the setup and particularly the examined tasks to be too limited. Although the high level question is interesting I feel like much more could be done in order to shed light on it and connect it to actual scenarios in which in-context learning is used in practice. \nThe authors chose to focus on linear regression (synthetic and 60 real world tasks filtered down to 42) / synthetic bandit problems. The abstract and intro describe that in-context learning is a very important feature of LLMs which facilitated their success; while true it is not related to these types of tasks but rather to natural language ones. I think that the interesting high level question raised in this paper could easily have been studied for natural language tasks, and in particular relating it to papers showing multitask benefits which study which tasks are related and reinforce each other such as Sanh et al ICLR 2022, Aribandi et al, ICLR 2022 and others. \nI acknowledge that there is a growing literature of works theoretically analyzing and experimenting with in-context learning on synthetic / mathematical problems, the authors mention some of it. However, in this case I think that the merits of the synthetic / simple / mathematic setup were not fully utilized – for example, the authors could have studied in this simple setting how the relatedness of different tasks in the same context affects their observed signal (eg by defining a formal metric of distance / similarity between linear regression tasks). A small subset of the results in section 3.3 is dedicated to a statement about task similarity and I think it should be expanded for deeper insight. In the lack of such deeper investigations in the synthetic realm, and on the other hand no experimentation on language tasks, this paper misses out on real insight relevant for language related scenarios. For the scenarios studied in the paper, LLMs are not the go-to tool. I will note as an outlier to the above that the study of  in-context learning strategies in the bandit case is deep, novel, and leverages the clean mathematical structure of the synthetic experiment. However, even in this case, I see no conclusions applicable to real world scenarios in which LLM in-context learning is the go-to tool.\n\n* The paper is not segmented into “method”, “experimental setup”, “results” sections, but rather includes one section describing all of the above linearly according to the different tasks that were ran. This is not a problem a-priori, but I found the presentation hard to follow at times, and the authors may want to consider structuring the paper a bit more.\n\n* Minor remarks:\n** There are some some typos (eg, end of parentheses in line 77, named citations without a year)\n** The name “meta in-context learning” is used for a different method presented by Min et al. 2021 in this influential paper: “MetaICL: Learning to Learn In Context”. You may want to rename this paper. \n** I think that the authors overstate when they claim meta in-context learning to be related to psychological experiments.\n\nQUESTIONS:\nWhat are the main practical consequences or implications of your findings for specific real world scenarios in which in-context learning in LLMs is currently used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that large language models (LLMs) are capable of meta-in-context learning: updating their in-context-learning abilities when prompted with examples of several tasks\nThe authors empirically show this capability on several learning paradigms, including supervised learning (1D linear regression), reinforcement learning (2-armed bandits), and several real-world linear regression problems.\nThese experiments show that LLMs, and GPT-3/GPT-4 more particularly, can effectively adapt their in-context learning algorithm while also matching simple baselines like Bayesian linear regression (BLR), upper-confidence bound (UCB), and random forests.\nThey also show that meta-in-context can outperform standard in-context learning, which is especially relevant given the popularity of this technique with LLMs.\n\nSTRENGTHS:\n1. I found the paper an enjoyable read. The research is relatively well motivated, the analysis are carefully detailed, and the method is clearly explained. In fact, I believe I could easily replicate the results from the paper because all experimental testbeds are precisely described and the exact prompts are provided (which also help illustrate the method; see blue panels). While more baselines could be included for completeness, the experiments clearly show the meta-in-context learning effect so I don't think they are necessary.\n2. Despite its simplicity, the presented idea is elegant and I could see it impacting research beyond large language models. For exampe it is not difficult to imagine extensions beyond text inputs (say, vision, audio, or robotics), or theory extensions (in the flavor of \"Transformers learn in-context by gradient descent\"), or even applying a similar strategy beyond in-context learning (eg, for prompt or prefix tuning). As such, I think it'll garner interest from the NeurIPS community.\n\nWEAKNESSES:\n1. Experimental design: given that there is no theory, I find the experimental section a bit light. First, the authors only include results with OpenAI models (GPT-3 and GPT-4) which are not open-source. So it's not clear if meta-in-context learning works as well (or at all?) with publicly available LLMs. Second, can *any* LLM meta-learn in-context? How does the meta-learning ability improve with model/data size? Third, meta-in-context is only tested on small scale and toy datasets -- even the \"real-world\" ones are only require simple linear regression models. It'd be much more compelling if the authors could show that meta-in-context learning also improves upon in-context-learning on standard NLP tasks. NLI-type tasks would be a promising start but I'd find question answering (eg, SQuAD) or even GSM8K much more interesting.\n2. Novelty: The proposed method is similar to \"MetaICL: Learning to Learn In Context\" by Min et al. and published at NAACL in 2022 -- yet, the authors don't even mention this work. As far as I understand, MetaICL pretrains the model such that the LLM weights are trained to adapt quickly to in-context prompts (hence they have learned to learn), while this work demonstrates that meta-learning emerges even with the standard LLM training objective. It'd be insightful to compare the two approach to quantify how much meta-training helps to adapt quickly. Min et al.'s work is especially relevant because it demonstrates benefits on real NLP tasks.\n3. Scholarship: Many references in the bibliography are broken. This might seem a detail but not when a third of the references don't even state a venue.\n\nQUESTIONS:\n- Do open-source models also exhibit meta-in-context learning? Why or why not?\n- How does meta-in-context learning depend on the size of the model / data it is trained on?\n- Can meta-in-context learning outperform in-context learning on standard NLP tasks like NLI, SQuAD, or GSM8K?\n- How is this work different from the Min et al.'s MetaICL paper?\n- Please fix your bibliography.\n\nLIMITATIONS:\nThe limitations of the work are appropriately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}