SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meta-in-context learning reshapes learning strategies: We found a positive main effect of the value difference  $\mathrm{V}_t$  ( $\beta = 0.307 \pm 0.027$ ,  $z = 21.759$ ,  $p < 0.001$ ), indicating that GPT-3 engages in Boltzmann exploration. GPT-3 becomes more greedy during meta-in-context learning as shown by the positive interaction effect between task number and  $\mathrm{V}_t$  ( $\beta = 0.128 \pm 0.025$ ,  $z = 9.633$ ,  $p < 0.001$ ). Furthermore, we found negative main effects for both relative uncertainty  $\mathrm{RU}_t$  ( $\beta = -0.160 \pm 0.010$ ,  $z = -31.788$ ,  $p < 0.001$ ) and the uncertainty-scaled value difference  $\mathrm{V}_t / \mathrm{TU}_t$  ( $\beta = -0.400 \pm 0.025$ ,  $z = -5.393$ ,  $p < 0.001$ ), suggesting that GPT-3 avoids uncertain options by default. However, we also found a slight, but significant, increase in UCB-based decisions during meta-in-context learning ( $\beta = 0.046 \pm 0.010$ ,  $z = 9.279$ ,  $p < 0.001$ ). Figure 3E shows a visualization of all regression coefficients involved in this analysis.
The results presented in this section corroborate those obtained from the supervised setting. GPT-3 generally performed better in a two-armed bandit task after meta-in-context learning. We again observed that GPT-3 accomplishes this by adapting its priors across tasks. In addition, we investigated changes in strategies during meta-in-context learning and found that GPT-3 learned to perform better by exploiting more consistently.
6 Regression on real-world data
Next, we wanted to investigate whether our results obtained in the artificial domains scale up to real-world applications. To test this, we considered a multi-dimensional regression benchmark which consists of 60 different real-world datasets introduced in [28].
6.1 Methods
For each simulation, we randomly selected five different tasks from the benchmark and then sampled five points without replacement for each task. We used five features for all tasks. If a dataset contained less than five features, we omitted it from our analysis, which yielded 42 remaining datasets. For tasks exceeding five features, we used a sub-selection procedure that retained only the top five features based on their F-value with respect to the target variable evaluated in a univariate linear regression. To maintain a consistent regression loss across all tasks, we normalized both the feature and target spaces to the interval of  $[-1, 1]$ . The resulting prompts follow the general template outlined earlier:
Regression on real world data
You observe an input vector  $x$  and have to predict the corresponding output  $y$  as accurately as possible. You are given 5 different tasks.
Task 1:
Task 5:
$$
x = \left[ - 0. 8 1, - 0. 1 6, - 0. 7 8, - 0. 7 7, - 0. 4 5 \right], y = - 0. 3 4;
$$
$$
x = \left[ - 0. 8 1, - 0. 6 3, - 0. 7 5, - 0. 8 3, - 0. 5 5 \right], y = - 0. 6 8;
$$
$$
x = \left[ - 0. 9 7, - 0. 9 2, - 0. 9 7, - 0. 9 7, - 0. 8 2 \right], y = [ \text {i n s e r t} ];
$$
6.1.1 Results
GPT-3 does in-context and meta-in-context learning: Following the previous experiments, we investigated the learning curves of GPT-3 and GPT-3 after meta-in-context learning. We additionally included two baselines in our analysis: BLR and a random forest model. We measured performance by the root-mean-squared error (RMSE). Figure 4A shows the learning curve for all these models. Like in the previous cases, we found that GPT-3 improves with additional data-points, confirming that it is capable of in-context learning in this setting. In addition, meta-in-context learning further improved performance. GPT-3 after meta-in-context learning matched the performance of BLR and was only slightly outperformed by the random forest model.
Figure 4: Meta-in-context learning on the real-world regression benchmark (42 - 50 simulations). Errors bars represent  $95\%$  confidence intervals. A: RMSE across trials for different models. B: Percentage of predictions outside or equal to the extremes of the squashed target range. C: Effects of trial and task similarities for estimating the RMSE.
Meta-in-context learning constrains predictions: Most of the improvement of meta-in-context learning seems to come from zero-shot performance. We hypothesized that this is the case because GPT-3 acquires an understanding of potential target value ranges during meta-in-context learning. To test this hypothesis, we plotted the proportion of GPT-3's outputs that fell at the extreme values of the target space or outside of it in Figure 4B. We found that meta-in-context learning substantially reduces the number of predictions that are outside of this range, indicating that meta-in-context learning helps to learn a better initial guess.
Meta-in-context learning works better when task similarity is high: However, learning a good initial guess is not the full picture. We speculated that meta-in-context learning is more effective if there is a higher similarity to previously encountered tasks. To verify this, we fitted a linear regression model with trial and task similarity on the RMSE. We computed task similarity as the average similarity between the current data-point and the data-points from all previously observed tasks (each individual similarity measure was obtained using a radial basis function kernel). We found a significant effect of trial  $(\beta = -0.089 \pm 0.020, t = -9.204, p < 0.001)$  as shown in Figure 4C, confirming that performance within a task improves with additional observations. Furthermore, we found a significant effect of task similarity  $(\beta = -0.147 \pm 0.020, t = -15.223, p < 0.001)$ , suggesting that meta-in-context learning works best if similarity to previously encountered tasks is high. In previous experiments, we showed that meta-in-context learning improved performance. This experiment delves into the underlying reasons by proposing how the relatedness of different meta-in-context tasks affects their observed signal. Therefore, we expanded our analysis by also incorporating task similarity considerations to both Experiments 1 and 2 (see Figure 7, Appendix A.2). The analysis reveals the same observation: the more related the task, the more meta-in-context learning is beneficial.
The findings outlined in this section provide further evidence for meta-in-context learning in LLMs.4 Notably, GPT-3 exhibited superior performance in a real-world multi-dimensional regression task following meta-in-context learning. We identified two reasons for this. First, meta-in-context learning constrained initial model predictions to the range of plausible values, and second, GPT-3 was able to leverage similarities to previously encountered tasks in order to improve its predictions.
7 Meta-in-context learning on natural language processing benchmarks
Finally, we examined whether meta-in-context learning also improves upon in-context-learning on standard natural language processing tasks. To test this, we conducted an experiment on the Massive Multitask Language Understanding (MMLU) benchmark [29].
Note that while we were running our model simulations, OpenAI released a new model - GPT-4. We repeated the analysis from this section on this new model and described our results in Appendix A.4.
7.1 Methods
We focus on the tasks from the STEM supercategory as other supercategories – together with the addition of meta-in-context learning – cause prompt lengths to exceed the limits of GPT-3. For the in-context learning simulations, we provided the model with  $k \in \{0,1,2\}$  examples from the same category before prompting it on the test question. For the meta-in-context learning simulations, we additionally prepended three examples of two tasks from different categories. Examples of two STEM tasks and some extra analysis on open-source models on the entire MMLU can be found in Appendix A.5.
Figure 9 in Appendix A.5 summarizes our results. We found that meta-in-context learning was in general beneficial in terms of performance. The biggest benefit was observed in the zero-shot case, in which meta-in-context learning reached an accuracy of  $55.1\%$  outperforming in-context by  $22.4\%$ . This illustrates that LLMs do not necessarily have to be prompted by examples from the same category but that they can also transfer some knowledge from different categories.
We have demonstrated that LLMs can improve their in-context learning abilities via in-context learning itself, i.e., that they are capable of meta-in-context learning. Meta-in-context learning was not only able to overwrite an LLM's priors but also changed its learning strategies, as demonstrated in two artificial domains. Finally, we applied our approach to two benchmarks. First, a real-world benchmark of regression tasks where we found that meta-in-context learning leads to algorithms that are competitive with standard learning algorithms. Then, we verified the applicability of our results in an NLP benchmark, providing further evidence of the versatility and effectiveness of our approach across diverse contexts.
Perhaps the most significant shortcoming of our model simulations is that they all relied on learning tasks with just a handful of observations. This limitation is mainly due to the practical constraint of a finite context window coupled with meta-in-context learning's rapid prompt length increase. To ensure that we remain within the allowed context length (and to keep the monetary costs for our simulations at a reasonable level), we had to make this design choice. However, we believe that - despite this restriction - our simulations were sufficient to illustrate the potential of meta-in-context learning. We hope that future LLM iterations with longer context lengths and lower inference costs will allow us to extend our simulations to larger datasets.
In addition, the tasks we probed were rather simplistic in their nature. That being said, we think we have reasonably covered the space of fundamental learning paradigms, including a supervised problem, a reinforcement learning problem, a collection of real-world datasets and an NLP benchmark. With the increasing availability of multi-modal models, it will furthermore become feasible to apply our approach to other domains. In this context, two obvious candidates are classification tasks with visual stimuli or grid-based navigation tasks.
A promising direction for further research involves delving into the factors that contribute to the emergence of the meta-in-context learning phenomenon. One potential factor, as mentioned earlier, is the length of the context window. Another avenue to explore is to assess whether specialized models fine-tuned for in-context learning, like the framework proposed by Min et al. [30] (Meta-training for In-Context Learning), yield optimal performance in the context of meta-in-context learning.
In summary, our work has both near- and long-term consequences. In the near term, it indicates that it is not strictly necessary to engineer the perfect prompt for an LLM so that it can solve a given learning problem [31]. Instead, LLMs are – to some degree – able to infer the required information from just a handful of related-tasks examples. In the long term, it could point to a paradigm where we adapt these models to the environment they are applied in purely through meta-in-context learning rather than finetuning them using traditional means.
We would like to thank Zeb Kurth-Nelson for the helpful discussions. Zeynep Akata acknowledges partial funding by the ERC (853489 - DEXIM).
[18] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850. PMLR, 2016.
[19] Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vlademyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
[20] Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598, 2022.
[21] Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.15661, 2022.
[22] Openai api. https://platform.openai.com. Accessed: 2023-05-10.
[23] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
[24] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[25] MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023. Accessed: 2023-06-22.
[26] Samuel J Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34-42, 2018.
[27] Marcel Binz and Eric Schulz. Modeling human exploration through resource-rational reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[28] Jan Malte Lichtenberg and Ozgür Simsek. Simple regression models. In IDM@NIPS, 2017.
[29] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
[30] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943, 2021.
[31] Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022.
[32] Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.