From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

Authors: robert  vacareanu
vlad-andrei  negru
vasile  suciu
mihai  surdeanu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 22:46:23

Abstract: We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman \#2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting.
We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

Main paper: 
 introduction: 
 Figure 1 : Mean Absolute Error (↓) comparison between three large language models (LLMs) and four traditional supervised methods for learning a linear regression function with one informative variable out of two. Given only in-context examples and without any additional training or gradient updates, pre-trained LLMs such as Claude 3, GPT-4, or DBRX can outperform supervised methods such as Random Forest or Gradient Boosting.Large Language Models (LLMs) are capable of learning to perform a task when given examples of that task in their context, without any additional training. This surprising capability, called in-context learning (ICL) Brown et al. (2020) , emerges just from next-token prediction for sufficiently large models.We use regression tasks to analyze the incontext capabilities of already pre-trained large language models (LLMs), such as Llama2, GPT-4, or Claude 3. Garg et al. (2022) have previously explored the range of functions that transformers, when trained specifically for incontext learning, are capable of learning. However, contemporary LLMs emerge as capable in-context learners without being specifically trained for it. We extend previous work and analyze the extent to which LLMs, decoderonly transformers trained auto-regressively for next-token prediction, are capable of learning regression functions when given in-context exemplars, without any additional form of supervision or training.Similar to previous work (Garg et al., 2022) , we use (synthetic) regression datasets. Synthetic regression datasets have the following advantages:(i) Algorithmically generated: The data is guaranteed to be generated deterministically, by a well-determined (and logical) formula. This property makes them suitable to use when investigating whether a given model is capable of unraveling the underlying structure of the data.(ii) Difficulty control: The user has direct access to the difficulty of the synthetic regression problem and can investigate the cases of simple linear regressions of the form y = ax + b, to more difficult problems such as Friedman #2, a highly non-linear function used for benchmarking:y = (x 2 1 + (x 2 • x 3 - 1 x 2 • x 4)2 )(iii) Data availability: Lastly, synthetic datasets present the advantage of allowing the user to generate novel data in large(r) quantities. Additionally, it ensures that models are less likely to have been previously exposed to these specific problems.Formally, let D n be a dataset consisting of n input-output examples: D n = {(x 1 , y 1 ), . . . , (x n , y n )}, where x i ∈ R d , with 1 ≤ d ≤ 20 typically. We have y i = f (x i ), where f : R d → R and y i ∈ R. We do not put any restrictions on f and study functions ranging from simple linear predictions (i.e., f (x) = ax + b) to more complex and highly non-linear functions (e.g., f (x) = x + 10sin( 5πx 100 ) + 10cos( 6πx 100 )). We study how well various models such as LLMs (e.g., , traditional supervised models (e.g., Random Forest), and unsupervised baselines (e.g., random prediction) are capable of predicting y n+1 when given access to n input-output examples (i.e., D n ) and x n+1 .tOur study shows that pre-trained large language models (LLMs) display a surprisingly good performance on various regression tasks. For example, in Figure 1 , without any parameter update, Claude 3 approaches the performance of a Linear Regression model and largely outperforms other supervised methods such as Random Forest or Gradient Boosting on a randomly generated linear regression dataset, with one informative variable out of two. 
experimental setup: 
 We describe the models and the datasets we use in our experiments below.We experiment with 3 types of datasets: (1) linear regression, (2) non-linear regression, and (3) regression datasets with non-numerical inputs. We describe each below.We experiment with linear regression tasks of the form y = wx + b, where w, x ∈ R d , b ∈ R, and y ∈ R, with 1 ≤ d ≤ 10. We vary both d, the dimension of the input x, and the number of informative variables (i.e., the number of non-zero elements in w).When generating a dataset, we sample the input x from N (0, 1). We sample the weight vector w from Uni f orm(0, 100). 2For non-linear regression problems, we use the three problems introduced by Friedman, called Friedman #1, Friedman #2, and Friedman #3 (Friedman, 1991; Breiman, 1996) . For example, Friedman #1 is defined as follows: y(x) = 10 * sin(x 0 x 1 π) + 20(x 2 -0.5) 2 + 10x 3 + 5x 4 + ϵ * N(0, 1) Where y ∈ R, x ∼ Uni f orm(0, 1) d and ϵ ∈ R. We have 5 ≤ d. When d > 5, the extra dimensions are ignored for the computation of y.While we create these datasets with different random seeds, resulting in different D n , making a particular D n very unlikely to have been seen by the LLMs during training, it is still possible that they have seen different D n originating from the same generator function f . In an attempt to mitigate this risk, we created 5 new non-linear datasets. We describe them in the Appendix C. For example, one of these functions is: y(x) = x + 10sin( 5πx 100 ) + 10cos( 6πx 100 ), where x ∼ Uni f orm(0, 100) (plotted in Figure 5 ).To supplement the non-linear regression datasets and following Garg et al. (2022) , we create datasets using randomly initialized neural networks. We explore the outputs of 2 types of neural networks: (1) a sequence of simple linear layers with ReLU non-linearity in-between, and (2) the output of a randomly initialized transformer encoder block.To further investigate whether the models are able to learn abstract tasks beyond those subsumed by numerical regression (Razeghi et al., 2022) , we design the following tasks. We (randomly) map symbols (i.e., characters) to numbers (e.g., a → 1). We then randomly sample a subset of these symbols in order to keep the context size manageable and to not need a large number of examples. We map the symbols to a numerical value by sampling a weight vector w ∈ R d and doing a dot product between it and the corresponding values of each symbol. We use lowercase ASCII letters as our symbols (i.e., a . . . z). We randomly sample 5 symbols which will serve as our vocabulary. We include the pseudocode in Appendix C.11.We experiment with three types of models: (1) large language models such as GPT-4, (2) supervised models such as Random Forest, and (3) heuristic-based unsupervised models such as random sampling. All models have access to the same train data and are evaluated on the same test partition. They have access to an input dataset D n and are asked to predict the y n+1 corresponding to the x n+1 . The train partition is used for in-context exemplars for LLMs and supervised training for supervised methods. Due to budget constraints and the context size limitations of the LLMs, we round input values to two decimal places. 3 We repeat each experiment with different random seeds.We use a total of 12 large language models (LLMs), both open and private. Specifically, we use Mistral7B, Mixtral8x7B, CodeLlama70B, Llama2 70B, Yi 34B, DBRX (weights available) and ChatGPT, GPT-4 (OpenAI) and Claude 3 Opus, Claude 3 Sonnet (Anthropic) , Gemini Pro (Google), and Mistral Medium (Mistral) (weights not available). The models we use cover a wide range of parameters, from 7B or less (Mistral) to 132B or more (DBRX). 4 Unless otherwise specified, we interact with the models only through prompting and in-context exemplars. We use the same prompt for all models and do not do any prompt tuning. The prompt is of the form Feature 1: <number>\nFeature 2: <number>\nOutput: <number>. Incontext exemplars are separated with two new lines "\n\n". For the test example, the model is asked to predict the number corresponding to the Output variable. We observed that some models tend to provide additional explanations, before outputting the final number. To prevent this behavior, we add an additional text in the beginning, instructing the LLM to only output the number. We give a complete example in Appendix D.1.1. Additionally, we analyze the explanations provided by the models in Appendix L, finding that there is sometimes a discrepancy between the rationale given for their predictions and the actual predicted values. Unless otherwise specified, we use a temperature of 0.We use a total of 10 traditional supervised models, available in most statistical learning packages. We use: Linear Regression (4 versions: (i) no regularization, (ii) Ridge regularization, (iii) Lasso Regularization, and (iv) no regularization and with polynomial features), Multi-Layer Perceptron (6 versions, 3 versions with different widths (Cybenko, 1989; Hornik et al., 1989 ) and 3 versions with different depths (Lu et al., 2017) ), Random Forest, Bagging, Gradient Boosting, AdaBoost, SVM, KNN, Kernel Ridge, and Splines. Similar to the LLM case, we do not tune any hyperparameters and use the defaults available in sklearn. It is important to note that these supervised baselines are very strong: (1) many of them are the results of algorithms specifically designed for regression (e.g., Splines);(2) all perform parameter updates (unlike an LLM with ICL); and (3) the default hyperparameters, as set in widely-used statistical packages, have been refined over time to offer reliable and generally strong performance across a variety of scenarios.Unsupervised Baselines: In order to contextualize the performance of the LLMs and to evaluate their effectiveness relative to basic heuristics, we incorporated the following series of heuristic-based unsupervised baseline:1. Average: Predicts the next value, y n+1 , as the mean of all preceding outcomes: y n+1 = 1 n ∑ n i=1 y i . 2. Last: Uses the most recent tuple (x n , y n ) for prediction, such that y n+1 = y n . 3. Random: Predicts y n+1 by randomly selecting from the set of prior observations {y 1 , . . . , y n }. The final prediction is thus y n+1 = sample([y 1 , . . . , y n ])Additional details on the models are provided in Appendix D. We include results with additional models, such as the latest release of GPT-4 at the time of submission (gpt-4-2024-04-09) or Mixtral Mixture of Experts 8x22B in Appendix G, where we present the average rank obtained by each model. 
large language models can do linear regression: 
 Our first experiment intends to capture how well LLMs can do linear regression when given only in-context examples. To this end, we experiment with 4 different types of regression problems, varying the number of total variables and the number of informative variables. We provide a total of 50 tuples D 50 = {(x, y) i |i = 1 . . . 50} as in-context exemplars and ask the model to generate y 51 , corresponding to x 51 . We repeat the experiments with 100 different random seeds and present the average and 95% confidence intervals.We present two bar plots in Figure 2 , corresponding to two different datasets: (1) a dataset consisting of three variables, with a single informative variable (Regression NI 1/3), and(2) one dataset containing two random variables, where both variables are informative (Regression NI 2/2). For LLMs, we selected Claude 3 Opus (Claude 3), GPT-4, and Gemini Pro, as they are the flagship closed-source models currently available, and Mixtral8x7B (Mixtral), Llama2 70B (Llama 2), Yi 34B (Yi) and DBRX (DBRX) as the flagship open-weights models. Traditional supervised models in our analysis included Linear Regression (LR), Multi-Layer Perceptron (MLP), Random Forests (RF), and Gradient Boosting (GB). Additionally, we include a fifth supervised method, the one resulting in the best performance. 5We would like to remark that this is a very strong baseline, as it highlights the best performance obtainable with hindsight information. For the unsupervised baselines we included (i) Average, and (ii) Random Sampling. We draw the following observations from this experiment:First, LLMs, when given in-context examples of input-output pairs, exhibit a (perhaps surprisingly) good overall performance. When compared with unsupervised baselines, the large language models always outperform them, indicating that the underlying mechanism at play is more sophisticated than such simple heuristics.Second, we remark that LLMs in some cases outperform even supervised methods. For example, for the regression task with one informative variable out of a total of 3 (Regression NI 1/3), Claude 3 ranks 3 out of a total number of 31 models, only (slightly) behind Linear Regression and Linear Regression + Poly. For example, when averaging the mean absolute error across all runs, Claude 3 obtains 0.14, while Linear Regression obtains 0.12. It largely outperforms other supervised methods such as Random Forest or Gradient Boosting, even though it no gradient updates were performed, nor it was specifically designed for linear regression.6 Lastly, we remark that this strong performance is not only specific to the current closedsource flagship models. For example, Mixtral outperforms supervised methods such as Random Forest or Gradient Boosting on the Regression NI 2/2 dataset.Alongside the two bar plots, we include a heatmap in Figure 3 to show how each model ranks across different datasets. We show the datasets vertically and the models horizontally. Overall, these results reveal that large language models, whether closed-source (e.g., Claude 3, GPT-4) or open-weights (e.g., DBRX, Mixtral 8x7B), are capable of performing linear regression tasks using in-context exemplars composed of (x, y) pairs, all without the necessity for gradient updates. While the performance across these models varies, it consistently outperforms that of unsupervised baselines, suggesting that the underlying mechanism at play is more sophisticated than these simple heuristics. Moreover, specific LLMs (e.g., Claude 3, GPT-4) consistently exceed the performance of strong supervised baselines such as Random Forests, Gradient Boosting, or KNN.We present extended results, encompassing a wider array of models and datasets, in Appendix E. 
large language models can do non-linear regression: 
 We extend our previous analysis to non-linear regression problems.We use the 3 synthetic regression benchmarks introduced by Friedman (1991) . Below, we provide the definition of the Friedman #2 dataset, with complete definitions for all datasets available in Appendix C.EQUATIONwhere ϵ represents noise added to the system, modeled as a Gaussian distribution N (0, 1), and the variables x 1 , x 2 , x 3 , and x 4 are drawn from uniform distributions as follows:x 1 ∼ U (0, 100), x 2 ∼ U (40π, 560π), x 3 ∼ U (0, 1), and x 4 ∼ U (1, 11).Our findings for the Friedman #1, #2, and #3 benchmarks are presented in Figure 4 . The selection of methods follows to the same procedure used in Section 3: three leading closedsource LLMs, four leading open-weights LLMs, and five conventional supervised modelsincluding the best performing model-and two unsupervised baselines. We remark that the strong performance of LLMs persists in the non-linear case as well. For example, Claude 3 outperforms all but the Linear Regression with Polynomial Features (LR + Poly) on Friedman #2. In an effort to mitigate the potential familiarity of models with pre-existing datasets encountered during their pretraining phase, we experiment with two new non-linear regression datasets which are unlikely to have been part of the pre-training phase. Our methodology is as follows. Our first novel dataset (called Original #1), plotted in Figure 5 , is created to resemble a line with oscillations:y = x + 10sin( 5πx 100 ) + 10cos( 6πx 100 ) (2)Where x ∼ U (0, 100).For the next dataset (called Original #2), we draw inspiration from the datasets introduced by Friedman, but we modify the domain of x and change the operands (e.g., 2 → 4 ). We provide an example below:y = (x 4 1 + (x 2 • x 3 - 2 √ x 2 • √ x 4 ) 2 ) 3 4(3)It is important to underscore that the primary goal of these novel datasets is not to construct inherently difficult challenges for the LLMs, but rather to minimize the probability of evaluating them on datasets they could have already seen during their training phase. We provide additional details on these datasets in Appendix C, along with additional datasets.For an in-depth analysis of potential data contamination concerns, including additional experiments conducted to address these issues, please refer to Appendix P.We summarize all our results in the form of a heatmap in Figure 6 . For each dataset, we record the relative rank of each method with respect to all the others. For example, Claude 3 Opus performs the best on Original 1 (rank=1). We structure our results in 3 blocks: (1) LLMs (left), (2) Traditional Supervised Methods (middle), and (3) Unsupervised Methods (right). We make the following observations:First, on Original 1 (see Figure 5 ), LLMs largely outperform traditional supervised methods.Remarkably, eight out of the ten highest-ranking methods in this context are LLMs. This strong performance on this dataset is exhibited by both private and open-weights models.For example, DBRX outperforms all traditional supervised methods, despite no gradient update.Second, we remark that the LLMs show a strong performance on all datasets introduced by Friedman (Friedman #1, Friedman #2, Friedman #3) and on all datasets introduced by us (Original #1, Original #2).Overall, our results show that LLMs with ICL are capable of performing non-linear regression. For example, Claude 3 Opus outperforms Gradient Boosting and KNN on all 5 datasets. We present extended results, encompassing a wider array of models and datasets, in Appendix F. We observed that LLMs struggle on the datasets generated with randomly initialized neural networks (e.g., Simple NN #1, Transformer #1), although they remain, generally, better than the unsupervised methods. Due to space constraints, we included in Appendix M an analysis of the performance of LLMs on non-numerical regression datasets. We found that even in this regime, LLMs outperform the unsupervised baselines. Borrowing from the Online Learning community, we empirically analyze how the cumulative regret (i.e., cumulative loss) grows with respect to the time step (number of examples in context) (Orabona, 2019) . Ideally, a good model should, over time, approach the quality of decisions that the best fixed strategy, informed by hindsight, would have made. In other words, the cumulative regret should, ideally, grow sub-linearly over time. To empirically estimate how the regret grows, we fit 3 curves: (1) Linear Fit: a * x + b, (2) Sqrt Fit: a * sqrt(x) + b and (3) Log fit: a * log(x) + b. 7 We then use the R 2 coefficient to determine which curve fit is better. We show two qualitative plots in Figure 7 . We summarize the results in Table 1 , recording the curve fit with the highest R 2 coefficient for each model. Since simply picking the best curve fit according to the R 2 score might tell an incomplete story, we include additional plots in Appendix H, covering multiple models and all seven datasets. We draw the following observations. First, the performance of large language models improves with the number of examples, suggesting the mechanism at play is capable of effectively leveraging more data. Second, we remark that very capable LLMs, such as Claude 3 or GPT-4 can obtain sub-linear regret, meaning that the predictions made by the LLM approach the quality of decisions that the best algorithm would have made, leading to near-optimal performance in the long run.We remark that there are differences between our empirical analysis and online learning. Firstly, while online learning often focuses on establishing theoretical regret bounds, our approach is empirical, we only empirically show that the regret of certain LLMs grow sub-linearly by using curve fitting and R 2 . To address potential concerns of overfitting and enhance the robustness of our empirical findings, we repeated the experiment 3 times and averaged the cumulative regret. Second, our results are only for finite (and relatively small) time steps, diverging from the online learning norm of analyzing behavior as T approaches infinity. To provide further evidence that the results are not an artifact of small T, we performed the following experiment. We used GPT-4 and recorded its performance across multiple training dataset sizes, ranging from 20 to 500. We have observed that the performance of GPT-4 continues to improve as the number of in-context exemplars increases, suggesting that, our results are not an artifact of limited time steps. We include the associated plots in Appendix Q.Following the empirical evidence that LLMs are very capable regressors, despite not being trained for it, we hypothesize that (very capable) LLMs emerge from their training as very good online meta-learners (Finn et al., 2019; Mirchandani et al., 2023) .  
related work: 
 Many previous papers studied in-context learning (Min et al., 2022; Dai et al., 2023; Pan et al., 2023; Kossen et al., 2024) , either from a more theoretical standpoint (von Oswald et al., 2022; Xie et al., 2022; Wies et al., 2023; Ahn et al., 2023; Vladymyrov et al., 2024) or from a more empirical one (Garg et al., 2022; Aky ürek et al., 2023; Zhang et al., 2023) . For a more general discussion of in-context learning, we refer the reader to Dong et al. (2023) . Our work fits into the empirical camp, showing through experiments that large language models pre-trained for next-token prediction on web-scale datasets are capable of doing regression when shown input-output examples in their context, sometimes with performance rivalling that of supervised methods such as Gradient Boosting or Random Forests. Previous studies Garg et al. (2022) ; Bai et al. (2023) ; Guo et al. (2024) ; Bhattamishra et al. (2024) have explicitly demonstrated the capacity of transformers to be trained for in-context learning, enabling them to implement a wide range of algorithms when provided with relevant examples. For example, it has been shown that transformers can be trained to in-context learn linear functions, with performance similar to that of optimal least squares (Garg et al., 2022; Zhang et al., 2023) . Bai et al. (2023) extended this, showing that transformers can implement a broad class of standard machine learning algorithms in context when pre-trained for it. Li et al. (2023) proved generalization bounds for ICL under specific conditions. Different from this line of work, we do not train any model specifically for in-context learning or regression. Instead, we simply use large, powerful models (e.g., Claude 3, GPT-3, Llama2) and empirically show that they are capable of performing linear and non-linear regression when given in-context examples, despite not being specifically trained for it. 
conclusion: 
 In this paper, we examined the extent to which large language models such as Claude 3, GPT-4, or DBRX are capable of performing the task of regression, when given input-output pairs as in-context examples, without any gradient updates.We showed that large language models are capable of doing both linear and non-linear regression, with performance rivaling that of supervised methods such as Linear Regression or Gradient Boosting. We then analyzed how their performance approaches that of the best possible fixed strategy as the number of in-context examples grows, showing how very capable models such as Claude 3 Opus or GPT-4 are capable of approaching the quality of decisions that the best algorithm in hindsight would have made. Our results demonstrate that large language models are capable of doing regression when given incontext examples of (input, output) pairs, despite not being explicitly trained to do so. We leave the exploration of augmenting LLMs' training with synthetic regression and math datasets, during either pre-training or fine-tuning, to future work. We release our code and results at https://github.com/robertvacareanu/llm4regression. 
ethics statement: 
 In this work we explored the extent to which large language models (LLMs) are able to perform regression tasks. We did not perform any additional training. We do not envision any negative impact of our results. 
limitations: 
 This study focuses primarily on regression tasks, including an exploration into regressionlike scenarios where inputs are symbolic rather than numeric, yet the outputs remain numeric.A second limitation is the reliance on several large language models, including proprietary ones whose performance may change over time, potentially affecting reproducibility. To address this, we also included leading open-weight models in our analysis, though we note that their performance is generally behind that of private models. Additionally, we release our intermediate results.Third, the issue of data contamination poses a challenge, given the opaque nature of training datasets for many LLMs. We have taken several steps to mitigate this risk: (i) Our analysis spans multiple LLMs, reducing the likelihood of all models being contaminated in the same way; (ii) We evaluated models with multiple random seeds on newly introduced datasets (alongside known ones like Friedman #1). In this way, we diminish the chance that these models have been directly exposed to the exact datasets during training; (iii) We included results with Falcon 40B, whose training data is publicly available (please see Appendix P for more details). We acknowledge that these measures do not eliminate the potential for data contamination entirely.Fourth, while we showed empirical evidence that large language models are capable to perform regression tasks, we did not provide theoretical explanations to support these observations. Min et al. (2022) shows that GPT-3 retains a strong performance even when the labels of the in-context exemplars are random. On the other hand, recent work (Hendel et al., 2023; Liu et al., 2023) Although not the main focus of their work, they also experiment with pre-trained models such as Llama 2 and GPT-4, showing that they obtain a performance similar to nearest-neighbor baselines for boolean functions. A post on the AI Alignment Forum (Pesut) also explored the types of models that GPT-3 can fit in-context. While their focus was primarily on classification, they did examine regression scenarios as well.While in this work we focus on in-context learning, there are other previous works that designed specific neural network modules (or cells) for arithmetic computations (Trask et al., 2018; Heim et al., 2020; Mistry et al., 2022) .Different from previous work, we investigate how pre-trained models, such as GPT-4 or Claude 3, without any gradient updates, can learn various linear and non-linear function classes when given examples in-context and thoroughly compare them against multiple traditional supervised methods (Ruppert, 2004 ) such as Gradient Boosting (Schapire, 1989; Friedman, 2001) or Random Forests (Breiman, 2001) . 
 
 Appendix: We provide the formulas for all datasets used below. We set the noise to 0 for all datasets.In order to generate the linear regression datasets, we use the function make regression, available in sklearn (Pedregosa et al., 2011) .C.2 Friedman # 1 f (x) = 10 • sin(x 0 x 1 π) + 20(x 2 -0.5) 2 + 10x 3 + 5x 4 + ϵ • N (0, 1) Where x 0 , x 1 , x 2 , x 3 , x 4 ∼ U(0, 1) C.3 Friedman # 2 f (x) = x 2 1 + x 2 • x 3 - 1 x 2 • x 4 2 + ϵ • N (0, 1)Wherex 1 ∼ U (0, 100) x 2 ∼ U (40π, 560π) x 3 ∼ U (0, 1) x 4 ∼ U (1, 11) C.4 Friedman # f (x) = arctan x 1 x 2 -1 x 1 x 3 x 0 + ϵ • N (0, 1).WhereEQUATIONWhereEQUATIONC.6 Original # y = (x 4 1 + (x 2 • x 3 - 2 √ x 2 • √ x 4 ) 2 ) 3 4Wherex 0 ∼ U (0, 3) x 1 ∼ U (4π, 56π) x 2 ∼ U (0, 2) x 3 ∼ U (1, 11) C.7 Original # y = e x 0 + x 1 • x 2 √ x 3 + (x 0 • x 3 ) C.8 Original # 4 y = x 1 10 • sin(x 0 ) + x 0 10 • cos(x 1 ) + √ x 0 log(x 1 ) √ x 1 log(x 0 ) Where x 0 , x 1 ∼ U (2, 100)C.9 Original # 5EQUATION))Wherex 0 , x 1 , x 2 ∼ U (-25, 25)For the random datasets induced by neural networks, we randomly initialize a neural network and create a dataset by feeding random data to it. The dataset Simple Random NN 1 was created by using a neural network with one hidden layer with ReLU non-linearities. The dataset Transformer 1 was created by a randomly initialized Transformer encoder block.We provide the code to generate the non-numerical regression datasets in the Listing 1.Essentially, we assign a random number (0 to 26) to each lowercase letter. Then we sample a weight vector. The expected output is generated by doing a dot product between the underlying assigned value of each character and the generated weight vector.Listing 1: The python code to generate non-numerical regression datasets import random import numpy as np import s t r i n g max num vars = 5 n samples = 51 def g e t c h a r a c t e r r e g r e s s i o n ( random state = 1 ) : r = random . Random ( random state ) a l p h a b e t = l i s t ( s t r i n g . We also experimented with several real-world datasets: (i) Liver Disorders (UCI, 1990), (ii) Real Estate Valuation (Yeh, 2018) , (iii) Diabetes (Efron et al., 2004) , (iv) Servo (Ulrich, 1993) , (v) Movies (Ahmed, 2017) . The corresponding results are presented in Section I.In the following, we provide additional details of the models we used for our main experiments and how we used them. We used three different types of models, as follows:(a) Large Language Models, (b) Traditional Supervised Methods, and (c) Heuristic-Based Unsupervised Methods. We describe them bellow.This section outlines the 12 Large Language Models (LLMs) featured in our main experiments, which include a mix of open-weights and private models. We also include additional models, such as the newest GPT-4 version (gpt-4-20240409), multiple Claude variants, and the most powerful model released by Cohere, Cohere Command R Plus. We tried with Cohere Command R and Cohere Command and observed their performance to be lower, albeit still the unsupervised baselines (except for Cohere Command).In Table 2 , we categorize the models by their names, availability of weights, and developers, dividing them into two distinct sections. The first section lists the models featured in the main paper's experiments (referenced in Sections 3, 4, and 5). The second section introduces additional models that were utilized for the extended analysis included in the Appendix.Developer GPT-4 (Achiam et al., 2023) Not available OpenAI Chat GPT (Ouyang et al., 2022) Not available OpenAI Claude 3 Opus (Anthropic) Not available Anthropic Claude 3 Sonnet (Anthropic) Not available Anthropic Gemini Pro (Team et al., 2023) Not 2 : Details of the large language models (LLMs) used in our study, divided into two sections. The first section contains the LLMs used for the experiments from the main body of the paper, while the second section includes additional models explored in the extended results presented in the Appendix.We list in Table 3 the models we used through OpenAI, together with their corresponding model code.8 GPT-4 gpt-4-0125-preview Chat GPT gpt-3.5-turbo-1106GPT-4 (20240409) gpt-4-turbo-2024-04-09 GPT-3 Davinci davinci-002 GPT-3 Babbage babbage-002 Figure 9 : The prompt we use for all LLMs. Concretely, we use an initial instruction to prevent the models from explaining their prediciton, something which we observed to happen for some models (e.g., Claude 3 Opus). Then, we give each input-output pair and finally ask the model to predict the value corresponding to the test input.For the experiments performed with 100 random seeds and 50 random input-output tuples (i.e., D 50 ), we simply skip the invalid generations. This is not problematic because it rarely occurs. For example, it has never occurred for the any of the experiments showed in Section 3 and Section 4. Moreover, given that we run each experiment with 100 random seeds, skipping the invalid generations will still leave us with enough samples to estimate the statistics. For example, out of the results presented in Section 3 and 4, the largest number of invalid generations for a given dataset was 11, by Mistral Medium on Regression NI 1/2. Claude 3 Opus produced only 2 invalid generations, for Friedman #2, and GPT-4 did not produce any invalid generations. One exception is for the results shown in Appendix J, as we observed that Claude 2.0 and Claude 2.1 produced invalid generations very often. For example, Claude 2.1 generated invalid generations for the dataset Regression 1/1 in 71% of the cases. The reason for invalid generations was usually because Claude 2.1 refused to give "potentially misleading numerical estimates without proper context". A second exception is for Striped Hyena Nous 7B (Appendix R). We add a cross (×) to the cell in the rank heatmap for every model-dataset pair where the number of valid generations is under 20.For the experiments where we investigated how the performance of the models scale with the number of examples, we average 3 random runs for each dataset size. In this set of experiments, only Llama 70B generated invalid outputs a total of 3 times, for Original #2 and Friedman #2. We skip the random runs with invalid generations.We use a total of 11 traditional supervised methods, resulting in over 20 different configurations. Specifically, we used the following models:1. Linear Regression: We used 4 variants of Linear Regression: (i) standard linear regression (Linear Regression), (ii) ridge (Ridge), (iii) lasso (Lasso), and (iv) Linear We used the sklearn implementation for each model.12 Similar to the LLM case, we do not tune any hyperparameters. We use the default hyperparameters available in sklearn. We remark that these supervised baselines are very strong, as (1) many of them are the results of algorithms specifically designed for regression (e.g., Spline), (2) all perform parameter updates, and (3) the default hyperparameters, as set in widely-used statistical packages, have been refined over time to offer a reliable and generally strong performance across a variety of scenarios.We use three heuristic inspired unsupervised models:1. Average: Predicts the next value, y n+1 , as the mean of all preceding outcomes: y n+1 = 1 n ∑ n i=1 y i 2. Last: Uses the most recent observation tuple (x n , y n ) for prediction, such that y n+1 = y n 3. Random: Predicts y n+1 by randomly selecting from the set of prior observations {y 1 , . . . , y n }. The final prediction is thus y n+1 = sample([y 1 , . . . , y n ])The goal of these unsupervised models is to better put the performance obtained by LLMs into perspective.We expand the barplots shown in Figure 2 with more models and more datasets. In particular, we show in Figures 10, 11, 12, 13, 14, 15 the performance of the models on six datasets for linear regression. Specifically, we used the following datasets:1. Regression 1/1, a linear regression with only 1 variable, which is informative 2. Regression 1/2, a linear regression with 2 variables, and only 1 informative variable 3. Regression 1/3, a linear regression with 3 variables, and only 1 informative variable 4. Regression 2/2, a linear regression with 2 variables, both which are informative 5. Regression 2/3, a linear regression with 3 variables, and only 2 informative variables 6. Regression 3/3, a linear regression with 3 variables, all which are informativeWe included the corresponding rank heatmap in Figure 16 .We make the following observations. First, Claude 3 Opus performs among the best for the linear regression case where there is only one informative variable (Regression 1/1, Regression 1/2, Regression 1/3), ranking among top 3 best performing models. The performance drops when there are more informative variables.Second, we remark that all large language models perform better than all the unsupervised methods on all datasets.Third, we remark that the large language models display a good overall performance. For example, there are 4 LLMs (i.e., Claude 3 Opus, Claude 3 Sonnet, GPT-4, DBRX) which perform better than all 3 variants of KNN over all the datasets used.Fourth, there are specific LLMs which always perform better than Gradient Boosting, such as Claude 3 Opus and GPT-4. We remark that DBRX and Code Llama 70B outperform Gradient Boosting in 4 out of 6 datasets. 13 13 In order to see how the performance of LLMs scale with more variables, we additionally experimented with Claude 3 Opus on Regression NI 1/10, 3/10, and 5/10 respectively. For example, in NI 5/10, Claude 3 Opus outperforms methods like Gradient Boosting or Random Forest but lags behind methods like Logistic Regression, which is expected given the nature of the datasets. For NI 3/10, Claude 3 Opus is outperformed by Random Forest and Gradient Boosting. Overall, we observed the performance to be similar to the performance on NI 2/3 from Figure 16 . We expand the barplots shown in Figure 4 with more models and more datasets. In particular, we show in Figures 17, 18, 19, 20, 21, 22, 23, and 24 the performance of the models on the eight datasets for non-linear regression.Additionally, we include in Figures 25 and 26 the performance of the models on datasets generated by randomly initialized neural networks, similar to the methodology of Garg et al. (2022) . We can see that the performance of the LLMs decreases on the datasets created using the neural networks, although it (generally) remains above that of the unsupervised baselines.Similar to Figure 6 , we include the corresponding rankings in Figure 27 .We would like to remark that Claude 3 Opus obtains an average rank of 7.7, the best out of all models we have investigated. The next best is Gradient Boosting, with 8.1. The next best LLM is Claude 3 Sonnet, with 9.3, then GPT-4 with 12.7. To provide a comprehensive overview of model performance across a diverse array of datasets, this section aggregates the average ranks obtained by each model. In this section we show the average ranks obtained by each model across: (1) all linear regression datasets (Linear), (2) all original benchmarking datasets introduced by us (Original), (3) all benchmarking datasets introduced by Friedman (Friedman), (4) all neural network induced datasets (NN), (5) all non-linear datasets (Non-Linear), (6) all datasets (Overall).We show our results in Table 7 . We divide the table into three blocks, separated by horizontal lines, corresponding to the results for (1) LLMs (e.g., GPT-4), (2) Traditional Supervised Methods (e.g., Gradient Boosting), and (3) Unsupervised Baselines (e.g., Average). We remark that LLMs obtain, overall, a strong performance. For example, Claude 3 Opus ranks second overall, outperforming methods such as Gradient Boosting, KNN, or multi-layer perceptrons. Overall, this strong performance is present in both private and open models. For example, DBRX and Mixtral 8x22B achieve average ranks that surpass conventional methods including AdaBoost, KNN, or Random Forests. Table 7 : We show the average rank of each model we investigated, across multiple types of datasets. We divide the table into three blocks, separated by horizontal lines, corresponding to the results for (1) large language models (e.g., GPT-4), (2) Traditional Supervised Methods (e.g., Gradient Boosting), and (3) Unsupervised Baselines (e.g., Average). Overall, the large language models (LLMs) obtain a strong performance. For example, Claude 3 Opus ranks second overall, outperforming very strong methods like Gradient Boosting or multi-layer perceptron. This strong performance is present in both private (e.g., Claude 3 Opus, GPT-4) and open-weights models (e.g., DBRX, Mixtral 8x22B).We expand Table 1 to include more models. We show the corresponding results in Table 8 .Additionally, we include curve fit plots for models. To keep the number of plots to a manageable amount, we selected a subset of the models as follows. We selected Claude 3 Opus and GPT-4, as they are the flagship closed-source models. We selected Yi 34B Chat for the open-weights model. Lastly, we selected Gradient Boosting, Linear Regression, and Linear Regression + Poly. We present the corresponding plots in Figure 28 and 29. Since in these experiments we vary the number of in-context examples starting from 1, we could not include variants of KNN that uses more than one neigbhors for their prediction. We included KNN v4 which uses only one neighbor. Additionally, we included KNN v5, where we use a small number of neighbors when the amount of data is small, then gradually increase it. We can see that their performance is much worse than that of the LLMs, suggesting that the LLMs are doing something more than what KNN do. In the following, we include the performance of GPT-4 and Claude 3 Opus on real-world datasets. Nevertheless, our aim in this work has not been to suggest that LLMs should replace (at least currently) traditional regression methods. 14 We evaluate on the following datasets:1. Liver Disorders (UCI id=60 (UCI, 1990)2. Real Estate Valuation (UCI id=477 (Yeh, 2018)) 3. Diabetes (sklearn (Efron et al., 2004 ))15 4. Servo (UCI id=87 (Ulrich, 1993) )5. Movies (UCI id=424 (Ahmed, 2017) )We evaluated the following models:• LLMs: GPT-4, Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku;• Traditional Supervised Methods: AdaBoost, Bagging, Gradient Boosting, KNN, KNN v2, KNN v3, Kernel Ridge, Lasso, Linear Regression, Linear Regression + Poly, MLP Deep 1, MLP Deep 2, MLP Deep 3, MLP Wide 1, MLP Wide 2, MLP Wide 3, Random Forest, Ridge, SVM, SVM + Scaler, Spline;• Unsupervised Methods: Average, Last, Random.We show our results in Tables 9, 10, 11, 12, and 13. In order to keep the tables easy to read, we selected: (i) two LLMs, Claude 3 Opus and GPT-4; (ii) three traditional supervised methods, Gradient Boosting, Random Forest, and Linear Regression with Polynomial Features (LR + Poly); and (iii) three unsupervised methods, Average, Last, and Random. We show both the mean absolute error (MAE) and the rank. The rank is shown against all the methods ran. For example, on Liver Disorders (Table 9 ), SVM ranks first. 2.1, and Claude 3 Haiku. We include a rank heatmap for all the models from the Claude family currently available in Figure 30 . For comparison, we also included the corresponding performance of two strong models with open-weights: DBRX and Mixtral 8x7B. Claude 2.0 and Claude 2.1 were sometimes generating invalid outputs (e.g., "I apologize, upon reflection I do not feel comfortable providing output values without context. Could we have a constructive discussion about the meaning and implications of this exercise?"). Therefore, we omit those problematic configurations. For all the other cases, the average performance is the result of at least 20 runs. 16 We note that the performance of Claude 3 models is much better than that of older models. We estimate the total cost for all our experiments to under $1200. We spent approximately $300 on OpenRouter. We spent approximately $600 on OpenAI. The cost for OpenAI is higher because we used it in our preliminary experiments. Additionally, the preliminary experiments used gpt-4, which is more expensive than gpt-4-0125-preview. We switched to gpt-4-0125-preview after we added the additional text to the prompt, instructing the model to only output their best estimate. 17Without adding the prefix instruction text (see prompt in Appendix D.1.1), which instructed the models to output only its best estimate, some LLMs (e.g., Claude 3 Opus) started to provide explanations, in an attempt to justify their prediction. 18 Analyzing these "explanations" revealed that there is a discrepancy between their explanation and their prediction. For example, for a non-linear regression problem, Claude 3 Opus suggests to train a Linear Regression model and gives the code to do so. Then it elaborates on how to use it for inference and gives the output. However, manually running the code suggested by the model results in a very different output. We provide some examples in Figures 31, 32, , 33, and 34 . We describe each one below. 16 These invalid outputs are specific to Claude 2.0 and Claude 2.1. For example, for Claude 3 Opus, there exist only 2 instances where it does not generate a valid output, out of a total of over 1000 runs. 17 We did not need this additional instruction in our initial experiments. We added it when we expanded the number of LLMs used, as some of them (e.g., Claude 3) would provide justifications before giving the final output. All models use the same prompt. 18 We hypothesize that this is because their system prompt instructs them to provide explanations.To find the output for a given Feature 0 value, we need to determine the relationship between Feature 0 and the Output. Looking at the provided data points, it appears that the Output is approximately equal to Feature 0 multiplied by a constant value.Let's calculate this constant value using a few data points:For Feature 0 = -2. Our investigation has centered on conventional regression tasks characterized by inputs and outputs represented as numerical values. However, the performance on these tasks might be influenced by the quality of numerical token embeddings, which can serve as a confounding factor Razeghi et al. (2022) . To address this and broaden our analysis, we shift our focus to datasets generated following the methodology outlined in Section 2.1.3. This allows us to evaluate the models' capabilities in contexts where inputs are symbolic rather than numerical.Specifically, we define an input vocabulary V, consisting of a small number of (random) symbols. 19 Each symbol is randomly assigned a number {1, . . . , 26}. We map the symbols to a numerical value by sampling a weight vector w ∈ R d and doing a dot product between it and the corresponding values of each symbol. We present our results in Figure 8 . The large language model display, once again, a performance superior to that of unsupervised models.Due to computational budgets and context limits,20 we rounded both the input and the output to two decimals. To validate that our conclusions are not an artifact of the rounding mechanism, we re-ran GPT-4 on Friedman #2 and Friedman #3, rounding to five decimals. We selected GPT-4 because it obtained overall strong results and offers a good latency. We selected Friedman #2 and Friedman #3 because LLMs obtained generally good performance. We also ran the traditional supervised methods. We include the updated results in Figure 35 .Comparing it with Figure 27 , we can see that the performance of GPT-4 remains strong. This time it even outperforms Gradient Boosting on Friedman #3 and ranks first. The performance on Friedman #3 is only under Linear Regression + Poly, similar to Figure 27 . All in all, the strong performance we observed is unlikely to be just an artifact of rounding.Figure 35 : The ranks of the models when rounding to five decimals. The performance of GPT-4 remains strong, ranking above many traditional supervised methods. This is in line with our previous observations, where we rounded to two decimals. (best viewed in color)We can see from Appendix E and F that the performance of certain LLMs is almost always better than that of all three variants of KNN presented. For example, in the case of linear regression, Claude 3 Opus, Claude 3 Sonet and GPT-4 always perform better than the three variants of KNN we used. Similar, in the case of non-linear regression, only for Simple NN 1 and Transformer 1 is any of the KNN variants outperforming Claude 3 Opus. Moreover, from Table 8 , which records the best curve fit over the cumulative regret, we can see that both variants of KNN perform worse than Claude 3 Opus or GPT-4. We summarize our results in Figure 36 . To keep the plot comprehensible, we chose the best-performing KNN configuration for each dataset. However, the ranking considers the performance of all models. We draw the following conclusions.First, we remark that except on Friedman #1, for every other dataset, the top 9 best performing models are all LLMs. In other words, both closed-source models (e.g., Claude 3 Opus, GPT-4) and open-weights models (e.g., DBRX, Mixtral) outperform all KNN models on all the datasets except Friedman #1. This suggests that the mechanism implemented by in-context learning might be something more complex than KNN.Last, we remark that for Friedman #1, only Claude 3 Opus and Claude 3 Sonnet outperform the KNN variants. Moreover, Claude 3 Opus and Claude 3 Sonnet outperform all KNN variants we experimented with on all datasets.The very large datasets that are used to train contemporary large language models (LLMs) raise concerns about potential contamination (Sainz et al., 2023; Golchin & Surdeanu, 2024) .In our study, we have attempted to mitigate this as follows. First, we used many different random seeds. However, this does not nullify the risk that the LLM has seen similar data (e.g., data from Friedman #1, but with other random seeds). To mitigate this, we explored the performance of the models on regression functions of our own creation. This makes it unlikely that the model has seen data coming from the exact same function. For example, To further analyze the data contamination issue, we perform two additional experiments. We provide results with Falcon, an LLM whose training data is publicly available. Second, we perform an experiment similar to the approach proposed in Golchin & Surdeanu (2024) , where we compare the performance of LLMs with and without knowing the dataset where the data comes from.In this section we expand our analysis to include Falcon 40B and Falcon 40B Instruct, comparing their performance with both traditional statistical methods and other LLMs. To keep the figures comprehensible, we added only the following LLMs: Claude 3 Opus, Chat GPT, Mixtral 8x7B, and Mistral 7B.We remark that the Falcon LLM team has released their training data, 21 41 .We make the following observations. First, Falcon 40B outperforms our unsupervised baselines. Second, Falcon 40B outperforms Gradient Boosting and Random Forests on Regression NI 1/1.Overall, Falcon 40B displays, as well, the capability of doing regression when given incontext examples, albeit to a smaller degree than when compared to more powerful (and newer) models. To further investigate potential data contamination, we conducted an experiment inspired by the methodology described in Golchin & Surdeanu (2024) . This involves assessing model performance under two conditions: with and without explicit knowledge of the dataset being evaluated. Specifically, we modify the prompt shown in Figure 9 to mention the name of the dataset (e.g., Friedman #1, Friedman #2, Friedman #3), as detailed in Figure 42 . This approach allows us to discern the impact of dataset awareness on the model's predictive accuracy, providing insights into the extent of potential contamination in the training data.We present the comparative results in Table 14 , which shows the average absolute error under conditions of dataset awareness versus unawareness. Notably, the mean absolute errors (↓) remain closely matched across all scenarios. To statistically substantiate these observations, we performed paired t-tests for each dataset comparison. Given the multiplicity of tests performed, it became imperative to apply an adjustment for multiple comparisons to our p-values (Dunn, 1961; Benjamini & Hochberg, 1995) . Following this adjustment, none of the p-values remained below the (typically used) 0.05 threshold, suggesting that the knowledge of the dataset name does not significantly affect model performance. Prior to adjustment, in two cases the resulting p-value was under 0.05: GPT-4 on Friedman #3 (p-value 0.045) and Claude 3 Sonnet on Friedman #3 (p-value 0.018). Note, however, that only in the case of GPT-4 was the performance corresponding to the Dataset Aware setting better. This analysis indicates that, within the bounds of statistical significance, there is no substantial evidence to suggest that the performance of the models is influenced by explicit knowledge of the dataset name, something which has been linked to contamination (Golchin & Surdeanu, 2024) . 23The task is to provide your best estimate for "Output" (Friedman #1). Please provide that and only that, without any additional text.Feature 0: . . .Figure 42 : The prompt we use to further investigate whether the large language models have seen instances of the datasets we tested them on.Friedman In the following, we investigate whether the performance of LLMs continues to improve as the number of in-context exemplars increases. Because this experiment is dependent on the context size of the LLM, this imposes additional constraints on which LLMs we can use. To this end, we experimented with the following dataset sizes: {20, 50, 60, 70, 80, 90, 100, 150, 200, 250, 300, 400, 500}. We experiment with ChatGPT and GPT-4. For GPT-4, we found that the performance keeps improving with the number of in-context exemplars at least until 500. We present our results in Figures 43, 44, 45, 46, 47. We repeated each experiment 20 times for each dataset size. We report the mean and 95% confidence.Aggregating the results, we observed that GPT-4 performed better than Random Forest in 92% of the cases, better than Gradient Boosting in 51% of the cases, and better than Linear Regression with Polynomial Features (Linear Regression + Poly) in 40% of the cases, across all 5 datasets and all dataset sizes.For example, from Figure 44 we can observe that while Linear Regression + Poly performs much better than GPT-4 in small data regimes, this performance gap decreases as the number of examples increases, suggesting that the model is indeed capable of leveraging a larger number of examples. In our study, we initially focused on transformer-based large language models (LLMs).To broaden our scope, we explore the capabilities of non-transformer LLMs, including a RWKV-based 14B LLM (Peng et al., 2023) and with StripedHyena (Poli et al., 2023a; b) , a 7B LLM. The performance rankings for these models, along with the transformer-based Mistral 7B for comparison, are illustrated in the heatmap provided in Figure 48 . We tried running Falcon 7B as well, but it produced invalid outputs for almost all examples and all datasets, therefore we skip it. StripedHyena also encountered difficulties, producing invalid outputs in certain scenarios, such as 98% invalid responses for the Friedman #2 dataset. Consequently, we omitted Friedman #2 and Original #2 from its evaluation. However, it is important to highlight that the other models evaluated did not exhibit these issues and were able to generate valid outputs consistently. We make the following observations.First, we remark that performance-wise, RWKV is worse than traditional transformer-based LLMs, although it generally remains better than our unsupervised baselines, with the exception on two linear regression datasets: Regression NI 1/3 and Regression NI 2/3. Nevertheless, we remark that on Original #1, RWKV outperforms many MLP variants, despite no gradient updates.Second, we remark that the performance of Striped Hyena 7B is generally lower than than some of our unsupervised baselines. We note that there is a notable exception for Original #1. For this dataset, KNN approaches work well, as evident by the good performance obtained by nearest neighbor approaches. We used sklearn. Please see make regression for more details.We provide extra experiments without rounding in Appendix N to show that the strong results we observed are not an artifact of rounding.Since the number of parameters for some models is not disclosed, it is possible that certain closed models may have fewer than 7B or more than 132B parameters.If this method coincides with one previously selected, the subsequent best performer is chosen.Comparatively, Random Forest obtains 5.32, Gradient Boosting obtains 2.58, and GPT-4 obtains 2.26.The choice of linear, square root, and logarithmic fits is motivated by their common appearance in theoretical regret bounds within the online learning literature.2Wherex 0 ∼ U (1, 3) x 1 ∼ U (1, 10) x 2 ∼ U (0, 10) x 3 ∼ U (1, 20)https://openai.com/https://openrouter.aihttps://deepinfra.comhttps://fireworks.ai/We used sklearn 1.4.1.post1.Although we can foresee some interesting advantages for LLMs, such as allowing for natural language description of what each feature represents.https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load diabetes. htmlWe used 5 symbols, in the form of letters (e.g., a)For example, Llama2 context size is only 4096.https://huggingface.co/datasets/tiiuae/falcon-refinedwebThe context size of Falcon is 2048.We investigated why the performance of Claude 3 Sonnet degraded on Friedman #2 when given the dataset name. We found that it is (mostly) because of an outlier: the absolute difference between the model's prediction and expected output is > 300. 


Decision: This paper demonstrates several LLMs can be effective at performing regressions in-context, in some cases outperforming supervised methods.

The reviewers noted several strengths of the paper:
- The paper makes an interesting point about the emergent capabilities of LLMs.
- The paper provides an important expansion on prior work by studying pretrained LLMs rather than ones trained specifically to be able to do ICL.
- The paper is clearly written.
- The experimental setup is strong, with clear baselines, a comprehensive set of experiments, and enough information provided to make for strong reproducibility

Several limitations also came up:
- The paper does not provide a theoretical understanding of why LLMs can perform regression tasks.
- LLMs using ICL may still not be effective in real-world regression tasks or as replacements for supervised methods
- There are unexplored directions about several details of the experiments such as the number of in-context examples and the numbere of dimensions.

However, I found the following comment from the author response to be compelling in response to the first and third points:  “We acknowledge that there are open questions regarding the limits of LLMs in regression tasks, such as whether their strong performance is restricted to specific conditions like a small number of examples or dimensions. However, these questions should not detract from the merits of our findings. Instead, they highlight potential avenues for future research that our work enables.” And regarding the second point, I was convinced by the authors’ response that these directions are not the goal of the work (the goal is to understand LLMs, not to propose a new approach for doing regressions).

Overall, I am recommending acceptance due to the interesting results that make important points about ICL in LLMs, and due to the fact that I found the response from the authors to be generally compelling in addressing reviewer concerns. The response from the authors also added some new results with real-world datasets that will enhance the paper.

I thank the reviewers and authors for a very productive discussion period.

Reviews: review: paper_summary: paper_summary: The paper looks at the ability of 11 pre-trained LLMs like GPT-4, Claude 3, and others to perform both linear and non-linear regression tasks using in-context learning (ICL). The results show that some LLMs can perform on par or outperform traditional supervised methods such as Random Forest, Gradient Boosting, and AdaBoost on challenging synthetic regression datasets - without additional training or parameter updates. Additionally, the paper explores how the performance of LLMs scales with the number of in-context examples, empirically showing that some LLMs obtain sub-linear regret.
strength_weakness: strength_weakness: * Because of the nature of the experiments on synthetic data to avoid encountering data that may have been in the training set, there are no experiments with benchmark/ real world regression problems.
* In the last study on regret, it is not clear how regret correlates to MAE, and why the regret values or so different between the different models evaluated. 
* It does not consider the computational requirements for scaling these LLMs to even larger datasets or higher-dimensional regression tasks, leaving a question of practical feasibility for real-world applications of larger problems.
paper_summary: paper_summary: The paper evaluates how LLMs like GPT-4 perform linear and non-linear regression tasks using in-context examples, without further training. The authors present evidence that these models can match or exceed traditional supervised methods on several synthetic datasets. They explore how the performance of LLMs scales with the number of in-context examples and demonstrate that these models can achieve sub-linear regret, suggesting effective learning over time.
strength_weakness: strength_weakness: - While synthetic datasets provide control over experimental conditions, the results might not generalize to real-world data, which can be noisier and less predictable.
- The paper does not extensively address whether the in-context learning approach could lead to overfitting, especially with an increasing number of context examples.
- The paper lacks a theoretical framework explaining why LLMs perform well in these tasks, which could strengthen the overall contribution.
paper_summary: paper_summary: This work conducts a detailed empirical analysis of the performance of several state-of-the-art pretrained Large Language Models for in-context learning of linear and non-linear regression. Around 11 of existing LLMs with 50 in-context exemplars are evaluated on synthetic datasets over 7 tasks comprising of Friedman regression benchmark, and linear and non-linear regression, along with regression with non-numerical inputs. The LLMs are compared with baseline methods of supervised methods from sklearn package such as linear regression (with lasso and ridge regularization), Bagging, GBDTs, Adaboost, etc, and unsupervised methods such as average and random. The synthetic data has inputs with dimensionality between 1-20, and around 50 in-context exemplars are used. The performance of the models is measured using Mean Average Error (MAE) and rank of the models. The authors expound on a sub-linear regret as the number of in-context exemplars scale. 

The performance of these models is comparable to Random Forests and GBDTs, which are surprisingly poorer than linear and non-linear regression approaches. This could be the case as the dimensionality of the models is quite low (< 20). Since the datasets are synthetically generated from a generator function, the amount of data that can be generated an fed to train the supervised baselines are quite large in numbers. It would be useful to know how to extend these models to higher dimensionality of features, which is typically the case in real world scenarios, and  where Random Forest and GBDT baselines usually outperform other regression approaches. Given that baselines such as Garg et. al. 2022 are not considered for comparison, the nature of selecting the 50 in-context exemplars could play a vital role. Given the synthetic nature of the datasets, it should be possible to measure the variance of choosing a different subset of in-context exemplars.
strength_weakness: strength_weakness: 1. The novelty is limited, the impact of which 50 in-context exemplars to select is not looked into. Domain adaptation of LLMs for in-context learning has already been studied in several works (Garg et al 2022, etc), and this work essentially works without any domain adaptation, and might not generalize to higher dimensions or other more complex datasets.

2. It is unclear how to extend this to work with higher dimensional datasets (>20) which are common for regression tasks.

3. More details on how long and many examples are used to train the baselines is important given the datasets are synthetic.
paper_summary: paper_summary: This paper empirically compares the performance of LLMs on linear and non-linear regression tasks against traditional supervised models. The evaluations of LLMs are conducted in an ICL manner, where the training input and output are used as exemplars. Several different supervised methods (with their variations) are studied as the baselines, including linear regression, multi-layer perceptron, Random Forest, Bagging, Gradient Boosting, AdaBoost, SVM, KNN, Kernel Ridge, and Splines. To mitigate the risk of evaluation data being exposed in LLM pretraining, new non-linear regression datasets are also proposed in the evaluation. Overall, this study reveals that LLMs are capable of performing both linear and non-linear regression on the proposed datasets. Strong LLMs can outperform some supervised algorithms like linear regression and gradient boosting. They also approach the performance of the best supervised algorithms in some specific tasks.
strength_weakness: strength_weakness: Please correct me if my understanding is wrong. I think an important property missing in the paper is how the LLM’s performance change with different number of variables in the regression task. Based on Figure 2, there are at most 2 (informative) variables in the evaluations. This is significantly simpler than the dataset description of $1 \leq d \leq 20$ . Also, it is also not clear why there is no noise term $\epsilon * N(0, 1)$ in the linear regression task. With only 2 variables, it would be hard to claim that LLMs are capable of doing general linear regression tasks.
questions: questions: 1. Bar plot might not be the best choice when the scale differences of MAEs are big (e.g., Figure 2a).
2. Sec. 2.1.2 how is $\epsilon$ determined in the experiments?
3. Sec. 2.1.2 re: 
> When $d > 5$, the extra dimensions are ignored

why not just set $d=5$?

Comments: title: Thank you to all Reviewers
Regarding the over-claiming
Reply to Reviewer Gng7
Request for Feedback on Rebuttal and Consideration for Score Re-evaluation
Thank You for Your Feedback and Updated Rating; Additional Data and Insights Provided
Expanding on the Rebuttal
Response to the rebuttal
Thank you for your Feedback
Open Source Code and Intermediate Results; Summary of Changes
Request for Feedback on Rebuttal and Consideration for Score Re-evaluation
Expanding on the Rebuttal
Results on Real-World Datasets
comment: Since the discussion period will end soon, we want to thank the reviewers again for their feedback and engagement. 
We are glad that the reviewers appreciated the novelty of our work [Reviewer Gng7], its thorough empirical analysis [Reviewers y3a3, KJ4G], found it well motivated [Reviewer y3a3]. The regret analysis presented in Section 5 was praised as well [Reviewers y3a3, n2HA].
Additionally, it was judged to broaden the understanding of ICL [Reviewer Gng7]. Reviewers also found our work to be well-written [Reviewers y3a3, KJ4G], practical and useful [Reviewer y3a3, Gng7].

 
We will summarize below the steps we took to ensure that what we have presented in the main manuscript is robust.

Our findings suggest that LLMs are indeed capable of doing regression. Besides the main highlights shown in the main manuscript, we took many precautions to ensure that what we have observed is not simply an artifact of our design choices. The details of these experiments can be found in the Appendix. To this end:
- All the results we represent an average of as many as 100 runs; For example, all the results in Section 3 and 4 are with an average of 100 runs with different train-test splits, ensuring robustness;
- Experimented with 5 original datasets (post revision; 2 in the original submission) to show that what we have seen is not simply because the LLMs saw that data on the Internet
    - We also added a new section in the Appendix titled "Could It Be Just Contamination?" where we discuss the contamination problem even further by: (1) using LLMs with known training data (Falcon 40B) and (2) employing data contamination checks
-  Increased the number of in-context examples given to the LLM to 500 to ensure that this strong performance is not present only in low-data regimes (Appendix L, "Does The Performance Plateau?")
- Compared the performance against 70 different variants of KNNs to test whether the LLMs are just better KNNs or not (Appendix K "Is It Just Better KNN?", extended post revision).
- We ablated the choice of rounding the numbers to two digits (Appendix J, "Effects of Rounding")

In all the experiments we performed, the same conclusion prevailed: LLMs are indeed capable of performing regression, with some LLMs being more effective than others (e.g., Claude 3 Opus).



We believe we have adequately addressed your concerns and suggestions. We are happy to incorporate any additional feedback, especially with the availability of an extra page.


As detailed in the individual comments, we will make the following changes:
1. Section titled "Performance on Real-World Datasets", which will contain a discussion of the performance of LLMs on real world datasets. Following the reviewer's suggestions, we provided results on 5 real-world datasets in our rebuttal (please see the relevant comment [here](https://openreview.net/forum?id=LzpaUxcNFK&noteId=EZ3Xl5zZan))
2. A section dedicated to limitations of using LLMs for regression in general and of this work in particular
3. Improve the clarity of certain sections, as mentioned by the reviewers, and include the clarification experiments.
I appreciate the additional real world evaluations and the detailed rebuttal from the authors. The empirical evaluations are detailed, the utility of applying this for real world tasks at high dimensional features can be further explored in current or future work. I raised the rating to 6 to reflect this.
We will provide our thoughts on this below.

We understand your concern regarding the perceived over-claiming in some parts of the paper. Our intention was to highlight the surprising capability of LLMs to perform regression tasks, even if they do not always outperform traditional methods. The fact that LLMs can consistently outperform unsupervised baselines (Random, Average, Last) demonstrates their ability to handle regression tasks. Furthermore, we compared between LLMs and 70 variants of KNNs (Appendix K, expanded in revision). These KNN Regressor variants were created by modifying: (i) number of neighbors, (ii) weights (uniform or weighted), (iii) p.
We added our conclusion in the bullet point below. We argue that this is further evidence that the LLMs can, indeed, perform regression. 

- We observed that except on Friedman #1, for every other dataset, the top 9 best performing models are all LLMs. In other words, both closed-source models (e.g., Claude 3 Opus, GPT-4) and open-weights models (e.g., DBRX, Mixtral) outperform all KNN models on all the datasets except Friedman #1. This suggests that the mechanism implemented by in-context learning might be something more complex than KNN.

Lastly, a low number of dimensions (i.e., $<10$) is reasonably common in practice. For example, from the UCI repository, there are 136 such datasets out of a total of 478 ([link](https://archive.ics.uci.edu/datasets?NumFeatures=0-10&skip=0&take=10&sort=asc&orderBy=NumInstances&search=&Types=Multivariate))


To summarize, the reasons we do not think this is overclaiming are: (1) The above experiment, comparing LLMs against 70 variants of KNNs, where the LLMs consistently outperform **all** KNN variants, (2) In almost all situations, the LLMs outperform our unsupervised baselines, (3) In many situations the LLMs outperform traditional supervised methods (Claude 3 Opus, for example, ranks 1st overall). While there might be certain settings where the LLMs will consistently under-perform even the unsupervised baselines (although we did not find any such examples), we believe that their strong performance, at least within the limits of our experimental setting, justifies the claim that they are indeed capable of regression. We believe this is a surprising and important observation.

Nevertheless, we are open to revising the section names if there is consensus among the reviewers/AC that they may be perceived as over-claiming.
Thank you for raising your score and engaging with us!

We would like to further elaborate on the motivation of this paper.

Understanding the underlying mechanism of In-Context Learning (ICL) has spawned many recent works. Importantly, these works predominantly train a model specifically to do ICL. We complement these works by exploring the performance of off-the-shelf Large Language Models (LLMs) without any further training. This is important for the following reasons: (i) ICL emerges in LLMs without them being specifically trained for it, and (ii) it is more practical, as practitioners will typically use off-the-shelf LLMs rather than training one from scratch.

Furthermore, as you noted in the "Reasons to Accept" section of the original review, our work broadens the understanding of their potential uses and could streamline analytics processes in various fields.

In other words, the motivation of this work is to further enhance our understanding of the underlying mechanisms at play in ICL. We expand upon previous work aimed at improving our understanding of ICL by investigating it under a more realistic scenario, with off-the-shelf LLMs.
Thanks for your response. While I am still skeptical about the motivation, I'd like to raise my score to reflect the efforts the authors have made during the rebuttal period.
Dear Reviewer,

Since the discussion period will end soon, we would appreciate your feedback on our Rebuttal ([link](https://openreview.net/forum?id=LzpaUxcNFK&noteId=j9hEPsGrdM)) and the expanded points posted as a comment ([link](https://openreview.net/forum?id=LzpaUxcNFK&noteId=caL65oNRj8)). We believe we have thoroughly addressed all your points.

If you find our responses satisfactory, we respectfully request you to reconsider your score.

If there are any remaining issues or questions, we would be grateful for the opportunity to clarify them further.

Thank you for your time and consideration.
Dear Reviewer,

Thank you for your reply and for raising your score! We appreciate your engagement with our work and your constructive comments.

We included a table with the average ranks across various types of datasets to summarize the results more concisely. We will include a similar table in the manuscript. 

More concretely, we show below a comparison between a subset of the models we used:
- LLMs: 9 large language models (LLMs), both open and private:
    - Open: DBRX, Mixtral 8x22b, Mixtral 8x7B
    - Private: Claude 3 Opus, Claude 3 Sonnet, GPT-4, GPT-4 (20240409), Chat GPT, Gemini Pro
- Traditional Supervised Methods: 5 traditional methods:
    - Linear Regression + Poly, Linear Regression, Gradient Boosting, Random Forests
- Unsupervised Methods: 3 unsupervised methods:
    - Average, Random, Last


We attached the table below (sorted by Overall). 

| Model Name | Average Rank Across Linear Datasets (6 datasets) | Average rank Across Original Datasets (5 datasets) | Average Rank Across Friedman Datasets (3 datasets) | Average Rank Across NN Datasets (2 datasets) | Average Rank Across Non-Linear Datastes (10 datasets) | Overall (16 datasets) |
| -----------| ------------------------------------------------ | -------------------------------------------------- | -------------------------------------------------- | -------------------------------------------- | ----------------------------------------------------- | ------- |
| Claude 3 Opus| 2.50  | 3.8| 2.00| 5.5| 3.6| 3.18|
| Linear Regression + Poly| 2.33  | 6.4| 2.33| 2.5| 4.4| 3.62|
| Claude 3 Sonnet| 5.33  | 4.0| 2.66| 7.0| 4.2| 4.62|
| GPT-4| 5.00  | 5.8| 6.00| 8.0| 6.3| 5.81|
| Linear Regression| 1.16  | 11.0| 9.00| 2.5| 8.7| 5.87|
| GPT-4 (20240409)| 5.50  | 6.2| 6.00| 10.5| 7.0| 6.43|
| Gradient Boosting| 9.50  | 5.6| 5.33| 2.0| 4.8| 6.56|
| DBRX| 7.83  | 8.2| 8.66| 10.5| 8.8| 8.43|
| Mixtral 8x22B| 9.66  | 7.0| 9.00| 9.0| 8.0| 8.62|
| Gemini Pro| 7.66  | 7.6| 10.66| 12.0| 9.4| 8.75|
| Random Forest| 12.33 | 8.8| 7.66| 5.5| 7.8| 9.50|
| KNN| 12.66 | 10.2| 11.33| 3.0| 9.1| 10.43|
| Mixtral 8x7B| 11.50 | 10.2| 12.33| 13.0| 11.4| 11.43|
| Chat GPT| 12.00 | 13.0| 12.00| 15.0| 13.1| 12.68|
| Average| 15.00 | 12.2| 15.00| 14.0| 13.4| 14.00|
| Random| 16.50 | 16.6| 16.33| 16.5| 16.5| 16.50|
| Last| 16.50 | 16.4| 16.66| 16.5| 16.5| 16.50|


For example, Claude 3 Opus achieves an average rank of 2.5 on linear datasets, better than Random Forest, Gradient Boosting, or KNN, and only behind Linear Regression and Linear Regression + Poly. 
Similarly, it achieves an average rank of 3.6 on non-linear datasets, outperforming all the other methods in this comparison. Moreover, Claude 3 Opus achieves the best overall rank, outperforming all the other methods. Overall, at least in the settings we investigated, the LLMs indeed prove to be capable of regression.


We acknowledge that there are open questions regarding the limits of LLMs in regression tasks, such as whether their strong performance is restricted to specific conditions like a small number of examples or dimensions. However, these questions should not detract from the merits of our findings. Instead, they highlight potential avenues for future research that our work enables. 

We believe that future LLMs could perform even better on regression tasks for the following reasons:

- (i) Synthetic regression data might be included during pre-training to enhance the in-context learning algorithm.
- (ii) Synthetic regression tasks could help evaluate how well LLMs leverage their contexts, with the expectation that error rates should predictably decrease with more examples if the LLMs can effectively use their context (therefore, performing well might become a target)
Dear Reviewer,

We will provide an expanded discussion on the rebuttal.

(1) Regarding the novelty, we argue that the novelty lies in the fact that, to the best of our knowledge, it has been unknown that off-the-shelf LLMs can do regression reasonably well, just by giving them in-context examples, without any additional training. While the task of regression with transformers and in-context exemplars has been studied in previous work, they all trained the model for the specific task at hand. Here, we show that capable LLMs already emerge with an underlying in-context learning algorithm that is capable of learning the regression task on-the-fly. 

We believe that the observation the off-the-shelf LLMs perform well even on synthetic datasets is a significant finding in itself and can pave the way for future work to: (i) better understand the limits of in-context learning, and (ii) develop better LLM-based approaches for regression. Other potential avenues for future work include: (i) adding synthetic regression data during pre-training to improve the in-context learning algorithm; and (ii) testing how well LLMs use their context by evaluating their performance on synthetic regression tasks, where the error should predictably decrease with more examples if the LLM is indeed capable of using all its context well.

As you have acknowledged in your review, while domain adaption of LLMs has been studied, this work essentially works without any domain adaptation. We believe this is a novel observation.
> Domain adaptation of LLMs for in-context learning has already been studied in several works (Garg et al 2022, etc), and this work essentially works without any domain adaptation

Regarding the impact of which 50 in-context exemplars are selected, we repeat each experiment 100 times and report the average. Each time, we select train/test exemplars. Therefore, we argue that our results are robust and representative.

We discuss higher-dimensional datasets in (2). Regarding their applicability to more complex datasets, we would like to remark that our analysis encompassed the Friedman benchmark datasets. We leave the exploration of other datasets to future work.

(2) Regarding higher-dimensional datasets, we believe that techniques typically used for feature selection could be effectively applied to LLMs as well. Although we agree that this remains to be proven, this can be a potential area for future research. Given the current manuscript's length of approximately 50 pages with the additional results, we leave this for future work.

Our primary goal was not to propose LLMs as replacements for traditional regressors. Instead, we showed that LLMs are capable of performing regression, despite not being explicitly trained to do so. We want to emphasize that off-the-shelf LLMs performing well even on lower-dimensional datasets is a significant finding in itself.

We did not claim that LLMs would necessarily perform well on higher-dimensional datasets. Our focus was on showing their potential in lower-dimensional scenarios, which we believe is significant in itself. Furthermore, while high-dimensional datasets are indeed common, there are many practical datasets with a modest number of features where our findings remain highly relevant.

For example, on UCI datasets, there are 478 regression datasets ([link](https://archive.ics.uci.edu/datasets?skip=0&take=10&sort=desc&orderBy=NumHits&search=&Types=Multivariate)). Out of those, there are 136 with less than 10 features ([link](https://archive.ics.uci.edu/datasets?NumFeatures=0-10&skip=0&take=10&sort=desc&orderBy=NumHits&search=&Types=Multivariate)) and 206 with a number of features between 10 and 100 ([link](https://archive.ics.uci.edu/datasets?NumFeatures=10-100&skip=0&take=10&sort=desc&orderBy=NumHits&search=&Types=Multivariate)). Counting the ones with less than 20 features (inclusive), there are over 225 such datasets, just under 50% of the total number of datasets. 
Therefore, even if LLMs might not generalize well to high-dimensional data (although this is not certain), there are still a large number of datasets that fall well within their realm of applicability.

Lastly, we would like to emphasize that the contributions of this work should not be readily dismissed by speculation on how it would work in settings not investigated or mentioned in the paper.

(3) We used exactly the same examples as for LLMs. The training of the baselines has been handled by sklearn with its default values, which are battle-tested and should provide a good start for most practitioners. Similarly, we did not perform any tuning for the LLMs either. All methods were used without any tuning to ensure a fair comparison. We will add more details in the Appendix B, where we describe the models we used. We also investigated in "Is It Just Better KNN?" (Appendix K) with a large number of different KNN variants. For example, Claude 3 Opus always outperform all 70 variants of KNNs we investigated.
Thanks for the updates! 

I appreciate the extra effort in the additional experiments. I agree that the observations from this paper, although they are based-on low-dimension regression tasks, are valuable, and inspiring further research. But I do feel the several parts in the paper, (e.g., "Large Language Models Can Do Linear Regression", "Large Language Models Can Do Non-Linear Regression"), are a bit over-claiming.

I raised my rating to 5.
Dear Reviewer,

Thank you for your positive review and the high score. We are glad that you found our work valuable.

If you have any further questions or need additional clarifications, please let us know. We would be happy to provide more information.

Thank you again for your thoughtful feedback.
We open sourced our code together with the intermediate results and examples of how to re-create some of the plots in the paper. This way, if anyone is interested in performing other analysis or using other metrics, they can do so. 

We mentioned this in our rebuttal. In this comment, we included the Anonymous Github [link](https://anonymous.4open.science/r/llm4regression-BD81/README.md). The Github page also contains: (1) Local Jupyter notebooks, (2) Links to Google Colab notebooks, (3) Examples on how to read the data and re-construct some of the plots in the paper. We believe (1) and (2) will be especially helpful, allowing interested readers to quickly try do to regression with LLMs.

We also made small changes to the Appendix. Our motivation for including these changes is to further strengthen the paper. These changes complement the paper without altering its main conclusions or the overall message. The changes include:
- Small reorganization to ensure that relevant figures stay closer to the corresponding sections
- Expanding on the number of original synthetic datasets
- A full section in the Appendix titled "Could It Be Just Contamination?" where we expand on the measures we took to ensure that the results presented in the paper are not an artifact of contamination. This is has also been a motivation for including more "Original" synthetic datasets. Additionally, we also:
(1) included LLMs with known training data (Falcon), and 
(2) employed techniques from the literature to detect contamination in black-box models. 
- A full section in the Appendix titled "Beyond Transformer-Based LLMs", where we investigate the performance of LLMs with architectures different from transformers. Specifically, we used `RWKV-based 14B` and `StripedHyena 7B`. We found that RWKV is worse than traditional transformer-based LLMs, although it generally remains better than our unsupervised baselines, with the exception on two linear regression dataset.
Dear Reviewer, 

Since the discussion period will end soon, we would appreciate your feedback on our Rebuttal ([link](https://openreview.net/forum?id=LzpaUxcNFK&noteId=onU0uvgdVQ)) and the expanded points posted as a comment ([link](https://openreview.net/forum?id=LzpaUxcNFK&noteId=ZgNCzfxwhA)). We believe we have thoroughly addressed all your points.

If you find our responses satisfactory, we respectfully request you to reconsider your score. 

If there are any remaining issues or questions, we would be grateful for the opportunity to clarify them further.

Thank you for your time and consideration.
Dear Reviewer,

We will provide an expanded discussion on the rebuttal.


(1) Please refer to our comment ([link](https://openreview.net/forum?id=LzpaUxcNFK&noteId=EZ3Xl5zZan)). We included 5 additional real-world datasets. 

(2) We do not expect overfitting to be problematic in the regime of an increasing number of examples. Regarding overfitting in general and how it affects in-context learning vs traditional methods, we believe this is an interesting question, but it is outside the scope of the paper (which already grew to ~50 pages with the additional results and experiments). We explore in Appendix L (Figures 31,.., 35) how the performance of GPT-4 compares with Gradient Boosting, Random Forest and LR + Poly when given more examples (20 <= x <= 500). We observed that GPT-4 behaves “normally”, in the sense that its performance improves with the number of examples at least as well as the other methods. 

Additional evidence that the LLMs are capable of leveraging can be found in the literature [1, 2].


(3) We would like to remark that this paper has not intended to be an explanation on why ICL works in general or why it works for regression. We included in the revision an extended abstract in the Appendix, expanding on the papers included in the Related Work, some of which attempt a theoretical understanding of the ICL mechanism. We would like to highlight that even though in-context learning took off with GPT-3 and many papers attempted to explain it [3, 4, 5, inter alia], the exact mechanisms at play in real-world LLMs, such as the ones we used in this work, are still not fully understood [6].

Furthermore, we want to emphasize that the primary focus of our paper is on exploring the extent to which transformers can perform regression, rather than providing a theoretical framework for in-context learning (ICL). 
We believe that our findings can be helpful for a future theoretical framework, as it further describes what transformers can do. The fact that off-the-shelf LLMs are capable of performing regression has not been previously known.
This discovery is particularly surprising because it is not limited to proprietary models like GPT-4 and Claude 3; it also extends to open-weight models such as Yi 34B, DBRX (Appendix), and Mixtral 8x22B (added in the Revision). This suggests that, perhaps, once an LLM reaches a certain level of capability, it might universally acquire the ability to perform regression, despite not being explicitly trained to do so. 

In our paper, we hypothesize that LLMs emerge from their pre-training as highly capable online meta-learners. This capability may explain their ability to perform a wide variety of tasks, including some that were not encountered during their pre-training phase, such as our original regression datasets. This hypothesis motivated the exploration in Section 5.




[1] [Agarwal, Rishabh et al. “Many-Shot In-Context Learning.” ArXiv abs/2404.11018 (2024): n. pag.](https://arxiv.org/pdf/2404.11018)

[2] [Jiang, Yixing et al. “Many-Shot In-Context Learning in Multimodal Foundation Models.” (2024).](https://arxiv.org/pdf/2405.09798)

[3] [Oswald, Johannes von et al. “Transformers learn in-context by gradient descent.” International Conference on Machine Learning (2022).](https://arxiv.org/pdf/2212.07677)

[4] [Xie, Sang Michael et al. “An Explanation of In-context Learning as Implicit Bayesian Inference.” ArXiv abs/2111.02080 (2021): n. pag.](https://arxiv.org/abs/2111.02080)

[5] [Ahn, Kwangjun et al. “Transformers learn to implement preconditioned gradient descent for in-context learning.” ArXiv abs/2306.00297 (2023): n. pag.](https://arxiv.org/pdf/2306.00297)

[6] [Shen, Lingfeng et al. “Do pretrained Transformers Learn In-Context by Gradient Descent?” (2023).](https://arxiv.org/pdf/2310.08540)
We present the new results obtained from testing the LLMs on real-world datasets. Two reviewers highlighted the absence of real-world evaluations, and we recognize that future readers may also wonder about the LLMs' performance on real-world datasets. To address this, we ran additional experiments on the following datasets.

- (1) Liver Disorders (UCI id=60)

| Dataset         | Method                   | MAE  | Rank |
| --------------- | ------------------------ | ---- | ---- |
| Liver Disorders | GPT-4                    | 2.55 | 6.0  |
| Liver Disorders | Gradient Boosting        | 2.56 | 8.0  |
| Liver Disorders | Random Forest            | 2.61 | 12.0 |
| Liver Disorders | Linear Regression + Poly | 2.67 | 19.0 |
| Liver Disorders | Average                  | 2.79 | 22.0 |
| Liver Disorders | Claude 3                 | 3.04 | 24.0 |
| Liver Disorders | Random                   | 3.23 | 27.0 |
| Liver Disorders | Last                     | 3.46 | 28.0 |



- (2) Real Estate (UCI id=477)
  
| Dataset     | Method                   | MAE   | Rank |
| ----------- | ------------------------ | ----- | ---- |
| Real Estate | Gradient Boosting        | 4.30  | 1.0  |
| Real Estate | Claude 3                 | 5.08  | 4.0  |
| Real Estate | GPT-4                    | 5.254 | 6.0  |
| Real Estate | Linear Regression + Poly | 5.37  | 7.0  |
| Real Estate | Random Forest            | 5.72  | 10.0 |
| Real Estate | Average                  | 11.12 | 26.0 |
| Real Estate | Random                   | 14.38 | 27.0 |
| Real Estate | Last                     | 15.26 | 29.0 |

- (3) Diabetes (from sklearn)

| Dataset  | Method                   | MAE    | Rank |
| -------- | ------------------------ | ------ | ---- |
| Diabetes | Random Forest            | 41.30  | 1.0  |
| Diabetes | Gradient Boosting        | 47.95  | 16.0 |
| Diabetes | GPT-4                    | 50.26  | 17.0 |
| Diabetes | Claude 3                 | 54.13  | 19.0 |
| Diabetes | Linear Regression + Poly | 55.84  | 20.0 |
| Diabetes | Average                  | 68.02  | 25.0 |
| Diabetes | Random                   | 94.34  | 27.0 |
| Diabetes | Last                     | 103.96 | 28.0 |

- (4) Servo (UCI id=87)

| Dataset | Method                   | MAE  | Rank |
| ------- | ------------------------ | ---- | ---- |
| Servo   | Gradient Boosting        | 0.24 | 2.0  |
| Servo   | Claude 3                 | 0.27 | 4.0  |
| Servo   | GPT-4                    | 0.43 | 10.0 |
| Servo   | Random Forest            | 0.44 | 11.0 |
| Servo   | Linear Regression + Poly | 0.62 | 13.0 |
| Servo   | Average                  | 1.42 | 25.0 |
| Servo   | Last                     | 1.60 | 27.0 |
| Servo   | Random                   | 1.77 | 29.0 |

- (5) Movies (UCI id=424)

| Dataset | Method                   | MAE (e+07) | Rank |
| ------- | ------------------------ | ---------- | ---- |
| Movies  | Random Forest            | 3.02       | 2.0  |
| Movies  | Gradient Boosting        | 3.02       | 3.0  |
| Movies  | Claude 3                 | 4.50       | 17.0 |
| Movies  | GPT-4                    | 5.47       | 20.0 |
| Movies  | Average                  | 6.19       | 21.0 |
| Movies  | Linear Regression + Poly | 6.39       | 22.0 |
| Movies  | Random                   | 8.16       | 23.0 |
| Movies  | Last                     | 8.25       | 24.0 |


  


Except (4) Servo and (5) Movies, all the other datasets have numeric features. (4) Servo has categorical features. (5) Breast Cancer has categorical features and missing features. The LLMs can implicitly handle both: (i) categorical features and (ii) missing features without any additional pre-processing. For missing features, we simply put "unknown". 
For the traditional supervised methods, we encoded the categorical features with One Hot Encoder. For missing features, we imputed them with the mean. We ran all the traditional supervised methods described in the paper.

Lastly, we do not intend to suggest LLMs should replace traditional supervised methods for regression or that they obtain a superior performance in all situations. We believe that the observation the off-the-shelf LLMs perform well even on synthetic datasets is a significant finding in itself and can pave the way for future work to: (i) better understand the limits of in-context learning, and (ii) develop better LLM-based approaches for regression. Other potential avenues for future work include: (i) adding synthetic regression data during pre-training to improve the in-context learning algorithm; and (ii) testing how well LLMs use their context by evaluating their performance on synthetic regression tasks, where the error should predictably decrease with more examples if the LLM is indeed capable of using all its context well.