From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
Authors: Robert Vacareanu
Vlad Andrei Negru
Vasile Suciu
Mihai Surdeanu
Venue: COLM (2024)
Keywords: regression, emergent capabilities
TL;DR: Large Language Models can do linear and non-linear regression when given in-context examples, sometimes better than specialized methods such as Gradient Boosting or Random Forest

Abstract: We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman \#2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting.
We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.

Published as a conference paper at COLM 2024
From Words to Numbers: Your Large Language Model Is Se- cretly A Capable Regressor When Given In-Context Examples
Robert Vacareanu"
Vlad-Andrei Negru
University of Arizona
Technical University of Cluj-Napoca
Technical University of Cluj-Napoca
Vasile Suciu
Mihai Surdeanu
Technical University of Cluj-Napoca
University of Arizona
Abstract
We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g,, GPT-4, Claude 3) are able to perform regression tasks with performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such AdaBoost, SVM, Random Forest, KNN, Gradient Boost- ing. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining sub-linear regret.
1 Introduction
Large Language Models (LLMs) are capable of learning to perform task when given exam- ples of that task in their context, without any additional training. This surprising capabil- ity, called in-context learning (ICL) Brown et
Mean Absolute Erro
20
100
(2020), emerges just from next-token prediction for sufficiently large models.
Claude
GPT.4
Linear Reg Regression essiiio
DBRX
Randem Forest
Gradient Boosting
13N
We use regression tasks to analyze the in- context capabilities of already pre-trained large language models (LLMs), such as Llama2, GPT- 4, or Claude 3. Garg et al. (2022) have pre- viously explored the range of functions that transformers, when trained specifically for in- context learning, are capable of learning. How- ever, contemporary LLMs emerge as capable in-context learners without being specifically trained for it. We extend previous work and analyze the extent to which LLMs, decoder- only transformers trained regressegrys for next-token prediction, are capable learning regression functions when given in-context ex- emplars, without any additional form of su- pervision training.
Model
Figure 1: Mean Absolute Error (L) compar- ison between three large language mod- els (LLMs) and four traditional supervised methods for learning a linear regression function with one informative variable out of two. Given only in-context examples and without any additional training or gra- dient updates, pre-treee trained LLMs such as Claude 3, GPT-44, or DBRX can outperform supervised methods such as Random For- est or Gradient Boosting.
"Correspondence to ivacareanuearizon izona.edu "Code available https: / thub. www........oo..oooo...oooooooooe

---

Published as a conference paper at COLM 2024
Similar to previous work (Garg et al., 2022), use (synthetic) regression datasets. Synthetic regression datasets have the following advantages:
(i) Algorithmically generated: The data is guaranteed to be generated deterministically, by a well-determined (and logical) formula. This property makes them suitable to use when investigating whether given model is capable of unraveling the underlying structure of the data.
(ii) Difficulty control: The user has direct access to the difficulty of the synthetic regression problem and can investigate the cases of simple linear regressions of the form = ax + b, to more difficult problems such as Friedman #2, highly non-linear function used for benchmarking:
y -ー
V (x9 (27 (x2 x3
1
)2)
X2 X4
(iii) Data availability: Lastly, synthetic datasets present the advantage of allowing the user to generate novel data in large(r) quantities. Additionally, it ensures that models are less likely to have been previously exposed to these specific problems.
Formally, let Dn be a dataset consisting of input-output examples: D2 ((x,,)),..., bn where x; E R2, with I S 20 typically. We have y; = f(x,), where f: IR - R and yi E IR. We do not put any restrictions on and study functions ranging from simple linear predictions (i.e., f(x) ax + b) more complex and highly non-linear functions f() x 10sin( 5az
100 + 10cos( 6xx )). We study how well various models such as LLMs (e.g,, GPT-4), traditional supervised models (e.g., Random Forest), and unsupervised baselines (e.g,, random prediction) are capable predicting Vn+1 when given access to input-output examples (i.e., D,)) and x=+1-
tOur study shows that pre-trained large language models (LLMs) display a surprisingly good performance various regression tasks. For example, in Figure 1, without any parameter update, Claude 3 approaches the performance of a Linear Regression model and largely outperforms other supervised methods such as Random Forest or Gradient Boosting on a randomly generated linear regression dataset, with one informative variable out of two.
2
Experimental Setup
We describe the models and the datasets we use in our experiments below.
2.1
Datasets
We experiment with types of datasets: (1) linear regression, (2) non-linear regression, and (3) regression datasets with non-numerical inputs. We describe each below.
2.1.1 Linear Regression Datasets
We experiment with linear regression tasks of the form wx + b, where w, x E Rd, b E R, and IR with 1 < d < 10. We vary both d, the dimension of the input x, and the number of informative variables (i.e,, the number of non-zero elements in w).
When generating a dataset, we sample the input x from (0, 1). We sample the weight vector w from Uniform(O, 100).
2.1.2 Non-Linear Regression Datasets
For non-linear regression problems, we use the three problems introduced by Friedman, called Friedman #1, Friedman #2, and Friedman #3 (Friedman, 1991; Breiman, 1996). For
We used sklearn. Please see make regression for more details.
2

---

Published as a conference paper at COLM 2024
example, Friedman #1 is defined as follows:
y(x) 10 sin (x0x1 20(x2 0.5) + 10x3 + 5x4 + N(0,
Where y E R,x Uniform(0, 1)a and e E R. We have S d. When d > 5, the extra dimensions are ignored for the computation y.
While we create these datasets with different random seeds, resulting in different Da, making a particular Dr very unlikely to have been seen by the LLMs during training, it is still possible that they have seen different D, originating from the same generator function f. In an attempt to mitigate this risk, we created 5 new non-linear datasets. We describe them in the Appendix C. For example, one of these functions is: y(x) = + 10sin 5 where x ~ Uniform(O, 100) (plotted in Figure 5).
+ 10cos( 100
100
To supplement the non-linear regression datasets and following Garg et al. (2022), we create datasets using randomly initialized neural networks. We explore the outputs of 2 types of neural networks: (1) sequence of simple linear layers with ReLU non-linearity in-between, and (2) the output of randomly initialized transformer encoder block.
2.1.3 Regression With Non-Numerical Inputs
To further investigate whether the models are able to learn abstract tasks beyond those subsumed by numerical regression (Razeghi et al., 2022), we design the following tasks.
We (randomly) map symbols (i.e., characters) to numbers (e-g., - 1). We then randomly sample a subset these symbols in order to keep the context size manageable and to not need large number of examples. We map the symbols to numerical value by sampling weight vector w E Rd and doing a dot product between it and the corresponding values of each symbol. We use lowercase ASCII letters as our symbols (i.e., z). We randomly sample 5 symbols which will serve as our vocabulary. We include the pseudocode in Appendix C.11.
2.2 Models
We experiment with three types of models: (1) large language models such as GPT-4, (2) supervised models such Random Forest, and (3) heuristic-based unsupervised models such as random sampling. All models have access to the same train data and are evaluated on the same test partition. They have access to an input dataset ID, and are asked to predict the Va+1 corresponding to the X,+1- The train partition is used for in-context exemplars for LLMs and supervised training for supervised methods. Due to budget constraints and the context size limitations of the LLMs, we round input values to two decimal places. We repeat each experiment with different random seeds.
We use total of 12 large language models (LLMs), both open and private. Specifi- cally, we use Mistral7B, Mixtral8x7B, CodeLlama70B, Llama2 70B, Yi 34B, DBRX (weights available) and ChatGPT, GPT-4 (OpenAl) and Claude 3 Opus, Claude 3 Sonnet (Anthropic), Gemini Pro (Google), and Mistral Medium (Mistral) (weights not available). The models we use cover a wide range of parameters, from 7B or less (Mistral) to 132B or more (DBRX). 4 Un- less otherwise specified, we interact with the models only through prompting and in-context exemplars. We use the same prompt for all models and do not do any prompt tuning. The prompt is of the form Feature 1: <number> Taabbee>>Feeeurrrr 2: <number> AnOutput: <number>. In- context exemplars are separated with two new lines "\n\n". For the test example, the model is asked to predict the number corresponding to the Output variable. We observed that some models tend to provide additional explanations, before outputting the final number. To prevent this behavior, we add an additional text in the beginning, instructing the LLM to only output the number. We give a complete example in Appendix D.1.1. Additionally,
LLMs:
We provide extra experiments without rounding in Appendix N show that the strong results we observed are not an artifact of rounding. "Since the number of parameters for some models is not disclosed, it is possible that certain closed models may have fewer than 7B more than 132B parameters.
3

---

Published as a conference paper at COLM 2024
1IM
Terhional Siperised Model Nedel
Trichree Superdied Hotel
Unoperakee Midel
West 400ee Eroo
Hour Alockin
Coods onla
cant
Pay
Avsrage
fansdn
Goude
GPL4
y47
oge Uk Ray
Acaage
Rindion
Model
Model
(a) Informative Variables: 1; Total Variables:
(b) Informative Variables: 2; Total Variables:
The performance, as measured by the Mean Absolute Error (4), across large language models (LLM), traditional supervised models and unsupervised models on two different random regression tasks: (a) sparse linear regression, where only out of a total 3 variables is informative, and (b) linear regression with two informative variables. The results are averages with 95% confidence intervals from 100 runs with varied random seeds. All LLMs perform better than the unsupervised models, suggesting more sophisticated underlying mechanism at play in ICL. Furthermore, some LLMs (e.g, Claude 3) even outperform traditional supervised methods such as Random Forest or Gradient Boosting.
Figure 2:
we analyze the explanations provided by the models in Appendix L, finding that there is sometimes discrepancy between the rationale given for their predictions and the actual predicted values. Unless otherwise specified, we use a temperature of 0.
Supervised Baselines: We use total of 10 traditional supervised models, available in most statistical learning packages. We use: Linear Regression (4 versions: (i) regular- ization, (ii) Ridge regularization, (iii) Lasso Regularization, and (iv) no regularization and with polynomial features), Multi-Layer Perceptron (6 versions, versions with different widths (Cybenko, 1989; Hornik et al., 1989) and 3 versions with different depths (Lu et al., 2017)), Random Forest, Bagging, Gradient Boosting, AdaBoost, SVM, KNN, Kernel Ridge, and Splines. Similar to the LLM case, we do not tune any hyperparameters and use the defaults available in sklearn.
important to note that these supervised baselines are very strong: (1) many of them are the results of algorithms specifically designed for regression (e.g., Splines); (2) all perform parameter updates (unlike an LLM with ICL); and (3) the default hyperparameters,
set in widely-used statistical packages, have been refined over time to offer reliable and generally strong performance across variety scenarios.
Unsupervised Baselines: In order to contextualize the performance of the LLMs and to evaluate their effectiveness relative to basic heuristics, we incorporated the following series of heuristic-based unsupervised baseline:
1. Average: Predicts the next value, Yn+1, as the mean of preceding outcomes:
Yn+1 = 1 Ei
2. Last: Uses the most recent tuple (xn, Vn) for prediction, such that Vn+1 Yn-
3. Random: Predicts Vn+1 by randomly selecting from the set of prior observations Uy....,yu) The final prediction thus yn+1 sample((y1..... (y,
Additional details on the models are provided in Appendix D. We include results with additional models, such as the latest release of GPT-4 at the time of submission 200t-4222240--000 or Mixtral Mixture of Experts 8x22B in Appendix G, where we present the average rank obtained by each model.
3 Large Language Models Can Do Linear Regression
Our first experiment intends to capture how well LLMs can do linear regression when given only in-context examples. To this end, we experiment with different types of regression problems, varying the number total variables and the number of informative variables. We provide a total of 50 tuples 1550 {(x,y) I 1 1..550 as in-context exemplars and
4

---

Published as a conference paper at COLM 2024
Teat Sppertadee
Uargeedded
10
19
18
24
24
20
14
28
10
27
222
10
22
20
26
36
24
25
22
Clonlo open
GPP4
Hearsl Mediion
64228
Hattoal wert
Hatre T3
Coile LABION 73B
Ltemat: Cles
Gradlent Eaasting
Beoging
Fermet kidee
Licar Neprresoo
Lande
MEL Diese
MAS Dresd
HILH sedde
Paesdor Frrmis
Nodon
State
Spine
Artrop
Ranidom
Liseer Regrssaaon
NET
Nethod
The rank of each method investigated over all four linear regression datasets. Rankings are visually encoded with color gradient, where green means better performance (higher ranks) and red indicates worse performance (lower ranks). Notably, very strong LLMs such as Claude 3 and GPT-4 consistently outperform traditional supervised methods such as Gradient Boosting, Random Forest, or KNN. (best viewed in color)
Figure: 3:
ask the model to generate 1/5I, corresponding to X51- We repeat the experiments with 100 different random seeds and present the average and 95% confidence intervals.
We present two bar plots in Figure 2, corresponding to two different datasets: (1) dataset consisting of three variables, with single informative variable (Regression NI 1/3), and (2) one dataset containing two random variables, where both variables are informative (Regression
2/2). For LLMs, we selected Claude Opus (Claude 3), GPT-4, and Gemini Pro, as they are the flagship closed-soure models currently available, and Mixtral8x7B (Mixtral), 1lama2 70B (Ilama 2), Yi 34B (Yi) and DBRX (DBRX) as the flagship open-weights models. Traditional supervised models in our analysis included Linear Regression (LR), Multi-Layer Perceptron (MLP), Random Forests (RF), and Gradient Boosting (GB). Addi- tionally, we include a fifth supervised method, the one resulting in the best performance." We would like to remark that this is a very strong baseline, it highlights the best perfor- mance obtainable with hindsight information. For the unsupervised baselines we included (i) Average, and (ii) Random Sampling. We draw the following observations from this experiment:
First, LLMs, when given in-context examples of input-output pairs, exhibit (perhaps surprisingly) good overall performance. When compared with unsupervised baselines, the large language models always outperform them, indicating that the underlying mechanism at play is more sophisticated than such simple heuristics.
Second, we remark that LLMs in some cases outperform even supervised methods. For example, for the regression task with one informative variable out of a total of (Regression NI 1/3), Claude 3 ranks 3 out of a total number of 31 models, only (slightly) behind Linear Regression and Linear Regression + Poly. For example, when averaging the mean absolute error across all runs, Claude 3 obtains 0.14, while Linear Regression obtains 0.12. It largely outperforms other supervised methods such as Random Forest or Gradient ent Boosting, even though it no gradient updates were performed, nor it was specifically designed for linear regression."
Lastly, we remark that this strong performance is not only specific to the current closed- source flagship models. For example, Mixtral outperforms supervised methods such as Random Forest Gradient Boosting on the Regression NI 2/2 dataset.
Alongside the two bar plots, we include a heatmap in Figure 3 to show how each model ranks across different datasets. We show the datasets vertically and the models horizontally.
TI this method coincides with one previously selected, the subsequent best performer is chosen. *Comparatively, Random Forest obtains 5.32, Gradient Boosting obtains 2.58, and GPT-4 obtains 2.26.

---

Published as a conference paper at COLM 2024
For instance, Claude 3 Opus achieves the top rank (rank=1) the NI 1/1 dataset. Notably, Claude Opus and GPT-4 consistently perform better than methods such as AdaBoost, Gradient Boosting, KNN, or Random Forest. Out of the LLMs with open-weights, both Mixtral 8x7B and Yi 34B Chat outperform methods such as KNN or SVM on all four datasets.
LLM
Traditional Supervised Model
Unsupervised Model
1500
Mean Absolute Error
500
Claude
GPT-4
Gemini Pro
Mixtral
Liama2
Y DBRX
LR
MLP
RF
GBR
LR + Poly
Average
Random
Model
(a) Friedman #
LLM
Traditional Supervised Model Unsupervised Model
Mean Absolute Error
Claude
GPT-4
Gemini Pro
Mixtral
Liama2
Y DBRX
LR
MLP Wide 3
MLP
RF
GBR
Average
Random
Model
(b) Friedman #
LLM
Traditional Supervised Model
Unsupervised Model
500
Mean Absolute Error
300
200
100
Claude 3
GPT-4 Gemini Pro
Mixtral Llama2
Y DBRX
MLP
RF
GBR
LR + Poly
Average
Random
Model
(c) Friedman #
Figure 4: The performance of large language models (LLM), traditional supervised models and unsupervised models on Friedman #1, #2, and #3. The results represent the averages with 95% confidence intervals over 100 different runs.
Overall, these results reveal that large language models, whether closed-source (e-g., Claude 3, GPT-4) or open-weights (e.g,, DBRX, Mixtral 8x7B), are capable of performing linear re- gression tasks using in-context exemplars composed of (x,y) pairs, all without the necessity for gradient updates. While the performance across these models varies, it consistently outperforms that of unsupervised baselines, suggesting that the underlying mechanism at play is more sophisticated than these simple heuristics. Moreover, specific LLMs (e.g., Claude 3, GPT-4) consistently exceed the performance strong supervised baselines such as Random Forests, Gradient Boosting, or KNN.

---

Published as a conference paper at COLM 2024
We present extended results, encompassing wider array of models and datasets, in Ap- pendix
Large Language Models Can Do Non-Linear Regression
We extend our previous analysis to non-linear regression problems.
4.1 Friedman Benchmarks
We use the synthetic regression benchmarks introduced by Friedman (1991). Below, provide the definition of the Friedman #2 dataset, with complete definitions for all datasets available in Appendix C.
2 + + (
f(x)
2 -1)=-19+
x2 X3
(1)
X2 X4
where represents noise added to the system, modeled as Gaussian distribution (0, 1), and the variables X1, X2, X3, and X4 are drawn from uniform distributions as follows: X1 ~ (0, 100), ~ U (40m,, 5607r), (1, 1), and X4 ~ (1),
Our findings for the Friedman #1, #2, and #3 benchmarks are presented in Figure 4. The selection of methods follows to the same procedure used in Section 3: three leading closed- source LLMs, four leading open-weights LLMs, and five conventional supervised models- including the best performing model-and two unsupervised baselines. We remark that the strong performance of LLMs persists in the non-linear case as well. For example, Claude 3 outperforms all but the Linear Regression with Polynomial Features (LR Poly) on Friedman #2.
4.2 New Regression Datasets
In an effort to mitigate the potential fa- miliarity of models with pre-existing datasets encountered during their pre- training phase,
experiment with two new non-linear regression datasets which are unlikely have been part of the pre-training phase. Our methodol- ogy as follows. Our first novel dataset (called Original #1), plotted in Figure 5, created to resemble a line with oscilla- tions:
80
40
5xx
67LX
y = x + 10sin
+ 10cos(
(2)
100
100
Where 100).
Figure 5: An example of of our new linear regression functions. The function was de- signed to mimic a linear trend with oscillations.
For the next dataset (called Original #2), we draw inspiration from the datasets introduced by Friedman, but we modify the domain of and change
the operands (e.g., 2 + 4). We provide an example below:
y (xf (X2 X3
)2)
/X2 /x4
is important to underscore that the primary goal of these novel datasets is not to construct inherently difficult challenges for the LIMs, but rather to minimize the probability of
7

---

Published as a conference paper at COLM 2024
evaluating them datasets they could have already seen during their training phase. We provide additional details on these datasets in Appendix C, along with additional datasets. For an in-depth analysis of potential data contamination concerns, including additional experiments conducted to address these issues, please refer to Appendix
4.3
Discussion
We summarize all our results in the form heatmap in Figure 6. For each dataset, we record the relative rank of each method with respect to all the others. For example, Claude Opus performs the best on Original (rank=1). We structure our results in 3 blocks: (1) LLMs (left), (2) Traditional Supervised Methods (middle), and (3) Unsupervised Methods make the following observations:
(right).
First, on Original (see Figure 5), LLMs largely outperform traditional supervised methods. methods in this context are LLMs. This
Remarkably, eight out of the ten strong performance this dataset is exhibited by both private and ppenhweights models. For example, DBRX outperforms all traditional supervised methods, despite no gradient update.
Second, we remark that the LLMs show a strong performance on all datasets introduced by Friedman (Friedman #1, Friedman #2, Friedman #3) and all datasets introduced by (Original #1, Original #2).
Overall, our results show that LLMs with ICL capable of performing non-linear regression. For example, Claude 3 Opus outperforms Gradient Boosting and KNN on all datasets. We present extended results, encompassing wider array of models and datasets, in Appendix F. We observed that LLMs struggle on the datasets generated with randomly initialized neural networks (e-g., Simple NN #1, Transformer #1), although they remain, generally, better than the unsupervised methods. Due to space constraints, we included in Appendix M an analysis of the performance LLMs on non-numerical regression datasets. We found that even in this regime, LLMs outperform the unsupervised baselines.
Orignal
20
Orginal
Filemass
Cludie toua
Cheate Stmmat
tarta
carmon
Hestrel Meriion
Poatrel bec:3
Wetrral 6r7a
allevi 7B
Comn Llana 11B
Librect T80 3036 Chest
tamte
Groddyt Broutegg
Geosung
Karmll Pudge
Unaar srrporciion
Hin peep
lott Camm
10. teep 3
NEL upidle
HRP tpide
Bedgen
Spline
avsrage
Lest
senstom
Linaer Ruprassion
aur
Method
Figure 6: Rank of each model investigated on the non-linear regression datasets. LLMs are capable non-linear regression. For example, for Original #1, eight out of the ten highest-ranking methods are LLMs. (best viewed color)
How Fast Do Large Language Models Adapt?
Following the surprising results that LLMs are capable doing regression, when given the training data in their context in the form of in-context exemplars, we investigate next how
8

---

Published as a conference paper at COLM 2024
Camubahee Rupee Uh lep tit (C.45)
Oums stme Lnear Lop Et 00,354
140
100
26
(a) GPT-4 on Friedman #2
(b) Claude 3 on Original #1
Figure 7: The cumulative regret of two large language models on two different non-linear regression dataset. Both show sub-linear regret grow, indicating that as more data points are observed, the models become increasingly efficient at predicting outcomes closer to the optimal strategy derived in hindsight.
their predictions improve with the number of examples given. Specifically, we empirically analyze whether the performance of the LLMs approaches that of the best possible fixed strategy over time.
Borrowing from the Online Learning community, we empirically analyze how the cumula- tive regret (i.e., cumulative loss) grows with respect the time step (number of examples in context) (Orabona, 2019). Ideally, good model should, over time, approach the quality of decisions that the best fixed strategy, informed by hindsight, would have made. In other words, the cumulative regret should, ideally, grow sub-linearly over time. To empir- ically estimate how the regret grows, we fit 3 curves: (1) Linear Fit: a * x + b, (2) Sqrt Fit: a (x) + h and (3) Log fit: * log(x) + b.7 We then use the R² coefficient to determine which curve fit is better. We show two qualitative plots in Figure 7. We summarize the results in Table 1, recording the curve fit with the highest R² coefficient for each model. Since simply picking the best curve fit according to the R² score might tell an incomplete story, we include additional plots in Appendix H, covering multiple models and all seven datasets. We draw the following observations. First, the performance of large language models improves with the number of examples, suggesting the mechanism at play is capable of effectively leveraging more data. Second, we remark that very capable LLMs, such as Claude GPT-4 can obtain sub-linear regret, meaning that the predictions made by the LLM approach the quality of decisions that the best algorithm would have made, leading to near-optimal performance in the long run.
We remark that there are differences between our empirical analysis and online learning. Firstly, while online learning often focuses establishing theoretical regret bounds, our approach is empirical, we only empirically show that the regret of certain LLMs grow sub-linearly by using curve fitting and R.. To address potential concerns of overfitting and enhance the robustness of our empirical findings, we repeated the experiment 3 times and averaged the cumulative regret. Second, our results are only for finite (and relatively small) time steps, diverging from the online learning norm of analyzing behavior as T approaches infinity. To provide further evidence that the results are not an artifact of small T, we performed the following experiment. We used GPT-4 and recorded its performance across multiple training dataset sizes, ranging from 20 to 500. We have observed that the performance of GPT-4 continues to improve as the number of in-context exemplars increases, suggesting that, results are not artifact of limited time steps. We include the associated plots in Appendix Q.
Following the empirical evidence that LLMs are very capable regressors, despite not being trained for it, we hypothesize that (very capable) LLMs emerge from their training as very good online meta-learners (Finn et al., 2019; Mirchandani et al., 2023).
The choice of linear, square root, and logarithmic fits is motivated by their common appearance in theoretical regret bounds within the online learning literature.

---

Published as a conference paper at COLM 2024
Model (Dataset
Priedman
Friedman
Friedman
Original
Original
Regression 1/3 Regression 2/2/2
Claude 3 Opus
linear
sqet
sqrt
log
sqrt
log
log
GPT-4
linear
sqrt
log
sqrt
log.
sqrt
Gemini Pro
linear
sqrt
linear
log
sqrt
sqri
sqrt
Yi 34B Chat
linear
sqrt
linear
squt
sqrt
sept
sqrt
Mixtral 8x/B
linear
linear
linear
sqrt
linear
linear
Mistral 7B
linear
squt
linear
linear
linear
DBRX
linear
log
linear
log
sqri
sqrt
sqrt
AclaBocst
lineat
sqrl
linear
sqrt
sqrl
sqrt
Gradient Boosting
sqrt
sqpt
log
sqet
log
sqrt
Linear Regression
linear
linear
linear
linear
linear
log
log
Linear Regression Poly
squt
log
log
linear
log
log
log
Random Forest
linear
sprt
linear
sqrt
sqri
sqri
linear
KNN
linear
linear
linear
log
linear
sqrt
sqrl
Table 1: We show which curve-fit obtained the highest R² for multiple models and datasets.
The slower the growth of the function, the better (i.e., log sqrt >> -inear). (best viewed in color)
6 Related Work
Many previous papers studied in-context learning (Min et al., 2022; Dai et al., 2023; Pan et al., 2023; Kossen et al., 2024), either from a more theoretical standpoint (von Oswald et al., 2022; Xie et al., 2022; Wies et al., 2023; Ahn et al., 2023; Vladymyrov et al., 2024) more empirical one (Garg et al., 2022; Akyürek et al., 2023; Zhang et al., 2023). For a more general discussion of in-context learning, we refer the reader to Dong et al. (2023). Our work fits into the empirical camp, showing through experiments that large language models pre-trained for next-token prediction on web-scale datasets are capable of doing regression when shown input-output examples in their context, sometimes with performance rivalling that of supervised methods such as Gradient Boosting or Random Forests. Previous studies Garg et al. (2022); Bai et al. (2023); Guo et al. (2024); Bhattamishra et al. (2024) have explicitly demonstrated the capacity of transformers to be trained for in-context learning, enabling them to implement wide range of algorithms when provided with relevant examples. For example, it has been shown that transformers can be trained to in-context learn linear functions, with performance similar to that of optimal least squares (Garg et al., 2022; Zhang et al., 2023). Bai et al. (2023) extended this, showing that transformers can implement broad class of standard machine learning algorithms in context when pre-trate pre-trained for it. Li et al. (2023) proved generalization bounds for ICL under specific conditions. Different from this line of work, we do not train any model specifically for in-context learning or regression. Instead, we simply use large, powerful models (e.g,, Claude 3, GPT-3, Llama2) and empirically show that they are capable of performing linear and non-linear regression when given in-context examples, despite not being specifically trained for it.
from
Conclusion
In this paper, we examined the extent which large language models such Claude 3, GPT-4, or DBRX are capable of performing the task of regression, when given input-output pairs as in-context examples, without any gradient updates.
We showed that large language models are capable of doing both linear and non-linear regression, with performance rivaling that of supervised methods such as Linear Regression or Gradient Boosting. We then analyzed how their performance approaches that of the best possible fixed strategy as the number of in-context examples grows, showing how very capable models such as Claude 3 Opus or GPT-4 are capable of approaching the quality of decisions that the best algorithm in hindsight would have made. Our results demonstrate that large language models are capable of doing regression when given in- context examples of (input, output) pairs, despite not being explicitly trained to do so. We leave the exploration of augmenting LLMs' training with synthetic regression and math datasets, during either pre-training or fine-tuning, to future work. We release our code and results at https: //gi wttphh/.cc//ooorreeaaaaaaeeaaaaa
10

---

Published as a conference paper at COLM 2024
Acknowledgments
This work was partially supported by the Defense Advanced Research Projects Agency (DARPA) under the ASKEM and Habitus programs. Mihai Surdeanu declares financial interest in lum.ai, ai. This interest has been properly disclosed to the University of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.
8 Ethics Statement
In this work we explored the extent which large language models (LLMs) are able to perform regression tasks. We did not perform any additional training. We do not envision any negative impact of our results.
Limitations
This study focuses primarily on regression tasks, including an exploration into regression- like scenarios where inputs are symbolic rather than numeric, yet the outputs remain numeric.
A second limitation is the reliance on several large language models, including proprietary ones whose performance may change over time, potentially affecting reproducibility. To address this, we also included leading open-weight models in our analysis, though we note that their performance is generally behind that of private models. Additionally, we release our intermediate results.
Third, the issue of data contamination poses a challenge, given the opaque nature of training datasets for many LLMs. We have taken several steps to mitigate this risk: (i) Our analysis spans multiple LLMs, reducing the likelihood way; (ii) We evaluated models with multiple random seeds on newly introduced datasets (alongside known ones like Friedman #1). In this way, we diminish the chance that these models have been directly exposed to the exact datasets during training; (iii) We included results with Falcon 40B, whose training data is publicly available (please see Appendix for more details). We acknowledge that these measures do not eliminate the potential for data contamination entirely.
all models being contaminated in the same
Fourth, while showed empirical evidence that large language models capable to perform regression tasks, we did not provide theoretical explanations to support these observations.
References
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, and Lama Ahmad et al. Gpt-4 technical report. 2023. URL httph:tt///eee //api. semanti cosnantiscchlar... wrg/CorppsII: 20 257532815.
Mehreen Ahmed. CSM (Conventional and Social Media Movies) Dataset 2014 and 2015. UCI Machine Learning Repository, 2017. DOL:
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openrevieee net/ forum?id=LziniiXE11
01. AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, and Guanwei Zhang et al. Yi: Open foundation models by 01 ai, 2024.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What leam- ing algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL httt:::///penreieew.eee.. forumPid=0g004488yMII
11

---

Published as a conference paper at COLM 2024
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra-Aimée Cojocaru, Daniel Hesslow, Julien Launay, Quentin Malartic, Daniele Mazzotta, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon
series of open language models. ArXio, abs/2311.1680 2023. URL https://api. semanticscholar cemanttisscholaar.rr////oooupplp wwr.aaarr..rrrorousDD255555555 26544666..
Anthropic. The claude 3 model family: Opus, sonnet, haiku. URL https://api.
pommatttiicccoolo..oog///ooolp/ wtg//orrpuID::28222299
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. In Workshop on Efficient Systems for Foundation Models @ JCML2023, 2023. URL / http:///pppeeeeeeeee//eoo net/forumm
id=vICG5HKEKI.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series (Methodological), 57(1))289 300, 1995. ISSN 00359246. URL http:/ http:://ww.jjtttt jstor org/ stable/2346101.
Satwik Bhattamishra, Arkil Patel, Phil Blunsom, and Varun Kanade. Understanding in- context learning transformers and LLMs by learning to learn discrete functions. In The Twelfth International Conference on Learning Representations, 2024. URL https:t/oppeereeiieee
nel/forrumiicrekeyyyeRRR...
L. Breiman. Random forests. Machine Learning, 45:5-32, 2001. semanticscholar hwtpi/laarrrg/o/oreooeee httg:/CoppsID::11111
URL https://api.
Leo Breiman. Bagging predictors. Machine Learning, 24(2):123-140, 1996. ISSN 1573-0565.
doi: 10.1007/b/000088e
URL http:
org/1/..rr1.... 100700007/BB /B00058500
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL filee/147700dbffdb447744 (ttpe/44888888cccccc1 1266 .dff
wwwinngn...nurii neurips wttps/ccppppe.iliceeeee www.ffflllsppaaeee p
Stephanie C. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers, 2022.
Xiang Cheng, Yuxin Chen, and Suvrit Sra. Transformers implement functional gradient descent to learn non-linear functions in context. ArXiv, abs/2312.0657 2023. URL
https::/app. wew.nticscholla........ orgg/orppsID::26112222
George V. Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989. URL https://api. wew.nticsmmmoooooo....... org/ CArpusID:39583350
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta- optimizers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Ōkazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 4005-4019, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10.18653/v1/2000 2t0./////11/222.idnnng----- findings-acl. 247. URL https: / https:///cclnnnooooloooo//////// weeela.fiiinggg findings- acl. 247.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, and Baobao Chang et al. A survey for in-context learning. ArXiv, abs/2330.0003333 2023. URL https: //ppi. semanticscholarr. org/CorpusID: 2638860774
12

---

Xiaochuang Han, Daniel Simig, Todor Mihaylov, Yulia Tsvetkov, Asli Celikyilmaz, and Tianlu Wang. Understanding in-context learning via supportive pretraining data. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12660-12673, loronto, Canada, July 2023. Association for Computational Linguistics. doi: 001/553/v1/2223...0 Published as a conference paper at COLM 2024
Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical As- sociation, 56:52-64 64, 1961. URL httpsss//ap...ssntttccchoo... wwwpoaaarrrrr/ooousosssooa 122009246.
semanticscholar.cmm
Bradley Efron, Trevor Hastie, lain Johnstone, and Robert Tibshirani. Least angle regression.
The Annals of Statistiic,, 32(2):407 - 499, 2004. doi: 10.2114/009053440000000000 URL
https://doi org/10 101.1...111101111 2214/00996564440000000666
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Inter- national Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1920-1930. PMLR, 09-15 Jun 2019. URL https: httt::://proceedingg wwg.mms......mm..mmm www.mmr.ppperr / .rrss/
v97/finn19a 19. html.
Jerome H. Friedman. Multivariate Adaptive Regression Splines. The Annals of Statistics, 19(1):1 67, 1991. doi: 11121//a0s/11733477 URL https://dof org/10. /214/aos/// 1176347963.
Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. An- nals of Statistics, 189-1222,, 2001. URL https://api. semanticscholart wwaccholar.rr//oooooo org/CCrpusID:
39450643.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? URL https://api. wet.ssiismaaaooooo...e wwwpcchllaa.og//orrrrreeee Sorg/CorpssID::20 org//orppss1::22155333. 251253368.
study of simple function classes. ArXiv, abs 2208.00 066, 2022.
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs:
Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //pterev//p net/forund 2d-22226333....
Tianyu Guo, Wei Hu, Song Mei, Huan Wang, Caiming Xiong, Silvio Savarese, and Yu Bai. How do transformers learn in-context beyond simple functions? case study on learning with representations. In The Twelfth International Conference on Learning Representations, 2024. URL httt::://opprreviiie www.nntt/forrmmiidiwweeEEe
URL httpps://acee https://aclanttoolgy/ org. acl long.llog... 708. 708.
Niklas Heim, Tomas Pevny, and Vasek Smidl. Neural power units. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 6573-6583. Curran Associates, Inc., 2020. URL https://prooceedings newrrirriiinnnnnnnii........... cctpcper.pppcceeceecc http/pper.files/ppppppseee htt.leeepapee//2///iLl. Bee80oatrffffccctteeea0000333323 .dd.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9318 1118,9933, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v1 2008553/1122233333 22/2222..fmm findings-emnjly 6244. URL https: /ttp///aaannollooorrrrr////////e
findings-emnlp 624.
K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Netw., 2(5): (6665966666,16 jul 1989. ISSN 0893-6080.
Yu Huang, Yuan Cheng, and Yingbin Liang. In-context convergence of transformers. ArXiv, abs/2310.0522 2023, URL httpp:://pp...mmmtiisssceeo ptt.s/eatnnttcccoooooooooooorooo :rr.://///rrruSS..2223333333333 263829335.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, and Chris Bamford et al. Mixtral of experts. ArXiv, //api. semanti cscholar
2024. URL https:
oow.aattttcsssl.....o.oooooorooo .rtg//CorpusID:226444777777
13

---

Published as a conference paper at COLM 2024
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamffrd, Deven- dra Singh Chaplot, and Diego de Las Casas et al. Mistral 7b. ArXio, abs/ bbbs/231000685,, 2023. URL https://api. semanti semmattiissssol..rrrro/ooossssp 263830494.
Jannik Kossen, Yarin Gal, and Tom Rainforth. In-context learning learns label relation- ships but is not conventional learning. In The Twelfth International Conference on Learning Representations, 2024. URL httpss://openrevllee wttp://ww.ee/oooonndddooaaaaaaaa
Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, and Pin-Yu Chen. Training nonlinear transformers for efficient in-context learning: theoretical learning and generalization analysis. ArXiv, abs/ 0atps/402.15600) 2024.
Yingcong Li, M. Emrullah Ildiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: generalization and stability in-context learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Sheng Liu, Lei Xing, and James Y. Zou. In-context vectors: Making in context learning more effective and controllable through latent space steering. ArXiv, abs 0/ns/2311.6666 2023. URL https://apt: semanticscholar httprriscccooaaooorrrrrrrruuu
2652265511999 665149781.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: ptttoe///procceeinngs. wewr.ns....p... cc/paper files/paper/2917/ File/32cbf687880ed 674a07bf717 674487bf11766ddda--ppppp .pdf.
Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024. URL httph: //ppnreviieee net/ net/forum?idd8p3fuee fu56k...
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, PP. 11048- 11064, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. (0.18653/v1/ http://2222...mmmmaiaa.. 2092.emnlp-maing 759.
URL setps://aclaantholog. org/
Suvir Mirchandani, Fei Xia, Pete Florence, brian ichter, Danny Driess, Montserrat Gonzalez Arenas, Kanishka Rao, Dorsa Sadigh, and Andy Zeng. Large language models as general pattern machines. In 7th Annual Conference on Robot Learning, 2023. URL https:// openreview. www.nnt/ffromiid--Rccomm SMSyE.
Bhumika Mistry, Katayoun Farrahi, and Jonathon Hare. A primer for neural arithmetic logic modules. Journal of Machine Learning Research, 23(185):1-58, 2022. URL http://jmmr.orgg papers/v23/21-0222 1mm...
Tomer Bar Natan, Gilad Deutch, Nadav Magar, and Guy Dar. In-context learning and gradient descent revisited. ArXiv, abs/2311.07772 2023.
Francesco Orabona. A modern introduction to online learning. ArXiv, Abs/1191..32213 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, and Pamela et al. Mishkin. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL https: httpss:/ newriiignnnnneiiisppeee....r newrips.ccccm cc/paper files/ paass//aper///e 2222//2022/ fflebbeeeee530033432944444488113
pdf.
14

---

Published as a conference paper at COLM 2024
Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. What in-context learning "learns" in- context: Disentangling task recognition and task learning. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 8298-83989 8319, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: (0.18653/v1 210////11223.imm// 22003.indingss-cc..5222 URL httpsss//acllatthooolooo org/ 2023. findings--ininn -aliggs-acl... ac1.52c.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 7:2825-2880, 2011.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.186633vv12023 findings-emnlp.9 9966. URL https://aclantthlloge org 2023. findinng--emllp.90.
Lovre Pesut. Who models the models that model models? an exploration of gpt- 3's in-context model fitting ability. URL https://www/
httpeeennnooroooooooootttttt modeee-ooelss----eeaeeeeeee c2ReFadrxkzyRRFKa/ wwwp://aww..///ooooooeeeeeeeeeee wwwaa//wwoooootttttoooooeoe theedeel-----meel-----------e
www.cammpignnpost..hhreee www.sshttrraeeeeee. sos.kttet...rrreiin......
Michael Poli, Stefano Massaroli, Eric Q. Nguyen, Daniel Y. Fu, Tri Dao, Stephen A. Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, 2023a.
URL httpps://pp semanticscholar. set.annissscoll..rrroo/o/oosssss 1tte/CooppuDD:25750333
Michael Poli, Jue Wang, Stefano Massaroli, Jeffrey Quesnelle, Ryan Carlow, Eric Nguyen, and Armin Thomas. StripedHyena: Moving Beyond Transformers with Hybrid Signal Pro- cessing Models, 12 2023b. URL https: / //g/tttttt thub. com/ cogettherccompteer//srrpiieeeeee
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. Impact of pre- training term frequencies on few-shot numerical reasoning. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840-854, Abu Dhabi, United Arab Emirates, December 2022. Associ- ation for Computational Linguistics. doi: 10.18653/ /v1 211/2222..fiiii diw.fiinnnnnnnlppp. URL 200p ftne22.finndd findings-emnlp 59.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, and Xiaoqing Tan et al. Code llama: Open foundation models for code. Arviv, https://api. semant commantisscoo.. cscholar. wwwpllaarrrrg//ooo/ 2ttg:/ooppsSD::211111 26110919.
2023, URL
David Ruppert.
The elements of statistical learning: Data mining, inference, and pre- diction. Journal of the American Statistical Association, 99:567 567, 2004. URL https: seapi..seanticseeee cscholar wwt.:ogg////o/// http//ooppss::::oo 18901 444.
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contamina- tion for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2000 findings-emmlp.722 URL https:://acllaa
// http:////ollaaaooog....o.o org/ 2023. http://.o//.2.....nn-o-eeoomem findings-emmlp.222
Robert E. Schapire. The strength of weak learnability, Machine Learning, 5:197-227, 1989.
URL https: //api. semanticscholar htt.:/oaa...oooecoooss/////00000
15

---

Published as a conference paper at COLM 2024
Lingfeng Shen, Aayush Mishra, and Daniel Khashabi. Do pretrained transformers really learn in-context by gradient descent? ArXio, abs/2310.08540, 2023. URL https://api. semanticscholar.com wrw.:orgg/o/ooooeooe http://orpss:::::: (orpusIDD228499122 268499126.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, and Jean-Baptise Alayrac al. Gemini: A family of highly capable multimodal models, 2023.
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon Shaolei Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //oppenrevew..nee/eeeeeeeeeeeeee
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, and Yas- mine Babaei et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307 009888, 2023, URL httpp:://pppp semanticscholar wwtpsschoaaaaorr/////ooo
wwwp/orppssI::::2......mm.. 259950998.
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arith- metic logic units. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biaaa Biianci, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL
www.nng....uiiii. neurips. wttppperrrcpeeeppepppppppce.eeep fiaepp/rriles///ppr///// f1e2/86447000333226666662222266 Papere--pppppppppp
UCL. Liver Disorders. https://doi.orgg/// /tt/:/002432//AACCAA
UCI Machine
Learning Repository, 1990.
DOI:
Karl Ulrich. Servo. UCI Machine
Learning Repository, 1993.
DOI:
Max Vladymyrov, Johannes von Oswald, Mark Sandler, and Rong Ge. Linear transformers are versatile in-context learners. ArXiv, abs/ 09///2422.1418,0 2024.
Johannes von Oswald, Eyvind Niklasson, E. Randazzo, João Sacramento, Alexander Mord- vintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2022. URL
http://ppp..emmanicsch.....a.a wwwpchhoaaarorr/oouusssiiie orgg/ooppsSD::544855..
Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymy- rov, Razvan Pascanu, and João Sacramento. Uncovering mesa- oostmmitimnzttion algorithms in transformers. ArXiv, abs/ 2e5///20900555 2023.
Jerry W.
Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanx- iao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently. ArXiv, abs//300.0338 2023, URL https://app semanticscholar. org/CorpusID:257311 25737877.
Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https:
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022. URL
Yue Xing, Xiaofeng Lin, Namjoon Suh, Qifan Song, and Guang Cheng, Benefits of trans- former: In-context learning in linear regression tasks with unstructured data. ArXiv, abs/2402.00743, 2024.
I-Cheng Yeh. Real Estate Valuation. UCI Machine Learning Repository, 2018 DOL: https://doi.org/ http://doirrr/00.24/5133 /1.24440/ CC5J30W.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. In RO-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL https: //tpenreview.w..///////// wtwpeeewwwwwewwweeeeeeeeeeeeeeee nett/Forummid-ggoooRRlll
16

---

Published as a conference paper at COLM 2024
A Appendix Structure
17
B Related Work (Expanded)
18
C Datasets
19
C.1 Linear Regression
19
C.2 Friedman 1
19
C.3 Friedman 2
19
C.4 Friedman #
20
C.5 Original #1.
20
C.6 Original #
20
C.7 Original #3
20
C.8 Original #4
21
C,9 Original #
21
C.10 Neural Network Induced
21
C.11 Non-Numerical Regression
21
C.12 Real-World Datasets
22
D Models
22
D.1
LLM
22
D.1.1 Prompt
24
D.2
Traditional Supervised Models
25
D.3
Unsupervised Models
26
E Large Language Models Can Do Linear Regression (Expanded)
26
Large Language Models Can Do Non-Linear Regression (Expanded)
30
G Average Model Ranks
H How Fast Do Large Language Models Adapt? (Expanded)
35
Performance on Real-World Datasets
38
Claude Performance
39
K Costs
40
L LLMs Justifying Their Prediction
40
M Beyond Numerical Regression
46
N Effects of Rounding
46
O Is Just Better KNN?
46
P Could It Be Just Contamination?
47
P.1 LLMs With Known Training Data
48
P.2 Performance When Knowing The Dataset Name
50
Q Does The Performance Plateau?
51
R Beyond Transformer-Basee LLMs
54
A Appendix Structure
We organized this appendix as follows.
In Appendix B we expand the related work discussion.
17

---

Published as a conference paper at COLM 2024
In Appendix C we provide additional details of the datasets we used, including their underlying formulas.
In Appendix D provide additional details of the models used, together with their exact API code.
In Appendix E, provide additional experiments and results to complement Section 3:
Large Language Models Can Do Linear Regression.
In Appendix F, provide additional experiments and results to complement Section 4:
Large Language Models Can Do Non-Linear Regression.
In Appendix G, we show the average ranks obtain by each model across different dataset types.
In Appendix H, we provide additional experiments and results to complement Section 5:
How Fast Do Large Language Models Adapt?
In Appendix I, we provide additional experiments and results on real-world datasets.
In Appendix J, we provide results with more models from the Claude family.
In Appendix K we detail the total costs of running all the large language models.
In Appendix we detail how LLMs provided justifications for their prediction.
In Appendix M we include another experiment: regression task with non-numerical inputs. In Appendix we analyze the effects of rounding.
In Appendix O we analyze whether the performance of LLMs is similar with KNNs or not.
In Appendix P we analyze whether the results we have seen could be the effect of data contamination.
In Appendix Q we analyze whether the performance of LLMs plateaus after given number of in-context examples or not.
In Appendix we analyze the performance of LLMs whose backbone architecture is different from Transformers.
B Related Work (Expanded)
The in-context learning capability of large language models has garnered significant at- tention Brown et al. (2020). How this capability emerges during a standard next-token prediction pretraining and how it operates still up for debate. A substantial body of research is dedicated to exploring the parallels between in-context learning mechanisms and traditional algorithms like gradient descent (Akyürek et al., 2023; von Oswald et al., 2022; Dai et al., 2023; Ahn et al., 2023; Cheng et al., 2023; Mahankali et al., 2024; Vladymyrov et al., 2024). For example, Akyürek et al. (2023) and von Oswald et al. (2022) prove that transformers could theoretically implement gradient descent. Bai et al. (2023) shows that the transformer architecture can implement more complex in-context learning procedures, involving algorithm selection. Cheng et al. (2023) argue that non-linear transformers learn to implement gradient descent in function spaces. von Oswald et al. (2023) suggests that performance ransformer-baseee models may be due an architectural bias towards mesa-optimizaiton. Nonetheless, the extent to which pre-trained transformers actually implement gradient descent when given in-context examples remains topic of debate (Natan et al., 2023; Shen et al., 2023).
Other lines of work investigate the convergence of in-context learning (Wies et al., 2023; Huang et al., 2023). Li et al. (2024) analyzes the training dynamics of transformers with nonlinear attention and nonlinear MLP, expanding upon previous work which considered simpler transformer-based architectures (Huang et al., 2023; Tian et al., 2023). However, for natural language tasks such as sentiment analysis, it is unclear how much learning occurs with in-context examples (Min et al., 2022; Pan et al., 2023; Kossen al., 2024). For example,
18

---

Published as a conference paper at COLM 2024
Min et al. (2022) shows that GPT-3 retains a strong performance even when the labels of the in-context exemplars are random. On the other hand, recent work (Hendel et al., 2023; Liu et al., 2023) investigated how in-context learning creates task vectors, which can then be applied to produce the output.
Another question investigated in recent work is where does the in-context learning (ICL) emerges from (Chan et al., 2022; Xie et al., 2022; Han et al., 2023). For example, Chan et al. (2022) shows that in-context learning appears when the training data has particular properties. Xie et al. (2022) analyzes in-context learning through small scale synthetic dataset (GINC). Han al. (2023) identified subset of the pre-training data that supports in- context learning, showing how continuing pretraining on this subset increases the model's ICL abilities.
Another line of research, which is close to our work, is that of investigating what types of "functions" can be learned through in-context learning (Garg et al., 2022; Zhang al., 2023; Xing et al., 2024). Notably, all these works do not use pre-trained LLMs, but specif- ically train a transformer for the task. Garg et al. (2022) shows empirically that standard transformers can be trained from scratch to perform in-context learning of linear functions. Guo et al. (2024) investigates more complex function classes. Wei et al. (2023) shows that larger language models are able to overcome their semantic priors when shown input-label mappings. Zhang et al. (2023) train transformers with a single linear self-attention layer to in-context learn linear regression tasks, showing that transformers are capable of obtaining performance competitive with the best linear predictor. Bhattamishra et al. (2024) experi- ment with training various models to in-context learn boolean functions. Although not the main focus of their work, they also experiment with pre-trained models such as Llama 2 and GPT-4, showing that they obtain a performance similar to nearest-neighbor baselines for boolean functions. A post on the AI Alignment Forum (Pesut) also explored the types of models that GPT-3 can fit in-context. While their focus was primarily on classification, they did examine regression scenarios as well.
While in this work we focus on in-context learning, there are other previous works that designed specific neural network modules (or cells) for arithmetic computations (Trask et al., 2018; Heim et al., 2020; Mistry et al., 2022).
Different from previous work, we investigate how pre-trained models, such as GPT-4 or Claude 3, without any gradient updates, can learn various linear and non-linear function classes when given examples in-context and thoroughly compare them against multiple traditional supervised methods (Ruppert, 2004) such as Gradient Boosting (Schapire, 1989; Friedman, 2001) Random Forests (Breiman, 2001).
C Datasets
We provide the formulas for all datasets used below. We set the noise to 0 for all datasets.
C.1 Linear Regression
In order to generate the linear regression datasets, we use the function regressnnss available in sklearn (Pedregosa et al., 2011).
C.2 Friedman #1 #1
f(x) = 10 sinxx1 + 20(x2 - 0.5)² + 10x3 + 5x4 + (N,N(O,))
Where Xo, X1, X2, X3, X4 ~ L(O,1))
C.3 Friedman #2
2
f(x) = = /2++
(x2- xx3
+e- +0,1(00,)
X4
19

---

Published as a conference paper at COLM 2024
Where
(0,1000)
(400r,56017)
x3 ~ ((,0
X4 ~ (1,
C.4 Friedman 3
X1 N135
f(x)
arctan
(0,1(
X0
Where
X1 U (0, 100)
X2 (40mz,
~ (0,1)
X4 (1,111
C.5 Original 1
5xxx
6
x + 10sin
+ 10cos
(4)
100
100
Where
x~ (10,100)
(5)
C.6 Original #2 #2
y (xf + (x2 X3
/x2 /x4 )2)
Where
Xo ~ U(0,3)
x1 ~ U(4T,,56m)
X2 ~ ((0,2)
X3 ~ (1,,11
C.7 Original #3
y = 20 +
X1 X2
(xx x3)
X3
Where
Xo (11,3)
X1 (1)
X2 2 10,
X3 (1122))
20

---

Published as a conference paper at COLM 2024
C.8 Original # 4
No
cos(x1) + v (x1)
y 10
sin(xo) +
10
v (x1)
Where
X1 ((2, 100)
C.9 Original
X 100 ma fImax(
10
Where
Xo, x1, X2 ~ (-(22,,25)
C.10 Neural Network Induced
For the random datasets induced by neural networks, we randomly initialize a neural network and create a dataset by feeding random data to it. The dataset Simple Random NN was created by using neural network with one hidden layer with ReLU non-linearities. The dataset Transformer was created by a randomly initialized Transformer encoder block.
C.11 Non-Numerical Regression
We provide the code to generate the non-numerical regression datasets in the Listing 1. Essentially, we assign random number (0 to 26) to each lowercase letter. Then we sample weight vector. The expected output is generated by doing dot product between the underlying assigned value each character and the generated weight vector.
Listing 1: The python code to generate non-numerical regression datasets import random
2 import numpy as np import string
4
5
n_samples 51
def get.character.regrest
(random_state=1))
r = random Random (random_state
10
alphabet list (string ascii.lowercase) shuffled r. sample (alphabet, 26)
11
12i [(c, i) for c, in enumerate (shuffled))
13
gen = np random. RandomState (random
14
15
letters sample (12i, max_num_vars
16
sample r. choices ( (letters) veaavas n.samples)
17
sample np. array (sample) [; 1]. reshapee---, max.num.vars)
19
weight
10 gen. uniform (size= num vars))
21

---

Published as a conference paper at COLM 2024
LLM
Unsupervised Model
100
80
Mean Absolute Error
60
40
20
Claude
GPT-4
Gemini Pro
Mixtral
Lama2
Yi
Average
Random
Model
Figure 8: Performance Comparison between LLMs and unsupervised baselines on non- numeric regression dataset. LLMs are outperforming our unsupervised heuristics even in this regime.
20
21
i21 dict (icttette
22
12i = {v: for k, v in i21 items ())
23
24
y = weigght.eec@n array (1 [[] for in xI) for x in sample]
C.12
Real-World Datasets
We also experimented with several real-world datasets: (i) Liver Disorders (UCI, 1990), (ii) Real Estate Valuation (Yeh, 2018), (ii) Diabetes (Efron et al., 2004), (iv) Servo (Ulrich, 1993),
(v) Movies (Ahmed, 2017). The corresponding results are presented in Section
D Models
In the following, we provide additional details of the models we used for our main ex- periments and how we used them. We used three different types of models, as follows: (a) Large Language Models, (b) Traditional Supervised Methods, and (c) Heuristic-Based Unsupervised Methods. We describe them bellow.
D.1 LLM
This section outlines the 12 Large Language Models (LLMs) featured in our main experi- ments, which include mix of open-weights and private models. We also include additional models, such as the newest GPT-4 version (2Dt-4-22240489) multiple Claude variants, and the most powerful model released by Cohere, Cohere Command R Plus. We tried with Cohere Command R and Cohere Command and observed their performance to be lower, albeit still the unsupervised baselines (except for Cohere Command).
In Table 2, we categorize the models by their names, availability of weights, and developers, dividing them into two distinct sections. The first section lists the models featured in the main paper's 's experiments (referenced in Sections 3, 4, and 5). The second section introduces additional models that were utilized for the extended analysis included in the Appendix.
22

---

Published as a conference paper at COLM 2024
Model Name
Weights
Availabil-
Developer
ity
GPT-4 (Achiam et al., 2023)
Not available
OpenAl
Chat GPT (Ouyang et al., 2022)
Not available
OpenAI
Claude 3 Opus (Anthropic)
Not available
Anthropic
Claude 3 Sonnet (Anthropic)
Not available
Anthropic
Gemini Pro (Team et al., 2023)
Not available
Google
Mistral Medium
Not Available
Mistral
Mixtral Mixture Experts 8x7B (Jiang et al., 2024)
Available
Mistral
Mistral 7B (Jiang et al., 2023)
Available
Mistral
Llama2 70B (Touvron et al., 2023)
Available
Meta
Code Llama2 70B (Rozière et al., 2023)
Available
Meta
Yi 34B (Al et al., 2024)
Available
0l.ai
DBRX
Available
Databricks
GPT-4 (20240409) (Achiam et al., 2023)
Not available
OpenAl
GPT-3 Davinci (Brown et al., 2020)
Not available
OpenAl
GPT-3 Babbage (Brown et al., 2020)
Not available
OpenAl
Claude 3 Haiku (Anthropic)
Not available
Anthropic
Claude v2.1¹
Not available
Anthropic
Claude v2.0
Not available
Anthropic
Claude v1.2
Not available
Anthropic
Cohere Command R Plus"
Available
Cohere
Mixtral Mixture of Experts 8x22B
Available
Mistral
Falcon 40B Almazrouei et al. (2023)
Available
TII
Falcon 40B Instruct Almazrouei et al. (2023)
Available
TII
RWKV v4 14B Peng al. (2023)
Available
Various
StripedHyena Nous 7B Poli et al. (2023b)
Available
TogetherAl
https://www.anthropiic.
htt.::/www...lk...c.o//oo////ttt newprr/dwdeciiidedddeeeeeee--eee seae--ne-ssttte art-open- 10pe--11m
https://cohere. com/ command
Developed collaboratively by numerous contributors across 29 different affiliations
Table 2: Details of the large language models (LLMs) used in our study, divided into two sections. The first section contains the LLMs used for the experiments from the main body of the paper, while the second section includes additional models explored in the extended results presented in the Appendix.
We list in Table the models we used through OpenAl, together with their corresponding model code.
Model Name
API Model Code
GPT-4
gpt-4-0125-prevv 11--33--tuub--1110
Chat GPT
GPT-4 (20240409)
11144tturb-2224-0445 davinci-002 babbage-002
GPT-3 Davinci
GPT-3 Babbage
Table The specific model used for each model family through the OpenAl API.
9nttps://openai.co com/
23

---

Published as a conference paper at COLM 2024
We list in Table 4 the models used through OpenRouter, together with their correspond- ing model code."
Model Name
API Model Code
Claude Opus
a
Claude Sonnet
anthropicccllaud-335omm
Gemini Pro
google/gemini-pro
Mistral Medium
mistralaai/mistrralmmddduuu
Claude 3 Haiku
anthropice/claaude3-hhiktt
Claude v2.1
anthropic/ccladde22.11
Claude v2.0
anthropic/ccauue-220
Claude v1.2
anthropic/claude-11.
Cohere Command R Plus
cohere/ccomand-rrplluss
StripedHyena Nous 7B
Table 4: The specific model used for each model family through the OpenRouter API.
We list in Table 5 the models we used through DeepInfra, together with their corresponding model code.10
Model Name
API Model Code
Mixtral Mixture of Ex-
perts 8x7B
Mistral 7B
mistralaiMMittral-----nnstttttv
Llama 70B
meta-1Laamm/Lammm22-777--hhhhhh codellllnnn/ccceeellmmm//00o11sr 01 01-ai/Vii33BB-hht
Code Llama 70B
Yi 34B
Mistral 7B v2
mistrrlla/Mistrrl----iissttvvv
Table 5: The specific model used for each model family through the DeepInfra API.
We list in Table 6 the models we used through Fireworks, together with their corresponding model code.!! 11
Model Name
API Model Code
DBRX
Mixtral 8x22B
Table 6: The specific model used for each model family through the Fireworks API.
D.1.1 Prompt
We show the prompt we used in Figure 9. Importantly, we used the same prompt for all large language models. We did not tune the prompt.
We encountered cases where the large language model would not produce valid output. For instance, some models would occasionally output an empty string (i.e., ""). In the following, detail the way handled them across the experiments showed in this paper.
9nttps://openrouter.as ai WPhttps://deeiinrra.cmm com "https://firreworsssu
24

---

Published as a conference paper at COLM 2024
The task is to provide your best estimate for "Output". Please provide that and only that, without any additional text.
Feature 0: -2.06 Output: -81.93
Feature -0.64 Output: -25.33
Feature O: 1.62 Output:
The prompt we use for all LLMs. Concretely, we use an initial instruction to prevent the models from explaining their prediciton, something which we observed to happen for some models (e.g,, Claude 3 Opus). Then, we give each input-output pair and finally ask the model predict the value corresponding the test input.
Figure 9:
For the experiments performed with 100 random seeds and 50 random input-output tuples (i.e., Ds0), we simply skip the invalid generations. This is not problematic because it rarely occurs. For example, has never occurred for the any of the experiments showed in Section 3 and Section 4. Moreover, given that run each experiment with 100 random seeds, skipping the invalid generations will still leave us with enough samples to estimate the statistics. For example, out of the results presented in Section and 4, the largest number of invalid generations for a given dataset was 11, by Mistral Medium on Regression NI 1/2. Claude Ópus produced only 2 invalid generations, for Friedman #2, and GPTT4 did not produce any invalid generations. One exception is for the results shown in Appendix J, as we observed that Claude 2.0 and Claude 2.1 produced invalid generations very often. For example, Claude 2.1 generated invalid generations for the dataset Regression 1/1 in 71% of the cases. The reason invalid generations was usually because Claude 2.1 refused to give " "potentially misleading numerical estimates without proper context". A second exception is for Striped Hyena Nous 7B (Appendix for every model-dataset pair where the number of valid generations is under 20.
R). We add cross (x) to the cell in the rank heatmap
For the experiments where we investigated how the performance the models scale with the number of examples, we average 3 random runs for each dataset size. In this set of experiments, only Llama 70B generated invalid outputs a total of times, for Originai #2 and Friedman #2. We skip the random runs with invalid generations.
D.2 Traditional Supervised Models
We use total of traditional supervised methods, resulting in over 20 different configura- tions. Specifically, we used the following models:
1. Linear Regression: We used 4 variants of Linear Regression: (i) standard linear regression (Linear Regression), (ii) ridge (Ridge), (iii) lasso (Lasso), and (iv) Linear Regression with Polynomial Features (Linear Regression Poly), where we used polynomial features of degree 2
2. Multi-Layer Perceptron: We used variants of multi-layer preceptrons: 3 with different widths (MLP Wide 1, MLP Wide 2, MLP Wide and 3 with different depths (MLP Deep 1, MLP Deep 2, MLP Deep 3)
3. Random Forest
4. Bagging
5. Gradient Boosting
25

---

Published as a conference paper at COLM 2024
6. AdaBoost
7. SVM: We used both single SVM and an SVM paired with Scaler (SVM Scaler)
8. KNN: We used multiple variants of KNN, where vary the number of neighbors, the type of distance used, and the power parameter for the Minkowski metric; We distinguish between them with a vf [
9. Kernel Ridge
10. Spline
We used the sklearn implementation for each model. 12 Similar to the LLM case, we do not tune any hyperparameters. We use the default hyperparameters available in sklearn. We remark that these supervised baselines are very strong, as (1) many of them are the results of algorithms specifically designed for regression (e.g,, Spline), (2) all perform parameter updates, and (3) the default hyperparameters, as set in widely-used statistical packages, have been refined over time to offer a reliable and generally strong performance across variety of scenarios.
D.3 Unsupervised Models
We use three heuristic inspired unsupervised models:
1. Average: Predicts the next value, Yn+1, as the mean of preceding outcomes:
Vn+1 = N 1
2. Last: Uses the most recent observation tuple (xn) for prediction, such that
Mn+1 = 3/2
3. Random: Predicts Vn+1 by randomly selecting from the set of prior observations
(y1....,y.) yn). The final prediction thus yn+1 sample((y1......)
The goal of these unsupervised models is to better put the performance obtained by LLMs into perspective.
E Large Language Models Can Do Linear Regression (Expanded)
We expand the barplots shown in Figure 2 with more models and more datasets. In particular, we show in Figures 10, 11, 12, 13, 14, 15 the performance of the models on six datasets for linear regression. Specifically, we used the following datasets:
1. Regression 1/1, a linear regression with only 1 variable, which is informative
2. Regression 1/2, linear regression with 2 variables, and only informative variable
3. Regression 1/3, linear regression with 3 variables, and only informative variable
4. Regression 2/2, linear regression with 2 variables, both which are informative
5. Regression 2/3, linear regression with 3 variables, and only 2 informative variables
6. Regression 3/3, linear regression with 3 variables, all which are informative
We included the corresponding rank heatmap in Figure 16.
We make the following observations. First, Claude 3 Opus performs among the best for the linear regression case where there is only one informative variable (Regression 1/1, Regression 1/2, Regression 1/3), ranking among top best performing models. The performance drops when there are more informative variables.
Second, we remark that all large language models perform better than all the unsupervised methods all datasets.
1e used sklearn 1.4. 1.4.1.postl.
26

---

Published as a conference paper at COLM 2024
Third, we remark that the large language models display good overall performance. For example, there are 4 LLMs (iee,, Claude 3 Opus, Claude Sonnet, GPT-4, DBRX) which perform better than all 3 variants of KNNN over all the datasets used.
Fourth, there are specific LLMs which always perform better than Gradient Boosting, such as Claude 3 Opus and GPT-4. We remark that DBRX and Code Llama 70B outperform Gradient Boosting in 4 out of 6 datasets.
30
Cauee
Cluce Tooee
oPea
Ganar
Meche sadior
Useaa the
Hal caa
menar
bee Rowerl free drdder
Zane Ritte the Napee
Liber degrestoo
Figure 10: Regression 1/1
UP
bediinal bape vied Hald
Uesaavid Madd
10
Olone O0pe
Glsle Tmup
onia
Genar no
Lseal ros te
HMR out
Represto
Grudenn Assiiaa
Ldi
Bandonn
Dale
Model
Figure 11: Regression 1/2
Ueueevied
onea
Gembi
Metoa Medion
Llmer rot the
Hadd cue
Resor
Cose tana
Dou ciege Ueear iepieatoo
Usver
Figure 12: Regression 1/3
In order to see how the performance of LLMs scale with variables, we additionally experi- mented with Claude Opus on Regression NI 1/10, 3/10, and 5/10 respectively, For example, in NI 5/10, Claude Opus outperforms methods like Gradient Boosting or Random Forest but lags behind methods like Logistic Regression, which is expected given the nature of the datasets. For NI 3/ 10, Claude 3 Opus is outperformed by Random Forest and Gradient Boosting, Overall, we observed the performance to be similar to the performance NI 2/3 from Figure 16.
27

---

Published as a conference paper at COLM 2024
UP
Vod Sinal Sape vieed Hadd
Moce
Oloee
Ohcle
Gange Po
Wier 2378
Leeal nas the
oud
Mose 1d Dale ciom
and plst Reveeal
Gudeen Acciooa
Senill
LHd
Rarcon
Model
Figure 13: Regression 2/2
Ueuuervied
Godd
orea
7ot the
H.dd
Heow
Caie tang
Ueea seginato
Figure 14: Regression 2/3
Glsde touer
Cook
sesial lndios
lanal tha
cama nas
Wost
Resseol
Figure 15: Regression 3/3
28

---

Published as a conference paper at COLM 2024
Cloude 3 Opus
1
3
13
Claude Sonnet
11
19
Claude Holku
t
14
1/
20
GPT-4
11
15
15
Chat GPT
26
25
23
31
Gemini Pro
22
Mistral Medlum
12
21
27
20
Mixtral 8x22B
13
14
24
19
Mixtral 8x7B
19
28
28
23
Mistral 7B
27
Liama2 70B Chat HF
21
30
33
YI 348 Chat
22
23
26
Code Liama 708
20
22
25
21
DBRX
18
19
17
Cohere Command R Plus
16
20
Lirea ear Regression
2
Ridge
25
17
Lasso
24
15
11
Method
MLP Wide
10
MLP Deep
10
Deep
Deep
15
Traditional
Bagging
30
27
Supervised
Gradient 3costing
Regression Poly
Spline
Average
Last
Unsuperveed
Random
1/2
1/3
3/3
MI
Ni
Ni
Dataset
Figure 16: The rank of each model investigated on linear regression datasets, (best viewed color)
29

---

Published as a conference paper at COLM 2024
F Large Language Models Can Do Non-Linear Regression (Expanded)
We expand the barplots shown in Figure 4 with more models and more datasets. In particular, we show in Figures 17, 18, 19, 20, 21, 22, 23, and 24 the performance of the models on the eight datasets for non-linear regression.
Additionally, we include in Figures 25 and 26 the performance of the models on datasets generated by randomly initialized neural networks, similar to the methodology of Garg et al. (2022). We can see that the performance of the LLMs decreases on the datasets created using the neural networks, although it (generally) remains above that of the unsupervised baselines.
Similar to Figure 6, we include the corresponding rankings in Figure 27.
We would like to remark that Claude Opus obtains an average rank of 7.7, the best out of all models we have investigated. The next best is Gradient Boosting, with 8.1. The next best LLM is Claude 3 Sonnet, with 9.3, then GPT-4 with 12.7.
Coule hore
6914
Motot to
moce 73 Uleaa
Lnew Peprostoo
Sita Koda
Credtoal Fevoing
Repping
Scate
kvarge
Unae
Figure 17: Extended results on the Friedman #1 dataset
UM
Tectional Super Model
Unsupanvee Moedl
10-
Caoaee 100
Cbode Some
6n14
Senor
Keee telch
nurral to
canso not that chs thet
tecre ta
Laeoa YOu
Lnear Represtor
Boging Gradin Aebon Mood reeous
Nen Stzte
dan
Unear Regertmnn poly
Kinue
Led
Model
Figure 18: Extended results on the Friedman #2 dataset
0440
Clodle 1090
Tuone
0714
Genor
Nlow medin
coona Not Char or
Cas Usro
Lsea Heponsoo
hdle
Daen Kandam
dayping Godtas Focciny Matast
Jmn Srite
Unaer fopporee Spiee
Gonel Nape
Model
Figure 19: Extended results on the Friedman #3 dataset
30

---

Published as a conference paper at COLM 2024
UU
Todiiial boacreee Poak
Deseavood Nord
Cloie Oria
Cooes lavel
and
Gembrr Po
stooe Malion
Apsnal to
Jane nok Chat
a- Nore to
Cor PM
Reppet froec trising
sie eve
Lnder Reprnation 1o))
Accrel coye
ModN
Figure 20: Extended results on the Original #1 dataset
110-
Giole T0ps
6P14
Gaoted
atinel Haciun
Apstaal 10
Char
Ho To
Lsese Hit
dicce
Liman Regoneni
bide
tride
Coep
Dogp Diwn
Aaping Crostng Actiuaa
gote
Stale
Loser Repetsioon
hope
Spice
Figure 21: Extended results on the Original #2 dataset
1914
Masae Madion
Norel to
Llme Yoa oft
K.d cur
to
Cont Ueeaa
Lieer Pepressoo
Mardinn Reginn Credto
Unsar Sopinatio
Sallo
Figure 22: Extended results on the Original #3 dataset
UM
Fudlinal beaeriee Pork!
Ueseunsel Mord
Cluil Oyys
4914
Gemaar
Mooe Redion
Aoniaa
clal
Aoacel 70
PM
oiice
Reproomo
Aepping Cropnig ateias
Side
Respesdooo caly
Mode
Figure 23: Extended results on the Original #4 dataset
Craal 10pk
C bome
chat
1894
Lnew Repreter
den
Deso
Dapping Fouttey
Averet
Unear Hopressio
Figure 24: Extended results on the Original #5 dataset
31

---

Published as a conference paper at COLM 2024
LU
Taciiionll Super Model
Olale 1000
Obcle Soont
4P14
Dtnocr
Mijoa Melion
hot
Represton
eoper Gadden Kodu Acaani
5en
Unai fegeormn Pi
600
Model
Figure 25: Results random regression dataset generated using randomly initialized Neural Network
UM
Teditonal Super Model
Clole Opm
Dbude Samet
6P-4
tenoi
Mieue Melon
Hureal to
clens not thar
You
Leew Reprosto
Addy
6n Eacrel Kime
Unoar
Model
Figure 26: Results random regression dataset generated using randomly initialized Transformer Encoder Block
32

---

Published as a conference paper at COLM 2024
Claude 3 Opus
1
1
2
2
26
15
19
Claude 3 Sonnet
13
2
2
2
3
3
23
21
22
GPT-4
5
22
23
Mixtral 8x22B
11
5
23
24
LLM
Mixtral 8x7B 29
21
13
11
25
21
15
22
26
27
Mistral 7B
32
31
21
18
30
31
33
27
33
33
Llama2 708 Chat
19
19
32
34
Code Liama
DORK
100
24
25
Lirear Regression
11
13
18
29
18
19
32
13
Ridge
15
16
30
16
18
12
Lasso
27
HLP Wide
28
23
20
Method
MLP Wide
27
35
24
22
10
32
MLP Deep
29
36
26
12
5
18
5
16
MLP Deep
26
33
22
7
28
MLP
21
Traditional
Bagging
9
11
10
Supervised
Methods
Gradient 3costing
10
4
9
8
5
AdaBoost
20
14
17
11
15
12
19
14
14
SVM
22
32
32
32
19
SVM Scaler
15
KNN v2
23
6
27
24
11
KNN v3
15
24
26
3
28
26
13
16
15
Kerrel Ridge
10
19
34
27
26
2.81
36
35
20
Linear Regression Poly
28
13
Spline
20
Average
34
Unsupervised
Last
35
32
36
35
35
34
36
Methods
Random
35
36
31
35
36
36
35
35
35
Friedmen
Priedman
Friedman 3
Original
Original
Original
Original
Original
Simple NW
Tranesormmrr
Dacaset
Figure 27: Rank of each model investigated on the non-linear regression datasets. (best viewed
color)
33

---

Published as a conference paper at COLM 2024
G Average Model Ranks
To provide a comprehensive overview of model performance across a diverse array of datasets, this section aggregates the average ranks obtained by each model. In this sec- tion we show the average ranks obtained by each model across: (1) all linear regression datasets (Linear), (2) all original benchmarking datasets introduced by us (Original), (3) all benchmarking datasets introduced by Friedman (Friedman), (4) all neural network induced datasets (NN), (5) all non-linear datasets (Non-Linear), (6) all datasets (Overall).
We show our results in Table 7. We divide the table into three blocks, seakkkste by horizontal lines, corresponding to the results for (1) LLMs (e.g,, GPT-4), (2) Traditional Supervised Methods (e.g., Gradient Boosting), and (3) Unsupervised Baselines (e.g., Average). We remark that LLMs obtain, overall, a strong performance. For example, Claude Opus ranks second overall, outperforming methods such as Gradient Boosting, KNN, or multi-layer perceptrons. Overall, this strong performance is present in both private and open models. For example, DBRX and Mixtral 8x22B achieve average ranks that surpass conventional methods including AdaBoost, KNN, or Random Forests.
Model
Linear
Original
Friedman
NN
Non-Linear
Overall
Claude 3 Opus
7.67 5.89
7.2 4: 12.76
4,00 3.61
17.00 +: 2,83
8.2 1: 9.99
8.00 1: 8.45
Claude Sonnet
13.33 4.89
7 10.41
5.67 6.35
21.5 0.71
9.7 9.82
11.06 + 8.31
Claude 3 Haiku
13.33 7.69
18.00
1 8.63
19.00 +: 10.44
29.5 0.71
20.6 1: 8.92
17.88 1 8.98
GPT-4
12,83 4:07
8.02
11.33 - 11.85
0.71
13.6
13.31 7.
GPT-4 (20240409)
13.5
12.00
99.40
11.33
1633+10..21 10.21
25.00
+ 1.41
9.74
14.06 + 7.83
Chat GPT
2.35
30.6 00:2
24.67 12.9
39.00
2.83
30.5 10.23
30.12 8.17
Davinci 002
26.17 6.27
2188
5..03
36.00
35.5 - 1.11
8.77
25.5 7.72
Rabbage 002
40.17
25.8 6.06
0.58
35.00
Gemini Pro
16.5
10.92
2.09
2.75
19.3
18.25 + 8.49
Medium
23.17 6.71
20.6
2..33
0.7
22.38
Mixtral 8x22B
20.5 5.65
12.4 1:551
16.67 1: 11.72
23.5 1 0.71
15.9 1: 7.71
17.62 1 7.18
Mixtral
22.4 6.66
26.00
Mistral 7Bv2
35.00 1:000::: 2.53
2.53
31.6 1 4.72
7.66
43.5 1: 2.12
3244 1: 7.96
33.38 1 6.47
Mistral 7B
37.33
4.97
7.577
41.5 0.71
5.49
DBRX
18.67 1.51
15,00 8.0
15.67
1: 13.43
25.00 0.0
17.2 1: 9.25
17.75 1 7.25
Cohere Command Plus
25.00
26.4 8.79
24.67
Llama2 70B Chat HF
3198
7:22
28.67 3.511
40.5 + 3.54
33.1 + 6.79
32.62 + 5.76
Code Llama 70B
2.88
23.67
11.59
7.27
Y1 1134B Chat
28.00 13.00:22 2.76
2.76
21.2 9.36
25.00 3.46
33.00 + 2.83
24.7 1 8.04
25.94 + 6.64
Gradient Boosting
7.41
7.33
8.67 5.51
8.5
9.1 1:
15.00
AdaBoost
29.00 8.63
17.6222..2
16.00
1 4.58
14.00 - 0.0
16.4 - 4.03
21.12 8.62
Bagging
6.48
14.6
0.71
13.00
KNN
32.67 5.05
13.62100.21
27.33
8.96
1100 + 1.41
22.2 10.11
26.12 9.86
KNN v2
29.67 + 4.84
18.2
8.39
10.00
18.8 11.07
22.88
10.33
KNN v3
28.67 5.28
1 14.62
27.33
10.79
15.5 - 0.71
20.4 12.04
23.5
10.65
Linear Regression
25.8 13.85
16.00
10.00
19.7
12.75
13.04
Lasso
13.5 7.66
24.8 14.86
15.7 30.67
33.5 4 0.71
28.3 + 12.94
22.75
13.22
Ridge
15.38
25.00 14.47
16.33
8.5
12.1
17.69
10.44
Linear Regression + Poly
2.5 0.84
16.00
15.87
3.00 2.65
8.00 + 7.07
10.5
+ 12.49
7.5
10.48
ML.P Deep
5.67 3.39
16.00
12.88
31.33
20.11
11.5 9.19
19.7
15.51
14.44
14.05
MI.P Deep 2
8.83 4.17
18.8 + 13.37
28.67
+ 20.98
6:00 0.0
19.2
+ 15.68
15.31
+ 13.42
ML.P Deep
11.00 3.35
16.2 14.77
29.00
19.16
1.71
18.3
15.66
15.56
1281
MLP Wide
4.67 1.75
18.00 + 142
21.67
: 14.29
9.
17.3 1 12.95
12.56
11.9
ML.P Wide
7,83 1.83
25,2 11.3
25.00
1. 19.16
3.00 1.41
20.7 1 15.02
15.88
13.34
MLP Wide
6.33 1.21
4
27.67
23.86
1.00 : 0.0
20.7 + 17.25
15.31
: 15.19
Random Forest
3.05
17.2 9.58
14.33
6.881
0.0
16.3 7.22
22.00 + 10.3
Kernel Ridge
14.1716699
30.00
15.72
26.33
1:33::10 18.23
34.00 +: 19.8
29.7 1 15.33
23,88 + 14.74
SVM
+..8
29.67
+ 12.01
11.00
+ 11.31
27.5 14.77
34.00 14.4
SVM Scaler
45.5 1.76
33.4 15.39
24.33
16.65
12.00
11.31
26.4
15.99
33.56
15.68
Spline
14.00 3.95
29.00 2 6.75
10.33 7.77
19.5 2.11
21.5 = 10.38
18.69 :: 9.16
Average
48.5 1.38
34.2 17.43
43.00
1.73
33.5 1: 0.71
36.7 1 12.44
41.12 1 11.32
Random
51.17 0.98
44.8 2.42
46.67
3.21
4.00
45.8 +: 2.94
47.81 1: 3.56
Last
51.33 -: 0.52
43
47.00
17.00:2::66
47.00 1.41
46.00 + 3.33
48.00 3.72
Table 7: We show the average rank of each model we investigated, across multiple types of datasets. We divide the table into three blocks, separated by horizontal lines, corresponding to the results for (1) large language models (e.g., GPT-4), (2) Traditional Supervised Methods (e.g., Gradient Boosting), and (3) Unsupervised Baselines (e-g., Average). Overall, the large language models (LLMss obtain strong performance. For example, Claude 3 Opus ranks second overall, outperforming very strong methods like Gradient Boosting or multi-layer perceptron. This strong performance is present in both private (e.g., Claude 3 Opus, GPT-4) and open-weights models (e.g., DBRX, Mixtral 8x22B).
34

---

Published as a conference paper at COLM 2024
H How Fast Do Large Language Models Adapt? (Expanded)
We expand Table 1 to include more models. We show the corresponding results in Table 8.
Additionally, we include curve fit plots for models. To keep the number of plots to manageable amount, we selected subset of the models as follows. We selected Claude Opus and GPT-4, as they are the flagship closed-source models. We selected Yi 34B Chat for the open-weights model. Lastly, we selected Gradient Boosting, Linear Regression, and Linear Regression Poly. We present the corresponding plots in Figure 28 and 29. Since in these experiments we vary the number of in-context examples starting from 1, we could not include variants of KNN that uses more than one neigbhors for their prediction. We included KNN v4 which uses only one neighbor. Additionally, we included KNN v5, where we use a small number of neighbors when the amount of data is small, then gradually increase it. We can that their performance is much worse than that of the LLMs, suggesting that the LLMs are doing something more than what KNN do.
Model Dataset
Friedman
Friedman
Friediman
Original
Original
Regression NI 1/3
Regression 2/2
Opus
sprt
sqrt
log
sqri
log
log
Claude Sonnet
linear
log.
linear
log
log.
log
sqri
GPT-4
linear
sqri
log
sqri
Chat
sqrt
sqrt
sqrt
Gemini Pro
linear
sqrt
linear
log
sqrt
sqpt
sqrt
Mistral Medium
linear
linear
linear
sqri
sqrtt
LJama2 Chat
linear
linear
sqrt
sqri
Code Llama 20B
linear
sgrt
linear
sqrt
sqel
linear
sqrl
Mistral 7B
linear
linear
linear
sqrt
linear
linear
linear
Mixtral 8x7B
linear
linear
linear
sqrt
linear
linear
linear
sqrt
linear
sqri
sqri
sqri
sqrt
DBRX
linear
log
linear
log
sqrt
sqrt
sqrt
AdlaBocst
lineat
linear
sqrt
sqrt
Bagging
sqpt
log
sqrt
sqrt
sqrt
Gradient Boosting
sqrt
sqrt
linear
log
sqrt
log
sqrt
Linear Regression
lineat
linear
linear
linear
Linear
log
log
log
Linear Regression
Poly
sqet
log
log
linear
log
log
log
linear
linear
lincar
lnear
squt
log
log
MI.P Deep
linear
linear
linear
linear
sqrt
log
log
MLP Deep
linear
linear
sqrt
linear
sqet
log
log
MLP Wide
linear
linear
linear
linea
sqrt
log
log
MLP Wide
sqrt
linear
linear
linear
linear
log
ML.
sqrt
log
log
Random Forest
linear
sqri
linear
sqrt
sqrl
sqri
linear
Ridge
linear
sqrt
linear
linear
linear
log
SVM
linear
sqrt
linear
Kernel Ridge
linear
linear
linear
linear
linear
log
log
SVM Scaler
linear
linear
linear
linear
linear
linear
KNN
linear
linear
linear
log
linear
sqri
sqri
KNN
linear
linear
linear
log
linear
sgrt
sqrt
Average
linear
linear
linear
linear
linear
linear
linear
Last
linear
linear
linear
linear
linear
linear
linear
Table 8: We show which curve-fit obtained the highest R2 for multiple models and datasets, expanding Table 1. The slower the growth of the function, the better (i.e., log > sqri >
linear). (best viewed in color)
35

---

Published as a conference paper at COLM 2024
2011
Lag
leg
(a) Claude Opus, Friedman #1
(b) GPT-4 Friedman #1
(c) Yi 34B, Friedman #1
(d) Claude Opus, Friedman
(e) GPT-4, Friedman
(f) 348, Friedman
(g) Clauud Opus, Friedman
(h) GPT-4 Friedman
34B, Friedman #3
i Claude Opus, Original #1
(k) GIT-4, Original #1
(I) Yi 34B, Original #1
pach
(m) Claude 3 Opus, Original #2
(n) GPT-4, Original #2
(o) Yi 34B, Original #2
(p) Claude 3 Opus, Regression 1/3
(q) GPT-4, Regression 1/3
(T) Yi 34B, Regression 1/3
(s) Claude 3 Opus, Regression 2/2
(t) GPT-4, Regression 2/2
(u) Yi 34B, Regression 2/2
Figure 28: Curve Fits for Claude 3 Opus, GPT-4, and Yi 34B on seven (linear and non- -in-linear) datasets.
36

---

Published as a conference paper at COLM 2024
(a1 leg
Log
(a) LR, Friedman #1
(b) GB, Friedman #1
(c) LR Poly, Friedman #1
(d) LR, Friedman
(e) GB, Friedman
(f) LR + Poly, Friedman N2
(g) LR, Friedman
(h) GB3 Friedman
LR Poly, Friedman
i LR, Original #1
(k) GB, Original #1
(I) LR Poly, Original 41
(m) L.R, Original #2
(n) CB, Original N2
(o) I.R + Poly, Original #2
(p) LR, Regression 1/3
(q) CB, Regression 1/3
(T) LR Poly, Regression 1/3
(s) LR, Regression 2/2
(t) GB, Regression 2/2
(u) LR + Poly, Regression 2/2
Figure 29: Curve Fits for Linear Regression (LR), Gradient Boosting (GB), and Linear Regression with Polynomial Features (LR Poly) on seven (linear and non-linear) datasets.
37

---

Published as a conference paper at COLM 2024
I Performance on Real-World Datasets
In the following, we include the performance of GPT-4 and Claude 3 Opus on real-world datasets. Nevertheless, our aim in this work has not been to suggest that LLMs should replace (at least currently) traditional regression methods. We evaluate on the following datasets:
1. Liver Disorders (UCI id=60 (UCI, 1990)
2. Real Estate Valuation (UCI id=477 (Yeh, 2018))
3. Diabetes (sklearn (Efron et al., 2004))
4. Servo (UCI id=87 (Ulrich, 1993))
5. Movies (UCI id=424 (Ahmed, 2017))
We evaluated the following models:
LLMs: GPT-4, Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku;
Traditional Supervised Methods: AdaBoost, Bagging, Gradient Boosting, KNN, KNN v2, KNN v3, Kernel Ridge, Lasso, Linear Regression, Linear Regression + Poly, MLP Deep 1, MLP Deep 2, MLL Deep 3, MLP Wide 1, MLP Wide 2, MLP Wide 3, Random Forest, Ridge, SVM, SVM + Scaler, Spline;
Unsupervised Methods:
Average, Last, Random.
We show our results in Tables 9, 10, 11, 12, and 13. In order to keep the tables easy to read, we selected: (i) two LLMs, Claude 3 Opus and GPT-4; (ii) three traditional supervised methods, Gradient Boosting, Random Forest, and Linear Regression with Polynomial Features (LR + Poly); and (iii) three unsupervised methods, Average, Last, and Random. We show both the mean absolute error (MAE) and the rank. The rank is shown against all the methods ran. For example, on Liver Disorders (Table 9), SVM ranks first.
Model
MAE
Rank
GPT-4
2.599
6
Gradient Boosting
1.57
8
Random Forest
2.62.11.57
12
Linear Regression + Poly
2.67
19
Average
2.79+11.0
22
Claude 3 Opus
3.04
24
Random
3.23
26
Last
3.46
27
Table Results on the Liver Disorders dataset, sorted by the MAE (L)
14 Altthough we can foresee interesting advantages for L.Mss, such allowing for natural language description what each feature represents. //tttss//scckitttt http://ww..a.arree/////////////s htw.o//lllesggeertttts//lleeeaae datasets. foad_diabete. html
38

---

Published as a conference paper at COLM 2024
Model
MAE
Rank
Gradient Boosting
4.30
Claude 3 Opus
5.08
GPT-4
5.25+45.88
Linear Regression Poly
5.38
Random Forest
5.72.15.32
10
Average Random
11.12+7..9
26
14.39
27
Last
15.27+11.66
28
Table 10: Results on the Real Estate Valuation dataset, sorted by the MAE (L)
Model
MAE
Rank
Random Forest
41.31+277.8
Gradient Boosting
47.95:128.36
GPT-4
50.26:1:4.255
17
Claude Opus
54.1331.9977
Linear Regression + Poly
55.84::52.87
20
Average Random
68.02:137.57
94.34+63.17
26
Last
103.96=:9.557
27
Table 11: Results the Diabetes dataset, sorted by the MAE (1)
Model
MAE
Rank
Gradient Boosting
0.25+10.29
2
Claude 3 Opus
0.27100.66
4
GPT-4
0.44
10
Random Forest
0.45
11
Linear Regression Poly
0.62:40.38
13
Average
1.43
24
Last
1.61+1.56
26
Random
1.77+1..74 27
Table Results on the Servo dataset, sorted by the MAE (L)
Model
MAE (1e+07)
Rank
Random Forest
3.02+
Gradient Boosting
3.03+:2.51
3
Claude Opus
4.51+6.20
GPT-4
5.4817..2
20
Average
6.20
21
Linear Regression + Poly
6.40=7.10
22
Random
8.17+1003
23
Last
8.26
24
Table 13: Results on the Movies dataset (scaled by le+07), sorted by the MAE (L)
J Claude Performance
Following the (perhaps surprisingly) strong performance of the Claude family of large language models on various regression tasks (e.g., Figure 27, when averaging the ranks of each model over each dataset, Claude Opus performs the best), we provide results with additional models from the Claude family, namely: Claude 1.2, Claude 2.0, Claude
39

---

Published as a conference paper at COLM 2024
2.1, and Claude 3 Haiku. We include rank heatmap for all the models from the Claude family currently available in Figure 30, For comparison, we also included the corresponding performance of two strong models with open-weights: DBRX and Mixtral 8x7B. Claude 2.0 and Claude 2.1 were sometimes generating invalid outputs (e.g. "I apologize, upon reflection I do not feel comfortable providing output values without context. Could we have a constructive discussion about the meaning and implications of this exercise?"). Therefore, we omit those problematic configurations. For all the other cases, the average performance the result of least runs. We note that the performance of Claude models much better than that of older models.
111
24
34
35
20
38
20
13
22
30
21
12
104
14
25
25
Pretean
22
Cheake Server
Cande Faleo
Lovver Lnpeeeeeeess
Sopeesion
Metel
Figure 30: The rank of each model from the Claude family currently available, For com- parison, we also included the ranks of two (strong) models with open-weights: DBRX and Mixtral 8x7B. (best viewed in color)
K Costs
We estimate the total cost for all our experiments to under $1200. We spent approximately $300 on OpenRouter. We spent approximately $600 on OpenAl. The cost for OpenAl is higher because we used it in our preliminary experiments. Additionally, the preliminary experiments used gpt-4, which is more expensive gpt-4-01255previee We switched to 11144-0025-ppeiiiee after we added the additional text to the prompt, instructing the model to only output their best estimate." 17
L LLMs Justifying Their Prediction
Without adding the prefix instruction text (see prompt in Appendix D.1.1), which instructed the models to output only its best estimate, some LLMs (e.g,, Claude 3 Opus) started to provide explanations, in an attempt to justify their prediction." 18 Analyzing these "explana- tions" revealed that there is discrepancy between their explanation and their prediction. For example, for a non-linear regression problem, Claude 3 Opus suggests to train a Linear Regression model and gives the code to do Then it elaborates inference and gives the output. However, manually running the code suggested by the model results in a very different output. We provide some examples in Figures 31, 32, 33, and 34. We describe each one below.
how to use it for
16These invalid outputs are specific to Claude 2.0 and Claude 2.1. For example, for Claude Opus, there exist only instances where it does not generate a valid output, out of a total of over 1000 runs. 1e did not need this additional instruction in our initial experiments. We added when expanded the number LLMs used, as some of them (e-g., Claude 3) would provide justifications before giving the final output. All models use the same prompt.
1e hypothesize that this is because their system prompt instructs them provide explanations.
40

---

Published as a conference paper at COLM 2024
In Figure 31, Claude 3 correctly identifies that the output is generated by multiplying the input with a constant. Then, calculates the constant and gives the final output.
In Figure 32, can that Claude 3 first calculates the mean of each feature. Manually inspecting the true values, they are: Mean Feature O: 51.3180, Mean Feature 1: 846.6326, Mean Feature 2: 0.4764, Mean Feature 3: 5.2438. We remark that the values given by Claude are surprisingly close. Then, Mean Output: 374.04. Then, Claude 3 calculate the covariance between each feature and the output. These estimates are much worse. For example, Cov(Feature 0, Output) is actually 2516.3683, not 8958.8469. Next, Claude gives the variance of each feature. These values are close. For example, Var(Feature 0) is 729.420751 and Claude 3 generates 729.99999999 9052. Then, the model calculates the coefficients. These calculations are close to their true value, except for bo. Lastly, the model gives the final formula to calculate the output, which is (according to the model): 227.4744 + 25.9730 42.54 + 1.2777 851.9 931.93 + 1648.5958 0.51 + 229.2408 6.26 436.5981, However, this calculation is wrong. The output of that equation is 4070.70. However, we would like to remark that what the model generated (wrongly, from mathematical point of view), is actually much closer to the true value of 434.54. In other words, the explanation offered by the model was not faithful.
that Claude suggests that there is strong linear relationship (please refer to Figure 5 for a plot of the data). Then, Claude 3 fits a linear regression 0.9102 and 12.1615. However, manually fitting a linear regression model on the corresponding data yields the following values 0.97 and 2.94. Then, Claude 3 calculates the final output. The calculation is correct, however it is far off from the true value of: 30.86. We would like to remark that instructing the model give its best estimate without any additional information gives much better prediction: 30.91.
In Figure 33, we
y = y= mx + and gives the approximate values:
In Figure 34, the solution generated by Claude 3 involves calculating the nearest neighbor. The problem in this approach is that the dataset given in-context contain only 50 examples, while the solution generated Claude involves taking the examples 50 and 54, which are non-existent.
All all, we remark that the explanations provided by the model are not always faithful. We also remark that the predictions of the model in two cases: (i) when it outputs explanation and (ii) when it does not output an explanation can vary.
41

---

Published as a conference paper at COLM 2024
To find the output for given Feature value, need to determine the relationship between Feature and the Output. Looking at the provided data points, it appears that the Output is approximately equal to Feature 0 multiplied by constant value.
Let's calculate this constant value using few data points:
For Feature -2.06, Output -81.93
Constant = Output / Feature 0 = -81.93 -2.06 ~ 39.77
For Feature 0= 1.46, Output 58.14 Constant = Output / Feature 0 58.14 1.46 39.82
The constant values are very close, so we can assume that the relationship is:
Output = Feature 39.8 (rounded to one decimal place)
Now, for Feature 0 1.62, we can calculate the Output:
Output 1.62 39.8 64.48
Therefore, when Feature 0 1.62, the Output should be approximately 64.48.
Figure 31: Claude 3 Opus justifying its output on Regression 1/1/1. In this dataset, the final output constructed by multiplying the input feature with a constant value. Claude 3 correctly identified the constant. The answer is very close to the correct one (differences might appear due to rounding).
42

---

Published as a conference paper at COLM 2024
To predict the output value, I'll use a linear regression model with the given features. The model is:
Output b0 Feeatuue + F2* b3* + Feature
First, I'll calculate the mean of each feature and the output:
Mean Feature 0: 51.7054
Mean Feature 1: 836.5202
Mean Feature 2: 0.4844
Mean Feature 3: 5.3154
Mean Output: 385.2352
Next, I'll calculate the covariance between each feature and the output:
Cov(Feature 0, Output): 18958.848 8469
Cov(Feature 1, Output): 262416.1385
Cov(Feature 2, Output): 138.6469
Cov(Feature 3, Output): 1117.1448
And the variance of each feature:
Var(Feature 0): 729. 299.9052
Var(Feature 1): 205416.6328
Var(Feature 2): 0.0841
Var(Feature 3): 8.6469
The coefficients are:
b1 Cov(Feature 0, Output) / Var(Feature 0) 25.9730
b2 Cov(Feature 1, Output) Var(Feature 1) 1.2777
b3 Cov(Feature 2, Output) Var(Feature 2) 1648.5958
b4 Cov(Feature 3, Output) /
Var(Feature 3) 129.2408
b0 Mean(Output) - b1 Mean(Feaaurre 0) beeaeeaeeea 1) -3) Meaneaaaee 2) b="MMan(Featuure 2744
So, the linear regression model is:
Output 222..4744 + Feetrr0 + 1.2777
1648.5958
ture2 110.2408*Featuun
Plugging in the given feature values:
Output = 2227.4744 25.9730042.54 1. 1.2777 =33..5981 Therefore, the predicted output value is 436.60.
+ 1648.5958*0.51 229.2408
Figure 32: Claude 3 Opus justifying its output on Friedman #2. Except for covariance and bo, the calculations made by Claude are close to their true values (e-g., Cov(Feature 0, Output) is 2516.3683, not 8958.8400 8469). The model gives the final formula to calculate the output: 227 227.4744 + 25,9730 * 42.54 + 1.2777 * 851.93 + 1648.5958 0.51 229.2408 6.26 = 436.5981. However, this calculation is wrong. The output of that equation is 4070.70. However, we would like to remark that what the model generated (wrongly, from a mathematical point of view), is actually much closer to the true value of 434.54. In other words, the explanation offered by the model was not faithful, even though the final prediction was good.
43

---

Published as a conference paper at COLM 2024
To find the relationship between the input Feature and the Output, let's plot the data points and see if there's clear pattern trend.
Plotting the data, it appears there is strong linear relationship between Feature and Output. As Feature increases, the Output also tends to increase fairly straight line.
To quantify this linear relationship, we calculate the line of best fit ing linear regression. The equation for line is typically written as:
y mx + b
where: y is the Output value
m is the slope of the line x is the Feature value
b is the y-intercept (the Output value when Feature 0 is 0)
Using the given data points, the linear regression equation comes out to ap- proximately:
Output 0.9102 Feature 0 + 12.1615
So for the given Feature 0 value of 31.34, plugging that into the equation:
Output 0.9102 31.34 + 12.1615 =88.53 12.1615
40.69
Therefore, based on the strong linear trend in the data, the predicted Output value corresponding to Feature value of 31.34 would be around 40.69.
Figure 33: Claude 3 Opus justifying its output on Original #1. In this dataset, the final
output is constructed as follows: f(x) = 10x sin(5xx) + cos(xxx). The solution proceeds by fitting a linear regression and giving the approximate values. However, manually fitting linear regression model yields the following values: m = 0.97 and b 2.94. Then, the model proceeds by calculating the final output. The calculation is correct, but the final value is far off from the true value, which is 30.86. We would like to remark that instructing the model to give its best estimate without any additional information or explanation gives much better prediction: 30.91.
44

---

Published as a conference paper at COLM 2024
To predict the output for the given input features, I'll calculate the average output for the two nearest neighbors the training data.
Input: Feature O: 0.7, Feature -2.03
Calculating Fuclidean distances to all training examples:
(1 2.27
2. - 3.45
49. /(0.7 - (--38)))+4(--00300000) 2.35
50.
(-277 0.62
51. V(0.7 - (0.7-(-0.46))3 (-044)) 0.35) 2.76
52. V(0.7 - (-0.7))) 0.01) 2.64
53. <(0.7 - (-2.03 (-0.22) 1.88
54. - - (- (-1.38)) 1.41
The two nearest neighbors are examples 50 and 54, with distances 0.62 and 1.41.
Example 50 output: 31.76 Example 54 output: -49.65
Predicted output (31.76 + (-49.65)) / -8.945
So for input features 0.7 and -2.03, the predicted output using 2-nearest neighbors is -8.945.
Claude 3 Opus justifying its output on Regression NI 2/2, linear regression dataset with two variables, where both are informative. However, the dataset given as in-context exemplars Claude contains only 50 data points, while the solution generated by Claude involves taking the examples 50 and 54. Those examples are non-existent.
Figure 34:
45

---

Published as a conference paper at COLM 2024
M Beyond Numerical Regression
Our investigation has centered on conventional regression tasks characterized by inputs and outputs represented as numerical values. However, the performance on these tasks might be influenced by the quality of numerical token embeddings, which can serve confounding factor Razeghi et al. (2022). To address this and broaden our analysis, we shift our focus to datasets generated following the methodology outlined in Section 2.1.3. This allows us to evaluate the models' capabilities in contexts where inputs are symbolic rather than numerical.
Specifically, we define an input vocabulary V, consisting of a small number of (random) symbols. 19 Each symbol is randomly assigned number (1,...,6)) 26). We map the symbols to numerical value by sampling weight vector wo E R and doing dot product between and the corresponding values of each symbol. We present our results Figure 8. The models.
large language model display, once again, performance superior to that of unsupervised
N Effects of Rounding
Due to computational budgets and context limits, 20 we rounded both the input and the output to two decimals. To validate that our conclusions are not an artifact of the rounding mechanism, we re-ran GPT-4 on Friedman #2 and Friedman #3, rounding to five decimals. We selected GPT-4 because obtained overall strong results and offers good latency. We selected Friedman #2 and Friedman #3 because LLMs obtained generally good performance. We also ran the traditional supervised methods. We include the updated results in Figure 35. Comparing it with Figure 27, can that the performance of GPT-4 remains strong. This time even outperforms Gradient Boosting on Friedman #3 and ranks first. The performance Friedman #3 is only under Linear Regression Poly, similar to Figure 27. All in all, the strong performance we observed is unlikely to be just an artifact of rounding.
Telters/Soppr Mettoes
Ura Sowveoovonl
Meedmmn
15
13
14
32
26
10
20
27
13
23
22
20
20
24
(eddmss
15
23
10
22
20
25
12
10
MaARoost
Bngging
condent Boristog
Liiea Rxgresztio
Lasse
Llovge deggn Repredatee Paay
Hur Dessp
Ser thepp
steer
atr siide
lel spila
Randioo Porest
Rodger
aott Bosder
Spiee
Random
cernet
Figure 35: The ranks of the models when rounding five decimals. The performance of GPT-4 remains strong, ranking above many traditional supervised methods. This is in line with our previous observations, where we rounded to two decimals. (best viewed in color)
O Is It Just Better KNN?
We can see from Appendix and that the performance of certain LLMs is almost always better than that of all three variants of KNN presented. For example, in the case of linear regression, Claude 3 Opus, Claude Sonet and GPT-4 always perform better than the three variants of KNN used. Similar, in the case of non-linear regression, only for Simple NN and Transformer 1 is any of the KNN variants outperforming Claude Opus. Moreover, from Table 8, which records the best curve fit over the cumulative regret, can that both variants KNN perform worse than Claude 3 Opus or GPT-4.
We used 5 symbols, in the form of letters (e.g., 8) For example, Ilama2 context size is only 4096.
46

---

Published as a conference paper at COLM 2024
Hisinas
Mdedmss
Dutuve
222
10
Original
8
79
25
Orgina
cnet
deeatt Oeen
closde Sooree
Gade Laena rel
Uamid Gadl
Natrral
Met chet
Eent vep
E00 vdd
kent uvg
Hastral
Nethod
Figure 36: Comparison between the ranks of LLMs and the ranks of best KNN on each
dataset. (best viewed in color)
To further investigate the extent to which what LLMs are internally implementing to do regression is variant of KNN, we compare the performance of the LLMs against total of 70 KNNs in a setting similar to that from Section and 4: we randomly sample a dataset of size 50, which is given to both LLMs and KNNs and we ask them to predict the output corresponding to a testing data point. We repeat each experiment 100 times, with different random seeds. We describe what KNNs we considered in Listing 2.
Listing 2: The python code to create the KNN configurations
for n in [1, 2, 3, 5, 7, 9, 11]:
2
for w in [' I'uniform'
'distance '1:
3
for in 10.25, 0.5, 1, 1.5, 21:
4
yield KNeighborsRegresor ( meighborss=: weights=w, p=p)
We summarize our results in Figure 36. To keep the plot comprehensible, we chose the best-performing KNN configuration for each dataset. However, the ranking considers the performance of all models. We draw the following conclusions.
First, we remark that except on Friedman #1, for every other dataset, the top best per- forming models are all LLMs. In other words, both closed-source models (e.g., Claude 3 Opus, GPT-4) and ween-weights models (e.g,, DBRX, Mixtral) outperform all KNN models all the datasets except Friedman #1. This suggests that the mechanism implemented by in-context learning might be something more complex than KNN.
Last, we remark that for Friedman #1, only Claude Opus and Claude 3 Sonnet outperform the KNN variants. Moreover, Claude Opus and Claude Sonnet outperform all KNN variants experimented with all datasets.
P Could It Be Just Contamination?
The very large datasets that are used to train contemporary large language models (LLMs) raise concerns about potential contamination (Sainz et al., 2023; Golchin Surdeanu, 2024). In our study, we have attempted to mitigate this as follows. First, we used many different random seeds. However, this does not nullify the risk that the LLM has seen similar data (e.g., data from Friedman #1, but with other random seeds). To mitigate this, explored the performance of the models on regression functions of our own creation. This makes unlikely that the model has seen data coming from the exact same function. For example,
47

---

Published as a conference paper at COLM 2024
LUM
Vedional ieprenlrrae
Grspevted Nidd
Maind
Say
Rope
Setur Sovnng nodioo
Unea Rage
Ue
tep
see
Rodel
Figure 37: Falcon performance on the Regression NI 1/1 #2 dataset
across all our newly introduced datasets (e-g,, Original #1, Original #2, Original #3, Original #4, Original #5), Claude 3 Opus obtains the highest average rank of 6.4. Second place is Clade 3 Sonnet, with an average rank of 6.8, then Gradient Boosting with 8.4. Furthermore, our empirical evidence of consistent high performance across diverse array of LLMs.
To further analyze the data contamination issue, we perform two additional experiments. We provide results with Falcon, an LLM whose training data is publicly available. Second, perform experiment similar to the approach proposed in Golchin & Surdeanu (2024), where we compare the performance of LLMs with and without knowing the dataset where the data comes from.
P.1 LLMs With Known Training Data
In this section expand our analysis to include Falcon 40B and Falcon 40B Instruct, comparing their performance with both traditional statistical methods and other LLMs. To keep the figures comprehensible, GPT, Mixtral 8x7B, and Mistral 7B.
added only the following LLMs: Claude 3 Opus, Chat
We remark that the Falcon LLM team has released their training data, 21 offering further insights into how the training environments of contemporary LLMs can result into LLMs being capable of regression.
Due the context size limitations of Falcon, 22 we only evaluated it on the linear regression datasets and on the Original #1 dataset. The other datasets have larger number of input variables (e.g., Friedman has input variables) and we could not fit 50 in-context examples. We show our results on four datasets, Regression NI 1/1, Regression NI 1/2, Regression 2/2, Original in Figures 37, 38, 39 and 40. Additionally, we include the corresponding rank heatmap in Figure 41.
We make the following observations. First, Falcon 40B outperforms our unsupervised baselines. Second, Falcon 40B outperforms Gradient Boosting and Random Forests Regression 1/1.
Overall, Falcon 40B displays, well, the capability of doing regression when given in- context examples, albeit to a smaller degree than when compared to more powerful (and newer) models.
21 hhttps://hugging: http////aaaeetttttttttt////ieeie wtt.//aateee/tiiaaaaaaaoooreeeee 22The context size of Falcon is 2048.
48

---

Published as a conference paper at COLM 2024
Unagenteed Nadel
Homeo
Uneo vepeisinn
Gwep Hauless
syn Steee
Levver Bugge
Trise
Unwe fagi Raguesser
Figure 38: Falcon performance on the Regression NI 1/2 dataset
Unsipentted Nedel
Iotret
Mcchel
Ledar tepresio
Owep
See Stow
Lecill Uneee Fog begiessoe
Figure 39: Falcon performance on the Regression NI 2/2 dataset
Unugenited Nodel
Naize botuus
Uses Aipresio
besa
Seet?
Resbos hoadl
Bogin
andons
Vye Soat
bepustits
oio di
Figure 40: Falcon performance on the Original #1 dataset
Supersised Mathods
Unnaprsiseed Methods
2/1
10
11
20
17
18
19
15
15
12
13
?
21
14
23
22
24
26
25
19
323
211
21
15
1.5
19
1/
18
11
10
/
6
16
12
22
25
24
2b
25
Dataset
2/3
20
21
15
14
13
19
17
9
11
10
30
22
23
24
2:
20
Original 1
2
17
23
14
1.5
16
12
1.1
1.5
3.5
1.4
20
23
77
10
24
26
Pakcan 408 Instruct
Falcon 40B
Adalious
Gradient Bocsting
Bapgingg
K30N
KNN v2
KNW 42
Karnel Ridge
Liner Regression
Laxaso
MEP Despp
Poly
MLF Deep
MLP Despp
HeLD Wide
MLP Wede
MLB Wide
Rerdom Forest
Pldgee
StM
Sirater
Spline
Average
Lasit
Rendom
Linear regresstent
Method
Figure 41: The rank of Falcon models, compared with traditional supervised methods and unsupervised heuristics, (best viewed in color)
49

---

Published as a conference paper at COLM 2024
P.2 Performance When Knowing The Dataset Name
To further investigate potential data contamination, we conducted an experiment inspired by the methodology described in Golchin & Surdeanu (2024). This involves assessing model performance under two conditions: with and without explicit knowledge of the dataset being evaluated. Specifically, we modify the prompt shown in Figure 9 to mention the name of the dataset (e.g., Friedman #1, Friedman #2, Friedman #3), as detailed in Figure 42. This approach allows us to discern the impact of dataset awareness on the model's predictive accuracy, providing insights into the extent of potential contamination in the training data.
We present the comparative results in Table 14, which shows the average absolute error under conditions of dataset awareness versus unawareness. Notably, the mean absolute errors (J) remain closely matched across all scenarios. To statistically substantiate these ob- servations, performed paired t-tests for each dataset comparison. Given the multiplicity of tests performed, it became imperative to apply an adjustment for multiple comparisons to our p-values (Dunn, 1961; Benjamini & Hochberg, 1995). Following this adjustment, none of the p-values remained below the (typically used) 0.05 threshold, suggesting that the knowledge of the dataset name does not significantly affect model performance. Prior to adjustment, in two cases the resulting p-value was under 0.05: GPT-4 on Friedman #3 (p-value 0.045) and Claude 3 Sonnet on Friedman #3 (p-value 0.018). Note, however, that only in the case of GPT-4 was the performance corresponding to the Dataset Aware setting better. This analysis indicates that, within the bounds of statistical significance, there is no substantial evidence to suggest that the performance of the models is influenced by explicit knowledge of the dataset name, something which has been linked to contamination (Golchin & Surdeanu, 2024).23 23
The task is to provide your best estimate for "Output" (Friedman #1). Please provide that and only that, without any additional text.
Feature ...
Figure 42: The prompt use to further investigate whether the large language models have seen instances the datasets we tested them on.
We investigated why the performance of Claude 3 Sonnet degraded on Friedman #2 when given the dataset name. We found that it is (mostly) because of an outhier the absolute difference between the model's prediction and expected output is > 300.
50

---

Published as a conference paper at COLM 2024
Friedman #1
Friedman #2
Friedman #3
MAE Dataset Aware
2.820+2.332
39.086:164.787 32.787+66.165
0.089:10.433 0.102+0.156
MAE Dataset Unaware
2.798++2.22
(a) GPT-4
Friedman #1
Friedman #2
Friedman #3
MAE Dataset Aware
1.933.1
5.4351 6.374=10:440
0.074:10.163 0.070=0.148
MAE Dataset Unaware
2.002::1.400
Claude Opus
Friedman #1
Friedman #2
Friedman #3
MAE Dataset Aware
2.118+1.791
11.332++33.119 5.375-15.940
0.095+:0.175 0.088110.166
MAE Dataset Unaware
2.192.1..78
(c) Claude Sonnet
Table 14: Comparison of Mean Absolute Error (MAE 4) for Models With and Without
Dataset Name Awareness.
Q Does The Performance Plateau?
In the following, we investigate whether the performance of LLMs continues to improve as the number of in-context exemplars increases. Because this experiment is dependent the context size of the LLM, this imposes additional constraints on which LLMs we can use. To this end, we experimented with the following dataset sizes: {20, 50, 60, 70, 80, 90, 100, 150, 200, 250, 300, 400, 500).
We experiment with ChatGPT and GPT-4. For GPT-4, we found that the performance keeps improving with the number of in-context exemplars at least until 500, We present our results in Figures 43, 44, 45, 46, 47. We repeated each experiment 20 times for each dataset size. We report the mean and 95% confidence.
Aggregating the results, we observed that GPT-4 performed better than Random Forest in 92% of the cases, better than Gradient Boosting in 51% of the cases, and better than Linear Regression with Polynomial Features (Linear Regression Poly) in 40% the cases, across all 5 datasets and all dataset sizes.
For example, from Figure 44 we can observe that while Linear Regression Poly performs much better than GPT-4 in small data regimes, this performance gap decreases as the number of examples increases, suggesting that the model is indeed capable of leveraging a larger number of examples.
51

---

Published as a conference paper at COLM 2024
GPT-4
Lincar Acgression Poly
Gradient Boosting
Random Forest
10-
Moan Absolute Error
Examples
Figure 43: Performance comparison between GPT-4 and three traditional supervised models on the Friedman #1 dataset. The performance of GPT-4 remains good, outperforming Random Forest. (best viewed color)
GPT-4
Linear Regressi on + Poly
Gradient Boosting
Random Forest
150
125
Mean Abselute Error
100
75
50
100
400
Exampies
Figure 44: Performance comparison between GPT-4 and three traditional supervised models on the Friedman #2 dataset. The performance of GPT-4 improves with the number of examples given, approaching that of Linear Regression Poly.
(best viewed color)
52

---

Published as a conference paper at COLM 2024
GPT-4
0.35
Linear Regression on + Poly
Gradient: Boosting
Random Forest
0.25
Mcan Absolute Eroor
0.15
0.10
0.00
250
500
Number Examples
Figure 45: Performance comparison between GPT-4 and three traditional supervised models on the Friedman #3 dataset. (best viewed in color)
GPT-4
Linear Regression + Poly
Gradient Boosting
Random Forest
Mcan Absolute Error
Examples
Figure 46: Performance comparison between GPT-4 and three traditional supervised models on the Original #1 dataset. The performance of GPT-4 increases with the number of examples given. (best viewed colur)
53

---

Published as a conference paper at COLM 2024
GPT-4
700
Lincar Regression Poly
Gradient Boosting
Random Forest
300
Mcon Absolute Error
400
300
200
100
200
400
500
Number of Exampies
Figure 47: Performance comparison between GPT-4 and three traditional supervised models on the Original #2 dataset. The performance of GPT-4 no worse to that of Random Forest (or, to a lesser extent, to that of Gradient Boosting) the number of examples increases. (best viewed in color)
R Beyond Transformer-Based LLMs
In our study, we initially focused on transformer-based large language models (LLMs). To broaden our scope, we explore the capabilities of non-transformer LLMs, including RWKV-based 14B LLM (Peng et al., 2023) and with StripedHyena (Poli et al., 2023a;b), 7B LLM. The performance rankings for these models, along with the transformer-based Mistral 7B for comparison, are illustrated in the heatmap provided in Figure 48. We tried running Falcon 7B as well, but produced invalid outputs for almost all examples and datasets, therefore we skip it. StripedHyena also encountered difficulties, producing invalid outputs in certain scenarios, such as 98% invalid responses for the Friedman #2 dataset. Consequently, we omitted Friedman #2 and Original #2 from its evaluation. However, it important to highlight that the other models evaluated did not exhibit these issues and were able to generate valid outputs consistently. We make the following observations.
First, we remark that performance-wise, RWKV is worse than traditional transformer-based LLMs, although it generally remains better than our unsupervised baselines, with the exception on two linear regression datasets: Regression NI 1/3 and Regression NI 2/3. Nevertheless, we remark that on Original #1, RWKV outperforms many MLP variants, despite no gradient updates.
Second, remark that the performance of Striped Hyena 7B is generally lower than than some of our unsupervised baselines. We note that there is notable exception for Original For this dataset, KNN approaches work well, evident by the good performance obtained by nearest neighbor approaches.
54

---

Published as a conference paper at COLM 2024
1/1
22
16
10
II
24
MI 222
25
15
10
26
Original
Modmun
Meddms
23
strened Hypeo
Mistinal
Adsllsse
Graalent Strating
Bagsiig
kaenel Bidge
Libeal Segression
Ubver Aeguescooo Phir
HEP toep
tiwap
lec resie
Biogee
Seliee
Last
Random
Sook
Nethod
Figure 48: The ranks of RWKV V4 14B and (RWKV Architecture) and StripedHyena Nous 7B (Hyena Architecture) compared with traditional supervised method. We included Mistral 7B for comparison, as is model similar in size. Notably, RWKV's performance, while generally lower than that of transformer-based LLMs, still surpasses our unsupervised baselines. (best viewed in color)
55


Reviews:
[{"review_id": "74AzzHWyxT", "reviewer": "Reviewer_y3a3", "summary": "The paper looks at the ability of 11 pre-trained LLMs like GPT-4, Claude 3, and others to perform both linear and non-linear regression tasks using in-context learning (ICL). The results show that some LLMs can perform on par or outperform traditional supervised methods such as Random Forest, Gradient Boosting, and AdaBoost on challenging synthetic regression datasets - without additional training or parameter updates. Additionally, the paper explores how the performance of LLMs scales with the number of in-context examples, empirically showing that some LLMs obtain sub-linear regret.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We address each reason to reject below.\n\n(1) While this choice has been deliberated, multiple reviewers have mentioned the lack of evaluation on real-world datasets as a weakness. Since future readers might wonder about the performance of LLMs on real datasets as well, we will include a new section titled “Performance On Real-World Datasets”. \n\nWe ran new experiments on the following datasets:\n- (1) Liver Disorders (UCI id=60)\n- (2) Real Estate (UCI id=477)\n- (3) Diabetes (from sklearn)\n- (4) Servo (UCI id=87)\n\nEach experiment was repeated 50 times, each time with different train/test splits. Results:\n\n- On 1, GPT-4 ranks 6, outperforming Gradient Boosting and Random Forests; Claude 3 Opus performs worse, ranking 23rd;\n- On 2, GPT-4 ranks 6 and Claude 3 Opus ranks 4th. Gradient Boosting ranks 1st and Random Forest ranks 9th;\n- On 3, GPT-4 ranks 16th and Claude 3 Opus ranks 19th. Random Forests ranks first;\n- On 4, this dataset contain features both categorical and numerical and we tested only Gradient Boosting and Random Forests. Claude 3 Opus 2nd. Both GPT-4 and Claude 3 Opus outperform Random Forests.\n\nOverall, the strong performance of LLMs persists over real-world datasets as well. Tables skipped due to length constraints.\n\n(2) In the regret experiment, we wanted to analyze how well the LLM are capable of adapting when increasing the number of in-context examples. Borrowing from online learning, ideally, a capable model would be able to achieve the performance of a model that has hindsight information, fast. Our motivation was our hypothesis that, perhaps, LLMs emerge from their pre-training as very capable online meta-learners, which is why they are capable of performing so many tasks, some of which were not seen during their pre-training phase (like our Original regression datasets). In Figure 7, for example, the Y-scale is different between the two figures because the datasets are different. Original #1, for example, has 0 <= f(x) <= 100. Friedman #2, on the other hand, can have values larger than 1000.\n\n(3) We agree and will include this in a dedicated section for limitations. We would like to remark that we did not intend to suggest that LLMs should be used, at least currently, for regression instead of more traditional methods like Gradient Boosting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper is clearly written, well motivated and easy to follow.\n* It nicely expands on earlier work for Transformers and ICL with off-the-shelf LLMs.\n* It conducts a thorough empirical analysis with clear experimental setup for reproducibility and strong baselines for comparison.\n* The analysis of scaling performance and use of regret shows how many examples are needed to get a certain level of performance.\n* The findings have practical use, suggesting that LLMs can potentially reduce the need for traditional supervised learning methods in certain regression scenarios.", "reasons_to_reject": "* Because of the nature of the experiments on synthetic data to avoid encountering data that may have been in the training set, there are no experiments with benchmark/ real world regression problems.\n* In the last study on regret, it is not clear how regret correlates to MAE, and why the regret values or so different between the different models evaluated. \n* It does not consider the computational requirements for scaling these LLMs to even larger datasets or higher-dimensional regression tasks, leaving a question of practical feasibility for real-world applications of larger problems.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JNuv4yN9x", "reviewer": "Reviewer_Gng7", "summary": "The paper evaluates how LLMs like GPT-4 perform linear and non-linear regression tasks using in-context examples, without further training. The authors present evidence that these models can match or exceed traditional supervised methods on several synthetic datasets. They explore how the performance of LLMs scales with the number of in-context examples and demonstrate that these models can achieve sub-linear regret, suggesting effective learning over time.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your review! We address each reason to reject below.\n\n(1) While this choice has been deliberated, multiple reviewers have mentioned the lack of evaluation on real-world datasets as a weakness. Since future readers might wonder about the performance of LLMs on real datasets as well, we will add a new section titled “Performance On Real-World Datasets”. \n\nWe ran new experiments on:\n- (1) Liver Disorders (UCI id=60)\n- (2) Real Estate (UCI id=477)\n- (3) Diabetes (from sklearn)\n- (4) Servo (UCI id=87)\n\nEach experiment was repeated 50 times, each time with different train/test splits. Results:\n\n- On 1, GPT-4 ranks 6, outperforming Gradient Boosting (GB) and Random Forest (RF); Claude 3 Opus performs worse, ranking 23rd;\n- On 2, GPT-4 ranks 6 and Claude 3 Opus ranks 4th. GB ranks 1st and RF ranks 9th;\n- On 3, GPT-4 ranks 16th and Claude 3 Opus ranks 19th. RF ranks first;\n- On 4, Claude 3 Opus 2nd. Both GPT-4 and Claude 3 Opus outperform RF. For this dataset, we only tested the LLMs against GB and RF.\n\nOverall, the strong performance of LLMs persists over real-world datasets as well. Tables skipped due to length constraints.\n\nLastly, we did not intend to suggest LLMs should replace traditional supervised methods for regression. We believe that the observation the off-the-shelf LLMs perform well even on synthetic datasets is a significant finding in itself.\n\n(2) We do not expect overfitting to be an issue with more examples. Overfitting in general and its effect on in-context learning vs traditional methods is interesting but outside the scope of this paper (now ~50 pages with additional results). In Appendix L (Figures 31-35), we show GPT-4's performance improves with more examples (20-500), similar to other methods.\n\n(3) We would like to clarify that this paper is not intended to explain why ICL works in general or for regression specifically. Instead, our focus is on exploring the extent to which LLMs can perform regression. We believe our findings can contribute to a future theoretical framework by further describing transformers' capabilities. We hypothesize that LLMs emerge from pre-training as highly capable online meta-learners, able to perform tasks beyond those seen during pre-training. This hypothesis motivated Section 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper leverages existing LLM capabilities in a novel application, showing that these models can perform regression tasks effectively, which broadens the understanding of their potential uses.\n- Demonstrating that LLMs can perform regression effectively without retraining could significantly reduce computational costs and streamline analytics processes in various fields.", "reasons_to_reject": "- While synthetic datasets provide control over experimental conditions, the results might not generalize to real-world data, which can be noisier and less predictable.\n- The paper does not extensively address whether the in-context learning approach could lead to overfitting, especially with an increasing number of context examples.\n- The paper lacks a theoretical framework explaining why LLMs perform well in these tasks, which could strengthen the overall contribution.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZrDZC3YZcC", "reviewer": "Reviewer_KJ4G", "summary": "This work conducts a detailed empirical analysis of the performance of several state-of-the-art pretrained Large Language Models for in-context learning of linear and non-linear regression. Around 11 of existing LLMs with 50 in-context exemplars are evaluated on synthetic datasets over 7 tasks comprising of Friedman regression benchmark, and linear and non-linear regression, along with regression with non-numerical inputs. The LLMs are compared with baseline methods of supervised methods from sklearn package such as linear regression (with lasso and ridge regularization), Bagging, GBDTs, Adaboost, etc, and unsupervised methods such as average and random. The synthetic data has inputs with dimensionality between 1-20, and around 50 in-context exemplars are used. The performance of the models is measured using Mean Average Error (MAE) and rank of the models. The authors expound on a sub-linear regret as the number of in-context exemplars scale. \n\nThe performance of these models is comparable to Random Forests and GBDTs, which are surprisingly poorer than linear and non-linear regression approaches. This could be the case as the dimensionality of the models is quite low (< 20). Since the datasets are synthetically generated from a generator function, the amount of data that can be generated an fed to train the supervised baselines are quite large in numbers. It would be useful to know how to extend these models to higher dimensionality of features, which is typically the case in real world scenarios, and  where Random Forest and GBDT baselines usually outperform other regression approaches. Given that baselines such as Garg et. al. 2022 are not considered for comparison, the nature of selecting the 50 in-context exemplars could play a vital role. Given the synthetic nature of the datasets, it should be possible to measure the variance of choosing a different subset of in-context exemplars.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your review! We address each reason to reject below.\n\n(1) We argue that the novelty lies in demonstrating that off-the-shelf LLMs can perform regression reasonably well using in-context examples, without additional training. \nWhile the task of regression with transformers and in-context exemplars has been studied in previous work, they all trained the model for the specific task at hand. Our work shows that capable LLMs emerge with an underlying in-context learning algorithm capable of learning regression on-the-fly. This novel observation has interesting implications for future research. \n\nRegarding the impact of which 50 in-context exemplars are selected, the results in our paper were constructed by repeating each experiment 100 times, selecting train/test exemplars each time, and reporting the average. \n\nWe did not claim that LLMs should be the go-to choice for any regression task. We acknowledge that LLMs currently have limitations. However, we demonstrate that off-the-shelf LLMs are more capable than previously thought. Their in-context learning algorithm, which emerges through pre-training, can perform regression tasks on-the-fly using in-context examples without any additional gradient updates.\nWe believe our findings can contribute to a future theoretical framework by further describing transformers' capabilities. Additional implications include: (i) adding synthetic regression data during pre-training to improve the in-context learning algorithm; and (ii) testing how well LLMs use their context by evaluating their performance on regression tasks, where the error should predictably decrease with more examples if the LLM is indeed capable of using all its context well.\n\n\n(2) Regarding higher-dimensional datasets, we believe that techniques typically used for feature selection could be effectively applied to LLMs as well. We want to emphasize that off-the-shelf LLMs performing well even on lower-dimensional datasets is a significant finding in itself. This can pave the way for further exploration on higher-dimensional data.\n\n(3) We used the same examples for LLMs as for baselines, with baseline training handled by sklearn using default, battle-tested values. Similarly, no tuning was performed for LLMs to ensure a fair comparison. We will add more details in the Appendix B, where we describe the models we used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written, and well presented.\n\n2. In-context learning is important to understand for capabilities of learning functions beyond natural language tasks, which this paper looks at.\n\n3. The empirical analysis is extensive and covers a large number of LLMs and baseline methods.", "reasons_to_reject": "1. The novelty is limited, the impact of which 50 in-context exemplars to select is not looked into. Domain adaptation of LLMs for in-context learning has already been studied in several works (Garg et al 2022, etc), and this work essentially works without any domain adaptation, and might not generalize to higher dimensions or other more complex datasets.\n\n2. It is unclear how to extend this to work with higher dimensional datasets (>20) which are common for regression tasks.\n\n3. More details on how long and many examples are used to train the baselines is important given the datasets are synthetic.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "flCgGu22rl", "reviewer": "Reviewer_n2HA", "summary": "This paper empirically compares the performance of LLMs on linear and non-linear regression tasks against traditional supervised models. The evaluations of LLMs are conducted in an ICL manner, where the training input and output are used as exemplars. Several different supervised methods (with their variations) are studied as the baselines, including linear regression, multi-layer perceptron, Random Forest, Bagging, Gradient Boosting, AdaBoost, SVM, KNN, Kernel Ridge, and Splines. To mitigate the risk of evaluation data being exposed in LLM pretraining, new non-linear regression datasets are also proposed in the evaluation. Overall, this study reveals that LLMs are capable of performing both linear and non-linear regression on the proposed datasets. Strong LLMs can outperform some supervised algorithms like linear regression and gradient boosting. They also approach the performance of the best supervised algorithms in some specific tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We address each reason to reject and question below.\n\nRegarding the reason to reject:\n\n(1) We show in Figure 14 results with six variants, covering the possible combinations with 3 total variables. Our main motivations were to (i) keep the total number of tokens lower to accommodate a wider range of LLMs and (ii) reduce costs. To further supplement the results for linear regression, we conducted new experiments with: (i) 1 informative variable out of 10, (ii) 3 informative variables out of 10, and (iii) 5 informative variables out of 10. The performance is similar to NI 2/3 from Figure 14, where LLMs still perform reasonably well, but are outperformed by various baselines. For example, in NI 5/10, Claude 3 Opus outperforms methods like Gradient Boosting or Random Forest but lags behind methods like LR, which is expected given the nature of the datasets. For NI 3/10, Claude 3 Opus is outperformed by Random Forest and Gradient Boosting. Overall, these results align with those from Figure 14.\n\nWe will update this in text. \n\n\nRegarding the Questions:\n\n(1) We included the rank heatmaps to address this issue. Additionally, we will include a table with average ranks across different dataset types. We also released our code and intermediate results, such that if any interested reader wants to compute other measures (or re-create ours), they will be able to do so. If the reviewer has other suggestions, we are willing to accommodate them. \n\n(2) In Sec 2.1.2 we included the definitions of each datasets in their most generic form (e.g., [make_friedman1](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_friedman1.html)). In our experiments, we set $\\epsilon$ to 0. We will add additional details in Appendix H to clarify this. Following the suggestions from the reviewers, we also included results with real-world datasets, which are expected to contain noise.\n\n(3) That detail is part of the Friedman #1 dataset definition (e.g., [make_friedman1](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_friedman1.html)). We indeed used $d=5$, but included the generic definition of the dataset. We will further clarify this in text to avoid any confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper studies an interesting task. While it is widely believed that LLMs are not good at math tasks, this paper revealed that strong LLMs can outperform simple supervised algorithms for certain problems. There is no widely accepted explanation for this yet, but it is an intriguing problem. From this perspective, the empirical findings from this paper are valuable.\n2. I like the regret analyses in Sec. 5. It gave a rough estimation of how fast LLMs can learn to estimate the underlying functions. This can help determine the number of examples needed in different tasks.", "reasons_to_reject": "Please correct me if my understanding is wrong. I think an important property missing in the paper is how the LLM’s performance change with different number of variables in the regression task. Based on Figure 2, there are at most 2 (informative) variables in the evaluations. This is significantly simpler than the dataset description of $1 \\leq d \\leq 20$ . Also, it is also not clear why there is no noise term $\\epsilon * N(0, 1)$ in the linear regression task. With only 2 variables, it would be hard to claim that LLMs are capable of doing general linear regression tasks.", "questions_to_authors": "1. Bar plot might not be the best choice when the scale differences of MAEs are big (e.g., Figure 2a).\n2. Sec. 2.1.2 how is $\\epsilon$ determined in the experiments?\n3. Sec. 2.1.2 re: \n> When $d > 5$, the extra dimensions are ignored\n\nwhy not just set $d=5$?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper demonstrates several LLMs can be effective at performing regressions in-context, in some cases outperforming supervised methods.

The reviewers noted several strengths of the paper:
- The paper makes an interesting point about the emergent capabilities of LLMs.
- The paper provides an important expansion on prior work by studying pretrained LLMs rather than ones trained specifically to be able to do ICL.
- The paper is clearly written.
- The experimental setup is strong, with clear baselines, a comprehensive set of experiments, and enough information provided to make for strong reproducibility

Several limitations also came up:
- The paper does not provide a theoretical understanding of why LLMs can perform regression tasks.
- LLMs using ICL may still not be effective in real-world regression tasks or as replacements for supervised methods
- There are unexplored directions about several details of the experiments such as the number of in-context examples and the numbere of dimensions.

However, I found the following comment from the author response to be compelling in response to the first and third points:  “We acknowledge that there are open questions regarding the limits of LLMs in regression tasks, such as whether their strong performance is restricted to specific conditions like a small number of examples or dimensions. However, these questions should not detract from the merits of our findings. Instead, they highlight potential avenues for future research that our work enables.” And regarding the second point, I was convinced by the authors’ response that these directions are not the goal of the work (the goal is to understand LLMs, not to propose a new approach for doing regressions).

Overall, I am recommending acceptance due to the interesting results that make important points about ICL in LLMs, and due to the fact that I found the response from the authors to be generally compelling in addressing reviewer concerns. The response from the authors also added some new results with real-world datasets that will enhance the paper.

I thank the reviewers and authors for a very productive discussion period.
Decision: Accept