MLPs Learn In-Context on Regression and Classification Tasks
Authors: William Lingxiao Tong
Cengiz Pehlevan
Venue: ICLR 2025 Poster (2025)
Keywords: In-context learning
relational reasoning
synthetic tasks
MLP
MLP-Mixer
Transformer
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: On a range of widely studied synthetic in-context learning tasks, we find that MLPs perform comparably with Transformers under the same compute budget.

Abstract: In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By examining commonly employed synthetic ICL tasks, we demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, MLPs, and the closely related MLP-Mixer models, learn in-context comparably with Transformers under the same compute budget in this setting. We further show that MLPs outperform Transformers on a series of classical tasks from psychology designed to test relational reasoning, which are closely related to in-context classification. These results underscore a need for studying in-context learning beyond attention-based architectures, while also challenging prior arguments against MLPs' ability to solve relational tasks. Altogether, our results highlight the unexpected competence of MLPs in a synthetic setting, and support the growing interest in all-MLP alternatives to Transformer architectures. It remains unclear how MLPs perform against Transformers at scale on real-world tasks, and where a performance gap may originate. We encourage further exploration of these architectures in more complex settings to better understand the potential comparative advantage of attention-based schemes.

Published as a conference paper at ICLR 2025
MLPs LEARN IN-CONTEX ON REGRESSION AND CLASSIFICATION TASKS
William L. Tong & Cengiz Pehlevan
School of Engineering and Applied Sciences
Center for Brain Sciences
Kempner Institute for the Study of Artificial and Natural Intelligence
Harvard University, Cambridge, MA 02138
(wtongêg, cpehlevan@seas) waw.aadd.dd.
ABSTRACT
In-context learning (ICL), the remarkable ability to solve task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By ex- amining commonly employed synthetic ICL tasks, demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, MLPs, and the closely related MLP-Mixen models, learn in-context comparably with Transformers under the same compute budget in this setting. We further show that MLPs outperform Transformers on series of classical tasks from psychology designed to test re- lational reasoning, which are closely related to in-context classification. These results underscore a need for studying in-context learning beyond attention- based architectures, while also challenging prior arguments against MLPs ability to solve relational tasks. Altogether, our results highlight the unexpected competence MLPs in synthetic setting, and support the growing interest in all-MLP alterna- tives to Transformer architectures. It remains unclear how MLPs perform against Transformers at scale on real-workes tasks, and where a performance gap may orig- inate. We encourage further exploration of these architectures in more complex settings better understand the potential comparative advantage of attention- based schemes.
INTRODUCTION
The last few years have witnessed meteoric progress in neural language models. Catalyzed by the Transformer architecture and driven by steady increase in scale, these aptly-named Large Language Models (LLMs) demonstrate unprecedented competence drafting grammatical text, answering questions, summarizing content, generating creative output, and even reasoning through non-trivial puzzles (Bubeck et al., 2023; Brown et al., 2020; Achiam et al., 2023).
Crucial to LLM's proficiency is its ability to learn in-context (Lu et al., 2023; Dong al., 2022; Brown et al., 2020). In-context learning (ICL) refers to a task paradigm where exemplars from novel task are presented during inference time rather than during training (Figure la). The model must then respond correctly to a query based only on these novel exemplars. No weight updates occur throughout this process; rather, the model infers the task from the input exemplars and, despite having fixed weights, produces the correct output.
ICL commonly assumed to be unique ability of Transformers, and explanations of the phenomenon often ground their constructions in attention-based architectures (Akyürek et al., 2024; Von Oswald al., 2023; Zhang et al., 2023; Reddy, 2024; Lu al., 2024). On controlled regression and classification tasks targeted specifically for evaluating ICL, we demonstrate that the simple multi-layer perceptron (MLP) can also learn in-context - and moreover, learn in-context competitively with the full Transformer given the same compute budget.' These results suggest that ICL is not an exclusive
Universal approximation by MLPs suggests that they may be able to learn in in-tontext, though at uncertain cost. We demonstrate that MI.Ps leam in-context without significantly larger compute (and occasionally quite hit smaller) than Transformers.

---

Published as a conference paper at ICLR 2025
feature of attention-baa based architectures, and highlights the need for studying the phenomenon in broader setting.
Tasks, We focus on controlled tasks commonly studied in the ICL literature, where the specific capacity for in-context learning can be precisely characterized. These tasks are necessarily synthetic approximations of natural language ICL prompting, but allow us to disambiguate a model's capacity for in-context learning from its ability to attain natural language fluency. In Section 2, we examine ICL versions of regression (Garg et al., 2022; Akyürek et al., 2024; Raventós et al., 2024; Zhang al., 2023) and classification (Reddy, 2024; Chan al., 2022). As the two primary task paradigms of machine learning, regression and classification form a representative basis for measuring ICL competency. In Section we consider series of classical tasks used in the psychology literature to probe relational reasoning (Campbell ct al.. 2023; Skinner, 1950; Sablé-Meyer et al., 2021), which are functionally in-context classification. On these tasks, we find that MLPs outperform Transformers, challenging common beliefs about MLPs' proficiency at relational reasoning (see Appendix A for an extended discussion). In focusing on controlled tasks, we avoid confounds irrelevant to ICL introduced by naturalistic settings like language and vision. Nonetheless, our findings remain consistent with existing results that do test MLPs in these more complex domains (Tolstikhin et al., 2021: Liu et al., 2021; Fusco al., 2022; Bachmann et al., 2024).
Ground rules. To ensure different architectures are comparable across tasks, we observe the following ground rules. First, we compare models based on the total compute required for training (measured in peta-floating point operations, PFLOPs), which summarizes influences like parameter count, training iterations, and architectural efficiency. Details on how we compute this quantity are provided in Appendix C.13. Measuring by compute reflects the practical use of these models, fairly compares architectures by performance per floating-poin cost, and is an established scale for defining neural scaling laws (Kaplan al., 2020). Second, where a single model required, we select the best model configuration as measured by loss, keeping compute cost equal across architectures. Data are presented online, reflecting the "one-pass" setting common in training large language models (Brown et al., 2020). Specific model and task configurations are enumerated in Appendix C.
1.1 RELATED WORK
In-context learning has been widely studied in number of controlled settings. particular, ICL has been reproduced for linear regression, where a Transformer trained to perform the task can extrapolate to novel input/label pairs provided in-context (Garg et al., 2022; Akyürek al., 2024; Raventós et al., 2024; Wu et al., 2024; Bai et al., 2024; Li et al., 2023; Lu et al., 2024). Proposed mechanisms whereby a Transformer accomplishes the feat include that the Transformer implements some form of gradient descent (Von Oswald et al., 2023; Akyürek et al., 2024) or recapitulates least-squares or Ridge regression (Zhang et al., 2023; Akyürek et al., 2024; Lu et al., 2024). has also been observed that Transformer interpolates between in-weight learning (IWL), the traditional paradigm where the model learns specific examples through training, to in-context learning, where the model uses only exemplars provided in the input context inference time (Raventós et al., 2024; Wu al., 2024). Such transition occurs as a function of data diversity, where datasets with more distinct examples encourage the development of ICL competency. Analogous phenomena have been observed in in-context classification tasks (Chan et al., 2022; Reddy, 2024). Impressively, the ICL performance attained in these tasks by Transformers approaches Bayes optimality (Xie al., 2021; Bai al., 2024; Li et al., 2023; Ahuja et al., 2024; Lu et al., 2024).
These studies nearly all ground their investigations in Transformer models, and explicitly assume that the model uses an attention mechanism to implement ICL. The exceptions include Chan et al. (2022), who discover that recurrent neural networks (both vanilla RNNs and L.TTMs) are unable to learn an in-context classification task under the same conditions where Transformer can, and Xie et al. (2021), who discover that LSTMs can in fact learn in-context on a synthetic language modeling task. Recently, Lee et al. (2024) found that wide variety of causal sequence models can learn in in-context on a broad array of toy tasks, with varying degrees of success. Park et al. (2024) support this finding by showing how state space models and their hybrids with Transformers can learn in-context competitively. To the best of our knowledge, no prior work has examined in-context learning in vanilla MI.Ps.
2

---

Published as a conference paper at ICLR 2025
The current resurgence of interest in applying MLPs to modern, complex tasks originates with Tolstikhin et al. (2021), which introduced the MLP-Mixer model. Mixers operate by alternating MLPs across the dimensions of the input, treating the remaining dimensions as batch dimensions. Despite their simplicity, Mixers attain state-of-the-ar performance on image classification, recalling the broad observation that "less inductive bias is better" (Sutton, 2019; Bachmann al., 2024). In the ensuing years, "all-MLP" models based primarily on MLP components have spread across many areas including vision (Bachmann et al., 2024) and natural language (Liu et al., 2021; Fusco et al., 2022). While strong performance has been documented on natural language, less is known about MLPs' specific proficiency for ICL, and how compares with Transformer models. In this study, we select series of controlled, representative tasks that clarify an MLP's surprising competence for ICL. Our findings underscore the ultimate utility MLPs, uncovering avenues of both theoretic and practical interest.
EXPERIMENT: IN-CONTEXT TASKS
We begin by exploring MLPs' behavior in a controlled ICL format, where their specific capacities and weaknesses can be precisely characterized. Specifically, we examine two tasks: in-context regression and in-context classification.
2.1 ICL REGRESSION
We present in-context regression following its common formulation (Garg et al., 2022: Zhang et al., 2023). The input consists of sequence of values (æ1, W1), (x2, w.),.... (ær,, Wr.), (r., where æ; E IR" and y/ E R. The Is y1 pairs are linearly related through a set of weights 3 E R" such that Y/ T; B + E, with noise ~ (O0, o²). Finally, the input includes query T.. The model output is a single scalar regressed against the corresponding 34. Crucially, the weights 3 vary between input sequences. The model cannot rely learning any one B. Rather, it must infer from context exemplars (æ;, y/) what the corresponding ,3 must be, and use this to predict the correct output Y4- Figure lb illustrates the task, with additional details in Appendix C.
In the main text, our task fixes the number of context points at L. A common variation on this tasks allows the number of context points vary, and trains the model autoregressively. Results on this autoregressive variation are presented in Figure 5, and are unchanged from the fixed-length case.
Following Raventós et al. (2024), we consider two different task distributions: finite and unrestricted. For the finite distribution, we fix a finite pool of weights before training 12.32..... Bg. where 3, ~ ~NO, I /n). For each input, we sample new ,3 by selecting uniformly random one weight from the pool { Larger A corresponds to higher data diversity. For the unrestricted distribution, new of weights sampled for each input (N I //n.. The unrestricted distribution can be thought of as the k + oo case, and requires full ICL competency in order to infer the correct weights dimensional inputs.
relating the context exemplars. Unless otherwise stated, we use
Results. We first consider how MLPs perform compared Transformers on in-context regression. To do so, we train and test using online samples drawn from the unrestricted task distribution, requiring all models learn an in-context solution. Figure Ic plots the MSE achieved by different architectures as a function of total compute. With sufficient compute, MLPs, Mixers, and Transformers all perform in-context regression with near optimal MSE, which given by Ridge regression on context points using the Bayes optimal regularization parameter (Appendix C.6). For smaller compute, Transformers attain somewhat better MSE than their MLP counterparts, though the difference is modest and performance across all three architectures overlaps substantially.
One domain in which a vanilla MLP is decisively worse than Transformer for long context length. Figure ld plots the excess MSE obtained after training and testing on the unrestricted task distribution for varying number of points in the context, where [excess MSE) = (model MSE) [Bayes optimal Ridge MSEJ. The Transformer generally approaches the optimal MSE regardless of context length, though it performs with less stability for longer contexts. The vanilla MLP worsens quickly with larger contexts and approaches the performance of an estimator that returns zero for every input. Strikingly, the MLP mixer does not exhibit the same sensitivity to context length, and continues attaining the Bayes optimal MSE consistently even for very long contexts.
3

---

Published as a conference paper at ICLR 2025
C
1.00
1.00
Excess MSE
MLP
MSE
0.75
Mixer
Transformer
0.50
0.25
0.00
#11 01. 022-..
10-1
10-1
107
Compute (PFLOPs)
2
2
Finite Task Distribution
Unrestricted Task Distribution
ICL Regression
0.4
1.5
MLP
Mixer
MSE
MSE
1.0
Transformer
0.2
dMMSE
Point Query
0.5
Ridge
2
2
2
2.
0.6
MLP
Mixer
ICL Classification
Loss
0.4
Transformer
0.2
10-4
10-2
10
0.0
Compute (PFLOPs)
2
2
2
2
Point Query
Accuracy
MLP
Mixer
Transformer
Test Task
ICL (swap)
ICL (resample)
IWL
2
2
21
2
2
21
2
2
Figure 1: ICL regression and classification results. (a) ICL presents context exemplars from novel task (red), followed by query input (blue). The model must infer the solution (green) based on the context. (b) ICL regression example. The model receives linearly-related input points, and must regress the query point. (c) Compute vs. MSE on the unrestricted task distribution. Each point represents single model, with particular parameters and training iterations. At large compute, MSE approximately equal across architectures. The red line corresponds to the Bayes optimal Ridge MSE. (d) Excess MSE (MSE above Bayes optimal) for varying context length L on the unrestricted task distribution. Excess MSE remains flat for Mixers, but rises somewhat for Transformers. MLPs fail to learn in-context all beyond 26 context exemplars. The grey line corresponds to the excess MSE incurred by always guessing zero. (e, f) IWI. to ICL transition with increasing data diversity. We train on finite distribution with & weights, then test on both the finite training distribution and the unrestricted distribution. All models exhibit transition from IWL (represented by dMMSE) to ICL (represented by Ridge) increases. Note: it possible to "outperform" " Bayes optimal Ridge on the finite training distribution by learning in- weight the underlying 3's. (g) ICL classification example, with burstiness 3. Multiple clusters may share the same label. (h) Compute vs. cross entropy loss on ICL classification, with A: 2048 clusters, B 4, and L 8, which pushes all models to learn in-context. large compute, all architectures attain near-zero cross entropy loss. The gray line corresponds to loss obtained from placing equal probability present in context. (i) Cross entropy loss for varying context length on the task configuration in (h). Loss is relatively flat for all architectures, though it increases little for Mixers. (j) IWL ICL transition with increasing data diversity, where L = 8 and B = 4. All models exhibit a transition from IWL to ICL the number of clusters & increases. (all) We use n 8 dimension inputs. All line plots feature 95 percent confidence intervals about the mean, estimated from 5 replications.
the 2 (of C' = 32) labels
4

---

Published as a conference paper at ICLR 2025
One final observation: data diversity increases, Transformers exhibit transition from in-weight learning (IWL), the traditional paradigm where the model learns specific examples through training. to in-context learning, where the model uses only context exemplars presented at inference time (Raventós et al., 2024). We next show that MIPs exhibit a similar transition. Following Raventós et al. (2024), we train each model on finite distribution with & fixed regression weights. As increase k, we record the MSE obtained by each model on both the finite distribution /3 ~ L (13)) using the same 3 from training (Figure le) and the unrestricted distribution 3 ~ (O, I/n) where I's are sampled anew (Figure If). We determine whether a model has learned the in-weight solution by comparing its MSE to that of the discrete minimum mean squared error (dMMSE) estimator, which is a Bayesian estimator derived from a prior matched to the finite training distribution (see Appendix C.6 for details).? We characterize the in-context solution by Ridge estimator with the Bayes optimal choice of regularization. For small k, all models demonstrate in-weight learning by tracing the dMMSE curve. As k increases, we observe swift transition to the Ridge curve, indicating a transition to in-context learning. The Transformer makes this transition at a somewhat smaller than the MLP models. We consider additional plots and parameterizations in Appendix D.
2.2 ICL CLASSIFICATION
Following Reddy (2024), we present in-context classification as follows. The input consists of sequence of context exemplars (z), (2) ( where ,; y, E IR". The æ points are sampled from a Gaussian mixture model Me consisting of k components. Each mixture component (i.e. cluster) is labeled by one among C' labels, where k 2 C. so multiple clusters may map to the same label. Labels are represented in the context by vectors 01; Qr2, ac E R. If æ; belongs to cluster j. then y = aj. The model must predict the correct label for g... and outputs C' logits corresponding to the C labels (not vector of values a, which are used only to represent labels in the context). Figure Ig illustrates this task, with additional details in Appendix C.7.
22) followed by a query point z.
Importantly, the query point B. shares a cluster with at least one of the context points 21, 222 Cr. Mixture components and cluster labels remain fixed throughout training. Hence, the model can learn either an in- weight solution by memorizing the label for each cluster, or an in-context solution by referencing the class label associated with og among the context exemplars. We also consider the input's burstiness B. which is the number repeats per cluster in the context (B must divide the context length L). For example, B = 3 means there are exactly three points from each cluster represented in the inputs. Data diversity corresponds to the number of clusters k. where larger k correspond to more diverse dataset. Unless otherwise stated, we use 8 dimensional inputs.
Results. We first compare how different architectures pertirr at ICL across different compute in Figure 1h. The task is parameterized by burstiness B and k 2048 with L points in the context, setting in which all models develop an in-context solution (see Figure 7d for details). As before, with sufficient compute Transformers do not outperform vanilla MLPs or Mixers. Indeed, at small compute, vanilla MLPs attain somewhat lower loss. Note: in this setting, there are L/ B 2 labels present in each context, out of C 32 total possible labels. As a baseline, we plot gray the loss obtained by placing equal probability the 2 labels present in-context. MLPs in particular appear to plateau at this baseline before approaching zero loss with higher compute.
We also measure how well each model handles increasingly long context lengths in Figure li. In surprising reversal, vanilla MLPs attain a relatively flat loss across context lengths, as do Transformers. Mixers' loss increases somewhat for longer contexts, though this blip vanishes at higher dimensions (Figure 7b). Overall, MLPs continue to perform at a comparable level with Transformers on in-context classification.
Finally, we observe a transition from IWL to ICL across the three architectures as data diversity increases. As in Reddy (2024), we measure IWI. by constructing test examples where the query point does not appear in the context. The only way the model correctly classifies these points is if memorizes the mapping from cluster label. To measure ICL, we consider two different strategies: 1) sample points from an entirely different mixture M e producing novel clusters, or 2)
2The dMMSE estimator averages across the & weights in the finite training distribution based on their fit to the current context exemplars.
5

---

Published as a conference paper at ICLR 2025
swap cluster/label mappings that clusters are labeled differently than they during training. Test examples from either strategy can only be solved if the model identifies cluster labels in- context, since the underlying cluster label assignment is different from training." We plot accuracy across all three types of test examples in Figure Jj for increasing k, and observe a transition from IWL. to ICI. across all three model architectures. The transition happens for somewhat lower data diversity in Transformers and Mixers, and somewhat higher in vanilla MLPs. Additional plots and parameterizations are explored in Appendix D.
Train scramble
True
Falco
Test scramble
Accuracy Accuracy
to-sample
Troer
Loss
Fals
Puit
Anseeet
10-6
10
0.25
0.5
0.25
0.5
2
Compute (PFLOPs)
Radius
Radius
g
1.0
10a
Xx dl
Sphere odoball
Accuracy
Logit
d
Loss
0.5
10a
Point Answer
0.0
10-6
10"4
1
2.5
5
20
80
80
Compute (PFLOPs)
Distance
Distance
Accuracy
Line oddbail
Loss
0.5
0.5
Test d
Test d
Accuracy
Train d'
MLP
10
10
RB MLP
Compute (PFLOPs)
RB MLP (deep)
0.5
Transformer
Figure 2: Relational reasoning results. Global legend is at the bottom right. (a) task. (b) Compute vs. cross entropy loss MTS task. Each point represents single model, with particular parameters and training time. RB MLPs attain the best loss with the smallest compute, followed by MLPs and Transformers. (c) OOD generalization
MTS. In-distribution radii highlighted in red. MLPs and RB MLPs generalize well on OOD radii. No model generalizes well on OOD test scrambling.
(d) Sphere oddball task. (e) Same in (b), for sphere oddball. (f) OOD generalization on sphere oddball. In-distribution distance is highlighted in red. Red dashed lines correspond to the accuracy obtained by guessing that the furthest point away from the cluster center is the oddball. (g) Logit of oddball point as its distance from the center increases. Dashed lines correspond to different polynomial scalings. Only the Transformer fails to increase its logit with distance. (h) Line oddball task. (i) Compute vs. loss on line oddball task. RB MLP no longer learns the task well, but appending additional MLP layers ("RB MLP (deep)") helps. (j) OOD generalization on line oddball. In-distribution distance is highlighted in red. Red lines indicate accuracy attained by a model guessing that the furthest point away from the center the oddball. MLPs continue to generalize stronger than Transformers, and match the deep RB MLP. (all) Shaded regions and error bars correspond to 95 percent confidence intervals estimated from 5 replications.
In practice, accuracy on these ICI. measures virtually identical across all models and settings.

---

Published as a conference paper at ICLR 2025
3 EXPERIMENT: RELATIONAL TASKS
We next consider classical tasks from psychology used to study relational reasoning in humans, animals, and neural networks (Campbell et al., 2023; Skinner, 1950; Sablé-Meyer al., 2021; Geiger et al., 2023). These tasks are functionally a subset of in-context classification, and rely on understanding similarity relationships between context exemplars.
In surprising advance from the tasks explored in Section 2, we find that MLPs perform better with less compute than Transformers, and generalize more effectively on out-of-distribution test sets. As a benchmark for gold-standard performance using hand-crafted features, we implement relationally bottlenecked MLP (KB MLP), an architecture demonstrated to solve many challenging relational tasks with competitive generalization and efficiency characteristics (Webb et al., 2023; Campbell et al., 2023). Relational bottlenecks are architectural components that prevent absolute information about the inputs from propagating downstream; rather, the RB computes set of (hand-crafted) relations between inputs (often simple dot products), and allows only these relations to flow downstream, forcing the model operate abstractions. Our RB MLP operates by first computing dot product relations between inputs, then propagating optionally through several MIP layers before a final readout (see Appendix C.5 for details). We find that while relational bottlenecks are helpful when the model's 's relations align well with the task structure, they may fail on tasks with deviating structure. All in all, these relational tasks demonstrate that MLPs can quite surprisingly outperform Transformers on certain in-context tasks.
The question of whether neural network models can reason relationally at all has been an enduring topic of heated debate (see Alhama and Zuidema (2019) for a recent review). Our results fall decisively in favor of the affirmative, and contrast a recent attempt formally proving that MLPs cannot reason relationally (Boix-Adsera et al., 2023). In Appendix A, we discuss our relationship with the relational reasoning literature, comment on the proof by Boix-Adsera et al. (2023), and demonstrate empirically that a vanilla MLP solves task posited by their arguments to be impossible.
3.1 MATCH TO SAMPLE
The match-to-samie sample (MTS) task paradigm originates in Skinnerian behavioral experiments (Skinner, 1950). A test subject first presented with a sample stimulus (c.g. an image). The subject is then shown a set of many stimuli, and must select the one that matches the original sample.
Our MTS task proceeds as follows. The model is presented with L context points 11, T2..... 3g R² followed by query point Tg. The context points are evenly spaced along circle with unit radius centered at the origin. The model must return the index of the context point closest to the query y arg max; (a:; a.y). The points can be thought of as idealized stimulus embeddings in neural representation space. model must reason correctly about distances between points, and ignore their absolute location (which varies from example to example). Framed this way, the MTS setup is an in-context classification task with one context point per class. In the results that follow, we use L == 5 points in the context. Figure 2a presents illustration of the task, with additional details in Appendix C.8.
In addition to the vanilla MLP and Transformer models, also consider relationally bottlenecked MLP architecture (RB MLP) (Webb et al., 2023). The RB MLP uses dot-product relations r between the query point and each of the five context points r = (ar. 21; B. - 122..... 3g .x)). The relations r are passed directly to a softmax readout, producing class predictions YRB = smax ( (W readour r). Note, a choice of weights W readout I solves the task perfectly, though it remains to be seen whether the RB model discovers this solution. Further details on the RB MLP model are in Appendix C.5.
Results. Figure 2b plots the loss obtained by each of the three models on the MTS task as function of compute. The vanilla MLP outperforms the Transformer by surprising margin. With relations that are well-aligned to the task, the RB MLP model achieves the best compute efficiency.
We also consider how well each model performs in different kinds of out-of-distrrbution test examples. Results are plotted in Figure 2c. We first try perturbing the radius of the circle after training with unit radius. As we vary the radius during testing, both MLP models continue perform well, but the Transformer quickly degenerates. We also try changing the order of context points. If the points ordered, they are presented in clockwise order along the circle. If the points scrambled, they
7

---

Published as a conference paper at ICLR 2025
presented in random order. Curiously, if the models trained first ordered points, then no model generalizes well when subsequently tested with scrambled points (not even the relationally bottlenecked model).
3.2 SPHERE ODDBALL
The oddball tasks described in the next two sections are based on work from Sablé-Meyer et al. (2021), who used it to measure geometrie relational reasoning in humans and baboons. In oddball task, the test subject is presented with six stimuli, originally geometric images. One image differs from the others. The subject must select this "oddball" to succeed. Like the MTS task, the oddball tasks are a subset of ICL classification where all-but-one point belong to the same class.
As before, our version of the oddball task simplify full visual stimuli into idealized stimulus repre- sentations. The model is presented with I. context points 21, #2...., 2T E TR.2_ (There are no query points.) In the sphere oddball task, the context points are sampled as N (1,,, T), for some nonzero center μ. One point in the context is randomly selected and perturbed in a random direction by distance d. The model must return the index of this oddball point. In the results that follow, we use L ==6 6 points in the context. Figure 2d presents an illustration of the task, with additional details in Appendix C.9.
In addition to the vanilla MLP and Transformer models, we again use an RB MLP with dot- dot-product relations. Given the average context point T _ Σ, mz, the relations R form matrix with entries Raj = (2; )) (x) H). These centered' dot-product relations are flattened and passed directly to softmax readout, forming class predictions ýкв smax W readout flat RR)) Note, the sphere oddball task can be solved by finding the point that is furthest away from the cluster center. Hence, choice of W readout that selects the diagonal relations R;; will solve the task, but it remains to be
seen whether the model will learn such a readout. Additional details on the RB MLP are provided in Appendix C.5.
Results. Figure 2e plots the loss obtained by each model on the sphere oddball task as a function of compute. Again, the vanilla MI.P outperforms the Transformer by wide margin. With well-suited relations, the RB MP achieves the best compute efficiency.
We also consider how each model performs on OOD test examples. Training examples consist of oddballs with fixed perturbation distance d = 5. As we vary towards longer distances, we observe in Figure 2f that both the vanilla and RB MLPs continue performing perfectly, while the Transformer's performance decays. We can also examine how the logit corresponding to the oddball index changes as we change the position of the oddball with respect to the cluster center (Figure 2g). Both the vanilla and RB MLPs learn strictly increasing relationships, suggesting they will correctly generalize any d provided the other logits do not also increase. The Transformer seems to asymptote to flat value, suggesting that it ultimately fails to distinguish the oddball logit for large d.
3.3 LINE ODDBALL
Rather than sample context points from spherical Gaussian, the line oddball task distributes context points along a line. For each training example, we select a line with random orientation that passes through the origin. Context points 2,, I2. ... Ey. R² Gaussian distributed along this line with zero mean and unit variance. One context point is selected at random to be the oddball, and is perturbed by distance d in the direction perpendicular to the line. The model must output the index of the oddball point. We use L = 6 points in the context. Figure 2h presents an illustration of the task, with additional details in Appendix C. We use the same models for the spherical oddball, including an RB MLP using the same relations R.
The line oddball task cannot be solved by simply selecting the point furthest away from all the others for small d. The relevant relations are more sophisticated, and must be sensitive to the linear structure between context points. The line oddball task also presents an alternative hypothesis for the structure of stimuli in representation space. Whereas the sphere oddball presumes that input
'Centering was not required in the MTS task above, since the context points in that task were already centered.
8

---

Published as a conference paper at ICLR 2025
stimuli are distributed isotropically in representation space, the line oddball task assumes that inputs fall along a favored direction. Neither is obviously more plausible than the other for a general task. However,
see below, the RB MLP from the past two tasks fails quickly this task, presenting a simple example in which well-aligned relations can be critical. We also experiment with a "deep" RB MIP, which features two additional MLP layers between the relations and readout, and now solves the task at a small compute cost.
Results. Figure 2i plots the loss for each model as function of compute. Vanilla MLPs perform just a little better than Transformers. (shallow) RB MLP fails to solve the task altogether, and loss remains high. A deep RB MIP, which features two additional fully-connected layers after the relational bottleneck, can solve the task.
We also compare how each model performs on different We vary the distance d between the oddball point and the line of context points on different training sets. At test time, we compare the accuracy of each model on the whole range of d. As saw above, MLPs continue to outperform Transformers on almost all OOD test cases. Unless equipped with further layers, the shallow RB MLPs fail to learn the task at all for small d. Though a relationally bottlenecked model can succeed with great efficiency on well-aligned tasks, without relations that are tailored the task's underlying structure, an RB model may be disadvantaged.
test examples in Figure 2j.
4 DISCUSSION
We observed that MLPs can learn in-context and moreover, perform at a level comparable to Trans- formers on in-context tasks. We also examined relational reasoning tasks, closely related to ICL. classification, which have historically been considered beyond the ability of simple neural architec- tures like MLPs (Alhama and Zuidema, 2019; Marcus, 1998; Boix-Adsera et al., 2023). Surprisingly, MLPs learn these relational tasks well and exhibit both better compute efficiency and generaliza- tion performance than Transformers. This observation diverges from earlier claims (Boix- -BsiraAdsera al., 2023; Webb et al., 2023), but is consistent with the emerging realization that, given sufficient data diversity and compute, an MLP can indeed learn to reason relationally (Geiger et al., 2023). We discuss our relationship with prior work in relational reasoning further in Appendix A.
Broadly, our work is consistent with the "bitter lesson" of Al research (Sutton, 2019): in the face of increasing compute and data resources, general methods with weaker inductive bias will outperform specialist methods endowed with stronger inductive bias. This heuristic speaks to the intuition that a strong inductive bias may be beneficial for particular tasks, but may misalign the model in different or more general settings. We see an extreme example of this in studying relationally bottlenecked MLPs, where hand-crafted relations strongly benefit the model specific cases where they align with the task. However, departing even slightly from the ideal task structure prevents the shallow RB MLP from learning the task at all, while a vanilla MLP continues to exhibit strong performance. In the absence of less than vanilla MLPs. As a result, for certain vell-suited tasks (like ICL regression), Transformers perform more efficiently for fixed compute budget. But for other tasks (relational reasoning, simple regression and classification in Appendix B), MLPs have the upper hand.
relations, Transformers are more general learners than RB MLPs, but
These results expand the range of possible tasks commonly thought solvable by MLP models. ICL may not be the exclusive domain of Transformers, and we encourage greater engagement with ICI. beyond -based architectures. The surprising success of MLPs for relational reasoning also encourages a change in perspective about how simple architectures may be capable of solving relational tasks, and under what conditions they fail. The impressive performance of MLPs hints at potential real-world benefits, and we watch the future outcomes of all-MLP approaches with interest.
Limitations and future directions. We consider only controlled, synthetic tasks designed to probe for specific characteristics. We never compare architectures on complex datasets like those found in language vision, though there are other studies that do, and find that MLPs continue to perform competitively (Tolstikhin et al., 2021; Liu et al., 2021; Fusco et al., 2022). The advantage of our approach is that we avoid confounds irrelevant to ICL introduced by complex data, and clarify the specific competencies of each model to learn in-context across representative settings. It remains overall unclear how MI.P-based architectures perform against Transformers scale in complex,

---

Published as a conference paper at ICLR 2025
real-world tasks, and where possible performance gap may originate. Future work should explore MLPs in more naturalistic settings to better understand the potential comparative advantage of attention-basee -aatio schemes.
We also work exclusively in an online setting where models have access to continuous stream of infinite data. As the bitter lesson heuristic predicts, this setup benefits the MLP, but imagine that data-limited scenarios, Transformers and other architectures with stronger inductive bias would dominate. Indeed, we have already observed that Transformers tend to learn in-context with comparatively less data diversity. Examining a data-limited setting represents another important future direction, and will potentially reveal important weaknesses in MLPs.
certainly
Where possible, we test on a wide array of parameterizations and task settings. The main text figures represent only an illustrative subset of our total results, with supplementary figures provided in Appendix D. However, as with any empirical study, we cannot test every infinite variation our models and tasks; there may be further unexpected results hiding behind a setting we have not tried.
Overall, we hope our results encourage further work studying ICL beyond attention-base archi- tectures, and the properties of simple architectures like MLPs that enable them to solve relational tasks. Important questions remain in quantifying how much data diversity is generally required to transition to ICI., how this threshold depends on architecture, varying sensitivity to context length across architectures, precisely characterizing differences in inductive bias for ICL, and more.
Ethics statement. Since this study focuses on synthetic tasks, is limited in direct negative societal impacts beyond that of general theoretical machine learning research. We do not conduct experiments in human or animal subjects.
Reproducibility statement. Full descriptions of all tasks are provided in Sections 2 and 3 of the main text. Additional details regarding specific task and model configurations, along with code availability, compute requirements, and software, comprehensively enumerated in Appendix C.
Acknowledgements. We thank Alex Atanasov, Hamza Chaudhry, Alex Meterez, Mo Osman, Sab Sainathan, Jacob Zavatone-Vethe members the Pehlevan Group, and the anonymous ICLR reviewers for many helpful comments and discussions our manuscript. WLT is supported by Kempner Graduate Fellowship. CP is supported by NSF grant DMS-213413 IIS-2239780, DARPA grant DIAL-FP-OSS
NSF CAREER Award Sloan Research Fellowship, and The William F. Milton Fund from Harvard University. This work has been made possible part by gift from the Chan Zuckerberg Initiative Foundation establish the Kempner Institute for the Study of Natural and Artificial Intelligence. The computations in this paper were run on the FASRC cluster supported by the FAS Division of Science Research Computing Group at Harvard University.
REFERENCES
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXhv:2303.08722 2023.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities large language models just in-context learning? arXiv preprint arXiv:2309.01809 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey in-context learning. arXiv preprint arXiv:2301.002. 2022.
10

---

Published as a conference paper at ICLR 2025
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. International Conference on Learning Representations, 2024.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMLR, 2023.
Ruiqi Zhang, Spencer Frei, and Peter Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306609997, 2023.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in in-context classification task. In International Conference on Learning Representations, 2024.
Yue M. Lu, Mary I. Letey, Jacob A. Zavatone- Veth, Anindita Maiti, and Cengiz Pehlevan. Asymptotic theory of in-context learning by linear attention. arXiv preprint arXiv:2405. / 1751, 2024.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30598. 2022.
Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-bayesian in-context learning for regression. Advances in Neural Information Processing Systems, 36, 2024.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances Neural Information Processing Systems, 1888-18818,,,, 2022.
Declan Campbell, Sreejan Kumar, Tyler Giallanza, Jonathan D Cohen, and Thomas L Griffiths. Relational constraints on neural networks reproduce human biases towards abstract geometric regularity. arXiv preprint arXiv:2309.17366
2023.
Burrhus Frederic Skinner. Are theories of learning necessary? Psychological review, 57(4) 1950.
Mathias Sablé-Meyer, Joël Fagot, Serge Caparos, Timo van Kerkoerle, Marie Amalric, and Stanislas Dehaene. Sensitivity to geometric shape regularity in humans and baboons: A putative signature of human singularity. Proceedings of the National Academy of Sciences, 118(16)e202332311 18, 2021.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
Hanxiao Liu, Zihang Dai, David So, and Quoc Le. Pay attention to mlps. Advances neural information processing systems,
2021.
Francesco Fusco, Damian Pascual, Peter Staar, and Diego Antognini. pnlp-mixer: An efficient all all-mip architecture for language. arXiv preprint 2rXiv:2202.043555 2022.
Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Sealing mips: A tale of inductive bias. Advances in Neural Information Processing Systems, 36, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (0X0:22000000361,, 2020.
Jingfeng Wu, Difan Zou, Zixiang Chen, Vladimir Braverman, Quanquan Gu, and Peter L Bartlett. How many pretraining tasks are needed for in-context learning of linear regression? In International Conference Learning Representations 2024.
11

---

Published as a conference paper at ICLR 2025
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. Advances in neural information processing systems, 36, 2024.
Yingcong Li, Muhammed Emrullah Iidiz, Dimitris Papailiopoulos, and Samet Oymak. Transformers as algorithms: Generalization and stability in in-context learning. In International Conference Machine Learning, pages 19594, PMLR, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2021.
Kabir Ahuja, Madhur Panwar, and Navin Goyal. In-context learning through the bayesian prism. In International Conference on Learning Representations, 2024.
Ivan Lee, Nan Jiang, and Taylor Berg-Kirkpatrick. Exploring the relationship between model archi- tecture and in-context learning ability. In International Conference on Learning Representations, 2024.
Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kang- wook Lee, and Dimitris Papailiopoulos. Can mamba learn how learn? comparative study in-context learning tasks. arXiv preprint 242.04442,04448, 2024.
Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019.
Atticus Geiger, Alexandra Carstensen, Michael C Frank, and Christopher Potts. Relational reasoning and generalization using nonsymbolic neural networks. Psychological Review, 130(2):308, 2023.
Taylor W Webb, Steven M Frankland, Awni Altabaa, Kamesh Krishnamurthy, Declan Campbell, Jacob Russin, Randall O' O'Reilly, John Lafferty, and Jonathan D Cohen. The relational bottleneck as an inductive bias for efficient abstraction. arXiv preprint arXiv:2309.06629, 2023.
Raquel G Alhama and Willem Zuidema. review computational models of basic rule learning: The neural-symbolic debate and beyond. Psychonomic bulletin & review, 26: 1 174-1194, 2019.
Enric Boix-Adsera, Omid Saremi, Emmanuel Abbe, Samy Bengio, Etai Littwin, and Joshua Susskind. When can transformers reason with abstract symbols? arXiv preprint arXiv:2310.097733, 20233.
Gary Marcus. Rethinking eliminative connectionism. Cognitive psychology, 37(3):243 1998.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis.
Cognition, 28(1-2):3-71, 1988.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint / 1806.0/261, 2018.
Andrew Ng. Feature selection, 1 vs. 12 regularization, and rotational invariance. Proceedings of the twenty-first international conference on Machine learning, page 78, 2004.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. arXiv preprint 13Xiv:1301.3781, 2013.
Grigory Khromov and Sidak Pal Singh. Some intriguing aspects about lípschitz continuity of neural networks. arXiv preprint 23X.::2302. /0886, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Gary F Marcus, Sugumaran Vijayan, Shoba Bandi Rao, and Peter M Vishton. Rule learning by seven-month-old infants. Science, 283(5398):77-80. 1999.
12

---

Published as a conference paper at ICLR 2025
Junkyung Kim, Matthew Ricci, and Thomas Serre. Not-so-clevr: learning different relations strains feedforward neural networks. Interface focus, 8(4):20180011, 2018.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-t---sqqueeee recurrent networks. In International conference on machine learning, pages 2873-2882. PMLR, 2018.
Thomas Serre. Deep learning: the good, the bad, and the ugly. Annual review of vision science, 5: 399 426, 2019.
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Iliia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- Milne,, and Qiao Zhang. JAX: composable transformations of Python Python programs, 2018. URL httt:///ithbp coom/gopg le/ jax.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL
http://git thub. com/ goo/googl e/ flax.
Michael L. Waskom. seaborn: statistical data visualization. Journal of Open Source Software, 6(60); 3021, 2021. doi: 10.21105/joss.0230
URL https::/doi..oooo/ 2111:10...111111111 030. 03021.
The pandas development team. pandas-dev/pande Pandas, February 2020. URL https: /doi.
org/10. /eeodooo . 3509134.
13

---

Published as a conference paper at ICLR 2025
A MLPs REASON RELATIONALLY
Relational reasoning is closely related to in-context learning, as solving ICL tasks requires reasoning about the relationships between inputs while ignoring their absolute characteristies. Indeed, the relational tasks we explore in the main text are functional subsets of ICL classification. At the same time, MLPs are commonly presumed to fail at relational reasoning (Marcus, 1998; Boix-Adser et al., 2023) or exhibit severe weaknesses (Fodor and Pylyshyn, 1988). While our primary focus remains on comparing in-context learning between Transformers and MLPs, offer this digression to contextualize our results within the broader relational reasoning literature.
The question of whether connectionist models able reason relationally at all has been an enduring topic of passionate debate (see Alhama and Zuidema (2019) for a recent review). Our empiries support the notion that classical architectures like vanilla MLPs can indeed reason relation- ally, consistent with recent findings in Geiger et al. (2023). However, many researchers presuppose that classical architectures cannot solve relational tasks, resulting in a zoo of alternatives aimed at endowing neural networks with relational capacities (Webb et al., 2023; Battaglia et al., 2018; Geiger et al., 2023; Alhama and Zuidema, 2019).
One especially strong claim that MLPs cannot reason relationally was advanced by Boix-Adsera et al. (2023), who formally prove that MLPs will never generalize to unseen symbols on relational tasks. Their proof however relies on pathological input scheme that hinders learning. Below, we discuss their analysis on MLPs and offer our own remarks on the learnability of relational tasks. We also demonstrate empirically that, under conventional settings, MLPs do generalize relational task claimed by Boix-Adsera et al. (2023) to be impossible.
A.1 SUMMARY OF BOIX- ADSERA ET AL.
We begin with brief summary of the relevant theorem in Boix-Adsera et al. (2023). Consider template z consisting of a sequence of wildcards
Or 012-2-20 Og E W
A string x consisting of symbols x = 01 Ty E X satisfies = there exists an injective map W X' such that s(a:) = 2; for all which we call substitution map. Finally, we have labeling function f" W + that assigns scalar labels to different templates.
concrete example of this setup is the same-differen task often used in probing relational reasoning (Geiger et al., 2023). Here, we consider two templates 21 aa and 22 = ox,3. The labeling function is (=12)) and I (22) = The string
AA matches template 21, and the string x'' = AB
matches template 22. We will abuse notation slightly and write (x) = (x1) = 1. Crucially,
matching a template depends only on relations between symbols and not on the symbols themselves. For example,
("(CC) = (88) = f ([platypss[pllatyus)] = 1 ([platypus]
[kangaroo)) ([platypus|kaaggrrooo]
regardless of the concrete meaning the symbols.
Boix-Adsera et al. (2023) consider MLP models with one-hot encodings of symbols
E(x) (Ez, Coss ..., eza) E(x)
where Cz, |R|x|| is vector with at an index corresponding to x; and 0 elsewhere. The one-hot encodings are then flattened Hat(E(()) e 4 before being passed directly into an MLP.
For notation, write fMu 0¹) an MLP applied to string with parameters 0' obtained after I steps of stochastic gradient descent (SGD). We denote Xuns as symbols that are unseen during training, and Xzeen symbols that are seen during training. The theorem is stated as follows
Theorem A.1 (From Poox-AAseea et al., failure MLPs at generalizing on unseen symbols). Suppose the label function I is non-constant. Then for all SGD steps there exists template z E W and a string x consisting of symbols I 2122.... ... E une which satisfy z such that
Eg (fmpp(;; 0¹) (z)) 2 c>0 0
where e is a constant that depends only on f³, and the expectation is taken over random initialization of parameters O and subsequent SGD steps.
14

---

Published as a conference paper at ICLR 2025
Their proof relies the permutation invariance property of MLPs and SGD (Ng, 2004). Summarizing briefly, they argue that if I1, Z2 E Xuns. we can swap their one-hot encodings without any impact on the ML.. More generally, we can construct permutation matrix IT [ X such that for all strings of symbols =(1), flat (a((())())) and II fflatEE(z
E X and x' E X Reen it remains true that II flat flattEE(a E(xx()) flat (x))). That is, we permute only the indices of unseen symbols, but leave the indices of seen symbols untouched. Then given the permutation symmetry of MLPs and SGD (Ng, 2004), because we preserve the indices of the seen symbols, we must have that
xx()) x2)
Eg fMLP (ze 0²) Egg fMLP
(x (2); 0*)
Hence, if f(((()) # fx(())) the MLP cannot approach arbitrarily close to both labels, incurring an irreducible cost 0 that depends on the difference. In this way, MLPs cannot generalize to unseen symbols on any relational task.
A.2 A DIFFERENT INPUT SCHEME
Is there a way to circumvent this impossibility result? One aspect of the proof that may seem suspect is its reliance on flattening one-hot encodings flat( E(æ))) direct input to the MLP. Going far back as Word2vec (Mikolov et al., 2013), a well-established convention for processing one-hot inputs is to instead pass them through embedding matrix W creating vector embeddings
(x ee, W W.C....... ee,,,)T hz(()) E R
where d is the dimension of single symbol's vector embedding. 5 A practitioner then flattens and operates on the resulting vector embeddings, not the one-hot encodings directly. As we will shortly see, if we consider the more conventional input scheme that uses vector embeddings ho and not the one-hot encodings directly, then the conclusion from Boix-Adsera et al. (2023) no longer holds.
In particular, consider an architecture where the input to the MLP is flat(ho (æ)), rather than (x)) We can attempt the same logic as before, and identify a permutation II such that for all strings of symbols (1),, (² E X and x' E X seen. we have that 1I flat(ho(==!)))
Unfortunately, if the embedding matrix W. is randomly initialized like most neural network parameters, it is virtually impossible to find permutation where TT (l)
flat(ha (x((x)))) and II (1ze())(z²)) = fla(ho((x')
(2:(())))))) while (1) o 22 This is because the probability that any two elements of W. are identical is zero for typical random matrix ensembles
used in practice, e.g. if the elements of W. are sampled i.i.d from a normal distribution.
Hence, it is clear that the original proof strategy of permuting the input, now flat(ho (x))) has become unviable. However, a skeptical reader might now wonder whether Theorem A.1 might still be saved if we apply permutations to the one-hot encodings before they are passed to the embedding matrix. That is, given a permutation matrix II E [R| x[X), we construct
h (z) = (W, (W.(Te,, (Ie,)) W. WIe.......... W. (Ie,,,))
In this way, Theorem A. might still be rescued through permutations on one-hots before the embedding matrix. This method sidesteps the issue with permuting flat(ho (x)) directly, and the MLP trained on SGD remains invariant to any permutation on the underlying one-hots. Hence, seems the proof may remain valid, and the impossibility result might still holds.
Unfortunately, this scheme runs into a different issue: it is impossible to find two inputs x (')) (2)) where (1) = (2), but that f (x) * (2)())). (Note, we have abused notation slightly and write lle,- = 1Ix.) Indeed, we next show that if æ satisfies template z, then any permutation II on the symbols of æ will also satisfy z. This can be seen quite simply by considering that 1) by definition, template satisfaction is invariant under a relabeling of symbols and 2) any permutation is a relabeling of symbols- - hence, template satisfaction must be invariant under permutation. We phrase this formally below.
Proposition A.1 (Permutation invariance of template satisfaction). For any template E Wk and any permutation II: : X - X, if the string x satisfies z. then Hæ also satisfies x.
$ Indeed, in their results on Transformers, Boix-Adsera et al. do use vector embeddings. It is unusual they would choose omit them in their analysis of ML.Ps.
15

---

Published as a conference paper at ICLR 2025
Proof. If symbols 21 ....... Dj satisfy the template 0102.- .k. then there exists an injective substitution map s such that s(a,) = z₁. Because permutations are bijective, there must also exist injective substitution map s' such that s' (aa) II(x;). Hence, Il satisfies the template 2.
In this way, it is not actually possible to find two strings x x01) xx(2) such that IIx = x but for which (x(1) # (x(22) since they both satisfy the same template. Permuting over one-hot could try permuting the output from an intermediate layer of the MIP, but this will fail for the same reason that permuting flat ho (a)))
encodings before the embedding matrix is not viable. Alternatively,
failed. All in all, if we replace the input flat (x)) with the more conventional (at(ha(=)), Theorem A.1 is no longer valid.
A.3 CAN MLPS REASON RELATIONALLY?
We have argued that the impossibility theorem of Boix-Adsera et al. (2023) can be circumvented, but it remains to be seen whether MLPs can truly reason relationally. We next identify coarse conditions that would in principle allow an MLP to generalize to unseen symbols given finite training data. Intuitively,
imagine that if the MLP's training data is sufficiently diverse and the model is sufficiently expressive and smooth, then any unseen input æ will fall somewhat close to a seen input x'. fMLP(w) z fMLP(Z' (x) f (x'). ). If æ and x' are labeled the same (not unreasonable, if they are close), then the MLP would indeed be generalizing on an unseen input example.
We formalize this intuition in the following proposition, which establishes coarse conditions for model f to generalize on unseen input. We adopt the same setting as above, but we now treat strings as real-valued E F". This is equivalent flattening the vector embeddings hz(x) generated from one-hot encoded symbols x1 ... Tg. Doing so simplifies the following discussion.
Proposition A.2 (Conditions for generalizing to unseen inputs). Fix and select < E/3. Given a model f RD - I and labeling function R IR, they satisfy the following three conditions
1. Smoothness: f and f are L- L-Lipschitz continuous
2. Expressivity: for all a that seen, (f(x) f*(x)) < 6.
3. Data diversity: for all x' that unseen, there exists that is seen such that | x'|l L/L.
then
If(x)) (x) <E
for all x (seen and unseen).
Proof. This statement is a simple consequence of the triangle inequality. For any unseen x' in the 6/ L-neighborrhood of seen x, we have that
f(x)) - (x)) - -(aa) If(x') f (w')1 S -())))
< 35.
Hence, if we select < </3, we must have that [f(x²) (x')) <e
In this way, if a model satisfies the above three conditions, generalizes to unseen inputs for task defined by the labeling function f*. The first condition for smoothness is regularly achieved by standard neural networks (Khromov and Singh, 2023). The second condition corresponds to a notion of expressivity - that is, model f should be able to approach arbitrarily close to zero on its training data. For modern neural network models trained on simple tasks, this a frequent occurrence (Zhang et al., 2021). The third condition corresponds to coarse description of data diversity. The training data should be sufficiently diverse such that all unseen examples are very close to an example seen during training. This condition may be difficult to achieve in practice, but it offers very coarse upper bound on the requisite data diversity required to generalize on unseen examples. Nonetheless, an MI MP trained online suitably constrained input space may very well achieve this condition.
16

---

Published as a conference paper at ICLR 2025
1.0
Accuracy on Xuts
0.9
0.8
0.7
0.6
0.5
26
2
210
2
2
[&]
Figure MLP accuracy on unseen symbols for the same-different task. The gray dashed line indicates chance-level performance. Shaded region indicates 95 percent confidence regions estimated from 5 replications. For higher data diversity (i.c. number of symbols in the task), the MLP generalizes progressively better. Beyond roughly 2 symbols in the task, the MLP performs substantially above chance, and approaches perfect generalization beyond 2 symbols.
Given further assumptions on I (e.g. ʃ an MLP), it is likely possible shrink this data diversity bound considerably.
Regardless whether an MLP achieves these conditions exactly, we next show that with sufficient data diversity, MLP equipped with an embedding matrix and trained through gradient descent does solve a relational task the form posited in Theorem A.I, generalizing perfectly to unseen data.
A.4 SAME-DIFFERENT TASK
We now demonstrate empirically that vanilla MLP trained with gradient descent will discover solution that generalizes to unseen symbols. While the tasks we explore Section already indicate that MLPs solve relational tasks, to address any remaining doubt, we pursue ostensibly impossible task as specified
Roix-Adsera et al. (2023). particular, we examine the same-different task.
As noted in Appendix 1. the safe-different task consists of two templates, with labels f (aa) = and (a33) = O. Following Boix-Adsera al. (2023), we consider input strings 21 2 that one-hot encoded before being passed through embedding matrix. As previously discussed, this embedding enables the model to circumvent the impossibility result (Theorem 1) from Boix-Adsera al. (2023). The remainder of our model is exactly the MLP described in Appendix C. We use an MLP with 4 hidden layers (in addition to an embedding layer) all of width 256 dimensions. The MLP trained on batches of 128 examples sampled from a set of symbols with varying size |X) with roughly even positive and negative examples. In each run, the MLPs are first trained for 10, 000 batches on half the symbols X. then tested on the other half. All remaining hyperparameters are specified in Appendix C.
and 22 aB,
We plot the performance of an MLP on this task Figure 3. For this task, data diversity refers the number of symbols in X. With higher data diversity, we see that the MLP improves progressively at generalizing on unseen symbols. Beyond about 212 symbols, the MLP generalizes pear-perfectly. confirming that these models do, indeed, learn to reason relationally. This result is particularly interesting because it shows that, with sufficient data diversity, the MLP generalizes to completely novel symbols.
A.5 DISCUSSION
Our results are consistent with Geiger et al. (2023), who also find empirically that MLPs (among other architectures) reason relationally and generalize robustly to unseen inputs. We complement their results by further evidencing the possible conditions where MLPs may continue to generalize successfully. Geiger et al. (2023) argue that neural networks require "non-fcatural input represen- tations" to generalize. A representation featural if it encodes interpretable features of the task in axis-aligned dimensions. One-hot token encodings featural, but randomized encodings are not.
17

---

Published as a conference paper at ICLR 2025
As in Geiger et al. (2023), show that featural representations like one-hot encodings remain usable provided they that pass through an embedding matrix, becoming non-featural and circumventing the impossibility result found by Boix-Adsera et al. (2023). In this way, with sufficient data diversity, an MI.P still generalizes to unseen inputs, even if the inputs are unseen one-hot encodings.
Despite success above, many earlier studies document cases where common neural network architectures fail to reason relationally (Marcus et al., 1999; Kim et al., 2018; Lake and Baroni, 2018; Alhama and Zuidema, 2019). One important reason for the failure may be that the task inputs are very large and complex, as in visual reasoning (Kim et al., 2018; Serre, 2019). Proposition A.2 suggests that the data diversity required for successful generalization scales exponentially with the dimension of the inputs in the worst case. It is possible that given sufficiently vast dataset, an MLP would perform well on visual reasoning tasks. Furthermore, having shown above that MI.Ps are decisively capable of relational reasoning (especially when presented with idealized stimulus embeddings, as in Section 3), their failure on complex tasks highlights a need to separate model's ability to reason relationally from its ability to learn sufficiently rich feature representations. In realistic data-limited scenarios, perhaps an MLP paired with a more bespoke module for feature learning would reason quite successfully. We anticipate further work that more closely investigates whether these failures stem from data limitations, insufficient feature learning, or some other cause, thereby building more complete and updated picture of relational reasoning in neural networks.
B EXPERIMENT: SIMPLE TASKS
In the main text, we showed that MIPs perform comparably with Transformers on ICL regression and classification, and better on relational tasks. In this separate set of experiments, we examine setting in which MI.ss are decisively superior. To do so, we depart from in-context tasks and consider simple (non-ICL) regression and classification.
B. SIMPLE REGRESSION
Following the classic regression setup, the model receives as input single point æ E R", and must output the corresponding y E which is related through y x: ,3. Note: this is not in-context regression, so the model receives only a single input æ and the weights ,3 remain fixed throughout the duration of the task. For the Transformer, unless otherwise stated, each input coordinate is processed "token" with depth 1. Additional details are provided in Appendix C.11.
Results. In Figure 4a, we plot the MSE of vanilla MLPs and Transformers function of compute on 64 dimensional regression. The gap between the two models substantial. The Transformer seems to struggle especially for larger inputs. For smaller the compute gap shrinks between MLPs and Transformers (Figure 10). your are stuck with large n, one potential strategy for improving the Transformer's performance is to manually chunk the inputs into larger tokens, reducing the total number of tokens. In the extreme case, we chunk the entire input into a single token (effectively transposing the input). As the token size increases, the Transformer's effiency smoothly improves until it reaches a level comparable to the MLP (Figure 4b). Indeed, in the extreme case of a single input token, the Transformer is almost identical to an MLP anyway.
B.2 SIMPLE CLASSIFICATION
We next consider classic classification setup. The model receives a single point E IR" that was sampled from of & different clusters. The model must output the correct label y of the corresponding cluster. This is not in-context classification, the model receives only single input and the cluster/label mapping remains fixed throughout the duration of the task. Additional details are provided in Appendix C.12.
The same results continue to hold. As shown in Figures 4(c,d), for 64 dimensional classification, there is a wide compute gap between a vanilla MLP and a Transformer model, though the gap can be narrowed by manually chunking the inputs into larger tokens. Figure 10 gives performance for inputs of different dimensions, where smaller n narrow the gap between the two models.
Results.
18

---

Published as a conference paper at ICLR 2025
Regression
Classification
a
Tiken Sze
10"
M35
MLPP Thansformrr
MSE
Loss
Loss
10-1-
10a3
MLP Trasformer
10
10
10
10t 10-3
10-2 10-4 Compute (PFLOPS)
10
10 (PFLOPS)
(PFLOPS)
Compute (PFLOPS)
Figure 4: Simple regression and classification results. (a) MLPs attain substantially lower MSE at lower compute than Transformers. The red line corresponds to the minimum attainable MSE. (b) Transformers attain performance given larger token sizes. (c, d) Same as in (a, b), for classification, with & 16 clusters. (all) We in Appendix D. Shaded regions correspond to 95 percent confidence intervals estimated from replications.
64 dimension inputs. Other parameterizations are explored
B.3 DISCUSSION
Evidently simple tasks with long inputs work against the Transformer's attention mechanism. Short- ening the context by reducing the task dimension, chunking inputs into larger tokens, bypassing the attention mechanism altogether by stacking the input into a single token all improve the Trans- former's efficiency. It is immediately obvious why the Transformer performs dramatically worse compared to the MLP for larger n,, though it is well-known that Transformers can struggle with long inputs (Tay al., 2020).
C MODEL AND TASK CONFIGURATIONS
In the following appendix, we provide all details on the specific model and task configurations used this study, including architecture, hyperparameter settings, training methodology, and more.
C.1
CODE
For the most precise information on setup, please refer to our GitHub code repository:
https://ghtt ttt.b cwm wwtp://tttggggg//oo 9
There, you will find all code used to reproduce the plots in this document, as well any minor implementation details omitted from this appendix. you notice an error, we welcome your pull requests!
C.2 MLP
The MLP accepts inputs æ E R", If task provides inputs of shape L x (length by token depth), the inputs first flattened size LDD before being passed the MLP. A model with & hidden layers then proceeds as follows:
h = @ (Waa a - -b1)
h2( o (W 22) + bz)
h Φ (W (W che- + bb)
[MP(#) W out b
For all tasks, ReLU activation functions applied pointwise d(x) max(æ,, 0). Widths of all hidden layers are fixed to the same value H. As with all models, all training examples are presented online with batch size 128. Training uses AdamW (Loshchilov and Hutter, 2017) with learning rate = 10 and weight decay = x 10 The hyperparameters used to train MLPs on each task presented in Table 1.
19

---

Published as a conference paper at ICLR 2025
Table MLP hyperparameters
Task
Depth (t)
Width (H)
Train iterations
ICL regression
2 8
128 124-
048, 000
ICL classification
2 - 8
64 1024
S 128, 000
Simple regression
1 4
256
S 64, 000
Simple classification
1 1-4
4-
S 64, 000
Match-to-sample
1-4
256
S 8, 000
Sphere oddball
1-4
256
S 8, 000
Line oddball
1-4
256
8, 000
C.3 MIXER
The MLP-Mixer accepts inputs X LR (length by token depth). If a task does provide tokenized inputs, we assume D = unless otherwise stated, and reshape accordingly. A model with é hidden layers then proceeds follows:
h1 (X) of (Z XW +.)
h2 (X) ol (Z h1(X)W2) + c2)
he (x(X) Φb (Z,(b] + hx-1 + ce)
f MIX (X)
W Wo + bout
The matrices W mix within token dimensions, and share fixed hidden width H, so W; E HxH for 1 < < L. The matrices Z mix across spatial dimensions, and share fixed channel width C, so Z, E RCXC for < < The bias vectors b and c are assumed to broadcast over unit dimensions expected. The index -1 in (x (-)(-1 refers to taking the last token in the layer, producing output vector with length H. We again use point-wise ReLU activations Φf [X) max (X, 0). Our Mixer is simplified version of the original model proposed in Tolstikhin al. (2021), and differs in number of small ways:
We use only a single hidden layer per Mixer layer, rather than two.
We apply the point-wise activation after the final spatial mixing, and not between spatial and token mixings.
We do not use layer norm or skip connections.
Using the full original model proved to be unnecessary in our setting, so we proceeded with this simpler version.
As with all models, all training examples are presented online with batch size 128. Training uses AdamW with learning rate used to train MLPs on each task are presented in Table 2.
10-4 and weight decay À 10 4, The hyperparameters
Table 2: Mixer hyperparameters
Task
Depth (f)
Hidden width (H) Channel width (C)
Train iterations
ICL regression
2
32- 512
64
s 500, 000
ICL classification
2 8
16 256
64
< 24, 000
C.4
TRANSFORMER
The Transformer accepts inputs X E R (length by token depth). a task does not provide tokenized inputs, assume D unless otherwise stated, and reshape accordingly. A model with
20

---

Published as a conference paper at ICLR 2025
é hidden layers then proceeds follows:
X X + PE(X)
a (X) = LN XV, X)
h1 (xX LN(cI [bb} (X) x)
ag () =
X)
h2 (X) = LN( + [bb] + a (X) (x) ww? +X)
fRR (X) W h, (X) + bout
The attention matrices A, are Lingl-hhedded and constructed as
A, = mask ( (Q.X))(()))
where "mask" corresponds to causal attention mask, and refers to softmax applied per query. As now popular, we use GeLU activations applied pointwise for Φ. We fix the hidden dimension across all key, query, value, and weight matrices be of width H. We use sinusoidal positional encodings for PE and layer normalization as indicated by LN. One exception is for ICL regression, which does not require positional encodings due the input format (Appendix C.6), so they omitted in this case. The bias vectors b and c are assumed to broadcast over unit dimensions as expected. The index -1 in h (X)(-1) refers to taking the last token in the layer, producing an output vector with length H. Our architecture is precisely the decoder-only Transformer architecture first described in Vaswani et al. (2017), with the exception that we do not use dropout.
As with all models, all training examples presented online with batch size 128. Training uses AdamW W with learning rate a = 10-4 and weight decay X = x 10-4 The hyperparameters used to train MLPs each task presented in Table 3.
Table 3: Transformer hyperparameters
Task
Depth (6)
Width (H)
Train iterations
ICL regression
2
-12
S 600, 000
ICI classification
2 8
16 16-256 256
16, 000
Simple regression
1 4
8-32
S 256, 000
Simple classification
1-4
8-32
S 128, 000
Mateh-to-sample
1-4
8-32
S 8, 000
Sphere oddball
1-4
8-32
S 000
Line oddball
1-4
8-32
S 88, 000
C.5 RB MLP
The caariinnlll-bottteeneete MIP is architecturally identically to the vanilla MLP described above in Appendix C.2, but with the crucial difference that the inputs are preprocessed to preserve only (dot-product) relations.
The RB MLP accepts inputs X E LRD (length by token depth). The inputs are processed into relation matrix R such that each entry
Rij = (x; T) (x; Øt
where x; E RDD refers to the ith row of X, and क = - E, æ; is the average across all æ; Relations R (as in the MTS task) or
vectors r are then generated by either selecting a specific column flattening the entire matrix of relations r = (lat The output of the RB MLP is then simply
fRB W out boat
For the "deep" RB MLP used in the line oddball task, there is additional set of two hidden layers between and the readout weights W with width 256. All other training parameters are equivalent to the above models.
21

---

Published as a conference paper at ICLR 2025
C.6 ICL REGRESSION
We prepare in-context regression in a setup that closely mimics Raventós et al. (2024), though without an autoregressive objective. The input consists of a sequence of values (x1. 31); (02, 122,.... (LL, VL), where 2; E Rn and yi E R. The 24 pairs are linearly re- lated through set of weights noise. Finally, the input includes a query 2q. The model output is single scalar regressed against the corresponding V/-- Inputs are sampled as æ ~ NNO, I) and weights are sampled as , ~ N (0, I/n). Before being presented to the model, all inputs are packed into an imput matrix X E ([++1)x(++1) with the following structure (Zhang et al., 2023)
R" such that y x; - where E g~ (O, o²) corresponds to
X
Ø1 T2 y/1
LL V
=
V/
The model returns scalar value estimate of yy, and is trained using the nean-squared-error, Note: this format does not require positional encodings. Following Zhang et al. (2023), we omit positional encodings for this task.
As in Raventós et al. (2024), we fix finite pool of weights before training 31. By, where 3 ~ (O, I/n). For each training example, we sample new by selecting uniformly random one weight from the pool [B,) We also consider the limit & - 0C, which corresponds to sampling ~ AO, I/n) afresh rather than drawing from fixed pool. During testing, we probe the model's performance both on the training distribution where the weights are restricted to a
finite pool / u (114)==) and an unrestricted distribution where the weights are drawn freely G (0,
Unless stated otherwise, all of our experiments use n = 8 dimensional regression with L = 8 points in the context, and noise level o² 0.05.
Bayes estimators. We compare our models to two different Bayes estimators that correspond to priors assuming finite or infinite k.
For finite h where weights 3 are sampled uniformly from a pool of k possibilities, the Bayes optimal estimator given by the discrete minimum mean- squared error (dMMSE) estimator, based on the estimator formulated in Raventós et al. (2024)
k
AMMSE
w.B
where the weights w; are given by
Σ (y)
2. 3,)²
Wi exp
20
j=l
normalized such that E, W0 1.
In the case k + oo, the Bayes optimal estimator is simply the familiar Ridge estimator with Bayes optimal regularization
BRide (xT + no=21 -1 XTy
where the rows of X are the context points, and y = (3/1, U2....... VL) are the corresponding labels.
C.7 ICL CLASSIFICATION
We prepare ICL classification in a setup that closely mimics Reddy (2024). We begin with a set of labels ar, 0k2, ... aC R" that correspond to class indices 1,2, C. Labels are sampled as a ~ (0, I/n). The model ultimately predicts the class index, but the real-valued labels provide content of the correct dimension to fill an input without arbitrary padding (described further below).
Points are sampled from a Gaussian mixture model M₁ consisting of k components, where A 2 C (we allow multiple clusters to have the same class label). Each component is associated with center
22

---

Published as a conference paper at ICLR 2025
M4 ~ ~40, A point is sampled from the kth component as
14 + Zyk v1 6
where ~ (0, and governs the within-cluster variability. Below in Figure 8, we also consider k + oo setting, where the number of mixture components is infinite. This settings corresponds to case where the mixture centers br are resampled for each example, always producing novel clusters. In the finite case, mixture centers remain fixed throughout the duration of the task.
An input sequence consists of L context exemplars (11, (22, Uy)...., (æ vr) followed by query point zg where æ; M₁ and y, E originated the point. The model must predict the corresponding query label y. and output the class index associated with this label. The imputs are packed into an input matrix X TR (2L+1)) which has structure
{a²} is the corresponding label for the cluster that
x
(
V1
222
y22
yL Fgg)
The model outputs logits over class indices, and is trained using cross-entropy loss.
We also parameterize the inputs by burstiness B, which is the number of repeats per cluster in the context (B must divide the context length L). For example, B means there are exactly two points from each cluster represented in the inputs.
Unless otherwise stated, we use dimensional inputs, C 32 class labels, and within-cluster variability e = 0.1.
C.8 MATCH-TO-SAMPLE.
The match-to-sample task proceeds as follows. The model is presented with context points T1, T2... ................................ #L E R followed by query point Tq. The inputs packed into input matrix X
(1), 22.......................... Lb, xx) ([L+1)xx before being passed to the model.
The context points are evenly distributed along sphere S with unit radius centered at the origin. Points are rotated by a random angle so that their absolute positions vary from input input. The model must return the index of the context point closest to the query y = arg max, (æz xg), and is trained using cross-entropy loss.
Unless otherwise stated, we use L. 5 context points and 2 dimensional inputs.
C.9 SPHERE ODDBALL
The sphere oddball task proceeds as follows. The model is presented with L context points I1, .2,...., #1 E R". (There The center is sampled uniformly from a box l ~ 2 [-B, B)" One point the context is selected at random and perturbed in random direction with magnitude The model must return the index of the oddball point in the context, and is trained using cross- entropy loss. Both the center i and points 2; are sampled afresh from example to example, necessi- tating a general relational solution.
query points.) The context points are sampled
~ NM,, I).
|| so that Foddhall Foddhall t.
Unless otherwise stated, we use n dimensional inputs, points in the context, and a box size of B = 10.
C.10 LINE ODDBALL
The line oddball task proceeds as follows. For each training example, we select an n dimensional plane with random orientation that passes through the origin. Context points 1 Ø2..... æt I are Gaussian distributed along this subspace with zero mean and unit variance. One context point is selected random to be the oddball, and is perturbed by distance in the direction perpendicular to the line. The model must output the index y of the oddball point, and is trained using cross-entropy.
Unless otherwise stated,
dimensional inputs and I points in the context.
23

---

Published as a conference paper at ICLR 2025
C.11 SIMPLE REGRESSION
Simple (non-ICL) regression is the classic regression setup. The model receives as input a single point æ R", and must output the corresponding y E R which related through y x Weights are sampled as B ~ (0, I/n), and noise is sampled as & ~ [O0 o²). Weights B are sampled once, then remain fixed through the entire duration of the task. The model trained using MSE loss.
Unless otherwise stated, consider 64 dimensional regression with noise level o² 0.05.
In Appendix D, we also consider simple non-linear version of regression where y = (æ 3) + e for powers p and 3.
C.12 SIMPLE CLASSIFICATION
Simple (non-ICL) classification proceeds as follows. The model receives input single point E R" that we sample from different clusters. Cluster centers μæ are sampled as μ4; ~ (O,,I/n) The label y of x is given by
= arg min | 14:11
Cluster centers are sampled once, then remain fixed throughout the entire duration of the task. The model is trained using cross-entropy loss.
Unless otherwise stated, we consider dimensional inputs with classes.
C.13 COMPUTE
To measure the number of floating point operations (FLOPs) used to train model, Jax's cost analysis routines. Specifically, we compute the total number of FLOPs required to perform a single step of gradient descent, then multiply this quantity by the total number of gradient steps used train the model.
All experiment were run the Harvard FASRC research cluster. CPU requirements negligible compared to GPU time. so they are omitted. All experiments required no more than 16 GB of RAM. The per-experiment GPU time on an A100 generate the above figures are estimated at
ICL regression: 1500 GPU hours
ICL classification: 500 GPU hours
Simple regression: 50 GPU hours
Simple classification: 50 GPU hours
Mateh-to-sample: 10 GPU hours
Sphere oddball: GPU hours
Line oddball: 10 GPU hours
The total GPU time is therefore roughly 2130 GPU hours. The compute used to generate these results represents less than 5 percent of the total compute deployed through the life-cycle of this research project.
C. SOFTWARE
All models are implemented and trained using the Jax (Bradbury al., 2018) family of libraries, particularly Flax (Heck et al., 2023). Plots are created using Seaborn (Waskom, 2021) and Pandas (pandas development team, 2020).
24

---

Published as a conference paper at ICLR 2025
D ADDITIONAL FIGURES
a
ICL Regression
Excess MSE
1.0
MLP
MSE
0.5
Mixer
0.8
Transformer
0.6
0.0
10
10
10
2
6
Compute (PFLOPs)
L
C
Finite Task Distribution
Unrestricted Task Distribution
0.6
1.5
MLP
Mixer
MSE
MSE
Transformer
1.0
dMMSE
0.2
Ridge
0.5
7
10
4
7
k
k
Figure 5: ICL regression with autoregressive objective. For each input example (x), y1, 22 1/,....................... LL, )r), we compute the autoregressive loss E, L(f((11,)1 V1, 22, ..,) ax,).u ys). for neural network f and MSE loss L. For vanilla MLPs and Mixers, mniable-length inputs are handled by padding inputs with zero up to the max length L. (a) Compute vs. MSE on the unrestricted task distribution. Each point represents single model, with particular parameters and training iterations. Just as in the fixed input length case, at large compute, MSE is approximately equal across all architectures. The red line corresponds the Bayes optimal Ridge MSE. (b) Excess MSE (MSE above Bayes optimal) for varying context length L on the unrestricted task distribution. Excess MSE remains flat for Mixers and Transformers, but rises for MLPs. The grey line corresponds to the excess MSE incurred by the zero predictor. Given compute limitations, we plot on slightly narrower range context lengths, but the overall trends remain consistent with the finite-input-lnntth case. (c, d) IWL to ICL transition with increasing data diversity. We train a finite distribution with & weights, then test on both the finite training distribution and the unrestricted distribution. Just with finite input lengths, all models exhibit transition from IWL (represented by dMMSE) to ICL (represented by Ridge) as & increases. Note: it is possible to "outperform" " Bayes optimal Ridge the finite training distribution by learning in-weight the underlying (3's. (all) We use dimension inputs. All line plots feature 95 percent confidence intervals about the mean, estimated from 5 replications.
25

---

Published as a conference paper at ICLR 2025
MLP
Params
10t*
1.00
Mixer
Params
10ah
10a
1.00
Transformer
Params
1053
1045
0.75
0.75
1053
0.75
1054
10
0.50
10t2
1053
10
10-2
10P
10-3
10
10
10"
10
10a
Compute (PFLOPS)
Compute (PFLOPS)
(PFLOPs)
b
Excess MSE
n=
10a
10a
10a
Excess MSE
n=
8
16
Train
10aa
Mixer
1012
102
10
10a
10
102
101
10
10th
10
Transformer
Excess MSE
n-2
16
10a
102
10-
10a
10a 102
10a
10a
Figure 6: ICL regression supplementary figures. (a) MSE obtained across each architecture as function of compute. Lines connect common models, with colors denoting different parameter counts. Hence, single line traces the trajectory of model across different training iterations. The red dashed line corresponds to the Bayes optimal MSE. (b) Excess MSE across different context lengths T., for different input dimensions n. Line colors indicate the number of elapsed training steps. The gray dashed line corresponds to the MSE obtained from always guessing zero. Particularly for high dimensions, MLPs struggle to learn in-context with larger context lengths. After sufficient training, both Mixers and Transformers can learn in-context even for very large input contexts. (all) Shaded regions correspond to 95 percent confidence intervals computed across replications.
26

---

Published as a conference paper at ICLR 2025
a
MLP
Params
1043
3
Mixer
Params
1050
Transformer
Params
10t
1051
10
10
1055
10
1056
1060
100
0-
0
1043
10
10
10"
10
10
10
10-1
202
10
10 10-1
Compute (PFLOPs)
Compute (PFLOPs)
Compute (PFLOPS)
n=2
n
n
Train steps
10a2
MUP
10t6
1040
2
2
2
2
2
10
104a
Train steps
2.5
n= 2
n 4
n
n = 16
103.00
Mixer
Loss
10
10a
0.0
24
2
2
2
2
103/5
100000
104
Transformer
2
4
8
16
Train steps
103.00
Loss
10225
10250
24
2
24
2
2
1037
10
Accuracy
MLP: -1
Mixar: B=1
Transformer: B-1
Accurccy
MLP:
n=
MLP:
M=P:n=16
MLP: =
Mixer: =
Transformer: ==2
n=x
Mixer p=
Mixe: B
Mixer: = 16
Accuracy
Accurtty
Transformer. n=2
Transforeer.n
Transfermer.n
Transfomern
MLP:
Mixer:
Transfarmer: 8
Accurny
233
25
21
211
2
20
2
Test Task
ICL (swapl
ICL (resample)
IWL
Figure 7: ICL classification supplementary figures. (a) MSE obtained across each architecture as function of compute. Lines connect common models, with colors denoting different parameter counts. Hence, single line traces the trajectory of model across different training iterations. (b) Cross entropy loss across different context lengths L, for different input dimensions n. Line colors indicate the number of elapsed training steps. In these examples, B n//2, so there only 2 labels present in each context (out of C = 32 total possible labels). The gray dashed line corresponds to the loss obtained by placing equal probability plateau for a time at guessing one among the two correct labels, before eventually collapsing to the correct ICL solution. (c) IWL to ICL transition for different burstiness B. Consistent with prior work (Reddy, 2024; Chan et al., 2022), higher burstiness encourages ICL. Transformers transition to ICL for lower burstiness and lower number of clusters k. (d) ICL vs. IWL behavior for B n/2 and k 2048 clusters across context lengths L and input dimensions n. For the most part, these settings are sufficient to encourage ICI., including the configuration plotted in the main text Figure 1, though ICL appears to decay at higher dimensions and longer contexts. (all) Line plots feature 95 percent confidence intervals about the mean, computed across replications.
the 2 labels present in the context. All models
27

---

Published as a conference paper at ICLR 2025
MLP Mixer
3
Transformer
Loss
2
1
0
10-4
10-2
10
Compute (PFLOPs)
Params
10%
Mixer
# Params
1040
Transformer
Params
10t0
10t
1014
1043
10th
1060
1013
1014
Loss
205.0
10
1044
1060
2015
10
10-3
10P
10
10 10"1
10
10
10"
Compute (PFLOPs)
Compute (P/LOPS)
Compute (PFLOPs)
n ==
4
n 8
n = 16
Train steps
1022
MLP
Loss
10
1010
0.0
2 2'
24
2'
2
2'
2
2
10
Train steps
2
4
8
16
102.00
10225
Mixei
Loss
10
100
0.0
10400
2'
2
24
27
2
2
27
10
L
steps
Transfomerr
2
16
10
10221
Loss
10:
10
10"00
2
27
2
2'
2 2
2
2
L
Figure 8: ICL classification with infinite clusters. Just as we can consider a + oo limit for ICL regression, where regression weights are sampled afresh for each example, we can consider an analogous - oo limit for ICL classification where clusters are resampled for each new example rather than being fixed to underlying Gaussian mixture. Doing so forces each model to learn an in-context solution, but the learning outcomes turn out to be different. In particular, the task because substantially more difficult for longer contexts. For example, selecting context length L. 16 with infinite clusters is enough to block any model from learning the full ICL solution. In contrast, L= 16 with finite clusters can still push a model to learn the full ICL solution (Figure 7), even if an in-weight solution is also available. For this reason, we consider only finite but large k in the main text, enough to develop ICL without blocking learning for longer contexts. In this appendix figure, we examine more closely what happens if we attempt ICL classification with infinite clusters. (a) Loss obtained by each architecture as function of compute, for context length L 8 and dimensional inputs with burstiness B- 4, so 2 of the 32 possible labels appears in each example. The gray dashed line corresponds to the loss obtained by a model if assigns equal probability to the 2 labels present in the example. Like in Figure 1, we witness a plateau at the gray line, though it is somewhat more severe. Nonetheless, all models are able to perform the task perfectly with sufficient compute. (b) Line plot for each architecture in panel (a). Lines connect common models, with colors denoting different parameter counts. Hence, a single line traces the trajectory of a model across different training iterations. (c) Cross entropy loss across different context lengths L, for different input dimensions n. Line colors indicate the number of elapsed training steps. The gray dashed line corresponds the loss obtained by placing equal probability on the 2 labels present in the context among the 32 total labels. For context lengths L 2 16, all models plateau at the gray line and fail to learn further. Hence, it appears that even Transformers fail to learn the full in-context task, and remain stuck at local optima of guessing one of the two labels present in the context. In contrast, if we fixed the number of clusters to a large but finite value, all models will learn the full ICL solution even though an in- weiweghht solution is available (Figure 7 above). In this way, it appears that finite clusters afford some curricular benefit that leads model to the ICL solution, which the infinite case lacks. This discrepancy poses fascinating topic for future study. (all) Shaded regions correspond to 95 percent confidence intervals computed across replications.
28

---

Published as a conference paper at ICLR 2025
MLP
Transformer
Params
10aH
10
1.5
103a
10aa
Params 10
MTS
1.0
1042
10a6
1.0
10th
101a
1.0
104a
1012
0.0
10
10
10
10
Compute (PFLOPs)
Compute (PFLOPs)
Compute (PFLDPs)
Params
Transformer
Params
RB MLP
1024
10a0
Sphere Oddball
10a0
1030
1.5
1023
2050
1.0
Params 10ah
Loss
10a2
10aa
1042
0.5
0.5
10-5
10-1
10-4
10-
10-5
10-5
Compute (PFLOPs)
Compute (PFLOPS)
Compute (PFLOPs)
* Params
Transformer
Params
RB MLP
1030
1024
10:
10a0
1.90
# Params 1024
Line Oddbail
10ac
1.85
10
10
10
1.80-
0.0
10-4
10-1
20"
Compute (PFLOPs)
Compute (PFLOPs)
Compute (PFLOPs)
RB MLP (deep)
# Params
1025
1.5
1030
1.0
10a5
0.5
1045
10
10
Compute (PFLOPS)
Figure 9: Relational reasoning supplementary figures. We plot the loss obtained across each architecture function of compute. Lines connect common models, with colors denoting different parameter counts. Hence, a single line traces the trajectory of model across different training iterations. Note: the RB MLP does not have configurable widths or depths, so all MLPs have same parameter count. (all) Shaded regions correspond to 95 percent confidence intervals computed across 5 replications.
29

---

Published as a conference paper at ICLR 2025
n=
n=4
n= 8
100
MSE
Regression
10-1
MLP
16
n= 32
n= 64
Transformer
100
MSE
10
10-3 Compute (PFLOPs)
10-6
10-3 Compute (PFLOPs)
10-6
10-6 Compute (PFLOPs)
10-3
n 8
10"
Classifcationn
Loss
10
n = 16
n 32 aorieeee ex
n 64
Transformer
10a
Loss
10-1
-0 Compute (PFLOPs)
10-5
10 Compute (PFLOPs)
10"5
10-5 10
Compute (PFLOPs)
Figure 10: Simple regression and classification with varying input dimension. We plot the MSE (for regression) or cross entropy loss (for classification) as a function of compute across varying input dimension The red dashed lines in the regression plots correspond to the minimum attainable MSE. Each point corresponds to single model with particular parameter and training time. In all cases, reducing the dimension of the input reduces the gap between Transformers and MLPs, with the gap effectively vanishing for n = dimensional inputs.
30


Reviews:
[{"review_id": "KAll23dYxT", "reviewer": "Reviewer_KD93", "summary": "This paper presents experiments to argue that MLPs and MLP-Mixers are almost as effective as Transformer on many in-context learning (ICL) problems. Experiments in various settings like in context linear regression, classification and relational tasks from the field of psychology like sample match (akin to nearest neighbors in context), and finding the odd example out. In all of these settings, the paper presents experiments showing that MLP, MLP-Mixers and Transformers, by and large, perform similarly on these tasks at the same FLOPs spent on training. In particular, MLPs are slightly worse on the standard ICL problems, whereas they are better on the psychology inspired problems. Overall, through synthetic experiments, the paper makes that case that architectural biases may not play a huge role for ICL with enough compute.", "questions": "1. What is the distribution of inputs $x_i$ for Section 2.1 What could happen if the tasks were even more diverse by changing the data covariance? This is important because for a fixed input covariance, even a 1-layer Transformer suffices to solve incontext linear regression.\n\n2. Any understanding/analysis of why Transformers fair poorly in the relational tasks and why MLPs might be better? These seem right up the alley for Transformers, especially match-to-sample since attention directly computes all inner products.\n\n3. Are there previous papers that argue that attention is required for ICL?\n\n4. These experiments mix the role of expressivity (how large the model needs to be to solve this task) and optimization (can standard algorithms learn a good solution fast enough). \n\n5. Any reason to look at linear regression/classification and not other ICL (like decision trees, or fitting MLPs like considered in Garg et al.)? It raises the question whether the findings are an artifact of simplicity of the chosen problems.\n\n6: One strength of Transformers is the ability to handle different context lengths n with a single model. Is this property also true for MLPs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is the first paper, to my knowledge, that highlights that MLPs alone can lead to incontext learning. This is an interesting finding since, at least intuitively, the belief is that self attention helps with ICL. Verification of in-weight to in-context transition with task diversity, for all architectures, was also an interesting finding\n\n2. Presentation and discussion of results is clear\n\n3. For the set of ICL problems considered, the analysis seems quite extensive", "weaknesses": "1. The analysis is mostly in stylized and restricted settings. It is not entirely clear what this means for kinds of ICL that is observed in realistic settings (this is also mentioned in the limitations section of the paper). Even within simplistic settings, some more complex problems can be considered to make the claim that MLPs are competitive with Transformers. See questions 5 and 6 below.\n\n2. Some useful description of the experimental setup, like input distribution,  how MLP and MLP-Mixer were used, were either missing or in the appendix. There is also some theoretical discussion in the appendix that are not referred to in the main paper. See question 1 below.\n\n3. The paper mostly shows empirical evidence that MLPs can be competitive, however there is not much discussion about why this might be the case. See questions 2, 4 below\n\n\nOn the whole I still think the contributions are positive, but they could have been more extensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SUC0MqBoU7", "reviewer": "Reviewer_qhqq", "summary": "This paper provides more evidence that in-context learning (ICL) is a unique capability of Transformer models by demonstrating that Multi-Layer Perceptrons (MLPs) and MLP-Mixer models can also effectively learn in-context. The authors show that:\n\n1) On well studied regression and classification ICL tasks , MLPs perform competitively with Transformers when given the same compute budget. \n2) On classical relational reasoning tasks from psychology, MLPs actually outperform Transformers both in compute efficiency and out-of-distribution generalization. This challenges prior beliefs about MLPs' limitations in relational reasoning.\n3) The authors demonstrate that MLPs, like Transformers, show a transition from in-weight learning to in-context learning as data diversity increases across their experimental tasks.\n\nThe findings support the growing interest in Transformer alternatives, and studies there capabilties in controlled synthetic tasks  while acknowledging that their findings align with existing results showing MLPs' competitiveness on more complex natural language and vision tasks.", "questions": "1) Can you please provide additional experiments and provide analyses of these results when training autoregressively as Garg et al., 2022 or  von Oswald et al. 2023. \n\n2) If the MLPs / MLP mixers networks are chaning from in-weights to in-ciontext learning, do they approximate  e.g.  gradient descent in their archicture or mimic functionally of the self-attention layers. For the MLP mixer variants, I would find an analyses of functional similarity between architectures interesting. Can you e.g. train read-out layers at different depths of the trained network which approximate the solution / optimal prediction similarly when going down the network. It would be interesting to see if the networks are comptuing similar things at the same depth of the network (and gradually approximating the final solution). One could even try to see if the output of the self-attention and MLP mixers are similar.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is quite well writtin and in my opinion easy to follow. The experiments seems well executed and believable, some questions remain, see below.", "weaknesses": "The paper in my opinion overclaims the signifiance of the work, of how surprising the findings are. MLPs are universal function approximators, and ofc, can to some extend approximate self-attention layers. Its nevertheless somewhat interesting that gradient descent can install such solutions into architectures purely consisting of MLPs. \nIt is, especially on tractable problems such as linear regression / classification, clear that, if optimized well, neural networks will find / approximate the (known) Byaes optimal solution of these problems. I therefore not find the results very surprising. \n\nI would benefit the authors to highlight that Transformers architectures dynamically allocating compute / memory based on its in-context length. This is a unique feature, when comparing to RNNs or MLPs. Even if they might be performing similarlry to MLPs for a given sequence length given a fixed memory, compute budget, the flexiblity of Transformers is their strength.  \n\nThe authors, afaiu, do not run experiments in an autoregressive model as e.g. Garg et al., 2022 (What Can Transformers Learn In-Context? A Case Study of Simple Function Classes) or von Oswald et al. 2023 (Uncovering mesa-optimization algorithms in Transformers). I find this setting very important, see Questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c84mAV7Bw3", "reviewer": "Reviewer_2Mnt", "summary": "This paper studies the in-context learning capabilities of MLPs and mixer-MLP models, comparing them to Transformers on tasks such as synthetic regression and classification. The models are evaluated on limited training data and larger test sets to determine when they shift from in-weight learning to in-context learning. The authors also introduce unique relational tasks—match-to-sample, sphere oddball, and line oddball—revealing that MLPs and relationally bottlenecked MLPs outperform Transformers on these tasks. They suggest that the results may stem from the inductive biases of these architectures.", "questions": "Can you provide more insights on why transformers are failing in relational tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Every experiment in the paper is designed thoroughly. \n2. This is the first work encountered that explores the ICL capabilities of MLPs, which could be relevant to the literature on foundation models, especially in time series.\n3. The addition of relational tasks to the existing synthetic regression and classification experiments contributes valuable insights into Transformer limitations. Transformers perform poorly when test exemplars differ significantly from the training data.", "weaknesses": "The paper could have included real regression data. Most existing literature focuses on synthetic tasks, and exploring real data (even simple regression datasets) with somewhat complex underlying distributions would have added valuable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XolI4hrMmN", "reviewer": "Reviewer_v7Gh", "summary": "The authors empirically study the performance of MLPs, MLP Mixer Models, Transformers and some hand crafted models on a range of small synthetic in-context learning tasks. By carefully designing the tasks the authors show a transition from in \"weight learning\" to in \"context learning (ICL)\".", "questions": "Did you try comparing against a bi-directional transformer?\n\nDid you try applying the MLP or MLP mixer auto-regressively?\n\nIf you had to give a concrete definition of ICL what would it be? \n\nWhy is working out a linear projection for a “new point” not a new task (and not ICL) where as working out the projection of a point in a new linear regression problem ICL? To me the only different seem to be be how you define a \"task\"?\n\nDid you consider trying to get even more \"meta\" so each X,Y pair represents a type of regression problem? (say polynomials of a different order, or a mix of classification and regression problems). Each X would represent the concatenation of m examples of a different problems of the same type. The task is to in-context learn the class of the problem then work out the solution to a new point according to the class of problem? In other words apply the construction figure 1, recursively so X=(((x111,x112,...x11d,y11),....,(x1n1,x1n2,...x1nd,y1n)),....,((xm11,xm12,...xm1d,ym1),....,(xmn1,xmn2,...xmnd,ymn))(xqq1,xqq2,...xqqd)) Y = yqq. here you have m examples of problems from the same class each with n points with dimension d.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is thought provoking, well written and does a good job a questioning common assumptions, such as which Models can exhibit ICL? and what ICL even is?", "weaknesses": "The paper compares transformers applied auto-regressively against MLP and MLP mixer models that access the data all at once (are not auto-regressive). This seems like a strange choice for a fair comparison especially when trying to compare training FLOPs required. I strongly suggest including a bi-directional transformer as a fairer baseline. This would greatly strengthen any claim made about computational efficiency.\n\nThe paper does not really offer a concrete definition of what ICL is. “In-context learning (ICL) refers to a task paradigm where exemplars from a novel task are presented during inference time rather than during training. ” One could argue working out a linear projection for a “new point” is a new task and hence standard linear regression is “in-context learning”. For clarity I think promoting this sort of discussion about what is and isn’t ICL is a strength of the paper, but I would like to the see the authors add more discussion on this.\n\nThe paper only consider small toy problems which makes it hard to know if these observation generalise to more real world problems.\n\nSimilar to the above two points I'm fairly confident some researchers might not consider the problems tackled in the paper \"real in-context learning\", due to the scale and high levels of structure. I would like to see the authors add more discussion and try and push for a more concrete definition of ICL. The fact that MLP can learn a mapping that corresponds to the linear regression algorithm (or similar) is not that surprising. In my opinion the strength of the paper is questioning the existing definition of ICL and trying to push for a more rigours definition, I would like to see the authors lean into this a bit more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZLD7454fed", "reviewer": "Reviewer_3xiz", "summary": "This paper investigates the in-context learning ability of MLP and MLP-Mixer, and find it learns in-context competitively with Transformers given the same compute budget in some simple settings.", "questions": "Could the authors provide empirical results on real-world few-shot learning data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper is well written. The connection with related works are clearly presented.", "weaknesses": "The contribution of this work is hard to identify, and the results are not novel nor inspiring. First, it has been widely studied that MLP can learn relational data, i.e., can predict for set/sequence inputs, i.e., can learn in-context. Thus the studied topic is not novel. Second, it is also known that naive MLP for relational prediction (concatenating input) has disadvantages, e.g. lack of capturing permutation-invariance or variant input number. But this paper does not point out this keys, which would be helpful to explain why the MLP-Mixer and RB MLP can learn in-context better. Third, the result that MLP competitively with Transformers given the same compute budget is obtained with a synthetic setting, considering above nature of MLPs, it is not convincing enough to be generalizable.\n\nIn playful but intuitive words, while RB MLP and MLP-Mixer can be viewed as MLP Pro for relational data (including ICL), Transformer can be viewed as MLP Pro Max. So while it is mainstream to study ICL with transformer, it is not meaningful enough to study ICL with MLP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)