Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to study the ability of transformers model to predict values of a function given an input via sample prompting. Overall, this is a nice study that sheds some light on the capability of transformers model to deduce what function the context refers to and execute/compute the value of that function given an input in the context. 
strengths_and_weaknesses: Pros:

- Quite a thought-provoking paper studying transformers capabilities to learn explicit mathematical functions
- Nice analysis / experiments. Everything seems well designed given the scope

Cons:
- There's a lot of mentioning for 'in-context'. However, I do fear that this could be a bit misleading. All the model is doing is doing sequential modeling that given a series of inputs and the corresponding function values, the model predicts the function value of  the previous input. So in a way this can also be seen as sequence modeling. 
questions: - For the input format x_1, f(x_1), x_2, f(x_2), .. if x_1 is a 20-dim vector, how is it presented to the model. This is a sequence of 20 numbers followed by 1 number for f(x_1, and so on?
- any comparison regarding the overall loss function proposed?
limitations: I think the paper can easily be expanded to include complex functions. 2-layer ReLU is more complex than linear but also not as interpretable. Extending to something like degree-m polynomial and to observe trends with respect to capacity / how much it takes to train / etc can shed more light into the ability of transformers model for this type of sequence modeling.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: **Goal:** Investigate to what extent decoder-only transformer models are capable to in-context learn functions from simple functions classes, like linear functions, sparse linear functions and shallow neural networks. If the answer is positive, understand the training conditions that lead to such in-context learning capabilities. 

**Main results:**
* Transformers are capable of in-context learning all of the simple function classes tested in the paper to a surprising extend. The mean squared error values computed using the transformers often match the best-case known algorithms (such as min-norm linear regression and LASSO).  (The extent to which this is true is quite striking) 
* Carefully designed experiments show that this capability isn't explained by simple memorization of training examples. Rather, the transformers seem to learn in-context learning strategies that generalize to nontrivial distribution shifts. 
  * Distribution shifts in the in-context exemplars, as well as between in-context and query examples mostly leave the performance unchanged. 
  * In the case of learning linear functions, even the gradient of the learned transformer matches that of the min-norm linear regression solution. This is also a very strong and surprising finding. 

====================

**Post rebuttal update**

I thank the authors for their response. I still think that this is a strong submission that deserves a place in NeurIPS2022, hence maintain my score. 
strengths_and_weaknesses: STRENGTHS
* **Comprehensive experiments:** The paper is full of clever experiments that strongly support the claims made in the paper. A number of these experiments yield nontrivial and (in my opinion) unexpected and surprising results. 
* **Sound and convincing:** The claims made in the paper are sound and convincing (assuming the experiments were run correctly).
  * Some of the results look "too good to be true" (i.e. to the extent transformers match the performance of optimal algorithms like LASSO and min-norm linear regression), though the experiments look carefully done and I don't currently have any suspicions as to their validity. 
* **Well scoped and crisp problem formulation and analysis** The paper is very well scoped, and the goal and the contributions are very clear. 



WEAKNESSES
* **Lack of discussion on types of in-context learning** There are multiple forms of in-context learning that can perhaps be studied separately (Olsson et. al. [31] discuss this briefly). This paper tackles only one of those - it would be better if the authors disambiguated this earlier in the paper. 
* **More distribution shifts:** Perhaps one or two more distributions shifts in addition to the skewed Gaussian distribution could be helpful. 
(The paper is very well-scoped, and (in my opinion) doesn't have that many glaring limitations within that scope. Some of what's below can be regarded as recommendations to make the paper better instead of weaknesses)
* **Visualizing Sampled 2-layer neural network functions:** It'd be useful to provide some kind of visualization to make sure that the approximated neural network functions are non-trivial in the domain they're evaluated at. For example, all of the sampled neural nets could be roughly linear around the domain most of the samples fall, which'd make the result a bit less interesting (though still cool). 
* **Lack of mechanistic understanding:** After the authors make a very compelling case that transformers do indeed learn generic in-context learners, perhaps the authors could have spent some time analyzing how a GPT-2 like architecture could be achieving this mechanistically. 
questions: * Regarding curriculum learning: Do you think one can do away with curriculum learning by using a fixed dataset that contains a mixture of examples of different difficulties? Establishing this could be useful for facilitating future research in this topic, as one does not need to deal with the additional subtleties regarding a curriculum learning setup. 
* Inputs to the transformer: I don't quite get the description between lines 107 and 109. How exactly are the exemplars processed before being fed to the first transformer layer? 
* Consistency of results: Does training work all the time, or do you need to train a number of times before you come across something that displays the impressive in-context learning results? 
* Figure 3 (b): Is this plot averaged over multiple prompts? (This is a very interesting finding!) 


Nitpick: Typo at line 335: "when being trained" instead of "when being training" 
Another nitpick: In line 307, you write that you study "what classes of functions can a model learn in-context". The current set of results don't seem to address this exact framing, as answering that question would require experiments with many more types of functions. Perhaps rephrasing this would be helpful? 
limitations: I don't wish to report any glaring limitations beyond what has been listed as weaknesses above.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tries to study the in-context learnability of function classes in transformers. While the existing research is actively exploring this phenomenon in language models, this work presents a novel analysis of the phenomenon using function classes (in lieu of tasks) as the objects to be learnt. The paper shows that transformers can learn linear functions with high precision (obtaining almost the same loss as the least squares estimator) and that learning more complex  function classes (e.g. 2 layer neural nets) entails the in-context learning of simpler function classes (e.g. linear). The paper also shows that (similar to many language tasks in large-scale language models), learning function classes could also be classified as an "emergent" behavior during training. Further, the paper also analyze model behavior under distribution shift and how that in-context learning of function classes is surprisingly robust to two common forms of distribution shift. Altogether, this analysis is a solid contribution to the literature, with simple, easy to follow experiments that can shed more light into the mechanics of in-context learning.
strengths_and_weaknesses: The strengths of the paper are:

1. The paper is well written, with a clearly defined (and well motivated) experimental protocol. This simple setup could encourage further experiments.
2. The empirical results are intriguing and will be of interest to the broader ML research community, given transformer has become such an indispensable architecture. 
3. To my knowledge this is the first exposition of "emergent" in-context learning in function classes as well as curriculum learning of function classes.

The paper has the following weaknesses:

1. There is no analysis of the size of training data that is required for successful in-context learning. A surprising omission in the experiments is the lack of a plot of data size against in-context learning performance. I think this is an important component that is missing.
2. The paper does not shed any light on the in-context learning phenomenon for language tasks, and one of the missing pieces is it's lack of analysis on training data size, along with training on a mixture of function classes.

questions: 1. In section 5 (lines 229-232), does the model trained to learn the 2-layer neural net also learn sparse linear functions? 
2. For a model with a given capacity, how does the data size change in-context learning performance?
3. The paper shies away from analyzing the interaction of training data and model capacity. Does the in-context learning performance depend on effective model complexity and not just on model capacity (as characterized by number of parameters)? Any comments on incorporating training data size into the analysis?
4. Unless I missed something, is there an experiment where model training is done a mixture of function classes? How sensitively do the the results depend on the mixture ratios?

limitations: The authors do not include a section on limitations and potential negative societal impacts of the work. I urge the authors to add a limitations section, explaining any factors (e.g. training data size, effective model complexity, etc.) that could further complicate the analysis.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies in-context learning in transformers by training models to predict the output of a linear function given a few input/output examples presented in context. The main findings are that transformers trained in this way are able to match the performance of the optimal estimator in various settings, and show some ability to generalize out of distribution. Surprisingly, models trained on sparse linear functions have an in-context performance that matches Lasso. Also surprisingly, models trained on the output of 2-layer ReLU networks perform reasonably well on linear functions. Finally, the authors find that curriculum learning (starting with easier examples and progressing to harder ones) greatly speeds up training.
strengths_and_weaknesses: The paper studies in-context learning in a well-scoped and well controlled setting. The setup and results are presented clearly, and the experiments are designed to produce sharp results with clear baselines and conclusions. I found several of the results surprising and inspiring for future investigation. The experiments on OOD generalization were especially interesting. Understanding in-context learning in transformers is an important problem, and this paper is a good step in that direction.

In terms of weaknesses, the settings studied in this work are relatively simple, and it is still hard to see how they connect with examples of in-context learning using natural data.
questions: It's interesting to explore how models are able to achieve in-context learning of linear and sparse linear functions. For example, can one recover the linear function's weights from the transformer's activations? And does the transformer implement some version of Lasso, or does it work in a different way? I realize this is out of scope for the present work. I'm only proposing these questions in case the authors have more to share on this.

In Figure 2 "trivial error" is introduced but is not defined. I believe it is defined to be the zero estimator only in Figure 5.
limitations: This work has no obvious societal impact.
review_title: null

Meta-review: metareview: This paper demonstrates compellingly that transformers are able to in-context learn simple function classes (e.g., linear functions), to the extend that they can recover solutions from algorithms like LASSO. The experiments are well designed and executed, which lead to surprising and intriguing results. While the paper does not provide any explanation for why transformers exhibit such capabilities, it will spur both empirical and theoretical work studying how transformers learn algorithms from in-context examples. Congratulations on a nice work!
confidence: Certain
Decision: Accept