Meta-Learning General-Purpose Learning Algorithms with Transformers
Authors: Louis Kirsch
James Harrison
Jascha Sohl-Dickstein
Luke Metz
Venue: Submitted to ICLR 2023 (2023)
Keywords: meta-learning
general-purpose
transformers
learning-to-learn
meta-optimization
large-models
black-box

Abstract: Modern machine learning requires system designers to specify aspects of the learning pipeline, such as losses, architectures, and optimizers. Meta-learning, or learning-to-learn, instead aims to learn those aspects, and promises to unlock greater capabilities with less manual effort. One particularly ambitious goal of meta-learning is to train general purpose learning algorithms from scratch, using only black box models with minimal inductive bias. A general purpose learning algorithm is one which takes in training data, and produces test-set predictions across a wide range of problems, without any explicit definition of an inference model, training loss, or optimization algorithm. In this paper we show that Transformers and other black-box models can be meta-trained to act as general purpose learning algorithms, and can generalize to learn on different datasets than used during meta-training. We characterize phase transitions between algorithms that generalize, algorithms that memorize, and algorithms that fail to meta-train at all, induced by changes in model size, number of tasks used during meta-training, and meta-optimization hyper-parameters. We further show that the capabilities of meta-trained algorithms are bottlenecked by the accessible state size determining the next prediction, unlike standard models which are thought to be bottlenecked by parameter count. Finally, we propose practical interventions such as biasing the training distribution that improve the meta-training and meta-generalization of general purpose learning algorithms.

Under review as a conference paper at ICLR 2023
GENERAL-PURPOSE IN-CONTEXT LEARNING BY META-LEARNING TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern machine learning requires system designers to specify aspects of the learning pipeline, such losses, architectures, and optimizers. Meta-learning, or learning-to-leani instead aims to learn those aspects, and promises to unlock greater capabilities with less manual effort. One particularly ambitious goal of meta-learning is to train general-purpos. learning algorithms from scratch, us- ing only black-box models with minimal inductive bias. Such model takes in training data, and produces test-set predictions across a wide range of problems, without any explicit definition of an inference model, training loss, optimization algorithm. In this paper we show that Transformers and other black-box models can be meta-trained act as general-purpose in-context learners. We characterize phase transitions between algorithms that generalize, algorithms that memorize, and algorithms that fail meta-train at all, induced by changes in model size, number of tasks, and meta- cettaaopimizzaiin. We further show that the capabilities meta-trained algorithms are bottlenecked by the accessible state size (mem- ory) determining the next prediction, unlike standard models which are thought to be bottlenecked by parameter count. Finally, we propose practical interven- tions such biasing the training distribution that improve the meta-training and meta-generalization of general-purpose learning algorithms.
INTRODUCTION
Meta-learning is the process of automatically discovering new learning algorithms instead of de- signing them manually (Schmidhuber, 1987). An important quality of human-engineered learning algorithms is their applicability exceed those capabilities, the meta-learndd learning algorithms must be similarily general-purpose. Recently, there has been significant progress toward this goal (Kirsch et al., 2019; Oh et al., 2020). The improved generality of the discovered learning algorithms has been achieved by introducing inductive bias, such as by bottlenecking the architecture or by hiding information, which encourage learning over memorization. Methods include restricting learning rules 2019; Kirsch et al., 2019; Oh et al., 2020), symbolic graphs (Real et al., 2020; Co-Reyes et al., 2021), or parameter sharing symmetries (Kirsch & Schmidhuber, 2020; Kirsch al., 2021).
wide range of tasks or environments. For learning-to-lean to
gradients (Metz al.,
While enabling generalization, these inductive biases come at the cost of increasing the effort to design these systems and potentially restrict the space of discoverable learning algorithms. Instead, we seek to explore general-purpose meta-learning systems with minimal inductive bias. Good can- didates for this are black-box sequence-modelss as meta-learners such LSTMs (Hochreiter al., 2001: Wang et al., 2016; Duan et al., 2016) or Transformers (Vaswani et al., 2017). These memory- based or in-context learners take training data and produce test-set predictions without any explicit definition of an inference model, training loss, or optimization algorithm. This has lead to strong few-shot learning ability within the context of, for example, language modeling (Brown al., 2020).
In this work, we investigate how such black-box meta-learners can be trained to (meta-)generalize and learn on significantly different datasets than used during meta-training. For this we propose a General-Purpor In-Context Learner (GPICL) which is described with an as- sociated meta- meta-training task distribution in Section 3. In Section 4.1 we characterize transitions induced by scaling the number of tasks the model size used for meta-training traininggbbeteen between memo- rization, learning, and generalization. We further show in Section 4.2 that the capabilities of meta- trained algorithms bottlenecked by their accessible state size determining the next prediction
Transformer-basee

---

Under review as a conference paper at ICLR 2023
(such the hidden state size in recurrent network), unlike standard models which thought to be bottlenecked by parameter count, Finally, in Section 4.3, we propose practical interventions that improve the meta- training of general purpose learning algorithms.
BACKGROUND
What (supervised) learning algorithm? In this paper, we focus on the setting of meta- learning supervised learning algorithms. Consider mapping
No (fzz, Di Vili
y
(1)
from the training (support) set D = (Di,Wi)i No and a query input x' to the query's prediction y' where Te Rx=,, 3/s 3/ R.,, and No. Ng E N+. The subset of these functions that qualify learning algorithms those that improve their predictions y/ given increasingly larger training set D. Meta-learning then corresponds to finding these functions via meta- opttaiipttizziiin.. As in other black-box meta-learning models,
neural network represent such functions.
What is general-purpose learning algorithm? A learning algorithm can be considered general- purpose if learns on a wide range possible tasks D and their respective related queries x', y'- For example, gradient-descen on a suitable loss function can be considered very human-engineered learning algorithm (where the gradient is obtained via backpropagation or other means).
GENERAL-PURPOSE IN-CONTEXT LEARNING WITH TRANSFORMERS
Due the small number of inductive biases in black-box models, Ogeneralization when meta-training with an appropriately broad data distribution. Thus, changes in the data distribution affect whether and how model meta-learns and mueta-generalizes. We classify algorithms along two different dimensions: To what extent it learns (improving predictions given in- creasingly larger training sets), and to what extent it generalizes (performs well on instances, tasks,
only expect (meta-
or datasets not seen before). Algorithms can then be categorized as follows:
Learning
Generalization
Algorithm Description
Seen Tasks
Unseen Tasks
X No
X No
Instance memorization
System identification /
/ Yes
X No
Task memorization
Performance
No
Yes
Zero-shot generalization
General-purpon
Yes
Yes
learning algorithm
Examples seen
We demonstrate that sharp phase transitions occur between these learning modalities, and empiri- cally investigate these transitions.
3.1 GENERATING TASKS FOR LEARNING-TO-LEARN
Neural networks are known to require datasets of significant size to effectively generalize. While in standard supervised learning large quantities data are common, meta-learning algorithms may require similar number of distinct tasks in order to learn and generalize. Unfortunately, the number of commonly available tasks orders of magnitudes smaller compared to the datapoints in each task.
Previous work has side-stepped this issue by building-in architectural algorithmic structure into the learning algorithm, in effect drastically reducing the number of tasks required. For example, in Kirsch & Schmidhuber (2020); Kirsch al. (2021), the authors included symmetries into the
2

---

Under review as a conference paper at ICLR 2023
black-box model in the form of imput and output permutation invariances. An alternative this the generation of new tasks (Schmidhuber, 2013; Clune, 2019; Such et al., 2020; Parker-Holder et al., 2022). Unfortunately, it not easy to generate wide range of tasks that both diverse and contain structure as it can be found the real world.
In this work, take an intermediate step by augmenting existing datasets, in effect increas- ing the breadth of the task distribution based on existing task regularities. We generate a large number of tasks by taking existing supervised learning datasets, randomly projecting their inputs and permuting their classification labels.
While the random projection removes spa- tial structure from the imputs, this structure not believed to central the task (for instance, the performance of SGD-trained fully connected networks is invariant projection by random orthogonal ma- trix (Wadia et al., 2021)). Task augmen- tation allows tal questions about learning-to- to-mm in the regime of many tasks without relying on huge amounts of existing tasks or elabo- rate schemes to generate those.
Vi
VJ3
General-Purpose In-Context Learning Transformer
Third support set
investigate fundamen-
y
Y1
Third query set
Figure 1: Our Generul-Purpose In-Context Learner (GPICL) based on the vanilla Transformer which is trained make predictions for queries x' given any No prefix of dataset D {2i, We in Equation
A task or dataset D is then defined by its corresponding base dataset H {Te, y/ (linear) projection N=N== with A. N N.).
and output permutation p. {A p())). Unless noted otherwise, the distribution over output permutations p(p) is uniform.
3.2 META-LEARNING
Given those generated tasks. we then meta-train jointly on mini-batch sampled from the whole distribution. We minimize [()), the sum of losses on the query prediction after observing any prefix of dataset sampled from the augmented task distribution p(D)
ND Σ
((0) = EDDD(D)
(Uo((
(2)
where in the classification setting. / is the cross entropy loss between the label yj and prediction y' (1) fe is a neural network mapping predictions y' in Equation 1. Dur- ing meta-training. we take gradient steps in ((0) by backpropagation and Adam (Kingma & Ba, 2014). To investigate the effect of the data distribution, we train on various numbers of tasks (AI- gorithm 1). Finally, we need to choose black-box model for the function fa- We use a vanilla Transformer (Vaswani et al., 2017) with learned positional embeddings, visualized in Figure 1. We call it the Generul-P Purpose In- In-Contexi Learner (GPICL). Each token corresponds to concatenated and transformed input z; and one-ho hot encoded label y:- predicting the corresponding logits y' = y: for the current input z' 24. When querying for the first æ1. no label for the previous input is avail- able, so we feed a zero vector.
Algorithm Meta- DeeannTa for General-Purposs In-Context Learning (GPICL)
Require: Base dataset D {Z,, w.), Number of tasks K E N+
(A(k))
ij ( ~ (0, }
Sample input projections
{p(k)
1== p(p)
Sample output permutations
D(N) (A(K) p (T,))
p(D)= Uniform ( (1k) J
while not converged do + aVJ
Meta-train across tasks pDD) (Equation 2)
Meta-testing At meta-test time, no gradient- gradient-based learning is used. Instead, we simply obtain a prediction y by evaluating the neural network fo on the training dataset D and query point sr.
3

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS ON THE EMERGENCE OF GENERAL LEARNING-TO-LEARR
Multi-task training with standard classifiers Given a task distribution of many different classi- fication tasks, we first ask under what conditions expect "learning-to-leane to emerge. We train a single model across many tasks where each task corresponds to a random transformation of the MNIST dataset, but where the MLP only receives a single datapoint instead of whole sequence as input. This corresponds to ND = 1 in Equation 2. We would expect such a non-sequential classifier to be able correctly predict for the network capacity against the number of tasks, we indeed observe a linear boundary where an increasing number of tasks can be fit the larger the network (Figure 2a). This consistent with results in Collins et al. (2016), which found that a constant number of bits about the data distribution can be stored per model parameter, across a variety of model architectures and scales.
tasks its number of parameters increases. When plotting
Transformer
(a) Accuracy on seen tasks
(b)
Accuracy on seen tasks
(c) Accuracy on unseen tasks
2/4
2
222
222
24
0.8
2
21
236
234
244
0.6
222
27
23
0.4
24
28
24
27
128
756
512
1024
2
0.0
14
52
Hidden size drppacityl
Transforer
Figure 2: GPICL is able to generalize to unseen tasks. Each cell is a separate meta-trainin run. (a) An MLP classifier trained in a multi-task fashion across various numbers of tasks (generated based on MNIST) and network sizes is able to fit linearly more tasks, the larger its capacity. (b) sequence model (here the GPICL Transformer) that observes dataset D of inputs and labels transitions into generalizing an seemingly unbounded number of tasks with an increase in model size. This is achieved by switching from a memorization solution to a learning solution that (c) generalizes to unseen tasks. This generalization does not occur with the MLP.
Learning-to-learn with large sequential models and data contrast to the MLP classifier, a sequence model that observes multiple observations and their labels from the same task, could ex- ceed that linear performance improvement by learning at inference time. Indeed, we observe that Transformer that observe sequence of datapoints before making pre-
when switching diction about the query, more tasks can be simultaneously fit (Figure 2b). At a certain model size and number of tasks, the model undergoes phase transition, allowing generalize unbounded number of tasks. We hypothesize that this is due to switching the prediction strategy from memorization learning-to-learn. Further, when (meta-)testing the same trained models from the previous experiment on unseen task (new random transformation of MNIST), they generalize only in the regime of large numbers of tasks and model size (Figure 2c). As an in-context learner, meta-testing does not involve any gradient updates but only running the model in forward mode.
seemingly
Insight 1: It is possible to learn-to-learn with black-box models Effective learning algorithms can be realized using black-box models with few inductive biases, given sufficient meta- training task diversity and large enough model sizes. To transition the (earning-to-lcan regime, we needed at least 213 8192 tasks.
In the following, we study learning-to-lest le-raa from the perspective of the data distribution, the archi- tecture, and the optimization dynamics. For the data distribution, we look at how the data diversity affects the emergence and phase transitions of learning-to-leam generalization, and memorization. For architecture, we analyze the role of the model size and state size in various architectures. Finally, we observe challenges in meta-optimization and demonstrate how memorization followed by gen- eralization is important mechanism that can be facilitated explicitly biasing the data distribution.
4.1 THE LARGE DATA REGIME: GENERALIZATION AND PHASE TRANSITIONS
Simple invariances in data lead to the emergence of teaninggto-learn To verify whether the observed generalizing solutions actually implement learning algorithms (opposed e.g. zero-shot
4

---

Under review as a conference paper at ICLR 2023
Table 1: Meta-test generalization various datasets after meta-training on augmented MNIST and
seeing 99 examples, predicting the 100th. We report the mean across 3 meta-training seeds, 16 sequences from each task, 16 tasks sampled from each base dataset.
Fashion MNIST
Method Dataset
Inductive
MNIST
KMNIST
Random
CIFARIO
SVIIN
SGD
Backprop. SGD
70.31%
50.78%
37.89%
100.00%
14,84%
10.16%
MAML
Backprop, SGD
53.71%
48.44%
36.33%
99.80%
17.38%
11.33%
Parameter sharing.
VSML
Permutation invariance black-box
79.04%
68.49%
$4.69%
100.00%
24.09%
17.45%
LSTM
25.39%
28.12%
18.10%
58.72%
12.11%
11.07%
GPICL Transfommer (ours)
black. box
73.70%
62.24%
53.39%
100.00S
19,40%
14.58%
generalization), we analyze the meta-test time behavior. We plot the accuracy for a given query point given varying numbers of seen examples in Figure As it is typical for learning algorithms, the performance improves given an increasingly large set of seen examples (inputs and labels).
Generalization Naturally, the question arises to what ex- tent these learning algorithms general. While have seen generalization to unseen tasks consisting of novel pro- jections of the same dataset, do the learned algorithms also generalize to unseen datasets? In Figure we observe out- of-distribution performance on Fashion MNIST after having trained on MNIST (b, blue). In this direction, there is no gen- eralization gap directly train- ing on Fashion MNIST (b, or- ange). Similarly, when meta training on Fashion MNIST and meta testing on MNIST (a, with a larger generalization gap.
Meta-test learning curve on MNIST
Meta-test learning curve on FashionMNIST (b)
1.0
(a)
0.8
Accunay
0.4
Trained on mnis: tashion_mrist
0.0
1000
Numbber of examples seen
escemples seen
Figure 3: GPICL learns from examples at test time, and gener- alizes to unseen tasks and datasets. We meta-trained the Trans- former on a set of tasks defined by random transformations of ei- ther MNIST (blue) or FasHionMNIST (orange). We then meta-test on unseen tasks, and seen (ab) or unseen (ba) datasets. The plot shows the accuracy averaged across multiple runs at cach inner step, with shading indicating 95% confidence intervals. The in- crease in performance each step suggests have discovered learning algorithm.
orange) we observe that the learning algorithm generalizes, albeit
Comparison to other methods Other datasets and baselines are shown in Table 1. In particular, rather than focusing SOTA, we aim to validate whether methods with less inductive bias (such as our GPICL), can compete with methods that include more biases suitable to learning-to- te--m. This includes stochastic gradient descent (SGD) that updates the parameters online after observing each datapoint. MAML (Finn et al., 2017) proceeds like SGD, but uses a meta-learned neural network initialization. Both methods that rely on backpropagation and gradient descent, learn more slowly compared to our Transformer. In the case of MAML, this may be due to the main mechanism being feature reuse (Raghu et al., 2020) which is less useful when training across our wider task distribution. Among methods that do not hard-code gradient descent at meta-test time, we test VSML (Kirsch Schmidhuber, 2020) that discovered learning algorithms significantly generalizing between tasks. Our GPICL comes surprisingly close to VSML without requiring the associated inductive bias. Finally, we compare to standard LSTM that is trained with the same inputs as our Transformer. We observe that it performs worse, which we investigate further.
Insight 2: Simple data augmentations are effective for learning-to-learn The generality of the discovered learning algorithm can be controlled via the data distribution. Even when large task distributions are not (yet) naturally available, simple augmentations that promote permutation and scale invariance are effective.
Transitioning from memorization learning to generalizing When do the found solutions cor- respond to memorizing, learning, and generalizing solutions? In Figure 4, we plot the accuracy difference between the last and first prediction for seen task, an unseen task, and an unseen task with a different base dataset. We observe three phases: In the first phase, we memorize each instance, resulting in no within-sequence performance improvement. the second phase,
5

---

Under review as a conference paper at ICLR 2023
Inslance
Task memorization
memorize tasks and learn to identify tasks, resulting in a within-sequence improvement
memorization
Learning learn
Accurecy impooement within sequence
Seen MNSS
confined to seen task instances. In the final
Unseen MNIST
and third phase, we observe a more general leaming-to-leam,, performance improveent for unseen tasks, even different base datasets (here FashionMNIST). The last transition is very discrete with separate meta-training runs either finding solution of the task memoriza- tion or general learning-to-learn type (see Ap- pendix A.I).
0.5
Unseen asshionMNIST
0.4
0.2
0.1
2
222
255
Insight 3: The meta-learned behavior has phase transitions When increasing the num- ber of tasks, the meta-learned behavior transi- tions from instance memorization, to task iden- tification, to general learning- -o-arrrr...
Number of tasks
Figure 4: Transformers exhibit three different phases in terms of meta-learned behavior. (I) When training on a small number of tasks, spe- eific instances are memorized. orized, which is evident as a sequence in- crease of performance. (3) When training across many tasks, we discover a learning algorithm that generalizes to unseen tasks and unseen datasets.
(2) Tasks mem-
4.2 ARCHITECTURE:
LARGE STATE IS CRUCIAL FOR LEARNING
In the previous experiments we observed that given sufficient task diversity and model size, Trans- formers can learn general-purpose learning algorithms. This raises the question how essential the Transformer architecture is and whether other black-box models could be used. We hypothesize that for learning-to-les to-lernn the size of the memory at meta-test time (or state more generally) is particularly important in order to be able to store learning progress. Through self-attention, Transformers have a particularly large state. We test this by training several architectures with various state sizes in our meta-learning setting. In Figure 5a, observe that when vary the respective prrrmmparseeters which most influence the state size, we observe that for a specific state size we obtain similar per- formance of the discovered learning algorithm across architectures. In contrast, these architectures have markedly different numbers of parameters (Figure 5b).
Large state more crucial than parameter count This suggests that the model size in terms of parameter count plays a smaller role in the setting of learning-to-learn and Transformers have benefited in particular from an increase in state size by self-attention. Beyond learning-to teamm,,, this likely applies to other tasks that rely storing large amounts of sequence-specine information.
Insight 4:
4.3 CHALLENGES META- OPTIMIZATION
Meta-optimization is known to be challenging. Meta gradients (Finn et al., 2017; Xu et al., 2018: Bechtle et al., 2021) and works with parameter sharing or weight updates in their architecture (Kirsch & Schmidhuber, 2020; Pedersen & Risi, 2021; Risi, 2021) observed various difficulties: Slower convergence, local minima, unstable training, or loss plateaus at the beginning of training (see Appendix Figure 18). We show that some of these problems also occur with black-box models and propose effective interventions.
Sean MNIST
MNIST (unseen lask, seen datase!)
Unseen FachianMNIST (unseen task, unseen dataset)
Seen MNIST
(seen task seen dataset)
(seen task & seen dataset)
Dutee-anddut
0.6
0.4
0.2
02
85
0.0
2
2
2.1
2
2'
21
25
25
2
0.0
2
2/
24
State
State
Parameterr count
The state size (accessible memory) of architecture most strongly predicts its performance as a general-purpose learning algorithm. (a) A large state crucial for learning- to-learn emerge. (b) The parameter count correlates less well with learning capabilities.
Figure 5:

---

Under review as a conference paper at ICLR 2023
(a)
100% permuted labels
(b)
2.40
Training
Unseen FashionMNIST
Unseen MNIST
Training
Unseen FashionMNIST
2.35
2.30
2:25
0k
10k
20k
stop 50k
30k
40k
OK
20k
40k
Ok
20k
40k
Ok
20k
40k
stop
Figure 6: Meta-training dynamies often involve extended period where GPICL's perfor- mance stuck on plateau. (a) Meta-loss vs. meta-training step, for a uniform distribution over meta-training tasks. Training tasks are generated by random transformations of FashionMNIST. (b) zoomed in view of the plateau. The loss only decreases slightly and the model memorize small biases in the training data (decreasing generalization) before the loss drops sharply.
(b)
(c)
2
21
21A
20
29
Task
Pelese
25
25
Taak betch
Task Batch.ca C22
Number
Figure 7: Whether GPICL memorizes, generalizes, or remains trapped meta-loss plateau depends on the number of meta-training tasks, and the meta-training batch size. (a) A phase diagram showing GPICL's behavior at the end of meta-training (50k steps). Solutions either mem- orize, generalize and learn, or remain in the loss plateau. With additional training steps, configura- tions in the plateau might eventually transition to memorization or generalization. Generalization only occurs with large enough batch sizes and sufficient, but not too many, tasks. (b) This behavior explained by the plateau length decreasing with the increasing batch sizes (reducing the noise contribution). and (c) increasing with larger numbers of tasks.
Loss plateaus when meta-learning with black-hox models By training across large number of randomly transformed tasks, memorizing any task-specific information is difficult. Instead, the model is forced find solutions that are directly learning. We observe that this results in (meta- ploss plateaus during meta-training where the loss only decreases slightly for long periods of time (Figure 6a). Only after a large number of steps (here around 35 thousand) does drop in loss occur. In the loss plateau, the generalization loss increases on unseen tasks from both the same and a different base dataset (Figure 6b). This suggests that being able to first memorize slightly enables the following learning-to-lean phase. Furthermore, we observe that all gradients have very small norm with exception of the last layer (Appendix Figure 14).
Intervention 1: Increasing the batch size High variance gradients appear be one reason train- ing trajectories become trapped on the loss plateau (see Appendix Figures 12, 13). This suggests increasing the meta-batel size as straightforward solution. When plotting various batch sizes against numbers of tasks we obtain three kinds of solutions at the end of meta-training (Figure 7a): (I) Solutions that generalize and learn, (2) Solutions that memorize, and (3) Solutions that are still in the loss plateau (due to maximum of 50 thousand optimization steps). The larger the batch size, the more tasks we can train on without getting stuck in loss plateau. When plotting the length of the loss plateau against the task batch size (Figure 7b) observe a power-law relationship with increasing batch sizes decreasing the plateau length. At the same time, the batch size also increases the number of total tasks seen in the plateau (Appendix Figure 15). Thus, this intervention relies on parallelizability. An increase the number of tasks also increases the plateau length (Figure 7c). This may be due to larger number tasks making the initial memorization phase more difficult.
Intervention 2: Changes in the meta-optimizer Given that many gradients in the loss plateau have very small norm, Adam would rescale those clement-wise, potentially alleviating the issue. In practice, we observe that the gradients so small that the in Adam's gradient-rescaling de-

---

Under review as a conference paper at ICLR 2023
nominator (for numerical stability) limits the up-scaling of small gradients. Using smaller results in more than halving the plateau length. Alternatvely, discarding the magnitude of the gradient entirely by applying the sign operator to exponential moving average of the gradient (replacing Adam's is approximate magnitude normalization with direct magnitude normalization) has a similar effect while also increasing the numerical stability over Adam with small (Appendix Figure 16).
Intervention 3: Biasing the data distribu- tion / Currieula GPICL mainly relies on the data distribution for learning-to-leany This en- ables different kind of intervention: Bias- ing the data distribution. The approach is in- spired by the observation that before leaving the loss plateau the model memorizes biases in the data, Instead of sampling label permu- tations uniformly random, a specific permutation by using a fixed permu- tation for fraction of each batch. This com- pletely eliminates the loss plateau, enabling smooth path from memorizing learning (Fig- ure 8). Surprisingly, even when heavily bias- ing the distribution, memorization followed by generalization. This biased data distribution can be viewed as a curriculum, solving an easier problem first that enables the subsequent harder learning-to-learn. Further investigation is re- quired to understand how this transition occurs. 2022) which we investigate in Appendix A.7. We hypothesize that many natural data distributions- including language contain such sub-taske that easy memorize followed by generalization.
0% permuted labels
10% permuted labels
(b)
90% permuted labels
permuted la bebek
(d)
Training Unseen FashionMNIST
bias towards
Unseen MNIST
OK
10k
40%
50%
10%
20%
40k
50k
Figure 8: Biasing the training distribution is an effective intervention which prevents meta- loss plateau. A uniform distribution over tasks leads long plateau (d), while increasing the training fraction that corresponds to a single task reduces the plateau (abc).
This may be connected to grokking (Power et al.,
4.4 COMBINING DOMAIN-SPECIFIC AND GENERAL-PURPOSE LEARNING
We demonstrated the feasibility of meta- learning in-context learning algorithms that are general-purpose. An even more useful learn- ing algorithm would be capable of both gener- alizing, as well as leveraging domain-specific information for learning when it available. This would allow for considerably more ef- ficient in-context learning, scaling more difficult datasets without very long input se- quences. Toward this goal, we investigate simple scheme that leverages pre-trained neu- ral networks as features to Iearn upon. This could be from unsupervised learner frozen large language model (Radford et al., 2021; Tsimpoukelli et al., 2021). Here, first project the inputs F; of base-dataset D into some latent space using a pre-trained network, and then proceed with meta-training and meta- testing as before, randomly projecting these al- ternative features. For the pre-trained network, we use ResNet trained on ImageNet and re- move its final layer. In Figure 9 we have meta-trained GPICL on MNIST either with the randomly transformed raw inputs randomly transformed embedded features. At meta-test- time the learning algorithm generalizes to wide range of datasets, measured by the meta-test accuracy of the 100th example. At the same time, the pre-trained ImageNet helps accelerate learning datasets that have matching domain,
GPICL without feature embediing
Mots to5 acurscy of tast predictoo
GPICL with feature ombedsing
0.8
of
0.4
0.2
MNIST FashionMNIST KHNIST Random (unseen) Metasted task
CIFAR1O (unseer)
SVNNN (unseen)
Imeta-trained) Lurseen)
(unseen)
Figure 9: Using pre-trained networks allows leveraging domain-specific knowledge while still generalizing to other datasets GPICL is meta-trained on MNIST either with the randomly transformed raw inputs or randomly transformed pre- pre-trained features. Pre-training helps to acceler- ate meta-test-ting time in-context learning on datasets that have a matching domain, such as CIFAR1O. With only 100 examples, the learning algorithm can achieve about 45% accuracy on CIFAR I0. The learning algorithms still generalize to wide range of datasets. Error bars are 95% confidence intervals of the mean across meta-training runs.
8

---

Under review as a conference paper at ICLR 2023
such CIFARIO. We observe that with only 100 examples, the learning algorithm meta- veta-trained on MNIST, can achieve about 45% accuracy on CIFARIO.
5 RELATED WORK
Inductive biases in meta-learning Meta-learning approaches exist with a wide range of inductive biases, usually inspired by cxisting human- human-engineered learning algorithms. Some methods pre- wire the entire learning algorithm (Finn et al., 2017), pre-wire backpropagation and the structure of a gradient-based optimizer (Andrychowiez et al., 2016; Metz et al., 2019; 2020a), or hard-code gradient-based optimization but learn the loss function (Houthooft et al., 2018; Kirsch et al., 2019; Bechtle et al., 2021). Many methods search over hyper-parameters that alter existing learning algo- rithms (Xu et al., 2018; Metz et al., 2020b: Chen et al., 2022). Fast weight programmers or hyper- networks update the weights of the same or another neural network (Schmidhuber, 1992; 1993a; Ha et al., 2017: Irie et al., 2021; Sandler et al.. 2021; Kirsch & Schmidhuber, 2022; Zhmoginov et al., 2022). Our work aims to keep such inductive biases to a minimum.
General-purpose meta-learning There has been growing interest in meta-learning more general- purpose learning algorithms. The improved generality of the discovered learning algorithm has been achieved by introducing inductive bias, such as by bottlenecking the architecture or by hiding information, encouraging learning over memorization. Methods include enforcing learning rules to use gradients (Metz et al., 2019; Kirsch et al., 2019; Oh et al., 2020), symbolic graphs (Real et al., 2020; Co- Co-Reyes et al., 2021), or parameter sharing and symmetries (Kirsch & Schmidhuber, 2020; Kirsch et al., 2021). Parameter sharing and symmetries have also been discussed in the context of self-organization (Tang & Ha, 2021; Risi, 2021; Pedersen & Risi, 2022).
Black-box meta-learning: MetaRNNs, RL?, in-context learning In contrast to these inductive biases, neural networks can also learn-to-learn purely in their activations with little architectural and algorithmic biases (Hochreiter al., 2001; Wang et al., 2016; Duan al., 2016; Ortega al., 2019). This requires a feedback signal in the inputs that allows for learning such as the reward in reinforcement learning or label in supervised leamning (Schmidhuber, 1993b). While frequently used architecture is the LSTM (Hochreiter & Schmidhuber. 1997: Gers et al., 2000), this mechanism has also seen substantial recent attention in Transformer models (Brown et al., 2020; Chan et al., 2022) under the name of in-context learning. We refer to these networks simply as black-box meta learners. Our method GPICL in the class of these black-box meta learners. In contrast to previ- ous methods, GPICL implements general-purpose learning algorithms. Independently, Garg et al. (2022) recently studied generalization on synthetic functions compared to our augmented datasets. PFNs (Mikulik et al., 2020) demonstrated learning to learn on small tabular datasets when meta- training on synthetically generated problems. Experiments on more complex classification settings such Omniglot relied on fine-tuning. In comparison, our method investigated generalization of learning algorithms directly to datasets such as MNIST, Fashion MNIST, and CIFARIO.
6 DISCUSSION AND CONCLUSION
By generating tasks from existing datasets, we demonstrated that black-box models such as Trans- formers can be used to meta-learn general-purpose in-context learning algorithms (GPICL). We observed that learning-to-learn arises in the regime of large models and large numbers of tasks with several phase transitions from instance memorization, ing. The size of the memory or model state significantly determines how well any architecture can learn how learn across various neural network architectures. We identified difficulties in meta- optimization and proposed interventions in terms of optimizers, hyper-parameters, and biased data distribution acting as a curriculum. We believe our findings open up new possibilities of data-driven general-purpose purpose meta-lcarning with minimal inductive bias.
system identification, to general learn-
A current limitation is the applicability of the discovered learning algorithms to arbitrary input and output sizes. Appropriate tokenization to unified representations may solve this (Chowdhery et al., 2022). Furthermore, learning algorithms often process millions of inputs before outputting the final model. In the current black- box setting, this is still difficult to achieve. Recurrency-basee models usually suffer from accumulating errors, whereas Transformer's 's computational complexity grows quadratically in the sequence length.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 109139999,, 2016.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti. Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. 25th International Conference Pattern Recognition (ICPR), pp. 4161-4166 IEEE, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2et.200005.11..... 2020.
Stephanie CY Chan, Adam Santoro, Andrew K Lampinen, Jane X Wang. Aaditya Singh, Pierre H Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent in- context learning in transformers. arXiv preprint 2rXiv:2205.05053 2022.
Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski. Arnaud Doucet, Mare' aurelo Ranzato, al. Towards learning universal hyperparameter optimizers with transformers. arXiv preprint arXiv:2205.1332 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint wwXiv:2204.023:11 2022.
Jeff Clune. Ai-gas:
Ai-generating algorithmss alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv:1905. 10985, 2019.
John D Co-Reyes, Yingjee Miao, Daiyi Peng, Esteban Real, Quoc V Le, Sergey Levine, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Con- ference Learning Representations, 2021.
Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability in recurrent neural networks. arXiv preprint 16Xiv:1611.00 1.09913, 2016.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartletr, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. arXiv preprint arXiv:220..01066 2022.
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. Neural computation, 12(10):445122777 2000. 2000.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sepp Hochreiter, A Steven Younger, and Peter Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint arXiv: 0442/) 2018.
Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear trans- formers with recurrent fast weight programmers. Advances in Neural Information Processing Systems, 34:7733-777 2021.
10

---

Under review as a conference paper at ICLR 2023
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. arXiv preprint arXiv:2012. 14905, 2020.
Louis Kirsch and Jürgen Schmidhuber. Self-referential meta learning. In Decision Awareness in Reinforcement Learning Workshop ICML 2022, 2022.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint arXiv: 100,0099, 2019.
Louis Kirsch, Sebastian Flennerhag, Hado Hasselt, Abram Friesen, Junhyuk Oh, and Yu- tian Chen. Introducing symmetries to black box meta reinforcement learning. arXiv preprint arXiv:2109.1073 2021.
Luke Metz, Niru Mahes waramaaarnn Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and conrecting pathologies in the training of learned optimizers. In International Conference on Machine Learning, pp. 4556 4565. PMLR, 2019.
Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009. /243, 2020a.
Luke Metz, Niru Maheswaranathan, Ruoxi Sun, C Daniel Freeman, Ben Poole, and Jascha Sohl- Dickstein. Using thousand optimization tasks to learn hyperparameter search strategies. arXiv preprint arXiv:2002.11887, 2020b.
Vladimir Mikulik, Grégoire Delétang, Tom McGrath, Tim Genewein, Miljan Martic, Shane Legg, and Pedro Ortega. Meta-trained agents implement bayes-optime agents. Advances in neural information processing systems, 33:18691-18703, 2020.
Junhyuk Oh, Matteo Hessel, Wojciech M Czamecki, Zhongwen Xu, Hado van Hasselt, Satin- der Singh, and David Silver. Discovering reinforcement learning algorithms. arXiv preprint arXiv:2007.08794, 2020.
Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, 7eb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arXiv: (aae:::00500000 030300 2019.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint Stti::2003.0100222 2022.
Joachim Winther Pedersen and Sebastian Risi. Evolving and merging hebbian learning rules: in- creasing generalization by decreasing the number of rules. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 892-900, 2021.
Joachim Winther Pedersen and Sebastian Risi. Minimal neural network models for permutation invariant agents. arXiv preprini arXiv:2205.07866 2022.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint 27X1v:2201.02177. 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8766 PMLR, 2021.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Rep- resentations, 2020.
11

---

Under review as a conference paper at ICLR 2023
Esteban Real, Chen Liang, David So, and Quoe Le. Automl-zero: evolving machine learning algo- rithms from scratch. International Conference on Machine Learning, pp. 8007-8019. PMLR, 2020.
Sebastian Risi. The future of artificial intelligence is self-organizing and self-assembling. sebas- fianrisi.com, 2021. URL https: /eseeesaaattantiieaa
com/
ing_ai.
Mark Sandler, Max Vladymyrov, Andrey Zhmoginov, Nolan Miller, Andrew Jackson, Tom Madams, et al. Meta-learning bidirectional update rules. arXiv preprint arXiv:2104. 04657, 2021.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meia-... hook. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. Learning control fast-weight memories: An alternative dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
Jürgen Schmidhuber. Reducing the ratio between learning complexity and number of time varying variables fully recurrent nets. In International Conference on Artificial Neural Networks, pp- 460-463. Springer, 1993a.
Jürgen Schmidhuber. A 'self-referential weight matrix. In International conference artificial neural networks, pp. 446-450 Springer, 1993b.
Jürgen Schmidhuber. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers psychology, 4:313, 2013.
Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Genera- tive teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In International Conference on Machine Learning, pp. 9206-9216. PMLR, 2020.
Yujin Tang and David Ha. The sensory neuron as a transformer: Permutation commutationinnvaiinn neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34: 22574-22587, 2021.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Mul- timodal few-shot learning with frozen language models. Advances in Neural Information Pro- cessing Systems, 34:200-212, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, pp. 5998-5908 2017.
Neha Wadia, Daniel Duckworth, Samuel S Schoenholz, Ethan Dyer, and Jascha fohl-Dickstein. Whitening and second order optimization both make information in the dataset unusable dur- ing training, and reduce prevent generalization. In International Conference Machine Learning. pp. 10617-10629, PMLR, 2021.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprini arXiv: /6/ 1.05663, 2016.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradien reinforcement learning. Ad- vances in neural information processing systems, 31, 2018.
Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertransformer: Model generation for supervised and semi-supervised few-shot learning. In International Conference Machine Learning. pp. 77775-27709 PMLR, 2022.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 SOLUTIONS ARE MEMORIZING OR GENERALIZING
1.6
When do the found solutions correspond to memorizing
1.4
Training loss
vs generalizing solutions? In Figure 2 we observe fairly discrete transition between memorizing and generalizing solutions a function of the number of tasks. To an-
1.2
alyze this transition, we perform multiple training runs with varying seeds and numbers of tasks in Figure 10, re- porting the final training loss. We find that the distribution bi-modal. Solutions at the end of training are memoriz- ing or generalizing. Memorization cluster: The larger the number of tasks, the more difficult is to memorize all of them with a fixed model capacity. Generalization cluster: At certain number of tasks (here 6 thousand), a transi- tion point is reached where optimization sometimes dis- covers a lower training loss that corresponds to a general- izing solution. For larger numbers of tasks the solutions always settle in the gencralizing cluster.
0.8
0.6
0K
2k
5k
Bk
10k
12k 15k
Number of tasks
Figure 10: Solutions found by GPICL after meta-training are bi-modal, with a memorization and generaliza- tion mode. Each point represents the training loss at the end of meta-training for runs with different seeds and for various numbers of tasks that include the transition boundary previously ob- served. Almost all solutions are either in memorization cluster in gener- alization cluster.
A.2 WHAT CORRESPONDS
TO STATE (MEMORY) IN VARIOUS ARCHITECTURES,
We hypothesize that for learning-to-learn the size of the memory Ns at meta-test time (or state more generally) particularly important in order this by training several architectures with various N' in meta-learning setting. Memory in the context of recurrent neural networks corresponds to the hidden state or context vector of size NH, thus Ng E O(NH). More generally, we can describe the state the information bottleneck that the sequence has to pass through before making predictions. In the context of learning-to- to-leam, this state has to hold information about everything that has been learned so far. Standard learning algorithms such as neural networks trained via SGD would have a state that corresponds to the neural network parameters, iteratively updated via SGD. In transformers, self-attention allows for particularly large state of Ns E N(N where Nx is the size of key, value, and query, NL the number of layers, and Nr the length of the sequence.
able to store learning progress. We test
A.3 SUMMARY OF INSIGHTS
Insight 1: It is possible to learn-to-learn with black-box models Effective learning algorithms can be realized using black-box models with few inductive biases, given sufficient meta-training task diversity and large enough model sizes. To transition the -arning-to-learn regime, needed least 213 8192 tasks.
Insight 2: Simple data augmentations are effective for general learning-to-learn The general- ity of the discovered learning algorithm can be controlled via the data distribution. Even when large task distributions are not (yet) naturally available, simple augmentations that promote permutation and scale invariance are effecttive.
The meta-learned behavior has phase transitions When increasing the number of tasks, the meta-learned behavior transitions from instance memorization, task identification, to general learning-to-learn. The last transition is discrete, with two unique clusters.
Insight 3:
Large state is more crucial than parameter count We conclude that the specific inductive biases of each architecture matter to smaller degree. The driving factor behind their ability to learn how to learn is the size of their state. Furthermore, this suggests that the model size in terms of numbers of parameters plays smaller role in the setting of learning-to-learn and Transformers have benefited in particular from an increase in state size by self-attention. In non- meta-learning sequence tasks parameter count is thought to be the performance bottleneck (Collins et al., 2016). Beyond learning- to-lamm, this likely applics to other tasks that rely on processing and storing large amounts of sequence-specific information.
Insight 4:
13

---

Under review as a conference paper at ICLR 2023
A.4 LIMITATIONS
Varying input and output sizes Compared to some previous works in meta-learning (Andrychow- icz et al., 2016; Finn et al., 2017; Kirsch & Schmidhuber, 2020), the discovered learning algorithms are not applicable to an arbitrary input and output size which makes it more difficult to apply the learning algorithm
new, unseen problem. This problem also applies Transformers applied to multiple tasks and modalities. Related work has solved this problem by tokenizing inputs to com- patible, unified representations (Chowdhery al., 2022). We expect these techniques or others to be useful in the Bearning-to-lean context too.
Processing large datasets Learning algorithms often process millions of inputs before outputting the final model. In the black-box setting, this is still difficult to achieve. Recurrency- caarrenccy-bseed mod- els usually suffer from accumulating errors, whereas Transformers computational complexity grows quadratically in the sequence length. Additional work is required to build models capable of pro- cessing and being trained on long sequences. Alternativll, parallel processing, similar to batching in learning algorithms,, may be useful building block.
A.5 ARCHITECTURAL DETAILS AND PARPMEERRS
Transformer details By default, all Transformers have a key, value, and query size of 32, heads, and 4 layers, and model size of NM 256. The model size defines the dimensionality of each token, and the MLP between layers scales this size to a hidden representation of x Nas where Nas corresponds to the model size.
Outer-product LSTM We slightly modify an LSTM by replacing the context state with an outer- product update and inner-product read-out.
x __and_h = jnp concatenate (Linputs, prev pstaee hidden], ], axis=-1)
gated = hk. (i & size self num heads (x and h)
gated = gated reshape (( (batchhsizee self num sum-heads. 8 size))
gated = checkpoint. name (gated, 'gated')
# input,
cell-gate, f forget-gale, q = query. o = output-gate
sizes = (3 size, 3 size, size, size)
indices = up cumsumm [sizes[:- --]-
kl, k2, q, o jnp. split indices, axis=-1)
scale = jax nn. an. softplus
hk. get-parameter (' ('key-scale', shape=(), dtype=kl. dtype. init=jnp (et==jpp.zeros))
i. g. f = jjp-- j einsum ('bhki,bhkj->>
jax mn mn. amm.tan/e (split.axxi (kl, (3, size ))) scale,
jax nn (split axis (k2, (3, size)))
= jax on sigmoid + 1) # Forget bias
prev_state cell + jax_nn. nn sig
read = imp.einsume (' bbij bbi bhi->bhj', C. q)
hk. Flatten ()axx nn sigmoid * jnp tanh (read))
VSML We use version of VSML with single layer and self-messages (Kirsch et al., 2021) of size Each LSTM
hidden size of For each LSTM update we use two micro-ticks. We train on 2 tasks with a 90% biased permutation distribution. The task batch size is 8. All images are scaled to a size of x 32 x
VSML without symmetries Before activations are fed to standard instantiation of VSML, all inputs are projected using a learnable linear projection. Logits are generated using another lin- projection, followed by softmax. We use version of VSML with single layer and self- messages (Kirsch et al., 2021) of size 8. The LSTMs are on a grid of & x l LSTMs, where k E {1, 4, 8, 16, 24). Each LSTM has a hidden size of 64. For each LSTM update we two micro-tick We train on 25 tasks with a 90% biased permutation distribution. The task batch size is 128. All images scaled size of 14 x 14.
14

---

Under review as a conference paper at ICLR 2023
LSTM For the results in Table 1, used hidden size of 256 and 10 optimization steps. Larger hidden sizes were harder to optimize. We train on 2 tasks with a 90% biased permutation distri- bution. The task batch size is 128. All images are scaled to size of 32 x 32 x
A.6 EXPERIMENTAL DETAILS
Most experiments can be run on a single GPU, some require 16 GPUs due to sequence length and large batch sizes, with sufficient GPU memory (around GB each). Some experiments, such as Figure 2, require up to 1000 runs of that kind to produce the final heat-map.
Input normalization Each dataset is z-normalized by its mean and standard deviation across all examples and pixels.
Number of seeds and shading If not noted otherwise, line plots use 8 seeds for meta-training and least 512 seeds for meta-testing. Shading indicates 95% confidence intervals.
Figure 2
The MLP has two hidden layers of varying size with relu activations. The Transformer has the default parameters as defined above.
Figure We use transformer model with a model size of 256. We train on 2 tasks with 90% biased permutation distribution. The task batch size 128. All images are scaled to a size of 32 x 32 x Inputs are z-normalized across the dataset and all input dimensions.
Table 1 The SGD baseline was obtained by sweeping over learning rates from 10 to 0.5, opti- mizers SGD, Adam and Adam with weight decay, one or two layers, and hidden sizes of 32, 64, or 128 on MNIST. The best configuration (most sample efficient) corresponds to a learning rate of 10-3, Adam, and no hidden layers. SGD performs updates online each one out of the 100 data points. MAML is equivalent to SGD up to the difference that we meta-train the weight initialization according to Equation 2 where 0 are the initial parameters of the classifier that is then updated using SGD at meta-test time. All black-box approaches do not use gradient descent at meta-test time. All meta-learning approaches where meta-traie meea-ttaineed and tuned via grid search on MNIST.
Figure 10 We trained a Transformer with model size 64 and 32 seeds for each number-of-tashe configuration.
Figure 4
Input normalization is disabled.
Figure 5 The Transformer uses a task batch size 512.
Figure 6
Trained on 26 tasks generated from FashionMNIST with labels fully permuted. Trained on 26 tasks generated from FashionMNIST with labels fully permuted.
Figure 7
Figure 8 Trained on 26 tasks generated from FashionMNIST with label permutations varied.
A.7 ADDITIONAL EXPERIMENTS
Sequence length In all experiments of the main paper have meta-trained on sequence length (number examples) of 100. This is a small training dataset compared to many human- -eman-engineered learning algorithms. In general, long the learning algorithm does overfit the training data, more examples should increase the predictive performance. In Figure 11 we investigate how our model scales to longer sequence lengths. We observe that the final accuracy of the last query in the sequence consistently increases with longer sequences. The generalization to longer sequences than those seen during meta-training is another important direction for future work.
Gradient and update statisties To better understand the properties of the loss plateau, we visu- alize different statistics of the gradients, optimizer, and updates. In Figure 12, we track the expo- nential moving average statistics of Adam before the loss plateau and after (dashed vertical line). In Figure 13 we investigate how gradients differ between settings with a plateau and settings with a biased distribution where the plateau is avoided. We plot the cosine similarity between consecutive optimization steps, the gradient L2- norm, and the similarity and norm of the weight updates after normalization with Adam. The statistics plotted cumulatively smoothed with Gaussian filter for better readability. The gradient and update cosine similarity differ only marginally between cases with plateau and cases without. We observe that the gradient 1.2-norm in the plateau is half as big
15

---

Under review as a conference paper at ICLR 2023
0.8
0.6
accurat
0.4
0.2
Seen FashionMINIST
0.1
Unseen FashionMINIST
Unseen MNIST
0.0
sequen-co_ sequan-colaangth
Figure 11: Increasing the sequence length during meta-training and meta-testing improves the pre- dictive performance of the final query in the sequence. Error bars indicate 95% confidence intervals.
norm, typu
Movsug gradient norm
norm, type Mowlug gradient squared norm
0.07
0.0030
0.0025
0.0020
mmmm
0.04
ramm
0.0015
0.03
0.02
0.00
0. 0.0000
10000
20000
32000
40000
50000
100000
20000
30000
40000
50005
Figure 12: L2-norms of the gradient and squared gradient exponential moving average in Adam. The dashed line corresponds the loss drop at the end of the loss plateau.
as in the biased distribution case, although the updates that Adam applies are going towards zero. This also results in not moving far from parameter initialization when the plateau. We hypothesize this has to do with varying gradient norms when looking at individual parameter tensors (Figure 14). We observe that the gradients have small norm for most tensors, except for the last layer.
Batch size and number of tasks influence on plateau length Instead of looking at the plateau length in terms of the number of steps (Figure 7), may also be concerned with the total number of tasks seen within the plateau. This is relevant in particular when the task batch is not processed fully parallel but gradients are accumulated. Figure 15 shows the same figure but with the number of tasks in the plateau on the y-axis instead. It can be observed that larger batch-sizes actually increase the data requirement number of optimization steps. Similarly, larger task training distribution requires larger number of tasks to be seen within the plateau.
leave the plateau, despite decreasing the plateau in terms of the
Adjusting Adam's or changing the optimizer As discussed in the main paper and visualized Figure 16b, decreasing significantly shortens the plateau. This is due to the rescaling very small gradient magnitudes being limited by At the same time it incurs some instability. Directly normalizing the gradient by applying the sign function element-wise (Figure 16a) to the exponential gradient average shortens the plateau even further.
When memorization happens, can we elicit grokking? In Figure 7a we have seen that an insuf- ficiently large task distribution can lead to memorization instead of general learning-to-lear.. At the same time, Figure showed that biasing the data distribution helpful to avoid loss plateaus. Power et al. (2022) observed a phenomenon which they called "grokking" in which after having converged in terms of training loss, test loss may suddenly decrease. Large amounts of regulariza- tion, like weight decay with coefficient of 1.0 were found to facilitate this behavior. Is grokking
16

---

Under review as a conference paper at ICLR 2023
(b)
0.75
2.0
twiing lows
1.8
pormute labels prob
Smooth grad 5km
0.25
smooth podd nom
1.6
0.0
0.1
0.00
1.4
-025
1.2
-050
1.0
-0.7
0.1-
0.8
-1.00
0.0
200
100
120
150
cum www.g//doo.. am
80
60
50
1.00
0.30
0.5
smeodh updade 5m
pomn
0.0
0.15
-025
smooth
0.0
-0.50
-0.75
-1000
0.00
400
100
300
CUmL Un5ate
200
100
10000 20000 30000 400.00 50000 0
10000 20000
30000 40000 50000
step
Figure 13: Gradient and Adam update statistics for differently biased data distributions. (a) Plateaus in the loss are influenced by the bias in the data distribution. Plateaus result in moving away slowly from the parameter initialization. (b) The cosine similarity of both gradients and updates in consec- utive steps is only marginally different with or without a loss plateau. (c) While the gradient norm about half big when a plateau exists, the updates are going towards zero
17

---

Under review as a conference paper at ICLR 2023
transformetn fooftetierad nom
tannformeehs miplinssnbkrrad nom
frandfrmectss 210fisnnrrr nom
transfomenhh 1/sca/e/rad
transformenha attnveleubbrrddd nom
transfomert11 altre/quer//wggrrddd nom
trandforenti attiogueryybbhrrr
transformerhs..o www.linnesnbirrddd
Ponttormenhl.etr//ee///yyy _t__atttikyybb/rrr..mm nom
maniformerh2 mig-linar/wwrrddd
tandformen22 2lofiselgrad nom
transformenh2
transfomen12 attn/quey/wwirrad nom
transferent22 attnquerrbbirrad nom
transformnhh2 sttnilnssrwiirrad
trandfrmenhs snlp/hneeewerrrddd homm nom
randsormenhi, wwp.liiss..ibbbe.. Libigrad nom
transferments mipliinsneegggrr...oo norm
transforment1) attnkkaylllradd nom
transformenhi, 20.2oosetggrdddd
transtomenth1 In hnw.eeeehni___sseeeeeeeer iss_scceeggraddcoo nom
transfomeehh/ miplineer nor norm
ransformenh miptineer
Jioffettirad
trewifommehho.oo attte/querybirrrdd....
transformehhi attnoiuerryhhrrre...o nom
transforentii attike//lllrddd ngm
transformenho miplineer 1/bigrad
tenoor
transfomentho attn/calue/wigge nom
trandfermehh alteeocleebbirraa nom
transtamenhil wttp://urr//wwgrrddd nom
transtormenhio
transfomenh1 altrvvalue/wirrad
transformarti attnllinsarwwirrrd no
transformenhn. in loofieeiggrdd nom
transformehhi siikke//ysnn nom
out'wgrad rorm
linerriiirrd nom
transformehhi attnkeyysigrrddd nom
mransformeth11 afniinenrrarrrd
transfermenhh. wtt..nnnee///wrrdddom
transfomenthi In 2iscareggadd nom
transfomanhid In 2iscaeigrad nam
transformeshs_ In ji nom
transformenhi, aitw/queeyyworrrddm.m
transfoment2 In 2iscaloigrad nom
transforant12 Liscale/radd nom
trensforneshi iripiineer 1/iigrdd nom
transformettóo strllingssbiirad
trandfomenh3 In 2/scaleggadd nom
-/pos embaigrad nom
Irandormedda riplinser Libgrad
transforeennn eicalelgrad
innenbbgrrdd norm
transformeh22 miptinear 1/wigrad nor
tranforment2 attnkkyliilrrd nom
transformerhi, jntriiieeeewirrre
transformerhO sioffeeigrad nom
tranformerhh, mipliinsnbbirrdd norm
tensor www.d.rddd......
tenser graddd sim
Figure 14: Gradient L2 norms (left) and gradient cosine similarity for consecutive optimization steps
(right) for different parameter tensors. The last (output) layer has the largest gradients. Most other gradients are small.
227
Pistaau number tadks 200n
246
seen
245
2
245
213
2
224
224
22
224
22a
22a
23a
24
2
242
2.5
2 Number tasks
2
25
Task batch size
Figure 15: Instead of plotting the loss plateau length in terms of optimization steps, we look the total number of tasks seen within the platcau as function of the task batch size and the number of tasks in the training distribution. An increase in the task batch size leads to more tasks to be processed to leave the plateau.
18

---

Under review as a conference paper at ICLR 2023
21
29
Patex angth
Palee: wmpth
214
20
20
21
38te7 E20
Taac
21a
314
214
200
201
25
20
Optimizo
21
24
Adam
20-01
Sign
10-4
274
26
Tycs boWh 5te
27
Taes bowc Siee
3M4
25
Octinttr
29
Adam
20
20
Number
Figure 16: (a) When replacing Adam with sign normalization of the gradient or (b) reducing the plateau length significantly shorter.
19

---

Under review as a conference paper at ICLR 2023
namusalts 1024 wwggh.dayy
tanio 1024) wedgg
mamutalk 1224 wodght
4.2
33
22-
Thanneg the
2.5
8
Uteesin HILST
2.0
1.2
FATT bacc weght decay 2.01
PART, SaiCE 14304 (wiight dacay
FARL tada 25384 weight docay
4.2
3.5
3.2
2.5
1.5
5.5
4.2
3.5
3.0
8
3.0
1.5
0.3
25000 506c0 75301190100 50000050000000000
23000 5000E 75000 00000000000000000000000000000000
25302 50000 73600 10
Mep
Figure 17: We investigate whether grokking as defined in Power et al. (2022) can be produced when we observe memorization on smaller numbers of tasks. This would correspond the test loss decreasing long after the training loss has converged. We have not been able to elicit this behavior when looking at different numbers of tasks and weight decay coefficients.
connected to the optimization behavior observe, and if so, do similar interventions help in our setting? We look in particular at the boundary of memorization and generalization (214 = 16384)
where doubling the number of tasks few more times would lead to generalization. Figure 17 shows three task settings, 200 214 2 and three different weight decay coefficients, 0.01, 0. 1.0. The setting of 26 tasks shows generalization by default and only serves as baseline for the weight decay coefficient analysis. In the cases of memorization due to too few tasks, we have not been able to produce grokking behavior.
Optimization difficulties in VSML Previous work has observed several optimization difficulties: Slower convergence, local minima, unstable training, or loss plateaus at the beginning of training. Figure 18 shows some of these difficulties in the context of VSML (Kirsch & Schmidhuber, 2020). Because VSML. has permutation invariance built into the architecture as an inductive bias, changing the number of tasks has only small effect. We observe that in particular deeper architectures make meta-oppimizztion more difficult.
20

---

Under review as a conference paper at ICLR 2023
tasks
2 1 layers
of tasks 2" layers
# of tasks 2 layers
of tasks 2 layers
2.25
0% permuted sermuted
2.00
0
1.50
1:25
0.75
0.50
# of tasks 2 layers
tasks 225 layers
tasks 2 1 layers
# of tasks layers
2.25
Traning 1653
1:75
1.50
1.25
1.00
43k
50k
20k
30k
40k
10k
20k
40k 50k
Figure 18: Loss plateaus and slow convergence with deeper variants of VSMI.
21


Reviews:
[{"review_id": "y0y2reDIyYF", "reviewer": "Reviewer_RW6w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use transformers to meta-learn a general-purpose learning algorithm. They train a transformer on many related tasks such that it is able to make a prediction for x given an entire dataset and x as an input to the transformer. They analyze when under which circumstances the transformer memorizes the data seen during meta-training and when it generalizes and note that many tasks and large models are a requirement for generalization.", "summary_of_the_review": "The authors address a challenging research question without providing a motivation for solving it. There are some missing details which prohibit reproducibility and ability for the reader to interpret the results themselves. Finally, the method provides no empirical advantage over other methods as well.", "strength_and_weaknesses": "**Strengths**\n- Method is well-described\n- Interesting problem\n\n**Weaknesses**\n- No motivation provided (explain why no bias is useful and provide an example where this actually led to something more useful)\n- Experimental setup at times unclear (how do you create the different tasks exactly?)\n- No improvements over state-of-the-art (explain how the proposed method could be interesting nevertheless)\n\nThe authors tackle the ambitious problem of creating a black-box meta-learning algorithm. However, they provide neither theoretical, empirical or anecdotal evidence that this might be of any use. Therefore, it appears that this is a challenging problem but might be of no relevance.\n\nI have problems interpreting the results since it is nowhere described what a random transformation is. Are these rotations, shifts, crops, everything together? Maybe the effects observed are related to the fact that given enough tasks, we just have seen every possible transformation? So maybe we keep seeing memorization where test tasks are simply very close to another train test. There is no discussion on how task similarity between train and test tasks changes with growing number of tasks and whether that might have an impact on the results.\n\nHow does the work relate to TabPFN: https://arxiv.org/pdf/2207.01848.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VCBipWLnL-", "reviewer": "Reviewer_Ujps", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work performs a detailed investigation into the training process and qualities of a black-box meta-learning algorithm in a 'general-purpose learning' framework.\nThe focus is on a setup where the data is a collection of 'tasks', that consist of a sequence of (input, label) pairs for which the learner needs to predict the current label, given the prefix of labeled pairs. Hence, a transformer was chosen as a model that can process the task sequentially.\nThe tasks are generated as randomly projected and label-permuted MNIST images, and the dynamics of the standard meta-training scheme are studied empirically, mainly by monitoring the accuracy signal over training (seen), testing (unseen) and other-domain (Fashion-MNIST) data. \nThe training is claimed to go through stages of 'memorization' (first of instances, then tasks) and finally 'generalization' or 'learning-to-learn' to unseen or cross-domain tasks.\nThe main emphasis is on the transition between memorization and generalization - when does it occur (if at all) and how it is influenced by the properties of training and structure of the network. Mainly, the data size (number of tasks) and model state-size (rather than parameter count). \nThe phase transition is also studied in terms of the loss function, which reaches a plateau before it occurs (or doesn't), and several improvements in training are suggested in order to promote the transition at an earlier stage.", "summary_of_the_review": "Overall, I am very positive about this paper. I think that it very nicely demonstrates interesting phenomena in meta-learning. The new insights are very interesting and might have an impact on related areas as well.\nMy main concern is about the generality of the findings, given the very specific setting used for the experiments, as I described above.", "strength_and_weaknesses": "Strengths:\n* The paper presents a very interesting purely-empirical study of the learning dynamics in 'general purpose black box' meta-learning. \n* The experiments are well designed to demonstrate in a simple and clear way how learning progresses through the different stages, towards being able to learn-to-learn. Details are all super clear, very nicely presented and well explained, with nice insights about memorization and generalization.\n* The two findings that are most interesting, in my opinion: (i) How the existence of the phase shift (ability to generalize) is determined, very sharply, by the transformer model size and by the number of tasks. (ii) That the ability to learn is this setting is highly correlated to the state (rather than model) size.\n* I also find very interesting the in-depth look into what is happening during the (previously observed) training plateau, that is typical before moving to generalization. The 3 suggested 'interventions' in the process are well motivated and demonstrated.\n\nWeaknesses:\n* Although I find the observed phenomena and their explanations very interesting, I am concerned about how general they are, or whether they are very specific to the very limited setup that was chosen. This, in my view, limits the scope and impact of the findings, and I think there should have been more effort to relate to this point.\n* This is especially true, as the setup is different from the most standard one in (non-RL) meta-learning, which is few-shot learning. It is closely related, but focuses on the sequential version, rather than a support/query split.\n* In that regard too, the choice of producing tasks by random projections and orderings of MNIST, rather than the more common subsets of classes of a data-set with large variety of classes like imagenet should be justified. \n* Also, would these results and conclusions generalize to non-black-box settings? to the transductive one? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LCCVXC9El1", "reviewer": "Reviewer_momp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs an extensive empirical analysis of transformers used as general-purpose meta-learning algorithms in which transformers take a sequence of training data and a test points to output a prediction. The authors make a number of interesting findings including phase transitions when a transformer is able to generalize to unseen tasks and when a transformer is able to generalize to unseen instances within a task. The authors also demonstrate the importance of a large state to be able to learn a task. Finally, the authors propose a number of interventions to increase the ease of meta-optimization including increasing meta-batch sizes.", "summary_of_the_review": "Overall, the paper makes a number of interesting observations on transformers used as meta-learning algorithms. To my knowledge, these insights are novel, and thus the paper could have a significant impact in the field in its current form. The paper's significance could be further improved by conducting experiments on more complex datasets.", "strength_and_weaknesses": "**Strengths**\nThe paper's empirical analysis is extensive and comprehensive. The authors make a number of interesting observations that to my knowledge have not been made by prior literature. Moreover, the suggestions on interventions to improve meta-optimization are insightful and may be valuable to the community.\n\n**Weaknesses**\nOne weakness of the paper is that experiments are primarily conducted on simple datasets like MNIST. This work would be more significant if experiments could be conducted on more complex datasets.\n\nSince the paper's main focus is on explaining the properties of transformers as meta-learners, it would be helpful to have an expanded related work section explaining in more detail the previously known properties of transformers.\n\nFinally, the distinction between learning and generalization in the introduction of Section 3 is a little unclear to me. It would be helpful if the authors could provide examples of the 4 algorithms in the table here to make this more concrete.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject