A Picture of the Space of Typical Learning Tasks
Authors: Rahul Ramesh
Jialin Mao
Itay Griniasty
Rubing Yang
Han Kheng Teoh
Mark Transtrum
James Sethna
Pratik Chaudhari
Venue: Submitted to ICLR 2023 (2023)
Keywords: Information Geometry
Space of learning tasks

Abstract: We develop a technique to analyze representations learned by deep networks when they are trained on different tasks using supervised, multi-task, meta- and contrastive learning. We develop a technique to visualize such representations using an isometric embedding of the space of probabilistic models into a lower-dimensional space, i.e., one that preserves pairwise distances. We discover the following surprising phenomena that shed light upon the structure in the space of learning tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our visualization technique is consistent with parts of the Wordnet phylogenetic tree; (4) fine-tuning a model upon a sub-task does not change the representation much if the model was trained for a large number of epochs; (5) episodic meta-learning algorithms fit similar models eventually as that of supervised learning, even if the two traverse different trajectories during training; (6) contrastive learning methods trained on different datasets learn similar representations. We use classification tasks constructed from the CIFAR-10  and Imagenet datasets to study these phenomena.

Under review as a conference paper at ICLR 2023
A PICTURE OF THE SPACE OF TYPICAL LEARNING TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
We develop technique to analyze representations learned by deep networks when they are trained on different tasks using supervised, meta- and contrastive learning. We develop a technique to visualize such representations using an isometric embedding of the space of probabilistic models into lower-dimensional space, i.., one that preserves pairwise distances. We discover the following surprising phenomena that shed light upon the structure in the space of learning tasks: (1) the manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional; (2) supervised learning on one task results in a surprising amount of progress on seemingly dissimilar tasks; progress on other tasks is larger if the training task has diverse classes; (3) the structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree; (4) fine-tuning a model upon a sub-task does not change the representation much if the model was trained for a large number of epochs; (5) episodic meta-learning algorithms fit similar models eventually as that of supervised learning, even if the two traverse different trajectories during training; (6) contrastive learning methods trained on different datasets learn similar representations. We use classification tasks constructed from the CIFAR- 10 and Imagenet datasets to study these phenomena.
INTRODUCTION
Exploiting data from related tasks to reduce the sample complexity of learning a desired task, is an idea that lies at the heart of burgeoning fields like transfer, multi-task, meta, few-shot, and sulf-supervised learning. These algorithms have shown an impressive ability to learn representations that can predict well on new tasks. The algorithms are very diverse in how they work but it stands to reason they must be exploiting some shared structure in the space of learning tasks. Although there is a large body of work that seeks to understand relatedness among tasks and how these algorithms exploit it (see $4 for discussion of related work), we do not know what this shared structure precisely is. Our work makes the following contributions to advancing this line of research.
We develop a technique to analyze the learned representation on task, and its relationship to other tasks. Our key technical innovation is to use ideas from information geometry to characterize the geometry of the space of probabilistic models fit on different tasks. We develop methods to embed training trajectories of probabilistic models into a lower-dimensional space isometrically. i.e., while preserving pairwise distances. This allows to faithfully visualize the geometry of these very high dimensional spaces (for Imagenet, our probabilistic models are in ~00 dimensions) and thereby interpret the geometry of the space of learning tasks. These technical tools are very general and shed light on the shared structure among tasks.
We point these technical tools to study how algorithms that learn from multiple tasks work. We provide evidence for the following phenomena.
(1) The manifold of probabilistic models trained on different tasks using different representation learning methods is effectively low-dimensional, and this dimensionality is rather small; For Imagenet, a 3-dimensional subspace preserves 80.02% of the pairwise distances between models, which we define (in Appendix D) as the "explained stress";
(2) Supervised learning
task results in surprising amount of progress (informally, "progress" means that the representation learned on one can be used to make accurate predictions on other tasks; this is defined precisely in (4)) on seemingly dissimilar tasks;
progress on other tasks larger if the training task has diverse classes;
(3) The structure of the space of tasks indicated by our analysis is consistent with parts of the Wordnet phylogenetic tree;
(4) Fine-tuning a model upon a sub-task does not change the representation much if the model trained for large number of epochs;

---

Under review as a conference paper at ICLR 2023
(5) Episodic meta-learning algorithms fit similar models eventually as that of supervised learning, even if the two traverse different trajectories during training;
Contrastive learning methods trained on different datasets learn similar representations. We demonstrate these findings on image classification tasks constructed from CIFAR-10 and Imagenet datasets.
(6)
2 METHODS
Modeling the task We define a task P as a joint distribution on inputs x Rd and outputs y ()) corresponding to C' classes. Suppose we have N independent and identically distributed samples ((nny5)))== ) from P. Let øī (y1..... : UN) denote any sequence of outputs these N samples and y² denote the sequence of ground-truth labels. We may now model the task as
Pw II Piol (ym) ==1
(1)
where w are the parameters of the model and we have used the shorthand p (wn) pur 2n). The true probability distribution which corresponds to the ground-truth labels is denoted by P. P(y*) ). In the same way, let us denote by Po the probability distribution that corresponds to p" (v) 1/C for all n and all y, i.e., Po predicts accurately on a fraction 1/C of the samples.
Bhattacharyya distance Given two models Pa and Pe parameterized by weights and v respectively, define the Bhattacharyya distance (Bhattacharyya, 1946) between them averaged over samples as
dn Po) - log EII
Pu (u())) po(yn)
N
C
N-N log
II
v (C(() po(c)
(2)
n= c=1
Σ log Σ Vpl(c) (c]c
Appendix C for more details on (+). Our model (1) involves product the proba- bilities of N samples. Typical distances for probability distributions, e.g., Hellinger distance 2 2 (I. Ee Vpa(c) pe()) saturate when the number of samples N is large (because random high- nigh-dimmesiioal vectors are nearly orthogonal). is thus difficult to use such distances understand high-dimensional probabilistic models. However, the Bhattacharyya distance is well-behaved for large N due to the logarithm (Quinn et al., 2019; Teoh et al., 2020), and that is why it is well suited to our problem.
Distances between trajectories of probabilistic models Consider a trajectory (w=(k))
that records the weights after T updates of the optimization algorithm, e.g., stochastic gradient descent. This trajectory corresponds to a trajectory of probabilistic models Fur We are interested in calculating distances between such training trajectories. First, consider (u(0), u(1), u(2)..... u(T)) and another trajectory f = (u(0), u(2), a(4), u(T), (T))..... (T)) which trains twice as fast but to the same end point. If we define the distance between these trajectories as, say. Σg dn (Pu(()) )o then the distance between Fu and i, will be non-zero zero-even they are fundamentally the same. This issue is more pronounced when calculate distances between training trajectories of different tasks. arises because we are recording each trajectory using a different time coordinate, namely its own training progress.
To compare two trajectories correctly, we need a notion of time that can allow us to uniquely index any trajectory. The geodesic between the start point Po and the true distribution P. a natural candidate for this purpose. Geodesics are locally obtain closed-form formula for the geodesic as follows. We can
curves in a metric space. For the
product manifold in (1), think the square root of the probabilities /pl (c) as a point on C-dimensional sphere. Given two models P. and P, the geodesic connecting them under the Fisher information metric is the great circle on this sphere (Ito & Dechant, 2020, Eq. 47):
sin ((1 - (jd2) (da)
sin (Ad sin (dg)
Pd =
II 11
VpB
for A E [0, 1].
(3)
2

---

Under review as a conference paper at ICLR 2023
where dG En cos (E. VFi(C) is half of the great-circle distance on the N-way product manifold of probability distributions p-(+) and (+).. Every point w on the trajectory Fur can now be reindexed by new time which we call "progress"
tw inf do Pd..). Xe
Note that the progress bo E [0, 1] for any point on any trajectory. In practice, we solve (4) using bisection search (Brent, 1971). Observe that (3) also allows us calculate the geodesic between two successive points (x) and (u((++) of the trajectory. Any point on this geodesic Pa (e(k), ((++1) for À j0, 1] can be assigned progress (k) + A(Hw(k+1)) - (w())) We have effectively converted our trajectory which a discrete sequence of models fw = (()())))== into a continuous curve Tor = (Ejoa)) We can now calculate the distance between trajectories Tu and Ty as
duy Tv) J." (a(()(()) Po(t)) dt;
(5)
approximate this integral in practice using uniform grid on [0,
Riemann length of a trajectory of probabilistic models Divergences like the Bhattacharyya distance or the Bullback-Leibler (KL) distance (which is exactly the cross-entropy loss)
dKL(P., Pw) = -N Σ ΣΣeh pd (c) log H(P));
where (P.) is the entropy of P. (and independent of w) can be used to define a Riemannian structure in the space of probabilistic models (Amari, 2016). The distance between two models Pwr and Pwtdw parametrized by infinitesimally different weights is ds² = 4dg Pw-tww (dw, .gww) dw) + o(lldw])²)
where g(w) = NN Epp -2 the Fisher Information Matrix (FIM). This FIM is therefore the metric of the space of the probability distributions and weights w play the role of the coordinates in this space. The Bhattacharyya distance and the KL-divergence induce the same metric up to a
scalar factor. We can therefore calculate the Riemann length of trajectory Tw by integrating the infinitesimal lengths
Length(rw) =2 /da(Punto) ((CPPut))-Puctaar (w(c+aa)).
(6)
Observe that we do not need the FIM to calculate the length. We can think of the length of a trajectory taken by model to reach the solution P. compared to the length of the geodesic measure of the inefficiency of the training procedure. This inefficiency can arise because: (a) not all probability distributions along the geodesic can be parametrized by our model class (approximation error), and (b) the training process may take steps that are misaligned with the geodesic (e.g., due to the loss function, mini- batch updates, supervised vs. some other form of representation learning, etc.).
Mapping a model trained on one task another task using "imprinting" We will consider different tasks )==)),_...... with the same input domain but possibly different number of classes C*. Given model P. parametrized by weights for one task, we interested in evaluating its learned representation on another task, say, p2. There are many ways of doing so, c.g., one could fine-tune the weights using data from 22 We use simple technique that re-initializes the final layer of the model. Let us separate the weights into two parts w = (W], w2) for the backbone and the classifier respectively. Let (æ(;; w1) denote the features of the penultimate layer corresponding to an input To clarify, the logits given by RC¹ u2 "p(;;; w1) (with perhaps added bias) and the output (w æn) for c 1....,C? Crl is computed using a softmax applied to these logits.
If we have learned w from one task p1 then we can re-initialize each row of the classifier weights (w2Y for c I,..., C² to maximize the cosine similarity with the average feature of samples from task p2 with ground-truth class et (w2Yk h/1lA122 where Σ (x:og=e) 90 w1).
The new model w = (wi, w2') can now be used to make predictions on the task p2, This technique for mapping the learned representation from one task to another is motivated from previous work (Dhillon al., 2020: Qi et al., 2018; Hu et al., 2015) which initializes the final layer of a model before adapting it to new classes using fine-tuning. is called imprinting because rows of the classifier's weights can be thought of as templates of different classes against which features are compared.
3

---

Under review as a conference paper at ICLR 2023
Using imprinting, can map trajectory Tu of model being trained on task p to another task p by mapping each point along the trajectory, let us denote this mapped trajectory by T 71,+22
Remark 1 (Imprinting vs. training the final layer). We are interested in mapping 1000s of models across different trajectories to other tasks. Training the final layer for all these models cumbersome and we can avoid doing so using imprinting. Note that if we were to train the classifier w2 (with backbone wi fixed) using samples from the other task under the constraint that rows of w2 have unit Eg norm, then we would obtain the imprinted weights as our solution (see Appendix E).
How to choose an appropriate task to map different models to? Consider the training trajectory Tu of model trained on task p' and another trajectory ne of model trained on task pa. Using the procedure in (7), we can map these trajectories to the other task to get Tu 1-+2 and To ?-+11 This allows us to calculate (Td 1 +- )e) which is the distance of the trajectory of the model trained on p and
then mapped to p² with respect to the trajectory of a model trained on task p2; we can calculate day (7ā-11 7 analogously. Roughly speaking, if the two learning tasks pl and p2 are very different, (e.g., Animals in CIFAR-10 and Vehicles in CIFAR-10), then these distances will be large.
The distances (u -2, ,,,) and duag (T 2-1 TT)) are reasonable candidates to study similarities
between tasks lº and 2², but they are not equal to one another. We are also interested in doing such calculations with models trained on many different tasks, and mapping them to each other will lead to an explosion of quantities. To circumvent this, we map to unique task whose 12 space the union of the output spaces of the individual tasks, e.g., to study pl (Animals) and (Vehicles), we will map both trajectories pU which is all of CIFAR-10. We calculate quantities like
(n -1 .7 -.
(8)
Embedding a probabilistic model in lower-dimensions We use a technique called intensive principal component analysis (InPCA) (Quinn et al., 2019) to embed a probabilistic model into lower-dimensional space. This enables us to visually inspect relationships between different tasks and representation learning methods. For m probability distributions, consider matrix D E R"* with entries Dav dg (Pu, Po) and
= --DT/2
where Lij = 6,j 1/m is the centering matrix. An eigen- -ecempeeoooooitionn of W = UEUT where the eigenvalues are sorted in descending order of their magnitudes |Eool 2 |Enl 2 ... allows us to compute the embedding of these m probability distributions into an m-dimensional space as Rm X UVE. Unlike standard PCA where eigenvalues InPCA can be both positive and negative. There are technical reasons for this (Quinn ct al., 2019). But such an embedding into a Minkowski space allows InPCA to be an isometry:
non-negative, eigenvalues of
(10) for embeddings Xu, X2 of any two probability distributions Pa and Po. This property of InPCA is akin to that standard PCA, where if we preserve all the eigenvectors (i.c., k = n), then (10) holds, but if we use fewer eigenvectors to project the points, then pairwise distances can be distorted. We only use InPCA for visualization, all our findings are quantitative and do not rely on embedding into lower dimensions. We quantify how well pairwise distances are preserved by k-dimensional InPCA embedding using a quantity called "explained stress" (xx):
Ex x1)² dg Pb) 0
Xk W- Σt_ Ene U.U
EX
[W],
Appendix A.2 describes how we implement InPCA for very high-dimensional probabilistic models.
3 RESULTS
We next discuss our key results. Our experiments were conducted using multiple neural architectures and image-classing classfiaasiiiiatoonnn tasks created from the CIFAR- 10 and Imagenet datasets; see Appendix A.
All the analysis in this paper (except Fig. 2) was conducted using the test data. All models were trained using the training data, but all mapped models, distances between trajectories. quantitative evaluation of progress and InPCA embeddings were computed using the test dataset. The reason for this is that we would like to study the geometry of tasks as evidenced by samples that were not a part of training. To emphasize, we do not develop any new algorithms for learning in this paper and therefore using the test data to quantify relationships between tasks is reasonable; see similar motivations in Kaplun et al. (2022) among others. Our findings remain valid when training data is used for analysis; this is primarily because in most of our experiments, a representation is trained on one task but makes predictions completely new task after mapping.
4

---

Under review as a conference paper at ICLR 2023
(a)
(f)
Iragenel
Ranberr 323 classes
Po
Comvepance
Intrumentadly
Doge
Vartidraacs
Random labe's
0.65
P
P
PC
(b)
-0.8%
PC
-0.0
0,89
P0
(e)
.0.8
2in
Strves
(d)
(c)
silnnc Comeeyanze
Prggee Thepeetlal tae mented
Geodegic progress (fa)
Velidaees
10.0
0.2
Venlic
7.5
Rondom lobols
0.0
onagines
350 diator
20
40
Epoch
Figure 1: (a) InPCA embedding of training trajectories of probabilistic models trained on 6 tasks from Imagenet. Each point is one network, bold lines connect the averages of probabilities of 5 training trajectories of each task random initializations in weight space). The point Po denotes start training (complete ignorance) and P. drawn to show the model that predicts perfectly (truth). All trajectories move towards the end point of training the entire Imagenet. Training one task makes remarkable amount of progress dissimilar, classes. Trajectories of models trained on random set of 333 classes are similar to those of entire Imagenet. Some classes (Instrumentality) are closer to this trajectory while others such as Vertebrates and Dogs farther away. Dogs is semantic subset of Vertebrates; it splits the beginning but seems eventually reach a similar representation as one of the intermediate points of Vertehrates. See Result 1.
unseen, seemingly
(b) Percentage explained stress (defined (12)) captured by subspace spanned by the top & InPCA eigenvectors. Explained stress measures if the Bhattacharyya distances hetween models are accurately captured in this sub-space. Top 3 dimensions capture than 80% of the explained stress, and thus the manifold is effectively low-dimensional. See Result
(c) If train model on Vertebrates, do make progress unseen classes? This plot measures the progress made by each model on classes seen during training (left half, lighter shade) and on novel classes (right half, darker shade). We compute t which is the progress L, of images restricted to single class This quantity i measures the quality of the representation that we make more progress on classes seen during training. If the model sees a larger diversity of classes (like with random 333 classes), more progress is made the novel classes. Surprisingly, even if we train on just the "Dogs", we make some progress on novel classes.
class c. The violin plots that denote the distribution of t. indicate
(d) Progress (4) the Y-axis against the number of epochs of training on the X-axis. The progress tw increases with more epochs of training- all models make non-trivial progress towards the truth P. (tw = 1). Surprisingly, even if we train only Dogs (118 classes) we make progress on the entire Imagenet. (e) Validation accuracy as function of epochs. See Result
(f) Trajectories of models trained on different phyla of Wordnet (colors match those of the inset graph). The model manifold is again effectively low-dimensional (78.72% explained stress in dimensions). Trajectories of tasks that are nearhy in Wordnet are also nearhy, and vice-versa. (g) Bhattacharya distance between the mean trajectories (over random initializations) normalized by the average of the tube radii (maximum distance of one of the trajectories from the mean, computed at each progress) of the two trajectories. Such quantities allow us to make precise statements about the differences hetween representations and show some very surprising conclusions. E.g., trajectories of random 333 classes (light orange) are closer to Conveyance (as expected), but those of Vertebrates (red) are equally far away for more than 60% (tw, zs 0.25) of the progress. In other words, training on Vertebrates (reptiles, dog, bird) makes remarkable progress on Conveyance (cars, planes). Result
different tasks and the trajectory of Conveyance. This distance
5

---

Under review as a conference paper at ICLR 2023
Result 1. The manifold of models trained on different tasks, and using different representation learning methods, is effectively low-dimensional. We trained multiple models on different sub-tasks of Imagenet (from 5 random initializations each) to study the dimensionality of the manifold of probabilistic models along the training trajectories (100 points equidistami in progress (4)) after mapping all models to all Imagenet classes (~ 10 dimensions). We use the explained stress (defined in Appendix D). to measure if the distances of the models. The first 3 dimensions of InPCA (Fig. 1a) preserve 80% of the explained stress (Fig. 1b shows more dimensions). This is therefore remarkably low-dimensional manifold. It is not exactly low-dimensional because the explained stress is not 100%, but it is an effectively low-dimensional manifold. This experiment also indicates that the individual manifolds of models trained on one task are oow-dimesiional, even if they start from different random initializations in the weight space. Such low-dimensional manifolds are seen in all our experiments, irrespective of the specific method used for representation learning, namely, supervised, transfer (fine-tuning), meta and contrastive learning.
preserved by the first k dimensions of the embedding
Result 2. Supervised learning on one task results in a surprising amount of progress on seemingly dissimilar tasks. Progress on other tasks is larger if the training task has diverse classes. We studied the progress fur (4) made by models (Fig. Id) trained on tasks from Result 1. Training on the task "Dogs" makes progress on other tasks, even seemingly dissimilar ones like "Instruments" (which contains vehicles, devices and clothing). In fact, it makes progress on the entire Imagenet (about 63.38% of the progress of a model trained directly on Imagenet). Progress is larger for larger phyla of Imagenet (Vertebrates and Instruments). But what is surprising that if we train on a random subset of 333 classes (a third of Imagenet), then the progress on the entire Imagenet is very large (92%). This points to a remarkably strong shared structure among classes even for large datasets such as Imagenet. Note that this does not mean that tasks such as Vertebrates and Instruments are similar each other. Even if training trajectories are similar for while, they do bifurcate eventually and the final models are different (Fig. Ig).
To study this further, we projected models trained on one task onto the geodesics of unseen classes (Fig. Ic) calculated using (3)). A model trained on the entire Imagenet makes uneven progress on the various classes (but about 80% progress across them, progress highly correlated with test error of different classes). Models trained on the 6 individual tasks also make progress on other unseen classes. As before, training on Instruments, Vertebrates, Dogs makes smaller progress unseen classes compared to training on a random subset of 333 classes. This is geometric evidence that the more diverse the training dataset, the better the generalization to unseen classes/tasks; this phenomenon has been widely noticed and utilized to train models on multiple tasks as we discuss in $4.
The structure of the space of tasks indicated by our visualization technique is consistent with parts of the Wordnet phylogenetic tree To obtain more aine-graind characterization of how the geometry in the space of learnable tasks reflects the semantics of these tasks, we selected two particular phyla of Imagenet (Animals, Artifacts) and created sub-tasks using classes that belong to these phyla (Fig. 1f). Trajectories of models trained on Instruments and Conveyance are closer together than those of Animals. Within the Animals phylum, trajectories of Vertebrates (Dog, Reptile, Bird) are closer together than those of Invertebrates (Fig. Ig for quantitative metrics). Effectively, we can recover a part of the phylogenetic tree of Wordnet using our training trajectories. We speculate that this may point to some shared structure between visual features of images and natural language- based semantics of the corresponding categories which was used to create Wordnet (Miller, 1998) of the corresponding categories. This alignment with a natural notion of relatedness also demonstrates the effectiveness of our methods to understand the structure in the space of visual tasks.
Result 3.
Result 4. Fine-tuning a pre-trained model on a sub-task does not change the representation much. To understand how models train on multiple tasks, we selected two binary classification sub-tasks of CIFAR- 10 (Airplane vs. Automobile, and Bird vs. Cat). We selected models at different stages of standard supervised learning on CIFAR- 10 (i.e., using 10-way output and softmax cross- cross-entropy loss) and fine-tuned each of these models on two sub-tasks (the entire network is fine-tuned without freezing the backbone). As Fig. 3 shows, models that were fine-tuned from earlier parts of the trajectory travel a large distance and move away from trajectories of the supervised learned CIFAR- 10 models. As we fine-tune later and later models, the distance traveled away from the trajectory is smaller and smaller, i.e., changes in the representation are smaller. For a fully-traines CIFAR- 10 model which interpolates the training data, the distance traveled by fine-tuning is very small (the points are almost indistinguishable in the picture); this is because both pa and p² are subsets of CIFAR-10. 10.
Algorithms for transfer learning train on source task before fine-tuning the model on the target task. If two tasks share a large part of their training trajectory, then we may start the fine-tuning from many shared intermediate points there are many such points. If the chosen point is farther along
6

---

Under review as a conference paper at ICLR 2023
(8)
(a)
Par.
tade
Spity tosk
They task
CTARID
Ps
pk
-10
Po
(b)
(c)
KC1
(d)
Agg Aakaa wwwwe ten k-ver
2.0
and
Dbloros bshroon & ard
4.5m
Figure 3: (a) Finc-tuning trajectories on Airplane vs. Automobile, and Bird vs. Cat sub-tasks of CIFAR-10 10 (warm and cold hues) pre-trained from differ- points along the trajectory of su- pervised learning. If the pretrained model has progressed further towards the truth I'., then fine-tuning on a sub- task does not change the representation much. The final trajectory (fine- ttne-t from epoch 100) is indistinguishable from P.. (b) Bhattacharyya distance between the mean trajectories normal- ized by the average of the tube radii (like Fig. 1g). Models (say, fine-tuned after Epoch on Task 1) go backwards in progress, i.c., they unlearn the pre- trained representation in order to fit the new task. This occurs as early as Epoch here. It suggests that learning occurs extremely rapidly at the beginning and determines the efficiency of fine-tuning Some curves here not visible be- cause they are overlapping heavily.
Numtan
Figure 2: (a) Training trajectories for supervised learning (black), 2-way (pink) and S-way episodic meta-learning (pur- plc). Trajectories of 5-way meta-learning are very similar to those of supervised learning and eventually reach very similar models and high test accuracy. In contrast, 2-way meta-learning has a much longer trajectory (about 40x longer in Riemann length than black) and does not reach a good test accuracy (on all 10 CIFAR-I0 10 classes). Representations are similar during early parts of training even if these are quite different learning mechanisms. (b) Trajectories of 2-way (hlue), 5-way (green), 7-way (yellow) meta-learning compared to supervised learning (red). For large "way",, trajectories are similar to supervised learning but they deviate from the red trajectories quickly for small ways. (c) Average distance between two k-way meta-lcarning trajectories decreases with k, this is geometric evidence of the variance of predictions of learned representations. (d) Training with a small way leads models that predict poorly on test data (large distances from truth). These embeddings were calculated using the training dataset. The rationale being that we wanted to show how different meta-learning and supervised learning are during training.
in terms of progress then the efficiency resulting from using the source task is higher because the trajectory required to fit the target task is shorter; such trajectories were used in (Gao & Chaudhari, 2021) to define a distance between tasks. As we saw in Result 2, trajectories of different tasks bifurcate after a shared part. The resultant deviation less for related tasks and more for dissimilar tasks (Fig. 3a, Fig. la,f). Therefore it is difficult to know a priori from which point one should start the fine-tuning from without knowing the manifold of the target task. In particular, our geometric picture indicates that fine-tuning from a fully-trained model can be detrimental to the accuracy on the target task. This has been noticed in number of places in the transfer learning literature, e.g., Li et al. (2020), and has also been studied theoretically (Gao & Chaudhari, 2020).
Result 5. Episodic meta-learning algorithms fit a similar model eventually as that of supervised learning, even if the two traverse very different trajectories during training. Meta-learning methods build representation which can be adapted to a new task (Thrun & Pratt, 2012). We studied a common variant, the so-called episodic training methods (Bengio et al., 1992), through the perspective of few-shot learning (Vinyals et al., 2016). In these methods, each mini-batch consists of samples from C classes (called "way"" chosen randomly out of the C' classes) split into two parts: a "support set" D. of s samples/class (called "shot"), and a "query set" D, of q samples/class. Typical methods, say prototypical networks of Snell et al. (2017b), implement a clustering loss on features the query samples using the averaged features of the support samples i = 8 ExzED.. (z)=cc) s0(2; w1) for all 1,..., C as the cluster centroids. This involves maximizing the likelihood
Var E Dg, if 3 (z) p(yp(x; w1)) x exp (- 11pee plx; wI)ll² /2)
(11)
7

---

Under review as a conference paper at ICLR 2023
over query samples. If features up lie on & ball of radius 1, then maximizing this likelihood is akin to maximizing the cosine similarity between cluster centroids computed from the support samples and features of the query samples. At the end of training, the same clustering loss with the learned backbone wi is used to predict on unseen classes (using "few" support samples to compute centroids).
We compared trajectories of episodic meta-learning to the trajectory taken by supervised learning using the cross-entropy loss over all C' classes in Fig. 2. Episodic meta-Icarning methods are quite different from supervised learning so it seems surprising that both arrive at the same solution; see Fig. 2a,b and Fig. A3 for distances between trajectories. But this is consistent with recent literature which has noticed that the performance of few-shot learning methods using supervised learning (followed by fine-tuning) is comparable or better than episodic meta-learning (Dhillon et al., 2020; Kolesnikov et al., 2020; Fakoor et al., 2020). The supervised learned representation also minimizes the clustering loss (11). We find that the trajectory length of episodic training is about 40x longer than that of supervised learning.
Few-shot accuracy of episodic training is better with a large way (Gidaris & Komodakis. 2018). We trained models with different ways to study this (Fig. 2b). We find that the radius of the tube that encapsulates the models of 2-way meta-learning around their mean trajectory is very large, almost as large as the total length of the trajectory, i.e., different models trained with a small way learn very different representations. Tube radius decreases as the way increases (Fig. 2c). Further, the distance of models from the truth P. (which is close to the end point of the supervised learning model) is higher for small way (Fig. 2d). This is geometric evidence of the widely used empirical practice of using a large number of way in episodic meta-learning. Observe that as the way increases, the trajectory of the episodic meta-learning becomes more and more similar to that of supervised learning.
Result 6. Contrastive learning methods trained on different datasets learn similar representations Contrastive learning (Becker & Hinton, 1992) learns representations without using ground-truth labels (Gutmann & Hyvärinen, 2010; Chen et al., 2020a). It has been extremely effective for self-supervised learning (Doersch & Zisserman, 2017; Kolesnikov et al., 2019), e.g., prediction accuracy with 1-10% labeled data is close that of supervised learning using all data (Chen et al., 2020b). We compared representations learned using contrastive learning with those from supervised learning to understand some aspects of why the former are so effective.
Consider a task P and a set of augmentations G (c.g., cropping, resizing, blurring, coloricontrass/brihhtnsss distortion etc.). Given inputs (say images) æ from P. contrastive learning forces the representation ((g(z); w1) and (og (x); (shortened to ((g())) below) of the same input for two different augmentations g.9' to be similar. And forces it to be different from representations of other augmented inputs æ' (Zbontar et al., 2021; Bachman et al., 2019; Dosovitskiy et al., 2014).
We used popular method called SimCLR (Chen al., 2020a) to perform contrastive learning on bird-cat, ship-truck) and all of CIFAR-1O. 10. We
images from four sets of classes carplane--atomoobe compared the learned representation that from supervised learning on two tasks hirrlanne-auomobnn and all of CIFAR- 10) in Fig. 4. Models trained using contrastive learning on two-class datasets learn very different representations from models trained on the same task but using supervised learning. Models trained using contrastive learning on different datasets learning similar representations (trajectories of all three two-class datasets are very close to each other). This is reasonable because contrastive learning does not use any information from the labels. It is surprising however that the trajectory of models from contrastive learning on these two-class datasets is similar to trajectories of models from contrastive learning on the entire CIFAR-10.
We also performed an experiment where we compare the representations of semi-supervised (Fixmatch (Sohn et al., 2020)), contrastive (SimCLR (Chen al., 2020a), Barlow-twins (Zbontar et al., 2021)) and supervised learning; see Appendix G. All three trajectories are similar to the trajectory of supervised learning. We find that the trajectory for semi-supervised learning deviates from the supervised learning trajectory initially, but the two are very similar for larger values of progress (tw.).
4 RELATED WORK AND DISCUSSION
Understanding the space of learning tasks. A large body of work has sought to characterize relationships between learning tasks, e.g., domain specific methods (Zamir et al., 2018; Cui et al., 2018; Pennington et al., 2014), learning theoretic work (Baxter, 2000; Maurer, 2006; Ben-David et al., 2010; Ramesh & Chaudhari, 2022; Tripuraneni et al., 2020; Hanneke & Kpotufe, 2020; Caruana, 1997), random matrix models (Wei al., 2022). neural tangent kernel models (Malladi al., 2022) and information-theoreti analyses (Jaakkola & Haussler, 1999; Achille et al., 201 9a;b). Broadly speaking, this work has focused on understanding the accuracy of model on new task when it is
8

---

Under review as a conference paper at ICLR 2023
trained upon related task, e.g., relationships between tasks are characterized using the excess risk of a hypothesis.
(a)
Like these above works, our methods also allow us to say things like "task pl is far from p2 as compared to p3". But we heavily exploit information geometry and can go much further. Our methods shed light on the geometric struc- ture in the space of tasks using the geometry of probabilistic models of these tasks. This al- lows us make quantitative conclusions such as "the divergence between pl and pa eventually is more than that of pl and pa, but representations learned on these tasks are similar for 30% of the way". Therefore our methods allow us to paint a picture of the space of the tasks that is quantitative and globally consistent.
(b)
Nereekzee dietece betieee meen trojctstie
Disiance from Consasthe- -IEIEEI-----RRRRR-RI 10
20.0
17.5
18.0
There is strong structure in typical inputs, e.g., recent work on understanding generaliza-
Supardden APpeno
10.0
Auppordo Toes
tion (Yang et al., 2022; Bartlett et al., 2020) as well as older work such as (Simoncelli & Olshausen, 2001; Field, 1994; Marr, 2010) has argued that visual data is effectively low- dimensional. Our works suggests that tasks also share a low-dimensional structure. Just like the effective low-dimensionality of inputs enables generalization on one task, effective low-dimensionality of the manifold of models trained on different tasks could perhaps explain generalization to new tasks.
7.5
Supardiend - Taak
puparviod Tusk
Contrsttive Task
2.5
Conmestivve Task 2
Contrrative Tank
0.0
0.2
0.4
Progresa (7a)
Figure 4: (a) Trajectories of contrastive learning (Sim- CT.R) on 3 datasets (two classes cach) and entire CIFAR- 10 compared to those of supervised learning. SimCLR on entire CIFAR-10 10 learns a similar representation as that of the supervised learned model P. (which fits the training data perfectly). SimCLR trajectories are close to each other even if different datasets were used to train them. It may seem from the embedding that SimCLR trajectories are similar to that supervised leareinn which would be very surprising because the former does not any labels, but see below. tance between the mean trajectories of all models and the mean trajectory of SimCLR all 1I0AR-10. This distance is normalized by the average of the tube radii (like Fig. 1g). SimCLR trajectories of two-class datasets are indeed very close to cach other (mean distance is ~ more than their tube radii for about 45% of the way (tw. z 0.2)). This plot indicates that two-class Sim- CLR trajectory (light blue) is close to SimCLR all of CIFAR- 10. But two-class supervised learning trajectory (darker blue) is much farther away from SimCLR on all of CIFAR-10.
Relationships between tasks in computa- tional neuroscience. Our results are conceptu- ally close to those on organization and represen- tation of semantic knowledge (Mandler & Mc- Donough, 1993). Such work has primarily used simple theoretical models, e.g., linear dynamics of Saxe et al. (2019) (who also MDS)) Our tools are very general and paint similar picture of ontologies of complex tasks. There is some resemblance in how our different networks learn the task in a similar fashion (low- now-dimensioonl manifold, trajectories across different random seeds and similar tasks are very close to each other), and how different individuals share the same notion of what constitutes a task (Rosch & Mervis, 1975). Concept formalization and specialization over age (Vosniadou & Brewer, 1992) also resembles our experiment in how fine-tuning models trained for longer periods changes the representation marginally. Our goals are similar to those of works like Sorscher et al. (2021) but our focus very different. We have focused our techniques on studying how representation learning on multiple tasks works.
(b) Bhattacharyya dis-
Visualizing training trajectories of deep networks. A large number of works have investigated trajectories of deep networks and the energy landscape during or after training using dimensionality reduction techniques (Horoi et al., 2021; Li et al., 2018; Huang et al., 2020) and study if learning occurs in oiw-dimensional sub-space (Gur-Ari et al., 2018: Antognini & Sohl-Dickstein, 201 8). The key distinction with respect to this existing work-an the function space, not the weight space. We study the function space of deep networks by evaluating the underlying probabilistic model finite samples. While the weight space has symmetries (Freeman & Bruna, 201 7; Garipov et al., 2018) and nontrivial dynamics (Tanaka & Kunin, 2021; Chaudhari & Soatto, 2018), 1)1 the function space, i.e,. our large vector [o, N xC 3 {p I xx)} completely characterizes the output of the model. In comparison, the loss or the error which are typically used to
our key technical innovation is that we visualize
reason about relationships between tasks, are very coarse summaries of the predictions. Any two

---

Under review as a conference paper at ICLR 2023
models trained on the same task can be rigorously studied in our to understand their predictions. Another key innovation that our mapping procedure allows us to study the representation of the penultimate layer of any two models, even if they were trained on different tasks.
function space"
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439, 2019a.
Alessandro Achille, Glen Mbeng, and Stefano Soatto. Dynamics and Reachability of Learning Tasks. vrXiv:/810.02440 [cs, statj, May 2019b.
Shun-ichi Amari. Information Geometry and Its Applications, volume 194 of Applied Mathernatiaa Sciences. Tokyo, 2016.
Joseph Antognini and Jascha Sohl-Dickstein Pea high dimensional random walks with comparison to neural network training. Advances in Neural Information Processing Systems, 31, 2018.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Peter L Bartlett, Philipp M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 1177499:30033330077 2020.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-do stereograms. Nature, 355(6356::611-63,, 1992.
Shai BeviDavid, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):1111175, 2010.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6-8, 1992.
Anil Bhattacharyya On measure of divergence between multinomial populations. Sankhyā: the indian journal of statistics, pp. 401-406, 1946.
Richard P. Brent. An algorithm with guaranteed convergence for finding a zero of a function. The computer journal, 14(4):422-425 1971.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. ICLR, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models strong semi-supervised learners. Advances in Neural Information Processing Systems, 53:22243-22255 2020b.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4109-4118. 2018
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scalt scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A bascline for few-shot image classification. In ICLR, 2020.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.

---

Under review as a conference paper at ICLR 2023
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks, Advances neural information processing systems, 27, 2014.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta- Meta-Q-Learning In ICLR, 2020.
David Field. What the goal of sensory coding? Neural computation, 6(4): 601, 1994.
Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In ICLR, 2017.
Yansong Gao and Pratik Chaudhari. A free-energy principle for representation learning. In Proc. of International Conference of Machine Learning (ICML), 2020.
Yansong Gao and Pratik Chaudhari. An information- distance on the space of tasks. In ICML, 2021.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson Loss surfaces, mode connectivity, and fast ensembling of dnns. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1803-88122 2018
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367-4375, 2018
Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace. arXiv preprint arXiv:1812.004 2018.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Steve Hanneke and Samory Kpotufe. A no-free-lunch theorem for multitask learning, arXiv preprint 2tfi:2000615788. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Stefan Horoi, Jessie Huang, Guy Wolf, and Smita Krishnaswamy, Visualizing high-dimensional trajectories on the loss-landscape ANNs. arXiv preprint arXiv:2102.00455 2021.
Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Deep transfer metric learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 325-333, 2015.
W Ronny Huang, Zeyad Emam, Micah Goldblum. Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. 2020.
Sosuke Ito and Andreas Dechant. Stochastic time evolution, information geometry, and the Cramér-Rao
bound. Physical Review X, 10(2)021056,, 2020.
Tommi Jaakkola and David Haussler. Exploiting generative models in discriminative classifiers. In Advances in Neural Information Processing Systems, pp. 487-493, 1999.
Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework learning. arXiv preprint arXiv:2202.09931 2022.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting sele-super visual representation learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 1920-1929, 2019.
Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big Transfer (BiT): General Visual Representation Learning. arXiv: 19/2. 1370 [es], May 2020.
12

---

Under review as a conference paper at ICLR 2023
A. Krizhevsky. Learning Multiple Layers of Features from Tiny Images. PhD thesis, Computer Science, University of Toronto, 2009.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. fiev. https://gi //ithbb. com/libffcv/ffcv/ 2022. commit хXхXXXX.
Hao Li. Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape of neural nets. In ICLR, 2018.
Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Rethinking the hyper-parameters for fine-tuning. In Proc. of International Conference of Learning and Representations (ICLR), 2020.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning, arXiv preprint arXiv:2210.05643, 2022.
Jean Mandler and Laraine McDonough. Concept formation infancy. Cognitive development, (3):291-318, 1993.
David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. 2010.
Andreas Maurer. Bounds for linear multi-task learning. The Journal of Machine Learning Research, 7:117-139, 2006.
George A Miller. WordNet: An Electronic Lexical Database. 1998.
Frank Nielsen and Sylvain Boltz. The burbea-rao and bhattacharyya centroids. JEEE Transactions on Information Theory, 57(8):5455-5466 5466, 2011.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP),, pp. 1532-1543, 2014.
Hang Qi. Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9822-5830. 2018.
Katherine N Quinn, Colin B Clement, Francesco De Bernardis, Michael D Niemack, and James P Sethna. Visualizing probabilistic models and data with intensive principal component analysis. Proceedings of the National Academy of Sciences, 116(28):13762-13366 2019.
Rahul Ramesh and Pratik Chaudhari. Model Zoo: A Growing "Brain" That Learns Continually, In ICLR, 2022.
Eleanor Rosch and Carolyn B Mervis. Family resemblances: Studies in the internal structure of categories. Cognitive psychology. 7(4):573-605, 1975.
Andrew Saxe, James I. McClelland, and Surya Ganguli. mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 1 116(23); 11537-11546, 2019.
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroxcience, (4(1):11933112 2001.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference Neural Information Processing Systems, pp. 4080-4090, 2017a
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087 2017b.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
13

---

Under review as conference paper at ICLR 2023
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. The geometry of concept learning. bioRxiv the preprint server for biology, 2021.
Hidenori Tanaka and Daniel Kunin. Noether's s learning dynamics: The role of kinetic symmetry breaking in deep learning. arXiv preprint arXiv:2105.02716, 2021.
Han Kheng Teoh, Katherine N. Quinn, Jaron Kent-Dobias, Colin B. Clement, Qingyang Xu, and James P. Sethna. Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding. Physical Review Research, 2(3):033221, August 2020. ISSN 2643-1564.
Sebastian Thrun and Lorien Pratt. Learning Learn. 2012.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning linear representations. arXiv preprint arXiv:2002.11684 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630-3638, 2016.
Stella Vosniadou and William F Brewer. Mental models of the earth: A study of conceptual change in childhood. Cognitive psychology, 24(4):535-555 1992.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. arXiv preprint arXiv:2203.061 061206 2022.
Rubing Yang, Jialin Mao, and Pratik Chaudhari. Does the data induce capacity control in deep learning? ICMI, 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016, 2016.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, pp- 12310-12320) 2021.
Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pp. 7324-7334 PMLR, 2019.
14

---

Under review as a conference paper at ICLR 2023
A DETAILS OF THE EXPERIMENTAL SETUP
Data
We performed experiments using two datasets.
1. CIFARIO (Krizhevsky, 2009) has 10 classes (airplane, automobile, bird, cat, deer, dog, frog. horse, ship, truck) with RGB images of size x22 and
2. Imagenet (Deng et al., 2009) has 1000 classes each with about 1000 RGB images of size 224x224.
Architectures We use Wide-Resnet (Zagoruyko & Komodakis, 2016) architecture for supervised learning experiments on CIFAR-I0 (WRN-16-4 with depth 16 and widening factor of 4) and a R8 18 (He et al., 2016) to train a model using SimCLR. All experiments on Imagenet use the Resnet-50 architecture.
All convolutional layers are initialized using the Kaiming-Norman initialization. For the Wide- Resnet, the final pooling layer is replaced with an adaptive pooling layer in order to handle input images of different sizes.
We make three modifications to these architectures.
1. We remove the bias from the final classification layer; this helps keep the logits of the different tasks on a similar scale.
2. In the experiments pertaining to Results 4 and 5, we replace batch normalization with layer norm in the batch-norman normalizatoon parameters make training meta-leerning models very sensitive to choices of prree-pparamttrs (e.g. the support or query shot), and that the learned representations of new tasks were quite different in terms of their predictions (and thereby the Bhattacharyya distance) but all the difference was coming from modifications to the BN parameters.
This is because we found in preliminary experiments that
3. In the Resnet-50, we replace the pooling layers with BlurPool (Zhang, 2019). The bias parameter in batch normalization is set to zero with the affine scaling term set to one.
Training procedure All models are trained in mixed-precision (32-bit weights, 16-bit gradients) using stochastic gradient descent (SGD) with Nesterov's 's acceleration with momentum coefficient set to 0.9 and cosine annealing of the learning rate schedule. Patch-normmllizttion parameters are excluded from weight decay.
CIFARIO datasets use padding (4 pixels) with random cropping to an image of size 28 x 28 or x2x32 respectively for data augmentation. CIFAR10 images additionally have random left/right flips for data augmentation. Images are finally normalized to have mean 0.5 and standard deviation 0.25. Supervised learning models (including fine-tuning) for CIFAR1O are trained for 100 epochs with a batch-size of 64 and weight decay of 10 using the ReddeRR
Episodic meta-learners are trained using Wide-Resnet and with the prototypical loss (Snell et al., 2017a). For the 2-way meta-learner, each episode contains 20 query samples and 10 support samples. For the 5-way meta-learner, each episode contains 50 query samples and 10 support samples. We found Result 5 to hold across different choices of these hyper-parameters in small-scale experiments. Models are trained for around 750 epoch and the episodic learner is about 5 times slower to train with respect to wall-clock time.
We train models using SimCLR on CIFARIO and on tasks created from CIFARIO. For the augmentations, we use random horizontal flips, random grayscale, random resized crop and color jitter. Models trained for 200 epochs for 2-way classification problems and for 500 epochs when trained on the entirety of CIFARIO with the Adam optimizer and an initial learning rate of 0.001.
A.I EXPERIMENTS ON IMAGENET
We make use of FFCV (Leclere et al., 2022). which is data-loading library that replaces the pytorch Dataloader. FFCV reduces the training time on Imagenet to a few hours, which allows us to train 100s of models Imagenet, or on tasks created from it. Our implementation of Imagenet training builds on the FFCV repository !.
Imagenet models are trained for 40 epochs with progressive resizing - the image size increased from 160 to 224 between the epochs 29 and 34. Models are trained on 4 GPUs with a batch-size of mandom-resized crop and random horizontal
512. The training uses two types of augmentations flips. Additionally, we use label smoothing with the smoothing parameter set to O.
https::/ggithbb.cmmmiiffff
15

---

Under review as a conference paper at ICLR 2023
A.2 IMPLEMENTING INPCA VERY HIGH DIMENSIONS
We calculate an InPCA embedding of models along multiple trajectories, e.g., typical experiment has about 25 trajectories (multiple random seeds, tasks, or representation learning methods) and about 50 models (checkpoints) along each trajectory. Each model is a very high -igeediinnsional object (with dimensionality NC where N 10 and C 10-10"). Even if the matrix in (9) is relatively manageable with 1250, each entry of D is dn Po) and therefore requires ~ 10 operations
to compute. Implementing InPCA- even PCA-for such large matrices requires a large amount of RAM. We reduced the severity of this issue to an extent using Numpy's 's memmap functionality hww.mp/..rr/doo//tabc/eeeeeeeee
wwwprerttteeemmy.mmmmm.iae html.. Also note that calculating only the top few eigenvectors of (9) suffices to visualize the models, we not need to calculate all.
The formula (2) is an effective summary of the discrepancies between how the predictions made by two probabilistic models differ; even small differences in two models, e.g., even if both Pa and P. make mistakes on exactly the same input samples, if pu (c) is slightly different than pa (e) for even one of n or c, the divergence non-zero. InPCA is capable of capturing the differences between two such models (9). However, when the number of classes is extremely large, the number of terms in the summation prohibitively large and analyzing the discrepancies or calculating the embedding becomes rather difficult.
We also developed a method to work around this issue. We can use a random stochastic matrix (whose columns sum up to 1) to project the outputs for each sample {p" (c)))c...... c into smaller space before calculating (2). This amounts to pretending as the model predicts not the actual classes but a random linear combination of the classes (even if the model is trained on the actual classes). This is a practical trick that is necessary only when we are embedding a very large number of very nigh-ddmensional probabilistic models. We checked in our Imagenet experiments that using this trick gives the same embeddings.
In this paper, we did not need to use this projection trick.
B CALCUUATTING MEAN TRAIECTORIES
We defined the distance between two trajectories to be (rnj . -22- i.e., the integral of the Bhattacharyya distance between the trajectories after mapping them to the same task and re-indexing them using the geodesic. This quantity is key in our work. Say we wish compare a model trained on two tasks: cats vs. dogs and airplane vs. truck from CIFAR-10. We initialize multiple models for each of these two supervised learning problems (and do so for every experiment in this paper) and train these 10 models. We can now calculate the mean trajectory of models on a task
argmin
1 K
Σ dnsi TH)-
This optimization problem is very challenging because the variable is trajectory of probabilistic models a high-dimensiooal space. Even if we were to split this minimization and do it independently across time, this is still difficult because the solution is the so-called Bhattacharyya centroid on the product manifold defined in (1) and cannot be computed in closed form. See (Nielsen & Boltz, 2011) for an iterative formula. We therefore simply take the arithmetic mean of the probability distributions, i.., Pu(t) k Ek= Pwr(t)- This is similar to ensembling. We use the radius of the tube around the mean trajectory, i.e.,
max (Te (TuLe Tb)
to normalize distances (more precisely, we normalize using the average of the radii of the two trajectories being compared). Note that this radius depends upon time (and is computed after mapping and reindexing the trajectories). If the distance between the means of two sets of trajectories is smaller than their individual average radii, then the tubes around the means intersect each other. In such cases, one can say that the representations learned (at that time point) are not distinguishable. We next show all distances between reindexed points along the trajectories discussed in Figs. 1, 3 and 4. Note that each curve gives the integrands in (5), not the integral.
C BHATTACHARYYA DISTANCE
In this section, we provide additional details regarding (2). Let y = (. uN)), denote the labels assigned to each of the N samples. Since there are C classes in total, can take a total of N" different values denoted by the set yCC Given, two models Pu and P., the Bhattacharyya averaged over the samples is
16

---

Under review as conference paper at ICLR 2023
Thac Eppovo
1e Epech
Lo Thik -ppm
10
0.30
0.25
Teaks Eaoohi
Tosk O- Ezoch 10
Took D: Epean 100
Toon E-Epeoh
0.0
3.55
0.93
6.2
-395
3.33
03
107
347
207
Took det
booc Rocha
Tex: 2
Tea: Caee
0.5
Text 1-220013
Tasc -auch 1
(Fa)
(a)
Dista 1c0 From Bird
Distance from Conveyance
Distance fhom Dog
Distance From Instuments
15.0
w
Noraatzed dreaanee betwoon mann
2.5
2.5
0.0
0.0
Bd4
0.0
02
D4
0.0
02
0.4
0.0
02
04
Distane from Invertebrate
Distonce from Rand 388 cosgge
Distance from Reptle
Distanoe from Vertebratos
15.2
15.0
12.5
12.5
12.5
12.5
Bid
Conweyance
10.2
Dog
7.5
7.5
Inllumet to
Inertbbats
5.0
Hand 333 canous
Racilla
Vatobbslic
02
Progress (ta)
(b)
Supontisd QIFR10
Supentisd: Task1
Suponvised Task
Supervisdd Task
Nomalzed distanes trajoctooiis
20
20
20
20
G-
0.0
0.5
0.1
0.2
0.0
Corinstha GIFAR
Controntive Taak
Contixatioon Task
Contiratioe Trak
Superised. OIAR11
20
20
Superited Taak
Superieed: Taak
Sipenieed: Tank
Cantret ve Taxc
Contrat ve Take:
Contraatve Taee
Controste ve CIFAR -IFFAR
13
0.
05
0.
0.2
04
00
62
0.4
0.0
0.
Progress (tw)
(c)
Figure A1: This figure shows the extended version of distances between trajectories of probabilistic models; two of them are identical to the ones in Fig. 3b and Fig. 4b.
17

---

Under review as a conference paper at ICLR 2023
Pp),u,Po) == -N- log
Σ
Pol
NØEC
=-N log
Σ
II
v Vul(Uo) (olun)
1=1
C
C
N
--N log
Σ
Σ
Σ
II
V (u((u)) polum)
M1
1/2m
V/
Amml
C
(N-1
VPu(UN) pulun)) ))
-N log
Σ
II
v pulun) pu(vm)
Σ
1m1
c
- log
Σ
Σ
II in=1
v Pala)) Po(um)
II
Σ
v (u(s)) po(ui)
=1
BN DN-2=1
81=1
N-h+1
N
- log
Σ
Pu((i) (vs)
N
N
log
Pu((e)) (v6)
===1
D MEASURING GOODNESS-OF-EI OF AN INPCA EMBEDDING USING EXPLAINED STRESS
We would like to measure if a k-dimensional sub-space accurately preserves the true distances. For this purpose, define quantity called the "explained stress" that estimates the fraction of pairwise distances in the original space that are preserved in the k-dimensional embedding. This is analogous to the explained variance in principal component analysis (PCA); but explained variance is a measure of the how well the original points are preserved in the embedding whereas explained stress approximates how well pairwise Bhattacharyya distances are preserved. If we consider the embedding to be given by first & digen-vectors then the explained stress (XR) is
W Σkm En U.U,"
23 Σm E
(12)
Xk
||W||F
Note that InPCA finds an embedding that exactly maximizes the objective Xk.
E IMPRINTING RESULTS IN WEIGHTS THAT MAXIMIZE THE LOG-PROBABILITY UNDER NORM CONSTRAINT
Consider a total of C' classes. In this section, we show that imprinting results in weights (we ) that maximize the log-probability of the samples while satisfying the constraint [|wel] = 1. Hence, imprinting can be used instead of training the last layer.
Let the set of all samples in class c be (af) 1- We define the log- rigpabbbaylliyy with which sample æy belongs to c, as we p(a;), where sp(a;) are the feature of sample æ;. Hence,
The
Σ logg(y
(x))
Σ
We Σ
(13)
If we don't have a constraint on the norm of we, then we can maximize (13) by sending we + co. Let
Ne
W argmax
p(af),
w=||=|
18

---

Under review as a conference paper at ICLR 2023
denote the weights that maximize the bog-probability under the E2 norm constraint. Since 2/ ||]]] maximizes the objective w x, subject to the constraint that [ = 1, the optimal value for we is
Σ,
COMPARING DIFFERENT SEMI-SUPERVISED, CONTRASTIVE AND SUPERVISED LEARNING METHODS
Normalized ditannee between mean trajectorios
Darton Faaroron
Distance from Supervised
23.0
SimCLR
17.5
Barlow Twins
15,0
FlxMatth
12.5
5.0
Progress (ta)
(b)
(a)
We consider 4 methods for training on CIFARIO: supervised learning, SimCLR (Chen et al., 2020a), Barlow-twins (Zbontar et al., 2021) and Fixmatch (Sohn et al., 2020). Fixmatch is a semi-supervised learning method and has access 2500 labeled samples in addition 47500 unlabeled samples. SimCLR and Barlow-twins are contrastive learning methods that use all the 50,000 unlabeled samples for training.
Figure A2:
(a) We plot the trajectories for supervised, semi-supervised and contrastive lcarning. The trajectory of semi- supervised learning (Fixmatch) eventually resembles supervised learning in comparison contrastive learning methods. All methods result in remarkably similar trajectories although some of these methods are clearly trained using only unlabeled data.
(b) We plot the normalized distance trajectories corresponding to contrastive and semi-supervised learning to the trajectory of supervised learning. Semi-supervised learning (Fixmatch) deviates considerably from the other methods at the beginning. We speculate that this is because the trajectory of Fixmatch is influenced by the 2500 labeled samples. As, training progresses, Fixmatch becomes increasingly similar to supervised learning evidenced by the dip in the blue line for larger values of progress (fw)
G DISTANCE BETWEEN TRAJECTORIES OF EPISODIC META-LEARNING AND SUPERVISED LEARNING
19

---

Under review as conference paper at ICLR 2023
Normalized distance brelween trajediores
Supervieed Leamiin
Meta lemming S-way
Mett-leaming 2-way
60
Superiens Learing
Metn-bem bemeeeemn ng Eweay
Mota leam 2-way
700
0.00
0.25
0.50
275
1.00
0.00
22
0.50
25
10
0.50
0.25
01500
0.75
1.00
Progress (Ca)
Figure A3: Distance between trajectories of supervised and meta-learning at different values of progress. Distances between the average trajectories of different algorithms (e.g., 2-way episodic learning and supervised learning, and 5-way episodic learning and supervised learning in the lefimost panel) are normalized by the average of the radii of the tubes corresponding each trajectory. We find that trajectories of 2-way meta-learning deviate significantly from those of supervised learning for a large fraction of the trajectory. On the other hand, 5-way meta-learning is similar to the supervised learning trajectory for almost the entirety of the trajectory.
20


Reviews:
[{"review_id": "zrVZMRuWUTp", "reviewer": "Reviewer_hj5S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyses representations learned by neural network. The analysis is done via isometric mapping of the probabilistic model into a lower dimensional maniforld such that it preserves the pairwise distance between models. They propose a distance metric to compare different learning trajectories.  They also use InPCA to embed the model to a lower dimensional space. By doing so they can visualize a trajectory of the learning process and compare trajectories of different models.  \nThe main findings are: 1)the manifold of  the models trained on different tasks are low-dimensional 2)supervised learning help training of dissimilar tasks especially if it has diverse classes. 3) pre-training for longer epochs could hinder fine-tuning of the downstream task. 4) episodic meta-learning fits similar models than supervised learning even if they take different trajectories. 5) contrastive learning trained on different datasets learn similar representations. \n", "summary_of_the_review": "I think the paper has merits for publication. Analysis of deep learning representations and models can help us better understand what goes on the learning process. It seems to me the main novelty of the paper is distance metric between training trajectories which can help us measure similarity of models and tasks. The paper could have performed more extensive experiments (see weaknesses for details. )", "strength_and_weaknesses": "Strengths:\n- I generally find this sort of analysis interesting which can help us better understand what goes on the learning process. \n- Having a quantitative method to compare learning trajectories is very interesting. \n- The paper is generally organized well. The experiments support the contribution claims. \n\nWeakness:\n- The authors mention in the abstract that they have surprising findings, however, all their findings are inline with the literature.\n- They could apply their method on more diverse dataset instead of sampling classes from the same dataset or 2 relatively similar datasets (i.e. CIFAR10 and Imagenet)\n- CIFAR10 is a very small dataset to apply self-supervised learning. \n- It would have been more interesting if they had conducted experiments on more self-supervsing methods and to compare different self-superviseing methods. \n- Fig 4 does not seem to be referenced in the text. \n- Organization of results is very condense. It could be done better. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YWldBsucStU", "reviewer": "Reviewer_wHo9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper develops a set of tools to measure model distances, and visualize model embeddings/training trajectories, and thus make measurements on different tasks' relationship. Techniques used includes Bhattacharyya distance, linear layer imprinting, InPCA and others. The paper then uses these tools to make several interesting observations on self-supervised learning, meta learning, supervised learning and fine-tuning and find the representations are quite shared. ", "summary_of_the_review": "To summarize, the paper proposes a set of useful tools to analyze and compare different models, and present interesting findings with them from the perspective of task relationships. But the paper also has many points to improve on. Therefore I will recommend \"marginally above\".", "strength_and_weaknesses": "Strengths:\n\n-- The paper proposes a set of tools that can be used to analyze neural network training, trajectories and similarities in general, even for models on different tasks.\n\n-- The paper makes several interesting observations on the representations learned.\n\nWeaknesses:\n\n-- The metric Bhattacharyya distance relies only on the output y, so even if the intermediate representation is very different it still calculates two models as very close. This may not be ideal, especially when two models agree on the output but not representations. The later tools all rely on this distance so this affects all metrics and observations in this paper.\n\n-- Given the point above, when comparing models trained on the same task (e.g., supervised learning and meta-learning), it may not come as a surprise that their eventual similarity is high, since both of them will get high accuracy in the end. \n\n-- I feel the presentation and clarity can be greatly improved, especially for audience without a prior background in this topic. \n\n-- The paper's introduction does not cite any prior work which is weird. I also feel the six listed findings do not have to repeat in introduction and abstract, maybe only one place is enough.\n\nOther comments:\n-- Remark 1: \"Note that if we were to train the classifier w2 (with backbone w1 fixed) using samples from the other task under the constraint that rows of w2 have unit l2 norm, then we would obtain the imprinted weights as our solution.\" I get that imprinting is a way to substitute linear probing, but Is there a mathematical proof for this statement?\n\n-- Equation (2's) \"*\" step is still not that clear to me. Maybe it could be made more clear by having an additional intermediate step. \n\n-- \"where Lij = δij − 1/n is the centered version of D.\" The centered version of D seems to be W.\n\n-- The last equation in Section 2, can you make it more clear why it indicates isometry?\n\n-- The axes and ticks in figures can be made thinner and more transparent to improve the visualization and highlight the trajectories. \n\n-- I don't quite understand the \"n\" for equation (9) and the trajectories plotted in figure 1. Given that each epoch produces a standalone model, for an intermediate model, where to select the other n-1 models to construct a joint model embedding?\n\nTo summarize, the paper proposes a set of useful tools to analyze and compare different models, and present interesting findings with them from the perspective of task relationships. But the paper also has many points to improve on. Therefore I will recommend \"marginally above\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FupZKjCijwt", "reviewer": "Reviewer_opzC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors use the Bhattacharyya distance to define notions like the distance between learning trajectories on classification tasks. They also define a simple notion of transfer learning, and use that to compute learning trajectory distances across different transfer tasks and ", "summary_of_the_review": "Overall, the use of information geometry + InPCA embedding is a very nice way to visualize learning trajectories with different setups. However, the overall results seem to mostly be known, and the presentation, particularly for the figures, can be hard to parse.", "strength_and_weaknesses": "Using the Bhattacharyya distance gives a nice way to generate normalized comparisons between different approaches. In addition, mapping sub-tasks to the union task gives a simple way to compare aspects of representation learning.\n\nIt is not clear to me that the time parameterization using the geodesic from the random initialization to the final solution is a good normalization in all situations. It may provide too much weight to early time learning dynamics. However, the fact that the authors derived a relatively simple normalization procedure is to their credit.\n\nThe claim of result 1 feels like an overreach. Dimensionality reduction techniques should not be conflated with dimensionality counting techniques. In addition, 33% of the variance explained is not a large quantity. While low dimensional dynamics exist in many ML settings (including perhaps this one), it is not clear what fractions of the dimensions are necessary to capture the relevant information. For example if much of the captured variance is early in the trajectories (when accuracy of the classifier is low), the low-dimensional picture may not capture the more interesting aspects of feature learning later in the trajectories.\n\nResult 2 is perhaps less surprising, given results that the early layers of convolutional networks learn similar features in different tasks (e.g. https://link.springer.com/chapter/10.1007/978-3-319-10590-1_53). Result 3 is interesting in principle, but is very hard to see from the plot - again suggesting that the normalization may not be well-suited to answer certain questions.\n\nResult 4 is also known. Result 5 is potentially interesting; however, the fact that the models arrive at similar points in information space is not surprising if both methods are known to produce solutions with similar accuracy. Result 6 is also potentially interesting, but I had trouble understanding the text and the figure, and their relationship.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject