Transferring Learning Trajectories of Neural Networks
Authors: Daiki Chijiwa
Venue: ICLR 2024 poster (2024)
Keywords: neural networks
learning dynamics
permutation symmetry
loss landscape
Primary area: general machine learning (i.e., none of the above)
TL;DR: We formulate the problem of transferring learning trajectories between neural networks, and derive the first algorithm to approximately solve it.

Abstract: Training deep neural networks (DNNs) is computationally expensive, which is problematic especially when performing duplicated or similar training runs in model ensemble or fine-tuning pre-trained models, for example. Once we have trained one DNN on some dataset, we have its learning trajectory (i.e., a sequence of intermediate parameters during training) which may potentially contain useful information for learning the dataset. However, there has been no attempt to utilize such information of a given learning trajectory for another training. In this paper, we formulate the problem of "transferring" a given learning trajectory from one initial parameter to another one (named *learning transfer problem*) and derive the first algorithm to approximately solve it by matching gradients successively along the trajectory via permutation symmetry. We empirically show that the transferred parameters achieve non-trivial accuracy before any direct training, and can be trained significantly faster than training from scratch.

Published as a conference paper at ICLR 2024
TRANSFERRING LEARNING TRAJECTORIES OF NEURAL NETWORKS
Daiki Chijiwa" NTT Computer and Data Science Laboratories, NTT Corporation
ABSTRACT
Training deep neural networks (DNNSs is computationally expensive, which is problematic especially when performing duplicated or similar training runs model ensemble or fine-tuning pre-trained models, for example. Once we have trained one DNN on some dataset, we have its learning trajectory (i.e., sequence of intermediate parameters during training) which may potentially contain useful information for learning the dataset. However, there has been no attempt to utilize such information of given learning trajectory for another training. this paper, we formulate the problem of "transferring" given learning trajectory from one initial parameter to another one (named learning transfer problem) and derive the first algorithm to approximately solve it by matching gradients successively along the trajectory via permutation symmetry. We empirically show that the transferred parameters achieve non-trivial accuracy before any direct training, and can be trained significantly faster than training from scratch.
INTRODUCTION
Enormous computational cost is major issue in deep learning, especially in training large-scale neural networks (NNs). Their highly non-convex objective and high-dimensional parameters make their training difficult and inefficient. Toward a better understanding of training processes of NNs, their loss landscapes (Hochreiter Schmidhuber, 1997: Choromanska et al., 2015) have been actively studied from viewpoints of optimization (Haeffele & Vidal, 2017; Li & Yuan, 2017: Yun et al., 2018) and geometry (Freeman Bruna, 2017; Simsek et al., 2021). One of the geometric approaches to loss landscapes is mode connectivity (Garipov et al., 2018; Draxler et al., 2018), which shows the existence of low-loss curves between any two optimal solutions trained with different random initializations data ordering. This indicates surprising connection between scemingly different independent trainings.
Linear mode connectivity (LMC), special case of mode connectivity, focuses on whether not two optimal solutions are connected by low-loss linear path, which is originally studied in relation to neural network pruning (Frankle et al., 2020). It is known that the solutions trained from the same initialization (and data ordering in the early phase) tend to be linearly mode connected (Nagarajan & Kolter, 2019; Frankle al., 2020), but otherwise they cannot be linearly connected in general. However, Entezari et al. (2021) observed that even two solutions trained from different random initializations can be linearly connected by an appropriate permutation symmetry. Ainsworth et al. (2023) developed an efficient method to find such permutations and confirmed the same phenomena with modern NN architectures. These observations strengthen the expectation on some sort of similarity between two independent training runs even from different initializations, via permutation symmetry.
In this paper, motivated by these observations, we make the first attempt to leverage such similarity between independent training processes for efficient training. particular, we introduce a novel prob- lem called learning transfer problem, which aims reduce training costs for seemingly duplicated training runs on the same dataset, such as model ensemble or knowledge distillation, by transferring learning trajectory for one initial parameter another one without actual training. The problem statement is informally stated as follows:
"Corresoonding author; da i ki ki.chiij wa@ntt.coo

---

Published as a conference paper at ICLR 2024
Learning Trajectory for #g (Source)
8
02
Learning Trojectory
Pormuss neurons & connections
(Iransferred)
lpen
Ise
6
Dutputs are invariant.
Transferred by
Permutation symmetry of neural networks. (Section 2.2)
Figure I:
Figure 2:
Transfer learning trajectory from one to an- other NN by a permutation symmetry я. (Section
Learning transfer problem (informal). Suppose that a source learning trajectory (0'.... o is given for some initial parameter 0f. Given another initial parameter 6%, called target initialization. how can we synthesize the learning trajectory (02, ... ,6,] for 89 efficiently?
To tackle this problem, as illustrated in Figure 2, we take approach to transform the source trajectory ......... 0f') by an appropriate permutation symmetry π as in the previous works of LMC. In Section 3, we formulate the learning transfer problem non-linear optimization problem for т. We also investigate how much the source trajectory for 6% can be transformed close to the target trajectory for 0g by the optimal æ, both theoretically and empirically. We then derive theoretically- grounded algorithm to approximately solve it, and also develop practical techniques to reduce its storage and computational cost. Since our final algorithm requires only several tens of gradient computations and lightweight linear optimization in total, we can transfer given source trajectory very efficiently compared training from scratch. In Section 4, first we empirically demonstrate that learning trajectories can be successfully transferred between two random or pre-trained initializations, which leads non-trivial accuracy without direct training (Section 4.1). Next we further confirmed that the transferred parameters can indeed accelerate the convergence in their subsequent training (Section 4.2). Finally, we investigate what properties the transferred parameters inherit from the target initializations (Section 4.3). Surprisingly, observed that, even when the source trajectory trained from poorly generalizing initialization 8 (and thus inherits poor generalization) the one transferred onto more generalizing initialization 09 acquires better generalization inherited from 09.
In summary, our contributions are as follows: (1) we formulated the brand new problem of learning transfer with theoretical evidence, (2) derived the first algorithm solve it, (3) empirically showed that transferred parameters can achieve non-trivial accuracy without direct training and accelerate their subsequent training, and investigated the benefit/inheritance from target initializations. Finally, more related works are discussed in Appendix C.
BACKGROUND
2.1 NEURAL NETWORKS
Let I., N E N. Let (x;; () be I.-layered neural network (NN) parameterized by 0 R with non-linear activation function R and intermediate dimensions (dg,... .d,) ) E N4+1, Given
an input x de the output fx; 0) xz E Rd4 is computed inductively follows:
x, oWW bb), 1..0--++br-. (i=1)
(i= 0)
24
(1
where W, Rd:xd-111 R4: weight matrices and bias vectors. Under these notation, the parameter vector 0 is described as (W,,b,,... ..... Wt. by.) E R.
Stochastic gradient descent (SGD) a widely used approach for training neural networks. Let X' be the input space de and y be the output space Rdz,, Let D be probabilistic distribution over the input- output space X' XxV, and L yxxy E be differentiable loss function. SGD trains neural network f(;; 0) by iterating the following steps: (1) Sampling mini-batch B = ((æ1.V1), -.. (2s, yb)) ~ D of size 5 E N. (2) computing estimated gradient 9B Σi_ VoL(f 0), v.) for the mini-batch and (3) updating the model parameter by f agn + (momentum) where Rso is a fixed or scheduled step size.
2

---

Published as a conference paper at ICLR 2024
2.2 PERMUTATION SYMMETRY OF NNS
For simplicity, we assume that all bias vectors b, are zero by viewing them part of the weight matrices. Let ④ be the parameter space {0 (Wi, Wt. E R for the I-layered neural network [2; 8). Now we introduce permutation group action the parameter space O For n N. let Sn denotes the symmetric group over {1...... : n C N, which the set of all bijective mapping We define our permutation group G by G Sd, x... x Sa,-1- The group action G x è > è is defined as follows: For π = (01, ... ,L-1) E G and E 6, the action To is defined by
(1, n fI, mn..
n = (o, ( ... WLoT-1))
where each o; is viewed the corresponding permutation matrix of size d, dz. We call this group action the permutation symmetry of L-layered neural networks.
Simply speaking, the action Te just interchanges the axes of the intermediate vector 2; of the neural network fa(x; f) with the corresponding hase change of the weight matrices and bias vectors (Figure 1). Thus we can see that this action does not change the output the neural network, i.e., f(; TQ) = [a; 0) for every I E X, 0 E O, π E G. In other words, the two parameters and r can be identified from the functional perspective of neural networks.
2.3 PARAMETER ALIGNMENT PERMUTATION SYMMETRY
Previous work by Ainsworth et al. (2023) attempts merge given two NN models into model by leveraging their permutation symmetry. They reduced the merge problem into the parameter alignment problem:
min |O; XEG
02112 ==(m)." (1-11) ISSSL
min
E
Ilo -..11
where 81 = (W....., WL) and 02 = (Zi.... ZL) are the parameters to be merged. To solve this, they also proposed coordinate descent algorithm by iteratively solving the following linear optimizations regarding to each az's:
max (a,, W, + 2/41 W++1) o
The form of this problem has been well-studied as a linear assignment problem, and we can solve it in very efficient way (Kuhn, 1955). Although the coordinate descent algorithm originally proposed for model merging, we can also use it for other problems involving the parameter alignment problem (eq. 2).
3 LEARNING TRANSFER
In this section, first formulate the problem of transferring learning trajectories (which call learning transfer problem) as a non-linear optimization problem. Next, we derive an algorithm solve it by reducing the non-linear optimization problem sequence of linear optimization problems. Finally, we introduce additional techniques for reducing the storage and computation cost of the derived algorithm.
3.1 PROBLEM FORMULATION
Let f(z; 0) be some NN model with an N-dimensional parameter 0 RN, A sequence of N- dimensional parameters (00 ... 00)) E (x++1) is called a learning trajectory of length T for the neural network [a;; 0) if the training loss for 6 decreases to the convergent one 0T, Note that we do not specify what represents priori; it could be iteration, epoch or any notion of training timesteps. Now we can state our main problem:
increases, from the initial parameter do
In our experiments, due practical constraints, mainly consider cases where represents training epach linear interpolating step explained in Section 3.4, which leads the length 7' heing smaller than hundred.
3

---

Published as a conference paper at ICLR 2024
Learning transfer problem (informal). Suppose that learning trajectory (69. .0] is given for an initial parameter 8% which we call a source trajectory. Given another initial parameter f2 "similar" to 8% in some sense, how can synthesize the learning trajectory (62.... ..2). which call a transferred trajectory, for the given initialization f?? (Figure 2)
To convert this informal problem into computable one, we need to define the notion of "similarity" between two initial parameters. As a first step, in this paper, we consider two initializations are "simi- lar" if two learning trajectories starting from them are indistinguishable up permutation symmetry of neural networks (Section 2.2). In other words, for the two learning trajectories (ff...,, and (0g. .0%).. we consider the following assumption:
Assumption (P). There exists permutation и satisfying (0) - 02-02 fort 1,... , where the transformation (06 0 ") is as defined in Section 2.2.
Under this assumption, we know the permutation л providing the equivalence between the source and transferred trajectories, we can recover the latter one (09. .... 05) from the former one (ff.... ... of and the permutation æ,, by setting 0 02 :(0 - 0f 0 inductively t (Figure 2). Therefore, the bearning-traeccton problem can be reduced to estimating the permutation π from the source trajectory (69. ... ,fT) and the given initialization 08.
Naively, to estimate the permutation π, we want to consider the following optimization problem:
min Σ [|( 1 (0) 112
However, this problem is ill-defined in our setting since each 0 is not available for S S T in advance. Even if we defined 03 :- 05-1 + (0; - -0 in the equation (4) as discussed above, the optimization problem became trivial since any permutation æ makes the L norm to be zero.
Thus we need to fix the optimization problem (eq. 4) not to involve unavailable terms. We notice that the difference 8 - 62 can be roughly approximated by a negative gradient at 8;-1 averaged over a mini-batch if the trajectory enough fine-grained. Therefore, we can consider the approximated version of equation (4) as follows:
Pr min Σ
TV Co Vo where 6 := 62,1 + (0) -01-1)
(5)
In contrast to the equation (4), this optimization problem is well-defined even in our setting because each 85,r, is defined by using the previous parameter 0/-1 inductively.
3.2 WHEN AND HOW DOES ASSUMPTION (P) HOLDS?
Before delving into the optimization problem Pr, here we briefly investigate when and how our Assumption (P) holds. To measure the similarity of two vectors T(6 - 01-1) and 0 0 g--1 we mainly use variant of normalized distance || 02 veell/v ||:
[02|| for two vectors U1 22 R". |22
which can also be considered as cosine distance when [1
R
First of all, we study the case of 2-layered ReLU neural networks (w) = Σm u, (E) Wiyay) with N hidden neurons, where a (2) max(2, 0) is ReLU activation. Let 00 (Wig)ish (u:).) and 80- ((w))) )ay. (uf))) be initialized by Kaiming uniform initialization (He et al., 2015), i.e. Wijs why Va lal) and Vi, oy U(- VN h Then it is theoretically shown that Assumption (P)
U[[ 1
holds at initialization with high probability when the hidden dimension N is sufficiently large:
Theorem 3.1 (See Theorem A.1 for details) Given two pairs of randomly initialized parameters (w, v) and (wl, ,,)), with high probability, there exists a permutation symmetry π E Sx such that the normalized distance between the expected gradients (V..,)) and B(z,z) ( W L), where (w", ,'") is the permuted one of (w', o') by n, can be arbitrarily small when N is sufficiently large.
Although this result only validates Assumption (P) initialization, can naturally expect that same similarity holds for more iterations (t > 0) due to (almost everywhere) smoothness of neural networks with respect to parameters. Figure (3a) empirically validates this expectation.
4

---

Published as a conference paper at ICLR 2024
Pesphhet
100
200
Souder
(a) 2-MLP on MNIST.
(b) Conv8 on different datasets.
(c) ResNet18 on different datascts.
Figure We evaluate cosine similarities between x00 - 03-1) and 05 - 002-1, where 0f or 05 represents parameter trained for 10t iterations, averaged over timesteps t. Solid lines plotted for the solution of equation (4), and dotted lines are for т being the identity transformation (i.e., the case not permuted) baseline. Fag. Experiments using 2-layered MLP with various hidden dimensions. Over-parameetrrzattin leads to higher cosine similarities of learning trajectories. Fig.(h, c): Experiments with modern network architectures (8-layered CNN ResNet18) on different datasets.
Also, empirically investigate the case of modern network architectures in Figure (3b), (3c). The results show that the overall trends of similarity between learning trajectories heavily depend on net- work architecture rather than datasets, which is also indicated by Theorem 3.1. Moreover, additional experiments Appendix E.8 show that Assumption (P) also holds even when the initialization is pre-trained, instead of a random one, in the very early phase of training. Nevertheless, it remains for future work to make the assumption more appropriate for modern architectures so that it holds for longer iterations beyond initialization, for example in the case of Conv8 on CIFAR-IO in Figure (3b), or in the case of pre-trained initializations rather than random initializations.
3.3 ALGORITHM: GRADIENT MATCHING ALONG TRAJECTORY
Now our goal to solve the optimization problem PT (eq. 5). However, the problem Pr seems hard to solve directly because the variable appears non-linearly in the second term Vør__ L. To avoid the non-linearity, we introduce sequence linear sub-problems (PahsssT whose solution converges to the solution for Pr. For each s E ".....TT) we consider the following problem:
P'
min
Vo V
(6)
Since the second term in P'd uses the solution T.-1 for the previous sub-problem P'_11 the unknown variable T. appears only in the first term n, Vo C in linear way. Morcover, the following lemma implies that the final solution कт from the sequence original problem Pr:
approximates the solution for
Lemma 3.2 Under some regularity assumption, we have 62,,, =s 62,,, for St
The proof will be given in Appendix B. Indeed, by this approximation, we find out that the solution Fr for the last sub- sub-problem P/r minimizes
Σ
n Voy Vol
Σ
o Vo Vo, Vog..p 1
where the right-hand side is nothing but the objective of the original problem Pr.
Algorithm gives a step-by-step procedure obtain the transferred learning trajectory (0). 03) by solving the sub-problems (P') (P' sequentially. In lines 2-6, it computes
average of gradients VoL over single mini-batch for each 0 00-2,0²-- (1 < / 1 s), which is required in the s-th sub-problem P'. (eq. 6). In line 7, the s-th permutation n, is obtained as a solution of the sub probllooblmmmmoo P'. which can be solved as linear optimization (eq. 3) using the coordinate descent algorithm proposed Ainsworth et al. (2023). Then we update the transferred parameter of for t 1,....,, in line 8.
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 Gradient Matching along Trajec- tory (GMT)
Algorithm 2 Fast Gradient Matching along Tra- jectory (FGMT)
Require: (0P.-.. 07) E (=) .09 E IR"
08 E IR. Require: (0.. 07"') E R
(=x(T+1),
1: for 8 1,... T do
for s 1,... T do
2: for 1,...,,.. do
2: Sample (21, (I..,)).... ... (1,, (b) ~ D.
3:
Sample (11, ... (26, ~ D.
) 8c) gi : Ei=ı Vor (2); ),-
4:
gi E 1
((f "--"); "),
4:
g 1 V [[f(2:; 0-- v6)
5:
Φ2 +- Σ 2im1
0 (x((( 12000/2'00 ())
5:
T. + arg min, Σi= 10g2 rgill 1
6:
end for
6:
for 1- = 1, do
7:
iTo c arg min,
Σ -وا Tglll
0 0 n. (0) 0 end
8:
for t = 1,...
do
9:
03 4- 02
+ ne 10 0
9: end for
10:
end for
10: return (0).... 05)
11:
end for
12:
return (02, 05)
3.4 ADDITIONAL TECHNIQUES
While Algorithm solves the learning transfer problem (eq. 5) approximately. it still has some issues in terms of storage and computation cost. Here we explain two practical techniques to resolve them.
Linear trajectory. In terms of the storage cost, Algorithm requires a capacity of times the model size to keep a learning trajectory of length T, which will be more substantial issue as model size increases or the trajectory becomes fine-g mne-grained. To reduce the required storage capacity, instead of keeping the entire trajectory, we propose to imitate it by linearly interpolating the end points. In other words, given an initial parameter 0f and the final of we define a new trajectory [0 0f) =
(0),... 01, , of) ) with 0f == (1 O= + XxO and 0 = An S ... S Av S AT = 1.
Previous studies on the monotonic linear interpolation (Good- fellow et al., 2015; Frankle, 2020) indicate that such a lincarly interpolated trajectory satisfies our definition of learning trajec- tories in SGD training of modern deep neural networks. Through- out this paper, we employ uniform scheduling for de defined by Az+1 d := 1/T. Next, we compare the transferred results between the linear trajectory |0_ 03' and the actual trajectory (09.... ) where each 8 is a checkpoint at the t-th training epoch on CIFAR- 10 with T = 60. Interestingly, the transfer of the linear trajectory more stable and has less variance than the transfer of the actual one. This may be because the actual trajectory contains noisy information while the linear trajectory is directed towards the optimal solution of Due to its storage efficiency and stability in accuracy, we employ the linear trajectory of the length T S 20 throughout experiments in Section 4.
60-
Phadery Tremt
Figure 4: Linear vs. actual trajec- tory (Conv8 on CIFAR-10).
Gradient caching, In terms of the computation cost, Algorithm requires O(T²) times gradient computation. To reduce the number of gradient computation, we propose to cache the gradients once computed instead of re-comuutin them for every = 1, 1.... .T In fact, the cached gradients V P::-1 L
and the re-computed gradients V Voj.1 L. are not the same quantity exactly since the intermediate
parameter 0/-_1 02 ., (0 8Q) takes different values for each s. However, they can be treated as approximately equal by Lemma 3.2 if we assume the continuity of the gradients. Now we can reduce the number of gradient computation from O(T2) to O(T') by caching the gradients once computed. We describe this computationally efficient version in Algorithm 2.
4 EXPERIMENTS
In this section, we empirically evaluate how learning transfer works on standard vision datasets. First, compare our proposed methods (GMT, FGMT) and two baselines (Naive, Oracle), which are

---

Published as a conference paper at ICLR 2024
2
Telee (3re
2
10
ud Spickin
(a) MNIST (2-MLP)
(b) CIFAR-IO (Conv8)
(c) ImageNet (ResNet-18)
Thakee itowcs
Tuanell tipac
15.
Soned
(d) ImageNet + Cars
(e) ImageNet CUB
(I) CIFAR-10 CIFAR-IIE 100
Figure 5: We plot the validation accuracies of the transferred parameter 02 for each f 1,... T with various datasets and NN architectures. We also provide the standard deviation over three for each experiment. The dotted bars show accuracies in standard training. (Upper) Transfer of learning trajectory on single dataset between random initial parameters. (Lower) Transfer of fine-tuning trajectory between pre- pre-trained parameters. For example, "ImageNet + Cars" means the transfer of the fine-tuning trajectory on the Cars dataset between the parameters pre-trained on ImageNet.
explained below, under the following two scenarios: (1) transferring learning trajectories starting from randomly initialized parameters and (2) transferring learning trajectories starting from pre-trained parameters (Section 4.1). Next, we evaluate how efficiently the transferred parameters can be trained in their subsequent training. (Section 4.2). Finally, we investigate what properties the transferred parameters inherit from the target initializations from viewpoints of loss landscape and generalization (Section 4.3). The details on experimental settings are provided in Appendix D.
Baselines. As baselines for learning transfer, we introduce two natural methods: Naive and Oracle.
Both in the two baselines, transfer given learning trajectory (09. .... ..,7)) by single permutation Thaive or Tezacle, according to the problem formulation Section 3.1. In the Naive baseline, we
define Waaive the identity permutation, which satisfies Tveive 0 0. In other words, the transferred
parameter by Naive is simply obtained as 05 2.5 = =0 + (0) - 89). On the other hand, in the Oracle baseline, first obtain a true parameter 0f by actually training the given initial parameter 0g with the same optimizer as training 0f Then we define Toracle by minimizing the layer-wise L distance between the actually trained trajectories 0%¹ 09 and Toracle (8f 09). where we simply apply the coordinate descent as explained Section 2.3. The Oracle baseline is expected to close to the optimal solution for the learning transfer problem via permutation symmetry. Finally, note that the Oracle depends only on the initial and final parameter, and thus not on the the whole trajectory or its length T.
Source trajectories. In experiments, as discussed in Section 3.4, we consider to transfer linear trajectories [0_ 0'] of length I' rather than actual trajectories for 0f due to the storage cost and instability emerging from noise. The transferred results for actual trajectories instead of linear ones can be found in Appendix E.I.
4.1 LEARNING TRANSFER EXPERIMENTS
Figure 5 shows the validation accuracies the transferred parameters 8y.,. for each timestep 1= T' during the transfer. For the baselines (Naive and Oracle), we set the t-th permutation T, by the fixed Teaive and Toracle for every t. For our algorithms (GMT and FGMT), the t-th permutation n, corresponds to n, Algorithm and 2.
7

---

Published as a conference paper at ICLR 2024
8
Faiing
60070
1000
(a) CIFAR-IO 10 (Conv8) (b) CIFARIO OCIFAROO
(c) ImageNet - Cars
(d) ImageNet + CUB
Figure 6: Subsequent training of the transferred parameters. In each figure, we plot the validation accuracies during the training of the transferred parameters, on the same dataset as the source trajectory being trained on. The transferred parameters obtained by solving the equation (5) can be trained faster than standard training, and Naive baseline in pre-trained initialization scenario.
In the upper figures 5a-5c, we transfer a learning trajectory trained with a random initial parameter on single dataset (MNIST (LeCun et al., 1998), CIFAR-10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009)) to another random initial parameter. We will refer to this experimental setting as the random initialization scenario. We see that methods successfully approximate the Oracle baseline. Also, we can see that FGMT, the fast approximation version GMT, performs very similarly previously computed gradients in GMT, but not FGMT, resulting in the stable behavior of FGMT.
outperforms GMT. This is probably because the update of ग. affects the
specialized dataset (a 10-classes subset of CIFAR-100 (Krizhevsky, 2009), Stanford Cars (Krause et al., 2013) and CUB- 200-2011 (Wah et al., 2011)) from an initial parameter that is pre-trained on ImageNet another pre-trained one. We refer to this experimental setting as the pre-trained initialization scenario. This scenario seems be more difficult to transfer the learning trajectories than the random initialization scenario shown in the upper figures, since the Naive baseline always fails to transfer the trajectories. We can see that, while our methods behave closely to the Oracle baseline up to the middle of the timestep, the accuracy deteriorates immediately after that. Nevertheless, the peak accuracies of our methods largely outperform those of the Naive baseline. By stopping the transfer at the peak points (i.e., so-called early stopping), we can take an advantage of the transferred parameters as we will see the next section.
In the lower figures 5f-5e, transfer learning trajectory of fine-tuning
4.2 ACCELERATED TRAINING OF TRANSFERRED PARAMETERS
In the previous section, we obtained the transferred parameters that achieve non-trivial accuracy without any direct training. Here we evaluate how efficiently the transferred parameters can be trained in their subsequent training, by training them on the same dataset for same epochs as the source trajectory. We started each training from the transferred parameter f22,,, at the best trajectory step in Figure 5. Figure 6 shows the validation accuracies for each epoch in the training of the transferred parameters. In all cases, the transferred parameter can be trained faster than standard training from random/pre-traaine trained initializations (denoted by Standard Training). In the random initialization scenario (6a), there seem almost difference between four transfer methods. This because the Naive baseline also achieves non-trivial accuracy already when transferred in this scenario. On the other hand, in the pre-trained scenarios (6b), (6c), (6d), the parameters transferred by our methods and the Oracle baseline learns the datasets faster than the parameters transferred by the Naive baseline. Thus the benefit of the transferred parameters seems be greater especially in the pre-trained initialization scenario than in the random initialization scenario.
4.3 WHAT IS BEING INHERITED FROM TARGET INITIALIZATION?
In previous sections, we have observed that transferring given learning trajectory to target initializa- tion can achieve non-trivial accuracy beyond random guessing and indeed accelerate the subsequent training after transfer. Then the following question arises: what does the transferred parameter differ from the source parameter and inherit from the target initialization? In particular, pre-traue pre-trained initializations may have their own "character" making them different from each other, due to their generalization ability or prediction mechanism.
8

---

Published as a conference paper at ICLR 2024
(a) CIFARIO-CCIFARION
(b) ImageNet->Carg
(c) ImageNet-
Figure 7: Inheritance of basin in loss landscape of fine-tunings. We plotted the validation accuracies
over the we- we-plane following the same protocol as Garipov et al. (2018).
(a) ImageNet (Full/10%) - CUB
(b) ImageNet (Full/10%) - Cars
Figure 8: Inheritance of generalization ability. We plotted validation accuracies and generalization gaps (i.e., gap between training/validation accuracies) CUB and Cars, fine-tuned from initial- izations pre-trained on (full or 10% subset of) ImageNet. FGMT (X = Y) means that the learning trajectory starting from an initialization X is transferred onto another initialization Y. (Even if X and Y are the same description, they are pre-trained with different random seeds.) )
Basin in loss landscape. In context of transfer learning, Neyshabur et al. (2020) empirically showed that fine-tuning one pre-trained initialization always leads to the same basin in loss landscape, which enables the fine-tumed models to share similar properties inherited from the same pre- pra-trai initialization. From this viewpoint, when a source trajectory with some pre-traine pre-trained initialization is transferred another (target) pre-trained initialization and then fine-tuned (referred FGMT+FT), is expected to arrive at the same basin as Target parameters actually fine-tuned from the target initialization. In Figure 7, we empirically validate this expectation, with comparison to the Source (Permuted) parameter, the end point of the source trajectory permuted with Git Re-basin (Ainsworth al., 2023) which transforms closest to Target. We can see that the transferred parameter lives in the nearly same basin as the target one, while there are mild barriers from the permuted source parameter. This implies that the transferred parameter inherits similar mechanism (Lubana et al., 2023) from the target initialization which cannot be obtained by simply permuting the source parameter.
Generalization ability. In Figure 8, we focus on the case that, unlike previous sections, the source initialization 89 has different (worse) generalization ability than target initialization 82. In particular, we use the source initialization 09 pre-trained on 10% subset of ImageNet, and the target initialization 09 pre-trained on full ImageNet. The former generalizes poorly (validation accuracy 50% on ImageNet) compared to the latter (validation accuracy > 72% on ImageNet). Surprisingly, the results show that (1) learning trajectories can be transferred between pre-trained initializations even with different generalization ability, and (2) the transferred parameter successfully acquires better generalization inherited from the target initialization 0% even though has never been trained on the target initialization before/during transfer.
5 CONCLUSION
In this work, we formulated the problem of how we can synthesize an unknown learning trajectory from the known one, named the learning transfer problem, and derived an algorithm that approx- imately solves it very efficiently. In our experiments, we confirmed that our algorithm efficiently transfers given learning trajectory to achieve non-trivial accuracy without training, and that the transferred parameters accelerate their subsequent training. Moreover, we investigated what properties the transferred parameters inherit from the target initializations from the viewpoints of loss landscape and generalization. The last observation potentially opens up new paradigm of deep learning in future: For example, when foundation model is newly updated with better generalization fixed vulnerability, its fine-tuned models may efficiently follow it by transferring their tine-tuning trajectory from old foundation model to the new one.

---

Published as a conference paper at ICLR 2024
REFERENCES
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re- rasin:::: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL https: /pppneeveeee net/ forum? 3dd=C@mmMY1115555
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-3894, 2020.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling, In International Conference on Machine Learning, pp. 769-779. PMLR, 2021.
Frederik Benzing, Simon Schug, Robert Meier, Johannes Von Oswald, Yassir Akram, Nicolas Zucchet, Laurence Aitchison, and Angelika Steger. Random initialisations performing above chance and how to find them. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022
Workshop), 2022.
Srinadh Bhojanapalli, Kimberly Wilber, Andreas Veit, Ankit Singh Rawat, Seungyeon Kim, Aditya Menon, and Sanjiv Kumar. On the reproducibility of neural network predictions. arXiv preprint arXiv:2102.03349, 2021.
Anna Choromanska, Mikael Henaff, Michael Mathieu, Gérard Ben Arous, and Yann LeCun. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. PMLR, 2015.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine-grained categorization and domain-specific transfer learning. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 4109-4118, 18, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, leee, 2009.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference machine learning, pp. 1309-1318. PMLR, 2018
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in lincar mode connectivity of neural networks. arXiv preprint arXiv:2110.06299 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint 1r1iv:1912.02757 2019.
Jonathan Frankle. Revisiting "qualitatively characterizing neural network optimization problems". In NeurlPS 2020 Workshop: Deep Learning through Information Geometry, 2020. URL https: / //openrevieww.oo net/ formm? id- 0mu8aLQ33y"
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269. PMLR, 2020.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations, 2017. URL. /tt.tttts////peeeeeee/ net/forunt iidiidd--n BFF-BBKFFWVggg
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems. 31, 2018.
Ian J. Goodfellow, Oriol Vinyals, and Andrew M. Saxe. Qualitatively characterizing neural network optimization problems. In International Conference Learning Representations, 2015.
10

---

Published as a conference paper at ICLR 2024
Benjamin D Haeffele and René Vidal. Global optimality in neural network training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7331-7339, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1.Xiv:1503.0253:: 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Weinberger. Snap- shot ensembles: Train 1, get for free, In International Conference Learning Representations, 2017. URL https: : / www.reveew......... net/ wtt.:/oormmmddiooe BJTwwYY
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song. Hannanch Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolat- ing weights. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /tttp::////eeeeveee. net/ 1d=CZ2FRxb014
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic, In The Eleventh International Conference on Learning Representations, 2023. URL htthttp:: htttttss//operreviiii net net/forum? 1d=6t0Kwf8-jri;
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. pmlr, 2015.
Keller Jordan. Calibrated chaos: Variance between runs of neural network training harmless and inevitable. arXiv preprint arXiv:2304.01910, 2023.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, 2023. URL https: http://///pppeeeee.e.eo wtt.:eewwweeefoo.....eeo
:d=gU5sJ62ggccx
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Linear connec- tivity reveals generalization strategies. In The Eleventh International Conference on Learning Representations, 2023. URL /ttttp://openneee revierevviee net ttt.:/ooooummmmhoo.ooo_ 3uI.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL
net fett/formm? oyRl Ygg.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL https: //www. cs toronto. edu adu/~krizz ai far. html.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics guarterly, 2(1-2):83-97, 1955.
11

---

Published as a conference paper at ICLR 2024
Yann LeCun, Corinna Cortes, Chris Burges, et al. Mnist handwritten digit database, 1998. URL hattp://yae lecun. com/ /eeeeeddbmmiisi
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only converges to minimizers. In Conference on learning theory, pp. 1246-1257. PMLR, 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train mergee Embarrassingly parallel training of expert language models, 2023. URL https:ttp//oo. http:///ppeeeevveeeeee met/forume d= 64641166555
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. Advances in neural information processing systems, 30, 2017.
Xingyu Liu, Alexander Leonardi, Lu Yu, Christopher Gilmer-Hill Hill, Matthew L Leavitt, and Jonathan Frankle. Knowledge distillation for efficient sequences of training runs. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022, 2022. URL https:
//openreview.o net forum??mmm? iid 100
Ilya Loshchiloy and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In International Conference on Learning Representations, 2017. URL https://openreviee
met//ozze Eae/mo/ommmmm 2id Skq89Scxx.
Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mecha- nistic mode connectivity. 2023.
James R Lucas. Juhan Bac, Michael R Zhang, Stanislav Fort, Richard Zemel, and Roger B Grosse. On monotonic linear interpolation of neural network parameters. In International Conference on Machine Learning, pp. 7168-7179, PMLR, 2021.
Michael S Matena and Colin A Raffel. Merging models with fisger- averaging, Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Seyed Iman Mirzadeh, Mchrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In International Conference on Learning Representations, 2021. URL https: http::::///peeeeieeeeee net. www.mmrroo...mmmmmmo.mo......mmm dd==F EQYUejf.
Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Shibani Santurkar, Dimitris Tsipras, Mahalaxmi Elango, David Bau, Antonio Torralba, and Alek- sander Madry. Editing a classifier by rewriting its prediction rules. Advances Neural Information Processing Systems, 14:23359-23311 2021.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symme- tries and invariances. International Conference on Machine Learning, pp. 2021.
PMLR,
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
12

---

Published as a conference paper at ICLR 2024
Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry Pyrkin, Sergei Popov, and Artem Babenko. Editable neural networks. In International Conference on Learning Representations, 2020. URL https: // //openrevieww.o wereet/ooummiiie Aet.uummmaaHUCCeeattta
Cecilia Summers and Michael Dinneen. Nondeterminism and instability in neural network opti- mization. In International Conference on Machine Learning, pp. 9913-9922. PMLR, 2021.
Tiffany J Vlaar and Jonathan Frankle. What can linear interpolation of neural network loss landscapes tell us? International Conference on Machine Learning, pp. 22325-22341. PMLR, 2022.
C. Wah, S. Branson, Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. (CNS-TR-2011-00)) 1-001), 2011.
Xiang Wang, Annie N. Wang, Mo Zhou, and Rong Ge. Plateau in monotonic linear interpola- tion a "biased" view of loss landscape for deep networks. In The Eleventh International Conference on Learning Representations, 2023. URL https: https::/openreeieet net/ forum?
(dd=2289SIQQ00
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In International Conference on Machine Learning, pp. 11217- 11227. PMLR, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari
Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23998, PMLR, 2022.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Global optimality conditions for deep neural networks. In International Conference on Learning Representations, 2018. URL https: /ttpss://penreviees net wowoouummidn3ee1000
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021. URL https: //penr/iiee......... net/ forum? id=mSAKLYYLse.
13

---

Published as a conference paper at ICLR 2024
Appendices
A VALIDATING ASSUMPTION (P) AT INITIALIZATION
In this section, we theoretically validate Assumption (P) in Section 3.3 for 2-layered ReLU neural network at initialization. Let d be the dimension of inputs = (zz))=1,... , E Rd, and (x) == Σ 2, ( wyzzz) be a ReLU neural network' with N hidden neurons, where w = (wiy) E RNxd (e:) E (o) == max
We assume that the input x R&d and its label y E R sampled from some bounded dis- tribution D whose density function pp(z,y) is continuous over (I, y) Re+1 The parame- ters w and v are initialized with Kaiming uniform initialization, i.c., w U(- Vd' val )xd
U(-VV- VN VN ) We employ the MSE loss ((), := 1y y21² and focus on the expected gradient ((z,g)=DVV (w,n) (x), (x), for this loss function at the initialization w, Here V of (x), y) lives in IR xx Hd which is obtained as concatenation of two vectors () w)/ommg i (1,... N).j E {1 .... ,d]) NRd and (0C(J m, (x)) y)/Ou; i {1...., N)) E R.N. Moreover, introduce the following assumption on the data distribution 'D for technical reason:
BL yo Σ miraa) 2 K for some K with high probability w.r.t Wijs
(8)
which is reasonable because it requires non-zero correlation between the input and its label y.
To validate our assumption, we introduce normalized distance
|11--22laa lalloll between two vectors V1
and U2 which can be considered as cosine distance when || || ||u||2 holds. Now our claim is that the normalized distance of gradients independent random initializations can be arbitrarily small by appropriate neuron-permutian permutation and sufficient over- parameterization. In this sense, the following theorem validates Assumption (P) at random initializations:
Theorem A.1 Under the above assumption, given two pairs of randomly initialized parameters (w, v) and (w', t²), with high probability, there exists permutation symmetry E Sy such that the normalized distance between the expected gradients Ey ww, and E(z.g) [Vu"rce y C), where (w", v²") ) is the permuted parameter of (w', v²) with n, can be arbitrarily small when is sufficiently large.
Proof:
The expected gradient at (w, v) can be computed follows:
OC (f(fw.v(z))))
E (fr. (x) - 31) Ofuc (2)
B(Z.g)D
Owci
Owis
= E [(f - -y)20 (1, m)
TD [LL(F,r(x))))))
(f -1) (s
((2,a)~D
E
Ov;
dr:
E
(f - v)oa
Σ Way
E
(fm,v(x) -1)
Σ WijCi
112, 101
d
Wij E [xf - y) 2 (2,, mlo
'We can ignore the non diffeeeffeniible locus of network because we only consider expected gradients over a distribution with a continuous density function, where such low-dimensional locus has zero measure.
14

---

Published as a conference paper at ICLR 2024
vN vN), with high probability, we assume fu,,(2) 2 0 for sufficiently large N. Thus by letting C* (w w = E 1 (E mpmm we can simplify the expected gradients as follows:
By Hoeffding's inequality and 02 U(- >
@f(fu,
(.C. ,rr,
Jwiy
(
Σ WiyCC : wid),
Ous
which are also valid for the counterpart (w', e²) instead of (w. v). Next, take a permutation vectors (ydwa,,.. Vdwid, VNv:) and (Vdwl,,"" Vdwca vNv!))
SN by applying Lemma A.2 to (d + 1)-dmeensional random
(([-1,1]6+1) where
i = L,...,NN Let w" == (w (ay))s and o" == (C())) Then we want to evaluate the squared normalized distance between E[V (1) C] and E V ow"","") C]:
V ww C] - E V (w" , C
(9)
E V (w,o) 111, E V (wo .0n) C111
We can evaluate each factor | (w.w)
E(V (w wy) 4 2
|E[V (w,))
||
(w vor) 41112 in
Eq (9) as follows:
Claim (1): V (m, C]
V (wr" an C 2 = O(Ne3) + o(N) with high probability.
Claim (2): E (wr,)) 211 = S?(N), E (wo .ny c112 =
=Q(N) =1 with high probability.
Proof of Claim (1): Let 7 E E (1, (I,... ,N. |wig w - vde, l0s o/ S VNe). We have
(wr,) G]
(w) an 2/
Σ
2
Σ wiy -w,
+
N
[1 ( + +
-
Σ wiy C w
+
(1/- - -i' (C'(w |u!| ((() ((
Σ wiy - (((
NEI
+0(119)
(10)
where 1 stands for the complement (1, .N3 By Lemma A.2, the size of the complement |/ is upper bounded by O(VN). Also we note that C2 is bounded by some constant since D is bounded, and [1yl. lo'] & 1/VN, |ayl. |og] < 1/vd. Combining these facts and Lemma A.3 for the second term of (10), it follows that (10) O(N + O(VN)..
15

---

Published as a conference paper at ICLR 2024
Proof of Claim (2):
N
l|
E
2
((a.a)) D yo
> wbz2,,)
KN.
Here used (z) R 0 as explained above and the assumption Eq (8) on distribution D.
Finally, by combining Claim (I) and (2), we obtain that the normalized distance (Eq 9) converges to Cie² when N + co, with some constant C₁ > O, which can be arbitrarily small by controlling e. O
Lemma A.2 Lete > O be any small real number. Let 21.... . 21: i..dd uniform random variables where K is a fixed constant. When is sufficiently large, with high probability, there exists a permulation π E S such that the number of indices E (1,-.,,0 ,N satisfying |z45 (2(0s) for some is upper bounded by OVVN).
IN U(I-K, KT*) be
Proof: We can assume K without loss of generality by re-scaling 2, 2 and E. Here we follow the argument given in Entezari et al. (2021). For simplicity, we take M E N satisfying M E M For each 1 (4,-... 4) E L f1, 2M}* 2t we consider
Qi = (-1+ + 4 1 -1
4
x (-1+ 4 -
4 M
1+
M
M
M
f1 =
E Qt).
Note that any 2; and 2 are contained in Q1 for some E L with probability 1. For each 1 L, the number ni and n/ can be considered as the sum of random binary variables by +... by where each by sampled from Bernoulli distribution Ber( 1/4. ). By Hoeffding's inequality for the Bernoulli random variables, we have
t2
2
2exp
M
2N
for each E L. Thus, with probability 1 5, we obtain
N
N M
4
n4
2N log
M
To construct the desired correspondence x between (2: S i < N) and (2) : 1 S j s
N), each z; E Qi should be mapped to some z/J E Q1. The number of ii E f1. ... N}
zt does not have its counterpart zý}} can be upper bounded by
Σ LEL
m
11
+
r4
N M
S 2
2N log
4 8
= O(VN).
M
LEL
Lemma A.3 Assume that
(w1, ... w'') E [- 1 1
Va val satisfy sup wwy - w/l S
follows that Σ, W. (W wa) E, ( ,w/) I O(e).
Proof: Let G(w,, ... wa) = E, w (w) ... w.) = Bi-yoo()) wyzz)). By applying trian- gle incquality iteratively, we have
G(wn, ... wa) G(w(,''.. w(]] "(w), wa) G(w/, w2, wa)l + [G(w], "....,, ,.) ",,"".,,a))
G(w(, ... ,.,)))
G(wr,...
wa) ,wa) G(w's W2, wa)l + |G(W), w/s wa) C(w/,..-,,dy
16

---

Published as a conference paper at ICLR 2024
Thus the proof can be reduced to the case where [wkk- wk/ol S for some ky and wy w/J for k # Ko- We can assume ky 1 and W1 I w I wi + without loss of generality. Then we have
G( ... wa) - G( ... ,'')
L yo Σ wzzz) ]] - | yo Σ
w
TR. o E mjzts) 0 Σ w(sar))
/ lul {{ (wn
Σ
w
uff21 Ewjz;) }
p(x, v)dady
j=2
d Σ
/ 1ll /ueaa
2/22 m5 p(xx, v)dady
j=2
d
+ / |ll /uiar
+
Σ
WyDiz
"pp 20 p(a, v)dady
j=2 d
+ f 1ul 10l1(1110
Σ
(wfmm Σ Euy j=2
p(x, y)drdy
(11)
On the first term of the last inequality (11), since w4 + Σj= Jadj < holds on the integrated region, it follows that < wix1 + " Wjay < 1 wf 2₁ Thus we have
/ |yl w + Σ w [wn waz 202,20 + waa p(z, ydddy
/ |ol | - w/ mil p(x, ydrrd S / |vl e||rl p(x, ddddy
S
The same argument holds for the second term of inequality (11). Furthermore, the third term is also bounded by J lyl e|ail- p(æ, drdd.. Therefore, by the boundedness of the distribution D, we have (11) < 3a f Iyzz pl y)dxdy O
B PROOF OF LEMMA 3.2
In this section, we employ the same notation as Section 3.3. We assume that
1. |10 - -;) (-as Vo; Vo; D)22 <E
2. ||,, Vo L - Ves. L1l < c. for 1-
3. The gradient V,L is K-Lipschitz continuous with respect to the parameter 0, i.e., V Vo K 0'
Lemma B.1 Under the above assumptions, we have
0
2,2
2nH 0
(12)
for 1005SS<58 ST.
17

---

Published as a conference paper at ICLR 2024
Proof: We prove by induction:
8 2,, = 02 + n. (0) 89)
02 + TT
Σ
av Vor C O(te)
(by Assumption 1.)
t==0
M iMIM MM
80 +
TT vo O(tz)
L + 0(s) + O(te)
62 +
Ou V 05 1-
(by Assumption 2.)
02 +
Qy V 05
- ((E)) + O(ts)
(by induction hypothesis.)
02 +
V 05 0/ c)
+ + K*
(by Assumption 3.)
1-1
82 + n Σ or Vo
+O(T" E)
= 02 + n. (0)- 09) + +0 k*
= n, O(T k*'s).
C
RELATED WORK
Loss landscape, linear mode connectivity, permutation symmetry. Loss landscape of training deep neural network has been actively studied in an effort to unravel mysteries of non-convex optimization in deep learning (Hochreiter Schmidhuber, 1997; Choromanska et al., 2015; Lee et al.. 2016; Keskar et al., 2017; Li et al., 2). One of the mysteries in deep learning is the stability and consistency of their training processes and solutions, despite of the multiple sources of randomness such as random initialization, data ordering and data augmentation (Fort et al., 2019; Bhojanapalli et al., 2021; ;; Summers & Dinneen, 2021; Jordan, 2023). Previous studies of mode connectivity both theoretically (Freeman & Bruna, 2017; Simsck et al., 2021) and empirically (Draxler et al., 2018; Garipov et al., 2018) demonstrate the existence of low-loss curves between any two optimal solutions trained independently with different randomness.
Linear mode connectivity (LMC) is a special case of mode connectivity where two optimal solutions connected by low-loss linear path (Nagarajan & Kolter, 2019; Frankle et al., 2020; Mirzadeh et al., 2021; Entezari et al., 2021; Benzing et al., 2022; Ainsworth et al., 2023; Juneja et al., 2023; Lubana et al., 2023). In this line of research, Entezari et al. (2021) observed that even two solutions trained from different random initialization can be linearly connected by an appropriate permutation symmetry. Ainsworth al. (2023) developed efficient method find such permutations, and Jordan et al. (2023) extends it to NN architectures with Batch normalization (Ioffe & Szegedy, 2015). Their observations strength the expectation on some sort of similarity between two training processes even from different random initializations, via permutation symmetry. In our work, based on these observations, we attempt to transfer one training process to another initial parameter by permutation symmetry.
Another line of research related to our work is the studies of monotonic linear interpolation (MLI) between initialization and its trained result. Goodfellow et al. (2015) first observed that the losses are monotonically decreasing along the linear path between an initial parameter and the trained one. Frankle (2020) and Lucas et al. (2021) confirmed that the losses monotonically non-increasing even with modern network architectures such as CNNs and ResNets (He et al., 2016). Vlaar & Frankle (2022) empirically analyzed which factor in NN training influences the shape of the non-increasing
18

---

Published as a conference paper at ICLR 2024
loss curve along the linear interpolation, and Wang et al. (2023) theoretically analyzed the plateau phenomenon in the early phase of the linear interpolation. Motivated by these observations, we introduced the notion of linear trajectories in Section 3.4 reduce storage costs in our learning transfer.
Model editing. Our approach of transferring learning trajectories can be also considered as a kind of model editing (Sinitsin et al., 2020; Santurkar et al., 2021; Ilharco et al., 2022; 2023) in the parameter space because we modify a given initial parameter by adding an appropriately permuted trajectory. In particular, recent work by Ilharco et al. (2023) is closely related to work. They proposed to arithmetically edit a pre-trained NN with a task vector, which is defined by subtracting the initial pre-trained parameter from the parameter fine-tuned on specific task. From our viewpoint, task vectors can be seen as one-step learning trajectories (i.e., learning trajectories with T = 1). Model merging (or model fusion) (Singh & Jaggi, 2020; Matena & Raffel, 2022; Wortsman et al., 2022; Li et al., 2023) is also related in the sense of the calculation in the parameter space.
Efficient training for multiple NNs. There are several literatures that attempt to reduce the computation costs in training multiple NNs. Fast ensemble is approach to reduce the cost in ensemble training by cyclically scheduled learning rate (Huang et al., 2017) or by searching different optimal basins in loss landscape (Garipov al., 2018: Fort et al., 2019; Wortsman et al., 2021; Benton et al., 2021). A recent work by Liu et al. (2022) leverages knowledge distillation (Hinton et al., 2015) from one training to accelerate the subsequent trainings. Our approach differs from theirs in that we try to establish general principle to transfer learning trajectories. Also, the warm- sorm--tarting technique investigated by Ash & Adams (2020) seems to be related in that they subsequently train from a once trained network. There may be some connection between their and our approaches, which remains for future work.
Gradient matching. The gradient information obtained during training has been utilized in other areas outside of ours. For example, in dataset distillation, Zhao et al. (2021) optimized a distilled dataset by minimizing layer-wise cosine similarities between gradients on the distilled dataset and the real one, starting from random initial parameters, which leads to similar training results on those datasets. Similarly, Yin et al. (2021) successfully recovered private training data from its gradient by minimizing the distance between gradients. In contrast to their problem where input data is optimized, our problem requires optimizing unknown transformation for NN parameters. In addition, our problem requires matching the entire learning trajectories, which are too computationally expensive to be computed naively.
D DETAILS FOR OUR EXPERIMENTS
D.1 DATASETS
The datasets used in our experiments (Section 4) are listed below. For all datasets, we split the officially given training dataset into 9:1 for training and validation.
MNIST. MNIST (LeCun et al., 1998) is dataset of 28 x 28 images of hand-writter digits, which is available under the terms of the CC BY-SA 3.0 license.
CIFAR-10, CIFARI00. CIFAR-10 10 and CIFAR-100 (Krizhevsky, 2009) are datasets of 32 x 32 images with 10 and 100 classes respectively.
ImageNet. ImageNet (Deng et al., 2009) is a large-scale dataset of images with 1000 classes, which provided for non-commercial research or educational use.
Stanford Cars. Stanford Cars (Krause et al., 2013) is dataset of images with 196 classes of cars, which is provided for research purposes. We refer to this dataset as Cars for short.
CUB-200-2011. CUB-200-2011 (Wah et al., 2011) is a dataset of images of 200 species of birds. We refer this dataset as CUB for shot.
D.2 NETWORK ARCHITECTURES
The neural network architectures used in our experiments (Section 4) are listed as follows:
19

---

Published as a conference paper at ICLR 2024
2-MLP. 2-MLP is two-layered neural network with the ReLU activations. The design of this architecture shown in Table 1.
Conv8. Conv8 is 8-layered CNN followed by three linear and ReLU layers. The design of this architecture is shown in Table 2.
ResNet-18. The ResNet family (He et al., 2016) is a series of deep CNNs with skip connections. We employed the standard 18-layered one for ResNet-11.
Table I: The architecture of 2-MLP.
No.
Layers
Output dimensions
1
Flattening
784 (= 28 28)
Linear ReLU
4096
4
Linear
10
Softmax
Table The architecture of Conv8.
No.
Layers
Conv(input=3, output=64, kerne_ size= 3). stride=1, padding=1) - ReLU Conv(input=64,, output=64, kernel_sizee=(3 3), stride=1, padding=1) ReLU
2
3
MixPPolliggkemmesssiee=-2222222 2))
Conv(input=64, output=118, kerne_ kermel_size=(3,3) stride=1, padding-) + ReLU
5
Conv(input=128,, output=128,, 2 kemel_ kemel__szee=3..3) stride=1, padding=1) - ReLU
MaxPPooliggKerreeLssizez 2))
7
manvfinput=128,oupput-- kemel (imeelsizee=3.33) stride=1, padding=1) ReLU
Conviinpt==256, outputt:66 (emmelsizze(3,3) stride=1, padding=1) ReLU
9
MaxtPoolingkeemmeLsiiee---2 2))
Conv(inuut=256,, output=512, comnelsiize=(3 stride=1, padding=1) ReLU
11
manviinput=5122 kemel kemel_size==3.3 3). stride=1, padding=1) ReLU
12
MaxPPoling(kerrel_izze size=(4, 4))
13
Conv(input=512, output=512, kemne_ (iaeee__iiee333) 3), stride=1, padding=1) ReLU
14
Linear(input=512, output=256) - ReLU
15
Linear(input2256, output=256) - ReLU
16
Linear(input=512, output=10)
Softmax 17
D.3 TRAINING DETAILS
D.3.1 DETAILS ON IMPLEMENTATION AND DEVICES
We implemented the codebase for all experiments in Python 3 with the PyTorch library (Paszke et 2019). Our computing environment is machine with 12 Intel CPUs, 140 GB CPU memory and a single A100 GPU.
al.,
D.3.2 TRAINING OF SOURCE TRAJECTORIES
In training for source trajectories, we used SGD with momentum in PyTorch for the optimization. It has the following hyperparameters: the total epoch number E, batch size B, learning rate a, weight decay X, momentum coefficient μ. We used the cosine annealing (Loshchilov & Hutter, 2017) for scheduling the learning rate except for MNIST, For the random initialization, we used the standard Kaiming initialization (He et al., 2015), which is also default in Pyror.. The details the hyperparameters are as follows:
2-MLPP on MNIST. We used 15, B 128, a 0.01, X 0.0, 0.9.
Cony8 CIFAR-10. We used F. 60, B 128, 0.05, À 0-0001, 0.9.
20

---

Published as a conference paper at ICLR 2024
Conv8 CIFAR-100. We used E 30, B = 128, 0.05, À 0.0001, / 0.9. starting from the pre-trained parameter on CIFAR-10.
ResNet18 on ImageNet. We used = 100, = 128, a - 0.1, X 0.0001, 14 0.9. For the first 5 epochs, we gradually increased the learning rate as y = 0.1 x (i/5) for each i-th epoch = 1,... 5). For the 95 epochs, we decayed the learning rate by cosine annealing starting from 1) 0.1.
ResNet18 on Cars. We used E = 30, B = 128, = 0.1, À = 0.0001, = 0.9, starting from the pre-trained parameter on ImageNet.
ResNet18 CUB. We used E 30, B 128, 0.1, À 0.0001, /r 0.9, starting from the pre-trained parameter on ImageNet.
D.3.3 HYPERPARAMETERS FOR TRANSFERRING LEARNING TRAJECTORIES
Our methods (Algorithm 1, 2) have the following hyperparameters: the length T of learning tra- jectories, the batch size B for each gradient matching. Also, for NN architectures with the Batch normalization (loffe & Szegedy, 2015) such as ResNets, the batch size B3' for resetting the means and variances in the Batch Normalization layers (Jordan al., 2023) is also hyperparameter.
2-MLP on MNIST. We used 128 and T 5.
Conv8 CIFAR-10. We used B 128 and T 30.
Conv8 on CIFAR-100. We used B 128 2 and T 15.
ResNet18 on ImageNet. We used B = 128 x 100, B' = 128 x 20 and T = 40.
ResNet18 on Cars. We used B = 128 5, 128 x 2 and T = 15.
ResNet18 on CUB. We used B 128 5, 128 x 2 and T 15.
D.3.4 COMPUTATIONAL COST OF GMT/FGMT
Total computational cost FGMT (resp. GMT) consists of: computing 2BT (resp. 2BT*) gradients. which is the same budget as 2T iterations of standard training, and solving lightweight optimizations (eq. 6) for T times. Thus, for FGMT. the required computational cost should be significantly smaller than standard training (if implemented efficiently). We also provide the wall-clock time in our experiments in Appendix E.6, although the results are worse than the above estimate due to inefficiency of our implementation, non-negligible I/O delays and etc.
D.3.5 SUBSEQUENT TRAINING OF TRANSFERRED PARAMETERS
For the subsequent training in Section 4.2, we used the same optimizer and learning rate scheduler as training of source trajectories, with slightly small initial learning rates (a 0.01 on CIFAR- 10, = 0.05 on CIFAR-100, 0.05 Cars, = 0.01 for CUB), which are selected based on the validation accuracy of Naive baseline for fair comparison.
depends model architectures, typically or seconds for each optimization in environment
21

---

Published as a conference paper at ICLR 2024
E ADDITIONAL EXPERIMENTAL RESULTS
E.1 LEARNING TRANSFER FOR ACTUAL TRAJECTORIES
NOW Turch
10/4 Tarteer
(a) MNIST (2-MLP)
(b) CIFAR-I0 (Conv8)
(c) ImageNet (ResNet-
Faaed Specni
Sagestaa Teancty
(d) ImageNet Cars
(e) ImageNet + CUB
(f CIEAR-IO CIFAR-100
Figure 9: We plot the validation accuracies of the transferred parameter 0, for each timestep
t 1,... T with various datasets and NN architectures as in Figure 5, except that the actual trajectory 8f at t-th epoch of SGD is transferred instead of linear trajectories. Compared the results for linear trajectories in Figure 5, the transferred results for actual trajectories tend to have more variance in validation accuracy and fail to transfer in fine-tuning scenario.
E.2 LEARNING TRANSFER ON SUN397 AND INATURALISTT201
FESTT Tando
Sater Saeddn
15
(a) SUN (Transfer)
(b) iNat (Transfer)
(c) SUN (After Transfer)
(d) iNat (After Transfer)
Figure 10: Experiments on large-scale scale fine- tine-grained datasets, SUN-397 (SUN, Xiao al. (2010)) and iNaturalist-2011 (iNat, Cui et al. (2018)) with ResNet- 18, transferring fine-tuning trajectories on each dataset starting from ImageNet pre-trained initializations. Figs. (a, b) show validation accuracies during transfer, and Figs. (c, d) show the results of subsequent training after transfer, as in Section 4. Although validation accuracies during transfer are worse than standard results obtained for epoch, observed that it still accelerates the subsequent training bit. In practice, method still needs to be improved for such scage complex datasets, but the results indicate the potential effectiveness of learning transfer for such datasets.
22

---

Published as a conference paper at ICLR 2024
E.3 LEARNING TRANSFER WITH RESNET-34
TGTT Tertlr
FMMT Tande
(a) CUB (Transfer)
(b) Cars (Transfer)
(c) CUB (After Transfer)
(d) Cars (After Transfer)
Experiments with ResNet-34. Figs. (a. b) show validation accuracies during transfer, and Figs. (c, d) show the results of subsequent training after transfer, as in Section We can see that the overall trends are similar the results with ResNet-18.
Figure 11:
E.4 LEARNING TRANSFER WITH ADAM
(a) 10 (Transfer)
(b) CUB (Transfer)
(c) Cars (Transfer)
FQNT Puicce
FSMT Tuaik
FONT To sir
sa0 19019
19 1000
25
(d) CIFAR 10 (Afler Transfer)
(e) CUB (After Transfer)
(l) Cars (After Transfer)
Figure 12: Experiments of transferring learning trajectories obtained by Adam optimizer (Kingma
Ba, 2015). In Figs. (a-c), the validation accuracies during transfer are terribly worse than the case of SGD Section 4.1. This may be because optimization trajectories of Adam are more complex than SGD as known in studies of monotonie linear interpolation (Lucas et al., 2021), where MLI fails many cases with Adam. However, in Figs. (d-f), we observed that the acceleration in subsequent training after transfer (Section 4.2) still works bit even in these cases. Thus, although our proposed method should improved for Adam, the results indicate the potential effectiveness of learning transfer even for Adam optimization.
23

---

Published as a conference paper at ICLR 2024
E.5
ENSEMBLE EVALUATION
GMT
FGMT
Standard Ensemble
CIFAR-10
92.01 (@50ep)
92.11 (0ep))
92.26 (66ep)
CIFAR-100
70.41 41 (@ 50ep)
69.53 @ 50ep)
69.81 (@60ep)
Cars
86.83 20ep)
86.61
@ (0ep)
87.10 (@ 30ep)
CUB
74.71 @ 10ep)
75.62 @ 10ep)
74.02 @ (03pp)
Table We evaluate ensembles of three transferred models with fewer subsequent training epochs, compared to standard ensemble of three models trained from scratch. "@ X ep" means that each member of the ensembles are trained for epochs after transferred (in GMT/FGMT) or from scratch (in Standard Ensemble).
Matching Probubility
Calibration Error (L2) Standard Ensemble
Datasets
FGMT
GMT
Standard Ensemble
Single
FGMT
GMT
Single
CIFAR-10
0.96
0.96
0.94
1.0
0.09
0.083
0.082
0.078
CIFAR-100
0.77
0.77
0.8
1.0
0.17
0.17
0.18
0.18
Cars
0.89
0.89
0,91
1.0
0.050
0.045
0.053
0.052
CUB
0.83
0.83
0.84
1.0
0.082
0,084
0.092
0.099
Table 4: We evaluate the diversity of the members of ensemble. Matching Probability measures the probability that a prediction from each member of the ensemble matches the ensembled prediction. From this evaluation, we can see that the diversity in the ensemble by FGMT/GMT is the same level as the standard ensemble, compared to non-ensemble one (Single). We also evaluate the calibration error, but this metric does not give any insight in experiments because it cannot distinguish between ensemble results (Standard Ensemble) and non-ensemble results (Single).
E.6 WALL-CLOCK TIME FOR LEARNING TRANSFER
FGMT GC
GMT
SGD (Reference)
Models
Datasets
GM
Total
GM
GC
Total
1-epoch
Full epochs
2-MLP
MNIST
0.66
16.63
20.62
0.54
17.22
20.60
3.69
55.35
Conv8
CIFAR-10
44,02
57.73
132
57.74
572
645
7.66
459
Conv8
CIFAR-100
71.50
92.10
82.03
323
8.53
511
ResNet-18
ImageNet
253
3178
4239
480
59053
60259
5000
500030
R8
SUN397
26.52
541
779
26.21
3694
3914
403
12117
ResNet-18
NNauuralis2017
23.4
974
1321
30.23
7000
7343
1926
57787
ResNet-18
Cars
46.09
158
269
52,95
1080
1201
16,90
507
ResNet-18 18
CUB
38.33
121
217
44.87
850
949
12.94
388
ResNet-34
Cars
36,84
174
282
39.26
1669
1808
20.43
612
ResNet-34
CUB
37.38
144
243
34.75
979
1070
15.71
471
Table Wall-clock time (seconds) FGMT/GMTT benchmarked on computing node with Intel
Xeon Gold 6148 CPU and V100 GPU. Since the computing node is shared with other research groups and has non-negligible I/O delay due to distributed storage, the results are highly unreliable and should not be taken seriously. GM stands for total time of Gradient Matching (i.e., line 7 in Algorithm or line in Algorithm 2) during the algorithms, and GC stands total time of Gradient Computation (i.c., lines 4-5 in Algorithm of the whole algorithms. Note that the computational complexity for GM (TT) in both FGMT and GMT, and for GC is ((T) in FGMT and O(T2) in GMT.
lines 3-4 in Algorithm 2). Total stands for the total time
24

---

Published as a conference paper at ICLR 2024
E.7 ABLATION ANALYSIS ON THE TRAJECTORY LENGTH T
Talael Seot
Tat Bachh
50
(a) T 5
T
T
(d) T 30
Figure 13: Ablation analysis of 7 on CIFAR- 10 with Conv8.
a T 5
(b) T 10
(c) T 15
(d) 7' 20
Figure 14: Ablation analysis of T on CUB with ResNet18.
E.8 ADDITIONAL EXPERIMENTS ON ASSUMPTION (P)
Partenio or CiM025
de OUR
(a) ResNet34 with random initial-
(b) Conv8 with random/pre-trained initializations.
(c) ResNet18 with random/pre- trained initializations.
izations.
Figure As an addendum to Figure 3, we provide additional experiments on Assumption (P). Figure (a) shows that Assumption (P) holds weakly for ResNet34 compared to other network architectures. Figure (b, c) show how Assumption (P) behaves for random/pre-traite
initializations. The label X + Y means that the initialization is random pre-trained on X, and then trained on Y. In pre-trained initialization case, we can see that Assumption (P) only holds in the very early phase of training.
25

---

Published as a conference paper at ICLR 2024
E.9
EMPIRICAL VALIDATION FOR LEMMA 3.2
1.
0.1
8.4
POIT
Figure 16: Here we empirically validate Lemma 3.2 on CIFAR-10 10 with Conv8, by observing the cosine similarity between 2², and 2 during GMT FGMT. The cosine similarity tells us how the (s + 1)-th permutation T is consistent with the previous permutation TT, on the previous source parameter or trajectory (O..... ,0)) The results show that Lemma 3.2 actually holds in the early phase of transfer, and also holds in the later phase FGMT, due its stability as discussed in Section 3.4.
E.10 2 LOSS IN GRADIENT MATCHING
Figure 17: Comparison of L losses in FGMT between the linear/actual trajectory CIFAR-10 with Conv8. The loss is computed by equation (6) for each timestep t. The loss for the actual trajectory is accumulated constantly from the beginning, while the loss for the linear one is suppressed in the early phase. The result indicates that, due to its stochasticity, each gradient direction along the actual trajectory is too different from each other to be aligned.
26


Reviews:
[{"review_id": "qwvAAnE3BH", "reviewer": "Reviewer_gCAh", "summary": "This paper proposes a novel algorithm for transferring a learning trajectory from one initial parameter to another, which can significantly reduce the computational cost of training deep neural networks. The algorithm formulates the learning transfer problem as a non-linear optimization problem for the policy function and matches gradients successively along the trajectory via permutation symmetry to approximately solve it. The empirical results show that the transferred parameters achieve non-trivial accuracy before any direct training and can be trained significantly faster than training from scratch. However, the algorithm's limitations include the assumption that the source and target tasks are related, and the lack of a detailed analysis of the computational cost of the algorithm.", "questions": "1. How sensitive is the algorithm's performance to the assumption that the source and target tasks are related, and how well does it perform when the tasks are unrelated?\n2. How does the proposed algorithm compare to other methods for transferring learning trajectories, such as fine-tuning or transfer learning?\n3. How well does the algorithm perform on datasets that are not included in the empirical evaluation, and how does its performance compare to other methods on these datasets?\n4. Can the algorithm be extended to handle more complex neural network architectures?\n5. How does the computational cost of the algorithm compare to other methods for transferring learning trajectories, and how does it scale with the size of the neural network and the number of training epochs?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed algorithm is a novel approach to the problem of transferring a learning trajectory from one initial parameter to another. The idea is interesting.\n2. The algorithm is theoretically grounded and can be solved efficiently with only several tens of gradient computations and lightweight linear optimization.\n3. The empirical results show that the transferred parameters achieve non-trivial accuracy before any direct training and can be trained significantly faster than training from scratch.", "weaknesses": "1. The empirical evaluation of the algorithm is conducted on a limited set of benchmark datasets, and it is unclear how well the algorithm would perform on other types of datasets or in real-world scenarios.\n2. The paper assumes that the source and target tasks are related, and it is unclear how well the algorithm would perform when the tasks are not such related.\n3. The paper does not provide a detailed analysis of the computational cost of the algorithm, which may be a concern for large-scale neural networks.\n4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XHSJysb1UN", "reviewer": "Reviewer_rULw", "summary": "### Problem Statement\n\nThe paper introduces a novel problem called the \"learning transfer problem\", aimed at reducing training costs for seemingly duplicated training runs on the same dataset by transferring a learning trajectory from one initial parameter to another without actual training.\n\n### Main Contribution\n\nThe paper's contributions include: (1) formulating the new problem of learning transfer with theoretical backing, (2) deriving the first algorithm to solve it, (3) empirically demonstrating that transferred parameters can accelerate convergence in subsequent training, and (4) investigating the benefits and inheritances from target initializations. Through these contributions, the paper presents a promising avenue to significantly reduce the computational cost and time required to train DNNs, especially in scenarios involving model ensembles or fine-tuning pre-trained models.\n\n### Methodology\n\nThe authors approximate the solution to the learning transfer problem by matching gradients successively along the trajectory via a permutation symmetry technique. The updates along the \"source trajectory\" are applied to update a target network with a different random initialization after being permutated such that the gradients of the two networks at the same \"time step\" are best matched under the permutation, resulting in the Gradient Matching along Trajectory (GMT) algorithm. To further optimize the space and time complexity of the algorithm, the authors propose to use linear interpolation of the initial and final parameters of the source network in place of the acutal training trajectory, and to re-use the mini-batch gradients evaluated along the trajectories to search for the best-matching permutations. The optimized verison of the algorithm is named \"Fast Gradient Matching along Trajectory\" (FGMT). The best permutations for parameter alignment at each time step are solved with a coordinate descent algorithm, iteratively optimizing for the permutation in each layer of the network by solving a linear assignment problem.\n\n### Experiments\n\nThe learning transfer methods are evaluated on standard vision datasets with various architectures including CNN and MLP. Both random initializations and pre-trained initializations are used to evaluate and demonstrate the effect of GMT and FGMT in terms of 1) the performance after transfer; 2) the fine-tuning efficiency and performance after transfer.\n\nEmpirical evaluations reveal that the transferred parameters achieve non-trivial accuracy before any direct training and can be trained significantly faster than training from scratch, while inheriting the properties (e.g. generalization ability) from the parameter initialization.", "questions": "1. Data augmentation is widely used for the training of DNNs, which essentially (often randomly) modifies the dataset. Does this violate the assumption that the dataset is the same in the problem statement?\n\n2. Even gradient-based, many optimization methods for DL can have updates very different from SGD (for example, Adam). Considering that the parameter alignment is done with gradient matching in GMT and FGMT, does the choice of optimization method changes the effect of GMT / FGMT? Ablation study regarding this could be worthwhile to add.\n\n3. Why does the performance deteriorate in Figure 5d-5f? It is simply acknowledged without analysis and discussion.\n\n4. In Figure 7, to better understand the effect of FT and FGMT, where would the FGMT-only parameters be in the landscape? What about permuted-source + FT?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Originality and significance\n\nThe proposed task of learning transfer problem is novel and very interesting, with potentially wide applications, as the foundation-model paradigm prevails in many AI / DL fields. The proposed method is to progressively merge the target network with the source network using Git Re-basin, which is straightforward and efficient.\n\n### Quality\n\nTheoretical analysis is performed to justify the adopted method, in addition to a series insightful experiments. The experimental details are in general well documented. However, I find the experiments are not enough to support some of the claims, and will expand on this in the Weakness section.\n\n### Writing\n\nThe writing is overall good, despite minor grammar problems. I find the mathematics in the paper is clear with consistent and self-explanatory notations. The problem is well motivated and formulated, and the main thesis is well conveyed.", "weaknesses": "I am open to change my score if the authors can address the following concerns:\n\n### Lack of experiments more closely demonstrating the actual usage of the proposed method\n\n1. One potential usage of the proposed method suggested by the authors is to transfer the update of a foundation model to its fine-tuned versions. However, all experiments are limited to network architectures of relatively smaller scale, and to the cases where fine-tuning task shares exactly the same number of classes as the pre-training task, which differs from the realistic use-case of foundation models, which are of typically larger scale, and are used for various down-stream tasks with task-specific heads.\n\n2. The authors claim that method can accelerate the training of an ensemble of neural networks. Although the computational cost of the proposed method is briefly described in the appendix, there is no figure or table systematically comparing the cost with traditional training / fine-tuning approaches. More crucially, no experiment compares the performance of the ensemble obtained with GMT / FGMT and traditional approaches. One concern is that, transfering one (or a limited number of) source trajectory, the diversity of the resulted target networks is limited (this is partially endorsed by the landscape visualization in Figure 7), which could hurt the performance of the ensemble.\n\n### Lack of experiments verifying the Assumption (P), Theorem 3.1, and Lemma 3.2\n\nAlthough the theoretical analysis abounds, it would be much more convincing to show these statements hold in real experiments.\n\n### Lack of experiments and discussion on the choice of $T$\n\nIt is not clear how $T$, the numebr of time steps, or rather, the number of samples taken from the source trajectory, influences the performance of the transfer result. It seems that $T$ does not really matter in the Naïve baseline and Oracle baseline, where the parameter-aligning permutation $\\pi$ remains the same across time steps, which 1) would be good to be verified by the authors in the main text and 2) makes it interesting to explore the value of $T$ that GMT / FGMT requires to have good performance, because the computational cost is proportional to $T$.\n\n### The \"Generalization ability\" part in section 4.3 is not clear\n\nIn the \"Generalization ability\" part in section 4.3, the experiments and figures should be further clarified. From Figure 7, I assume that the task is to fit the CUB or Cars dataset (for the previously chosen 10-class subset), and the two \"Standard\" curves are for the target trajectories, which are transferred through FGMT to initialization pretrained with ImageNet-10%, leading to the green curves, but the explanations are really not clear. However, the main text explicitly states that the target initialization $\\theta_2^0$ is \"pre-trained on full ImageNet\", which implies only *1* (instead of *2*) possible combination for FGMT for each fine-tuning dataset: starting from ImageNet-10%-pretrained, transferring the finetuning trajectory which starts from the ImageNet-Full-pretrained initialization). Another example is, I am not sure what validation the authors refer to when they mention that the ImageNet-Full pretrained initialization has \"validation accuracy $\\approx$ 72%\" and the ImageNet-10% one has \"validation accuracy $\\approx$ 50%\".\n\n\n### Minor\n- In the second last line of the second paragraph in Introduction: strength -> strengthen\n- On Page 5, right below Lemma 3.2: please make it explicit in which Appendix the proof is\n- On Page 6, in the \"Linear trajectory\" section, \"such a linearly interpolated trajectory *satisfies* ...\"\n- On Page 9, in the \"Generalization ability\" section, \"generalizes poorly *than* ...\" -> \"generalizes poorly *compared to* ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGSxEKTmez", "reviewer": "Reviewer_EPpK", "summary": "The computational cost of training the neural network is high. \nTo reduce these computational costs, this paper also intends to utilize a well-trained neural network. \nContrast to previous studies, this paper proposes to use the training trajectory of neural networks for model training because it contains a lot of information. \nThe authors dubbed this problem \"learning transfer problem,\" which is the transfer of trajectories from one initial parameter to another. \n\nTo this end, the paper proposes an algorithm that matches the gradient continuously along the trajectory through permutation symmetry. \nThe authors demonstrate efficiency of the algorithm in two ways to evaluate the validity of the proposed algorithm. \n\nThe first is the initialization scenarios that consists of random or pre-trained initialization to assess whether the transferred parameters via “learning transfer” without finetuning can enhance the accuracy. \nThe other is the fine-tuning scenario to validate whether the transferred parameters can improve the learning speed.\n\nWith two scenarios, this paper empirically demonstrates the proposed algorithm can train the model quickly and efficiently.", "questions": "1. Does it show the good performance to make the model to have the same trajectory in tasks that are not related to each other?\n2. Are these transferred models' ensembles better than scratch ensembles? What is it like from an ECE perspective?\n3. In Sec. 3.4, there are an explanation why the transfer of the linear trajectory is more stable and has less variance than the transfer of the actual one. The authors explain that it may be because the actual trajectory contains noisy information. I think the theoretical or empirical evidence is necessary to support the explanation. \n4. I think we need to do the work shown in Fig. 5 to select the optimal parameter. Then shouldn't we put this process into the pseudo-code?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The task, “learning transfer problem” the authors proposed is novel to me.\n\nTo address this problem, the authors proposed an algorithm to match the trajectories between source and target, which is seemly convincing. To evaluate the validity of the proposed algorithm, the authors, without any training, conducted an experiment that transfers the calculated parameter to match the trajectory, which performed somewhat successfully. \n\nIn addition, as a result of fine-tuning after transferring the parameters, it is revealed that the performance increased very quickly.", "weaknesses": "1. There is a lack of motivation albeit the promising results. It is a lack of the evidence whether having the same trajectory between tasks is always good. \n2. To experimentally prove that an initialization or architecture affects the similarity more than the dataset, it is necessary to verify it on more datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)