The Negative Pretraining Effect in Sequential Deep Learning and Three Ways to Fix It
Authors: Julian G. Zilly
Franziska Eckert
Bhairav Mehta
Andrea Censi
Emilio Frazzoli
Venue:  (2021)
Keywords: Transfer learning
Deep learning
Sequential learning
Critical learning periods
Curriculum learning

Abstract: Negative pretraining is a prominent sequential learning effect of neural networks where a pretrained model obtains a worse generalization performance than a model that is trained from scratch when either are trained on a target task. We conceptualize the ingredients of this problem setting and examine the negative pretraining effect experimentally by providing three interventions to remove and fix it. First, acting on the learning process, altering the learning rate after pretraining can yield even better results than training directly on the target task. Second, on the learning task-level, we intervene by increasing the discretization of data distribution changes from start to target task instead of “jumping” to a target task. Finally at the model-level, resetting network biases to larger values likewise removes negative pretraining effects, albeit to a smaller degree. With these intervention experiments, we aim to provide new evidence to help understand the subtle influences that neural network training and pretraining can have on final generalization performance on a target task in the context of negative pretraining. 

Under review as a conference paper at ICLR 2021
THE NEGATIVE PRETRAINING EFFECT IN SEQUENTIAL DEEP LEARNING AND THREE WAYS TO FIX IT
Anonymous authors Paper under double-blind review
ABSTRACT
Negative pretraining is prominent sequential learning effect of neural networks where a pretrained model obtains worse generalization performance than a model that is trained from scratch when either are trained on target task. We conceptualize the ingredients of this problem setting and examine the negative pretraining effect experimentally by providing three interventions to remove and fix it. First, acting on the learning process, altering the learning rate after pretraining can yield even better results than training directly the target task. Second, the learning task-level, we intervene by increasing the discretization data distribution changes from start to target task instead of "jumping" to a target task. Finally at the model-level, resetting network biases to larger values likewise removes negative pretraining effects, albeit to smaller degree. With these intervention experiments, we aim to provide new evidence to help understand the subtle influences that neural network training and pretraining can have on final generalization performance on a target task in the context of negative pretraining.
INTRODUCTION
Lifelong learning (Thrun & Mitchell 1995 Silver et al. 2013) holds both the promise of benefitting from past experiences and the challenge of having continually adapt new problem settings. The characteristics of this intriguing sequential learning problem make it a balancing act between retaining knowledge and adapting to new experiences referred as the stability- plaitiit-llatiiitty dilenma (Mermillod
er al. 2013). Similar to coach or teacher, we want to understand when sequence of tasks helps or hinders further learning and what the underlying mechanism are that influence this process given that
the cost of training very large models from scratch such as GPT-3 (Brown et al. 2020) has increased considerably (Sharir et al. 2020). This setting of training a model on sequence tasks to maximize its performance on a target task we coin the sequential learning problem.
The existing literature on sequential learning sheds some light on the characteristies of this problem setting. In some cases, following sequence of learning tasks can lead to better results than simply training on the target task from scratch (Bengio et al. 2009). Achille and Soatto (Achille et al.. 2017) and Mndn et al. (2019) contrast this picture by highlighting that neural networks suffer from pretraining on some tasks. When learning data is corrupted for sufficiently long period, even switching back to clean data does allow the model to recover its original performance (Maennel et al. 2020). Until compared, it may unforuntely be unclear that model is corrupted.
Contribution: We formalize the basic ingredients of sequential learning as following path on learning manifold as depicted in Fig. and provide three ways to remove the negative pretraining effect that can occur for certain task changes in neural network training as depicted in Fig. 2 We establish experimentally for variety of task changes and datasets that this final performance can be highly dependent on the way the chosen model is trained, the sequence of training tasks, and how the model is perturbed during the training process. First, we demonstrate that increasing learning rates after pretraining can remove negative pretraining effects. Secondly, we show that continuously changing from source to target task instead of "jumping" can reap significant benefits in performance and does not incur a deficit penalty. Thirdly, we display that resetting only the biases for each task in the training process appears decrease negative pretraining effects. Finally we speculate on plausible reasons such as qualitatively different training behavior during the "high learning rate phase" (Lewkowyez al. 2020) that may explain the observed effects.

---

Under review as a conference paper at ICLR 2021
f2
Model path (fo.....
final)
(T3, w'))
(T2,W2)
(T
Learning path T
Figure I: In the context of the negative pretraining effect, e.g. when pretraining blurred images before training on unblurred images (Achille et al., (2177), we explore the effects of interventions on "learning paths", shown the blue path the learning manifold M, on the generalization performance of neural networks. A point on the learning manifold is defined as a learning task == (p(z,)) v). L), composed of a data distribution of input and labels and loss function, and learning process w which determines how a model f is adapted given a task т. Tasks can vary by changing any part of the definition such as when the data distribution is changed along blurring nuisance as depicted above with cat images. Learning processes can be adapted by changing their update equation, c.g, via learning rate changes An. The learning path T determines model path changes from initial to final trained model fiimt. Via three interventions in the learning path we show that varying neural network learning paths can result in a removal of negative pretraining effects.
2 RELATED WORK
Sequential learning involves choosing a sequence of tasks and learning processes and following this
learning path improve performance a single set of target tasks. This strongly resembles the
objectives of lifelong learning (Thrun & Mitchell, 1995), continual learning (Lesort et al. 2019)
and meta-learning (Thrun Prati 2012: Schaul & Schmidhuber 2010 Hochreiter et al., 2001 and
connects to many time denendennddnnt learning aspects in relation to neural networks.
We are aware of several time-dependent phenomena in training neural networks such as catastrophic forgetting (French, 1999), critical learning periods (Achille et al. 2017). and time sensitivity of regularization and data augmentation (Golatkar et al., 2019). Achille et al. (2017) show that by dropping out certain frequencies during the beginning of training, networks are unable to recover full performance, even when those frequencies are reintroduced. Similarly, Liu et al. (2199) show that first optimizing on random labels ruins performance on clean data shown indefinitely to the same network afterwards. Recent information-theoreiic analysis (Shwartz-Ziv & Tishby 2017) suggests that neural network have distinct learning phases: First, phase where network parameters grow in information content, before, in a second phase, regflreggi and pruning away irrelevant information. More commonly known time-dependent sequential heuristics are learning rate schedules (Darken & Moody 199T), cyclical learning rates (Smith) 2017). and regularization annealing such as dropout annealing (Rennie et al. 2014). student-teacher model transfer (Vicente & Catichal 1997). and pretraining Erhan et al. 2009).
Time-dependent learning:
Curriculum learning and generation: Curriculum learning (Elmnn 1993 Bengio et 2009) is
the technique of training agents by showing them a progression of tasks from easier variants to harder examples attempt solve the final target task. Both the utility of transferring between similar or dissimilar tasks is debated, with some results pointing towards similarity (Achille et al. 2019 Ammar
al. 2014; Ritter et al. 2018) and others towards dissimilarity (Farquhar & Gal 2018 Nguyen
et al.
2019). Self-paced learning (Tang et al. 2015) extends curriculum learning by letting the model
itself influence the sequence of tasks (Graves et al. 2017). The order tasks, the curriculum, is
generally an input that is hand-designed by human experimenters and often based on heuristics to guide the scheduling (i.e. it may be easier to make an agent jump over small gaps before moving to larger ones (Heess et al. 2017)) Meta-lean Meta-learning (Schaul & Schmidhuber, 2010 Vilalta & Drissi 2002) is similarly related by focusing on speeding up or enhancing performance on a target task by pretraining on other tasks to first learn to learn.
2

---

Under review as a conference paper at ICLR 2021
Biological analogies: It is through neuroplasticity (Dayan & Cohen 2011) that animals and humans learn new skills and recover from changes or damage to their sensory organs or brain. Famously, Erismann and Kohler (Kohler & Pissarek 1960) asked study participants to wear special mirror goggles which flipped the participants visual field upside down as summarized in (Sachse et al.) 2017). Remarkably, the participants visual perception adjusted after a few days by returning the visual field back to their "upright" standard perception. In contrast to this, there are other stimuli people learn to discard in childhood if they are not needed. Young children can differentiate phonemes of both their own and foreign languages; however, older children are better at distinguishing native phonemes and have more difficulty distinguishing foreign phonemes (Cheour et al. 1998] Kuhl et al. 1992). A similar effect of "perceptual narrowing" is found for 6-month-old and 9-month-old children when comparing the ability to discriminate between human faces to the ability to discriminate between primate faces (Pascalis et al. 2002). Younger children are again found to be less specialized to human faces than slightly older ones. Summarizing many results (Costandi 2016). we currently know that neuroplasticity is "largest" at early age yet generally does not end in adulthood. It is instead always present in reorganization processes involved in new experiences and when adjusting to injury.
3 SEQUENTIAL LEARNING SETTING
We are interested in better understanding what influences the outcome of a sequential learning problem. In the following, we say that a pretrained model has plasticity if it can generalize similarly well when trained on target task as an uncorrupted model trained directly on the target task. If it cannot generlize similarly well, we refer this as negative pretraining effect. To facilitate further reading, we summarize essential elements such as the definition of a learning task т, constrained to supervised tasks in this work, the learning mapping w, the learning path as an ordered sequence of task-learning process pairs T, and the setup of its repeated application in sequential learning process Q in Tab. Tand visualize the overall setting in Fig.
Table 1: Summary of notation, definitions, and terminology
Symbol
Definition
E 24 y p(z,v)
Input, model output, and desired output Joint probability distribution of input and target label
f:Xx θ - 10
Model function with parameters in parameter space
f(p,f): ()
Loss objective of accumulated loss of model across distribution Learning path T as a sequence of tasks T, and learning processes wi
T= ((TI, (Tn, wn))
Task T = (p,
Definition 1 (Supervised task). We define supervised task as the opti- mization problem associated with the tuple of dataset distribution p(a, y) and loss function C(p, I') : (p. f) + R.. where in this work L(p, J) Byte. [e [x)) v)) with individual data point loss E: .. x y + B...
Learn w (T,f)++
Definition 2 (Learning). We define learning as a mapping of a task and an initial model fin to new learned model flcam: w (T, fmmi) + fream
2: (T,
Definition (Sequential learning process). We define a sequential learning process Ω of a learning path consisting of sequence of ordered task leakn process pairs T = (1).......... (Tn, wn)) as the sequential application of the task-learm process pairs to initial model fui-
fo Wr(Tn, 4n ( ( (........y fin) Q(T, fini)
task т consists of a probability distribution p(a, y) of data points and desired outputs as well as a loss function L that maps an onput-output behavior of model to cost score. The learning mapping w uses the information in task T and initial model fint to train a model optimized for the task fieam until convergence. Applying the learning mappings and tasks (T,,w) E T sequentially starting from an initial model fini is summarized via the symbol Ω. Based on these elements, we ask what influences the final generalization outcome task-learn process pairs T. Specifically, we examine these question via three interventions aimed at distinct ingredients of the learning process as visualized in Fig. 2
target task when following learning path of
3

---

Under review as a conference paper at ICLR 2021
Default
A) A Learning rate:
B) Higher task discretization
C) Model bias reset
To
To
Thmal
Thaal
4
Fasa
f:
Jami ft
3
Raset
Resel
Figure 2: We explore the negative pretraining effect for neural networks, where, as a default, a model is first trained via learning process w on a task To, e-g. with blurred images, and later trained on the same task with sharp images Tiial yet never regains the performance of only training on sharp images. We investigate three different interventions to this negative pretraining scenario: A) we vary the learning rate during training on blurred and sharp images; B) Instead of "jumping" from blurred to sharp images, we insert additional tasks with a finer discretization of blurring; C) we reset the biases of the network optionally before training on the first and/or second task.
3.1 GENERAL EXPERIMENTAL BENCHMARK SETUP
We will focus on stochastic gradient descent (SGD) learning applied to convolutional neural networks. As stopping criterion, we consider the scenario where training has already converged, where the validation loss cannot be decreased further by changing the model via SGD. Empirical task transfer often deals with tasks that are separate (aaiir al. 2018), and while fruitful empirically, becomes difficult to understand conceptually. Instead we employ controlled customized task changes, allowing us to better gauge the effect of task changes. With the basic building blocks in place, we ask in which cases, even with infinite number of data points. previous tasks can hinder learning and generalization on new tasks. We examine three intervention scenarios acting on various levels of abstraction visualized in Fig.
A) Learning rate changes: different learning rates both on the first and second task can influence pretraining effects.
Applying changes the learning process, we observe how employing
B) Task discretization: Intervening on the task-level, we consider the influence of inserting more tasks between first and last task by discretizing the changes from first to last task.
C) Model biases resetting: Finally, we act on the model and examine how final performance is affected when resetting biases before tasks.
We deal with four visual supervised classification datasets of which we use three every experimental section below thereby significantly extending smaller scale experiments by Achille et al. (2017):
CIFAR-10 Krizhevsky
2009):
A 32x32 full-color dataset with 10 classes.
MashionMNIST Xiao et al.
2017): ) A 28x28, black-and-white dataset with 10 classes
MNIST LeCCnn et al.
1998): A 28x28, black-and-white dataset with 10 classes.
SVHN Sermanet et al.
2012): : A 32x32 full-color dataset with 10 classes.
For all experiments if not mentioned otherwise, we use Pytorch Paszke et al. 2019 and run ResNet-18 (He al. 2016) with fixed initial learning rate of 10c-4 with momentum 0.9 and weight decay le-3. The networks are initialized using the standard Pytorch layer intialization discussed in Appendix 8.2 Each task is trained until convergence, for maximum of 200 epochs per task, using early stopping validation set with patience 50. As baseline, the model was trained only on the target task the standard uncorrupted dataset.
HIGHER LEARNING RATES PREVENT A LOSS OF PLASTICITY
The learning rate schedule is arguably one of the most important hyperparameters in neural network
training as attested to by many deep learning practitioners (Li al. 2019 Nakkiran 2020 Smith 2017). In several works (Seong et al. 2018 Zhu et al. 2018 Kleinberg et al. 2018 Lewkowycz et al. 2020), higher learning rate has been linked to better generalization due its increased probability to escape local minima during optimization. By escaping sharp minima, the network is deemed more likely to converge flat minimum (Hochreiter & Schmidhuber 1995). i.e. minimum with
4

---

Under review as a conference paper at ICLR 2021
CIFAR 10
MaSSI
SVHN
1.00
0.14
baseline
Test loss
0.25
0.75
0.12
0.50
0.10
0.20
[5,
5]
[10.
5]
[10.
100
[20.
20
[20,
20
[5.
5]
[10.
51
10
[20.
20)
[20,
20)
[5.
5]
[10,
[10.
10]
[20.
20)
[2n.
20)
Learning rate 7h. mn
[e-4]
Figure 3: Final test loss on unblurred images as swarm and violin plot. The x-axis displays the learning rates on the first task with blurred images and the second task with unblurred images. The y-axis shows the test loss on the unblurred images with dots colored to distinguish different random seeds. The dotted line indicates the mean baseline performance across 3 seeds. A clear trend that negative pretraining can overcome when increasing the learning rate is visible, particularly when moving from 10e-4 to 20e-4.
Hessian with lower maximal eigenvalue. Accordingly, we present number of experiments detailing the effect of the learning rate on outcomes of neural network training after negative pretraining.
We keep the number of tasks, i.e. two, fixed and focus on the blurring transformation for all three datasets. For the first task, all images are downscaled to 1/4 their size before rescaling to their original size. The second task has standard unblurred images. The default learning rate of 1e-3 is varied either for both tasks, meaning a constant learning rate over the learning path, or kept for the first and changed only for the second task. We picked 2c-3 higher and 5e-4 as a lower learning rate. Each experiment was performed for 10 random seeds. The baseline, trained directly on the standard uncorrupted data, is averaged across random seeds. Results are shown in Fig.
We observe the overall trend that higher learning rates appear to decrease the difference between models pretrained on blurred images and the baseline models trained sharp images. Remarkably and consistently across datasets, training first with a learning rate 1e-3 and then 2e-3 is better in expectation than only training the higher learning rate 2e-3. We conclude that in sequential learning, the precise order of learning rates across tasks can be crucial to obtain the best final learning loss.
5 SMOOTHLY CHANGING THE TASK CAN PREVENT A LOSS OF PLASTICITY
Reasoning that a more "continuous" change from source to target task provides the model with more opportunity adapt, we provide an intervention that changes the discretization of "jumping" " from source to target task. To create each learning path, we focus on effects of three common image transformations: blurring, contrast, and random shifts, as visually depicted in the appendix in Fig. Blurring applies a resizing operation to each image, before resizing it back to the original size (as done in (Achille et al. 2017)). Contrast changes an image's channel attributes: A contrast of 0% results in a fully gray image" whereas contrast of 100% results in the standard image. Finally, random shift changes the maximum width of the random affine transformation applied to the image: Each component of the shift is sampled from [-a, a] and then applied to the image, where depends on the task parameterization. For reproducibility further details are provided in appendix 8.1
Discretization of paths: Traditionally, curriculum learning is concerned only with the sequence of tasks. In contrast, to test plasticity, we are additionally concerned with how discretely we traverse the space from source to target tasks. To characterize this, we keep the learning method fixed and segment the one-dimensional task path space into tasks as shown in Fig. 4 and and focus on how N affects target task performance. We continuously transform tasks from source target task, without violating the definition that all tasks lie on a task manifold that can be derived from the definition of a task All discretization experiments in this section were with maximum of 100 epochs and patience of 20 epochs across 3 random seeds. As visualized in Fig. increasing discretization removes most
UUsing 10% minimum contrast to ensure meaningful gradients.
"hhis rules naive interpolation between two standard computer vision datascts.
5

---

Under review as a conference paper at ICLR 2021
CIPAR-10 Hll
MNNIST Bli
MNIST Biu
0.25
0,015
0.000
Test
(1250)
0.005
0.255
0.0
Shit
Saitt
Stift
Test lme
84
0.015
0.10
0.000
leadan
Xm2
X-6
Xw6
N-10
Took leattin N
Figure 4: Final test loss standard images swarm and violin plots. The x-axis records the number of discretization steps, where = 2 refers to training first, e.g. on blurred then on sharp images. Higher discretization provides more gradual task shift from initial final transformation. The y-axis shows the test loss on standard images with dots colored to distinguish different random seeds. The dotted line shows the mean baseline performance across 3 seeds. A clear decrease of negative pretraining is visible for CIFAR- 10. On FashionMNIST, average results do not change noticeably except for shift. MNIST does not show clear negative pretraining.
of negative pretraining blurring and contrast on CIFAR-10. Remarkably, the already improved results of random shift are further improved via discretization. On the other hand shows clear trend on the MNSS and SVHN datasets visible. Possibly on these datasets, the "nudge" that increased discretization provides is not enough to overcome entrenched model characteristics.
Cheinar
(T3
Contrast (T2, acl
0,0
(Tiww)
Dopperi
(r4:4) w)
(1 20)
to
0.3
Blurring
M
Ned
Vel
Yod.
Xad
Xell
Mot Ned Nort discretization
Nob. Nall
Noo Xel
Kell
Figure 5: Left: Various two-dimensional sequential learning paths and discretizations from source to target task. Right: Results of deterministic 2D-paths with different discretizations on CIFAR- 10. In most cases finer discretization improves results. We remark the surprising path-dependence of results.
Multi-dim. paths from start to target task: We extend the sequential learning setup multiple dimensions as visualized in Fig. 5 Previously we only varied tasks along one transformation "dimension".. Unlike the one-dimensional case, we now vary both contrast or random shift and blurring levels (see Fig. [S] left) at the same time. Each path starts with blurred images and either low contrast high random shift transformation. Then we reduce transformations in the order mentioned Fl
in the title of each subplot We present results of these paths CIFAR- 10 in Fig. 5 Across most experiments and transformations, we see that performance on the target task increases as we approach continuous density transformation between source and target tasks (i.e. as N increases). We observe that continuous changes in tasks can help networks adapt to a new task in contrast, jumpy paths induce distributional shift too large to overcome, leading to poor performance on the target task.
Example: Blur then Contrast means that start at blurred and low contrast images. The next tasks will first move to unblur the images, but keep them at low contrast. Once unblurred, task move increase the contrast. Diagonal: Refers lowering both transformations equally for each discretization step.

---

Under review as a conference paper at ICLR 2021
CIFAR 10
FashionMNIST
SVHN
baseline
Test loss
0.7
0.250
0.12
0.6
0.225
0.5
0.10
[No. No.
[No, Yeal
(Yes, Nof (Yess, Yeul
[No, Nof Nof [No, Yeal Noj
(Yen, No)
[Yia, Yoel
[No, No.
[No, Yesl
[Yes, Noj
[Yes, Yesl
Bias reset [BRg, BR]
Figure 6: Final test loss on unblurred images as swarm and violin plot. The x-axis denotes whether bias resetting was applied before the first task with blurred images and/or the second task with unblurred images. The y-axis shows the test loss on unblurred images with dots colored to distinguish different random seeds. The dotted line shows the mean baseline performance across 3 seeds. A clear decrease of the negative pretraining effect is visible for CIFAR-10. On FashionMNIST and SVHN, overall better performance is obtained however average results do not change very much compared to the [No, No] negatively pretrained network.
6 BIAS RESETTING CAN PREVENT LOSS OF PLASTICITY
The neural network literature is ripe with examples of initializing model to be more amenable to optimization. For sequence modeling. the LSTM network is commonly initialized by setting biases such that the forget gate is inactive (Gers et al. 1999 Van Der Westhuizen & Lasenby
2018). The same method has also found application in the first very deep networks such as Highway Networks (Srivastava al. 2015) and its recurrent extension Recurrent Highway Networks ZZilyy et al. 2017) to ensure that gradients flow through the entire network. Batch normalization (BN) (loffe Szegedy 2015) aims to normalize the bias and variance of network activations to improve training. Building upon this. Li et al. (016) adapted to a new domain solely by retraining the BN weights while leaving all other parameters unchanged. Likewise we reason that resetting biases may, albeit shortly, activate many Recu-activanionss in the network and thus allow gradients to flow freely.
In this section we foeus on the effect of resetting the network biases. Again, we use task path consisting of two tasks and blurring as transformation for all three datasets. Using bias reset (BR) implies, that before training on task all biases are reset by adding 0.01 to their value manually chosen to be of a similar order of magnitude as the largest biases in the network. Depending on which task bias resetting used, we distinguish the following set-ups:
1. No BR No BR: No bias resetting is used for both task, which is the default set-up.
2. No BR BR: Only before training on the second task the network biases are reset.
3. BR- No BR: Just right in the beginning, before any training, the biases are reset.
4. BR- BR: The biases are reset before both training on the first task and the second task.
Again we used 10 different random seeds for the experiments and provide as baseline which is averaged across three random seeds. Viewing results in Fig. we note that the removal of the negative pretraining effect is most pronounced on the CIFAR-10 110 dataset. Remarkable for this intervention is that resetting biases even before the blurred image task and not before training on sharp images ([Yes, No]) can remove the negative pretraining effect. On FashionMNIST and SVHN, the BR-innteveention improves the test loss slightly ([Yes, Yes] vs. [No, No]). For these two datasets, the negative pretraining effect is however also less substantial. Notable however is that with BR intervention some seeds now are better than the baseline whereas before [No, No] this was not the case. One observed difference between CIFAR-10 and the two other datasets is the amount of training until best validation epoch. Bias reset leads to an increase in the number of epochs (about +44% for [Yes, Yes] compared to [No. No]) for CIFAR- 10, while for FashionMNIST and SVHN the number of epochs decreases when using bias reset - 44%).To corroborate our understanding of these results we provide norms of biases, weight as well activation statistics in appendix 8.3
7

---

Under review as a conference paper at ICLR 2021
7 SPECULATION ON CAUSES OF THE NEGATIVE PRETRAINING EFFECT
What we have shown are three markedly distinct ways of removing negative pretraining effects. What remains unclear is why we observe such a learning and generalization behavior. In the following, we will attempt an explanation or speculation of the underlying causes of the observed learning phenomena. An important aspect to note before diving in deeper is that the negative pretraining effect not an optimization issue. In all cases the training loss is reduced either to zero or to the loss generally reaches upon convergence even when generalizing well. Negative pretraining causes a change in generalization on the test set but not a drop in performance on the training set.
Large update rate phase: A plausible explanation for the observed behavior is that the phase with larger updates similar to the large learning rule phase for neural networks is qualitatively different to updates in the low-loss setting. All interventions achieve settings where either A) a higher learning rate makes all updates larger. B) a higher loss setting is encountered more frequently, or C) resetting slows down convergence. Such high learning rate phase is associated with better generalization.
More specifically, Zhu et al. (018)) claim that the anisotropic noise of SGD helps the network escape sharp minima and thus generalize better. The escape probability can be higher, if the training loss is still large compared to the loss at the minimum of the SGD optimization. [Seong et al. (2018) evoke the same image, plausibly showing that large learning rates make it more likely escape sharp minima. In the same spirit Nakkiran (020) provides an example of how a mismatch between training and test loss landscape can favor optimization with an initially large learning rate. Lewkowyez et al. (020) argue that in the large learning rate phase the curvature of the loss landscape is lowered thus yielding a tangible difference optimization behavior. Focusing on bias resetting, centered biases are known to accelerate convergence (Clevert et al. 2015 Cun et al. 1991; LeCun et al., 2012 Schraudolph 1998). Ensuring that biases are not centered is thus likely to have the opposite effect of slowing convergence which may ensure training remains in the large update rate phase longer.
Taking the opposite viewpoint when gradient updates are not large, optimization is likely to occur within the "oottiizaaionn valleys" in which the optimization started due to pretraining. In the limit case of a converged network, such a scenario is described for very wide neural networks (Lee et al. 2019). These wide networks are characterized by the neural tangent kernel (NTK). The largest eigenvalue of the NTK is in turn tied to generalization behavior (Lewkowycz et al. 2020: Jiang et al. 2019: Dyer & Gur-Ari 2019). Possibly, the tangent kernel is not updated enough to overcome the negative pretraining effect when retraining on the target task. Corroborating this point, Oymak & Soltanolkotabi (019)) argue that SGD, under certain conditions, takes direct path to the minimum such that final parameters may change little from initial parameters. Thus all interventions may serve increase the training time, given the network more time and opportunity to adapt not only its output but also its tangent kernel to the target task.
Preshaped learning landscape:
8 DISCUSSION AND CONCLUSION
The proposed three interventions provide us with a host of insights on what determines plasticity as the ability to adapt in sequential lcarning. We learned that increasing the learning rate after pretraining can consistently benefit learning to the point of surpassing the performance of only training on standard data. In the discretization experiment setting, we learn that pretraining on another task can have a significant biasing effect on final results. By providing a finer discretization from source to target task, possible retraining issues are alleviated thus yielding a better target performance. Finally, perturbing the model after pretraining by resetting biases can similarly diminish the negative pretraining effect albeit not consistently. Resetting the biases of a network can increase the initial training loss for a new task and slow down convergence.
Several new questions arise from the experiments. What is the best task to start from? What is the best path of tasks and learning processes to follow thereafter? Which kind of model is best suited to make use of the advantages of carefully-chosen path?
In this work, we presented a novel perspective on the sequential learning behavior of neural networks through the lens of negative pretraining. While casting light issues a learning model may face in sequential learning setup, we also discover a variety of new questions. The picture we are left with is an intriguing sequential learning problem setting offering fruitful ground for further inquiry.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint 17Xivv1711.08856, 2017.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439, 2019.
Haitham Bou Ammar, Eric Eaton, Matthew E Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An automated measure of mdp similarity for transfer in reinforce- ment learning. In Workshops at the Ewenty-Eightt AAAI Conference on Artificial Intelligence, 2014.
Yoshua Bengio. Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48 ACM, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. arXiv preprint 1rXiv:2005.14165. 2020.
Marie Cheour, Rita Ceponiene, Anne Lehtokoski, Aavo Luuk, Jüri Allik, Kimmo Alho, and Risto Näätänen. Development of language-specific phoneme representations in the infant brain. Nature neuroscience, 1(5):351, 1998.
Djork-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1rXiv:151 2015.
Moheb Costandi. Neuroplasticity, MIT Press, 2016.
Yann Le Cun, Ido Kanter, and Sara A. Solla. Eigenvalues of covariance matrices: Application to neural-network learning. Phys. Rev. Lent, 66:2396-2399 May 1991. doi: 10. Ro.ehhyyeeeee..... 66.2396. URL https: //link abs wrg. 11tp:011/////hhhheeeeeee
66.2396
Christian Darken and John E. Moody. Note on learning rate schedules for stochastic optimization. In Advances in neural information processing systems, pp. 832-838 1991.
Eran Dayan and Leonardo G Cohen. Neuroplasticity subserving motor skill learning. Neuron, 72(3): 443 454, 2011.
Ethan Dyer and Guy Gur-Ari. Asymptotics of wide networks from feynman diagrams. arXiv preprint arXiv:1909.1 11304, 2019.
Jeffrey L Elman. Learning and development neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Dumitru Erhan, Pierre-Antoine Manzagol, Yoshua Bengio, Samy Bengio, and Pascal Vincent. The difficulty of training deep architectures and the effect of unsupervised pre-training. In Artificial Intelligence and Statistics, pp. 153-160, 2009.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint 18X.001805509733. 2018
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (4):128-135, 1999.
F. A. Gers, J. Schmidhuber, and F. Cummins. Learning to forget: continual prediction with Istm. In 1999 Ninth International Conference on Artificial Neural Networks ICANN 99, (Conf. Publ. No. 470), volume 2, pp. 850-855 vol.2, 1999.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence, In Advances Neural Information Processing Systems, pp. 10677-10687, 2019.

---

Under review as a conference paper at ICLR 2021
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning- SrinninngVooumme Volume 70, pp. 1311-1320. JMLR. org. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.00286. 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems 7, pp. 529-536. MIT Press, 1995.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94, Springer, 2001.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03 50.5000...6,,.. 2015.
Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced curriculum learning. Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint 1xXiv:912.022177 2019.
Robert Kleinberg, Yuanzhi Li, and Yang Yuan. An alternative view: When does sgd escape local minima? arXiv preprint arXiv:1802.066/75 2018.
I Kohler and Pissarek. Brillenversuche zur vertikalentäuschung [spectacle experiments on horizontal-vertica illusion]. Psychol Beiträge, 5:117-140, 1960.
A. Krizhevsky. Learning multiple layers features from tiny images. Master's thesis. Department of Computer Science, University of Toronto, 2009.
Patricia K Kuhl, Karen A Williams, Francisco Lacerda, Kenneth N Stevens, and Björn Lindblom. Linguistic experience alters phonetic perception in infants by 6 months of age. Science, 255(5044): 606-608, 1992.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFE, 86(11 (6(11):2278-2322 1998.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Rober Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in neural information processing systems, pp. 8572-8583, 2019.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Diaz-Rodríguez. Continual learning for robotics. arXiv preprint
00/22, 2019.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218. 2020.
Yanghao Li. Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation, arXiv preprint 1rXiv:1603.04777 2016.
10

---

Under review as a conference paper at ICLR 2021
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks, 2019.
Shengchao Liu, Dimitris S. Papailliopoulos, and Dimitris Achlioptas. Bad global minima exist and SGD can reach them. CoRR, abs/1906. 02613, 2019.
Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? arXiv preprint arXiv:2006. 10455, 2020.
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plassicity dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in psychology, 4:504, 2013.
Prectum Nakkiran.
Learning rate annealing can provably help generalization, even for convex problems. arXiv preprint wwxxv:20050073360 2020.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint 1908,0099 2019.
Samet Oymak and Mahdi Soltanolkotabi. Overparameterized nonlinear learning: Gradient descent takes the shortest path? In International Conference on Machine Learning, pp. 4951-4960, 2019.
Olivier Pascalis, Michelle Haan, and Charles A Nelson. Is face processing species-specific during the first year of life? Science, (00665571):1211-110 2002.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Steven Rennie, Vaibhava Goel, and Samuel Thomas. Annealed dropout training of deep networks. In 2014 IEEE Spoken Language Technology Workshop (SLT), pp. 164. IEEE, 2014.
Hippolyt Ritter, Aleksandar Botev, and David Barber, Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 3738-3748, 2018
Pierre Sachse, Ursula Beermann, Markus Martini, Thomas Maran, Markus Domeier, and Marco R. Furtner. "The world upside down" The Innsbruck Goggle Experiments of Theodor Erismann (1883-1961) and Ivo Kohler (1915-1985). Cortex, 92:222-233 2017. ISSN 19738102. doi:
10.10166/..ortex.2011 wtt.ccorrxx.2....44.. 7.04.014.
Tom Schaul and Jürgen Schmidhuber. Metalcarning, Scholarpedia, 5(6):4550, 2010.
Nicol N Schraudolph. Centering neural network gradient factors. In Newral Networks: Tricks of the Trade, pp. 207-226. Springer, 1998.
Sihyeon Seong, Yegang Lee, Youngwook Kee, Dongyoon Han, and Junmo Kim. Towards flatter loss surface via nonmonotonic learning rate scheduling. In UAI, pp. 1020-1030, 2018.
Pierre Sermanet, Soumith Chintala, and Yann LeCun. Convolutional neural networks applied to house numbers digit classification. In Proceedings of the 2 International Conference on Pattern Recognition (ICPR2012), pp.
IEEE, 2012.
Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training nlp models: concise overview. arXiv preprint arXiv:2004.08900 2020.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100::00000000000 2017.
Daniel Silver, Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.

---

Under review as a conference paper at ICLR 2021
Leslie N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472. IEEE, 2017.
Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in neural information processing systems, pp. 2377-2385, 2015.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(1-2):25 46, 1995.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Jos Van Der Westhuizen and Joan Lasenby. The unreasonable effectiveness of the forget gate. arXiv preprint
2018.
Renato Vicente and Nestor Caticha. Functional optimization of online algorithms in multilayer neural networks. Journal of Physics A: Mathematical and General, 30(17): 1997.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
Zirui Wang, Zihang Dai, Barnabás Póczos, and Jaime Carbonell. Characterizing and avoiding negative transfer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 11293-11302, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 2017.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3712-3722, 2018.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint 1rXivv:8033.0 00195, 2018
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189-4198. JMLR. org, 2017
12


Reviews:
[{"review_id": "gaeMs7gLkdb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** This paper conducts an empirical study to examine the well-known negative transfer phenomenon (termed as a negative pretraining effect in this work) in neural networks. In particular, a network trained on a sequence of tasks performs inferior to a network trained from scratch on the intended target task. The main idea of the paper is to study this phenomenon by formulating and intervening on different constituents of the sequential learning process - (1) changing the learning rate across tasks, (2) number and type of tasks encountered in the learning process, and (3) resetting the model biases when going from one task to another. The paper conducts experiments on four visual classification datasets (CIFAR-10, FashionMNIST, MNIST, SVHN) and report their findings for sequential training of ResNet-18 architecture. They show that increasing the learning rate after training on the first task can alleviate the negative pretraining effect. They further showcase how different task discretization and resetting model biases help to reduce the effect. \n\n**Pros:** \nThe paper investigates an important problem of negative pretraining which has implications for different time-dependent learning paradigms (lifelong/ continual learning). Systematically studying how different constituents of the sequential learning process affect the severity of the negative pretraining effect is a reasonable approach and this paper attempts to take an initial step with this approach. \n\n**Cons:**\nThere are several concerns about the experimental setup which makes the results unconvincing:\n- The paper examines a single model (ResNet-18) on four datasets and through different experiments, the paper demonstrates that on MNIST, FashionMNIST, and SVHN datasets there is no clear (or less substantial) negative pretraining effect. However, this paper is about how different interventions can help mitigate the negative pretraining effect. If the current experimentation setup does not render the phenomena (except for CIFAR-10), this raises the question of whether the paper is analyzing the right setup (datasets and model)? Effectively a single model (ResNet-18) is examined on the CIFAR-10 dataset for the interventions, which is not representative enough. What happens if we increase the model complexity? What happens if we change the input modality? (see [1] for more details)\n- The paper concludes that increasing the learning rate on the subsequent task helps to **remove** the negative pretraining effect. In Figure 3 (discussion in Section 4) they report **a single case** of increasing the learning rate (10e-4 to 20e-4). It is unclear whether this is always the case. What happens if the learning rate is increased from 10e-4 to 50e-4 or 10e-4 to 10e-3? What is the general recipe to set the learning rate for the next task in the sequence?\n- The paper claims that it has proposed three distinct ways to **remove** the negative pretraining effect. Given the empirical evidence, this **(remove)** is too strong a claim to make. In some experiments, it alleviates the negative effect (see Figure 3, 4, 6: CIFAR-10) while in other experiments, results are inconclusive (see Figure 5: CIFAR-10, Shift-Then-Blur, Contrast-Then-Blue, and most of the experiments on MNIST, FashionMNIST, SVHN). \n\n**General comment:** There is no denying the fact that this paper studies an interesting phenomenon through systematic experiments. However, the authors should consider evaluating the proposed interventions on the setup (datasets/ models) where the negative pretraining is a clearly visible phenomenon to conclude generic applicability of the discussed interventions.\n\nPlease cite the below-mentioned work as it also empirically demonstrates the difficulty of warm-starting with pre-trained initialization.\n\n[1] Ash, Jordan T., and Ryan P. Adams. \"On the difficulty of warm-starting neural network training.\" arXiv preprint arXiv:1910.08475 (2019).\n\nPlease cite the peer-reviewed version of the related literature instead of the arXiv (for available ones), e.g.:\n\nDyer, Ethan, and Guy Gur-Ari. \"Asymptotics of Wide Networks from Feynman Diagrams.\" International Conference on Learning Representations. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g-NZLIsNBgM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors formalized the sequential learning problem and the negative transferring in this type of learning, and conducted empirical study on three interventions that can help to remove the negative transferring. Sequential learning (and its variants, such as continual learning and curriculum learning) is an important topic to study. \n\n\\+ formalization of sequential learning and the negative transferring of the learning, which facilitates future research on this topic\n\n\\+ Providing solid results (all on computer vision task though, more study on problems from diverse domains are desired) that demonstrates the high learning rate intervention is an effective approach to remove negative transferring. \n\n\\- The results for smoothing and bias resetting are somewhat limited. Out of the three datasets, only one clearly shows the impact of the intervention. To justify these two as useful approaches in general to removing negative pretraining effect, evaluation on more datasets and diverse learning problems is desired.\n\n\\- All studies were carried out on well controlled synthetic datasets. I understand that these datasets were generated from real life datasets and these well controlled studies are definitely needed.  However, a study (or more) of practical sequential learning problem is much needed to demonstrate the impact of the three interventions on real-word applications.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mz4W59bZ6d", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## 1. Brief summary\nThe authors study what they call a negative pretraining effect = models pretrained on task 1 and tuned on task 2 sometimes underperform compared to just training on task 2 from scratch. This is an important factor in many forms of life long learning, multi-task learning and curriculum learning. They investigate 3 potential remedies / setups: a) using different learning rates for task 1 and task 2, b) changing from task 1 to task 2 more smoothly, and c) resetting network biases at different stages of the process. The perform with a single ResNet18 architecture on MNIST, Fashion MNIST, SVHN and CIFAR-10.\n\n## 2. Positive things\n* I think the problem is very important and applicable in many problems in ML, especially in its practical deployment\n* The problem is important and interesting from the theoretical point of view as well\n* The interventions / experimental setup the authors use (a) LR changes, b) smooth task transition, and c) bias resetting) are all reasonable and potentially actionable if shown effective\n* I like that the authors report results of multiple random seeds and show the full distribution of the results. This allows the reader to better judge the validity of the claims made and \n* I like many aspects of your experimental setup\n* I like the idea with path discretization and studying the effect of the number of discrete steps\n\n## 3. Negative things / points of confusion\n\n1. The experiments are way to limited in scope to establish generally applicable results. I'm not asking you to go all the way to ImageNet, but several things would greatly improve the potential value of your work:\n\na. Adding datasets with a different number of classes than 10. All the datasets you used have 10 classes (MNIST, Fashion MNIST, SVHN, CIFAR-10) and it is at least plausible that some of the effects you see could be caused by some sort of class specialization. It would be very helpful to see smaller number of classes (you can restrict some of the datasets you have already) and let's say CIFAR-100 on the other end. I think these would not be crazy hard to add but would greatly increase the value of your paper.\n\nb. You only use a single architecture. It would be good to try others. Especially the learning rate considerations and experiments could be influenced by the fact that you chose a ResNet which are often hard to train from initialization at low learning rate (people sometimes preheat them by high LR initial phase of training). You could try a simple multi-layer CNN, and fully-connected multi-layer net. It would be good to see that your conclusions hold generally, not just for ResNets.\n\n2. I am not extremely convinced by some features of your experiment A where you use different learning rates for the blurred task 1 and the sharp task 2. Firstly, why is the baseline performance a fixed LR = 10e-4? Did you use a hyperparameter search of sorts of find it? Because if it were optimal, the fact that e.g. panel 1 in Figure 3 the first two results perform worse could be just because their LRs are smaller, and not because of the effect you're trying to observe (the negative pretraining). It is at least suspicious that the blurred -- sharp task starts performing equally to the default sharp task *precisely* when the LR of the blurred -- sharp task is at least as big as the default task. It is certainly a confounding factor that does not allow you to draw the conclusion you do -- namely that it is the LR change that helps, rather than just increasing the LR. I suggest you show multiple baselines with different LRs so that we can compare to them. Also, please show the errorbars for the baseline performance as well, so that we can judge the overlap of the uncertainty intervals.\n\n3. In Figure 3 there is essentially no signal there for Fashion MNIST and a week signal for SVHN. How does that fit with the causal explanation you are offering. I'm really not seeing how you can draw a conclusion as strong as \"the precise order of learning rates across tasks can be crucial\". I don't think Figure 3 shows that clearly, or possibly at all. Especially considering the potential confounding that I described in point 2.\n\n4. For experiment A in Figure 3, wouldn't you expect that the ordering of the LRs would matter. In panel 1 [5,5] and [10,5] look the same, why? and [10,20] and [20,20] also look the same. How could both be true if you're expecting that you should either increase/decrease the LR to get good performance.\n\n5. This is a point that I am not sure about, so please correct my if I'm wrong. I don't get how precisely you do the bias resetting. Do you set all biases to 0, or reinitialize them again from a distribution. Why would doing this at the begging before the blurred task have any effect. And if it does, doesn't it just mean that your original initialization was not good enough and now you made it better. It might not have much to do with the sequential learning but rather with this. Let me know if I'm wrong here.\n\n6. In Section 7 you say \"negative pretraining is not an optimization issue\". That is possible, but I don't think you show that. The fact that it's a question of generalization and not the training loss going to zero does not mean it's not a question of optimization -- optimization includes the generalization properties of whatever optimum it found, so it definitely does deal with this. This might be an issue of semantics and if so just let me know, this is not a major point.\n\n7. Figure 6 results don't look very conclusive to me, which probably means that the bias reset doesn't work too well (method C).\n\n## 4. Tiny issues and suggestions (I'm not judging the paper based on those):\n* You use e.g. 1e-1 instead of $1\\times10^{-1}$ and it doesn't look very good (it's also kind of hard to read). It might be worth typesetting those values in a more aesthetic way.\n\n## 5. Potentially relevant papers\nWhen you discuss the effect of learning rate on updates, it sounds similar to this paper:\n[1] Stiffness: A New Perspective on Generalization in Neural Networks (https://arxiv.org/abs/1901.09491) where they measured how \"stiff\" a NN is when trained with different LRs by comparing the learning effect of one image on another (also between datasts).\n\nFor the bias resets the class structure the DNN learns might be crucial. Here's a good overview of such class-specific effects:\n[2] Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra (https://arxiv.org/abs/2008.11865)\n\nIn the NTK discussion [3] discusses how the NTK behaves when you expand it around a partially pre-trained point in the weight space which sounds relevant to the way you study how the task 2 training depends on where task 1 got you:\n[3] Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel (https://arxiv.org/abs/2010.15110, NeurIPS 2020). It also discussed the importance of the early phases of training that you discuss.\n\n[4] The Break-Even Point on Optimization Trajectories of Deep Neural Networks (https://arxiv.org/abs/2002.09572, ICLR 2020) also looks at the crucial effect of the early phases of training.\n\n## 5. Conclusion\nI think the question asked is interesting and the approach the authors took promising. However, the breath of experiments is not large enough and I think there are significant potential confounding effects in at least the setup A (changing the learning rate) that make it really hard / inconclusive to draw strong causal conclusions about the effect of the intervention on the negative pretraining effect. I encourage the authors to improve the paper and resubmit -- this seems like a really promising piece of research, but as it is it's not strong enough.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E9DyIP1jevS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summarize what the paper claims to contribute. Be positive and generous.\nThe paper claims to contribute three ways of mitigating negative pretraining:\n  (1) altering the learning rate after pretraining,\n  (2) increasing the discretization of data distribution changes from start to target task instead of \"jumping\" to a target task, and\n  (3) resetting network biases to larger values.\nInteresting hypotheses! Agreed that negative pretraining effect is a great topic to be studied further.\n\n2. List strong and weak points of the paper. Be as comprehensive as possible.\n (1) Strengths\n    a. The paper focuses on a single topic of exploring ways to mitigate negative pretraining effects.\n    b. Easy to read and follow.\n (2) Weaknesses\n    a. Empirical results aren't strong enough to back the three main claims; Need further analysis. \n      i. For example, changing learning rate didn't seem to help with the learning task on FashionMNIST dataset. \n      ii. Increasing the discretization of data distribution doesn't seem to improve the learning for (FashionMNIST, Contrast), (FashionMNIST, Blur), (MNIST, Blur), (MNIST, Contrast), and (MNIST, Shift). \n      iii. The results of resetting network biases are not consistent across different datasets; need further analysis to make a more clear conclusion. \n      iv. Some test loss differences seem too small and made me wonder about their statistical significance.\n\n3. Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.\nReject. Because of the aforementioned weaknesses, it is unclear what conclusion to make and apply to other research topics and/or applications.\n\n4. Provide supporting arguments for your recommendation.\nSee 2.(2).a.\n\n5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \n  (1) Could you provide further analyses to refute i-iv in 2.(2).a?\n\n6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.\n  (1) It will be helpful to qualitatively understand what the fine-tuned network learned if you visualize convolution features of both pretrained and fine-tuned models.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "DghCIDYrsF7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates the intriguing phenomenon where pretraining on one task hurts the finetuning performance of another. Besides being interesting in general, this phenomenon has practical relevance as pretraining becomes increasingly popular with large-scale models. Here, the authors present a clean case for “negative pretraining effect” on images, and propose three ways to mitigate it.\n\nHowever, the investigation is limited to supervised image classification on four small scale datasets with blurring as the only deficit studied. The authors noted that the setup expanded that of Achille et al (2017); nonetheless, it would take more thorough experimentation to establish mitigations of a phenomenon, compared to merely observing one, especially given the empirical nature of this work. It’s not convincing to me that the proposed approaches are fundamental fixes that can be generally applied to other learning scenarios where one wishes to get rid of the “negative pretraining effect”.\n\nThe authors should consider expanding the set of “changes” they consider and experiment on a variety of tasks, ideally beyond vision, to establish the general efficacy of the approaches.\n\nSome other comments: 1) The background on biological analogies is nice but has no explicit connection to the main idea of the paper; 2) Why is it interesting to apply “bias resetting” to the initial task, as the model starts from random anyway? 3) Why adding 0.01 to biases to “reset” it, which is hacky and hard to generalize, as opposed to “resetting” biases to 0? 3) The only case where “bias resetting” seems to help nontrivially is for CIFAR-10, but at which point this “resetting” is applied doesn’t seem to matter; why is that the case? Overall the results on “bias resetting” appear negative and perhaps shouldn’t be included as a mitigation, especially given that the idea itself lacks any concrete theoretical motivations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Taking all reviews and the work in consideration, unfortunately the work does not present the breadth it needs to sustain the claims it makes. In particular, there work requires to analyse more architectures/variations of datasets with different properties and to provide more careful ablation studies that shows the efficiency of the 3 different proposed methods. Potentially removing one of this methods in order to give more space to analyse the others that seem more promising. 
Decision: Reject