Retrospection: Leveraging the Past for Efficient Training of Deep Neural Networks
Authors: Ayush Chopra
Surgan Jandial
Mausoom Sarkar
Balaji Krishnamurthy
Vineeth Balasubramanian
Venue:  (2020)
Keywords: Deep Neural Networks
Supervised Learning
Classification
Training Strategy
Generative Adversarial Networks
Convolutional Neural Networks

Abstract: Deep neural networks are powerful learning machines that have enabled breakthroughs in several domains. In this work, we introduce retrospection loss to improve the performance of neural networks by utilizing prior experiences during training. Minimizing the retrospection loss pushes the parameter state at the current training step towards the optimal parameter state while pulling it away from the parameter state at a previous training step. We conduct extensive experiments to show that the proposed retrospection loss results in improved performance across multiple tasks, input types and network architectures.

Under review as a conference paper at ICLR 2020
RETROSPECTION: LEVERAGING THE PAST FOR EFFICIENT TRAINING OF DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks are powerful learning machines that have enabled break- throughs in several domains. In this work, introduce retrospection loss to im- prove performance ing. Minimizing the retrospection loss pushes the parameter state at the current training step towards the optimal parameter state while pulling it away from the parameter state at a previous training step. We conduct extensive experiments to show that the proposed retrospection loss results in improved performance across multiple tasks, input types and network architectures.
neural networks by utilizing prior experiences during train-
INTRODUCTION
Large deep neural networks have enabled breakthroughs in fields such as computer vision (Krizhevsky al., 2012), speech recognition (Hinton al., 2012). natural language understand- ing (Mikolov et al. 2013) and reinforcement learning (Mnih al. 2015). Hence, recent times, sigmificant effort has been directed towards enhancing network efficieccy through data augmenta- tion, regularization methods and novel training strategies (Zhong al. (017) (Zhang et al. (017), (Huang et al.) (017)) (Noh et al.) 2017). (Wang et al. 2018) (Han et al, (016)). In this work, we introduce a technique to improve performance by utilizing prior experiences of the network during training.
Humans are efficient learners with the ability to quickly understand and process diverse ideas. A hallmark of human intelligence is the capability to internalize these complex ideas by actively refer- encing past interpretations to continually adapt understanding. Our artificial agents should be able to do the same, learning and adapting quickly. This kind of fast and flexible learning is challenging, since the agent must effectively integrate its prior experience with small amount of new informa- tion, while avoiding overfitting to the new data.
In this work, we introduce a new retrospection loss that utilizes prior training experiences of a deep neural network to guide parameter updates and improve performance. The idea for the retrospection loss is simple to ensure that the predictions at a training step are more similar to the ground truth than to the predictions from previous training step. As training proceeds, minimizing the loss constrains the network parameters to continually evolve towards the optimal state by successive constriction of the predictions into tighter spaces around the goal. The proposed retrospection loss simple, easy to implement and we empirically show that it works well across multiple tasks, input types and network architectures.
The key contributions of our work can be summarized as:
simple, easy to implement retrospective loss that is based on looking back at the trajectory of gradient descent and providing an earlier parameter state as guid- ance for further learning.
We propose
We exhaustively experiment on wide range of tasks including image classification (+ few- shot learning.. GANs, speech recognition, text classification and consistently beat state-of- the-art methods on benchmark datasets with the addition of this loss term.
To the best of our knowledge, this is the first such effort; our empirical studies showed a consistent improvement in performance across the tasks our multiple trials, demon- strating the potential of this method to have a strong impact on practical use in real-world applications across domains.

---

Under review as a conference paper at ICLR 2020
2 RELATED WORK
The retrospection loss leverages the parameter state from a previous training step guidance to compute the current gradient update. Correspondingly, one could find similarities with efforts in optimization, that utilize information from past training steps for future weight updates as well as methods that leverage guidance from other parameter states during training.
Techniques such as SVRG (Johnson & Zhang 2013), SARAH(Nguyen et al. 2017). ProxSARAH
(Pham et al., 2019) use gradients from earlier training steps to predict better weight updates. Other optimization methods like Momentum (Sutskever et al. 2013). Adam (Kingma & Bal 2014) Nes- terov Momentum (Jin et al., 2018) accumulate past gradients to accelerate weight updates in the right direction in order to achieve faster convergence. In contrast, our work introduces an additional training objective to guide convergence, and can be used to improve performance when used with different optimizer configurations, as shown in our results.
In reinforcement learning (RL), where techniques involve optimizing using moving (evolving) tar- gets, methods for Q-learning and policy gradients benefit from using a guidance network during training. The DQN algorithm proposed by (Mnih et al., 2015) uses an additional target network (same as online network) for Q-value updates, where parameters are updated by copying from the online network at discrete steps. Double Q-learning (Hasselt 2010) learns two Q functions, where each Q-function is updated with a value for the next state from the other Q-function. Policy gradient methods such as TRPO (Schulman et al., 2015). PPO (Schulman et al. (017) use KL--divergence objective during training that constrains the loss to ensure deviation from a previously learned policy small. In these techniques, leveraging a guidance during training results in improved convergence and sample efficiency. Note that all these efforts are constrained to the RL setting. Further, the objective in the RL setting is to control divergence from the guidance step to better handle moving targets. On the other hand, the proposed retrospection loss is formalized differently address the supervised learning setting. To the best of our knowledge, this is the first such effort that uses an idea such retrospection in supervised learning.
METHODOL.OGY
We now present the formulation of our retrospective loss. Consider a neural network, g()), param- eterized by its weights 0. Let the optimal parameters of the neural networks at the end of training be given by 0" The current parameters of the network at time step T during training are given by 0², The objective of the retrospective loss is to leverage the past states during training, and cue the network to be closer to the ground truth than a past state at time step Tp. Given an input data-label pair (x., y:), the retrospective loss is given by:
Eretrospective K * lg (x:) will - 1lger (X:) 90t, (x:)
The retrospective loss is designed such that minimizing it with respect to 0 over the training steps would constrain the parameter state at each reference step HT to be more similar to 0" than the parameter state from the delayed time step &T. The scaling term is required to obtain sufficient gradient signal in later stages of training when gor (x:) is close ye. and the first term becomes small.
Adding this loss term to existing supervised learning task loss provides for efficient training, as shown in our experiments later. The retrospective loss introduced to the training objective following a warm-up period wherein the neural network function can be considered stable for use of retrospective updates. The training objective at any training step T with the retrospective loss is hence defined as:
C={Crnst Llask Crask
T Iw T 2 Iw
C C=
(2)
LT Leetrospeccioee
where Lrask is the task task-specific training objective and Tw is the number of warm-up iterations. We simply use Tp F * [T/F] as the time step for retrospection in this work, and show gains in effi- ciency of training. One could however mine for T, intelligently to further improve the performance. Here, F is the retrospective update frequency, which gives an upper bound difference of the previous training step (Tp) from the current training step T at which we compute the retrospective loss.
Geometric Intuition. Figure Tillustrates the geometric intuition of the working of the retrospec- tive loss. By design (Eqn Eretrospective is negative when the current parameter state is farther away from the retrospective step, Tp, than the optimal solution (which is the desirable objective).
2

---

Under review as a conference paper at ICLR 2020
1,/<0
L,70
L/<c0
4/50
gr
0
er
L, >0
4, k* kllppp()) Bo4X) ([Br(()) Ban(x) II
L," k*)la-(()) -ex(x) II- l[(g(()) 8a1k)
Figure 1: Geometric intuition of the working of the proposed retrospection loss. The figures show polytopes in the weight parameter space. (Left) For all O inside the shown colored polytope, the retrospective loss is negative and is positive outside. Our objective is to push parameters of the current 8 further inside this polygon close to 0"; (Right) In a future time step T" > T, by design of the retrospective loss, the polytope region shrinks and our objective at this time step is to push parameters near-optimal region around 0*
One could view the loss term as dividing the pa- rameter space into two regions: a polytope around the optimal 0* where Cretrospective
Algorithu Retrosppcctive Training
1: Imput: Training Set. V. Current Model Parameters 8", Previous State Model Parameters 8*x, Update Frequency F, Warm-Up Iterations Iw.
for Step
gradiaa +- 0 (Initialising the gradieats w.r.l task-seeiific loss)
and the region outside the polytope 0.
grod.atrupeziive 4-0 (Initialising the gradients w.rt retrospective loss) Training Data of minibatch size B pains of (X()), Y(0).
S:
6. L(4r, Y(s) = Leoa( (X(()), Misi)
where Cretrosecctive Minimizing retrospective loss pushes the network towards parameters further inside the polytope, thus helping speed up the training process. As shown the right subfigure in Figure [ the polytope shrinks over time, since the retrospective support, Tp. is also updated to more recent parameter states. This helps further push the parameters into near-optimal region around d". The loss term helps in improved solution in most cases, and faster training certain cases, as shown our extensive empirical studies in Section Algorithm 1 summarizes the methodology.
7:
graduas V(L(6", (1( .X(),, Y(8)
then if L(0" X(t), YU) Enseroggeetin (0" (00))66+(((0) Y0)
grad gradgetrappetive V(L(0" " " X(8,, Y(i)
11:
end if
12:
Step
then 8% 6"
end
8" +- 8" 7 (grad grad grad
16: end for 17:
Connection with Triplet Loss. The triplet loss (CChechik et al., 2010; Schroff et al. 2015 Hoffer & Ailon 2015)) has been proposed and used extensively over the last few years learn high- quality data embeddings, by considering triplet of data points, x, (anchor point), x,, (point from the positive/same class the sample under consideration), and x,, (point from the negative class/class different from the sample under consideration). The loss is then defined as:
max (11.. spll Ilga gn. 10.1²+ +1) m.
where g the neural network model, and m is minimum desired margin of separation. The triplet loss, inspired by contrastive loss (Hadsell et al. 2006), attempts to learn parameters a neural network in such way that data points belonging to the same class are pulled together closer than data point from another class. One could view the proposed retrospection loss triplet loss the parameter space. While the traditional triplet loss consider a triplet of data samples, we consider a triplet parameters, 0", 0", and 0=,, We however believe that retrospection captures the proposed loss better, since we consider previous parameter states in time.
Connection with Momentum. Viewing retrospection from the perspective of previous gradients the training trajectory, one can connect it to the use of momentum, although more in contrasting sense. The use of momentum and variants such as Nesterov momentum (Jin et al. 20188) in training neural networks use the past gradient, say at 0f-1 or the gradient over the previous few steps, at 0), while updating the parameters in the current step. This assumes local
(0T-9.... .T-
3

---

Under review as a conference paper at ICLR 2020
MNIST LeNet
FashionMNIST ResNet-20
SVHN VGG-11
ResNet-18
92.5
95
88
90.0
Accuracy
93
83
92
80
Without Retrospection
Retrospection
82.5
80.0
Without Retrospection Retrospection
01
Without Retrospection
Retrospection
Without Retrospection Retrospection
25000
50000
75000
1000
20000
40000
60000
50000
100000 2500
50000 100000 150002000000
Steoss
Steos
Figure 2: Classification performance using retrospection on F-MNIST and SVHN datasets
consistency of the direction of the gradient update in the training trajectory, and that one can use these previous directions get a more robust estimate of the gradient step be taken currently. In contrast, retrospection leverages the same idea from the opposite perspective, viz., consistency of direction of the gradient update is only local, and hence the parameter state, 0 farther away from the current state 0", provides interesting discussions, and the possibility of analyzing retrospection as a thrust obtained from an undesirable parameter state, as opposed to momentum. We leave these as interesting directions of future work, and focus this work on proposing the method, and showing its effectiveness in training neural networks.
of what the next parameter must be far from. This raises
4 EXPERIMENTS AND RESULTS
We conduct experiments using retrospection the following tasks: image classification (Sec 4.1 image generation (See 4.2) speech recognition (Sec 4.3), text classification (Sec 4.4) and few- shot image classification (Sec 4.5 During experimentation, the original (without retrospection) and retrospective (with retrospection) configurations are trained using ensure consistency comparison. For all experiments, we use the L1- -oom as the choice of norm in our implementation for the retrospective loss (Bqn []. When retrospection is used without warm-up, the guidance parameters, 0Tr are initialized at random.
weight initialization,
4.1
IMAGE CLASSIFICATION
We perform image classification experiments using Fashion-MNIST (Xiao et al. 2017). SVHN (Netzer et al., 2011 and CIFAR-10 (Krizhevsky (209) datasets. The retrospection loss, for classi- ficatton, uses activations of the softmax layer. The default hyperparameter configurations for retro- spection include warm-up period of zero epochs and retrospective update frequency of fifty steps. The parameter, K, initialized 4 and increased by 2% at each retrospective update. Quantitative results for image classification are compiled in Table [
Fashion-MNIST. For experiments on Fashion MNIST, we use LeNet (Lecun et al. 2001 and ResNet-20 He et al. 2016) architectures. Models in each experiment are trained to convergence us- ing the SGD optimizer (1r-0.1, momentum=0.5, mini-batch=32) running over 70,000 steps. Results in Figure 2 (a)-(b) show that using the retrospective loss results in improved training.
SVHN. For experiments on SVHN, use VGG-11 (Simonyan & Zisser- man 2014) and ResNet-18 (He al.
Dataset
Model
Original
Retrospective
LeNet
10.8
9.4
2016) architectures. Models in each ex- periment are trained to convergence us- ing the SGD optimizer (Ir=0.001, mo- mentum=0.9, bini-batch=100) running over 200,000 steps. Results Figure (c)-(d) show that using the retrospective loss results in more efficient training.
F-MNIST
ResNet-20
7.6
6.8
VGG-I1
5.54
4.70
SVHN
ResNet-18
4.42
4.06
44
6.98 (7.17)
6.55
CIFAR-10
5e
6.86 (6.97)
6.52
ResNet- 10
6.55 (6.61)
6.29
Table 1: Classification error using retrospection on F- MNIST, SVHN and CIFAR-10 dataset
CIFAR-10. For experiments on CIFAR-10 (Krizhevsky 2009). use larger variants of ResNet including ResNet 44, 56, 1 110 (He et al. 2016). Models in each experiment are trained for 200 epochs, using
the training configuration (mini-batch, Ir policy) detailed in (He al. 2016). Here, observe that
4

---

Under review as a conference paper at ICLR 2020
using the retrospection loss in later stages of training results in best improvement in performance. Correspondingly, the retrospective loss introduced after a warm-up 150 epochs and the retrospective update frequency is one epoch. The parameter, is initialized at 4 and updated by 2% once every retrospective updates. Quantitative performance is reported in Table [ For sake of completion, we also mention (in brackets) the error rates for the corresponding experiments as reported by authors in the original work (He et al. 2016).
4.2 IMAGE GENERATION
Next, perform experiment with Generative Adversarial Networks (GAN) using Fashion-MNIST. (F-MNIST) (Xiao et al., 2017) and CIFAR-10 (Krizhevsky) 2009) datasets. Our study considers both unconditional (DCGAN, LSGAN) and conditional (ACGAN) variants of GANs. We adapt implementations from dcgg) for LSGAN (Mao al., 2016), DCGAN (Radford et al. 2015) and from (acg) for ACGAN (Odena et al. 2016). For our experiments, we train the generator and discriminator for 100 epochs, with initial learning rate of 0.0002 on minibatches of size 64 using Adam optimizer. We report performance using Inception Score (Salimans et al., 2016), standard metric for evaluating GANs. The inception score is calculated using implementation in (inc 2018) with predictions for CIFAR-10 generated using network in (Szegedy al. 2015) and features for F-MNIST using network Krizhevsky et al. 2012).
DCGAN CIPR-100
AOCAN CITAR-10
CEARCCO
Satropection
with hatraspeetion
3.
Insppter Stare
WEYEL Batvopectirr WEY Rcpsspecten
Epacts
Opecha
Opecha
DCGAL FIRAST
ACCAR TIRESST
LSGAN THNNET
Setropectien
Retroppcctin
with Rutraspection
5.:
t.
4.4
Tpeche
Opecha
Figure 3: Inception Scores using retrospection CIFAR-10 (Krizhevsky 2009 (row 1) and F- MNIST (Xiao et al. 2017) (row 2) datasets using DCGAN (Radford al. 2015) (col 1), ACGAN (Odena et al. (216) col 2), LSGAN (Mao et al. 2016 3).
Epoch
Epoch 45
Epoch 65
origina
retrospectiv
Figure 4: Images generated training epochs when ACGAN (Odena et al. 2016) trained
FMNIST dataset: (a) without retrospection (row 1) (b) with retrospection (row
5

---

Under review as a conference paper at ICLR 2020
For all experiments, the retrospection loss is initialized without any warm-up period (zero epochs). The loss is computed on outputs of the discriminator and is used to train the generator model. For DCGAN (Radford et al., 2015) and LSGAN (Mao et al. 2016) L2-norm as choice of norm. The retrospective update happens six times in one epoch. The scaling parameter, K is initialized at 4 and is not changed during training. For ACGAN (Odena et al. 2016), which is conditional, the retro- spective loss consists of both adversarial loss and class loss components. -nomm is used for class component and Lg-norm is used for adversarial component. Figure 3 presents comparative incep- tion score plots when the various dataset-network pairs are trained with (without) the retrospection loss. Additionally, Figure 4 presents images generated over epochs when training ACGAN (Odena et al. 2016). with and without retrospection, on F-MNIST (Xiao al. 2017).
4.3 SPEECH RECOGNITION
We perform speech recognition experiments using the Google Commands [Warden 2017) dataset. The dataset consists of 65,000 utterances, where each utterance is about one-second long and be- longs to one out of 30 classes. The classes correspond to voice commands such as yes, no, down, left, as pronounced by a few thousand different speakers. We follow (Zhang et al. (017) to prepro- cess the ufterances where first extract normalized spectrograms from the original waveforms a sampling rate of 16 kHz and subsequently we zero-pad the spectrograms to equalize their sizes at 160 x 101.
For this experiment, we compare LeNet(Lecun al. 2001 and VGG- (Simonyan & Zisserman 2014) architectrre, each of which is composed of two convolutional and two fully-connected layers. We train each model for 30 epochs
with minibatches of 100 examples, us- ing Adam as the optimizer. Training starts with a learning rate of 3x10-3 and
Model
Validation Set
Testing Sct
original
9.8
original
10.3
LeNet
retrospective
9.6
retrospective
9.9
is divided by 10 every 10 epochs. The retrospective loss is introduced after warm-up period of eight epochs, since we find it speeds up initial convergence. The retrospection update frequency is half epoch. The loss scaling margin, K. is initialized at 4, and is increased by 1% at each retrospective update. Results in Table 2 highlight that training using the retrospection loss decreases error rate for both LeNet (Lecun et al. 2001 and VGG-11 (Simonyan & Zisserman 2014) on both validation and testing sets.
original
5.2
original
5.0
VGG-11
retrospective
4.4
retrospective
4.2
Table 2: Classification error using retrospection on the Google Commands dataset
4.4 TEXT CLASSIFICATION
We perform text classification experiments on the task of emotion detection in dyadic conversations. We baseline our experiments against DialogueRNN (Majumder et al. 2019),, recent state-of-the work, which composed of an attentive network consisting of three Gated Recurrent Units GRU).
We perform experiments using AVEC (Schuller et al.) 2012) and IEMOCAP (Busso et al. 2008) datasets. While the datasets are multi-modal (image and text), following (Majumder et al.] 2019). we restrict scope of our experiments to using text. To feed into the network, the text data is pre- processed to obtain n-gram features detailed in (Majumder et al. 2019). We follow the same train- test split and training configurations as in the original work. Performance comparison reported against Bi Dialogge R.N N4 Att, the best performing variant from the original work.
For experiments on IEMOCAP, models in each experiment are trained for 60 epochs on enoss-entropy objective with F1-Score and accuracy as performance metrics. For retrospection, a warm-up of zero epochs used. On AVEC, models in each ex- periment are trained for 100 epochs using MSE loss with MSE and pear- pear-score(r) as the performance metrics. Here, introducing the retrospection loss after warm-up of sev- enty five epochs produces best performance. For experiments on both IEMOCAP and AVEC, the retrospective update frequency is one epoch. The loss scaling margin, R. is set to 4 at initialization and is updated by 2% at each retrospective update. Experiments are conducted using the official
Dataset
IEMOCAP
AVEC
Accuracy
F1-Score
MSE
Pear- Score (r)
original retrospective
62.66
62.75
0.179
0.318
64.60
64.75
0.177
0.332
Table Performance on using retrospection on task of dyadic emo-
tion recognition on DialogueRNN

---

Under review as a conference paper at ICLR 2020
code repository Coo 2019). Results in Table 3 show that using the retrospection loss when training DialogueRNN improves performance on both IECOMAP and AVEC datasets.
4.5 FEW-SHOT CLASSIFICATION
We conduct experiments the task of few shot classification using the CUB-200 (Wah al. 2011) dataset. The CUB-200 dataset consists of 1 1,7888 images from 200 bird species. In few-shot learning, the ability of a model is measured by its performance on n-shot, k-way tasks where the model is given query sample belonging
new, previously unseen class and a support set, S, consisting of n examples each from k different unseen classes. The model then has to determine which of the support set classes the query sample belongs to. We restrict the scope of our experiments to the 5- way 5-shot setting and baseline against closerlook (Chen et al. (019). a recent sfte-of-thhearr work, and protonet (Snell et al., 2017) another popular work from the domain. Our experiments follow from (Chen al., 2019) and implementations use code in (Chen 2019). We conduct experiments with backbones of varying depths Conv4, Conv6 and ResNet34, as presented in (Chen et al.
2019).
For our experiments, each model is trained on for 400 epochs and closerlook
protonet Snell et al. 2017)
(Chen et
al. 2019 for
200
epochs.
For Conv4
and Conv6 configura- tions on both closerlook and protonet, retrospection introduced without any warm-up pe- riod (zero epochs). For ResNet34, a warm-up period of 280 epochs for protonet and 150 epochs for closerlook is used. For all experiments, the retrospective update frequency is one epoch cach. The scaling parameter, K, is initialized at 4 and increased by 2% each retrospective update. For closerlook, we report comparative performance with baseline++,, the best performing variant. Results in Table Tablhihigghligh that training with the retrospective loss results in improved classification accuracy for all backbones configurations on both closerlook and protonet.
Model
protonet
closerlook
original
retrospective
original
retrospective
Conv4
75.26 1.05
77.42 + 1.25
79.03 +0.63
79.95 E 0.75 81.35 + 0.30
Conv6
80.71 +1.55
81.78 + 1.40
81.05 +0.55
ResNet34
88.75 +1.0
89.99 + 1.13
82.23 10.59
83.11 + 0.55
Table 4: Classification performance using retrospection for few-shot classification on CUB dataset
5 ANALYSIS
In this section, we presents ablation studies to analyse the impact of different hyperparameters batch size, optimizer, retrospective update frequency (F) and the scaling parameter k. The studies are conducted on the task of image classification on F-MNIST (Xiao et al. 2017) dataset using LeNet (Lecun et al. (201)) architecture. The default training configurations are used from Section 4.1. In all the studies, networks trained for each configuration are initialized with the same weights to ensure consistent comparison.
Impact of Batch Size We perform experiments analyse the invariance of the proposed retro- spection loss to batch size. For this study, we consider batch sizes 32, 64, 128. Results presented in Figure 5 highlight that the retrospection loss results in improved training performance, which is achieved much faster, across all the different batch sizes.
Impact of Optimizer We perform experiments to analyse the invariance of the proposed retro- spection loss to choice of optimizer. For this study, we use Adam (Kingma & Ba 2014) and SGD optimizers. The classification performance when using Adam and SGD (momentum=005) are re- ported in Figure [ (Row 2). The observed results highlight that the retrospective loss results in improved training performance across different optimizers.
We study the impact of different update frequen- cies (F) for the retrospective loss. We experiment with 150, 200, 250 steps. Results are presented in Figure (Row 1) with the best performance achieved using F = 250 steps. All configurations of the retrospection loss outperforms the configuration (in blue) trained without it. While experiments in the current work used randomized search to estimate update frequencies, retrospective mining can be an interesting future direction.
Choice of Retrospective Update Frequency, F.
Results in some experiments the original configuration do not match values (are higher lower) ported in (Chen al. 2019) even after using official code and same training contig. However, we ensure consistency of companson by using the same initializations for original and retrospective configurations
7

---

Under review as a conference paper at ICLR 2020
p mmmamm
WM
10303 20000 30000
18030
$208
1903
10:01
Figure 5: Classification performance using retrospection on LeNet(Lecun al. 2001 across differ- ent batch sizes on FMNIST (Xiao et al. (0177)
Choice of scaling mar- gin, x We conduct exper- iments using different ini- tial values of the loss scal- ing margin, k. For this analysis, the value of remains unchanged during the training. Results are presented in Figure 6 (Row I) with best performance achieved with 4. All configurations produce bet- ter performance than with =
peroesecciin aeto 50330
Reprospecii & 350 (OU0D
Performence with SGD
Performance with Adam
50
59
84 %0
Accuncy
&
3
CONCLUSION AND FUTURE WORK
Retrospection Retrospection
76
Retrospection Retrossection
Steps
Steps
In this work, we introduced retrospective loss that uti- lizes parameter states from previous training steps to condition weight updates and guide the network to- wards convergence. We conduct experiments across multiple tasks, input types and architectures to empiri- cally validate the effective- ness of the proposed loss. We perform ablation stud- ies to analyze its behaviour. As interesting future di- rection to explore the con- nection between retrospec- tion and momentum, conducted preliminary ex- periments on image classi-
FMNIST LeNet
FMNIST Resnet-20
00
86
Astuocc
86
82
Norrentum Momantur SGD With Retrosppction
80
With Momantum SGD With Retrosgrction
78 20020
80
S0000 100000
Slaps
Figure 6: Ablation studies of classification performance on the F- MNIST dataset: (Row 1) Varying loss scaling parameter, к (left), and retrospective update frequency (right); (Row 2) Using retrospection LeNet (Leeun et al., 2001) and SGD Adam optimizers; (Row 3) Using SGD, SGD +momentum, SGD retrospection for LeNet, ResNet-20 architectures
fication evaluate the impact of the retrospective loss on optimization. We contrast performance from three different configurations on image classification: (a) trained without retrospective loss (SGD); (b) trained without retrospective loss (SGD momentum); and (c) with retrospective loss (SGD). Results Figure 6 (Row highlight that introducing retrospection improves performance (blue vs green); morcover, using the retrospective loss improves convergence even when SGD is optimized without momentum.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Acgan-pytorch. URL
Degan-pytorehs
URL
dogan
Inception
score
code, 2018.
URL https///gitaub.co//bbaaaaaat
inceptioo-score www.onn--ororopooooooooo
Filippo Maria Bianchi, Daniele Grattarola, Cesare Alippi, and Lorenzo Livi. Graph neural networks with convolutional arma filters, 2019.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang. Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335, 2008.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11 (00-M))):109111333 109-1135, 2010.
Wei-Yu Yu Chen. http:///ttha...oo/////rrro//obot
URL, 2019.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. CoRR, abs/1904.04232, 2019. URL
abs/ abs/1904 04232
Sentic-Emotion Co. http://ggttaa.coommseernnooooocm URI,, 2019.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- fion (CVPR 06), volume 2. pp. 1735-1742. IEEE, 2006.
Song Han, Jeff Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, and William J. Dally. Dsd: Dense-sparse- dense training for deep neural networks. In 2016.
Hado V. Hasselt. Double qlearning. In J. D. Lafferty, C. L. Williams, J. Shawe-Taylor R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2613-2621. Curran Associates, Inc., 2010. URL httpp://page 1164-double-g--eezntie
nips. cc/paper/
pdf]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016
Geoffrey Hinton, li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Phuongtrang Nguyen, Tara Sainath, and Brian Kingsbury. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups Signal Processing Magazine, IEEE, 29:82-97, 11 2012. doi: 10.1109/MSP
2012.2205597.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. International Workshop on Similarity- Based Pattern Recognition, pp. 84 92. Springer, 2015.
Massed
Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, and Bo Li. Orthogonal weight normaliza- tion: Solution optimization over multiple dependent stiefel manifolds in deep neural networks. CoRR, 0bs/1709900079, 2017. httt::ttt///aatttt wttp://wwwwwooor//////a/aabass// aww.aa/9..0000 066799999970979 06079
Chi Jin, Praneeth Netrapalli, and Michael I. Jordan. Accelerated gradient descent escapes sad- dle points faster than gradient descent. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Proceedings of the 31st Conference On Learning Theory, volume 75 of Pro- ceedings of Machine Learning Research, pp. 1042-1085. PMLR, 06-09 Jul 2018. URL http: 7/proceed i ngs. ml www.srrssspee.... 75 8a. html

---

Under review as a conference paper at ICLR 2020
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 1, NIPS' 13, 2013.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In Proceedings of the 25th International Conference Neural Infor- mation Processing Systems- Volume 1, NIPS' 12, pp. 1097-1105, USA, 2012. Curran Associates aac. wttg/ciiaatiioooooooooooooo/oooo
Inc. URLhttpp: http::://allaamoorggggoe
fm? wtw.ffmmmm.mmm.....mmmmmm 2999257
Yann Lecun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. ciuuient-baseee learning applied to document recognition, pp. 306-351. IEEE Press, 2001.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. Dialoguernn: An attentive for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6818-6825, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks, 2016.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed represen- tations of words and phrases and their compositionality. In Proceedings of the 26th Interna- tional Conference on Neural Information Processing Systems Volume NIPS' 13, pp. 3111- 3119, USA, 2013. Curran Associates Inc. URL http:/ www.aaaaaaaammmmmmmmmmmmmmcooooa cwtatiiiiiinnoiiom cfm? id 2999959
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland. Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URI. http://dx doi www.orgg///.. http:///2......oor11111100333330 hww.0/////auuueee20
Yuval Netzer, Tiejie Wang. Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Lam M. Nguyen, Jie Liu, Katya Scheinberg, and Martin Takác. SARAH: A novel method for machine learning problems using stochastic recursive gradient. In Proceedings of the 34th In- ternational Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, pp. 2623-2621,222 2017.
Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5115-5124, USA, 2017. Curran Associates Inc. ISBN 978-1-5108-609644, URL httlhhtt::/dld acm. wwtp//ittiiii......o////.//.
cfm?1d=3295222.c 3295264
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans, 2016.
Nhan H. Pham, Lam M. Nguyen, Dzung T. Phan, and Quoe Tran-Dinh. Proxsarah: An efficient algorithmic framework for stochastic composite nonconvex optimization, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016.
10

---

Under review as a conference paper at ICLR 2020
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Pruceedings the IEEE conference on computer vision and pattern recognition, pp.
Björn Schuller, Michel Valster, Florian Eyben, Roddy Cowie, and Maja Pantic. Avec 2012: the con- tinuous audio/visual emotion challenge. In Proceedings of the 14th ACM international conference
on Multimodal interaction, pp. 449-456. ACM, 2012.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32Nd International Conference on International Con- ference Machine Learning Volume 37, ICML'11,, pp. 1889-1899 1897. JMLR.org, 2015. URL
http://dd al. acm, org/ wwt.ttiitttiinnnneoo www.w/fiimdddc.o
3045319
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, //////077003444 2017.
Prithviraj Sen, Galileo Mark Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad. Collective classificatton in network data. Al Magazine. 29(3):93-106, 2008. URL
http::/wwxx http://wwwwcc. cs. tit..ti edu/ httpm///p/////aaaaaa//aaaaaaa/a sen immm08 pdf
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of ini- tialization and momentum in deep learning. In Proceedings of the 30th International Confer- International Conference Machine Learning Volume 28, ICML' 13, pp. 111- III-1147. JMLR.org, 2013. URL http:/tt://// /dl acm. www.rr/aatttinn.oo////. www 0 3043064
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision, 2015.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD firds-200-2011 Dataset. Technical report, 2011.
Yong Wang, Xiao-Ming Wu, Qimai Li, Jiatao Gu, Wangmeng Xiang, Lei Zhang, and Victor O. K. Li. Large margin few-shot learning. CoRR, abs/1807.02872, 2018. URL http:
hwwp://www....o.o.ooooooooooooon brg/abs//807. (000b///0000072
Pete Warden.
Google Al Blog, I:URL, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.0944 2017.
Zhun Zhong. Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. ArXiv, abs/ 048.04 2017.

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL RESULTS ON GRAPH NEURAL NETWORKS
We study the impact of the retrospection loss on the task of semi-supervised node classification using CORA and CITESEER datasets (Sen ct al., 2008). For our experiments, we use two different models:
ARMA (Bianchi et al., 2019) (a recent state-of-the-ar method) and GCN (Kipf & WWelling 2016),
another popular variant. Our implementations follow from (Fey & Lenssen (019). Performance
is reported by averaging results over 30 experimental runs, each of which involves training the model for 100 epochs. For all experiments, the retrospective loss is introduced without any warm- up period (zero epochs). The hyperparameters, F' and K, used for training on both datasets (CORA and CITESEER) for each of the three networks are: GCN: F == 4: b) ARMA: F 1, K=3. Table 5 presents quantitative impact of using the retrospection loss.
Dataset
Contig original
GCN
ARMA
80.85 +0.53
78.53 1.5
CORA
retrospective
81.23 + 0.27
79.45 + 1.15
original
70.65 #0.93
63.63 #1.
CITESEER
retrospective
71.25 + 0.75
64.22 + 1.2
Table 5: Classification performance using retrospection loss on graph neural networks
B ROBUSTNESS OF RESULTS
To ensure consistency comparison, we reported performance in the main paper by initializing both retrospective and original experiments with the same weights. Now, for comprehensive analysis, we report experimental values for Image Classification, Speech Recognition and Text Classification tasks averaged over 10 runs. (Note that for few-shot learning, we already included this information in the main paper.) Table 6 Table 7 and Table 8 present the corresponding mean and standard deviation of the results for image, text and speech classification respectively. We also note that all the results in the submitted paper are in the same range as the mean + std in the results below, although these were separately performed showing the consistency.
Dataset
Network
Original
Retrospective
VGG-II
5.51 #0.08
4.75 E 0.09
SVHN
ResNet- 18
4.38 +0.09
4.01 + 0.07
ResNet-20
7.63 #0.04
6.85 + 0.06
F-MINST
LeNet
10.74 4.0.19
9.37 + 0.14
Table 6: Mean and Standard Deviation of error rates for Image Classification
IECOMAP
AVEC
Method
F1-Score
Accuracy
MSE
R (Pear Score)
Retrospective Original
64.40 + 0.4
64.97 + 0.5 62.70 0.77
0.1772 + 0.0006
0.332 E 0.008 0.317 +0.007
62.60 +0.9
0.99 +0.0005
Table Mean and Standard Deviation of results for Text Classification
Validation Set
Testing Set Retrospective
Model
Original
Retrospective
Original
LeNet
9.77 #0.05
9.60 - 0.03
10.26 E0.05
9.86 - 0.04 4.16 - 0.05
VGG-11
5.15 +0.08
4.37 + 0.04
5.03 #0.06
Table 8: Mean and Standard Deviation of error rates for Speech Recognition
C ABLATION STUDY: MOMENTUM
We also conducted an additional study to analyse the impact of choice of momentum values in the optimizer when using the retrospection loss in training. As in other ablation studies, we train LeNet for image classification on F-MNIST using SGD and experiment with different values of
12

---

Under review as a conference paper at ICLR 2020
the momentum parameter (0.5, 0.7, 0.9). The other parameter configurations remain the same as initially presented in Section 4.1 (Ir=0.1, batch_ .ize.)), As highlighted by results in Table g the retrospection loss is independent of momentum value since retrospective training results in better performance than original training (wo retrospection) for all the different momentum values.
Momentum = 0.5
Momentum = 0.9
Momentum = 0.7 Retrospective
Original
Retrospective
Original 10.05
Retrospective 9.06
Original
10.8
9.4
9.31
8.94
Table 9: Impact of choice of momentum value on retrospection loss
D ABLATION STUDY: WARM-UP PERIOD
As in other ablation studies, we train LeNet on the task of FMNIST (60,000 images) image clas-
sification for 70,000 iterations with batch_size = 32 using SGD (mom=0.9). The eπror rates with different warm-ups are presented in Table 10 We observed that on simpler datasets (like FMNIST) since networks start at a reasonable accuracy. retrospection is effective even when we introduce with very low warm-up period (Tw 0).
Network
Original In finity
Retrospective
L
1 0
10k
I,p
15k
1 20
LeNet
10.05
9.06
9.3
9.33
9.06
For simple datasets like FMNIST, the retrospection loss is effective even when intro- duce it with a low warm-up period. All different warm-up configurations improve performance over baseline.
Table 10:
Further, we observed that for tasks more complex datasets with bigger networks, it is best to introduce the retrospection loss after training the network for some epochs when the network has started to converge to some extent, empirically around 50-75% of the training epochs. While intro- ducing retrospection early also improves over bascline, later introduction of the retrospection loss further improves performance. Table TT presents results obtained when we trained ResNet-56 on the task of image classification using CIFAR-10 for 200 epochs. Here, when the network is trained without retrospection (the original config in the ResNet paper), we got is reported in ResNet paper). However, on using retrospection, performance improved to 6.78 when the warm-up period (!w) of 50 epochs was used and it further improved to 6.52 with a warm-up period of 150 epochs.
rate of 6.86 (6.97
Network
Original In finitty
Retrospective
I,=0
I,=50
-- 10:
150
ResNet-56
6,86 (6.97)
6.81
6.78
6.61
6.52
Table 11: All the different warm-up configurations improve over baseline trained without retrospec- tion loss. A higher warm-up period results in better performance when more complex datasets trained using large networks.
Next, we study the impact of the warm-up period on the task of image generation using GAN's. s. For the GAN experiments in the main paper, we used the retrospection loss with a warm-up period of zero epochs which resulted in improved performance over the baselines. We believe that since GANs are inherently unstable and do not train to fixed target, the warm-up period is unlikely to have a significant impact. Here, we present results from our study of the impact of different warm- up periods when training GANs. Figure Z plots the inception scores when DCGAN is trained on FMNIST dataset using retrospection loss with different warm-up periods (0, 10 and 30 epochs) and all other parameters are same as in Section 4.2. As highlighted by the results, retrospection trained with all the three warm-up configurations improves upon the bascline method (better max inception scores) trained without retrospection (blue). For training with the retrospection loss, the warm-up period does not have significant impact on overall performance with warm-up of 10 epochs (green) and 30 epochs (red) producing almost similar peak values.
13

---

Under review as a conference paper at ICLR 2020
DCGAN FUNIST
54
Inooybon Scoee
4.6
Withous
Warmup
Epocts
Figure 7: Training DCGAN on FMNIST dataset using retrospection loss with different warm-up periods.
E ABLATION STUDY: NORM
In this work, we seek to present the retrospection loss as a general concept that encourages using past parameter states as guidance during training to improve performance. Hence, we conduct an ablation study to analyse the effect of using different norms for the retrospection loss. Table T2 and Table 13 presents results of using retrospection loss with Ll-norm and L2-norm on the task of image classification. As highlighted by the results, both configurations of the retrospection loss (L.-norm, L2-norm) improved performance as compared to training without retrospection but using L Ll-morm resulted in the better performance.
Network
Original
Ll-norm
L2-norm
LeNet
10.8
9.4
9.7
ResNet-20
7.6
6.8
7.3
Table 12: Error rates using retrospection loss with different norms for Image Classification on F-MNIST dataset.
Network
Original
Ll-norm
L2-norm
ResNet-13 18
4.42
4.06
4.27
VGG-11
5.54
4.70
5.15
Table 13: Error rates on using retrospection loss with different norms for Image Classification on SVHN dataset.
Further, we even tried KL-divergence based formulation of the retrospection loss. Consider an input (I,, V:) and network gs parameterized by 8. Here go (x;); are the activations of the softmax layer and y/s is the ground-truth class embedding. For the loss, outprev = TT (x+) ; target = yt. For KL-divergence, we used the following formulation of the retrospective loss at training step T:
define: out ger (I;)) ;
Loss KL) = -1 * K L_div(oullarr: oulppee) + CrossEntrooyyyuuuaar largel)
(4)
In the above experiment SVHN, obtained 5,45 and 4.31 error rates for VGG-11 and ResNet- 18 respectively.
While all our variants, Ll-norm, L2-norm and KL-divergence, improved upon baselines, Ll-norm resulted in better performance across tasks, except in unconditional GANs, where I.2-nomm is used to apply the retrospective loss on the adversarial predictions of the generator (Sec 4.2). One hypothesis is that when the L1-norm is used, the gradient is simply a dimension-wise sign (+ve -ve), which provides a clearer direction to gradient updates, especially when training to fixed target embedding in predictive tasks.
14


Reviews:
[{"review_id": "rygRVjNAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the retrospective loss to optimize neural network training. The idea behind the retrospective loss is to add a penalization term between the current model to the model from a few iterations before. Extensive experimental results on a wide range of datasets are provided to show the effectiveness of the retrospective loss.\n\nThe retrospective loss is additionally controlled by two hyperparameters, the strength parameter K and the update frequency T_p. This loss, measured in L-1 norm, is added to the training objective. The geometric intuition of the added loss term is that this pushes the model away from the model at iteration T_p. The paper argues that this shrinks the parameter space of the loss function.\n\nOne of the concern regards the writing of the paper.\n- Algorithm 1 and Figure 6 look very blurry, which I think are both below the publication standard.\n- The introduction could be written to be more helpful, such as providing more context on why the obtained experimental results are important (e.g. getting state-of-the-art results on the datasets studied in the experiments)\n- The Related Work contrasts with previous work which is not clear because the precise contribution has not been stated at the point.\n\nMore detailed questions:\n- What are the standard deviations for the experimental results (as you reported in Table 4 but not in other experiments)?\n- I'm curious whether the use of L-1 norm is critical or not in the retrospective loss.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJePCn43YB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new loss function which adds to the training objective another term that pulls the current parameters of a neural network further away from the parameters at a previous time step.\nIntuitively, this aims to push the current parameters further to the local optimum.\nOn a variety of benchmarks, optimizing the proposed loss function achieves better results than just optimizing the training loss.\n\nThe paper is well written and easy to follow.  However, I am not entirely convinced about the intuition of the proposed method and I think further investigation are necessary.\nWhile the method is simple and general, it also seems to be rather heuristic and requires carefully chosen hyperparameters.\nHaving said that, the empirical evidence shows that the proposed loss function consistently improves performance.\nThe following details should be addressed further:\n\n- I am a bit confused by the definition of the loss function. In Equation 1 it seems that the term on the left represents the training objective. If that is correct than Equation 2 second case contains the training objective twice?\n\n- F in Section 3 after Equation 2 is not properly defined\n\n- Could it happen that the proposed loss function leads to divergence, for example if the parameter from a previous time step theta^Tp is close to the optimum theta_star?\n\n- What is the motivation to use the L1 norm? How does this choice affect convergence compared to let's L2 norm?\n\n- Section 4.1 typo in first paragraph: K instead of \\kappa\n\n- Section 4.1 the results would be more convincing if all networks were trained multiple times with a different random initialization and Table 1 would include the mean and std.\n\n- Why is no warm-up period used for the GAN experiments?\n\n- Section 4.3: why is \\kappa increase by 1% for the speech recognition experiments where as by 2% for all other experiments?\n\n- I suggest to increase the line width of all figures since they are somewhat hard to identify on a print version.\n\n- Why is the momentum set to 0.5 for SGD in the ablation study? Most frameworks use a default value of 0.9.\n\n- I would like to see the affect of the warm-up period to the performance in the ablation study.\n\n- How does the choice of learning rate schedule, such as for example cosine annealing, affect the loss function?\n\n\n\npost rebuttal\n------------------\n\nI thank the authors for clarifying my questions and providing additional experiments. I think that especially the additional ablation studies and reporting the mean and std of multiple trials make the contribution of the paper more convincing. Hence, I increased my score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a further regularizer, retrospection loss, for training neural networks, which leverages past parameter states.  The authors added several ablation studies and extra experiments during the rebuttal, which are helpful to show that their method is useful. However, this is still one of those papers that essentially proposes an additional heuristic to train deep news, which is helpful but not clearly motivated from a theoretical point of view (despite the intuitions). Yes, it provides improvements across tasks but these are all relatively small, and the method is more involved. Therefore, I am recommending rejection.
Decision: Reject