Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper presents the retrospective loss to optimize neural network training. The idea behind the retrospective loss is to add a penalization term between the current model to the model from a few iterations before. Extensive experimental results on a wide range of datasets are provided to show the effectiveness of the retrospective loss.

The retrospective loss is additionally controlled by two hyperparameters, the strength parameter K and the update frequency T_p. This loss, measured in L-1 norm, is added to the training objective. The geometric intuition of the added loss term is that this pushes the model away from the model at iteration T_p. The paper argues that this shrinks the parameter space of the loss function.

One of the concern regards the writing of the paper.
- Algorithm 1 and Figure 6 look very blurry, which I think are both below the publication standard.
- The introduction could be written to be more helpful, such as providing more context on why the obtained experimental results are important (e.g. getting state-of-the-art results on the datasets studied in the experiments)
- The Related Work contrasts with previous work which is not clear because the precise contribution has not been stated at the point.

More detailed questions:
- What are the standard deviations for the experimental results (as you reported in Table 4 but not in other experiments)?
- I'm curious whether the use of L-1 norm is critical or not in the retrospective loss.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper proposes a new loss function which adds to the training objective another term that pulls the current parameters of a neural network further away from the parameters at a previous time step.
Intuitively, this aims to push the current parameters further to the local optimum.
On a variety of benchmarks, optimizing the proposed loss function achieves better results than just optimizing the training loss.

The paper is well written and easy to follow.  However, I am not entirely convinced about the intuition of the proposed method and I think further investigation are necessary.
While the method is simple and general, it also seems to be rather heuristic and requires carefully chosen hyperparameters.
Having said that, the empirical evidence shows that the proposed loss function consistently improves performance.
The following details should be addressed further:

- I am a bit confused by the definition of the loss function. In Equation 1 it seems that the term on the left represents the training objective. If that is correct than Equation 2 second case contains the training objective twice?

- F in Section 3 after Equation 2 is not properly defined

- Could it happen that the proposed loss function leads to divergence, for example if the parameter from a previous time step theta^Tp is close to the optimum theta_star?

- What is the motivation to use the L1 norm? How does this choice affect convergence compared to let's L2 norm?

- Section 4.1 typo in first paragraph: K instead of \kappa

- Section 4.1 the results would be more convincing if all networks were trained multiple times with a different random initialization and Table 1 would include the mean and std.

- Why is no warm-up period used for the GAN experiments?

- Section 4.3: why is \kappa increase by 1% for the speech recognition experiments where as by 2% for all other experiments?

- I suggest to increase the line width of all figures since they are somewhat hard to identify on a print version.

- Why is the momentum set to 0.5 for SGD in the ablation study? Most frameworks use a default value of 0.9.

- I would like to see the affect of the warm-up period to the performance in the ablation study.

- How does the choice of learning rate schedule, such as for example cosine annealing, affect the loss function?



post rebuttal
------------------

I thank the authors for clarifying my questions and providing additional experiments. I think that especially the additional ablation studies and reporting the mean and std of multiple trials make the contribution of the paper more convincing. Hence, I increased my score.
review_title: Official Blind Review #1

Meta-review: This paper introduces a further regularizer, retrospection loss, for training neural networks, which leverages past parameter states.  The authors added several ablation studies and extra experiments during the rebuttal, which are helpful to show that their method is useful. However, this is still one of those papers that essentially proposes an additional heuristic to train deep news, which is helpful but not clearly motivated from a theoretical point of view (despite the intuitions). Yes, it provides improvements across tasks but these are all relatively small, and the method is more involved. Therefore, I am recommending rejection.
Decision: Reject