Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper explores how plateau occur in the loss curves of neural networks. Specifically, it shows that a significantly different last-layer bias may result in a long plateau in the loss and error curves. It also analyzes how gradient dynamics generates such bias gap during training with a simplified classification model. Numerical experiments support the proposed theory.
strength_and_weaknesses: Overall I enjoy reading this paper. The paper proposes a novel view of the MLI property, which is related to the last-layer biases. The observation is not previously explained, and could be a good supplementary of the existing theory of MLI. I personally like the implicit idea presented by the authors that a seemingly common optimization property is actually "not common", but dependent on specific factors instead. This might explain why it is so difficult to obtain a common but useful theory for neural nets.

I have some questions regarding the results in the paper and would like to learn the authors' opinions.

1. The paper's setting considers a classification problem with a cross-entropy loss. Do authors expect similar phenomenon happens in other problems such as regression problems, or with a different type of loss function?

2. It seems that by Theorem 1, we can only predict the length of the plateau after we train the network and obtain the final bias gap. Is there a way of knowing the length of the plateau before training (if the initialization is known)? 

3. In the experiments, will changing the initialization affect the sequence of the class learned during training? For example, assume that in one experiment, class 1 is learned first and then class 2 is learned. By theory, the bias term in class 1 is smaller than class 2. Is it possible that class 2 turns to be learned first when we change a different initialization? In this case, the bias term in class 2 is smaller than class 1, and thus the two experiments result in very different outputs (although they may have similar loss or even similar training dynamics). 
clarity,_quality,_novelty_and_reproducibility: As previously mentioned, the paper provides new insights into the MLI property and I believe it contains enough novelty.

The paper is clearly written. I did not fully check the proof but it looks technically sound to me.
summary_of_the_review: In summary, I think the paper is in the right direction and presents new ideas. I recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper analyses the shape of the loss curve along the linear interpolation from initialization to minimum for neural networks. In particular, in analyses under which circumstances the loss curve is monotonic and when it exhibits a plateau. The paper shows theoretically that for fully-connected neural networks the existence of the plateau can be linked to the gap in last-layer biases and the network's initialization, independent of the difficulty of the optimization problem. Moreover, the paper analyzes the training dynamics wrt. this bias gap. These theoretical results are supported by empirical evidence on MNIST and CIFAR100.
strength_and_weaknesses: Strength:
- strong theoretical contribution
- empirical support for theoretical results
- insightful conclusion that the shape of the loss curve along the linear interpolation is not indicative of the success of training or the simplicity of the optimization problem

Weaknesses:
- the results on training dynamics hold only for a two-layer FCNN with specific activation
- Thm .1 does not apply to the r-homogeneous-weight network used in the following section, since it requires a neural network with at least 3 layers
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, the proofs are presented understandably, despite their complexity. The experiments are described well and code is provided, so that results are reproducible. The theoretical results and their empirical support are novel.
summary_of_the_review: The paper presents novel theoretical results on the loss curve of the linear interpolation between initialization and minimum. These results indicate that the hardness of the learning problem plays little role in the existence of a plateau. Instead, the existence of a plateau can be linked to the bias terms, which in term can be linked to initialization and network depth. This is a valuable insight. The examples in the appendix that underpin this insight are great.

The theoretical analysis is limited to specific types of networks (fully-connected feed-forward networks with at least three layers and only an output bias for Sec 3 and two-layer fully-connected feed-forward networks with a specific polynomial activation function for Sec 4), but the empirical results are performed on standard DNNs (including VGG16). It would be great if the results in Sec. 3 and 4 could be linked, though, because now the assumptions on the network are mutually exclusive. 

Nonetheless, the paper is insightful and relevant to the community, technically sound, and well-written.

Detailed comments:
- In Thm. 2: network as defined in Section 2 -> should be section 4
- r is used both as number of layers in Sec 3 and as parameter of the activation function in Sec 4, this is a it confusing
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper considers the problem of monotonic linear interpolation (MLI) in deep neural networks. That is, one connects the initialization of the neural network with the fully trained neural network through a line. Then one considers how the loss function behaves when one is interpolation between the weights at initialization and the weights at the fully trained neural networks. What has been observed is that in modern architectures, the loss stays first for some time at a plateau and then suddenly decreases as one is moving from the initialization to the fully trained neural network.

This paper argues that the reason for the reason for this phenomenon is that the biases in the last layer scale linearly with the interpolation parameter $\theta$, whereas all the other parameter scale with $\theta^r$, where $r$ denotes the number of layers. Importantly, they claim that there is one bias in the last layer which is much larger than all the others.

The paper makes the following contribution which corroborates these claims.
1. They prove that if there is bias which is much larger than the others, then the phenomenon described above appears.
2. They prove that in a simple toy setting there is indeed one bias which is larger than all the others (and they can subsequently use the results to strengthen the results in point 1 for this simple toy model.)
3. They provide numerical experiments which corroborate 1. and 2.
strength_and_weaknesses: Strenghts: The problem in this paper is well-motivated and the paper is very nice to read. The explanations are convincing both in theory and practice. I think that this is a paper which should be of interest to the ICLR community.

Weaknesses: The toy model is a bit simplistic but I think that this is OK since our understanding of neural network training is still very limited.

Typos and minor issues:
1. p. 4: "$V_{\max}$for all" (blank space missing)
2. p. 5, third line: "where (the) weight vector"
3. p. 5: The work of Ge et al., 2021 is about tensor decomposition. Consider explaining how it connects to the setting in this paper and, in particular, to the chosen activation function.
4. p. 5, statement of Theorem 2: "Suppose the neural network, dataset and optimization procedure are as defined in Section 2." Are these things really all defined in Section 2? I cannot find them there. Please consider clarifying.
5. p. 5, Theorem 2, statements 1 and 2: By using the notation $\Omega (1)$, do you mean that there is a small constant $c>0$ such that these statements hold? Otherwise please clarify. 
6. p. 7: In the second statement of Theorem 3, consider replacing "monotonically decreasing" with "strictly monotonically decreasing"

Questions:
1. p. 4, Theorem 1: It seems to me that Theorem 1 makes only Assumption 1 regarding the (fully) trained neural network.
Is that really true? Do you not also need an assumption which says that the loss function is small at the end? Or am I missing something? Otherwise please consider adding the missing assumptions to the theorem.
2. p. 4: "Empirically, $\frac{\Delta_{\min}}{2 V^r_{\max}}$... does not change much when depth increases." Do you mean that you empirically notice that increases depth increases $\Delta_{\min}$?
3. p. 5, beginning of page: Why do you need $r\ge 3$? Why would $r=2$ not suffice?
4. Suppose all biases would be equal after the network has been fully trained. Why can't you then prove that there is a plateau? There would still be the different scaling, i.e. $\theta$ vs. $\theta^r$. Consider adding this explanation to the main text.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is in general well-written with clear notation. The problem is well-explained and motivated and the story line is very clear.

Quality: The theoretical experiments consider a rather simple setting. However, it is still quite challenging to derive meaningful theoretical results in those simplified setting. The theoretical predictions from those experiments seem to transfer to more realistic settings, as the carefully conducted experiments show.

Novelty: To the best of my knowledge, the results in this paper are novel and go beyond existing literature.

Reproducibility: I could not have a close look into the proofs, but the proof strategy which was suggested in the main text seems to be  very reasonable to me.

summary_of_the_review: I think that this is a good paper, which has good theoretical and numerical experiments and which gives insight into an interesting question regarding the behaviour. of neural network training.
For these reasons, I think that this is a good paper which should be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows that monotonic linear interpolations (MLIs), commonly used to study the loss landscapes of neural networks (NNs), may be misleading if used to estimate problem difficulty. Specifically, it is shown (both theoretically and empirically) that the plateaus on the MLI plots, attributed to problem “hardness”, are in fact related to the bias training dynamics. The authors argue that the flatness of the MLI curve can be easily manipulated by tweaking hyper parameters such as initialisation range and network depth. The authors also make an interesting observation that the class that happens to be learned last by the NN ends up with the strongest (largest) bias.
strength_and_weaknesses: Strengths:
* The paper challenges the naive use of a crude estimation to make far-reaching conclusions. I strongly resonate with the message: do not over-interpret, aim to fully understand first.
* The results are supported by both theory and practice. The experiments are realistic and do not over-simplify the application domain.

Weaknesses:
* The overall presentation flow is a bit incoherent, with numerous past and future references to sections and figures. I found myself furiously scrolling up and down in an attempt to follow the plot. 
* The authors “attribute the plateau to simple reasons as the bias terms, the network initialization scale and the network depth”. Specifically for the last two, it is shown that smaller initialisation and larger depth lead to bigger plateaus. But is it really that unfair to correlate these with problem difficulty? Intuitively, weights initialised in a smaller region would yield weaker gradients in a deep network, which may slow down the learning = make the problem harder. Similarly, deeper networks experience vanishing gradients, which once again increases the difficulty of learning. Stating these as “simple reasons” sounds like an oversimplification.
* The paper contains a few small grammatical and formatting mistakes, please refer to the following non-exhaustive list of suggested corrections:

		Page 2: “Under this model we can show” - the sentence ends abruptly with no punctuation, consider adding “…that:” or otherwise rephrasing.
		Page 4: “achieve loss approximately” -> achieve a loss of approximately…
		Page 5: “detailed proof in Section C” - do you mean Appendix C?
		Appendix A: “We give two examples that illustrates” -> that illustrate
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear, although the flow of the narrative can be improved (e.g. placing of figures in relation to the text). The theoretical work is thorough, and the experiments are well put together. In my opinion, this work is of high quality.  The observation (relating plateaus to biases) is novel, however, it remains to be seen whether the discovery of this relationship will be impactful. The correlation between network depth, initialisation range and the plateaus seems to support the standard interpretation of MLI that the authors set out to debunk.
summary_of_the_review: Overall, I think this is a good paper that may yield interesting discussions in the community. Developing theory of NN learning is crucial for the field, and this work adds to the overall understanding, even if in a humble way.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the monotonic linear interpolation (MLI), which characterizes a proerty of the loss between a random initialization and the point GD converges to. While MLI was studied in a few works before, the work shows a new result  that the plateau of MLI is not necessiary related to the hardness of optimization, but related to the last-layer bais term theoretically and experimentally. This paper provides new understanding about MLI and more generally, the optimization landscape of neural networks. It is clearly written.  I recommend acceptance.
justification_for_why_not_higher_score: The phenomenon found is not significant enough to be a spotlight paper.
justification_for_why_not_lower_score: This paper provides new understanding about MLI and more generally, the optimization landscape of neural networks.
Decision: Accept: poster