Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Pros
+ This work provided a good summary of observations and network properties that worth studying during the early stage of network’s training.
+ The authors conducted extensive and detailed experiments to study the statistics of weights and their gradients. Ablation studies also considered network’s accuracy under perturbation of weight signs, weight shuffling, and different standard deviations.
+ The authors also verified the effectiveness of weak labels used in self-supervised learning.

Cons
- The work itself did not propose any new network properties or any new metric to measure. Most experiments are designed for previous observations and mostly for verification purpose. I am concern about the core motivation of this work, like to identify or solve any new problems, in addition to experimentally verify the observations during network’s training.
- The conclusion in this work is still very empirical: it remains uncertain whether in other vision tasks and with more complex networks (e.g. multi-branch network) these conclusions would hold.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper aims at exploring the properties of neural network training during the early phase. By some studies on the lottery ticket hypothesis, something important happens during the early phase of training so rewinding the network should go to these early phases instead of the initial phase. So, what is important during training? The paper explores this problem from four aspects through empirical studies: 
1. By showing the various statistics through different training iterations, especially the gradient magnitude, the training phase is divided into three parts, and each part has different behaviors.
2. The paper explores what is more important for the early phase of training: signs of the weights or magnitude of the weights.
3. The paper explores what is more important for the early phase of training if we redistribute the weights, signs or magnitudes.
4. The paper explores how training depends on data. Giving weak information such as self-supervised information may work but giving wrong information such as random labels will hurt the performance.

This paper studies the properties of deep neural networks. Through a series of carefully designed experiments, the paper shows what is important for the weights (magnitude or signs), and what is important for the data (weak information or random information). It enables a deep understanding of neural networks and may motivate new neural network compression methods to be proposed. Generally, the paper is well-written, although some parts can be improved. I would vote for acceptance of the paper.

Some questions/suggestions to make the paper clearer:
1. It is better to have a table summarizing various results of the paper, to give readers an overall impression after going through so many detailed experimental results.
2. The results of Fig. 4 and Fig.6 can be inconsistent. In Figure 4, it says that signs are less important than magnitudes. In Figure 6, it says that signs are more important than magnitudes if shuffling filters and keep signs. Any explanation on the inconsistency?
3. There is no explanation of the “weight trace” in Figure 3.
4. It is not clear what is the difference between “Init” and “Final” in “L2 Dist” and “Cosine Similarity” in Figure 3. 

Finally,  it is said in the “call for papers”, “Reviewers will be instructed to apply a higher standard to papers in excess of 8 pages”. This paper is nearly nine pages, and higher standards should be applied. 

--------------------------------------
I am satisfied with the rebuttal. Since the paper is now within the 8 pages limit, I would not apply a high standard and increase my score. 

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Overview:

This paper is dedicated to examining the changes that networks undergo during the early phase of the network training. The author conducts extensive measurements of the network state and its updates during the early iterations of training. Based on the observations, they find that: i) deep network is not robust to reinitialization with random weights, but maintained signs; ii) the distribution is highly non-i.i.d after the early phase of network training. This is why permuting weight dramatically harms performance. iii) the changes in the supervised networks are label-agnostic. The author claims these results can play an important role in explaining the network changes in the initial critical period.

Strength Bullets:

1. The paper performs exhaustive experiments in the early phase of network training. And it has some interesting implications for lottery tickets. i.e. to some extent, then signs from the rewinding iteration are sufficient to recover the damage caused by permutation.
2. I am very like the analysis of whether weight distributions are i.i.d.. The results in the paper are aligned with my intuition that the weights in the early stage are highly dependent and they share some similarities in the distribution level. And weight in different training stages supposes in a different distribution. Networks aren't robust to these permutations. They also show that the perturbations can be roughly be approximated by adding Gaussian noise to network weights.s
3. The author offers detailed results to analysis the data-dependence of the early phase of network training. The experiment organization is complete and convincing.

Weakness Bullets:

1. Although the paper gives extensive measurements and observations about the early phase about network training, it doesn't provide useful and efficient guidance for the lottery tickets hypothesis. In other words, the observation is interesting but the novelty is arguable. How can we use these "implications" to find better tickets? I mean it should be efficient. It is not reasonable cost more the find the best weight magnitude point or sign point for a better initialization.
2. [Minor] The legend in figure 9 is missing. I guess it may be the same as the upper left. But it is still confusing and hard to read.

Recommendation:

I think this paper provides plenty of insightful observations. I still hope the author can solve the above weakness bullets. This is a weak accept.
review_title: Official Blind Review #3

Meta-review: This paper studies numerous ways in which the statistics of network weights evolve during network training.  Reviewers are not entirely sure what conclusions to make from these studies, and training dynamics can be strongly impacted by arbitrary choices made in the training process.  Despite these issues, the reviewers think the observed results are interesting enough to clear the bar for publication.
Decision: Accept (Poster)