Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
This paper empirically examines an interesting relationship between mode connectivity and matching sparse subnetworks (lottery ticket hypothesis). 

By mode connectivity, the paper refers to a specific instance where the final trained SGD solutions are connected by a linear interpolation path without loss in test accuracy. When networks trained with SGD reliably find solutions which can be linearly interpolated without loss in test accuracy despite different data ordering,  the paper refers to these networks as ‘stable.’ 

Matching sparse subnetworks refer to subnetworks within a full dense network that matches the test accuracy of the full network when trained in isolation.  

The paper introduces a novel improvement on the existing iterative magnitude pruning (IMP) technique that is able to find matching subnetworks even after initialization by rewinding the weights. This allowed the authors to find matching subnetworks for deeper networks and in cases where it could not be done without some intervention in learning schedule. 

The paper then finds a relationship that only when the subnetworks become stable, the subnetworks become matching subnetworks.
———

Although finding a connection between two seemingly distinct phenomena is novel and interesting, I would recommend a weak reject for the following two reasons: 
1) The scope of the experiment is limited to a quite specific setting, 
2) there are unsupported strong claims which need to be clarified.
———

1)
In the abstract the paper claims that sparse subnetworks are matching subnetworks only when they are stable, but the results are shown in a limited setting only at a very high sparsity. 
They tested stability on the highest sparsity level at which there was evidence that matching subnetworks existed, but how would the result generalize to other sparsity levels?
With lower sparsity level (if weights are pruned less), is stability easier to achieve? 

The paper also focused on cases where matching subnetworks were found by IMP, but matching subnetworks can also be found by other pruning methods. 
As acknowledged in the limitations section, other relationships may exist between stability and matching subnetworks found by other pruning methods, or in different sparsity levels,
which could be quite different from this paper’s claim.

In order to address this concern, I think the paper needs to show how the same relationship might generalize to different sparsity levels, 
or alternatively modify the claim (to what it actually shows) and highlight the significance of the connection between matching subnetworks and stability in this highly sparse subnetwork regime.

2) 
As addressed above, in the Abstract and Introduction, the paper’s claims are very general about mode connectivity and sparsity, claiming in the sparse regime, “a subnetwork is matching if and only if it is stable.” However, the experiments only show it is true in a limited setting, focusing on specific pruning method and at a specific sparsity level.
Furthermore, the statement is contradicted in Footnote 7: “for the sparsity levels we studied on VGG (low), the IMP subnetwork is stable but does not quite qualify as matching“

There are also a few other areas where there are unsupported claims.

“Namely, whenever IMP finds a matching subnetwork, test error does not increase when linearly interpolating between duplicates, meaning the subnetwork is stable.” 
-> Stability was tested only at one specific sparsity level, and it is not obvious it would be stable at all lower sparsity levels where IMP found matching subnetworks.

“This result extends Nagarajan & Kolter’s observation about linear interpolation beyond MNIST to matching subnetworks found by IMP at initialization on our CIFAR10 networks” 
-> Nagarajan & Kolter’s observation about linear interpolation was on a completely different setup: using same duplicate network but training on disjoint subset of data, whereas in this paper it uses different subnetworks and trains it on full dataset with different data order. 

Related to the first issue, I think some of these stronger claims can be modified to describe what the experiments actually show. 
The relationship found between stability and matching subnetworks in the high sparsity regime is a valuable insight that I believe should be conveyed correctly in this paper.

———

I also have some minor clarification question and suggestions for improvement. 

How was the sparsity level (30%) of Resnet-50 and Inception-v3 chosen in Table 1? (which was later used in Figure 5)

— In Figure 3 and 5, the y-axis “Stability(%)” is unclear and not explained how this is computed. I first thought higher amount of stability(%) was good but it doesn't seem to be true.

— The ordering of methods for plots could be more consistent. In some figures VGG-19 come first and then Resnet-20 while for others it was the other way around, which was confusing to read. (Also same for Resnet-50 and Inception-v3)

— There are same lines in multiple graphs, but the labeling is inconsistent, potentially confusing readers:
Figure 1: (Original Init, Standard) is the same as Figure 4: (Reset), 
and Figure 1: (Random Reinit, Standard) is the same as Figure 4: (Reset, Random Reinit)
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper empirically presents a very interesting connection between two also very interesting phenomena (mode connectivity and lottery ticket hypothesis), while removing a previous limitation of the lottery ticket hypothesis on larger networks. through a good amount of experiments, the authors empirically showed these two phenomena co-occur together (i.e. matching networks are stable) and have positive correlation (i.e. the more “matching” the network the more “stable”), under different network architectures and datasets.

Though it is unclear from the paper what are the immediate / straightforward applications, the findings do present interesting contributions. Several things I found that can further improve this paper.

First, this paper lacks a structured literature review. It is suggested that the findings may provide insights to our understanding of how SGD works in neural network. Laying some proper background knowledge on this area is needed in the literature review. 

There are several experiments that I’m curious to see. Though I must say the existing amount of experiments sufficiently validates the existence of the connection authors put forth and hence not required.

a) Provide some metrics on how “far” are the two final weights upon which mode connectivity (or stability) is explored. For the sake of comparison, distance between the initial weights and final weights can be added. 

b) First off, the introduction mentions connectivity was previously observed using “disjoint subsets” of data, whereas later in the paper only different orders of the same data are explored. I wonder if this is a typo. Regardless, exploring if the findings still apply on disjoint data and/or varying amount of data, besides different data orders, is helpful. 

c) Does the full network have the property of mode connectivity (when trained using different data orders), or this only occurs under sparsity.

Lastly, the writing of the paper doesn’t interfere with understanding, but can definitely use more work. Abstract can be tightened. Several typos throughout the paper:
- in the abstract, "with the no change" -> remove "the"
- bottom of page 1, subt -> sub
- second bullet point under "contributions": remove ":"?
- page 3, paragraph starting with "stability": "the increase in worst-case increase" -> "the worst-case increase"?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper works on empirically demonstrating the connection between model connectivity and the lottery ticket hypothesis, which are individually explored in the literature. Here the model connectivity refers to the fact that SGD produces different solutions (from the randomness, such as data ordering) that are connected through model parameter transition paths of approximately equal loss/accuracy. The lottery ticket hypothesis tells that there exist sparse subnetworks of the corresponding full dense network which can attain as strong loss / accuracy as the full dense network. 

As the primary contribution, the authors demonstrated that the following two observations often emerge together: 1) A sparse subnetwork can match the performance of the corresponding full dense network;  2) Running SGD on this sparse subnetwork produces solutions which are connected via a linear model parameter transition path of similar loss/accuracy; this is observed across both small tasks (using CIFAR10) and ImageNet-level tasks. Another contribution I can see besides the primary one is that the lottery ticket hypothesis still holds on large tasks, which is against the conventional wisdom demonstrated in recent papers (e.g. Gale et al., 2019); the authors show that it needs to rewind to weights after a short period of training instead of rewinding to the initialized weight in Iterative Magnitude Pruning to produce the "lottery ticket" in large tasks (such as CNN for ImageNet). 

I think the primary contribution on the connection between model connectivity and lottery ticket hypothesis is an interesting observation, but the content are poorly presented for me fully appreciate the importance and practical implications of this work. Thus I give weak reject. The major concerns and questions are as the following:

1. From the paper, I don't understand why the connection between model connectivity and lottery ticket hypothesis is an important one to reveal. Is it important because it implies some practical approaches / heuristics to figure out performant sparse subnetworks? Is it intrinsically interesting because it validates some hypothesis in the training dynamics of SGD? These are not clear to me.

2. I think the current presentation of the content is only limited to the empirical demonstration. And I can not extract useful intuitions/messages from the demonstration here on why this happens. These message should provide intuitions on why this connection exists. E.g. These message can be extracted from some SGD on some simple (toy) non-convex models with multiple local minimum regions.  

Minor comments for improving the paper:

1. At the end of line 1 in algorithm one, it is not clear what 1^|W0| means.

2. The terms in figure legend needs to be properly defined to enable clear reading. Currently words such as "reset" is not mentioned in the text but appears in the legend of figure 4 and etc. 










review_title: Official Blind Review #3

Meta-review: This paper investigates theories related to networks sparsification, related to mode connectivity and the so-called lottery ticket hypothesis.  The paper is interesting and has merit, but on balance I find the contributions not sufficiently clear to warrant acceptance.  The authors made substantial changes to the paper which are admirable and which bring it to borderline status. 

Decision: Reject