Venue: UAI 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This paper focuses on learning sparse neural networks. Instead of learning a sparse network from scratch, the paper proposes to learn multiple low-loss sparse subnetworks (cheap tickets) and then superpose them together to get one stronger sparse network while maintaining the sparse degree. Experiments show the effectiveness on different datasets with different baseline models. 
Q3 Main strengths: + The paper is well motivated and well written. It is easy to follow. The claim is clear.
+ Extensive experiments are conducted to show the performances of the proposed method and study the effectiveness of different subcomponents, such as the averaging method for superposing operation. 

Q4 Main weakness: - More detailed discussions are absent. 
- It is still unclear why more cheap tickets cause worse performance shown in Figure 5. Why “creating too many cheap tickets will reduce the time of the normal sparse training phase”? What if enforcing not “reducing the time of the normal sparse training phase”? Could it boost the performance with more cheap tickets? This observation conflicts with some basic assumptions. 
- How is the variance of the multiple cheap tickets? In case there may be high variance, does it influence the accuracy of the averaging-based method? 

Q5 Detailed comments to the authors: I put the details with main weakness. 
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: The paper is well motivated and well written. It is easy to follow. Abundant experiments prove the main claims.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: The paper proposes an approach to enable more efficient training of sparse neural networks. The main idea is to average parameters of multiple instances of trained sparse networks, which they call cheap tickets or sub tickets or sub networks. Thus, instead of spending N iterations to train one sparse network until convergence, we spend the same amount of compute to train K sparse networks (each for N/K iterations), but achieve superior performance by combining their parameters. 


Q3 Main strengths: - The core of this idea of using an approach to have the advantage of an ensemble method, without the need to keep all subnetworks is interesting and can be very useful in practical scenarios.  Averaging the parameters of these cheap tickets, instead of using a standard ensemble method of averaging activations/outputs, helps to avoid the cost of maintaining and running multiple networks during inference.
- The paper touches upon an interesting challenge of how/when to collect the cheap tickets.
- The paper is well organised and easy/clear to read.
- The results are properly reported, over different trials and with respect to different metrics (compute time, accuracy, confidence calibration). 
- The paper compares different strategies for averaging the parameters of the subnetworks and provide some intuitions about why/how they could lead to different outcomes.
Q4 Main weakness: - Lack of evidence and discussion about the similarity or dissimilarity of the cheap tickets collected with the proposed approach.
- Lack of discussion, theoretical or intuitive, that why averaging the networks at parameter level, in this particular case, should lead to well behaving models.
- While the main novelty of the paper, could be in the strategy used to collect the cheap tickets, this particular aspect of the proposed method is not compared to any baselines.
- The relation with prior work and the novelty of the contributions is not well established. 
Q5 Detailed comments to the authors: 
The methods consists of two main steps:
1. Training and collecting the subnetworks:  To do so, through the coarse of training sparse NNs (with) they apply a specific cyclic learning rate schedule is applied together with a parameter exploration technique. 
2. Averaging parameters of the subnetworks: since we are in the sparse training regime, each subnetwork could converge to a different sparsity patterns, and the authors examine how different averaging strategies  impacts the results.


My take is, the main contribution/novelty of the paper is in the way they collect the subnetworks. Unfortunately, in the experiments, the proposed strategy for doing this is not compared with any baseline. E.g., using different learning rate scheduling, or simply sampling subnetworks randomly during a standard coarse of training.

Moreover, my main concern about the second step: We know from prior work, e.g., that computing convex combination of parameters of instances of NNs can lead to better/more robust models. However, this happens under specific conditions. E.g., if instances are close enough and in the same basin in the loss landscape.  I think it’s worth it, if the authors can intuitively or theoretically explain why this approach should work in this case.  Basically, my understanding from the paper is that the strategy for collecting the sub networks is designed such that these instances belong to different basins in the loss landscape. Hence, it’s not clear to me why we should expect the average of the parameters of these models to be a well behaving model? Taking this into account, and the fact that the results indicate some improvements, I am guessing that the assumption that these cheap tickets belong to different basins in the loss landscape, does not hold. Maybe the authors can analyse this, and provide some evidence for whether this is actually the case or not.

Additionally, while it’s really nice that the paper compares different strategies for averaging the parameters of the subnetworks. I wonder how the results would compare to the ensemble output of these subnetworks (averaging outputs/activations). On this note, to avoid dealing with issues related to averaging parameters of the networks, maybe distillation ( distilling from all sub-networks into one), can be used as an alternative strategy.


Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: I have concerns about both novelty and correctness of the approach.
1. Whether averaging the parameters of subnetworks is a meaningful operation in the particular setup of this paper? The paper can have a much stronger point, if this is justified and the improvements in the results and discussion about why this should lead to any improvements are consistent. 
2. The experiments don't support the fact that this particular strategy of collecting cheap tickets is leading to better results.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: In this paper the authors provide a method for sparse training that efficiently finds very sparse sub-networks which still performs well. These sub-networks, also called cheap tickets, are then combined to create a better performing model that still retains the sparsity benefits. 

The authors then provide empirical evidence that their approach performs well on several classification tasks.
Q3 Main strengths: The provided approach manages to sparsely train neural networks keeping the computational costs low while providing highly accurate models.

The different approaches to merge the cheap tickets are simple, interesting and effective. The empirical results confirm that they work well without increasing the training costs. 

The author compares to other related-works showing improvements across the board.
Q4 Main weakness: The authors focus only on classification tasks, raising the question of how general this sparse training approach is. Would it also perform well on other architectures, e.g., GANs? VAEs?, etc.

Considering tables 1 and 2, we see that the values are although marginally better still impressive given the sparsity. However, are they statistically significant? This is not mentioned. This is important as some of the other approaches have a similar computational complexity.
Q5 Detailed comments to the authors: In general I find the paper interesting and useful. It is also easy to read and technically sound. 

The issues mentioned in Q4 are still relevant, especially the issues around the significance which would make the reader more confident about the results.

There are other minor questions that could also be interesting. Do you use regularization? I do not remember any discussion about that, nor whether you use dropout in your approaches. Would that help or hurt in this highly sparse scenario? A small comment on that or references would be helpful for reproducibility too.

I found the examination of the number of tickets, the cyclical length and the comparisons in terms of ECE and NLL interesting and carefully thought out.

Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: In general I find this is a good paper, covering the different aspects presented in the approach very carefully and easy to read. My only serious comments would be regarding the significance of your results.
review_title: null

Meta-review: Meta Review: The paper proposes a mechanism for learning sparse networks by combining multiple sub-networks.
Overall the reviewers appreciated the contribution and empirical results.
In the discussion, the issue of comparison to SWA came up. As noted there, the authors are asked to more clearly highlight the difference between their method and an SWA approach that simply learns sparse models and averages those.
Is the key difference that their method can allow some of the averaged models to be non-sprase?
A more detailed explanation and ablation would be very helpful for demonstrating the advantage of the method.

Decision: Accept (Poster)