Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper provides empirical evidence that sparse networks obtained using the lottery ticket hypothesis are also effective recognisers in settings where small datasets are available. Extensive experiments are conducted on multiple datasets and several interesting findings are reported, e.g., sparse networks in low data regimes are robust to synthetic noise, density is preserved in initial layers etc.
strengths_and_weaknesses: Strengths

The paper is well written.

Interesting findings are reported.

Extensive studies are conducted.


Weaknesses

Most of the experiments and findings are based on the CIFAR datasets. Why not using a more challenging dataset, like ImageNet? It would make the results more convincing.

Novelty is limited, existing methods are used, and extensive experiments are run (and the findings are definitely interesting) but it would be desirable if some novelty was included.

questions: 
See comments/questions in weakness section.
limitations: Does not apply to this paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper investigates whether sparse winning tickets (obtained using Iterative Magnitude Pruning) improve image classification performance in low-data tasks (defined as 50-100 samples per class). The paper shows that sparse winning tickets outperform the full network by 10-15%, *but* this requires training using data augmentation. Without data augmentation, the winning tickets perform similarly to the dense network.

The paper’s contribution is empirical: it shows that sparse winning improves performance on tiny datasets.
strengths_and_weaknesses: # Strengths
- The paper provides a **good empirical contribution** to the literature on sparse networks and learning from small datasets. The results raise further questions that the community might continue investigating. I find the paper to be **relevant to the conference**.
- The paper contains **extensive experiments**, investigating winning tickets in different low-data scenarios (standard classification, synthetic noise, synthetic transforms, domain shifts, highly imbalanced datasets). They show that winning tickets consistently improve performance when the network is trained from scratch.

# Weaknesses
- The paper’s core weakness is having a **very shallow analysis of the experimental results**. The paper has extensive experiments with useful results, but I found the provided analysis of the results to be weak. The authors provide little reasoning/explanations for their findings. Contribution 4 (line 60) is not well supported by the provided experiments and written analysis. The discussion section shows additional experiments and only briefly discusses the results without providing answers to any *why does this happen?* questions.
	- Section 5.1, and Figure 5, don’t answer why the network capacity and connectivity are important.
	- Section 5.2
		- The paper mentions: “Retaining more weights in the initial layers appears to allow the sparse models to keep the filters for detecting primitive features such as edges”. What does this result imply? Why do sparse winning tickets preserve the filters? Why don’t denser winning tickets preserve the filters? One hypothesis is that sparse tickets (on low-data) are simple functions (because they keep few parameters on the upper layers) of very well-extracted low-level features (because they keep most parameters on the first layers). Please discuss the findings if you want to keep Figure 7 in the discussion section.
		- “the identified sparse networks at lower data sizes have three peaks which correspond to the residual connection layers of each block”. What does this imply?
		- “the residual connections enable stronger gradients, and hence the maximum magnitude weights are expected to reside on the residual branch thereby increasing its density”. Why do you believe that large gradients imply large weight magnitude? There can be large gradients in both negative and positive directions (thus, the sum of the weights updates will be zero, and the weights don’t change).
		- “The ratio of norms are lower for the winning ticket suggesting perhaps the model learns more generalizable representations”. What’s the connection between the output norms and the generalization of the representations?
		- “In fact, the WT+Aug network showcases higher similarity to the network trained on 100% data when compared to the dense+Aug network. The above observations go in hand to further support our empirical analysis regarding the generalizability of the learned representations.” How does this observation support your empirical analysis on generalization? It’s not self-evident what you are implying.
	- **Potential solutions**: The paper would be significantly improved if the authors could explain their results more deeply. If not, then I suggest that the paper should claim *only* to identify an interesting phenomenon (i.e., sparse winning tickets outperform dense networks on small datasets). Also, the abstract/introduction/Contribution should clearly specify that the paper is purely empirical.

# Small changes
- In the title, replace “recognizers” with “classifiers” because the paper investigates only classification tasks
- line 33: what “inductive bias” does pruning provide?
- Improve citations. Many cited papers were published in conferences, but you’ve referenced the arxiv article. Please reference the published versions of the papers.
- In section 3.2, consider reordering the experiments: 1, 3, 2, 4, 5. I think 1 and 3 are related: 1 investigates data augmentation, and 3 investigates other training methods for small datasets (outside data augmentation). 2 and 4 also seem related because 2 analyses memorization and 4 investigates generalization.
- Consider moving section 3.5 (datasets) before section 3.4 (Augmentation strategies)
- Line 220: clarify what type of “memorization” are you referring to. As I understand, you test the “memorization” of the domains of the images because the experiments from Section 4.2 investigates synthetic transforms and domain shifts.
- Lines 235-237 provide an overstatement: “these results clearly demonstrate that winning tickets… are capable of avoiding memorization of training samples at the smaller data sizes, and much more so than the dense models”. There are many types of memorization, and the presented results don’t “clearly demonstrate” that winning tickets “avoid” it. Rephrase and reduce the overstatement.
- line 196 can be removed
- line 248 - add a short one-line about the argument [37] makes on why IMP doesn’t work well on pre-trained networks (i.e., because the magnitude of the weight doesn’t change much during fine-tuning, and the pre-training task largely determines the winning tickets)
- align tables 3 and 4
- Put figures in the order they are referenced in the paper. Figure 2 is referenced after Figure 3 .
- Move the title of Section 4.2 before the text starts (line 220)
- In line 316, add a short one-line description of how the CKA works

questions: All my questions are related to missing explanations/analyses for the empirical results.

1. Why do winning tickets improve performance? Is there any relation between the number of model parameters and the size of the dataset (including data augmentation)?
2. Why is data augmentation required for sparse winning tickets to improve performance on very small datasets (last two subfigures from Figure 2)?
3. Why do sparse winning tickets preserve the filters? (Figure 7)
4. Why do sparse winning tickets preserve the residual connections (Figure 7)?
5. What’s the connection between the output norms and the generalization of the representations? (Figure 6)
6. Figures 3a/b/c show that training using data augmentation provides a 3x improvement on the winning ticket compared to the dense network (in the 1% data scenario). Why does this happen? In contrast, for over 10% of the training data, the improvement is similar for the WT and the dense network.
limitations: I don't see any negative social impact on their work.

The paper should clearly mention that the findings only apply to obtaining sparse winning tickets using Iterative Magnitude Pruning (IMP). Thus, all limitations of IMP also apply to the current work (e.g., IMP doesn't work well for pre-trained networks, and the presented experiments also show that the identified tickets on pre-trained networks provide small performance improvement).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a thorough study of sparse wining tickets in the context of limited data settings. Using iterative magnitude pruning and rewinding, the authors train deep models with limited data, and show that sparse winning tickets outperform standard dense networks. The authors also show that augmentation helps improve the performance for the sparse tickets. In addition, the authors study a long-tailed classification setup and observe that sparse WT outperform dense models even when trained with auto-augment.
strengths_and_weaknesses: **Strengths**:

1. The paper makes an interesting observation on the few-shot learning capabilities of sparse models. In addition, the experiments mostly support their claims. 
2. The experiments in Sec. 5 reveal some interesting properties of sparse wining tickets that aid in data-efficient learning.
3. Technically, the paper appears to be sound, with principled experiments and observations.

**Weaknesses**:

1. While not a weakness per se, it appears that the combination of IMP + augmentation is actually the winning combination for data-efficient learning. Especially, fig. 2 and fig. 4 both show that augmentation appears to play a larger role. Perhaps the authors could add some points to their discussion.
2. Another point of improvement could be training other architectures (transformers, VGG) to see if these properties are dependent on the architecture.
3. Liu et al [1] ,have also studied pruning for few shot learning. The authors should possibly include a citation and discussion.


Overall, the paper is a strong contribution by itself and reveals interesting properties.

[1] Liu et al., An Embarrassingly Simple Baseline to One-shot Learning, 2022

questions: 1. Can the authors share the parameters for their augmentation setup? 
limitations: The authors have mostly addressed the limitations by including experiments for a variety of setups.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that winning tickets can be successfully used in data-limited regimes. The authors conduct several experiments to empirically validate their assumptions. The authors first show that sparse winning tickets have superior performance in data-limited regimes, also compared to a dense network of similar size or a randomly pruned network. Then, they also show that they are robust to distributional shifts, such as domain shifts and synthetic transforms applied to the data. Most of the results are based on the CIFAR10 dataset, but they consider also other low sample datasets from various application fields, including cases with unbalanced classes.
strengths_and_weaknesses: Strengths:
- The authors show an interesting application of lottery tickets. 
- The experiments presented to validate their assumption are accurate and quite convincing, considering also synthetic noise and domain shifts. The authors consider also the case where lottery tickets are used in combination of other techniques for limited-data regimes, such as data augmentation.
- The paper is well written and easy to follow.
- I think that the Discussion section is very interesting, since it tries to give some insights on why lottery tickets can be a valid option in low data regimes. Since the paper is an empirical study, it is important to provide also an analysis that tries to explain the performance shown in the experiments and identifies the properties of the lottery tickets that are useful in this context.

Weaknesses:
- Since this paper is based on empirical evidence, I would have appreciated to see some experiments on different architectures. In the paper the authors perform all the experiments using ResNet-18, which is a quite simple architecture. I am curious to see if the same results could be obtained using also other architectures.
- I found the results not very surprising. Even though there are no prior works on sparse networks for data-limited regimes in the context of image classification, there are some results in NLP.
questions: - Fig. 4 is not very clear, how do you identify the sparse winning tickets and the most robust models (indicated using stars and triangles in the plots)?
- In the paper the authors consider also the case where IMP is applied in combination with well-known pre-training methods for low data regimes, such as SimCLR. However, it is not clear to me how the fine tuning is performed, IMP usually works when the weights are rewinded to initialization (or to an early epoch), therefore I'm not sure if (and how) it can be combined with pre-training techniques.
limitations:  The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: metareview: The paper has received borderline and positive reviews. Overall, the reviewers find the empirical contribution of the paper to be interesting and solid enough (even though one reviewer find the explanations given in the paper to be a bit shallow). The rebuttal was nevertheless convincing. The area chair agrees with the reviewers' assessment and follows their recommendation.
confidence: Less certain
Decision: Accept