Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper empirically studies the lottery ticket hypothesis with limited or no supervision. First, the authors use self-supervised learning to generate winning tickets, showing that "good" (reasonable) winning tickets can be found without labels. Second, the authors show that finding "good" (reasonable) winning tickets can be accelerated by a factor 5 on ImageNet by using only a subset of the data. The authors also argue that using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading.

The experimental results are rich and provide more understanding of winning ticket generation with limited or no supervision. The results on self-supervised learning task (including the layer-wise pruning results) and a subset of training dataset are reasonable and kind of expected, but it is still good that this paper provide solid experimental results to verify this. As the paper observed, "none of the tickets found with limited access to labels and or data matches the accuracy of tickets found with all the labeled data when considering moderate pruning rates (more than 10% of unpruned weights)
on ImageNet. Indeed, we consistently observe a decrease in performance compared to the full overparametrized network as soon as we prune the network." In this sense, winning tickets are certainly label and data dependant. This undermines the *bold* claim in the abstract that "we provide a positive answer to both questions, by generating winning tickets with limited access to data, or with self-supervision". From my perspective, the ability to exactly perserve the accuracy while pruning the weights (see the flat regions of "Lables" curves in Figure 1,2,3,4,5) is the interesting part of the lottery ticket hypothesis. We have several different ways to achieve a descreased accuracy with a smaller network, the dynamics there may be a mixture of the lottery ticket hypothesis and standard model pruning, which needs more careful experiment design to separate different dynamics.

"using large datasets is important to study lottery tickets, since deep networks trained on CIFAR-10 are natually sparse, making conclusions potentially misleading." "The definition of “early in training” is somehow ill-defined: network
weights change much more for the first epochs than for the last ones." These two messages are important to future study of the lottery ticket hypothesis. This paper raises the issue of ill-definedness of “early in training”, but did not provide a solution. 

Overall, I found that the experimental results in this paper are solid and provide more understandings of the lottery ticket hypothesis. However, I feel that the novelty of this paper is limited, and do not provide much new insights. Therefore, it does not reach the bar of being published at ICLR, from my perspective. Therefore, I say "Weak Reject".
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors try to provide empirical answers to several important open questions on winning tickets. They conduct most of experiments on ImageNet and results show that winning ticket is robust, and few data samples can also obtain good winning tickets.

Generally, the paper has conducted extensive experiments on three open questions and results prove their assumptions.

As describe in page 7, lottery tickets are sensitive to data distributions. I’m wondering, whether there will be winning ticket for multi-task learning with limited data each task? Will this be helpful in distilling the model?
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: In the original lottery ticket paper, it points out that training the pruned architecture from scratch with initial weight can achieve the same performance compared to fine-tuning it. This work further discuss this phenomenon when data or label is not enough. It is good to see the few-data/label can still provide a comparable results. But some experiment’s results and its setting are confusing, while also makes me concerned about the conclusion solidness.

1) Usually in classification task (especially in cifar10 dataset), 0.5% to 1% accuracy could be a huge gap between two models. For example, in the original “Lottery Ticket Hypothesis” paper, using initial weight only has roughly 0.5% improvement compared to random initialization. But the figures in this paper do not contain a zoom-in details for each line, make me hard to distinguish the performance between each setting. If the author does not provide a detailed version, it will look like theses model have the same performance, which is actually wrong. The author should either plot a zoom-in figure especially when the pruning ratio larger than 50% or give a Table with accuracy of each setting. And it is better to complete the figure with several random seed and plot the error bar to avoid randomness.

2) Does the “Random - adjusted” item in Figure. 1 mean the correctly pruning architecture with random initialization? In "rethinking the value of network pruning", Liu et al. points that in the large learning rate setting (lr=0.1, which is also your setting), random initialization can achieve the same performance compared to the lottery ticket. In my perspective, I want to see whether few-data/label also works on random initialization instead of lottery tickets. I expect the author to explain the “Random -adjusted” experiment setting clearly in the response and I suggest the author to discuss the
random initialization part specifically.

3) Figure.3 only shows the “varying dataset size” experiments on ImageNet. The experiments on cifar10 is lacked. The author should complete this part in the response.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper studies the problem of finding sparse networks in a limited supervision setup. The authors build on the lottery ticket work of Frankle & Carbin and investigate the validity of their idea when one has few or no labels. This work is an immediate followup on Morcos et al. who investigated the transferability of lottery tickets.

This work is more observational rather than algorithmic or theoretical. Authors study various small sample/label setups where network sparsification works well. 

Main contribution is Section 4.1 where self-supervision is investigated. However given that lottery tickets are transferable (Morcos paper) it is really not that surprising that semisupervised learning algorithms will do a decent job as well. I also don't see a practical benefit beyond transfer learning setup.

Section 4.2 essentially sweeps through supervised problem parameters such as reducing sample size, adding noise etc and . The main application seems to be extracting lottery tickets faster by downsampling the data however this aspect is again fairly obvious. 

In short, unfortunately, this paper doesn't cut it for ICLR. As improvements, I would recommend adding standard semi-supervised training techniques to their comparison. I was surprised to not see pseudo-labeling or consistency training (e.g. virtual adversarial training).
review_title: Official Blind Review #3

Meta-review: The paper studies finding winning tickets with limited supervision. The authors consider a variety of different settings. An interesting contribution is to show that findings on small datasets may be misleading. That said, all three reviewers agree that novelty is limited, and some found inconsistencies and passages that were hard to read: Based on this, it seems the paper doesn't quite meet the ICLR bar in its current form. 
Decision: Reject