Signs in the Lottery: Structural Similarities Between Winning Tickets
Authors: Isabel Holler
Mats Leon Richter
Ulf Krumnack
Venue: Submitted to ICLR 2023 (2023)
Keywords: lottery ticket hypothesis
sparse networks
structural similarity
deep learning

Abstract: Winning tickets are sparse subnetworks of a deep network that can be trained in isolation to the same performance as the full network.  Winning tickets have been found in many different contexts, however their structural characteristics are not well understood. We propose that the signs of the connections in winning tickets play a crucial role. We back this claim by introducing a sign-based structural comparison
metric that allows to distinguish winning tickets from other sparse networks.  We further analyze typical (signed) patterns in convolutional kernels of winning tickets and find structures that resemble patterns found in trained networks.

Under review as a conference paper at ICLR 2023
SIGNS IN THE LOTTERY: STRUCTURAL SIMILARITIES BETWEEN WINNING TICKETS
Anonymous authors Paper under double-blind review
ABSTRACT
Winning tickets are sparse subnetworks of a deep network that can be trained in isolation to the same performance as the full network. Winning tickets have been found in many different contexts, however, their structural characteristics are not well understood. We propose that the signs of the connections in winning tick- ets play crucial role. We back this claim by introducing sign-based structural comparison metric that allows distinguishing winning tickets from other sparse networks. We further analyze typical (signed) patterns in convolutional kernels of winning tickets and find structures that resemble patterns found in trained net- works.
INTRODUCTION
The lottery ticket hypothesis (Frankle & Carbin, 2019) claims the existence of sparse trainable sub- networks in given deep network, the so-called winning tickets. While it has been known for long time that artificial neural networks can be pruned significantly (removing more than 90% of the parameters) after training without impacting their performance (Liang et al., 2021; Blalock et al., 2020), such sparse networks usually resist training when their weights are initialized randomly, meaning they need longer to converge and usually reach lower performance. One interpretation of winning tickets that they form the trainable core dense network, with the other parameters essentially heing dead weights. This overparameterizatioo of the dense network is still useful, it combinatorially expands the number of subnetworks and hence the chance of containing a wining ticket.
Winning tickets have been reliably found for different types of deep networks by pruning connec- tions from randomly initialized dense networks. However, the known pruning approaches are quite laborious, often requiring more resources than simply training the original dense network. A good characterization of what distinguishes winning tickets from other sparse networks still seems to be missing. Better understanding these properties could not only help in designing more efficient pruning algorithms, but it might also allow devising new initialization schemes for deep networks, leading to smaller networks and more efficient training.
In this paper, we address the question of whether winning tickets show specific structural character- istics that allow us to distinguish them from other sparse networks. We claim that it is not merely their sparse structure but also the sign of the connections that should be considered. After briefly reviewing previous work and motivating approach (section 2), introduce sign-aware struc- tural distance metric for sparse networks (section 3), explain our experimental setup (section 4) and apply our metric to the generated winning tickets. We then complement this quantitative analy- sis with a more qualitative inspection of spatial structures found in winning tickets (section 6) and conclude by summarizing findings (section 7).
RELATED WORK
Since their discovery by Frankle & Carbin (2019), winning tickets have attracted lot of attention and several works have shown their existence for different types of networks and datasets, although for larger networks, some additional warmup training seems to be required (Frankle et al., 2019). On the other hand, Ramanujan et al. (2020) have shown that there exist subnetworks that already

---

Under review as a conference paper at ICLR 2023
perform significantly above chance level without any training (specifically they extracted subnet- work of an untrained ResNet-50 that performs like trained ResNet-34), and Malach et al. (2020) show, based statistical arguments, that such networks should always exist. Frankle et al. (2021)
claim that there are networks that do not only show good initial performance but can also be trained.
Chen et al. (2021) propose a method to transform a wining ticket found in one network to be applied to another network architecture suggesting an underlying winning structure and Movva et al. (2021) analyze the structural overlap of winning tickets obtained from the same initial network.
Closest related to our study is the work by Zhou et al. (2019), who have conducted several experi- ments to analyze properties of winning tickets. These also include experiments investigating the role of connection signs. They tried different approaches to change the connection weights of a given wining tieket, including sampling weights from the original initialization distribution, reshuffling the connection weights, or assigning a constant value to all connections. While such a reinitialization typically destroys the winning property of a ticket, they observed that if the signs of the connections are kept, the resulting network still shows comparable performance to the original wining ticket. is this observation that motivates us to take the sign into account when comparing the network structure.
STRUCTURAL COMPARISON OF SPARSE NETWORKS
In contrast to dense networks, sparse networks open way to structural analysis based their connection graph. For example, one can use methods from graph theory to compare the structure of two sparse networks ./1 and ./5. One such approach is the Neural Network Sparse Topology Distance (NNSTD) by Liu et al. (2020). This metric operates in a layer-wise fashion, considering for each pair of units the sets of incoming connections Gi and G2. It uses the normalized edit distance (NED), normalized version of the graph edit distance, that counts the number of edits (additions and removals of connections) required transform one network into the other. To account for differences between two networks, the units in two corresponding layers can be permuted to minimize the overall edit distance. The distance between M and M/2 can then be obtained by averaging over the layer distances.
While the NNSTD can be used to find structural similarities between sparse networks, the structure of a network cannot be the sole factor distinguishing winning ticket from other sparse networks: reinitializing a winning ticket with random weights usually destroys its winning property while keeping the sign, winning ticket tends to stay trainable (Zhou al., 2019). Hence we propose to use a structural similarity metric that takes the sign of a connection into account.
To obtain such a sign-awart metric, we adapt NNSTD in the following way: instead of just consid- ering the set G of all incoming connections to a node, we split that set into two parts, G and G consisting of only the positive and negative connections, respectively. The sign-aware NED* is then defined as the arithmetic mean of the NED for the positive and negative parts:
NED* (G,, G2) = 1/2 (NED (GT GTt) + NND (G,
This metric penalizes connections with different signs in M and N/ more than a connection that simply missing in the other network. One may choose to put more emphasis on a mismatch in positive or negative connections (by taking a weighted average; indeed in our experiment we observe, that the sign are usually not equally represented in the winning tickets). However, for this paper we stick to the symmetric definition.
layers require a special treatment: such layers are considered to consist of multiple filters, each being a stack of 2-dimension kernels that are moved over the input tensor. We consider each filter as a unit, setting the set of incoming connections to be the entries in the filter that have not been pruned. The NED* is then computed for each pair of filters from the corresponding layers of Mi and ./2 to obtain the N for that layer.
Convolutional
4 EXPERIMENTAL SETUP
In designing our experiments, we follow the setting in the original paper by Frankle & Carbin (2019), using the same Conv-2 architecture, consisting of two convolutional layers with 64 kernels of size
2

---

Under review as a conference paper at ICLR 2023
Distribution of Distances for each Condition
NNSTD
NNSTD=
0.89
Condition
Distance (mean over all Layers)
WTS_CIFAR
0.82
0.88
WTS_CINIC
WTs_SVHN
mixed miFAd_CIFAR
0.87
mixed_CINIC
mixed_SVHN
0.81
random
0.86
Netween_ciii__CN
Between_SVHN_CHH
Between_SVHN_Ce
0.80
0.85
WTs
mixed
rand
betw_DS
WTs
mixed
rand
betw_DS
Category
Figure 1: The distribution of the aggnstiniticc NNSTD (left) and sign-aware NNSTD* (right) com- puted distances for each condition, grouped by their category. The comparison indicates that taking the signs into account results in a clearer distinction between winning tickets and randomly pruned networks.
3 x 3, two dense layers with 256 units and an output layer with 10 units. We run our experiments on three different datasets: CIFAR-10, CINIC-10, and SVHN, all being scaled to the same size (32 32 pixels, RGB) and labeled with 10 class labels, so that exactly the same network architecture can be used for all experiments.
We employ iterative magnitude pruning (IMP) with resetting strategy to obtain winning tickets. We run IMP for 11 rounds. in each round training for 10 epochs and then removing all connections not belonging to the top magnitude weights, to obtain the desired pruning rate after the final round. We use different pruning rates for the weights of the convolutional (68.6%, i.c., 10% per round), dense (91.4%, i.e., 20% per round) and output layers (68.6%), while not pruning the biases. Using this approach, we could reliably obtain winning tickets for each of the three datasets at total pruning rate of 91.119 91.2%. For each dataset, we run 20 different trials, each time starting with different random initialization. For comparison, we also create sparse networks by random pruning.
STRUCTURAL COMPARISON OF WINNING TICKETS
To find characteristic structures of winning tickets, we conduct several comparisons based on NNSTD and our extended N metric. In the within-datusel WT-WT comparison, compute the similarity of two different winning tickets found for the same dataset. With N 20 tickets, we have NVN - 1)/2 190 possible comparisons from which 50 were chosen at random. As bascline, we perform the same comparison also for the N 20 random tickets (sampled from the same initial weights the winning tickets). We then compare winning tickets random tickets in the mixed condition. Finally, in the between DSI DS2 condition, winning tickets from different datasets are compared.
Note that the scales in this plot are not identical and direct quantitative comparison of the two metrics is not intended. As NNSTD not distinguishing between positive and negative weights it returns lower distances for all conditions. What is of interest, the relative distance between the different groups in each plot.

---

Under review as a conference paper at ICLR 2023
Table I: The distribution of distances computed by NNSTD and NNSTD* for each condition is statistically compared to the "control" condition random. Columns show the absolute difference of both means, the p-value of a Welch Two Sample 1-test between both groups and a Cohen' 's d as the effect size. In contrast to the sign-agnostic NNSTD metric, the sign-aware NNSTD* metric consistently shows a huge effect size when comparing winning tickets to winning tickets within and also between datasets.
Condition
NNSTD
NNSTD-
difference mean
p-value
effect size
difference mean
p-value
effect size
CIFAC
-0.0006
0.307
0.27 (small)
0,0122
2.5e-25
.4779 (huge)
WTs.CINIC
-0.0016
0.016
-0.64 (medium)
-0,0102
2.1e-23
-4.27 (huge)
WT5_SVHN
-0.0084
1.5e-12
-2.49 (huge)
-000255
3.8e-32
-7.94 (huge)
mixed_CIFAR
0.0013
0.026
0.59 (medium)
0.001
0.066
0.48 (small)
mixed CINIC
0.0017
0.003
0.8 (large)
0.0015
0.006
0.74 (medium)
mixed_SVIII
0.0051
6.3e-10
1.93 (very large)
0.0035
1.3e-08
1.72 (very large)
CINIC
-3e-04
0.570
-0.15 (negligible)
-0.01
1.9e-21
-3.95 (huge)
SVHN. CIFAR
0.0019
0.009
0.7 (medium)
-0,0066
9.4e-22
-3.95 (huge)
SVHN.CINIC
0.0013
0.078
0.46 (small)
-0,0067
2.3e-14
-2.64 (huge)
Figure 1 shows an overview of all comparisons applying the sign-agnostic NNSTD and the sign- aware NNSTD* metrics: the 10 conditions are grouped into the categories WTs, mixed, random and between DS. The W7's category comprises all conditions, where winning ticket from one dataset is compared to another winning ticket from the same dataset. In the mixed category, a winning ticket compared to a random ticket. In random, two random tickets are compared. The category between DS includes all comparisons between winning ticket from one dataset, to winning ticket from a different dataset.
In general, it seems as if NNSTD* is showing a greater similarity (a significantly lower distance) between WTs from the same or different datasets compared to the random condition than NNSTD. For the following analyses, the random condition will be our control group, to which the other conditions are statistically compared.
Table 1 compares the ANSTD-distancee for the nine conditions to the random condition. The first column lists the differences in means. These differences seem very small throughout the conditions, but relative to the (very low) standard deviations of all groups, they can still be significant. To check this, a Welch Two Sample t-test was conducted, comparing the mean of each condition to the random condition and giving evidence of whether they are likely to originate from the same distribution. The Welch test was chosen, as the conditions were all approximately normally distributed, but did not necessarily have the same standard deviation. The null hypothesis is that both compared groups originate from a distribution with the same mean (No Hest condition = (random), hence two-tailed test was conducted. significance value of a = 0.01 was fixed, as comparing a lot of groups would lead to higher probability of reaching the significance level in at least some groups, a smaller significance value seemed appropriate. The last column lists a Cohen's 's to indicate the effect size. The effect size is calculated as: d x-- with being the pooled standard deviation and x; and x, being the means of the currently relevant and the random condition. It therewith represents how many standard deviations the means the two groups are apart. The effect size counts as: negligible < 0.2 < small < 0.5 < medium < 0.8 < large < 1.2 S very large < 2 < huge.
can be seen that, when comparing the conditions with the sign-agnostic NNSTD, the mixed condi- tions (at least two out of three) have a significantly higher distance than the random one. In the WTs conditions, only the WTs_SVHN has a significantly lower distance with a large effect size. From the between conditions, both conditions including SVHN-WTs have higher distance than the control and the between CIFAR and CINIC group, although only significant in the Berween.SHHN.C condition. 2
Mooe detailed plots and statistical analyses of each layer separately for NNSTD can he found in appendix
A.I.
4

---

Under review as a conference paper at ICLR 2023
Weigts per
Type
Hkmber
(a) Histogram of number of weights left per x 3 3 fil- ter (from to 27) for each group of subnetworks. Win- ning tickets show compared to random tickets.
(h) Visualization of how many weights are left in each position of the 3 x 3x filters for each group of subnetworks (each in row).
sparse and more dense filters
The distribution of remaining weights per filter for random tickets and winning tickets for the three datasets (CIFAR-10, CINIC-10, SVHN)
Figure 2:
We next apply the sign-aware NNSTD* + -metric with subnetworks (WTs and random tickets) from the same pool for the NNSTD- above. However, when computing the distances with NNSTD# of the "treatment" conditions have a significant difference in means the "control" (random) condition, with large effect size in most of these cases. Even more relevant, the different categories exhibit clear trends the same direction. The mixed conditions have a positive effect size of around 0.5 to 1.7, meaning that the distances between WT and random ticket (on average) about 0.5 to 1.7 standard deviations higher than the random condition. All conditions comparing a WT to WT (if within the same dataset between different ones) exhibit huge negative effect size, meaning that they have a lower distance (higher similarity) than random-random or WT-random (mixed) comparisons, making them structurally more similar each other, according NNSTD*
These results suggest, that (a) there are structural similarities between winning tickets, making them distinguishable from random tickets by applying the sign-aware similarities at least partly depend on the signs or the connections, as they are not present in the sign-unaware NNSTDD-copparison.
and (b) that these
TYPICAL SIGN STRUCTURES IN WINNING TICKETS
The ONSTD* measurements suggest that WTs share similarities in their signed connection struc- ture. This section analyzes these similarities in more detail, considering the winning tickets from all three datasets as well as the random tickets as a bascline. We focus on the first (convolutional) layer, as it exhibits very similar statistical results as the overall distances (the mean of all layers). Importantly, it shows significantly lower distance for all WT conditions in comparison to the ran- dom condition." 3 In the following, we analyze which structural components are (more) present the WTs (from all datasets) than in the random tickets.
The first aspect the count of unpruned weights per 3 3-filter. We observe that the winning tickets have a higher standard deviation in their count of remaining weights per filter than the random ticketst, meaning that winning tickets rather contain filters with many remaining weights or with only few with the distribution of unpruned weights per filter can be seen in figure 2a.
remaining weights, instead of pruning all filters to the same percentage. Histograms
3 Exact numbers be found appendix A.I.
*While the random subnetworks have standard deviation of 2.35 (on average, filter has 2.35 weights more or less than the mean of 8.5). the winning tickets have higher standard deviation of 3.05 (CIFAR-WTs), 3.03 (CINIC-WTs) and 3.45 (SVHN-WTs)

---

Under review as a conference paper at ICLR 2023
Thereupon analyzed the count of unprumed weights in all positions of the 3 3-filter. While the random subnetworks have a similar amount of remaining weights in each position, there are higher differences between positions in the winning tickets, can be seen in figure 2b Higher and lower numbers of remaining weights (brighter and darker spots) suggest higher importance of weights in some positions compared others. A special structure in the winning tickets is not evident though and these deviations might be dataset-dependant
As a next aspect, we consider the ratio of signs of the weights. If randomly pruned, we could expect the ratio of the number of positive weights divided by the number of negative weights to be (same amount of both signs). The random tickets have indeed ratio of 1.01. The other groups of subnetworks have ratios of 1.83 (CIFAR- -Tss). 1.57 (CINIC-WTs) and 2.02 EEVHN-WT's though, having 1.5 to 2 times as many positive weights remaining as negative ones.
Following these three general aspects, we now combine positions and signs and study the patterns that can be found the pruned kernels. For this, we will focus on the 3-kernels, not distinguish- ing between different colour channels.
Before considering specific patterns, we look the frequency of combinations of weights. Figure shows map for each group of subnetworks and for each position in the x 3 kernel, indicating how often another positive weight jointly present in the same kernel. The red field shows the reference position for each plot.
(a) random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure 3: Heatmaps indicating how many positive weights each position survived pruning when the weight at the reference position (the red field) is also positive.
In the random group, there do not seem to be more or less frequent combinations of unpruned weights. For winning tickets this is not the case though: the positions close reference point more often also contain positive weight than positions further away. When taking into account all kernels containing at least two positive weights and calculating the percentages of two positive weights being neighbours, the results are 34.5% for the random group, 45.9% for CIFAR-, 46.8% for CINIC- and 50.5% for SVHN-WTs. When doing the same computations with only negative values, the percentages similar. The pruned kernels from the winning tickets seem have "grouped" weights of the same signs left.
(a) random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure 4: Heatmaps indicating the number of negative weights at each position that survived pruning when the weight at the reference position (the white field) is positive.
Thiis is also shown by the standard deviation (from the mean of 281 weights per position), which is 14.89 for the random group, 20.12 for the CIFAR-WTI 24.84 for the CINIC-WTs and 54.11 for the SVHN-WTT

---

Under review as a conference paper at ICLR 2023
The next interesting question is whether negative and positive weights also somehow influence the position of the oppositely-sige weights. Figure shows the number of combinations of a positive weight in reference position with negative weight in any other position. As be seen, the negative weights rather occur further away from a positive weight when looking at the winning ticket groups.
After analyzing general combinations, we now turn to the actual patterns. Figure shows which pat- of three unpruned weights especially often encountered in the different subnetwork groups. The first nine most frequent patterns of each group are shown, red stands for a positive weight, blue for a negative weight. The same plot for four remaining weights can be seen in figure 6. For the other counts of weights left from 2 to 7, the corresponding plots can be found in appendix A.3.
(a) random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure 5: The most frequently encountered patterns with three weights left in 3x3 kemel. The number of occurrences is printed above the pattern. stands for positive weights, blue for negative ones.
(a) random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure 6:
The most frequently encountered patterns
weights 3x3 kemnel. The number
occurrences is printed above the pattern. Red stands for positive weights, blue for negative ones.
As was already seen in the previous analysis of combinations of weights, remaining weights of the same sign seem to rather close to each other and opposing signs rather further away. In general, one observe that the most frrequent patterns of the winning tickets do not seem random. Most kernels exhibit similarities to classical filters used in computer vision to detect an edge, a corner or surface.
7 CONCLUSION
We have analyzed the signed structure of winning tickets obtained by the iterative magnitude pruning algorithm before training. We introduced sign-aware metric NNSTD* compare sparse networks, by extending the NNSTD metric. Using NNSTD* we were able to show that winning tickets are structurally more similar to each other than to random tickets, and also more similar than two random tickets compared with each other. This suggests the distribution of signs in winning ticket to be a characteristic property. Interestingly, this is not only the case within, but also between different (image) datasets, suggesting that the structural aspects that make winning tickets special (to some extent) mataset-independenn
We have further taken look at the spatial arrangement of positive and negative weights in the first-layer convolutional kernels of winning tickets. These show characteristic patterns, with iden- tically signed weights close each other, resembling typical filters applied in computer vision.

---

Under review as a conference paper at ICLR 2023
These patterns would not be visible when ignoring the signs in the pruned kernels. Furthermore, these characteristic patterns are in clear contrast to randomly pruned kernels, indicating that these structures are advantageous for quick training, probably because they already anticipate the target configuration.
Both findings underline the significance of taking signs into account when analyzing trainable sub- structures dense networks. While the topological structure alone seems not to be sufficient to explain winning tickets, considering also the initial signs changes the picture. Adopting this idea could not only lead to more efficient algorithms for finding winning tickets in dense networks, or creating sparse "trainable" networks from scratch, utilizing advantageous sign structures. In the long run, it may help to explain what makes these networks trainable and to design weight assignment procedures that go beyond random initialization by choosing signs in a structured way, allowing for smaller models and faster training.
REFERENCES
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze (eds.), Proceedings of Machine Learning and Systems, volume 2. pp. 129-146, 2020.
Xiaohan Chen, Yu Cheng, Shuohang Wang, Jingjing Liu Zhe Gan, and Zhangyang Wang. The elastic lottery ticket hypothesis. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. urXiv:1903.01611 March 2019. URL https: httpt:://aaaaee /ttp:i/aaavvvviaa org/ /bs/ 1903.016100
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural net- works at initialization: Why are we missing the mark? In International Conference on Learning Representations (ICLR), 2021.
Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing, 461:370-403, 2021. ISSN 0925-2312. doi: 10.10066f.eeuccmm222...oo
Shiwei Liu, Tim van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada Sokar, Mykola Pechenizkiy, and Decebal Constantin Mocanu. Topological insights in sparse neural networks. In Proceedings of the European Conference on Machine Learning and Data Mining (ECML PKDD), pp. 279-294. Springer. 2020. ISBN 978-3-030-67663-66
doi: (0.1007/978-3-030- 3-030-67666--.117
Eran Malach, Gilad Yehudai, Shai Sehwar-S and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6682-6691. PMLR, July 2020.
Rajiv Movva, Jonathan Frankle, and Michael Carbin. Studying the consistency and composability o of lottery ticket pruning masks. In Workshop on Science and Engineering of Deep Learning (ICLR), 2021.
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari. What's hidden in a randomly weighted neural network? In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11890-11899, 2020. doi: 10.1109/ CVPR42660.2020.01199 191.
Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32, May 2019.
8

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
The appendix provides additional figures and tables, adding some details to the figures and tables presented in the main text.
A.1 SIMILARITY MEASURES PER LAYER (NNSTD*
Figure in the main text shows the accumulated distances computed by NNSTD and NNSTD* which result from averaging over the five layers of the network. Figure A.1 shows the distances per layer for the NNSTD* metric:
Distribution NNST ANSTDe-distances over all layers
Condition
WTs_CIFAR
WTs_CINIC
WTs_SVHN
Distance
mixed_CIFAA
mixed_CIND
mixed_SVHN
random
Between_CINICCIFAN
between_SVHN_CC
Detween_SVHN_Cm
conv1
conv2
dense1
dense2
dense3
Layer
Figure A.I: The distribution of the NNSTD= BOSTD=*ccopultt -computed distances over all layers for each condition.
The first thing one notices is that there is a huge difference between the distances of different layers, e.g. the first layers seem to be more similar each other, than the later ones. These differences are due to the definition of the NNSTD* the different number of weights per layer and the different pruning rates of each layer. However, we are primarily interested the differences between all conditions in the same layer, which are shown in mor detail in figure A.2.
The layers are called convl and conv2 for the first and second convolutional layer, densel and dense2 for the next dense layers and outpul for the last dense output layer. Interestingly, all layers exhibit a similar trend (mixed has higher or similar distance, WTs and between DS lower distance than random), except for the first dense layer, where every condition has a significantly higher distance than the random condition.
Table A.1 includes all statistical results for the individual layers computed with NNSTD* Layers conv2, dense2 and output show similar picture, all conditions comparing WTs (WTs and between conditions) show very significant lower mean in distance than the random condition. The mixed conditions show varying results, only in the dense2 layer, they all have significantly higher distances. The /ense/ layer shows higher distance for all other conditions compared to the random one, with a huge effect size in each case, as it did with NNSTD.
A.2 SIMILARITY MEASURES PER LAYER (NNSTD)
For comparison, we also document the similarity values computed by the unsigned NNSTD met- ric. Figures A.3 and A.4 show the distribution of the computed distances for each condition and

---

Under review as a conference paper at ICLR 2023
Distribution NNSTDt-Distances for all Layers
conv1
conv2
0.87
0.70
0.86
0.65
Condition
0.85
0.60
0.84
WTs_CIFAR
WTs
mixed
rand betw_DS
WTs
mixed
rand
betw_DS
WTS_CINIC
dense1
WTs_SVHN
0.898
Distance
0.972
0.896
mixed_CIFAR CIFAR
0.971
0.894
mixed_CINIC
0.892
0.970
0.890
mixed_SVHN
WTs
mixed
rand betw_DS
WTs
mixed
rand betw_DS
random
Between_CiNIC_CN
dense3
Between_SVHN_CHH
0.98
Between_SVHN_Cee
0.96
0.94
WTs
mixed
rand
betw_DS
Category
Figure A.2: The distribution of the NNSTD* AASTTD=dissee -distances for all layers separately
each layer separately. In the first plot, an overview of all layers is given. Again we can see a high difference between the distances of different layers, which appears to be quite similar to the same overview for NNSTD* shown in A.2. The second plot goes more into detail for each layer sep- arately. Table A.2 shows the statistical results of comparing the distances of each condition to the random condition for each layer.
The first two convolutional layers, as well as the dense2 layer, show similar results the overall distances did. The mixed and between conditions show slightly higher distances, while the WTs con- ditions show varying results. Only the WTs_SVHN condition mostly exhibits lower distance than the random condition. The densel layer, similar to the NNSTD results, shows a higher distance for all other conditions when compared to the random condition, with huge effect size in every case. The output layer shows rather lower distances for all conditions compared to the random condition, but with varying significance and effect size. In this layer, WT comparisons (the WTs and between conditions) have a large effect size and significant differences in their mean of distances.
A.3 FREQUENT PATTERNS IN WINNING TICKETS
Figures A.5, A.6, A.7, and A.8 show the most frequent kernels with 2, 6, and 7 weights left, respectively, in the random and winning tickets with their corresponding absolute frequency. For each group, a total of 14x +3 = 2, 688 kernels were considered. The total number of x kernels with (0-9) remaining weights can be found in table A.3, complementing the histogram in figure 2a in the main text.
10

---

Under review as a conference paper at ICLR 2023
Table A.I: The distribution of distances found by NNSTI> for each condition is compared to the "control" condition random. Columns show the absolute difference in mean, the p-value of Welch Two Sample 1-test between both groups and a Cohen's 's d as the effect size.
(a) Conv1
(b) Conv2
Condition
dill. p-value
eflect
dill.
p-value
effect
CIFAC
-0.0395
4.6c-27
-5.23 (huge)
-0.01
1.Te-25
-7.05 (huge)
WTs_CINIC
-0.0309
3.1c-20
-3.82 (huge)
-0.007
8.0c-22
-5.28 (huge)
WTs_SVHN
-0.0898
9.9e-26
-6.86 (huge)
-0.0264
2.2o-32
-13.12 (huge)
mixed.CIFAR
0.0027
0.151
0.38 (small)
-0.0002
0.335
-0.25 (small)
mixed CINIC
0.0038
0.030
0.58 (medium)
0.0004
0.108
0.42 (small)
mixed SVHN
0,0146
4.0c-11
2.1 (huge)
0.0015
6.0e-09
1.8 (very large)
between CINIC. CIFAR
-0.033
4.0c-20
-3.88 (huge)
-0.0733
9.1e-23
-5.6 (huge)
between_ SVHN_ CIFAR
-0.032
6.6e-23
-4,2 (huge)
-0.0087
2.6c-26
-6.91 (huge)
between_SSVie SVNICCIIII
-0.0186
5.4e-13
-2.43 (huge)
-0.0058
9.2c-20
-4.55 (huge)
Densel
Dense2
Condition
dill. mean
p-value
effect size
diff. mean
p-value
effect size
WTs. CIFAR
0.0003
1.ce-33
2.82 (huge)
-0.0188
1.7c-08
-1.7 (very large)
WTs.CINIC
0.0004
1.9e11
2.47 (huge)
-0.0026
1.0e-15
-2, 82 (huge)
WTs SVHN
0.0006
3.5e-18
4.13 (huge)
-0.0017
4.7c-09
-1.78 (very lurge)
MixeddCIFAL
0.0015
1.2c-15
15.05 (huge)
0,0011
3.0c-05
1.17 (large)
mixed CINIC
0.002
1.1e-27
9.33 (huge)
0.0013
3.4e-06
1.33 (very large)
miXed_SVHH
0.001
8.3e-38
9.97 (huge)
0.001
8.6e-05
1.09 (large)
between. Netween.CINIC.CO
0.0005
4.1e-19
4.18 (huge)
-0.0025
3.3e-13
-2.44 (buge)
between SVHN CIFAR
0.0006
3,2c-25
5.63 (huge)
-0.002
7.7e-12
-2.21 (huge)
between SVHeC CINIC
0.0009
2.4o222
5.7 (huge)
-0.0016
2.2c-08
-1.68 (very large)
(e) Outpat
Condition
diff. mean
p-value
effect size
WTs.CIFAR
-0.0101
1.7e-04
-1.04 (large)
WTs.CINIC
-0.0112
1.9c-06
-1.37 (very largo)
WTs_SVHN
-0.0102
4.3e-05
-1.15 (large)
mixed. CIFAR
0.0001
0.978
0.01 (negligible)
mixxd_CINIC
-0,0002
0.905
-0.03 (negligible)
mixed SVHN
-0.0007
0.735
-0.99 (negligible)
between. between.CINIC CIFAR
-0.0079
0.001
-0.91 (large)
hetween SVHN CIFAR
-0.006
0.009
..77 (medium)
beiween_SVin SVINN CINIC
-0.0083
0.001
-0.89 (large)
11

---

Under review as a conference paper at ICLR 2023
Distribution NNSTD-distances over all layers
1.0
Condition
WTs_CIFAR
WTs_CINIC
WTs_SVHN
Distance
mixed_CIFAR
mixed_CINIC
0.7
mixed_SVHN
random
between Saaveen_CNII_CCEE
0.6
between Saaeren_SVNN_CIAAA
+
Detween_SVHN_CINIC
conv1
conv2
dense1
dense2
dense3
Layer
Figure A.3: The distribution of the NNSTD-computed distances over layers for each condition.
Distribution NNSTD-Distances for all Layers
conv1
conv2
0.775
0.600
0.575
+
0.770
0.550
0.765
Condition
0.525
0.760
0.500
0.755
WTs_CIFAR
WTs mixed
rand betw_DS
mixed
betw_DS
WTs_CINIC
dense1
dense2
WTs_SVHN
0.949
0.854
D Distange
0.948
0.852
mixed_ CIFAR
0.947
0.850
mixed_CINE
0.946
0.945
0.846
mixed_SVHN
0.944
0.844
WTs
mixed
rand betw_DS
WTs
mixed
rand
betw_DS
random
between_CINIC_CIFAR
dense3
0.98
between_SVHN_Ee
0.96
Detween_SVHN_CIINC
0.94
0.92
WTs
mixed
rand
betw_DS
Category
Figure A.4: The distribution of the NNSTD-computed distances for each layer separately
12

---

Under review as a conference paper at ICLR 2023
Table A.2: The distribution of distances found by NNSTD for each Condition is compared to the "control" condition random. Columns show the absolute difference in mean, the p-value of Welch Two Sample t-test between both groups and a Cohen's 's d as the effect size.
(a) Conv
(b) Conv2
Conditon
diff. mean
p-valuee
effect
diff.
p-value
effect
WTs CIFAR
0.0026
0.220
0.32 (small)
0.0011
0.001
0.92 (large)
WTs.CINIC
-5e-04
0.823
-0.06 (negligible)
6c-04
0.050
0.52 (medium)
WTs.SVHN
-0,0232
6,5c-07
-1.54 (very large)
-0.008
3.0c-14 14
-3.17 (huge)
CCFAR
0,0046
0,016
0.64 (medium))
0.0012
3.9e-4
0.97 (large)
mixed CINIC
mIXId.CINIC
0.0019
0.332
0.25 (small)
0.001
0.001
0.88 (large)
miXed_SVHN
0.0206
6.9e-13
2.38 (huge)
0.0036
4.5e- 19
3.42 (huge)
between CINIC, CIFAR
hetwoen_CINIC.CCC
CINIC,CIFAR
0.0025
0.203
0.33 (small)
7c-04
0.016
0.64 (medium)
0.0159
2.6c-08
1.68 (very large)
0.0013
0.010
(medium)
between SVHN CINIC
0.0157
2.8c-07
1.53 (very large)
0.0022
9.9c-07
1.45 (very large)
Densel
Dense2
Condition
diff. mean
p-value
effect size
diff. mean
p-value
effect size
WTs CIFAR
0.0012
2.6e-25
6.28 (huge)
0.0004
0.161
0.37 (small)
CINICINIC
0.001
8.2e-17
3.97 (huge)
0.0001
0.786
0.07 (negligible)
WTs.SVII
0.0011
3.1e-15
3.61 (huge)
-0.0004
0.224
-0.32 (small)
miXed_CIFAR
0.0029
1.2c-41
16.17 (huge)
0.0023
1.9c-09
1.84 (very large)
mixed CCNIC
0.0041
3.5c-28
10.26 (huge)
0.0031
1.5c-12
2.33 (huge)
MVH
0.0018
2.5e-32
9.79 (huge)
0.0021
9.2c-08
1.58 (very large)
between. beIween,CINIC CINIC CINIC.CIFAR .CIFAR
0.0013
3.8e-26
6.8 (huge)
0.0008
0,020
0.62 (medium)
between SVHN CIFAR
0.0014
1.7c-18
4.62 (huge)
0.0004
0,255
0.3 (small)
between SVHN CINIC
0.0018
4.8c-27
7,81 (huge)
0.940
0.02 (ncgligible)
(0) Output
Condition
dilf. mean
p-value
effect size
WTs CIFAR
-0.0084
0.001
-0.91 (large)
WTs.CINIC
-0.0094
4.7e-4
(large)
WT' SVHN
-0.0113
9.8c-06
-1.26 (very large)
mixed_CIFAL
-0.0046
0.031
-0.57 (medium))
mixed.CINIC
CINIC
-0.0014
0.540
-0.16 (negligible)
mixed_SVHN
-0.0027
0,280
-0.28 (small)
between,
between,CINIC CINIC CINIC,CIFAR CIFAR
-0.0069
0.002
-0.84 (large)
Sovven_SVHN_CCIT CIFARR
-0.00%6
0.002
-0.86 (large)
between SVHN CINIC
-0.0132
4.2c-06
-1.33 (very large)
Table A.3: The number of 3x3 kernels with n weights left for 14 pruned subnetworks selected from each group (2, 688 kernels in total per group).
group
6
8
9
random
84.
361.
683.
764.
501.
223.
57.
12.
3.
0.
CIFAR- WTs
141.
376.
643.
686.
478.
252.
89.
22.
1.
0.
CINIC-WTs
149.
367.
632.
689.
489.
264.
76.
18.
4.
0.
SVHN-WT'
248.
300.
544.
686.
529.
281.
81.
17.
2.
0.
13

---

Under review as a conference paper at ICLR 2023
11
random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure A.5: The most frequently encountered patterns with two weights left in 3x3 kernel. The number of occurrences is printed above the pattern. Red stands for positive weights, blue for negative ones.
T
random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure A.6: The most frequently encountered patterns with five weights left in 3x3 kernel. The number of occurrences printed above the pattern. Red stands for positive weights, blue for negative ones.
LED
LEL
random
(b) CIFAR
(c) CINIC
SVHN
Figure A.7: The most frequently encountered patterns with six weights left in a 3x3 kernl.. The number of occurrences printed above the pattern. Red stands positive weights, blue for negative ones.
random
(b) CIFAR
(c) CINIC
(d) SVHN
Figure A.8: The most frequently encountered patterns with seven weights left in a 3x3 kerne. The number of occurrences is printed above the pattern. Red stands for positive weights, blue for negative ones.
14


Reviews:
[{"review_id": "rmKAeoU-sYc", "reviewer": "Reviewer_UJuy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of understanding the differences between various sparsely connected neural networks models. Consequently, it proposes an improvement over the current existing method (metric) which measures the distance between two sparse neural network models, i.e. Neural Network Sparse Topology  Distance (NNSTD), by considering also the sign of the weight values. The new proposed method is named NNSTD± and it is evaluated mainly on convolutional neural networks.", "summary_of_the_review": "The paper addresses a timely and relevant topic with a fair level of novelty. Unfortunately, given the above discussed limitations, I believe that the paper is not ready yet for publication in its current form.", "strength_and_weaknesses": "Strength:\n* The topic addressed is well understudied even if a small progress on it could help in understanding better the performance and the behavior of sparse neural networks\n* The proposed methodology has a fair level of novelty, in my opinion\n* The empirical validation shows that the proposed method, NNSTD±, can achieve a better granularity and clarity than the baseline method (e.g., better clustering of the output like in Fig 1) in illustrating the differences between sparse neural networks models.\n\nWeaknesses:\n* The proposed method, NNSTD±, could benefit in clarity by a more thorough mathematical description and qualitative discussions in Section 3. For instance, a graphical representation of the method to improve its readability, adding more mathematical details on how the method actually works, adding an algorithm, the range of the metric output, etc.\n* The empirical validation, even if it seems to be well executed, it is incomplete as it addresses just convolutional layers. A systematic study on other type of layers (e.g., fully-connected) could make stronger the results and bring more confidence in the metric. \n* The related work is very poor, and this leads to several statements in the paper which are not fully accurate. While, up to my best knowledge, there is very little work in the literature addressing this particular problem (i.e., the distance between two sparse neural networks), there is a considerable amount of work on dense-to-sparse and sparse-to-sparse training. With respect to the statements, here are some examples: “…such sparse networks usually resist training when their weights are initialized randomly…”; “…However, the known pruning approaches are quite laborious, often requiring more resources than simply training the original dense network…”; “…Since their discovery by Frankle & Carbin (2019), winning tickets have attracted a lot of attention…”. It was actually shown from 2017 [1], that a sparse neural networks trained from scratch can easily outperform its equivalent dense counterparts (where the latter has exactly the same amount of neurons) by using the sparse evolutionary training algorithm which put the basis of what is called today dynamic sparsity, prune and grow strategies, or dynamic sparse training [2]. Non-exhaustively, starting from these two missing references a proper related work discussion can be made.\n* Accordingly, the writing style can be improved to be more rigorous and more academic. All concepts have to be properly defined. E.g., the “winning ticket” term needs a proper definition somewhere in the introduction or in a background section.\n\nReferences:\n\n[1] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, Antonio Liotta, Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science, Nature Communications 2018, https://arxiv.org/abs/1707.04780 \n\n[2] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste, Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, JMLR 2021, https://www.jmlr.org/papers/volume22/21-0366/21-0366.pdf \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lbfxrh_CI8t", "reviewer": "Reviewer_nFr4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a common structural characteristic between winning tickets. The authors argue that the found winning tickets indeed share a similar structure, the sign of the connections. To analyze this feature quantitatively, they devise a new metric (sign-aware structural distance) that measures the degree of the sign and structural similarity between winning tickets better (than solely comparing the network structure). Based on their analyses and experiments on different image datasets, sign-aware structural similarities between winning tickets seem to be a dataset-independent characteristic. ", "summary_of_the_review": "Overall, the idea of the paper is interesting, but further research is needed beyond simple observation.", "strength_and_weaknesses": "Pros\n\nThe main argument of the paper is that the winning tickets share similar network structures and the sign of the connections is an important characteristic. It is interesting to see the characteristics and it is good that the authors provided a simple measure that can evaluate this feature quantitatively.\n\nCons\n\nHowever, the paper ends there. It only contains observations and there are few points that one can make out of the results. How can we benefit from knowing this? Can we somehow use this fact to find the winning ticket directly? Any insights? Any applications? Anything other than just observations? \nIn addition, the authors investigate the co-occurrence of signed weights in some specific convolution filters in the winning tickets and report that they seem to have some patterns. However, this is presented in a very abstract and qualitative way, which makes it hard to count on the results. \nOverall, the main argument needs more rigorous backup experiments. For example, would the results still hold for different methods, like edge-popup? Different initialization distributions (Kaiming, signed, Gaussian, etc)? Different architectures? More results for different datasets? In my humble opinion, it is difficult to convince the reader in its current state.\n\nThe presentation can be improved. For example, it is very hard to see the differences in figure 1, which is one of the main results of the paper. Please make the colors stand out by using more vivid ones and consider using different markers or patterns than only using colors. The size of the fonts and tick values should be enlarged. (This also applies to other figures)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PaW_QxjLVQ", "reviewer": "Reviewer_ghvk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a sign-based metric for comparing the structure of two sparse subnetworks, in order to understand the importance of signs in Winning Tickets (Lottery Tickets). Using two different metrics, one that is \"sign-aware\", they compare the distances between Winning Tickets and \"random tickets\", which are in this particular work defined as random masks/random weights. The authors further attempt to analyze differences between signed patterns in sparse subnetworks found for convolutional neural networks.", "summary_of_the_review": "While the direction of understanding the importance of the mask itself in sparse neural networks is interesting, the comparison of winning tickets with masks found from training to models with random masks is not well motivated. Furthermore, its not clear that the analysis presented in this paper is significant, and there are several reasons to question the comparison of models even within the same metric, as it differs for convolutional and fully-connected models.", "strength_and_weaknesses": "Strengths\n* There is less analysis about the structure (i.e. mask) of winning tickets than the weights and this paper, at least at a high level, proposes better understanding the importance of structure of winning ticket subnetworks.\n* The results in Table 1 showing differences between the metric distance between WTs and random tickets within the $NNSTD^{\\pm}$ are potentially interesting.\n\nWeaknesses\n* It appears the paper compares Winning Tickets (i.e. subnetworks with masks found from pruning a dense neural network (i.e. the pruning mask), and using the (original) Lottery Ticket Hypothesis, initialized with the original pruned dense initialization, with sparse subnetworks - with *random masks* (i.e. as described in Section 4, \"random pruning\"). While they appear to still use the original pruned dense initialization, this doesn't make much sense, as we don't expect completely random connectivity to perform as well as that found via training in winning tickets! Rather (as the authors point out themselves in section 3) the main question in this space is why we can't train from *random initialization* the *same* subnet, i.e. *same pruning mask*, to as good a generalization as in a winning ticket.\n* The paper only looks at winning tickets in the context of the original lottery ticket hypothesis - i.e. using dense initialization and tiny models. We now know in practice this work doesn't extend to larger models, and instead we must use weight-rewinding in practice. Similarly it's not clear that any analysis the authors make on these tiny models/original winning tickets would extend to real-world models.\n* The proposed \"metric\" itself ($NNSTD^{\\pm}$) is not directly comparable to vanilla NNSTD (expanded upon in the next weakness), as the authors themselves point out in the footnote on page 5, these are fundamentally different metrics. However, the author's claim that the \"relative distances between the different groups in each plot\" is comparable is also not necessarily true! These relative distances are still distances in different metric spaces. This is fine when the authors compare the separation of WT v.s. random within the same metric as done with Table 1.\n* It's not clear that the proposed (different) $NNSTD^{\\pm}$ metric for convolutional NNs is comparable to the one for fully-connected neural networks given the modifications suggested.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "48Te-4WlR3p", "reviewer": "Reviewer_MFkC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigate the structrual similarity between winning tickets. In particular, they argue that the signs in weight connection play a critical role.", "summary_of_the_review": "I have to reject this paper in the current shape. I can see the point in the results but overall, the writing need huge improvements.\n\nThe authors are welcomed to further revise this paper to be ready for review.", "strength_and_weaknesses": "Strength:\n\n* The topic of whether lottery tickets (LTs) is unique or share some similarity is of great interests to the communicty.\n\nWeakness:\n\n* The paper does not organized very well.\n* What is the conclusion of this paper? The sign is only thing matters or is one part of it. LTs may have lots of similarity or dissimilarity. For example, [1] aruges that the ratio of remaining neurons in each layer matter the most. How is this paper different from that one?\n\n[1] Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject