Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a new sparse convolution kernel, with the following properties:

* The filters are "outer-vector-wise" sparse. This means that the matrix of flattened filters has its columns split into vectors, and each vector is either zero or non-zero.
* This allows the kernel to use implicit matrix multiplication. (In explicit matrix multiplication the `im2col` operation is used to extract patches of the features map, which are then used in a batched matrix multiply. In implicit matrix multiplication, pointers into the feature map are used instead of explicitly extracting the patches. This is faster.)

When compressing a dense network, there are two ways that the sparsity pattern can be optimized:

* The sparsity pattern can be chosen (i.e., which vectors are zero and non-zero, under the constraint that each column of the flattened filter matrix must have the same number)
* The rows of the flattened filter matrix can be permuted (i.e., the output channels can be permuted), since this doesn't actually change the computation (as long as the next layer's input channels are permuted correspondingly).

The authors propose a row clustering algorithm which uses k-means to find a permutation of the rows that maximizes the weights of the unpruned entries.

The results show that this sparsity kernel can reach significant speedups at relatively low sparsity levels. The experiments also show that the test accuracy remains relatively high after pruning and fine-tuning. Lastly, it's shown that the permutations provide a small but consistent improvement in the test accuracy compared to not permuting.
strengths_and_weaknesses: I think this paper proposes a very neat sparse convolutional kernel. The outer-vector-wise sparsity seems like a good trade-off between fast structured sparsity (e.g., block-wise sparsity) and flexibility (e.g., unstructured sparsity). This works nicely with the implicit matrix multiplication approach, which gives impressive experimental results.

The main weakness of this paper is the presentation and reproducibility. The writing is not good, which makes it more difficult to follow along than it should be, and the pseudo-algorithms seem to be missing important details required to reproduce their results. Similarly, the figures are not as elucidating as they could be, and the experimental results are presented in a way that is difficult to parse.

All in all, the amount of editing that needs to be done to bring this paper to a publishable state is on the borderline of what is reasonable to expect from a conference rebuttal cycle. This makes it difficult to recommend this paper as is.
questions: **Writing**

Overall the text requires a significant amount of editing. For example, the abstract contains typos ("re[p]ly on") and grammar mistakes such as incorrect plurals/singulars ("pattern[s]", "speedup[s]"), missing articles ("[the] OVW pattern"), and incorrect conjugations ("achieve[s]"). I found the writing to be meandering and unstructured throughout. As a reader it was often unclear to me what the main point of each section was. I suggest the authors spend some time editing the text for both grammar and structure.

The text could also be more precise to help ensure reproducibility. Section 4.2 is a particular example of this, with vagues statements ("kmeans requires additional operations to meet this demand") and a lack of precision, e.g., "We employ Kuhn-Munkres algorithm to solve the distance matrix and optimize the mean square error of it". Which distance matrix? Mean square error of what? None of this seems to appear in algorithm 2.

**Illustrations**

I believe figure 1's illustration of "inter-vector-wise" sparsity contains a small mistake (the two top right vectors have 3 instead of 2 unpruned entries)

Figure 2 is confusing to me: The diagram makes it look as if the sparse filter is first densified to then be passed to `load_columns`, the output of which seems the same as the original input `data`? There's a function `recover` which seems to just be `reshape`? And the figure suggests that the feature offsets are a function of the filters' values, whereas they are actually computed based on the row indices (`row_idx`).

**Experimental results**

I believe it would be valuable to perform multiple runs and provide confidence intervals on the presented results, given the small margins (e.g., 65.49 and 65.46 for SqueezeNet OVW permuted/non-permuted).

The "V" in the table of results should probably be explained in the caption. I assume it's the vector length as in the text, but what does it mean for the vector length to be 44.48 or 32.02? And perhaps it is better to move this column to the end of the table, to make it easier to compare the OVW columns to the baseline columns.

The results are also a bit hard to parse since they require the reader to compare numbers across 6 columns to see which ones are bigger or smaller. Perhaps the authors could report "drop/increase in accuracy compared to baseline" instead? And perhaps the columns are better grouped by their sparsity level rather than by the sparsity type.

**Training**

This paper assumes a setting of dense training followed by compression for deployment. I would be interested to see the authors discuss using these sparse kernels during training as well.

**Title**

The title of the paper is overly general. Many papers discuss accelerating sparse convolutions after all. Perhaps something along the lines of "Column-wise sparsity and implicit matrix multiplication for efficient sparse convolutions" would be more descriptive of the content of the paper.
limitations: The authors claim to have discussed the limitations of their method, but it's not clear to me where they did so.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to speed up sparse convolutional neural networks on GPU devices. Instead of unstructured sparsity pattern, this paper proposes a vector-based sparsity pattern (OVW) to balance accuracy loss and computation efficiency. Based on the OVW pattern, a heuristic clustering method is introduced to obtain an appropriate channel permutation with smaller accuracy loss. Moreover, an efficient CUDA kernel is implemented for OVW-based sparse convolution. Extensive experiments on several network architectures demonstrate the efficiency of the method.
strengths_and_weaknesses: **Strengths:**
- This work is clearly motivated. The unstructured sparsity is hard to be accelerated on GPUs, while channel-wise sparsity tends to have a large accuracy drop. The sparsity pattern between these two would have a better trade-off.
- The proposed vector-based sparsity pattern (OVW) is reasonable and technically sound. It can preserve continuous memory access for efficient GPU implementation.
- Extensive evaluation and impressive results. This paper demonstrates the power of the proposed sparse convolution on different network architectures, outperforming representative unstructured sparse methods.

**Weaknesses:**
- The proposed method can achieve practical speedup on V100 GPU. It would be better to verify on more GPU types.
- The writing should be improved. There are some typos: 
Common issue: there should be a space before left bracket.
Line34: Missing citation number.
Line270: Fig5 -> Fig 5.

questions: - If I understand correctly, the hyperparameter $V$ should be an integer. In table 1, why $V$ is 44.48 (32.02) in the second last (last) row?
- The proposed OVW sparsity pattern is faster than unstructured sparsity. It would be better to include the comparison with unstructured sparsity in Figure 5.

limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an out-vector-wise (OVW) pattern-based sparse convolution for accelerating neural networks. The proposed OVW pattern can preserve continuous memory access. Moreover, the clustering-based channel permutation method is introduced to reduce error caused by sparsity. The sparse convolution is efficiently implemented using CUDA for GPU acceleration. The experimental results demonstrate the efficiency of the proposed method over unstructured sparsity.
strengths_and_weaknesses: Strengths:

- The proposed out-vector-wise (OVW) sparse pattern can preserve continuous memory access which is beneficial to GPU acceleration, while unstructured sparsity involves random memory access which is unfriendly for hardware.
- Sparse model acceleration is an important problem for mobile devices. The proposed method achieves a better tradeoff between inference speed and accuracy over unstructured sparsity and channel sparsity.
- The clustering-based channel permutation method can reduce error caused by sparsity.

Weaknesses:

- The paper claims that the OVW pattern can achieve a better tradeoff than channel pruning. please add the experimental comparison to evaluate this claim.
- The hyperparameter V determines the length of a vector. How does it affect the inference speed? It would be better to present an ablation study of it.

questions: please refer to the weaknesses.
limitations: Yes
review_title: null

Meta-review: metareview: While the reviewers had a difference of opinion in their scores of the paper, on balance I think the overall evaluation described in the text of the reviews leans strongly towards acceptance. This is especially the case because the only negative reviewer wrote "if these two changes are made (editing for language/grammar, and reporting confidence intervals in the main text) I would be willing to increase my score to accept" and I think that the authors have shown that they are going to do that in their updated version. The paper is well structured (if not so clearly written), and provides a novel approach to co-designed sparse convolution. It's really nice to see this sort of co-design, which is common in the ML systems space at the hardware-software level, being applied higher in the systems stack to achieve such significant speedups (even with not that much sparsity) for structured sparsity on existing commodity hardware. In general, when I see a technically strong paper where the weaknesses are presentation and spelling/grammar, I lean toward acceptance, with the understanding that the authors can incorporate the reviewers' feedback to improve the writing in the final version of their paper. Applying that reasoning here, I recommend acceptance.
confidence: Less certain
Decision: Accept