Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the use of random weights together with learnable masks. The learnable masks are learned with straight through operator. The authors argue that such training approach for neural network would reduce the model storage requirements and has applications to network compression. The model is validated on cifar 10 and cifar 100 datasets showing that the proposed layers underperforms (approximately) between 1 and 10 accuracy points wrt dense layers depending on the model architecture and number of parameters.

-------
Post rebuttal: Based on authors responses I updated the overall score from 3 to 6 and increased soundness and presentation both by 1.
strengths_and_weaknesses: Overall, the discussed ideas are interesting – using a random layer with learnable masks to achieve competitive performance. However, the validation and the presentation of the paper require improvements. For details see comments below.

**Title**
- The title might be a bit to generic (not very informative) and a bit misleading (for more complex datasets one would probably need more layer architectures). How about the following title:  On learning masking operators for network random weights. 

**Abstract**
- To strengthen the abstract, please add quantification of improvements in terms of space complexity. Also, based on the experimental section the improvements come at the expense of model accuracy, however, this trade-off is not captured in the abstract.

**Introduction**
- Introduction section is in general well written and easy to follow. However, it could benefit from some re-writing, shortening, and refocusing. 

- The introduction does not discuss the obtained results making it hard to assess the significance of the proposed approach. It is also unclear what the ML community gains with the results of this study. Please add such discussion to the introduction section.

**Methodology**

- This section would benefit the most from re-writing and restructuring. In the current form it is difficult to follow and do not allow to fully appreciate the presented ideas.

- Figure 2 should be better discussed and better formatted. Please extend the caption to clarify the figure. In general, all figures in the paper should be self-explanatory making it possible to understand the figures just by reading the captions. In its current form it is impossible to understand the drawings.

- Based on the description the process of updating the prototype weights into target weights is unclear. Could the authors clarify how different networks are updated?

- Eq 4 and Eq 7 differ only in the dimensionality of w. Why changing the dimensionality leads to new paradigm of random weights padding? Moreover, please boldface vectors in Eq 7 to differentiate them further from scalars in Eq. 4.

- The section lacks motivations behind different choices. 

**Results**

- The introduced layer is not compared to previously published models. Would it be possible to compare the model to Supermaks and Popup? Adding comparisons to previous art would make the validation stronger.

- Cifar datasets are small scale. Would the observations generalize to larger scale datasets? Adding another dataset would make the observations more compelling. 

- The results are missing stds, making it hard to assess the significance of the results.

- In general, the proposed layers are underperforming w.r.t dense layers. That would be expected. However, the results section lacks discussion and positioning of the reported results, e.g., Why the reported results are interesting? What do we learn as a community from the results? What is the impact of the reported results? Adding more in-depth discussion would make the paper stronger.

questions: See above (strengths and weaknesses).
limitations: - The paper does not discuss the limitations of the discussed ideas.

- The paper does not mention societal impacts.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper aims to handle the difficulty of restoring/transmitting models caused by the increasing model size for recent large-scale neural networks. Inspired by recent works (e.g., LTH, Popup) on random networks, the paper starts by answering a scientific question: what is the potential of a random network? Specifically, the authors propose a series of strategies to study the random network with different masks to map different features. Through the exploration for the answer, a new model compression paradigm is proposed by only restoring one-layer random weights and a bunch of masks to represent a model. Experiments were conducted based on using different CNN/transformer architectures. Extensive results validate the rationality of the motivation and show the feasibility of the new compression paradigm.
strengths_and_weaknesses: Strengths:
1) This work tries to reduce the model storage size, which is a clear and practical motivation. Compared with typical model size compression methods that remove partial parameters, it is a novel way to represent a model by using different masks on fixed random weights.
2) This work is driven by studying the random weight capacity, which is an interesting yet under-explored studying point. It is novel to use “one-layer” weights with different masks to learn a model.
3) Experiment is extensive using different model architectures. Firstly, it answers the question about random weight potential using a series of proposed strategies to construct a network using random weights. Secondly, it shows the feasibility of a new compression paradigm compared with the typical model compression method.

Weakness:
1) It is encouraged to revise the draft title to a more appropriate one. After reading the draft, I think the current title doesn’t convey the key factor of this paper. Iteratively selecting different masks on a set of fixed random weights for different feature mappings should be the main point, therefore, the usage of “one-layer” in the title is inaccurate. On the other side, “all you need” is a too vague description. It needs to be concretized to eliminate confusion.
2) Some related works are supplemented in the appendix, I suggest moving them into the main draft and providing necessary discussions about them. The discussion should include the difference between the submitted work with these existing works since they look highly related to this work, even if they are in a different setting.
3) Technically, the proposed random vector padding (RP) repeats the given set of random weights in the same order. If randomly shuffling the random set and then doing the padding to construct the model, can it improve the capacity?
4) Minors: (1) In Alg.3, it seems the output is written in the wrong way, which should be the output of MP strategy in Alg.2, but not consistent with RP strategy in Alg.3. (2) Around Eq. 5 and Eq. 6, the explanation of T^p is missing. It should be further clarified and consistent with Alg.1. (3) In Eq.9, the d_l should be the dimension of w_l instead of the number of vector v_pro. Please make it clear to eliminate confusion. 
questions: See the above weaknesses.
limitations: The authors have addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposed a new paradigm for neural network compression. The authors randomly initialize a set of weights. The actual parameters of each layer are represented as the initialized weights with binary masks. The weights are shared by multiple layers, while the masks are different for each layer. The weights are fixed, while the masks are learnable. In this way, the total bytes are significantly reduced. Experiments show that the proposed method achieves better compression than baselines.
strengths_and_weaknesses: Strengths:

1. This paper is well organized, and the core method is clearly represented.
2. This paper represents each layer as shared weights with different masks. The idea for model compression is interesting and novel.
3. Experiments show that the proposed method achieves good compression for image classification models.

Weaknesses:

1. The title of this paper is unsuitable and the authors should change it. First, people will not associate the title with model compression. Second, the word "one layer" in this paper is misleading. Although some parts are shared cross all layers, there are differences between layers. Thus, we can't say them "one layer". In my opinion, masks are also parameters of the model.
2. It is better to compare the compression performance with stronger baselines or bigger datasets such as ImageNet.
3. In general, the proposed method achieves compression by sharing some parts of parameters (while adjusting the others). Several previous works have explored this direction, such as [1] and [2]. The authors should discuss them in related works.

[1] Residual connections encourage iterative inference. International Conference on Learning Representations, 2017

[2] Recurrent convolutions: A model compression point of view. NIPS Workshops: Compact Deep Neural Network Representation with Industrial Applications, 2018


questions: None (refer to weaknesses)
limitations: Authors discussed the limitations and potential negative societal impact of their work in supplementary material.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new way of representing a neural network in a compressed way coined "One Layer is All You need". The idea is to keep a single fixed and randomly initialized weight vector as prototype for each layer of the network, whereas each layer is saved as a learned mask determining which weights of the prototype are used. Since saving bit masks is more memory efficient than floating point, the network can be efficiently stored. Experiments with ResNet32, ResNet56, ConvMixer and ViT on CIFAR10 and CIFAR100 show that this method achieves improved results in terms of accuracy compared to sparse network training baselines while maintaining larger compression ratios.
strengths_and_weaknesses: Strengths:
- {S1} The problem of storing neural networks in an efficient manner is significant and the proposed idea improves in this direction.
- {S2} The trade-off between network compression and accuracy is improved in comparison to sparse network training baselines.
- {S3} The writing is well-structured and easy to follow.

Weaknesses:
- {W1} Experiments only performed on low-resolution datasets (CIFAR10, CIFAR100, TinyImagenet).
- {W2} It is not clear if experimental settings are repeated with different seeds. The checklist refers to the supplementary material, but I cannot find any results for multiple seeds there either. I believe all experiments should be conducted for multiple seeds.
- {W3} No code is included for reproducibility.
- {W4} The writing should be improved in terms of typos and grammar (see below for some instances).

Typos:
- The sentence in lines 20-21 seems incomplete.
- Lines 126/150: rewrited -> rewritten
- Line 259: compreesion -> compression
- Line 263: The sentence is confusing, because you train two networks with different strategies and not one network with both.
- Line 307: foundamental -> fundamental

--------------------------------------------------------------------------

{W2} is addressed by the authors during the discussion. Furthermore, they ensured they will resolve {W3} and {W4}. I updated my score respectively.
questions: - {Q1} The paper mentions a predefined sparse selection ratio (line 129) which is not further mentioned in the experiments. What value did you set K to for your experiments?
- {Q2} Given the paper, it is not clear to me how the compression ratio is computed for MP/RP in Figures 4/5. I understand that Eq. 10 is used for previous work, however, the proposed method seems to decouple p in C_w (which depends on the fixed vector size) and C_m (which depends on K as mentioned in Q1). Could you please show how you calculated the compression sizes with an example?

I'm willing to raise my score if my questions and concerns ({W2}) are addressed.
limitations: The authors discussed limitations in the supplementary material. The fact that this method cannot be used to compress already pretrained models but requires training from scratch is an important limitation and should be mentioned in the main paper.
review_title: null

Meta-review: metareview: The paper studies the use of random weights together with learnable masks. Authors demonstrate that such training approach for neural network can reduce the model storage requirements and has applications to network compression.

Reviewer appreciated  the novelty of the idea and the extensive experiments on various architectures.  Adding experiments that would go beyond small-scale datasets would further strengthen the quality of the paper and its potential impact.
confidence: Certain
Decision: Accept