Parameter-Efficient Masking Networks
Authors: Yue Bai
Huan Wang
Xu Ma
Yitian Zhang
ZHIQIANG TAO
Yun Fu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Random weights representative capacity
A new network compression paradigm

Abstract: A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g., Transformer). They empower the network capacity to a new level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning diverse masks and introduce the Parameter-Efficient Masking Networks (PEMN). It also naturally leads to a new paradigm for model compression to diminish the model size. Concretely, motivated by the repetitive structures in modern neural networks, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive network modules. Therefore, the model can be expressed as \textit{one-layer} with a bunch of masks, which significantly reduce the model storage cost. Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights vector. In this way, our method can further lower the space complexity, especially for models without many repetitive architectures. We validate the potential of PEMN learning masks on random weights with limited unique values and test its effectiveness for a new compression paradigm based on different network architectures.
Code is available at \href{https://github.com/yueb17/PEMN}{\textcolor{magenta}{https://github.com/yueb17/PEMN}}.

Parameter-Efficient Masking Networks
Yue Bai
Huan Wang Xu Ma' Yitian Zhang'
Zhiqiang Tao Yun Fu 12.4
'Department of Electrical and Computer Engineering, Northeastern University
"Khoury College Computer Science, Northeastern University SScooo of Information, Rochester Institute of Technology
AInnovation Labs, Inc.
Project Homepage: https::/yuubl1 https///uebll.giitubb.//P.MM wwt.gttu.b.b..........
Abstract
A deeper network structure generally handles more complicated non-linearity and performs more competitively. Nowadays, advanced network designs often contain a large number of repetitive structures (e.g.. Transformer). They empower the network capacity
level but also increase the model size inevitably, which is unfriendly to either model restoring or transferring. In this study, we are the first to investigate the representative potential of fixed random weights with limited unique values by learning diverse masks and introduce the Parameter-Efffcient Masking Networks (PEMN). It also naturally leads to a new paradigm for model compres- sion to diminish the model size. Concretely, motivated by the repetitive structures in modern neural networks, we utilize one random initialized layer, accompanied with different masks, to convey different feature mappings and represent repetitive network modules. Therefore, the model can be expressed as one-layer with a bunch of masks, which significantly reduce the model storage cost. Furthermore, we enhance our strategy by learning masks for a model filled by padding a given random weights vector. In this way, our method can further lower the space com- plexity, especially for models without many repetitive architectures. We validate the potential of PEMN learning masks on random weights with limited unique values and test its effectiveness for a new compression paradigm based on different network architectures. Code is available at httpss//gthuubcco/yyye/77777//
Introduction
Deep neural networks have emerged in several application fields and achieved state-of-the perfor- mances 8 16 29). Along with the data explosion in this era, huge amount of data gathered to build network models with higher capacity framework to deal with multi-modal and multi-task problems
(25). addition, researchers also pursue a unified network
powerful intelligent model [25 37 All these trending topics inevitably require even larger and deeper network models to tackle diverse data flows, arising new challenges to compress and transmit models, especially for mobile systems.
Despite the success recent years with promising task performances, advanced neural networks suffer from their growing size, which causes inconvenience for both model storage and transferring. To reduce the model size of given network architecture, neural network pruning
typical technique [22 20, [2]. Pruning approaches remove redundant weights using designed criteria and the pruning operation can be conducted for both pretrained model (conventional pruning: [13] 12)
and randomly initialized model (pruning at initialization: (21 32)). Another promising direction is to obtain sparse network by dynamic sparse training [9] (24. They jointly optimize network architectures
and weights to find good sparse networks. Basically, these methods commonly demand regular training, and the final weights are updated by optimization algorithms like SGD automatically.
"Corresponding author: bai wwehnortheastern.cmm edu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Now that the trained weights have such a great representative capacity, one may wonder what is the potential of random and fixed weights or is it possible to achieve the same performance on random weights? If we consider whole network, the answer is obviously negative as random network cannot provide informative and distinguishable outputs. However, picking a subnetwork from a random dense network make it possible as feature mapping varies with changes of subnetwork structures. Then, the question has been updated as what is the representative potential of random and fixed weights with selecting subnetwork structures? Pioneer work LTH [10] shows the winning ticket exists in random network with good trainability but cannot be used directly without further training. Supermasks [39] enhances the winning ticket and enable it being usable directly. Recent work Popup [26) significantly improves subnetwork capacity from its dense counterpart by learning the masks using backpropagation. Following this insightful perspective, we further ask question - -what is the maximum representative potential of a set of random weights? In our work, we first make thorough exploration of this scientific question to propose our Parameter-Ffficieent Masking Networks (PEMN). Then, leveraging on the PEMN, we naturally introduce a new network compression paradigm by combining a set of fixed random weights with corresponding learned mask to represent the whole network.
We start with network architectures which recent popular design style, i.e., building a small-scale encoding module and stacking it to obtain a deep neural network [8 28 271. Based on this point, we naturally propose the One-layer strategy by using one module as a prototype and copy its parameter into other repetitive structures. More generally, we further provide two versions: max-layer padding (MP) and random weight padding (RP) to handle diverse network structures. Specifically, MP chooses the layer with the most number of parameters as the prototype and uses first certain parameters of prototype to fill in other layers. RP even breaks the constraint of network architecture. It samples random vector with certain length as the prototype which is copied several times to fill in all layers based on their different lengths. RP is architecture-agnosti and can be seen as most general strategy in our work. Three strategies are from specific to general manner and reduce the number of unique parameters gradually. We first employ these strategies randomly initialize network. Then, we learn different masks to explore the random weights potential and positively answer the scientific question above. Leveraging on it, we propose new network compression paradigm by using set of random weights with a bunch of masks to represent a network model instead of restoring sparse weights for all layers (see Fig. [T] We conduct comprehensive experiments to explore the random weights representative potential and test the model compression performance to validate our paradigm. We summarize our contributions as below:
We scientifically explore the representative potential of fixed random weights with lim- ited unique values and introduce our Anrameter-Efficien Masking Networks (PEMN). It leverages on learning different masks to represent different feature mappings.
A novel network compression paradigm is naturally proposed by fully utilizing the repre- sentative capacity of random weights. We represent and restore network based on a given random vector with a bunch of masks instead of retaining all the sparse weights.
Extensive experimental results explore the random weights potential by using our PEMN and test the compression performance of our new paradigm. We expect work can inspire more interesting explorations in this direction.
2 Related Works
2.1
Sparse Network Training
Our work is related to sparse network training. Conventional pruning techniques finetune the pruned
network from pretrained models 12 [3 with various pruning criteria for different applications (22
14 B3 B 35 36). Instead of pruning a pretrained model, pruning at initialization [32) approaches
attempt to find winning ticket from the random weight. Gradient information is considered to build pruning criteria in (0). Different from pruning methods above, sparse network training also can be
conducted in a dynamic fashion. To name a few, Rigging the Lottery [9] edits the network connections and jointly updates the learnable weights. Dynamic Sparse Reparameterization 24 modifies the parameter budget among the whole network dynamically. Sparse Networks from Scratch (6] proposes momentum based approach to adaptively grow weights and empirically verifies its effectiveness.
2

---

Conventional Sparse Network
Optimized weights
Paramdter-Effteien Masking, Networks
Random fixed weights
Feature mappings
Feature mappings
Pruning using criteria
Selecting using masks
sparse
One-layer with
weight matrix
several musks
Figure I: Comparison of different ways to represent neural network. Different features mappings are shown as blue rectangles. Squares with different color patches inside serve as parameters of different layers. Left is the conventional fashion where weights are optimized and sparse structures are decided by certain criteria. Right is our PEMN to represent network where the prototype weights are fixed and repetitively used to fill in the whole network and different masks learned deliver different feature mappings. Following this line, we explore the representative potential of random weights and propose novel paradigm achieve model compression by combining random weights and bunch of masks.
Most of the sparse network training achieve the network sparsity by keeping necessary weights and removing others, which reduces the cost of model storage and transferring. our work, we propose novel model compression paradigm by leveraging the representative potential of random weights accompanied with subnetwork selection.
2.2 Random Network Selection
Our work inherits the research line of exploring the representative capacity of random network. The potential of randomly initialized network is pioneeringly explored by the Lottery Ticket Hypoth- esis [10], and further investigated by [3] 34 It articulates that there exists winning ticket subnetwork in random dense network. This subnetwork can be trained in isolation and achieves comparable results with its dense counterpart. Moreover, the potential of the winning ticket is further explored in Supermasks 39 It surprisingly discovers the subnetwork can be identified from dense network obtain reasonable performance without training. It extends and proves the potential of subnetwork from good trainability to being used directly. More recently, the representative capacity of subnetworks enhanced by Popup algorithm proposed by (26). Based on random dense initial- ization, the learnable mask is optimized to obtain subnetwork with promising results. Instead of considering network with random weights, the network with the same shared parameters can also delivery representative capacity to some extent, which is investigated by Weight Agnostic Neural Network [II] and also inspires this research direction. We are highly motivated by these researches to validate how is the representative potential of random weights with limited unique values by learning various masks.
2.3
Weight Sharing
Our study also related several recent works about weight sharing. This strategy has been explored and analyzed in convolutional neural networks for their efficiency [8). In addition, several works also proposed for efficient transformer architecture using weight sharing strategy 19
5 I There
are two main differences between these works and our study: 1) They follow the regular optimization strategy to learn the weight in recurrent fashion, which is closer the recurrent neural network. Our work follows different setting. We use fixed repetitive random weights to fill in the whole network and employs different masks to represent different feature mappings; 2) They mainly conduct cross-layer weight sharing for repetitive transformer structure. In our work, explore the potential of random weight vector with limited length as much smaller repetitive granularity fill in the whole network, which is more challenging than cross-layer sharing strategy.

---

Regular
One-Layer
:000
1000
--1 I
Randomly ly initialized
Prototype
Maxer Padding
Random Vector Padding
H 1000
0 0
Prototype
Prototype
Figure Illustrations of different strategies in PEMN to represent network structures. Compared with regular fashion where all parameters randomly initialized, we provide three parameter-efficient strategies, One-layer, Max-layer padding (MP), and Random vector padding (RP), to fully explore the representative capacity of random weights.
Parameter-Efficien Masking Networks
3.1 Instinetive Motivation
Overparameterized randomly initialized neural network benefits network optimization to get higher performance. Inevitably, the trained network contains redundant parameters but can be further compressed, which defines the conventional neural network pruning. On the other side, the network redundancy also ensures large random network contains huge number of possible subnetworks, thus, carefully selecting specific subnetwork should obtain promising performances. This point of view has been proved by (26] 39]. These works demonstrate the representative potential of certain subset combinations of given random weights. Following this lane, we naturally ask a question: what the maximum representative potential of a set of random weights? or in another word: can we use random weights with limited unique values to represent usable network? We answer this question as positive and introduce our Parameter- Efficient Masking Networks (PEMN), Moreover, leveraging on I) compared with trained network where the weight values cannot be predicted, we can pre-access the random weights before we select the subnetwork; 2) selected subnetwork can be efficiently represented by bunch of masks, we can extremely reduce the network storage size and establish new paradigm for network compression.
3.2 Sparse Selection
We follow
26 to conduct the sparse network selection. We start from a randomly initialized neural network consisting of L layers. For each 1 21, L}, has
I (F[[; wi))
where I and Z,+1 are the input and output of layer 1. a is the activation. F represents the encoding
layer such as convolutional or linear layer with parameter parameter dimension of layer To perform the sparse selection, all the weights w = {w1, W2, } are fixed and denoted as w. To pick the fixed weights for subnetwork, each weight w/ is assigned a learnalle clement-wise score of indicate its importance in the network. The Eq. is rewrited as
(w/ w?...., ... w ). where di is the
I++1 oFF Wi @ h(s:))),
(2)
where s; {s/ s7 8/ is the score vector and h(-) is the indicator function to create the mask. It outputs when the value of belongs to the top K% highest scores and outputs O for others, where K is predefined sparse selection ratio. Through optimizing with fixed subset of original dense

---

weights is finally selected. Since h(-) is a non- derivebivalle function, the gradient of each sy cannot be obtained directly. The straight-through gradient estimator [3] is applied to treat ht-) as identity function during gradient backwards pass. Formally, the gradient of is approximately computed as
(($)) = OC ai++
Oc 01++1
(3)
0I,11 Os
0h++
where T++1 = (F un @ si)), which applied estimation. [(s/) is approximately estimated gradient of weight score of. In this way, the dense network is randomly initialized but fixed, but one of its subnetwork can be selected using Backpropagation. In work, name this optimization process as sparse selection.
3.3 Enrrmeter-Efficien Strategy
Following the logic of parameter-efficient exploration from specific to general scenario, PEMN utilizes three strategies construct the whole network based on given random weights: 1) One-layer 2) Max-layer padding (MP) (Sec. (3.4). and 3) Random vector padding (RP) (Sec. (Sec.B.4), which are detailedly introduced below.
3.3.1 One-Layer
Sparse selection initializes a dense network as pool to pick certain weights. It provides a novel di- rection to find admirable subnetwork without pre- training and pruning. However, with increasing of network scales, the cost of restoring and transfer- ing a neural network grows rapidly. Noticed that more popular network structures follow similar design style: proposing a well-designed model- ing block and stacking it several times boost network capacity, we are inspired to explore the feasibility of finding subnetworks by iteratively selecting different masks in a series of repetitive modules. Formally, a L-layer randomly initial- ized network Nz can be represented as series of parameters:
Algorithm One-Layer
1: Input: A random network with L-layer:
= {wi, W2, ..., WL)
2: Output: A L-layer network filled by
prototype layers with parameters: w"
[Wpros Wpro? ...., Wpror}
3: Randomly initialize layers from I to L
4: Record parameter space dimensions:
{TK, 1R,,,...TRR,
5: Initialize a prototype layers list: Listpro =
6: for l in L.,, do
if ViRprr E Listpro + R2 then
8: Append wi into Listpro
else
No w [w2, W2.... wLl: wi E R,,
(4)
10: Find Upror, where Rpro" = Ry
11: Replace wi with Wpror
where E 21, L.) and uy is used for various
12: end if
layers. R2 denotes different parameter spaces (e.g..
13: end for
H W for CNN layer, where R 1 for linear, IR," N
14: Return updated w w"
1/0 are input/output dimensions and N/H/W are CNN kernel dimensions). From shallow to deep layer, we first sample the prototype layers for the whole network with unique parameter spaces,
represented as Wprro [Wipro' , Wprro" .... [p For each Wproz, we use its parameters to replace its target layers Wtane which share the same parameter space:
Waar? +- Wpror: ; Rear? = Reror t E (1.
(5)
where TP is the number of target layers of prototype The whole network filled by several layers with unique weight size and Eq. 4 can be rewrited as
T' +++12+..7*
NL : w² prot Wpro Upror W. PrroP
ETe = L.
(6)
We take simple 5-layrr MLP to clarify this operation: its dimensions are [512, 100, 100, 100, 10) with 4 weight matrices wi E R512x100 We 10000x100 wa E IR 10000000 and w44 E R 100xx10 In this case, W1, W2 and w4 are three prototype layers. w2 has two target layers wa and itself. w/ /w/ only has itself as target layer. The general algorithm is summarized in Alg.
5

---

In this way, any repetitive modules in a given network structure can be represented by one bunch of random weights. Using the sparse selection strategy, we iteratively pick subnetworks in the same set of random weights to obtain diverse feature mappings. Therefore, the cost to represent the network significantly reduces, especially for deep network with many repetitive blocks. other words, one random layer with different masks can represent the majority of a complete network structure, which is named as one-layer.
3.4 Random Weights Padding
One-layer strategy efficiently handles networks with many repetitive modules. The majority of the whole network can be compressed into random layer with set of masks. However, in wool-world applications, various network architectures may not follow a tidy pattern resulting in different shapes for different layers. Hence, the one-layer is not flexible enough to efficiently represent such networks. To handle it, we naturally propose a enhanced strategy, Random Weights Padding. It consists of two versions, Max-layer padding and Random vector padding. We first formally rewrite Eq. 4as
Nr : w = [ W2, .... wr]; wy E Rd {1,2, L}.
where wi is flatten into a vector with dimension di (e.g.. di N x H x W for CNN layer).
O for linear layer and di
Max-Layer Padding (MP). It chooses the layer Wm the prototype where dm max [d,
dr.]) is the highest dimension. All other layers in Nr. have fewer parameters than Wm. We keep the prototype it is and simply pick the first di parameters from wno to replace the parameters in wy, which is described by
wi + w dij:/ E {1,2, L}.
(8)
Random Vector Padding (RP). Instead of picking a complete layer as prototype, RP further reduces the granularity of random prototype from layer random weights vector with relatively short length. We let Upro E Rd. as the random weights vector with length d,,, For each layer I, we repeat Upro several times to reach the length of wy. It can be formally described as
de
Wy
Pproo-. I (1, L}.
After the padding operation, weights in Eq. 7 are reshaped back into the format of Eq. 4 to perform as network. These two padding strategies are summarized in Alg. and Alg.
In the series of one-layer, MP, and RP, based on sparse selection, we explore using fewer unique weights represent the whole network. Leveraging the property of the fixed weight values, the cost of delegating a network keeps decreasing by using a random vector with a bunch of masks. By this mean, we fully explore the representative capacity of random weights with limited unique values. Furthermore, a novel model compression paradigm can be correspondingly established by restoring a set of random weights with different masks. Our three strategies compared to regular network setting are shown in Fig. 2.
Algorithm 2 Max- Padding
Algorithm 3 Random Vector Padding
1: Input: A L-layer random network with
1: Input: A L-layer random network with
parameters: w [w1, uP..... w.)
parameters: = {W1, W2, .., W.)
2: Output: A L--ayer network with MP: w" [wmll: d, w,,l: dz). wm dr.13
2: Output: A L-layer network with RP: w"
de
3: Randomly initialize layers from I to L
Erros Randomly initialize layers from to L
(prros ...] Ppro:....
4: Find the layer u' with the maximum weight dimension dm among all layers wi,l (1,2....,L]
4: Randomly initialize a weights vector Upro
Rdd with d, dimension
5: for 1 in 1, 2.., .... do
for in 1, 2, ..., do
6: Replace wi with the first di values in Wm
6: Repeat Upro until reaching the length d:
given by wm[[: de)
7: Replace w, with the repeated vector Upro
7: end for
8: end for
8: Return updated as n"
9: Return updated w as w²

---

94-
91
(96]
69-
Acc (9)
85
Mask 236
Dense,25 312
Donse_250
86-
Mask 256
Donse 256
Dense 532
Mask
86
83
byer_256
layer
Onalayer
RP 256
MP R2 256
NP RP 512
MP RP _512
79
Mask One-ayer
MP
RP.ted MP
le2 RP le3 R2_ie-4 RP_leS
Mask One One-
MP
RP.1e1 XP le2 RP le.3 RP_le4 RPlleI
(a) 6-block ConvMixer with 256/512 dimensions.
(b) 8- 8-block ConvMixer with 256/512 dimensions.
81
(96)
Acc (96)
79.
Donse_250
Dense_250
Dense, 312
Dente 512
736
Mask: 256
M35k
75-
Mask
One-byer_226 256
One-byer_200
Onelkyer_
Gneiayerr_ 512
RP 256
73
MP RP R2_256
NP R2 512
MP RP RP_512
66
Mask
One mye-ayer
MP
16-1 RP 1o2 RP 1e3 RP 1o4 1e.5
Mask
One byee
AP 16-1 RP 1e2 RP le3 1e4 RP le.
(c) block VIT with 256/512 dimensions.
(d) 8-block VIT with 256/512 dimensions.
Figure 3: Performances of ConeMixer and VIT backbones on CIFARIO dataset with different model hyperparameters. Y-axis represent the test accuracy and X-axis denotes different network parameter settings. Dense means the model is trained in regular fashion. Mask is the sparse selection strategy. One-layer, MP, and RP our strategies. The decimal after RP means the number of unique parameters compared with MP. From Mask to RP le-5, the unique values network decrease. Different experimental settings illustrate the representative potential of random weights.
4 Experiments
Our experiments conduct empirically validations on two aspects of our interests. Firstly, we validate how large is the representative potential of random weights with limited unique values to test the effectiveness of proposed Ceaunetr--ffficienn Masking Networks (PEMN), Secondly, leveraging on the pre-accessibility random weights and lightweight storage cost binary mask, it is promising to establish new model compression paradigm.
4.1 Preparation
We comprehensively use several classic or recently popular backbones for image classification task conduct general validations. Backbones include ResNet32, ResNet56 (16), ConvMixer (28), and ViT 8 We use CIFARIO and CIFAR 100 datasets [8] for our experiments.
4.2 Representative Random Weights in PEMN
We first explore the representative potential of random weights in PEMN based on our proposed strategies, One-Layer, Max-Layer Padding (MP), and Random Vector Padding (RP). We a CNN based architecture Conymixer 28 and MLP based model ViT g to conduct our experiments on CIFARIO dataset [18].
In Fig. 3 we show pairs of experiments based on backbones (ConvMixer, ViT) using depth numbers (6, 8) and 2 hidden dimensions (256, 512). Each pair includes a dense network performance and a series of results obtained by sparse selection with different random weighting strategies. Specifically, Mask learns the mask on the randomly initialized network. One-layer, MP, and RP represent our proposed strategies. To simplify the comparison, we use rate number after RP to show how many unique parameters used in RP compared with MP. From the left Mask to right
7

---

67
90
63
47
55
57
53
83
49-
kanden 20
47
ka 20
M
45
m1
Randon 26
39
Rander 26
37
77 89
35
96
89
91
93
Size Compression Ratio (%)
Size Compression Ratio (%)
(a) ResNet32/ResNetee on CIFARIO dataset.
(b) Resnet32/ResNet566 on CIFARIO0 dataset.
CIFAR datasets ResNet32/ResNee16 backbones. Y-axis denotes the test accuracy. X-axis means the network size compression ratio. Different colors represent different network architectures. The straight lines the top are performance of dense model with regular training. Lines with different symbol shapes denote different settings. For ResNet, our three points based on MP, RP le-1, and RP Ie-2, respectively, This pair of figures show that our proposed paradigm achieves admirable compression performance compared with baselines. In very high compression ratios, still maintain the test accuracy.
Figure 4: Compression performance validation
RP le-5, the number of unique parameters gradually decreases. Different settings show the similar patterns concluded as: 1) Compared with regular trained dense model, sparse selection approach generally obtains promising results, even if with performance drop caused by the constraint of fixing all the parameters; 2) From left to right on X-axis, the performance gradually drops. This is caused by the decreasing number of unique values in network, which makes network has less representative capacity; 3) However, performance drop arises when the number of unique parameters is extremely low (e.g., g.. RP le-4, RP le-5). The results remain stable for the most of random weights strategies; 4) Larger depth and hidden dimension boost the model capacity for different configurations. The performance drop of random weights strategies from their dense counterpart is also decreased. In addition, ViT shows some unstable fluctuation when fewer unique parameter, compared with ConvMixer with relatively stable patterns. This may caused by the difficulty of training MIP based network itself and will not affect our main conclusions.
The performance stability shown above illustrates the network representative capacity can be realized not only by overparameterizing the model, but also carefully picking different combinations of random parameters with limited unique values. In this way, the effectiveness of our PEMN is validated and can represent a network using a random parameters prototype with different learned masks, instead of typically restoring all the different parameters. This property inspires us introduce new model compression paradigm proposed in following section.
4.3 A New Model Compression Paradigm
Practically, our work proposes new network compression paradigm based group of random weights with different masks. We first elaborate the network compression and storage processes to clarify our advantages then report the empirical results.
4.3.1 Sparse Network Storage
Previous works aim remove redundant weights (c.g., unstructured pruning) among different layers. The trivial weights are set to zero based on different criteria. The ratio of zero-weight the whole network is regarded as sparsity ratio. Different approaches are compared based on their final test accuracy with a given sparsity ratio. Different from this conventional fashion which restoring sparse trained weights, instead use fixed random weights with different masks represent network. To compare these two paradigm, we calculate the required storage size as an integrated measurement.
Assuming we have a trained network with p parameter numbers and sparsity ratio. Due to its sparsity, we only need to restore the non-zero weight values accompanied with their position [12 denoted binary mask. The storage cost can be separated into two parts, C.,, for weight values and

---

Cm for mask given by
C= C= + Cm-
(10)
For conventional setting, C.. restores the values of kept sparse weights which is p. (1 r) for the whole network. It needs to be restore in float format. Cm restores sparse positions of these weights, which can be restored into compressed sparse column (CSC) or compressed sparse row (CSR) formats with cost around 2p- (1 r) [12. In our new paradigm, C records the values of the given random weights. For example, the one-layer requires to record all weights of non-repetitive layers and one prototype weights of all repetitive layers. MP requires to keep the values of layer with the largest number of parameters. RP only requires restore the values of random vector with given length. Cm is also for the sparse positions to record the selected subnetwork.
4.3.2 Compression Performance Validation
We test the compression performance on CI- FARIO and CIFARIO0 datasets using Con- VMixer with 6/8 depths, ResNet32, and ResNe156 backbones. The compression ratio is based on the storage size as we discussed in- stead the conventional pruning ratio. Since strategy to compress network, we involve two sparse network training base- train
96
90-
86
propose
Deere d6 Aasson
lines in our experiments. Specifically, a sparse network from scratch by removing ran- dom weight and minimum magnitude weights. For compression ratio, we set four settings for baselines: 90%, 92%, 94%, and 96%. We di- rectly refer to settings, MP, RP from Sec. Sec. for our paradigm. Their compression ratio is calcu- lated using the same measurement as baselines.
80
Maannnn
Janden
92 Size Compression Ratio (%)
95
96
Figure 5: Compression performance validation on CIFARIO dataset on ConvMixer backbone. Y-axis is the test accuracy. X-axis means compression ratio. Two pairs of comparisons are for different depths shown in different colors. Straight line on the top the dense model performance. Curves in different symbols represent baselines and our method. For ConvMixen, our three points are based on RP le-1, RP 1e-2, and RP le-3, respectively.
In Fig. 4 we show 4 pairs of comparison based on 2 backbones (ResNee32//ResNet556) and 2 Each pair in- cludes dense model, 2 baselines with 4 com- pression ratios, and results in 3 ratios. Two baselines are sparse network training by prun- ing random weights and minimum magnitude weights, respectively. For convenience, we do not follow exactly the same compression ratios as bascline but directly use settings from Sec 4.2 For ResNet, we use MP, RP le-1, and RP le-2. Their corresponding compression ratios are computed as shown in figures. Experiments based on different networks and datasets show the similar conclusions summarized as: 1) our method outperforms the baselines by a significant margin, with even higher compression ratio; 2); Compared with conventional sparse network training where compressed model performance decreases obviously along with increasing compression ratio, our method is relatively robust to the compressed model size; 3) If we compare cross different models, we find compressed small model by our method even performs better than baselines using larger model; 4) Network scale affects the compression performance, compared with ResNet32, NesNet56 basically contains more parameters and performance drop between compressed network with its dense counterpart is relatively small. In Fig. 5 we show compression performance on CIFARIO dataset using ConvMixer with depth 6 and 8. The settings are basically similar to Fig. We can also draw the similar conclusions: 1) Our method outperforms the baselines on ConvMixer with different depths; 2) Our method compresses network into lower size but maintains higher performance.
datasets
As a summary, in Sec. 4 our experiments can be separated into two parts. Firstly, to validate our PEMN, we investigate the representative potential of random weights which are used to fill in the complete network structure using different proposed strategies. Secondly, the promising conclusion (Sec. 2 for this investigation naturally leads to the newly proposed network compression paradigm. Different from conventional fashion restoring sparse weights, we instead restore the fixed random weights and different masks. Empirically, we validate the effectiveness of our paradigm for

---

network compression. Our experiments involve diverse network architectures to demonstrate the proposed paradigm can be generalized into different network designs.
5 Discussion and Conclusion
Discussion We summarize our intuitive logic and potential research direction in the future. Our fundamental insight is motivated by Supermasks (39) and Popup [26] showing random network encodes informative pattern by selecting subnetworks. They inspire us to understand neural network in decoupled perspective: the informative output is delivered by certain weight-structure combination. Even if weights are fixed, the flexibility of learnable masks still provides promising capacity to represent diverse semantic information. We the first to fully explore the representative potential of random weights, and practically, a new network compression paradigm is naturally established. We further discuss some research directions in the future following this study. Firstly, compared with conventional approaches need to record learned weights, our paradigm records random weights which can be pre- pre-accessed, can this property be used for improve the model security? In addition, leveraging on the property that repetitive random weights existing in networks for our strategies, is it possible to specifically design hardware deployment configurations to achieve further compression or acceleration? Moreover, our PEMN is based on random initialized weights but cannot be directly deployed on pretrained models, which are more powerful these days. How to further improve our strategy on large-scale pretrained models is another interesting point explore. We leave these topics in our future work.
Conclusion We first explore the maximum representative potential of a set of fixed random weights, which leverages different learned masks obtain different feature mappings. Correspondinglyy we introduce our proposed Parameter-Efficiaa Masking Networks (PEMN). Specifically, we naturally propose three strategies, one-layer, max-layer padding (MP), and random vector padding (RP), to fill in complete network with given random weights. We find that a neural network with even limited unique parameters can achieve promising performance. It shows that parameters with fewer unique values have great representative potential achieved by learning different masks. Therefore, we can represent a complete network by combining a set of random weights with different masks. Inspired by this observation, we propose a novel network compression paradigm. Compared with traditional approaches, our paradigm can restore and transfer network by only keeping a random vector with masks, instead of recording sparse weights for the whole network. Since the cost of restoring mask is significantly lower than weight, we can achieve admirable compression performance. We conduct comprehensive experiments based several popular network architectures to explore the random weights potential for PEMN and test the compression performance our new paradigm. We expect our work can inspire further researches for both exploring network representative potential and network compression.
Acknowledgments and Disclosure of Funding
We thank the anonymous NeurIPS reviewers for giving us valuable and constructive suggestions to improve our paper. This work supported by Northeastern University and finished by Yue working in SmileLab research assistant. There are competing interests to disclose.
References
[1] S. Bai, J. Z. Kolter, and V. Koltun. Deep equilibrium models. NeurlPS, 32, 2019.
[2] Y. Bai, H. Wang, Z. Tao, K. Li, and Y. Fu. Dual lottery ticket hypothesis. arXiv preprint arXiv:2003002448 2022.
[3] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3433 2013.
[4] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, al. Language models are few-shot learners. NewrIPS, 13:1877-1901, 2020.
10

---

[5] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. Universal transformers. arXiv preprint arXiv:1807.03819 2018.
[6] T. Dettmers and L. Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprini arXiv: arXiv:1907.000
2019.
[7] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488) 2018.
[8] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[9] U. Evci, T. Gale, J. Menick, P. S. Castro, and E. Elsen. Rigging the lottery: Making all tickets winners. In ICML, pages 2943-2952. PMLR, 2020.
[10] Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803.03635, 2018.
[11] A. Gaier and D. Ha. Weight agnostic neural networks. NeurIPS, 32, 2019.
[12] Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. arXiv preprint arXiv:1510.00149, 2015.
[13] S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. NeurIPS, 28, 2015.
[14] B. Hassibi and D. Stork. Second order derivatives for network pruning: Optimal brain surgeon. NeurlPS, 5, 1992.
[15] B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. IEEE international conference on neural networks, pages 293-299, 1993.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[17] S. Jastrzebski, D. Arpit, N. Ballas, V. Verma, Che, and Y. Bengio. Residual connections encourage iterative inference. arXiv preprint arXiv: 2017.
[18] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[19] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, Albert: A lite bert for self-supervised learning language representations. arXiv preprint 19Xiv:1909.11990 2019.
[20] Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. NeurlPS, 2, 1989.
[21] N. Lee, T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arX1v:/810.023400 2018.
[22] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv: 012.66000.71111 2016.
[23] E. Malach, G. Ychudai, Shhaav-SShwwrtz, and O. Shamir. Proving the lottery ticket hypothesis: Pruning all you need. In ICML, pages 6682-6691. PMLR, 2020.
[24] H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In ICML, pages 4646-4655, PMLR, 2019.
[25] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, pages 7748-7763. PMLR, 2021.
[26] V. Ramanujan, M. Wortsman, A. Kembhavi, Farhadi, and M. Rastegari. What's hidden in randomly weighted neural network? In CVPR, pages 11893-11902, 2020.

---

[27] I.O. O.O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung. A. Steiner, D. Keysers, Uszkoreit, al. Mip-mixer: An all-mlp architecture for vision. NeurIPS, 34, 2021.
[28] A. Trockman and J. Z. Kolter. Patches are all you need? arXiv preprint arXiv:2201.09792.
2022.
[29] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. NeurlPS, 30, 2017.
[30] C. Wang, G. Zhang, and R. Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arX1v:2002.07376, 2020.
[31] H. Wang and Fu. Trainability preserving neural structured pruning. arXiv preprint arXiv:2207. / /2534, 2022.
[32] H. Wang, C. Qin, Y. Bai, Y. Zhang, and Y. Fu. Recent advances neural network pruning at initialization. arXiv e-prints, pages arXiv-2103, 2021.
[33] H. Wang, Qin, Zhang, and Fu. Neural pruning via growing regularization. arXiv preprint 20Xiv:2012.09243, 2020.
[34] H. You, C. Li, P. Xu, Y. Fu, Y. Wang, X. Chen, R. G. Baraniuk, Z. Wang, and Y. Lin. Draw- ing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint a90iv: Ttt.11.99.1999991 11957, 2019.
[35] Y. Zhang, H. Wang, C. Qin, and Y. Fu. Aligned structured sparsity learning for efficient image super-resolution. NewrIPS, 34:2695-2786 2706, 2021.
[36] Y. Zhang, H. Wang, C. Qin, and Y. Fu. Learning efficient image super-resolution networks via structure-regulaarize pruning. In ICLR, 2021.
[37] Y. Zhang and Q Yang. A survey multi-task learning. TKDE, 2021.
[38] Z. Zhang and C. Jung. Recurrent convolutions: A model compression point of view. 2018.
[39] H. Zhou, Lan, R. Liu, and J. Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. NeurIPS, 32, 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] . You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you deseribe the limitations of your work? [Yes] See supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary material.
12

---

(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] See supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]] See Section
(b) Did you mention the license of the assets? [Yes] See Section 4
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 'r using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "cFl87AKce0", "reviewer": "Reviewer_jejx", "summary": "This paper proposed a new paradigm for neural network compression. The authors randomly initialize a set of weights. The actual parameters of each layer are represented as the initialized weights with binary masks. The weights are shared by multiple layers, while the masks are different for each layer. The weights are fixed, while the masks are learnable. In this way, the total bytes are significantly reduced. Experiments show that the proposed method achieves better compression than baselines.", "questions": "None (refer to weaknesses)", "limitations": "Authors discussed the limitations and potential negative societal impact of their work in supplementary material.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is well organized, and the core method is clearly represented.\n2. This paper represents each layer as shared weights with different masks. The idea for model compression is interesting and novel.\n3. Experiments show that the proposed method achieves good compression for image classification models.\n\nWeaknesses:\n\n1. The title of this paper is unsuitable and the authors should change it. First, people will not associate the title with model compression. Second, the word \"one layer\" in this paper is misleading. Although some parts are shared cross all layers, there are differences between layers. Thus, we can't say them \"one layer\". In my opinion, masks are also parameters of the model.\n2. It is better to compare the compression performance with stronger baselines or bigger datasets such as ImageNet.\n3. In general, the proposed method achieves compression by sharing some parts of parameters (while adjusting the others). Several previous works have explored this direction, such as [1] and [2]. The authors should discuss them in related works.\n\n[1] Residual connections encourage iterative inference. International Conference on Learning Representations, 2017\n\n[2] Recurrent convolutions: A model compression point of view. NIPS Workshops: Compact Deep Neural Network Representation with Industrial Applications, 2018\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KsrCUGqWTh", "reviewer": "Reviewer_9nMH", "summary": "This paper proposes a new way of representing a neural network in a compressed way coined \"One Layer is All You need\". The idea is to keep a single fixed and randomly initialized weight vector as prototype for each layer of the network, whereas each layer is saved as a learned mask determining which weights of the prototype are used. Since saving bit masks is more memory efficient than floating point, the network can be efficiently stored. Experiments with ResNet32, ResNet56, ConvMixer and ViT on CIFAR10 and CIFAR100 show that this method achieves improved results in terms of accuracy compared to sparse network training baselines while maintaining larger compression ratios.", "questions": "- {Q1} The paper mentions a predefined sparse selection ratio (line 129) which is not further mentioned in the experiments. What value did you set K to for your experiments?\n- {Q2} Given the paper, it is not clear to me how the compression ratio is computed for MP/RP in Figures 4/5. I understand that Eq. 10 is used for previous work, however, the proposed method seems to decouple p in C_w (which depends on the fixed vector size) and C_m (which depends on K as mentioned in Q1). Could you please show how you calculated the compression sizes with an example?\n\nI'm willing to raise my score if my questions and concerns ({W2}) are addressed.", "limitations": "The authors discussed limitations in the supplementary material. The fact that this method cannot be used to compress already pretrained models but requires training from scratch is an important limitation and should be mentioned in the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- {S1} The problem of storing neural networks in an efficient manner is significant and the proposed idea improves in this direction.\n- {S2} The trade-off between network compression and accuracy is improved in comparison to sparse network training baselines.\n- {S3} The writing is well-structured and easy to follow.\n\nWeaknesses:\n- {W1} Experiments only performed on low-resolution datasets (CIFAR10, CIFAR100, TinyImagenet).\n- {W2} It is not clear if experimental settings are repeated with different seeds. The checklist refers to the supplementary material, but I cannot find any results for multiple seeds there either. I believe all experiments should be conducted for multiple seeds.\n- {W3} No code is included for reproducibility.\n- {W4} The writing should be improved in terms of typos and grammar (see below for some instances).\n\nTypos:\n- The sentence in lines 20-21 seems incomplete.\n- Lines 126/150: rewrited -> rewritten\n- Line 259: compreesion -> compression\n- Line 263: The sentence is confusing, because you train two networks with different strategies and not one network with both.\n- Line 307: foundamental -> fundamental\n\n--------------------------------------------------------------------------\n\n{W2} is addressed by the authors during the discussion. Furthermore, they ensured they will resolve {W3} and {W4}. I updated my score respectively.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KKKSA4fougQ", "reviewer": "Reviewer_PU7f", "summary": "This paper aims to handle the difficulty of restoring/transmitting models caused by the increasing model size for recent large-scale neural networks. Inspired by recent works (e.g., LTH, Popup) on random networks, the paper starts by answering a scientific question: what is the potential of a random network? Specifically, the authors propose a series of strategies to study the random network with different masks to map different features. Through the exploration for the answer, a new model compression paradigm is proposed by only restoring one-layer random weights and a bunch of masks to represent a model. Experiments were conducted based on using different CNN/transformer architectures. Extensive results validate the rationality of the motivation and show the feasibility of the new compression paradigm.", "questions": "See the above weaknesses.", "limitations": "The authors have addressed the limitations and potential negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) This work tries to reduce the model storage size, which is a clear and practical motivation. Compared with typical model size compression methods that remove partial parameters, it is a novel way to represent a model by using different masks on fixed random weights.\n2) This work is driven by studying the random weight capacity, which is an interesting yet under-explored studying point. It is novel to use “one-layer” weights with different masks to learn a model.\n3) Experiment is extensive using different model architectures. Firstly, it answers the question about random weight potential using a series of proposed strategies to construct a network using random weights. Secondly, it shows the feasibility of a new compression paradigm compared with the typical model compression method.\n\nWeakness:\n1) It is encouraged to revise the draft title to a more appropriate one. After reading the draft, I think the current title doesn’t convey the key factor of this paper. Iteratively selecting different masks on a set of fixed random weights for different feature mappings should be the main point, therefore, the usage of “one-layer” in the title is inaccurate. On the other side, “all you need” is a too vague description. It needs to be concretized to eliminate confusion.\n2) Some related works are supplemented in the appendix, I suggest moving them into the main draft and providing necessary discussions about them. The discussion should include the difference between the submitted work with these existing works since they look highly related to this work, even if they are in a different setting.\n3) Technically, the proposed random vector padding (RP) repeats the given set of random weights in the same order. If randomly shuffling the random set and then doing the padding to construct the model, can it improve the capacity?\n4) Minors: (1) In Alg.3, it seems the output is written in the wrong way, which should be the output of MP strategy in Alg.2, but not consistent with RP strategy in Alg.3. (2) Around Eq. 5 and Eq. 6, the explanation of T^p is missing. It should be further clarified and consistent with Alg.1. (3) In Eq.9, the d_l should be the dimension of w_l instead of the number of vector v_pro. Please make it clear to eliminate confusion. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A7Csk-v-i6f", "reviewer": "Reviewer_vyrt", "summary": "The paper studies the use of random weights together with learnable masks. The learnable masks are learned with straight through operator. The authors argue that such training approach for neural network would reduce the model storage requirements and has applications to network compression. The model is validated on cifar 10 and cifar 100 datasets showing that the proposed layers underperforms (approximately) between 1 and 10 accuracy points wrt dense layers depending on the model architecture and number of parameters.\n\n-------\nPost rebuttal: Based on authors responses I updated the overall score from 3 to 6 and increased soundness and presentation both by 1.", "questions": "See above (strengths and weaknesses).", "limitations": "- The paper does not discuss the limitations of the discussed ideas.\n\n- The paper does not mention societal impacts.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the discussed ideas are interesting – using a random layer with learnable masks to achieve competitive performance. However, the validation and the presentation of the paper require improvements. For details see comments below.\n\n**Title**\n- The title might be a bit to generic (not very informative) and a bit misleading (for more complex datasets one would probably need more layer architectures). How about the following title:  On learning masking operators for network random weights. \n\n**Abstract**\n- To strengthen the abstract, please add quantification of improvements in terms of space complexity. Also, based on the experimental section the improvements come at the expense of model accuracy, however, this trade-off is not captured in the abstract.\n\n**Introduction**\n- Introduction section is in general well written and easy to follow. However, it could benefit from some re-writing, shortening, and refocusing. \n\n- The introduction does not discuss the obtained results making it hard to assess the significance of the proposed approach. It is also unclear what the ML community gains with the results of this study. Please add such discussion to the introduction section.\n\n**Methodology**\n\n- This section would benefit the most from re-writing and restructuring. In the current form it is difficult to follow and do not allow to fully appreciate the presented ideas.\n\n- Figure 2 should be better discussed and better formatted. Please extend the caption to clarify the figure. In general, all figures in the paper should be self-explanatory making it possible to understand the figures just by reading the captions. In its current form it is impossible to understand the drawings.\n\n- Based on the description the process of updating the prototype weights into target weights is unclear. Could the authors clarify how different networks are updated?\n\n- Eq 4 and Eq 7 differ only in the dimensionality of w. Why changing the dimensionality leads to new paradigm of random weights padding? Moreover, please boldface vectors in Eq 7 to differentiate them further from scalars in Eq. 4.\n\n- The section lacks motivations behind different choices. \n\n**Results**\n\n- The introduced layer is not compared to previously published models. Would it be possible to compare the model to Supermaks and Popup? Adding comparisons to previous art would make the validation stronger.\n\n- Cifar datasets are small scale. Would the observations generalize to larger scale datasets? Adding another dataset would make the observations more compelling. \n\n- The results are missing stds, making it hard to assess the significance of the results.\n\n- In general, the proposed layers are underperforming w.r.t dense layers. That would be expected. However, the results section lacks discussion and positioning of the reported results, e.g., Why the reported results are interesting? What do we learn as a community from the results? What is the impact of the reported results? Adding more in-depth discussion would make the paper stronger.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept