Title: ATOMIC COMPRESSION NETWORKS

Abstract: Compressed forms of deep neural networks are essential in deploying large-scale computational models on resource-constrained devices. Contrary to analogous domains where large-scale systems are build as a hierarchical repetition of smallscale units, the current practice in Machine Learning largely relies on models with non-repetitive components. In the spirit of molecular composition with repeating atoms, we advance the state-of-the-art in model compression by proposing Atomic Compression Networks (ACNs), a novel architecture that is constructed by recursive repetition of a small set of neurons. In other words, the same neurons with the same weights are stochastically re-positioned in subsequent layers of the network. Empirical evidence suggests that ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks (88× to 1116× reduction) with only a fractional deterioration of classification accuracy (0.15% to 5.33%). Moreover our method can yield sub-linear model complexities and permits learning deep ACNs with less parameters than a logistic regression with no decline in classification accuracy.

1. INTRODUCTION:

The universe is composed of matter, a physical substance formed by the structural constellation of a plethora of unitary elements denoted as atoms. The type of an atom eventually defines the respective chemical elements, while structural bonding between atoms yields molecules (the building blocks of matter and our universe). In Machine Learning a neuron is the infinitesimal nucleus of intelligence (i.e. {atom, matter} ↔ {neuron, AI}), whose structural arrangement in layers produces complex intelligence models. Surprisingly, in contrast to physical matter where molecules often reuse quasi-identical atoms (i.e. repeating carbon, hydrogen, etc.), neural networks do not share the same neurons across layers. Instead, the neurons are parameterized through weights which are optimized independently for every neuron in every layer. Inspired by nature, we propose a new paradigm for constructing deep neural networks as a recursive repetition of a fixed set of neurons. Staying faithful to the analogy we name such models as Atomic Compression Networks (ACNs). Extensive experimental results show that by repeating the same set of neurons, ACNs achieve unprecedented compression in terms of the total neural network parameters, with a minimal compromise on the prediction quality.

Deep neural networks (DNN) achieve state-of-the-art prediction performances on several domains like computer vision (Huang et al., 2018; Tan & Le, 2019) and natural language processing (Vaswani et al., 2017; Gehring et al., 2017) . Therefore, considerable research efforts are invested in adopting DNNs for mobile, embedded, or Internet of Things (IoT) devices (Kim et al., 2015) . Yet, multiple technical issues related to restricted resources, w.r.t. computation and memory, prevent their straightforward application in this particular domain (Han et al., 2016; Samie et al., 2016; Mehta et al., 2018) . Even though prior works investigate neural compression techniques like pruning or low-rank parameter factorization, they face fragility concerns regarding the tuning of hyperparameters and network architecture, besides struggling to balance the trade-off between compression and accuracy (Cheng et al., 2017) .

Contributions: In a nutshell, this paper introduces:

• a novel compression paradigm for neural networks composed of repeating neurons as the atomic network components and further motivated by function composition;

• compression rates of up to three orders of magnitudes compared to a cross-validated fullyconnected network on nine real-world vector datasets;

• first work to achieve sub-linear model complexities measured in the number of trained parameters compared to connected architectures on several computer vision tasks.

2. RELATED WORK:

2.1 MODULAR NEURAL NETWORKS Our approach of training a set of neurons and (re-)using them in building the network architecture is partially related to the existing scheme of modular neural networks (MNN). End-to-End Module Networks (Andreas et al., 2016a; b; Hu et al., 2017) are deep neural network models that are constructed from manual module blueprints defined for different sub-tasks in question answering. The Compositional Recursive Learner proposed by Chang et al. ( 2018) employs a curriculum learning approach to learn modular transformations while Routing Networks (RN) (Rosenbaum et al., 2017) consist of a set of pre-defined modules (which each can be a NN) and a meta controller (called router) that decides in which order these modules are consecutively applied to a given input. Modular Networks (Kirsch et al., 2018) are an extension to RNs employing conditional computation with Expectation-Maximization. Finally the recent work of Cases et al. (2019) focuses on the recursive formulation of the Routing Network, consecutively applying one of a set of modules to an input. The crucial difference to our approach is that our neurons are much smaller than the general modules of MNN and that we reuse the same components on the same network level (e.g. within the same layer) while the modules of MNN are only applied sequentially. A different extension of the RN is the model of Zaremoodi et al. (2018) which uses the router in a gating mechanism to control the input to a set of shared RNN modules similar to a Mixture of Experts model (Jacobs et al., 1991) . Although Mixture of Experts models are related, they normally do not stack and recursively reuse components within the same network but have a comparably shallow architecture.

Another related field of research is that of neural architecture search (NAS) (Zoph & Le, 2016) .

It is concerned with the automatic discovery of high performing neural network architectures for diverse tasks. There exists a multitude of search-approaches including (Neuro-)Evolution (Stanley et al., 2009; Real et al., 2017) and Reinforcement Learning (Zoph & Le, 2016; Pham et al., 2018) . A sub-field of NAS is the dedicated search for neural modules called cells or blocks (Liu et al., 2018; Zoph et al., 2018; Pham et al., 2018) , which can be reused in the network architecture. Although this is related to our approach, the discovered cells are usually much more complex than single neurons and only share their architecture while each cell has a different set of weights, therefore not reducing the total number of required parameters. Although parameter sharing approaches exist, they either share weights between different evolved macro-architectures (Real et al., 2017; Elsken et al., 2018) or only use them to warm start cells with a parameter copy on initialization, which then is refined in subsequent fine tuning steps (Pham et al., 2018; Xie et al., 2018) , missing the advantage of the recursive sharing scheme employed by ACNs. Although recent works like Stamoulis et al. (2019) achieve huge speed-ups, the large amount of time required to find suitable architectures remains a major disadvantage, while our approach can be trained from scratch in one go.

2.2. NETWORK COMPRESSION:

One popular way for network compression is the pruning of unimportant weights from the network, an idea first appearing several decades ago to prevent over-fitting by reducing the network complexity (LeCun et al., 1990) . The seminal work by Han et al. (2015b) proposes an iterative procedure consisting of training and pruning weights with small magnitudes. Li et al. (2016) focus on pruning filters of trained CNNs based on the L 1 -norm while Louizos et al. (2017b) and Li & Ji (2019) consider L 0 -norm regularization. In contrast Luo et al. (2017) employ a pruning schedule on the channel with the smallest effect on the activations of the subsequent layer. Molchanov et al. (2016; 2019) use Taylor expansion to calculate importance scores for different channels to prune CNNs. Similarly neuron importance scores based on the reconstruction error of responses in the final layers are used by Yu et al. (2018) while Wang et al. (2018) propose an adaptive pruning scheme for sub-groups of weights. Furthermore there also exist recent Bayesian approaches to weight pruning (Molchanov et al., 2017; Louizos et al., 2017a; Ullrich et al., 2017) . Another path of research utilizes factorization approaches to produce low-rank approximations to the weight matrices of DNNs (Denton et al., 2014; Gong et al., 2014) . Alternatively vector quantization methods can be applied to the parameter matrices, e.g. dividing the weights into sub-groups which are represented by their centroid (Denil et al., 2013; Gong et al., 2014) . SqueezeNet introduced by Iandola et al. ( 2016) adapts so-called squeeze modules which reduce the number of channels while TensorNet (Novikov et al., 2015) uses the Tensor-Train format to tensorize the dense weight matrices of FC layers. Finally Wu (2016) use the Kronecker product to achieve layer-wise compression.

Recent work combines several of the aforementioned methods in hybrid models. Han et al. (2015a) use pruning together with layer-wise quantization and additional Huffman coding. In Kim et al. (2015) Tucker decomposition supported by a variational Bayesian matrix factorization is employed while Tung & Mori (2018) jointly apply pruning and quantization techniques in a single framework.

A special variant of compression methods for DNNs is the teacher-student approach also known as network distillation (Ba & Caruana, 2014; Hinton et al., 2015; Luo et al., 2016) , which utilizes the relation of two models, a fully trained deep teacher model and a significantly smaller and more shallow student model to distill the knowledge into a much more compressed form. There exist different approaches to the design and training of the student model, e.g. utilizing special regularization (Sau & Balasubramanian, 2016) or quantization techniques (Polino et al., 2018) .

A crucial advantage of ACNs is that we do not waste time and computational resources on training a large, over-parameterized architecture which is prone to over-fitting and usually needs a lot of training data only to compress it afterwards (e.g. by pruning or distillation). Instead we directly train smaller models which bootstrap their capacity by recursively-stacked neurons and therefore are more compact and can be trained efficiently with less data. Furthermore, the aforementioned approaches are often quite limited in achieving large compression without degrading the prediction performance.

HashedNets (Chen et al., 2015) use random weight sharing within the parameter matrix of each layer and introduce an additional hashing formulation of the forward pass and the gradient updates. Our model is different from these approaches since we share neurons which can have an arbitrary number of weights and don't restrict the sharing to the same layer, but enable sharing across layers, too.

As an additional complementary method to the aforementioned approaches one can train networks with reduced numerical precision of their parameters (Courbariaux et al., 2014; Gupta et al., 2015) . This achieves further compression and experiments show that it leads only to a marginal degradation of model accuracy. Taking this idea to the extreme Courbariaux et al. (2016) constrain the values of all weights and activations to be either +1 or -1.

3. ATOMIC COMPRESSION NETWORKS:

A neuron can be understood as an arbitrary function f (x; θ) : R D → R 1 defined by its parameters θ ∈ R D . The idea is to stochastically apply the same neuron with the same weights to the outputs of neurons from the previous layer. The different functions or components f i then form a global set M , f 1 , f 2 , ..., f m ∈ M which defines the set of parameters (indexed by the neurons) which is available to the model. Additionally we add the output layer f out which is a unique FC layer that is not reused anywhere else in the network (see figure 1 ).

The most naive approach to recursion is repeating whole layers with shared weights. Each module is represented by a FC layer mapping from a layer input dimension D in to the output dimension D out .

The architecture is depicted in figure 1a . More generally this approach can also be written in terms of a function composition (see equation 1). Sharing layers forces them to have equal input and output dimensions, resulting in rectangular-shaped neural architectures. For comparison with our neuron level ACN in the ablation study in the experiment section we reference this approach with random sampling of the sequential network composition by LayerNet.

g(x) = f m • f m-1 • ... • f 1 (x) , f 1 , f 2 , ..., f m ∈ M (1)

In contrast to repeating entire layers, ACN reuses the most atomic components of a neural network which are represented by its single neurons. Consequently a full network layer in the model consists of several neurons which do not need to be all different, but the same neuron can be reused more than once in the same layer (see figure 1b ). Each neuron p ∈ M has an input dimension d p which we choose randomly from a preset range of dimensions D. When inserting the neuron into a layer , we randomly sample d p connections to the neurons of the previous layer forming a mask δ

( )

p on its activation z. While the trainable parameters (weights and bias) of the neurons are shared, this connection mask is different for every neuron in every layer. The procedure for sampling the ACN architecture is shown in algorithm 1. 

M := {f 1 (•; θ 1 ) , . . . , f N (•; θ N )} , θ i ∈ R di , d i ∼ U (1, D) // Create network architecture 2 for = 1, ..., L do 3 M ( ) := f i (•; θ i ) | f i (•; θ i ) ∈ M ∧ d i < K ( -1) 4 for i = 1, ..., K ( ) do 5 f j (•; θ j ) ∼ M ( ) 6 m ( ) i (•) := f j (•; θ j ) 7 δ ( ) i ∼ U 1, K ( -1) dj 8 return m, δ

Equation 2 shows the forward pass through a ACN neuron layer, where Π represents the projection (selection) of the mask indices δ from the activation of the previous layers:

EQUATION

To reduce the additional memory complexity of these input masks we can store the seed of the pseudo random generator used for sampling the masks and are able to reproduce them on the fly in the forward pass at inference time. Since the input masks of the neurons are completely random, situations may arise in which not all elements of z are forwarded to the next layer. However in our experiments we find that this case happens only very rarely granted that the number of neurons per layer and the greater part of the input dimensions D are sufficiently large.

4. EXPERIMENTS:



4.1. PRELIMINARY STUDY OF MODULE RECURSION:

To gain more insights into the potential of ACN we perform a small study on a synthetic curve fitting problem. We compare a simple FC network with a varying number of layers and neurons to an ACN with approximately the same number of parameters. The curve fitting problem is a regression task on the function f (x) = 3.5x 3 -3 sin(x) + 0.5 sin(8x 2 + 0.5π) 2 . To achieve the best possible fit for the models we perform a hyper-parameter grid search including the dimension of hidden layers, learning rate and the number of different modules for the recursive net. Details can be found in appendix A.2. Considering our knowledge about the true function which includes powers of x we boost the performance of both models by squaring the neuron activations.

Figure 2 : Plots of the model fit to the curve data. The first row shows the fit of the FC baseline, the second that of ACN. In the columns the respective number of model parameters as well as the achieved MSE on the test set are given above the plots. Since an arbitrary number of parameters cannot always be achieved with each model, the next nearest number with a competitive model was selected, e.g. in the first column with 17 and 18 parameters respectively.

As can be seen in figure 2 , ACN consistently achieves a better fit in terms of MSE than the FC baseline. Already an ACN with only 32 parameters is sufficient to approximately fit even the highly non-linear parts of the function while the FC network only achieves a comparable fit with actually more than twice the number of parameters. This result can be explained by some intuition about function composition. Consider a function f (x) = (αx + β) 2 , f : R → R with sole parameters α and β. Then we can create much more complex functions with the same set of parameters by composition f (f (x)) = α 3 x 2 + 2α 2 βx + αβ 2 + β 2 . Extending this to other functions g, h, ...

(which each can be a neuron) enables us to create complex functions with a small set of parameters. Furthermore compared to a standard FCN our ACN achieves much deeper architectures with the same number of parameters what further improves the fitting capability.

4.2. EXPERIMENTAL SETUP:

We evaluate the performance of the proposed algorithms using nine publicly available real-world classification datasets and three image datasets. The selected datasets and their characteristics are detailed in appendix A.1. The datasets were chosen from the OpenML-CC18 benchmark1 which is a curated high quality version of the OpenML100 benchmark (Bischl et al., 2017) . Details on the employed hyper-parameters and training setup can be found in appendix A.2.

We compare our model to six different baseline models:

• FC: A standard FC network of comparable size. It was shown in Chen et al. (2015) that this is a simple but strong baseline which often outperforms more sophisticated methods. • RER: Random Edge Removal first introduced in Cires ¸an et al. (2011) . For this baseline a number of connections is randomly dropped from a FC network on initialization. In contrast to dropout (Srivastava et al., 2014) the selected connections are dropped completely and accordingly the network is trained and also evaluated as sparse version. • TensorNet (Novikov et al., 2015) : This model builds on a generalized low-rank quantization method that treats the weight matrices of FC layers as a two dimensional tensor which can be decomposed into the Tensor-Train (TT) format (Oseledets, 2011) . The authors also compare their model to HashedNet (Chen et al., 2015) and claim to outperform it by 1.2%.

To train TensorNet on our datasets we introduce two FC layers mapping from the input to the consecutive TT layers and from the last TT layer back to the output dimension. • BC: The Bayesian compression method of Louizos et al. (2017a) uses scale mixtures of normals as sparsity-inducing priors on groups of weights, extending the variational dropout approach of Molchanov et al. (2017) to completely prune unimportant neurons while training a model from scratch. We achieve varying levels of compression and accuracy by specifying different thresholds for the variational dropout rate of the model. • TP: Molchanov et al. (2019) use first and second order Taylor expansions to compute importance scores for filters and weights which then are used to prune respective parts of the network. In our experiments we use the method which the authors report as best, introducing pruning gates directly after batch normalization layers. To realize different levels of compression we prune increasingly bigger parts of the network based on the calculated importance scores in an iterative fine tuning procedure. • LogisticRegression: A linear model.

4.3. EXPERIMENTAL RESULTS:

The results of the experimental study on the real-world datsets are shown in figure 3 while table 1 shows the results for the image datasets. All results are obtained by running each model-hyperparameter combination with three different random seeds and averaging the resulting performance metric and number of parameters. The standard deviation between runs can be found in table 5 in appendix A.3.

Table 1 : Classification test accuracy on image datasets -in each row the best model (selected based on validation accuracy) up to the specified maximum number of parameters is shown. Therefore the model and its test accuracy values do not change for greater numbers of parameters, if a smaller model has a better validation accuracy. For some models very small versions could not be trivially achieved (indicated by "-"). All trained models are fully connected architectures. We did not train ACN for more than 500,000 parameters. As can be seen in figure 3 , ACN consistently outperforms all other models on all of the nine real-world datasets for very small models, for datasets with a huge feature space even for models with up to 10,000 parameters. This shows that the ACN architecture is very efficient w.r.t. its parameters and the required amount of training-data and that on these datasets the recursive nature and parameter sharing throughout the architecture has a positive impact on the predictive efficiency compared to the number of parameters. For larger models the advantage of ACN decreases and the other models catch Figure 3 : Plots of the model classification error on the test set for different numbers of parameters. For each tick on the x-axis the model with the best validation accuracy under the threshold for the number of parameters is selected. In the titles next to the name of the dataset we indicate the dimension of the respective feature space. For some methods very small models could not be achieved so they have no points plotted on the left part of the plots (e.g. a fully connected network will not have less parameters than a linear model with number of inputs × number of classes many parameters)

up or even outperform it by a small margin, e.g. in case of the InternetAds dataset, however ACN remains competitive. TensorNet usually is at least a factor of two larger than the other models and therefore is only shown in the last bin. The same applies for TP and BC on the optdigits, theorem and nomao/spambase datasets respectively.

To evaluate the parameter efficiency of our technique we compare ACN and FC (as the best performing baseline) to a large optimal FC network trained on the same dataset (doing a hyper-parameter search with the same parameters mentioned in section 4.2 and five increasingly large architectures). We show the results regarding the relative compression rate and the respective loss in model performance in table 2. Say n is the number of model parameters of the large baseline and n that of the evaluated model, then the compression rate is given by ρ = n/n . Our model achieves compression rates of 88 up to more than 1100 times while the loss in test accuracy is kept at around 0.15% up to 5.33%. In comparison FC achieves compression of 67 up to 528 times with accuracy loss between 0.28% and 7.13%. For all datasets except spambase and theorem ACNs realize higher compression rates than FC as well as a smaller loss in accuracy in 5/9 cases. However since the comparison to a large model baseline can be biased by model selection and the thoroughness of the respective hyper-parameter search, we also investigate how the performance and capacity of the models compare w.r.t. a linear baseline. The results in table 2 confirm the trend that is observed in the comparison to the large model. On the image datasets ACN consistently and significantly outperforms all other models for up to 100,000 parameters. On MNIST our smallest model has 4091 parameters and outperforms a linear model with 7850 by 1.8% although the baseline has nearly twice as many parameters. The same trend can be observed for FashionMNIST (7057 / 7850 with a difference of 2.04%) and CIFAR10 where an ACN with 7057 parameters employs less than a quarter of the 30730 parameters needed by the simplest linear model and outperfoms it by a overwhelming 8.63%. Surprisingly the LayerNet outperfoms all other models for the category < 500, 000. For more than 500,000 parameters the FC network performs best. Although the results on CIFAR10 first might seem rather poor compared to the approximately 96% accuracy which is achieved by CNN (Zagoruyko & Komodakis, 2016) , the best results achieved by dedicated FC networks lie around 56% and are only topped by a specifically designed network with linear bottleneck layers and unsupervised pre-training (Lin et al., 2015) . While our model sacrifices some processing time for higher compression, the forward pass at inference time still works in a matter of milliseconds on a standard CPU.

4.4. ABLATION STUDY ON NEURON-RECURSION:

To investigate the effect of the chosen level of sharing and recursion we perform a small ablation study. We compare ACN with parameter-sharing and recursion on a neuron-level with the simpler approach of sharing and reusing whole layers described as the LayerNet architecture in section 3. Therefore we also report the performance for the LayerNet in an extra column in the tables 5 and 1. The results imply that in general the recursion on neuron-level is much more effective to achieve competitive models with high compression, outperforming the LayerNet in all but one case for less than 10,000 parameters. However the LayerNet seems to have a beneficial regularization effect on large models, what is notable especially on the image datasets.

5. CONCLUSION:

In this paper we presented Atomic Compression Networks (ACN), a new network architecture which recursively reuses neurons throughout the model. We evaluate our model on nine vector and three image datasets where we achieve promising results regarding the compression rate and the loss in model accuracy. In general ACNs achieve much tinier models with only a small to moderate decrease of accuracy compared to six other baselines. For future work we plan to include skip connections in the architecture and to extend the idea to CNNs and the sharing of kernel parameters as well as for the FC layers. Another interesting path of research is the combination of the ACN scheme with NAS methods to further optimize the efficiency and performance of the created architectures.

A APPENDIX:

A.1 DATASET DETAILS Table 3 shows detailed attributes for the datasets selected from the OpenML-CC18 benchmark. The selection criteria were datasets with more than 3000 instances and more than 50 original features.

The number of features reported in the table is the number of features after one-hot-encoding. For the image datasets ( 10)-( 12) we employ the original train/test splits and use a random 20% of the training set for validation leading to a train/val/test split of 0.64/0.16/0.2. The datasets (1)-( 9) are split randomly using the same fractions. We do not use any kind of additional transformation or data augmentation beyond those which was already originally applied to the respective datasets. Given the large computational demands of deep learning architectures, a proper full hyper-parameter search is infeasible for the models employed on the real-world datasets. Under these circumstances we follow the established trend of using some specific deep architectures designed based on expert knowledge together with a small selected parameter grid containing learning rate (0.01, 0.001), use of batch normalization (Ioffe & Szegedy, 2015) and dropout probability (0.0, 0.3). To enable a fair comparison the hyper-parameter search for every model type which is trained from scratch is done for the same number of combinations. Accordingly for each dataset we evaluate eight different architectures per model type where we vary the number of hidden layers and the width of these layers, for ACN M and D are different between some of the eight varying architectures. The size of the set of modules M is varied in the range between 3 and 5 for the LayerNet (ablation study) and between 16 and 512 for the ACN. The range of input dimensions D is set to a subset of (2, 4, 8, 16, 32, 64, 128) depending on the maximum width of the used hidden layers and the respective input dimension. In case of the TensorNet we vary the dimension of the tensor-train layers and the length of the tensor-train decomposition between different architectures per dataset together with the aforementioned parameter grid. All other parameters remain at their default values (Novikov et al., 2015) . For the two pruning approaches BC and TP we train two different large FC architectures with the described hyper-parameter grid and then prune the networks in small steps, selecting the best architecture within every parameter bin. The models are trained for 25 epochs on batches of size 64 for the real-world datasets and 128 for the image datasets. In order to minimize the loss we apply stochastic gradient descent update steps using Adam (Kingma & Ba, 2014) and halve the initial learning rate every 10 epochs. The only exception is Tp where we use SGD with a momentum term of 0.99 as proposed by the authors but employ the same learning rate schedule. Finally, in order to avoid exploding gradients the gradient norm is truncated to a maximum of 10. Apart from the TensorNet for which we adapt the publicly available Tensorflow Code2 , all models are implemented using the PyTorch 1.1 framework. For BC3 and TP4 we adapt the public PyTorch code base provided by the authors.

A.3 ADDITIONAL RESULTS:

In this section we show the results presented in figure 3 in tabular form in table 5 . Furthermore for completeness we present additional results regarding two simple sparsification baselines in table 6 employing L1-regularization (L1) and iterative hard tresholding (L1+HT) (Han et al., 2015b) but without explicit cardinality constraint (Jin et al., 2016) . The additional baseline models perform mostly on par with the small FC baseline, sometimes slightly better or slightly worst. The biggest change can be seen on the splice dataset where the additional baselines perform better than FC and ACN in the three bins with the highest number of parameters. The additional baselines were trained and evaluated following the same experimentation and hyperparameter protocol as the other models. 

:

https://docs.openml.org/benchmark/#openml-cc18

https://github.com/timgaripov/TensorNet-TF

https://github.com/KarenUllrich/Tutorial BayesianCompressionForDL

https://github.com/NVlabs/Taylor pruning

Figures and Tables:

Figure 1: (a) LayerNet architecture; (b) ACN with single neuron components. The input masks are shown by colored arrows to respective neurons, where f 1 (red) has 2, f 2 (orange) has 3 and f m (green) has 4 inputs. The output layer f out (blue) is an independent module which is not reused.

ACN Architecture Sampling input :D, number of neurons N , number of layers L, layer sizes K ∈ N L output :ACN architecture m, δ // Create neurons with random dimensions 1

Compression and accuracy change compared to a large FC model and a linear model baseline

Table content: <table><tr><td/><td/><td colspan="2">Compared to large FC</td><td/><td/><td colspan="3">Compared to Linear Model</td></tr><tr><td>Dataset</td><td colspan="2">small FC</td><td colspan="2">small ACN</td><td colspan="2">small FC</td><td colspan="2">small ACN</td></tr><tr><td/><td>ρ</td><td>∆Acc(%)</td><td>ρ</td><td>∆Acc(%)</td><td>ρ</td><td>∆Acc(%)</td><td>ρ</td><td>∆Acc(%)</td></tr><tr><td>1) bioresponse</td><td>67.41</td><td>-0.28</td><td>322.15</td><td>-1.18</td><td>0.998</td><td>+0.70</td><td>4.771</td><td>-0.90</td></tr><tr><td>2) HAR</td><td>155.59</td><td>-7.13</td><td>361.82</td><td>-4.40</td><td>2.942</td><td>-6.25</td><td>6.840</td><td>-3.50</td></tr><tr><td colspan="2">3) InternetAds 133.23</td><td>-1.09</td><td>1115.58</td><td>-2.66</td><td>0.998</td><td>+0.31</td><td>8.360</td><td>-1.26</td></tr><tr><td>4) isolet</td><td>74.49</td><td>-3.71</td><td>87.76</td><td>-3.45</td><td>3.042</td><td>-2.86</td><td>3.58</td><td>-2.60</td></tr><tr><td>5) nomao</td><td>218.67</td><td>-0.74</td><td>460.36</td><td>-1.43</td><td>0.972</td><td>+1.37</td><td>2.047</td><td>+0.68</td></tr><tr><td>6) optdigits</td><td>149.12</td><td>-6.15</td><td>168.06</td><td>-5.33</td><td>1.879</td><td>-4.41</td><td>2.117</td><td>-3.59</td></tr><tr><td>7) spambase</td><td>129.59</td><td>-2.12</td><td>92.46</td><td>-0.15</td><td>0.951</td><td>+0,23</td><td>0.678</td><td>+2.20</td></tr><tr><td>8) splice</td><td>211.46</td><td>-3.47</td><td>316.12</td><td>-5.33</td><td>1.467</td><td>-1.73</td><td>2.193</td><td>-3.59</td></tr><tr><td>9) theorem</td><td>97.47</td><td>-6.50</td><td>95.30</td><td>-4.03</td><td>0.647</td><td>+2.14</td><td>0.633</td><td>+4.61</td></tr></table>

Description of datasets

Table content: <table><tr><td>Dataset</td><td colspan="4"># Instances # Features # Classes % Minor. Cl.</td></tr><tr><td>(1) bioresponse (Kaggle, 2012)</td><td>3751</td><td>1776</td><td>2</td><td>45.77</td></tr><tr><td>(2) HAR (Anguita et al., 2013)</td><td>10299</td><td>561</td><td>6</td><td>13.65</td></tr><tr><td>(3) InternetAds (Kushmerick, 1999)</td><td>3279</td><td>3113</td><td>2</td><td>14.00</td></tr><tr><td>(4) isolet (Fanty &amp; Cole, 1991)</td><td>7797</td><td>617</td><td>26</td><td>3.82</td></tr><tr><td>(5) nomao (Candillier &amp; Lemaire, 2012)</td><td>34465</td><td>174</td><td>2</td><td>28.56</td></tr><tr><td>(6) optdigits (Dua &amp; Graff, 2017)</td><td>5620</td><td>64</td><td>10</td><td>9.86</td></tr><tr><td>(7) spambase (Dua &amp; Graff, 2017)</td><td>4601</td><td>57</td><td>2</td><td>39.40</td></tr><tr><td>(8) splice (Noordewier et al., 1991)</td><td>3190</td><td>287</td><td>3</td><td>24.04</td></tr><tr><td>(9) theorem (Bridge et al., 2014)</td><td>6118</td><td>51</td><td>6</td><td>7.94</td></tr><tr><td>(10) MNIST (LeCun et al., 1998)</td><td>70000</td><td>784</td><td>10</td><td>9.02</td></tr><tr><td>(11) FashionMNIST (Xiao et al., 2017)</td><td>70000</td><td>784</td><td>10</td><td>10.00</td></tr><tr><td>(12) CIFAR10 (Krizhevsky et al., 2009)</td><td>60000</td><td>3072</td><td>10</td><td>10.00</td></tr></table>

Hyper-Parameters for Preliminary Study (4.1)

Table content: <table><tr><td>Hyper-Parameter</td><td colspan="2">SearchGrid FC SearchGrid ACN</td></tr><tr><td>hidden layer size</td><td>2, 3, 4, 5, 8</td><td>3, 4, 5, 8</td></tr><tr><td>learning rate</td><td>0.1, 0.05, 0.01</td><td>0.1, 0.01</td></tr><tr><td>number of layers</td><td>1, 2, 3</td><td>2, 3, 4</td></tr><tr><td>max gradient norm</td><td>0.99, 1, 2, 5</td><td>0.99, 1, 2</td></tr><tr><td>use batch norm</td><td>True, False</td><td>True, False</td></tr><tr><td>number of modules</td><td>-</td><td>2, 4, 8</td></tr><tr><td colspan="3">HYPER-PARAMETER AND TRAINING SETUP FOR REAL-WORLD DATASETS</td></tr></table>

Test accuracy and its standard deviation on real-world datasets. The standard deviation of runs with FC, RER, BC and TP models are always below the displayed three decimals precision and therefore are not shown in the table.

Table content: <table><tr><td>Capacity</td><td>FC</td><td>RER</td><td>TensorNet</td><td>BC</td><td>TP</td><td>LayerNet</td><td>ACN</td></tr><tr><td/><td/><td/><td/><td colspan="2">(1) bioresponse</td><td/><td/></tr><tr><td>&lt; 1000</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.787 (±0.005)</td></tr><tr><td>&lt; 2500</td><td>-</td><td>0.546</td><td>-</td><td>-</td><td>0.548</td><td>-</td><td>0.787 (±0.005)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.803 0.570</td><td>-</td><td colspan="3">0.685 0.548 0.775 (±0.008)</td><td>0.797 (±0.005)</td></tr><tr><td colspan="3">&lt; 10000 0.793 0.628</td><td>-</td><td colspan="3">0.776 0.744 0.777 (±0.003)</td><td>0.797 (±0.005)</td></tr><tr><td colspan="7">≥ 10000 0.793 0.609 0.784 (±0.013) 0.784 0.797 0.800 (±0.004)</td><td>0.797 (±0.005)</td></tr><tr><td/><td/><td/><td/><td>(2) HAR</td><td/><td/><td/></tr><tr><td>&lt; 500</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.940 (±0.001)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.913 0.420</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.967 (±0.003)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.971 0.669</td><td>-</td><td>-</td><td colspan="2">0.171 0.763 (±0.108)</td><td>0.977 (±0.005)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.984 0.764</td><td>-</td><td colspan="3">0.974 0.438 0.943 (±0.044)</td><td>0.985 (±0.004)</td></tr><tr><td>≥ 5000</td><td colspan="6">0.983 0.764 0.975 (±0.001) 0.981 0.950 0.983 (±0.002)</td><td>0.989 (±0.002)</td></tr><tr><td/><td/><td/><td/><td colspan="2">(3) InternetAds</td><td/><td/></tr><tr><td>&lt; 1000</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.956 (±0.012)</td></tr><tr><td>&lt; 2500</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.842</td><td>-</td><td>0.970 (±0.002)</td></tr><tr><td>&lt; 5000</td><td>-</td><td>0.842</td><td>-</td><td>-</td><td>0.842</td><td>-</td><td>0.972 (±0.005)</td></tr><tr><td colspan="3">&lt; 10000 0.972 0.842</td><td>-</td><td colspan="3">0.964 0.842 0.929 (±0.062)</td><td>0.972 (±0.005)</td></tr><tr><td colspan="7">≥ 10000 0.977 0.909 0.902 (±0.002) 0.967 0.908 0.978 (±0.004)</td><td>0.972 (±0.005)</td></tr><tr><td/><td/><td/><td/><td>(4) isolet</td><td/><td/><td/></tr><tr><td>&lt; 2500</td><td colspan="2">0.494 0.254</td><td>-</td><td>-</td><td>-</td><td>0.420 (±0.124)</td><td>0.883 (±0.017)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.842 0.414</td><td>-</td><td colspan="3">0.480 0.141 0.773 (±0.030)</td><td>0.941 (±0.001)</td></tr><tr><td>&lt; 7500</td><td colspan="2">0.939 0.645</td><td>-</td><td colspan="3">0.893 0.188 0.918 (±0.009)</td><td>0.948 (±0.002)</td></tr><tr><td colspan="3">&lt; 10000 0.939 0.645</td><td>-</td><td colspan="3">0.934 0.188 0.918 (±0.009)</td><td>0.948 (±0.002)</td></tr><tr><td colspan="7">≥ 10000 0.941 0.645 0.917 (±0.006) 0.949 0.932 0.949 (±0.002)</td><td>0.948 (±0.002)</td></tr><tr><td/><td/><td/><td/><td>(5) nomao</td><td/><td/><td/></tr><tr><td>&lt; 250</td><td>-</td><td>0.771</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.950 (±0.003)</td></tr><tr><td>&lt; 500</td><td colspan="2">0.957 0.778</td><td>-</td><td>-</td><td>-</td><td>0.953 (±0.001)</td><td>0.954 (±0.001)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.958 0.840</td><td>-</td><td>-</td><td>-</td><td>0.955 (±0.001)</td><td>0.959 (±0.001)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.960 0.901</td><td>-</td><td>0.949</td><td>-</td><td>0.960 (±0.001)</td><td>0.961 (±0.003)</td></tr><tr><td>≥ 2500</td><td colspan="6">0.961 0.901 0.955 (±0.001) 0.960 0.940 0.962 (±0.001)</td><td>0.961 (±0.003)</td></tr><tr><td/><td/><td/><td/><td>(6) optdigits</td><td/><td/><td/></tr><tr><td>&lt; 250</td><td colspan="2">0.714 0.552</td><td>-</td><td>-</td><td>-</td><td>0.560 (±0.118)</td><td>0.840 (±0.012)</td></tr><tr><td>&lt; 500</td><td colspan="2">0.931 0.814</td><td>-</td><td>-</td><td>-</td><td>0.885 (±0.006)</td><td>0.939 (±0.011)</td></tr><tr><td>&lt; 750</td><td colspan="2">0.977 0.881</td><td>-</td><td>-</td><td>-</td><td>0.885 (±0.006)</td><td>0.963 (±0.004)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.977 0.881</td><td>-</td><td>-</td><td>-</td><td>0.966 (±0.005)</td><td>0.969 (±0.002)</td></tr><tr><td>≥ 1000</td><td colspan="6">0.984 0.881 0.965 (±0.004) 0.936 0.775 0.984 (±0.003)</td><td>0.980 (±0.004)</td></tr><tr><td/><td/><td/><td/><td>(7) spambase</td><td/><td/><td/></tr><tr><td>&lt; 250</td><td colspan="2">0.921 0.758</td><td>-</td><td>-</td><td>-</td><td>0.927 (±0.012)</td><td>0.941 (±0.003)</td></tr><tr><td>&lt; 500</td><td colspan="2">0.941 0.835</td><td>-</td><td>-</td><td>-</td><td>0.936 (±0.004)</td><td>0.943 (±0.008)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.945 0.835</td><td>-</td><td>-</td><td colspan="2">0.615 0.936 (±0.004)</td><td>0.948 (±0.004)</td></tr><tr><td>≥ 1000</td><td colspan="6">0.945 0.835 0.729 (±0.006) 0.915 0.714 0.939 (±0.007)</td><td>0.948 (±0.004)</td></tr><tr><td/><td/><td/><td/><td>(8) splice</td><td/><td/><td/></tr><tr><td>&lt; 500</td><td>-</td><td>0.549</td><td>-</td><td>-</td><td>-</td><td>-</td><td>0.917 (±0.025)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.936 0.596</td><td>-</td><td>-</td><td>-</td><td>0.947 (±0.013)</td><td>0.949 (±0.006)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.964 0.648</td><td>-</td><td colspan="3">0.533 0.533 0.954 (±0.006)</td><td>0.963 (±0.013)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.964 0.648</td><td>-</td><td colspan="3">0.957 0.632 0.962 (±0.004)</td><td>0.963 (±0.013)</td></tr><tr><td>≥ 5000</td><td colspan="6">0.964 0.648 0.902 (±0.036) 0.969 0.906 0.965 (±0.003)</td><td>0.963 (±0.013)</td></tr><tr><td/><td/><td/><td/><td>(9) theorem</td><td/><td/><td/></tr><tr><td>&lt; 250</td><td colspan="2">0.486 0.430</td><td>-</td><td>-</td><td>-</td><td>0.445 (±0.006)</td><td>0.500 (±0.013)</td></tr><tr><td>&lt; 500</td><td colspan="2">0.511 0.460</td><td>-</td><td>-</td><td>-</td><td>0.474 (±0.026)</td><td>0.535 (±0.007)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.536 0.460</td><td>-</td><td>-</td><td>-</td><td>0.526 (±0.004)</td><td>0.540 (±0.007)</td></tr><tr><td>≥ 1000</td><td colspan="6">0.536 0.460 0.520 (±0.003) 0.535 0.503 0.542 (±0.004)</td><td>0.553 (±0.013)</td></tr></table>

Test accuracy and its standard deviation on real-world datasets for the additional sparsification baselines. The standard deviation of runs with FC, L1 and L1+HT models are always below the displayed three decimals precision and therefore are not shown in the table.

Table content: <table><tr><td>Capacity</td><td>FC</td><td>L1</td><td>L1+HT</td><td>ACN</td></tr><tr><td/><td/><td colspan="2">(1) bioresponse</td><td/></tr><tr><td>&lt; 1000</td><td>-</td><td>0.548</td><td>0.548</td><td>0.787 (±0.005)</td></tr><tr><td>&lt; 2500</td><td>-</td><td>0.548</td><td>0.777</td><td>0.787 (±0.005)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.803 0.767</td><td>0.791</td><td>0.797 (±0.005)</td></tr><tr><td colspan="3">&lt; 10000 0.793 0.788</td><td>0.791</td><td>0.797 (±0.005)</td></tr><tr><td colspan="3">≥ 10000 0.793 0.788</td><td>0.791</td><td>0.797 (±0.005)</td></tr><tr><td/><td/><td colspan="2">(2) HAR</td><td/></tr><tr><td>&lt; 500</td><td>-</td><td>-</td><td>-</td><td>0.940 (±0.001)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.913 0.181</td><td>0.193</td><td>0.967 (±0.003)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.971 0.181</td><td>0.959</td><td>0.977 (±0.005)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.984 0.981</td><td>0.967</td><td>0.985 (±0.004)</td></tr><tr><td>≥ 5000</td><td colspan="2">0.983 0.981</td><td>0.975</td><td>0.989 (±0.002)</td></tr><tr><td/><td/><td colspan="2">(3) InternetAds</td><td/></tr><tr><td>&lt; 1000</td><td>-</td><td>0.916</td><td>0.913</td><td>0.956 (±0.012)</td></tr><tr><td>&lt; 2500</td><td>-</td><td>0.916</td><td>0.966</td><td>0.970 (±0.002)</td></tr><tr><td>&lt; 5000</td><td>-</td><td>0.969</td><td>0.966</td><td>0.972 (±0.005)</td></tr><tr><td colspan="3">&lt; 10000 0.972 0.977</td><td>0.966</td><td>0.972 (±0.005)</td></tr><tr><td colspan="3">≥ 10000 0.977 0.977</td><td>0.966</td><td>0.972 (±0.005)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.494 0.033</td><td>0.488</td><td>0.883 (±0.017)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.842 0.033</td><td>0.933</td><td>0.941 (±0.001)</td></tr><tr><td>&lt; 7500</td><td colspan="2">0.939 0.938</td><td>0.933</td><td>0.948 (±0.002)</td></tr><tr><td colspan="3">&lt; 10000 0.939 0.938</td><td>0.953</td><td>0.948 (±0.002)</td></tr><tr><td colspan="3">≥ 10000 0.941 0.938</td><td>0.953</td><td>0.948 (±0.002)</td></tr><tr><td/><td/><td colspan="2">(5) nomao</td><td/></tr><tr><td>&lt; 250</td><td>-</td><td>-</td><td>-</td><td>0.950 (±0.003)</td></tr><tr><td>&lt; 500</td><td>0.957</td><td>-</td><td>-</td><td>0.954 (±0.001)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.958 0.718</td><td>0.718</td><td>0.959 (±0.001)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.960 0.952</td><td>0.949</td><td>0.961 (±0.003)</td></tr><tr><td>≥ 2500</td><td colspan="2">0.961 0.952</td><td>0.951</td><td>0.961 (±0.003)</td></tr><tr><td/><td/><td colspan="2">(6) optdigits</td><td/></tr><tr><td>&lt; 250</td><td>0.714</td><td>-</td><td>-</td><td>0.840 (±0.012)</td></tr><tr><td>&lt; 500</td><td>0.931</td><td>-</td><td>-</td><td>0.939 (±0.011)</td></tr><tr><td>&lt; 750</td><td>0.977</td><td>-</td><td>0.104</td><td>0.963 (±0.004)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.977 0.104</td><td>0.961</td><td>0.969 (±0.002)</td></tr><tr><td>≥ 1000</td><td colspan="2">0.984 0.979</td><td>0.985</td><td>0.980 (±0.004)</td></tr><tr><td/><td/><td colspan="2">(7) spambase</td><td/></tr><tr><td>&lt; 250</td><td>0.921</td><td>-</td><td>-</td><td>0.941 (±0.003)</td></tr><tr><td>&lt; 500</td><td>0.941</td><td>-</td><td>-</td><td>0.943 (±0.008)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.945 0.733</td><td>0.739</td><td>0.948 (±0.004)</td></tr><tr><td>≥ 1000</td><td colspan="2">0.945 0.919</td><td>0.922</td><td>0.948 (±0.004)</td></tr><tr><td/><td/><td colspan="2">(8) splice</td><td/></tr><tr><td>&lt; 500</td><td>-</td><td>-</td><td>-</td><td>0.917 (±0.025)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.936 0.533</td><td>0.533</td><td>0.949 (±0.006)</td></tr><tr><td>&lt; 2500</td><td colspan="2">0.964 0.889</td><td>0.974</td><td>0.963 (±0.013)</td></tr><tr><td>&lt; 5000</td><td colspan="2">0.964 0.971</td><td>0.974</td><td>0.963 (±0.013)</td></tr><tr><td>≥ 5000</td><td colspan="2">0.964 0.976</td><td>0.977</td><td>0.963 (±0.013)</td></tr><tr><td/><td/><td colspan="2">(9) theorem</td><td/></tr><tr><td>&lt; 250</td><td>0.486</td><td>-</td><td>-</td><td>0.500 (±0.013)</td></tr><tr><td>&lt; 500</td><td>0.511</td><td>-</td><td>-</td><td>0.535 (±0.007)</td></tr><tr><td>&lt; 1000</td><td colspan="2">0.536 0.422</td><td>0.422</td><td>0.540 (±0.007)</td></tr><tr><td>≥ 1000</td><td colspan="2">0.536 0.488</td><td>0.493</td><td>0.553 (±0.013)</td></tr></table>

Peer review (iclr2020):
This paper explores the use of replicating neurons across and within layers to compress fully connected neural networks. The idea is simple, and is evaluated on a number of datasets and compared with fully connected, single layer, and several compression schemes. Strengths: a lot of nice experiments with clearly advantageous results are given. Weaknesses: One obvious baseline missing is sparse compression, which can be achieved using either l1 regularization, or hard thresholding + fine tuning, both of which are easy to implement and appear in several works, e.g. Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization (Yang, Ruozzi, Gogate) Training skinny deep neural networks with iterative hard thresholding methods (Yin, Yuan, Feng, Yan) ... many others just via googling ... Also, I think this work should be compared with compression schemes that work via kronecker product, which seem very similar to this scheme (but where the kronecker matrix is binary to produce replication) Compression of Fully-Connected Layer in Neural Network by Kronecker Product (Zhou, Wu) (more via google) One obvious advantage of replication over kronecker product is lower complexity, but nonetheless, the methods belong in a similar family. Otherwise, I think the work makes sense, the idea is nice, and the results show promise! After rebuttal: I have read the rebuttal and the authors have basically addressed all my concerns. It is a bit disappointing that simple L1 regularization can give competitive results, but the fact that the authors are willing to do the experiment and incorporate the results convinces me that there*s nothing being hidden here, and the reader can make a fair and informed conclusion, so I have no more complaints.