Atomic Compression Networks
Authors: Jonas Falkner
Josif Grabocka
Lars Schmidt-Thieme
Venue:  (2020)
Keywords: Network Compression

Abstract: Compressed forms of deep neural networks are essential in deploying large-scale
computational models on resource-constrained devices. Contrary to analogous
domains where large-scale systems are build as a hierarchical repetition of small-
scale units, the current practice in Machine Learning largely relies on models with
non-repetitive components. In the spirit of molecular composition with repeating
atoms, we advance the state-of-the-art in model compression by proposing Atomic
Compression Networks (ACNs), a novel architecture that is constructed by recursive
repetition of a small set of neurons. In other words, the same neurons with the
same weights are stochastically re-positioned in subsequent layers of the network.
Empirical evidence suggests that ACNs achieve compression rates of up to three
orders of magnitudes compared to fine-tuned fully-connected neural networks (88×
to 1116× reduction) with only a fractional deterioration of classification accuracy
(0.15% to 5.33%). Moreover our method can yield sub-linear model complexities
and permits learning deep ACNs with less parameters than a logistic regression
with no decline in classification accuracy.

Under review as a conference paper at ICLR 2020
ATOMIC COMPRESSION NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Compressed forms of deep neural networks are essential in deploying large-scale computational models on snource-constraing devices. Contrary to analogous domains where large-scale systems are build as hierarchical repetition of small- scale units, the current practice in Machine Learning largely relies on models with non-repetitive components. In the spirit of molecular composition with repeating atoms, we advance the state-of-the-ar in model compression by proposing Atomic Compression Networks (ACNs), novel architecture that constructed by recursive repetition of a small set of neurons. In other words, the same neurons with the same weights are stochastically re-positioned in subsequent layers of the network. Empirical evidence suggests that ACNs achieve compression rates of to three orders of magnitudes compared fine-tuned fully-connected neural networks (88x to 11 1166x reduction) with only a fractional deterioration of classification accuracy (0.15% to 5.33%). Moreover our method can yield sub-linean model complexities and permits learning deep ACNs with less parameters than a logistic regression with no decline in classification accuracy.
INTRODUCTION
The universe is composed of matter, a physical substance formed by the structural constellation of a plethora of unitary elements denoted as atoms. The type of an atom eventually defines the respective chemical elements, while structural bonding between atoms yields molecules (the building blocks of matter and our universe). In Machine Learning a neuron is the infinitesimal nucleus of intelligence (i.e. {atom, matter) <+ (neuron, AI)), whose structural arrangement in layers produces complex intelligence models. Surprisingly. in contrast to physical matter where molecules often reuse quasi-identical atoms (i.e. repeating carbon, hydrogen, etc.), neural networks do not share the same neurons across layers. Instead, the neurons are parameterized through weights which are optimized independently for every neuron in every layer. Inspired by nature, we propose a new paradigm for constructing deep neural networks as a recursive repetition of fixed set of neurons. Staying faithful to the analogy we name such models as Atomic Compression Networks (ACNs). Extensive experimental results show that by repeating the same set of neurons, ACNs achieve unprecedented compression in terms of the total neural network parameters, with a minimal compromise on the prediction quality.
Deep neural networks (DNN) achieve sfate-of-thhe-eeee art prediction performances on several domains like computer vision (Huang et al. 2018 Tan & Le 2019) and natural language processing (Vaswani et al. 2017; Gehring et 2017). Therefore, considerable research efforts are invested in adopting DNNs for mobile, embedded, or Internet of Things (IoT) devices (Kim et al. (0015). Yet, multiple technical issues related to restricted resources, w.r.t. computation and memory, prevent their straightforward application in this particular domain (Han et al. 2016 Samie et al. 2016 Mehta et al.) 2018). Even though prior works investigate neural compression techniques like pruning or low-rank parameter factorization, they face fragility concerns regarding the tuning of hyperparameters and network architecture, besides struggling to balance the trade-off between compression and accuracy (Cheng et al. 2017).
Contributions: In a nutshell, this paper introduces:
a novel compression paradigm for neural networks composed of repeating neurons as the atomic network components and further motivated by function composition;

---

Under review as a conference paper at ICLR 2020
compression rates of up to three orders of magnitudes compared to a cross-validated fully- connected network on nine real-world vector datasets;
first work to achieve sub-linear model complexities measured in the number of trained parameters compared to connected architectures on several computer vision tasks.
2
RELATED WORK
2.1 MODULAR NEURAL NETWORKS
Our approach of training set of neurons and (re-)using them in building the network architecture partially related to the existing scheme of modular neural networks (MNN). End-to-End Module Networks (Andreas et al. 2016a b Hu et al. (017)) are deep neural network models that are con- structed from manual module blueprints defined for different sub-tasks in question answering. The Compositional Recursive Learner proposed by Chang et al. (2018) employs a curriculum learning approach to learn modular transformations while Routing Networks (RN) (Rosenbaum et al. 2017) consist of a set of pre-defined modules (which each can be NN) and meta controller (called router) that decides in which order these modules are consecutively applied to a given input. Modular Networks (Kirsch et al., 2018) are extension RNs employing conditional computation with Expectation- formulation of the Routing Network, consecutively applying one of set of modules to an input. The crucial difference to our approach is that our neurons are much smaller than the general modules of MNN and that we reuse the same components on the same network level (e.g. within the same layer) while the modules of MNN are only applied sequentially. A different extension of the RN is the model Zaremoodi al. (2018) which uses the router in a gating mechanism to control the input to a set of shared RNN modules similar to a Mixture of Experts model (Jacobs et al.) 1991). Although Mixture of Experts models are related, they normally do not stack and recursively reuse components within the same network but have comparably shallow architecture.
Expectation-Maxiiizaatte Maximization. Finally the recent work of Cases et al. (2019) focuses on the recursive
Another related field of research is that of neural architecture search (NAS) (Zoph & Le 2016).
It is concerned with the automatic discovery of high performing neural network architectures for diverse tasks. There exists a multitude of aaprchaappooccces including
(Stanley 2009 Real et al. 2017) and Reinforcement Learning (Zoph & Le 2016 Pham et al.) 2018). A sub-field of NAS is the dedicated search for neural modules called cells or blocks (Liu et al. 2018 Zoph et al. 2018: Pham et al.. 2018). which can be reused in the network architecture. Although this is related to our approach, the discovered cells are usually much more complex than single neurons and only share their architecture while each cell has a different set of weights, therefore reducing the total number of required parameters. Although parameter sharing approaches exist, they either share weights between different evolved macro- macro-architectures (Real et al. 2017: Elsken al. 2018) or only use them to warm start cells with a parameter copy on initialization, which then is refined in subsequent fine tuning steps (Pham al. 2018 Xie al., 2018). missing the advantage of the recursive sharing scheme employed by ACNs. Although recent works like Stamoulis et al. (2019) achieve huge speed-ups, the large amount of time required to find suitable architectures remains major disadvantage, while our approach can be trained from scratch in one go.
al.
2.2 NETWORK COMPRESSION
One popular way for network compression is the pruning of unimportant weights from the network, an idea first appearing several decades ago to prevent over-fitting by reducing the network complexity (LeCun et al. 1990). The seminal work by Han et al.) (2015b) proposes an iterative procedure consisting of training and pruning weights with small magnitudes, Li et al. (2016) focus on pruning filters of trained CNNs based on the L -nom while Louizos et al. (201 and Li & Ji 2019 consider Lo-norm regularization. In contrast Luo et al. (2017) employ pruning schedule on the channel with the smallest effect on the activations of the subsequent layer. [Mochaaovv et al. (2016) 2019) use Taylor expansion to calculate importance scores for different channels to prune CNNs. Similarly neuron importance scores based on the reconstruction error of responses in the final layers are used by Yu et al. (2018) while Wang et al. (2018) propose adaptive pruning scheme for sub-groups of weights. Furthermore there also exist recent Bayesian approaches to weight pruning (Molchanov)
et al. 2017 Louizos et al. 2017a Ullrich et al. 2017).
2

---

Under review as a conference paper at ICLR 2020
71
f2
Figure 1: (a) LayerNet architecture; (b) ACN with single neuron components. The input masks are shown by colored arrows to respective neurons, where fi (red) has 2. f2 (orange) has 3 and fm (green) has 4 inputs. The output layer fout (blue) an independent module which is not reused.
Another path of research utilizes factorization approaches produce low-rank approximations the weight matrices of DNNs (Denton et al.) 2014 Gong et al., (014)). Alternatively vector quantization methods can be applied to the parameter matrices, e.g. dividing the weights into sub-groups which are represented by their centroid (Denil et al. 2013, Gon3.Gong et al. 2014). SqueezeNet introduced by landola et al. (2016) adapts so-called squeeze modules which reduce the number of channels while TensorNet (Novikov et al. 20155) uses the Tensor- Train format to tensorize the dense weight matrices of FC layers. Finally Wu (016) use the Kronecker product to achieve ayyerwiie compression.
Recent work combines several of the aforementioned methods in hybrid models. Han et al. (2015a) use pruning together with layer-wise quantization and additional Huffman coding. In Kim et al. (2013) Tucker decomposition supported by variational Bayesian matrix factorization is employed
while Tung & Mori 2018) jointly apply pruning and quantization techniques in a single framework.
A special variant of compression methods for DNNs is the teacher-studen approach also known as network distillation (Ba & Caruana 2014] Hinton et al., 2015 Luo et al.] 2016), which utilizes the relation of two models, fully trained deep teacher model and a significantly smaller and more shallow student model to distill the knowledge into a much more compressed form. There exist different approaches to the design and training the student model, e.g. utilizing special regularization (Sau & Balasubramanian 2016) or quantization techniques (Polino et al. 2018).
crucial advantage of ACNs that we do not waste time and computational resources on training large, over-paramend parr--araatteeeed architecture which is prone to over-fitting and usually needs lot of training data only to compress afterwards (e.g. by pruning or distillation). Instead we directly train smaller models which bootstrap their capacity by recursively-stacked neurons and therefore are more compact and can trained efficiently with less data. Furthermore, the aforementioned approaches are often quite limited in achieving large compression without degrading the prediction performance.
HashedNets (Chen et al. 2015) use random weight sharing within the parameter matrix each layer and introduce additional hashing formulation of the forward pass and the gradient updates. Our model is different from these approaches since we share neurons which can have an arbitrary number of weights and don't restrict the sharing to the same layer, but enable sharing across layers, too.
As an additional complementary method to the aforementioned approaches one can train networks with reduced numerical precision of their parameters (Courbariaux et al. 2014 Gupta et al. 2015). This achieves further compression and experiments show that leads only to a marginal degradation of model accuracy. Taking this idea to the extreme Courbariaux et al. (2016) constrain the values of all weights and activations to be either +1 or -1.
3 ATOMIC COMPRESSION NETWORKS
neuron can be understood as an arbitrary function f(; 0) .R. RR1 defined by its parameters
0 E IR The idea is to stochastically apply the same neuron with the same weights to the outputs of neurons from the previous layer. The different functions or components fa then form a global set M,
fi, f2- fm M which defines the set of parameters (indexed by the neurons) which is available to the model. Additionally we add the output layer fone which a unique FC layer that is not reused anywhere else in the network (see figure T)
3

---

Under review as a conference paper at ICLR 2020
The most naive approach to recursion is repeating whole layers with shared weights. Each module is represented by a FC layer mapping from a layer input dimension Dan to the output dimension Dout- The architecture is depicted in figure Th. More generally this approach can also be written in terms of a function composition (see equation] T). Sharing layers forces them to have equal input and output neural architectures. For comparison with neuron
dimensions, resulting in rectangular-shape level ACN in the ablation study in the experiment section we reference this approach with random sampling of the sequential network composition by LayerNet.
g (z) = fm fm-1 o fi f1(1)) (z), fi. fi....... E M
(1)
In contrast to repeating entire layers, ACN reuses the most atomic components of neural network which are represented by its single neurons. Consequently full network layer in the model consists of several neurons which do not need to be all different, but the same neuron can be reused more than once in the same layer (see figure [D). Each neuron p E M has an input dimension dp which we choose randomly from a preset range of dimensions
When inserting the neuron into layer f. we randomly sample dp connections to the neurons of the previous layer forming mask o on its activation z. While the trainable parameters (weights and bias) of the neurons are shared, this connection mask is different for every neuron in every layer. The procedure for sampling the ACN architecture shown algorithm [
Algorithm 1: ACN Architecture Sampling
input :D, number of neurons N, number of layers L, layer sizes K E N'. output :ACN architecture m,
// Create neurons with random dimens simensions
M := (f::))) (:0w)), 0s E Rd, d, ~ U (1,
// Create network architecture
for f= 1, do
(( (fi(:0,) A((00)) E < K(t-1)
for L....,,""" K(t) do
[z(;0,z) M(0)
m. (1) (+) fz(:
8/ ~ u (1,K(e-1))
return m,
Equation 2 shows the forward pass through a ACN neuron layer, where II represents the projection (selection) of the mask indices 5 from the activation of the previous layers:
( m (E) (I (z(e-1)). i= 1, K()
To reduce the additional memory complexity of these input masks we can store the seed of the pseudo random generator used for sampling the masks and are able reproduce them on the fly in the forward pass at inference time. Since the input masks of the neurons are completely random, situations may arise which not all elements of z are forwarded to the next layer. However in our experiments we find that this case happens only very rarely granted that the number of neurons per layer and the greater part of the input dimensions D are sufficiently large.
EXPERIMENTS
4.1 PRELIMINARY STUDY OF MODULE RECURSION
To gain more insights into the potential of ACN perform small study on synthetic curve fitting problem. We compare a simple FC network with a varying number of layers and neurons to an ACN with approximately the same number of parameters. The curve fitting problem is regression task on the function f(x: = 3.5x 3 sin(z) + 0.5 sin sin(8x2 (8x + 0.5T)) R. To achieve the best possible fit for the models perform hyper-parameter grid search including the dimension of hidden layers, learning
4

---

Under review as a conference paper at ICLR 2020
rate and the number of different modules for the recursive net. Details can be found in appendix
A.2
Considering our knowledge about the true function which includes powers of z we boost the performance of both models by squaring the neuron activations.
(17), MSE: 0.02046
(33), MSE: 0.01979
(41), MSE: 0.01242
(69), MSE: 0.00250
1.2
0.6
FC
0.0
(18), MSE: 0.01799
(32). MSE: 0.00427
(45), MSE: 0.00404
(62). MSE: 0.00052
0.6
ACN
0.0
0.6
0
Figure 2: Plots of the model fit to the curve data. The first row shows the fit of the FC baseline, the second that ACN. the columns the respective number model parameters as well as the achieved MSE on the test set are given above the plots. Since arbitrary number of parameters cannot always be achieved with each model, the next nearest number with a competitive model was selected, e.g. in the first column with 17 and 18 parameters respectively.
As can be seen in figure 2 ACN consistently achieves a better fit terms MSE than the FC baseline. Already ACN with only 32 parameters is sufficient to approximately fit even the highly non-linear parts of the function while the FC network only achieves a comparable more than twice the number of parameters. This result can be explained by some intuition about function composition. Consider a function (x) = (ax + 3) f TR + R with sole parameters and B. Then
with actually
create much more complex functions with the same set of parameters by composition f()()) = xa2x² + 2a 3x + a.5² + B)² Extending this to other functions h, (which each can be neuron) enables
create complex functions with small set of parameters. Furthermore compared to a standard FCN our ACN achieves much deeper architectures with the same number of parameters what further improves the fitting capability.
4.2 EXPERIMENTAL. SETUP
We evaluate the performance of the proposed algorithms using nine publicly available real-wor world classification datasets and three image datasets. The selected datasets and their characteristics are detailed in appendix A.1 The datasets a curated high quality version of the OpenML100 benchmark (Bischl et al. 2017). Details on the employed typer-parametert and training setup can be found in appendix A.2
chosen from the OpenML-CCC8 benchmark which is
We compare our model to six different baseline models:
FC: A standard network of comparable size. It was shown Chen et (2015) that this is simple but strong baseline which often outperforms more sophisticated methods.
RER: Random Edge Removal first introduced in Ciresan et al. (20IT). For this baseline number of connections randomly dropped from a FC network on initialization. In contrast to dropout (Srivastava et al.) 2014) the selected connections are dropped completely and accordingly the network is trained and also evaluated as sparse version.
TensorNet (Novikov et al. 2015): This model builds on a generalized low-rank quantization method that treats the weight matrices of FC layers as two dimensional tensor which

---

Under review as a conference paper at ICLR 2020
can be decomposed into the Tensor-Train (TT) format (Oseledtt 2011). The authors also compare their model to HashedNet (Chen et al. 2013) and claim to outperform it by 1.2%. To train TensorNet
datasets introduce two FC layers mapping from the input to the consecutive TT layers and from the last TT layer back to the output dimension.
BC: The Bayesian compression method of Louizos et al. (2017a) uses scale mixtures of normals sparsity-inducing priors on groups of weights, extending the variational dropout approach ooochhnn et al. (2017) to completely prune unimportant neurons while training model from scratch. We achieve varying levels of compression and accuracy by specifying different thresholds for the variational dropout rate the model.
TP: Molchanov al. (2019) use first and second order Taylor expansions compute importance scores for filters and weights which then are used to prune respective parts of the network. In our experiments we use the method which the authors report as best, introducing pruning gates directly after batch normalization layers. To realize different levels compression we prune increasingly bigger parts of the network based on the calculated importance scores in an iterative fine tuning procedure.
Logistic:egressionn: A linear model.
4.3 EXPERIMENTAL RESULTS
The results of the experimental study on the real-world datsets are shown in figure B while table 1 shows the results for the image datasets. All results are obtained by running each model-hyper parameter combination with three different random seeds and averaging the resulting performance metric and number of parameters. The standard deviation between runs can be found in table 5
appendix A.3
Table Classification test accuracy on image datasets in each row the best model (selected based on validation accuracy) up to the specified maximum number of parameters is shown. Therefore the model and its test accuracy values do not change for greater numbers of parameters, if smaller model has better validation accuracy. For some models very small versions could not be trivially achieved (indicated by "-"). All trained models are fully connected architectures. We did train
ACN for more than 500,000 parameters.
Parameters
FC
RER
TensorNct
BC
TP
LayerNet
ACN
5,000
0.9265
0.5114
0.3758
0.1434
0.9445
< 10,000
0.9350
0,5114
0.8772
0.2482
0,9433
0.9625
MNIST
< 50,000
0.9650
0.7508
0.9763
0,8249
0.9711
0.9774
100,000
0.9650
0.8806
0.9532
0.9764
0,9236
0.9711
0.9774
500,000
0.9844
0,9440
0.9619
0.9805
0,9709
0.9847
0.9774
500,000
0.9852
0,9440
0.9619
0,9805
0.9737
0.9854
5,000
0.6055
0.4765
0.2704
0.8513
FMNIST
10,000
0.8489
0.6055
0.8220
0.2714
0.8478
0.8669
50,000
0.8672
0.7514
0,8713
0.6750
0.8684
0.8795
100,000
0.8672
0,8137
0,8568
0,8713
0.8273
0,8684
0.8795
500,000
0.8960
0.8459
0,8676
0,8741
0.8612
0.8965
0.8795
500.000
0.8985
0.8459
0.8676
0.8741
0.8635
0.8963
5,000
0.0999
0.4557
CIFARIO
< 10,000
0.3932
0,2577
0.1508
0.4795
< 50,000
0.4183
0.3039
0,5015
0.2167
0.4195
0,5138
100,000
0.4745
0.3039
0.5015
0.3144
0,4195
0.5138
500,000
0.5343
0,4055
0.4750
0.5015
0.4133
0.5364
0.5138
500,000
0.5567
0.4293
0.4893
0.5141
0.4970
0.5559
As can be seen in figure 3 ACN consistently outperforms all other models on all of the nine real-world datasets for very small models, for datasets with huge feature space even for models with up to 10,000 parameters. This shows that the ACN architecture is very efficient w.r.t. its parameters and the required amount of training-das data and that on these datasets the recursive nature and parameter sharing throughout the architecture has a positive impact on the predictive efficiency compared to the number of parameters. For larger models the advantage of ACN decreases and the other models catch

---

Under review as a conference paper at ICLR 2020
bioresponse (1776)
har (561)
ietAms (3113)
0.3
0.15
0.42
Test Error
0.6
0.12.
0.36
0.4
0.09.
0.30
0.06
0.
0.24
0.03
0.0
10000
2500
5000
10000
10000
500
1000
2500
5000
5000
1000
2500
5000
110000
10000
isolet (617)
nomao (174)
optdigits (64)
0.8
0.4
0.20
Test Error
0.6
0.3
0.15
0.4
0.2
0.10-
0.2
0.1-
0.05
0.0
0.0
2500
5000
7500
10000
>= 10000
250
500
1000
2500
2500
250
500
750
1000
> 1000
spambase (57)
splice (287)
theorem (51)
0.40
0.57
0.32
0
0.54
Test Error
0.24
0.3
0.51
2
0.16
0.48
0
0.3
0.08
0.45
< 250
500
1000
> 1000
500
1000
2500
5000
5000
250
500
1000
>= 1000
<
>=
Capacity
Capacity
Capacity
FC
X
TensorNet
TP
ACN
RER
BC
LayerNet
LogisticRegression
Figure Plots of the model classification error on the test for different numbers of parameters. For each tick on the x-axis the model with the best validation accuracy under the threshold for the number of parameters is selected. In the titles next the name of the dataset we indicate the dimension of the respective feature space. For some methods very small models could not be achieved so they have no points plotted on the left part of the plots (e.g. a fully connected network will not have less parameters than linear model with number oof jinputs number_o .of _classes many parameters)
up or even outperform it by small margin, e.g. in case of the InternetAds dataset, however ACN remains competitive. TensorNet usually is at least a factor of two larger than the other models and therefore only shown in the last bin. The same applies for TP and BC on the optdigits, theorem and nomao/spambase datasets respectively.
To evaluate the parameter efficiency our technique we compare ACN and FC (as the best performing baseline) to large optimal FC network trained on the same dataset (doing a dlyper-parameter search with the same parameters mentioned in section 4.2 and five increasingly large architectures). We show the results regarding the relative compression rate and the respective loss in model performance in table 2 Say n is the number of model parameters of the large baseline and that of the evaluated model, then the compression rate is given by p n/nn'. Our model achieves compression rates of 88 up to more than 1100 times while the loss in test accuracy is kept at around 0. 0.15% up to 5.33%. In comparison FC achieves compression of 67 up to 528 times with accuracy loss between 0.28% and
7

---

Under review as a conference paper at ICLR 2020
7.13%. For all datasets except spambase and theorem ACNs realize higher compression rates than FC as well as a smaller loss accuracy in 5/9 cases. However since the comparison to a large model baseline can be biased by model selection and the thoroughness of the respective hyper-parameter search, we also investigate how the performance and capacity of the models compare w.r.t. linear baseline. The results in aable(cconfrmm the trend that observed in the comparison the large model.
Table 2: Compression and accuracy change compared to large FC model and a linear model baseline
Compared to large FC small ACN
Compared to Linear Model small ACN
Dataset
small FC
small PC
(%)
A Acc
Ac)(%)
A Ace)%
1) bioresponse
67.41
-0.28
322.15
-1.18
0.998
+0.70
4.771
-0.90
2) HAR
155.59
-7.13
361.82
-4.40
2.942
-6.25
6.840
-3.50
3) InternetAds
133.23
-1.09
1115.58
-2.66
0.998
+0.31
8.360
-1.26
isolet
74.49
-3.71
87.76
-3.45
3.042
-2.66
3.58
-2.60
5) nomao
218.67
-0.74
460.36
-1.43
0.972
+1.37
2.047
+0.68
6) optdigits
149.12
-6.15
168.06
-5.33
1.879
-4.41
2.117
-3.59
7) spambase
129.59
-2.12
92.46
-0.15
0.951
+0,23
0.678
+2.20
8) splice
211.46
-3.47
316.12
-5.33
1.467
-1.73
2.193
-3,59
theorem
97.47
-6.50
95.30
-4.03
0.647
+2.14
0.633
+4.61
On the image datasets ACN consistently and significantly outperforms all other models for up to 100,000 parameters. On MNIST smallest model has 4091 parameters and outperforms linear model with 7850 by 1.8% although the baseline has nearly twice as many parameters. The same trend be observed for FashionMNIST (7057 / 7850 with a difference of 2.04%) and CIFARIO where an ACN with 7057 parameters employs less than quarter of the 30730 parameters needed by the simplest linear model and outperfoms by a overwhelming 8.63%. Surprisingly the LayerNet outperfoms all other models for the category < 500, 000. For more than 500,000 parameters the FC network performs best. Although the results on CIFARIO first might seem rather poor compared to the approximately 96% accuracy which is achieved by CNN (Zagoruyko & Komodakis] 2016), the best results achieved by dedicated FC networks lie around 56% and are only topped by specifically designed network with linear bottleneck layers and unsupervised pre-training (Lin et al. (2015). While our model sacrifices some processing time for higher compression, the forward pass at inference time still works in matter of milliseconds standard CPU.
4.4 ABLATION STUDY ON NEURON-RECURSION RECURSION
To investigate the effect of the chosen level of sharing and recursion perform small ablation study. We compare ACN with parameter-sharing and recursion on neuron-level with the simpler approach of sharing and reusing whole layers described as the LayerNet architecture in section 3 Therefore we also report the performance for the LayerNet in an extra column in the tables 5 and The results imply that in general the recursion on neuron-level is much more effective achieve competitive models with high compression, outperforming the LayerNet in all but one case for less than 10,000 parameters. However the LayerNet seems have a beneficial regularization effect on large models, what is notable especially on the image datasets.
I
CONCLUSION
In this paper we presented Atomic Compression Networks (ACN), new network architecture which recursively reuses neurons throughout the model. We evaluate our model nine vector and three image datasets where we achieve promising results regarding the compression rate and the loss in model accuracy. In general ACNs achieve much tinier models with only small to moderate decrease of accuracy compared to six other baselines. For future work we plan to include skip connections in the architecture and to extend the idea to CNNs and the sharing of kernel parameters well for the FC layers. Another interesting path of research is the combination of the ACN scheme with NAS methods further optimize the efficiency and performance of the created architectures.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to Compose Neural Networks for Question Answering. arXiv:/601.011 100000005105 [es], January 2016a. URL http://
arxiv. stxiv.org/abs. /000s/160111 01705
arXiv: 1601.01705.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural Module Networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 39
48, 2016b. URL http: httt:::/openaccess..tmm thereennccesss..eef..... Andreas_Neural Modul Ba Www.weoosssve_eeaOOSS
wwwoonneetttto_o__oteeet wwtp:_cccrr__ppmmmmm_ html
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes- -rtezz public domain dataset for human activity recognition using smartphones. In Esann, 2013.
Jimmy and Rich Caruana. Do Deep Nets Really Need to be Deep? In Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Process- ing Systems 27, pp. 2654-2662. Curran Associates, Inc., 2014. URL URL htttp://paper:: nips CC httppaper/550 54 84 do deep nets rea yy need to be desp.deep.pf.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang. Rafael Manto- vani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites and the openml100. arXiv preprint arXiv: 00031, 2017.
James P Bridge, Sean B Holden, and Lawrence C Paulson. Machine learning for first-order theorem proving. Journal of automated reasoning, 53(2):141-172, 2014.
Laurent Candillier and Vincent Lemaire. Design and analysis of the nomao challenge active learning in the real-world. In Proceedings of the ALRA Active Learning Real-world Applications, Workshop ECML-PKDD 2012, Friday, September 28, 2012, Bristol, UK, 2012.
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, and Lauri Karttunen. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3631-3644 Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/v1/N19.1
htRhhttps:/ www. a www.wwwwbbbbb. wwtp://.wwwwbo./////.//o//
org/ wttp://///..otooooooooal------
Michael B. Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically Composing Representation Transformations as a Means for Generalization. arXiv: /007004400 [cs, stat), July 2018. URL httt:://aaxi....rss// abs 1807 04640 arXiv: 1807 04440.
Wenlin Chen, James Wilson, Stephen Tyree, Kilian Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick. In International Conference Machine Learning, pp- 2285-2294, 2015.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A Survey of Model Compression and Acceleration for Deep Neural Networks. arXiv: 1710.09282.
[cs], October 2017. URL
abs fas//110..m 09282
Dan C Cireşan, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High- performance neural networks for visual object classification. arXiv preprint arXiv: 2011.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. 1rX2 [cs], December 2014. URL http://arxiv erq/abs// /etea//44 1412. 7024
arXiv: 1412.7024.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yani,, and Yoshua Bengio. Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or 11 16Xiv:1602.0288. [cs], February 2016. URL http:/ /wrxivvvvvv//////////////////a.v org. wttp://.wrrrbbba........ 02830
arXiv: 1602.02830.

---

Under review as a conference paper at ICLR 2020
Misha Denil, Babak Shakibi, Laurent Dinh, Marc\ textquotesingle Aurelio Ranzato, and Nando de Freitas. Predicting Parameters in Deep Learning. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems nips. cc/ cttpppec//p///////// paper/
26, pp. 2148-2156. Curran Associates, Inc., 2013. Ut..tttp. httpppaappppeee/aa/ee itt.:ppppriicc.nn-gaeeep.e watg-paarmtteer-iinn dee----------------e--- eea htt.httpp:/pppprrs
Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Process- ing Systems 27, pp. 1269-1277. Curran Associates, Inc., 2014.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://rcchiy
ics. www.ccccec edu/ml
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution. arXiv: [cs, stat], April 2018. URL http://
aaxiv.orge abs / 1804. 09081 arXiv: 1804.09081.
Mark Fanty and Ronald Cole. Spoken letter recognition. In Advances in Neural Information Processing Systems, pp. 220-226, 1991.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Scarning-Volume 70, pp. 1243-1252. JMLR. org, 2017.
Yunchao Gong. Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing Deep Convolutional Networks using Vector Quantization. arXiv: 1..6111115 [cs], December 2014. URL http:
//arxiv.org/abs////
6115]
arXiv: 1412.6115.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. arXiv: 15.000149 [cs], October 201 5a. URL http:
org/ abs/15100 00149) arXiv: 1510.00149.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1135-1143. Curran Associates, Inc., 2015b.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A Horowitz, and William J Dally, Eie: efficient inference engine on compressed deep neural network. In 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), pp. 243-254, IEEE, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network. 02531
arXiv:/503.022 1503.0253 fcs, stat], March 2015. URL arXiv: 1503.02531.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learn- ing to Reason: End-To-End Module Networks for Visual Question Answering. In Proceedings of the IEEE international conference on computer vision, pp. 804-813, 2017. URL htttp://openaccesss. thecv...sseeee...c. ceamm/ooteettteen iccv 2017/html h////htm//H/. Learning to Rea_sReeeae____e2___ 2tw.cccvveeollle1 www.0111ppp111.1..1111 html
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint / 1.0965,,6 2018.
Forrest N. landola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5mb model size. (oe::602.0736 [cs], February 2016. / org. arXiv: 1602.07360.
07360
10

---

Under review as a conference paper at ICLR 2020
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. 0rXiv:1502.03/66 [cs], February 2015. URL http:
//arxiv. org/ aba/1502 03167
arXiv: 1502.03167.
Robert Jacobs, Michael I. Jordan, Steven Nowlan, and Geoffrey E. Hinton. Adaptive Mixtures of Local Experts. Neurul Computation, 3(1):79-87, February 1991. ISSN 0899-7667, 1530-888X. doi: 10. 10.1112/neco.. 1991.3.1.79 URL http:t/pwwww.mmmpprsssor...aaaaa ott.://orrraaaaa.ooooo/oo///// 1162/neco 999.3.1..75
www.
Xiaojie Jin, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Training skinny deep neural networks with iterative hard thresholding methods. arXiv preprint arXiv:1607.05422, 2016.
Kaggle. Predicting biological response, 2012. URL bioresponse/data
c/
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Com- pression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications.
Stti::/511.065333 [cs], November 2015. URL
arXiv: 151..0633..
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], December 2014.
6980 arXiv: 1412.6980.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. In Advances in Neural Information Processing Systems, pp. 2408-2418, 2018.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Nicholas Kushmerick. Learning remove internet advertisements. In Agents, pp. 175-181. Citeseer, 1999.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient Cradient-based learning applied to document recognition. Proceedings of the IEFE, (6(11):2278-2324, 1998.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning Filters for Efficient ConvNets. arXiv: 1608, 08710 [cs], August 2016. URL hRR.htttp///arxi.rrr 1608. 08710
arXiv: 1608.08710.
Yang Li and Shihao Ji. 1.-arr: Network sparsification via stochastic binary optimization. arXiv preprint arXiv::194.000
2019.
Zhouhan Lin, Roland Memisevic, and Kishore Konda. How far can we go without convolution:
Improving fully-connected networks. arXiv preprint arXiv:1511.02588 2015.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shiens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive Neural Architecture Search. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, volume 11 11205, pp. 19-35. Springer International Publishing, Cham, 2018. ISBN 978-3- 030-01245-8 978- 978-3-030-01246-5
doi: 10. 0000000000300000012000000 springes wwt. 00. 00000000773333 0-00-----0000------500 0104-0122655-2
2. URL http::/linkk
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian Compression for Deep Learning. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3288-3298. Curran Associates, Inc., 2017a. URL http:://ppppss..ppp///// nips cttpspercpppppr/rp/p/ www.-bayyiinncccoom. cotp-nooopresssiooooo for--ee---e-eeee www.eeeeeeeeennneeeeeeeeeeeeeeee
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through I regularization. arXiv preprint arXiv: 171.0117 2017b.

---

Under review as a conference paper at ICLR 2020
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017. URL http: wpt.accss..cccp..epeeppe.. / html A AfFi: TCCtetrrr__V____eee 2012 paperr.ppp..__ html
theovf. com. content iccv 2017/
Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, and Xiaoou Tang. Face model compression by distilling knowledge from neurons. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
Sachin Mchta, Mohammad Rastegari, Anat Caspi. Linda Shapiro, and Hannaneh Hajishirzi. Espnet: Efficient spatial pyramid of dilated convolutions for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 552-568, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2498-2507. JMLR. org, 2017.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning Convolutional Neural Networks for Resource Efficient Inference. urXiv:161/ 1eer:6//116440 1.06440 [cs, stat), November 2016. URL http::/atax.i....eeemm
www./asll./1 06440 arXiv: 1611.06440.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1126--1222, 2019.
Michiel O Noordewier, Geoffrey G Towell, and Jude W Shavlik. Training knowledge-based neural networks to recognize genes in dna sequences. In Advances in neural information processing systems, pp. 530-536,1 1991.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. In Advances in neural information processing systems, pp. 442 450, 2015.
Ivan Oseledets. Fensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):
2295-2317 2011. 2011.
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Lc, and Jeff Dean. Efficient Neural Architecture Search via Parameter Sharing. I8022.1802.00 03266 [cs, stat], February 2018. URL http:// 03268
arXiv: 1802.03268.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv:7802.00 05668 [cs], February 2018. URL
arXiv: 1802.05668.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Lcale-Scale Evolution of Image Classifiers. arXiv: [cs], March 2017. /RLLattp://azxee http://axxvvvv...//.//./ abs 1703. C1 01041 arXiv: 1703.01041.
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing Networks: Adaptive Selection of Non-linear Functions for -ulti-Task Learning. arXiv: 171.00229 [cs]. November 2017. URL
http::/ org abs 1711 01239
arXiv: 1711.01239.
Farzad Samie, Vasileios Tsoutsouras, Lars Bauer, Sotirios Xydis, Dimitrios Soudris, and Jörg Henkel. Computation offloading and resource allocation for low-power iot edge devices. In 2016 IEEE 3rd World Forum on Internet of Things (WF-loT), pp. 7-12. IEEE, 2016.
Bharat Bhusan Sau and Vineeth N. Balasubramanian Deep Model Compression: Distilling Know]- edge from Noisy Teachers. arXiv: [cs), October 2016. URL abs / /t1./161... 09650 arXiv: 1610.09650.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting, J. Mach. Learn. Res., 15(1): 1929-1958, January 2014. ISSN 1532-4435. URL htt.:ttpp.. /dl. acm. org/ citation. cfmoon...mm. fmm
id=2 2670313
12

---

Under review as a conference paper at ICLR 2020
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Lingl-ppah, nas: Designing hardwarr-eeficien convnets in less than hours. arXiv preprint wxXiv:194..02877. 2019.
Kenneth O Stanley, David B D' Ambrosio, and Jason Gauci. A hypereube- encoding for evolving large-scale neural networks. Artificial life. 15(2)1185-212, 2009.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:/905: 1946, 2019.
Frederick Tung and Greg Mori. CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning- Quantization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7883-7882, 2018. URL http: /ttp:://oppccccccccc thecvf. com/
2018/
html/Tung CLIP-Q Deep Network CVPR 2018
2018_paperr_____peeppppppppa_1e_ paper.pp..... htmll
Karen Ullrich, Edward Meeds, and Max Welling. Soft Weight-Sharing for Neural Network Compres- sion. arXiv:/702.0444 04008 [cs, stat), February 2017. URL http::/arxiv/ org/ aaaaab///1000
04008
arXiv: 1702.04008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Huan Wang, Qiming Zhang, Yuehai Wang, Yu Lu, and Haoji Hu. Structured Pruning for Efficient 0946/ arXiv: 1804.09461.
ConvNets via Incremental Regularization. arXiv: /8040094/ [cs, stat], April 2018. http
09461
Jin Wu. Compression of fully-com coly-cconnecte layer in neural network by kronecker product. In 2016 Eighth International Conference on Advanced Computational Intelligence (ICACI), pp. 173-179.
IEEE, 2016
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 2017.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic Neural Architecture Search. arXiv:1812.09926 /cs, stat], December 2018. URL htR/:ttt////rrav.orr///bbb/ arXiv: 1812.09926.
09926
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I. Morariu, Xintong Han, Mingfei Gao, Ching- Yung Lin, and Larry S. Davis. NISP: Pruning Networks Using Neuron Importance Score Propagation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 9194 9203, 2018 URL html//uu NISP Pruning Networks CVPR 2018 paper.htm.
ppppacccss.c.p..ppppppp/./...... thecvf..ec... com/ conteettottttt/ooooooemmmmo _et___ee______________ 2018/
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. arXiv:1605.0711 [cs], May 2016. URL htttt:p:t//aae// www.wiiw........ooo 61000. C7146 arXiv: (605.07146.
Poorya Zaremoodi, Wray Buntine, and Gholamreza Haffari. Adaptive knowledge sharing in multi- task learning: Improving low-resource neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 656-661, 2018
Barret Zoph and Quoc V. Le. Neural Architecture Search with Reinforcement Learning. a6Xiv: 1.11111578 [cs], November 2016. URL http:://arxiv..otg//a/////1// arXiv: 1611.01578.
Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning Trans- ferable Architectures for Scalable Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8697-8710, 2018. URL
http:// Learning thecvf...tteee..... cww/:ootenttttcceceeeeooooooeeoe
13

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
A.1 DATASET DETAILS
Table shows detailed attributes for the datasets selected from the OpenML-CCI8 benchmark. The selection criteria were datasets with more than 3000 instances and more than 50 original features. The number of features reported in the table is the number of features after one-hot-encoding. For the image datasets (10)-(12) we employ the original train/test splits and use random 20% of the training set for validation leading to a train/val/test split of 0.64/0. 16/0.2. The datasets (1)-(9) are split randomly using the same fractions. We do not any kind of additional transformation or data augmentation beyond those which was already originally applied to the respective datasets.
Table 3: Description datasets
Dataset
# Instances
# Features
# Classes
% Minor. CL.
(1) bioresponse (Kaggle
2012
3751
1776
45,77
(2) HAR (Anguita
2013
10299
561
13.65
(3) InterneTAds Kushmerick] 1999)
3279
3113
14.00
(4) isolet
(Fanty & Cale 1991)
7797
617
3.82
(5) nomao (Candillier & Lemaire
2012)
34465
174
28.56
(6) optdigits (Dua & Graft
2017
5620
64
10
9.86
spambase (Dua Grafi
2017
4601
57
39.40
(8) splice (Noordeweer et
3190
287
24.04
(9) theorem (Bridge et al.
2014)
6118
51
7.94
(10) MNIST (LeCun
1998
70000
784
(11) FashionMNIST (Xiao el al.
2017
70000
784
10
10.00
(12) CIFARIO (Krizhevsky el al.
2009
60000
3072
10
10,00
14

---

Under review as a conference paper at ICLR 2020
A.2 HYPER-PARAMETER CONHIGURATION
Table 4: Hlyper-Paramette for Preliminary Study [4.
Hyper- Papee-Parrmeter
ScarchGrid FC
SearchGrid ACN
hidden layer size learning rate number layers max gradient norm use batch norm numher of modules
2, 3, 4, 8
3,4,
0.1, 0.050.0,000 0.01
0.01 0.01
1, 2, 0.99, 1, 2, 5 True, False
2, 4
0.99, 1, 2
True, False 2, 4. 8
HYPER-PARAMETE PARAMETER AND TRAINING SETUP FOR REAL-WORLD DATASETS
Given the large computational demands of deep learning architectures, a proper full prper-paraeeter search is infeasible for the models employed on the real-world datasets. Under these circumstances we follow the established trend of using some specific deep architectures designed based on expert knowledge together with small selected parameter grid containing learning rate (0.01, 0,001). use of batch normalization (loffe & Szegedy 2015) and dropout probability (0.0, 0.3). To enable fair comparison the hyper-paramettrr search for every model type which is trained from scratch is done for the same number of combinations. Accordingly for each dataset we evaluate eight different architectures per model type where vary the number of hidden layers and the width of these layers, for ACN M and D are different between some of the eight varying architectures. The size of the set of modules M is varied in the range between 3 and 5 for the LayerNet (ablation study) and between 16 and 512 for the ACN. The range of input dimensions D is set to a subset of (2, 4. 8, 16. 32. 64, 128) depending on the maximum width of the used hidden layers and the respective input dimension. In case of the TensorNet we vary the dimension of the tensor train layers and the length of the tensor-train decomposition between different architectures per dataset together with the aforementioned parameter grid. All other parameters remain at their default values (Novikov jet al. 2015). For the two pruning approaches BC and TP we train two different large FC architectures with the described hyper-parameter grid and then prune the networks in small steps, selecting the best architecture within every parameter bin. The models are trained for 25 epochs on batches of size 64 for the real-world datasets and 128 for the image datasets. In order to minimize the loss apply stochastic gradient descent update steps using Adam (Kingma & Ba 2014) and halve the initial learning rate every 10 epochs. The only exception is Tp where use SGD with momentum term of 0.99 as proposed by the authors but employ the same learning rate schedule. Finally, in order to avoid exploding gradients the gradient norm truncated to a maximum of 10. Apart from the TensorNet for which we adapt the publicly available Tensorflow Code all models are implemented using the PyTorch 1.1 framework. For BC and TI we adapt the public PyToreh code base provided by the authors.
A.3 ADDITIONAL RESULTS
In this section we show the results presented in figure in tabular form in table Furthermore for completeness we present additional results regarding simple sparsification baselines in table 6 employing Ll-regularization (L1) and iterative hard tresholding (L1+HT) (Han et al. 2015b) but without explicit cardinality constraint (Jin et al., 2016). The additional baseline models perform mostly on par with the small FC baseline, sometimes slightly better or slightly worst. The biggest change can be seen on the splice dataset where the additional baselines perform better than FC and ACN in the three bins with the highest number of parameters. The additional basclines were trained and evaluated following the same experimentation and hyperparameter protocol as the other models.
BayssinCCompresiooFooDl..
pruning
15

---

Under review as a conference paper at ICLR 2020
Table 5: Test accuracy and its standard deviation on real-world datasets. The standard deviation of runs with FC, RER, BC and TP models always below the displayed three decimals precision and therefore are not shown in the table.
Capacity
RER
TensorNet
BC
LayerNet
ACN
(1) bioresponse
1000
0.787 (#:0.005)
2500
0,546
0,548
0.787 (=0.005)
< 5000
0.803
0,570
0.685
0.548
0.775 (+00008))
0.797 (+00005)
10000
0.793
0.628
0.776
0.744
0.777 (+000033
0.797 (+0.005)
10000
0.793
0.609
0.784 (+.00033)
0.784
0.797
0.800 (+0.004)
0.797 (+0.005)
(2) HAR
500
0.940 (+0.001)
1000
0.913
0.420
0.967 (100.003)
2500
0.971
0.669
0.171
0.763
(10.1
0.977 (1000005
5000
0.984
0.764
0.974
0.438
0.943 (10.044)
0.985 (100.00)
5000
0.983
0.764
0.975 (+0.001)
0.981
0.950
0.983 (+.000
0.989 (0.00)
(3) InternetAds
1000
0.956 (+00.012)
2500
0.842
0.970 (#00002)
5000
0.842
0.842
0.972 (=0.005)
10000
0.972
0.842
0.964
0.842
0.929 (#.0002)
0.972 (=0.005)
10000
0.977
0.909
0.902 (+00.002)
0.967
0.908
0.978 (+00.004)
0.972 (+0.005)
(4) isolet
2500
0.494
0.254
0.420 (10.1144)
0.883 (100011)
5000
0.842
0.414
0.480
0.141
0.773 (+0.030)
0.941 (+0.001)
7500
0.939
0.645
0.893
0.188
0.918 (+0.009)
0.948 (+0.002)
10000
0.939
0.645
0.934
0.188
0.918
(+00009)
0.948 (=0.002)
10000
0.941
0.645
0.917 (1.0.066)
0.949
0.932
0.949
(10.002)
0.948 (10.002)
(5) nomao
250
0.771
0.950 (=0.003)
500
0.957
0.778
0.953 (1:0.001)
0.954 (+00.001)
1000
0.958
0.840
0.955 (+:0.001)
0.959 (+0.001)
2500
0.960
0.901
0.949
0.960 (+:0.00))
0.961 (+0.003)
2 2500
0.961
0.901
0.955 (=00001)
0.960
0.940
0.962 (100001)
0.961 (=00.003
(6) optdigits
250
0.714
0.552
0.560 (+-0.118)
0.840 (+0.012)
500
0.931
0.814
0.885 (10.006)
0.939 (10.001)
750
0.977
0.881
0.885 (100006)
0.963 (100.004)
1000
0.977
0.881
0.966 (100005)
0.969 (100.002
N 1000
0.984
0.881
0.965 (+0.004)
0.936
0.775
0.984 (#0.003)
0.980 (+0.004)
(7) spambuse
250
0.921
0.758
0.927 (#0.012)
0.941 (#00.003)
500
0.941
0.835
0.936 (#0.004)
0.943 (=0.008)
1000
0,945
0.835
0.615
0.936 (+00004)
0.948 (#0.000)
N 1000
0.945
0.835
0.729 (+0.006)
0.915
0.714
0.939 (+0.007)
0.948 (=00.04)
(8) splice
500
0.549
0.917 (+0.0255)
1000
0.936
0,596
0.947 (+0.013)
0.949 (+0.006)
2500
0.964
0.648
0.533
0.533
0.954 (++0006)
0.963 (+0.013)
5000
0.964
0.648
0.957
0.632
0.962 (+0.004)
0.963 (++0.013)
5000
0.964
0.648
0.902 (1.0.036)
0.969
0.906
0.965 (10.0003
0.963 (10.0013
(9) theorem
250
0.486
0.430
0.445 (+00006)
0.500 (+00013)
500
0.511
0.460
0.474 (100026)
0.535 (100.07)
1000
0.536
0.460
0.526 (1:0.004)
0.540 (10.007)
1000
0.536
0.460
0.520 (+.0.003)
0.535
0.503
0.542 (4:0.004)
0.553 (40.013)
16

---

Under review as a conference paper at ICLR 2020
Table 6: Test accuracy and its standard deviation on real-world datasets for the additional sparsification baselines. The standard deviation of runs with FC, LI and LI+HT models are always below the displayed three decimals precision and therefore
shown the table.
Capacity
FC
L1
LI+HT
ACN
(1) bioresponse
1000
0,548
0.548
0.787 (1.0.005)
2500
0,548
0.777
0.787 (=00,005)
5000
0.803
0.767
0.791
0.797 (=0.005)
10000
0.793
0.788
0.791
0.797 (=0.0005)
10000
0.793
0.788
0.791
0.797 (+0.005)
(2) HAR
500
0.940 (=00.001)
1000
0.913
0,181
0.193
0.967 (+000033)
2500
0.971
0.181
0.959
0.977 (+0.0055)
5000
0.984
0.981
0.967
0.985 (+0.0004
5000
0.983
0,981
0.975
0.989 (+000002
(3) InternetAds
1000
0.916
0,913
0.956 (#000012
2500
0,916
0.966
0.970 (1:0.002)
5000
0.969
0.966
0.972 (100.005)
10000
0.972
0.977
0.966
0.972 (+.0.005)
10000
0.977
0.977
0,966
0.972 (=0.0005)
(4) isolet
2500
0.494
0.033
0,488
0.883 (=00.017)
5000
0.842
0,033
0.933
0.941 (=0.001)
7500
0.939
0,938
0.933
0.948 (=000002)
10000
0.939
0.938
0.953
0.948 (100.002)
10000
0.941
0.938
0.953
0.948 (+000002)
(5)
250
0.950 (=00003)
500
0.957
0.954 (#0.001)
1000
0.958
0.718
0.718
0.959 (=0.001)
2500
0.960
0,952
0.949
0.961 (+00.003)
2500
0.961
0.952
0.951
0.961 (+00.033
(6) opidigits
250
0.714
0.840 (++0.012)
0.931
0.939 (+-0.011)
750
0.977
0.104
0.963 (+-0.004)
1000
0.977
0.104 0.979
0.961
0.969 (100,002)
1000
0.984
0.985
0.980 (10.004)
(7) spambase
250
0.921
0.941 (110.003
0,941
0.943 (1.0.008)
1000
0.945
0.733
0.739
0.948 (1.0.004)
1000
0.945
0.919
0.922
0.948 (=0.0044
(8) splice
500
0.917 (#:0.025)
1000
0.936
0.533
0.533
0.949 (=0.006)
2500
0.964
0.889
0.974
0.963 (=0.001)
5000
0.964
0,971
0.974
0.963 (#000113
5000
0.964
0.976
0.977
0.963 (++0.013)
(9) theorem
250
0.486
0.500 (100.013)
500
0.511
0.535 (=0.007)
1000
0.536
0.422
0.422
0.540 (=00077)
1000
0,536
0.488
0.493
0.553 (+0.013)
17


Reviews:
[{"review_id": "rke2IBwPtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the use of replicating neurons across and within layers to compress fully connected neural networks. The idea is simple, and is evaluated on a number of datasets and compared with fully connected, single layer, and several compression schemes. \n\nStrengths: a lot of nice experiments with clearly advantageous results are given.\n\nWeaknesses: One obvious baseline missing is sparse compression, which can be achieved using either l1 regularization, or hard thresholding + fine tuning, both of which are easy to implement and appear in several works, e.g.\n\nScalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization (Yang, Ruozzi, Gogate)\nTraining skinny deep neural networks with iterative hard thresholding methods (Yin, Yuan, Feng, Yan)\n\n... many others just via googling ... \n\nAlso, I think this work should be compared with compression schemes that work via kronecker product, which seem very similar to this scheme (but where the kronecker matrix is binary to produce replication)\n\nCompression of Fully-Connected Layer in Neural Network by Kronecker Product (Zhou, Wu)\n(more via google)\n\nOne obvious advantage of replication over kronecker product is lower complexity, but nonetheless, the methods belong in a similar family.\n\nOtherwise, I think the work makes sense, the idea is nice, and the results show promise!\n\nAfter rebuttal: I have read the rebuttal and the authors have basically addressed all my concerns. It is a bit disappointing that simple L1 regularization can give competitive results, but the fact that the authors are willing to do the experiment and incorporate the results convinces me that there's nothing being hidden here, and the reader can make a fair and informed conclusion, so I have no more complaints.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJgIDBwJ9B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new way to create compact neural net, named Atomic Compression Networks (ACN). An immediate related work is LayerNet, where a deep neural net is created by replicating the same layer. Here, this paper extends replication down to the neuron level. \n\nI am leaning towards rejecting this paper because the experimental setup is not well justified and a few important details are missing before conclusions can be drawn. I would like to ask a few clarification questions. Depending on the authors’ answers, I might be willing to adjust my rating. \n\n(1) Is there missing a delta in the first half of line 6 in Algorithm 1? \n\n(2) Throughout the experiments, for the same hyperparameter (e.g. Table 4 in A.2) do you run Algorithm 1 more than once and select the best sample architecture? If the answer is yes, summarizing all masks as one parameter will not be reasonable. Given a yes answer, I would also like to ask if the same number of samples have been considered for FC (for the same hyperparameter). \n\n(3) Is there any intuition behind why FC does a much worse job of fitting curves than ACN with much less parameters? This refers to Fig. 2, if we compare FC with 41 parameters to ACN with 18 parameters. I am confused because MSE on sampled points often goes down when we increase the number of parameters for the application of curve fitting.\n\n(4) Convolution can be thought of as a special case of ACN. ConvNet is the default architecture for working on image datasets. Since MNIST and CIFAR are considered, why not also compare to ConvNet?\n\n(5) The claims that “ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks with only a fractional deterioration of classification accuracy” is quite misleading. Given fully-connected neural networks achieve up to 528 times with also a fractional deterioration (Sec. 4.3), by presumably having a shallower architecture. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "ByxlTSdttS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a new method called Atomic Compression Network for constructing neural networks. The idea is straightforward. Basically, firstly create some neurons in random fashion, then reuse a subset of those neurons in each layer. The experiments shows ACN produces better accuracy than baseline models including a FC network, a Baysesina compression method, etc. for MINIST, etc. The paper also show ACN uses much less numbers of parameters and achieves similar accuracy when comparing with a large optima FC network on a set of datasets. \n\nOverall, I don’t support accepting this paper. First, I don’t think the proposed idea is very innovative. Please elaborate why this method seems to work well when comparing baseline models. Is it just randomly constructed network also perform well?  Secondly, I’m not convinced we will use this method to build network in real world applications. The model size is small, but in what cases this small model size matters? Is this a reliable way to create useful models? \n\nOn page 7, in Figure 3, why logistic regression only has a single point in some of the plots?\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a very general idea called Atomic Compression Networks (ACNs) to construct neural networks. The idea looks simple and effective.  However, the reason why it works is not well explained.  The experiments are not sufficient enough to convince the reviewers.
Decision: Reject