Atomic Compression Networks

Compressed forms of deep neural networks are essential in deploying large-scale computational models on resource-constrained devices . Contrary to analogous domains where large-scale systems are build as a hierarchical repetition of smallscale units , the current practice in Machine Learning largely relies on models with non-repetitive components . In the spirit of molecular composition with repeating atoms , we advance the state-of-the-art in model compression by proposing Atomic Compression Networks ( ACNs ) , a novel architecture that is constructed by recursive repetition of a small set of neurons . In other words , the same neurons with the same weights are stochastically re-positioned in subsequent layers of the network . Empirical evidence suggests that ACNs achieve compression rates of up to three orders of magnitudes compared to fine-tuned fully-connected neural networks ( 88× to 1116× reduction ) with only a fractional deterioration of classification accuracy ( 0.15 % to 5.33 % ) . Moreover our method can yield sub-linear model complexities and permits learning deep ACNs with less parameters than a logistic regression with no decline in classification accuracy . 1 INTRODUCTION . The universe is composed of matter , a physical substance formed by the structural constellation of a plethora of unitary elements denoted as atoms . The type of an atom eventually defines the respective chemical elements , while structural bonding between atoms yields molecules ( the building blocks of matter and our universe ) . In Machine Learning a neuron is the infinitesimal nucleus of intelligence ( i.e . { atom , matter } ↔ { neuron , AI } ) , whose structural arrangement in layers produces complex intelligence models . Surprisingly , in contrast to physical matter where molecules often reuse quasi-identical atoms ( i.e . repeating carbon , hydrogen , etc . ) , neural networks do not share the same neurons across layers . Instead , the neurons are parameterized through weights which are optimized independently for every neuron in every layer . Inspired by nature , we propose a new paradigm for constructing deep neural networks as a recursive repetition of a fixed set of neurons . Staying faithful to the analogy we name such models as Atomic Compression Networks ( ACNs ) . Extensive experimental results show that by repeating the same set of neurons , ACNs achieve unprecedented compression in terms of the total neural network parameters , with a minimal compromise on the prediction quality . Deep neural networks ( DNN ) achieve state-of-the-art prediction performances on several domains like computer vision ( Huang et al. , 2018 ; Tan & Le , 2019 ) and natural language processing ( Vaswani et al. , 2017 ; Gehring et al. , 2017 ) . Therefore , considerable research efforts are invested in adopting DNNs for mobile , embedded , or Internet of Things ( IoT ) devices ( Kim et al. , 2015 ) . Yet , multiple technical issues related to restricted resources , w.r.t . computation and memory , prevent their straightforward application in this particular domain ( Han et al. , 2016 ; Samie et al. , 2016 ; Mehta et al. , 2018 ) . Even though prior works investigate neural compression techniques like pruning or low-rank parameter factorization , they face fragility concerns regarding the tuning of hyperparameters and network architecture , besides struggling to balance the trade-off between compression and accuracy ( Cheng et al. , 2017 ) . Contributions : In a nutshell , this paper introduces : • a novel compression paradigm for neural networks composed of repeating neurons as the atomic network components and further motivated by function composition ; • compression rates of up to three orders of magnitudes compared to a cross-validated fullyconnected network on nine real-world vector datasets ; • first work to achieve sub-linear model complexities measured in the number of trained parameters compared to connected architectures on several computer vision tasks . 2 RELATED WORK . 2.1 MODULAR NEURAL NETWORKS . Our approach of training a set of neurons and ( re- ) using them in building the network architecture is partially related to the existing scheme of modular neural networks ( MNN ) . End-to-End Module Networks ( Andreas et al. , 2016a ; b ; Hu et al. , 2017 ) are deep neural network models that are constructed from manual module blueprints defined for different sub-tasks in question answering . The Compositional Recursive Learner proposed by Chang et al . ( 2018 ) employs a curriculum learning approach to learn modular transformations while Routing Networks ( RN ) ( Rosenbaum et al. , 2017 ) consist of a set of pre-defined modules ( which each can be a NN ) and a meta controller ( called router ) that decides in which order these modules are consecutively applied to a given input . Modular Networks ( Kirsch et al. , 2018 ) are an extension to RNs employing conditional computation with Expectation-Maximization . Finally the recent work of Cases et al . ( 2019 ) focuses on the recursive formulation of the Routing Network , consecutively applying one of a set of modules to an input . The crucial difference to our approach is that our neurons are much smaller than the general modules of MNN and that we reuse the same components on the same network level ( e.g . within the same layer ) while the modules of MNN are only applied sequentially . A different extension of the RN is the model of Zaremoodi et al . ( 2018 ) which uses the router in a gating mechanism to control the input to a set of shared RNN modules similar to a Mixture of Experts model ( Jacobs et al. , 1991 ) . Although Mixture of Experts models are related , they normally do not stack and recursively reuse components within the same network but have a comparably shallow architecture . Another related field of research is that of neural architecture search ( NAS ) ( Zoph & Le , 2016 ) . It is concerned with the automatic discovery of high performing neural network architectures for diverse tasks . There exists a multitude of search-approaches including ( Neuro- ) Evolution ( Stanley et al. , 2009 ; Real et al. , 2017 ) and Reinforcement Learning ( Zoph & Le , 2016 ; Pham et al. , 2018 ) . A sub-field of NAS is the dedicated search for neural modules called cells or blocks ( Liu et al. , 2018 ; Zoph et al. , 2018 ; Pham et al. , 2018 ) , which can be reused in the network architecture . Although this is related to our approach , the discovered cells are usually much more complex than single neurons and only share their architecture while each cell has a different set of weights , therefore not reducing the total number of required parameters . Although parameter sharing approaches exist , they either share weights between different evolved macro-architectures ( Real et al. , 2017 ; Elsken et al. , 2018 ) or only use them to warm start cells with a parameter copy on initialization , which then is refined in subsequent fine tuning steps ( Pham et al. , 2018 ; Xie et al. , 2018 ) , missing the advantage of the recursive sharing scheme employed by ACNs . Although recent works like Stamoulis et al . ( 2019 ) achieve huge speed-ups , the large amount of time required to find suitable architectures remains a major disadvantage , while our approach can be trained from scratch in one go . 2.2 NETWORK COMPRESSION . One popular way for network compression is the pruning of unimportant weights from the network , an idea first appearing several decades ago to prevent over-fitting by reducing the network complexity ( LeCun et al. , 1990 ) . The seminal work by Han et al . ( 2015b ) proposes an iterative procedure consisting of training and pruning weights with small magnitudes . Li et al . ( 2016 ) focus on pruning filters of trained CNNs based on the L1-norm while Louizos et al . ( 2017b ) and Li & Ji ( 2019 ) consider L0-norm regularization . In contrast Luo et al . ( 2017 ) employ a pruning schedule on the channel with the smallest effect on the activations of the subsequent layer . Molchanov et al . ( 2016 ; 2019 ) use Taylor expansion to calculate importance scores for different channels to prune CNNs . Similarly neuron importance scores based on the reconstruction error of responses in the final layers are used by Yu et al . ( 2018 ) while Wang et al . ( 2018 ) propose an adaptive pruning scheme for sub-groups of weights . Furthermore there also exist recent Bayesian approaches to weight pruning ( Molchanov et al. , 2017 ; Louizos et al. , 2017a ; Ullrich et al. , 2017 ) . Another path of research utilizes factorization approaches to produce low-rank approximations to the weight matrices of DNNs ( Denton et al. , 2014 ; Gong et al. , 2014 ) . Alternatively vector quantization methods can be applied to the parameter matrices , e.g . dividing the weights into sub-groups which are represented by their centroid ( Denil et al. , 2013 ; Gong et al. , 2014 ) . SqueezeNet introduced by Iandola et al . ( 2016 ) adapts so-called squeeze modules which reduce the number of channels while TensorNet ( Novikov et al. , 2015 ) uses the Tensor-Train format to tensorize the dense weight matrices of FC layers . Finally Wu ( 2016 ) use the Kronecker product to achieve layer-wise compression . Recent work combines several of the aforementioned methods in hybrid models . Han et al . ( 2015a ) use pruning together with layer-wise quantization and additional Huffman coding . In Kim et al . ( 2015 ) Tucker decomposition supported by a variational Bayesian matrix factorization is employed while Tung & Mori ( 2018 ) jointly apply pruning and quantization techniques in a single framework . A special variant of compression methods for DNNs is the teacher-student approach also known as network distillation ( Ba & Caruana , 2014 ; Hinton et al. , 2015 ; Luo et al. , 2016 ) , which utilizes the relation of two models , a fully trained deep teacher model and a significantly smaller and more shallow student model to distill the knowledge into a much more compressed form . There exist different approaches to the design and training of the student model , e.g . utilizing special regularization ( Sau & Balasubramanian , 2016 ) or quantization techniques ( Polino et al. , 2018 ) . A crucial advantage of ACNs is that we do not waste time and computational resources on training a large , over-parameterized architecture which is prone to over-fitting and usually needs a lot of training data only to compress it afterwards ( e.g . by pruning or distillation ) . Instead we directly train smaller models which bootstrap their capacity by recursively-stacked neurons and therefore are more compact and can be trained efficiently with less data . Furthermore , the aforementioned approaches are often quite limited in achieving large compression without degrading the prediction performance . HashedNets ( Chen et al. , 2015 ) use random weight sharing within the parameter matrix of each layer and introduce an additional hashing formulation of the forward pass and the gradient updates . Our model is different from these approaches since we share neurons which can have an arbitrary number of weights and don ’ t restrict the sharing to the same layer , but enable sharing across layers , too . As an additional complementary method to the aforementioned approaches one can train networks with reduced numerical precision of their parameters ( Courbariaux et al. , 2014 ; Gupta et al. , 2015 ) . This achieves further compression and experiments show that it leads only to a marginal degradation of model accuracy . Taking this idea to the extreme Courbariaux et al . ( 2016 ) constrain the values of all weights and activations to be either +1 or -1 .

Summary: The paper describes a new method called Atomic Compression Network for constructing neural networks. The idea is straightforward. Basically, firstly create some neurons in random fashion, then reuse a subset of those neurons in each layer. The experiments shows ACN produces better accuracy than baseline models including a FC network, a Baysesina compression method, etc. for MINIST, etc. The paper also show ACN uses much less numbers of parameters and achieves similar accuracy when comparing with a large optima FC network on a set of datasets.