EuclidNets: Combining hardware and architecture design for Efficient Inference and Training

In order to deploy deep neural networks on edge devices , compressed ( resource1 efficient ) networks need to be developed . While established compression methods,2 such as quantization , pruning , and architecture search are designed for conventional3 hardware , further gains are possible if compressed architectures are coupled with4 novel hardware designs . In this work , we propose EuclidNet , a compressed network5 designed to be implemented on hardware which replaces multiplication , wx , with6 squared difference ( x w ) 2 . EuclidNet allows for a low precision hardware7 implementation which is about twice as efficient ( in term of logic gate counts ) as8 the comparable conventional hardware , with acceptably small loss of accuracy.9 Moveover , the network can be trained and quantized using standard methods,10 without requiring additional training time . Codes and pre-trained models are11 available at http : //github.com/anonymous/.12 1 Introduction13 While the majority of deep neural networks are designed to be implemented on GPUs , they are14 increasingly being deployed on edge devices , such as mobile phones . These edge devices require15 compressed ( more efficient ) , hardware aware architectures , due to memory and power constraints16 [ 7 , 11 ] , which seeks to compress the architecture for a given hardware design ( e.g . GPU or lower17 precision chips ) . However , special-purpose hardware is being designed with neural network inference18 in mind . This leads to a new problem formulation which we study here : design an efficient hardware19 architecture which allows networks to be trained on GPUs , then implemented on the hardware.20 The combined problem of hardware and network design is complex , and the precise measurement21 of efficiency is both device and problem specific , taking into account latency , memory , energy22 consumption . Here we deliberately oversimplify the problem in order to make it tractable , by23 addressing a fundamental element of hardware cost . As a coarse surrogate efficiency , we use the24 number of logic gates required to implement an arithmetic operation on chip . While this is very25 coarse , and full costs will depend on other aspects of hardware implementation , it nevertheless26 represents a fundamental unit of cost in hardware design [ 23 ] .27 In a standard architecture , weights are multiplied by inputs , so the fundamental operation is multi-28 plication Sconv ( x , w ) = wx . In our work , we replace multiplication with the EuclidNet operator,29 30 Seuclid ( x , w ) = 1 2 |x w|2 . ( 1 ) which combines a difference with a squaring operator . We will refer to the family of networks that use31 ( 1 ) as EuclidNets . EuclidNets are a compromise between standard architecture , and AdderNets [ 9 ] ,32 which remove multiplication entirely , but at the cost of a significant loss of accuracy as well as33 difficulty training . Replacing multiplication with squaring is about half the cost ( on chip ) , depending34 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . on the number of bits used to represent the integer . The feature representation of each of the35 architectures is illustrated in Figure 1 . EuclidNets can be implemented on 8-bit precision without36 loss of accuracy , see Table 1.37 The squaring operator is cheaper ( in terms of logic gates ) than multiplication and can be reduced38 to a tiny look up table if run on integer values . [ 5 , 14 ] prove replacing look up table can replace39 actual float computing , but results in practice do not translate to inference speed-up [ 28 ] . Works40 such as LookNN in [ 38 ] take the first step in designing hardware for look up table use . On a low41 precision chip , we can compute Seuclid for about half the cost as Sconv , because hardware efficiencies42 for squaring two a fixed precision integer more than offsets the additional cost of a difference . At the43 same time , the network does not lose expressivity , as explained below . To summarize , we make the44 following contributions45 • We design an architecture based on replacing the multiplication Sconv ( x , w ) = wx by the46 squared difference ( 1 ) . Quantized networks using this operation require about half the cost47 ( measured by gate operators ) on a custom chipset.48 • These networks are just as expressive as convolutional networks . In practice , they have49 comparable accuracy ( drop of less than 1 percent on ImageNet on ResNet50 going from full50 precision convolutional to 8-bit Euclid ) .51 • In contrast to other network compression techniques , we can train and quantize these52 networks on GPUs without additional cost or difficulty.53 2 Context and related work54 Neural compression comes at the cost of a loss of accuracy , and may also increase training time ( to55 a greater extent on quantized networks ) [ 19 , 12 ] . Part of the drop in accuracy comes simply from56 decreasing model size , which is required for IoT and edge devices [ 42 ] . Some of the most common57 neural compression methods include pruning [ 39 ] , quantization [ 21 ] , knowledge distillation [ 24 ] , and58 efficient design [ 27 , 25 , 47 , 41 ] . Here we focus on a small , unorganized sub-field of compression,59 that optimizes mathematical operations in the network . This approach can be combined successfully60 with common other compression methods like quantization [ 44 ] .61 The most natural approach is low bit quantization [ 21 ] . The inference gains improves with lowering62 bit size , at the cost of accuracy drop and longer training . In the extreme case of binary networks,63 operations have negligible cost at inference but exhibits a considerable accuracy drop [ 26 ] .64 Knowledge distillation [ 24 ] consists of transferring information form a larger teacher network to a65 smaller student network . The idea is easily extended by thinking of information transfer between66 different similarity measures , which [ 44 ] explore in the context of AdderNets . Knowledge distillation67 is an uncommon training procedure and requires extra implementation effort . EuclidNet keeps the68 accuracy without knowledge distillation . We suggest a straightforward training using a smooth69 transition between common convlotution and Euclid operation.70 3 Network architecture and similarity operators71 Consider an intermediate layer of a neural network with input x 2 RH⇥W⇥cin and output72 y 2 RH⇥W⇥cout where H , W are the dimensions of the input feature , and cin , cout the num-73 ber of input and output channels , respectively . For a standard convolutional network , represent the74 transformation from input to output via weights w 2 Rd⇥d⇥cin⇥cout as75 ymnl = m+dX i=m n+dX j=n cinX k=0 xijkwijkl ( 2 ) Setting d = 1 recovers the fully-connected layer . We can abstract the multiplication of the weights wijkl by xijkl in the equation above by using a similarity measure S : R⇥R ! R. The convolutional layer corresponds to Sconv ( x , w ) = xw . In our work , we replace Sconv with Seuclid , given by ( 1 ) . A number of works have also replaced the76 multiplication operator in a neural network . The most relevant work is the AdderNet of [ 9 ] , which77 instead uses78 Sadder ( x , w ) = |x w| . ( 3 ) replacing multiplication by the absolute value of the difference . This operation can be implemented79 very efficiently on a custom chipset : subtraction and absolute value of a different of n-bit integers80 cost order n gate operations , compared to order n2 for multiplication Sconv ( x , w ) = xw . However,81 AdderNet comes with a significant loss in accuracy , and is difficult to train.82 3.1 Other Measures of similarity in neural network architectures83 The idea of replacing multiplication operations to save resources within the context of neural networks84 dates back to 1990s . Equally motivated by computational speed-up and hardware requirement85 minimization , [ 17 ] define perceptrons that use the synapse similarity,86 Ssynapse ( x , w ) = sign ( x ) · sign ( w ) ·min ( |x| , |w| ) , ( 4 ) which is cheaper than multiplication.87 Although ( 4 ) has not been experimented with in modern models and datasets , [ 2 ] introduced a slight88 variation , the multiplication-free operator,89 Smfo ( x , w ) = sign ( x ) · sign ( w ) · ( |x|+ |w| ) ) . ( 5 ) Note that both ( 4 ) and ( 5 ) induce the l1-norm . [ 32 ] explains that the updated design choice allows90 contributions from both operands x and w. [ 1 ] studies the similarity in image classification on91 CIFAR10 . Other applications of ( 5 ) include [ 4 , 36 ] .92 [ 46 ] further combines this similarity with a bit-shift , and claims an improved accuracy with negligible93 added cost . However , the plotted results for AdderNet appear lower than those reported in [ 9 ] .94 Another follow-up work uses knowledge distillation to further improve the accuracy of AdderNets95 [ 44 ] .96 Instead of simply replacing the similarity on the summation , there is also the possibility to replace the97 full expression on ( 2 ) . [ 30 , 31 ] approximate the activation of a given layer with an exponential term.98 Unfortunately , it only leads to speed-up in certain cases and , in particular , it does not improve CPU99 inference time . Reported accuracy on benchmark problems is also lower than the typical baseline.100 In a recent work , [ 34 ] used three layer morphological neural networks for image classification.101 Morphological neural networks were introduced in 1990s by [ 15 , 40 ] and use the notion of erosion102 and dilation to replace ( 2 ) :103 Erosion ( x , w ) = min j S ( xj , wj ) = min j ( xj wj ) , Dilation ( x , w ) = max j S ( xj , wj ) = max j ( xj + wj ) . The authors propose two methods of stacking layers to expand networks , but admit the possibility of104 over-fitting and difficult training issues , casting doubt on scalability of the method.105 4 Theoretical Results for EuclidNets106 4.1 Expressivity of the EuclidNet network107 Networks using the EuclidNet operation as just as expressive as those using multiplication , thanks to108 the polarization identity,109 Sconv ( x , w ) = Seuclid ( x , w ) Seuclid ( x , 0 ) Seuclid ( 0 , w ) which means that any multiplication operation can be expressed using only Euclid operations.110 4.2 Logic Gate Cost for EuclidNet compared to ConvNet ( multipication ) 111 The above similarity may not come across immediately as an improved choice on the cost of112 convolutions . It requires personalized hardware to obtain gains in inference speed like the other113 similarities . For example , in a typical architecture , the cost of addition is very close to multiplication,114 and squaring is usually not considered distinctly from multiplication [ 30 , Table III ] . Hence , first we115 discuss what these gains are theoretically . As for training , unlike other competitors such as AdderNet116 that embodies a considerable slow training , we implement the Euclid similarity in a way that is only117 slightly slower than Sconv.118 Here we provide a brief theoretical analysis of119 basic binary operations on custom hardware that120 is optimized for model inference . Assuming121 equal cost between AND , XOR and OR gates,122 we first compute the cost of gate-level integer123 operations , defined in Appendix A.1 . See Fig-124 ure 2125 The following formula gives the gate count of126 n-bit operations:127 Sconv = 6n 2 8n+ 3 Seuclid = 3n 2 + n/2 3 ( with a minor modification to the second for-128 mula to 3n2 + n/2 3/2 when n is odd ) , refer129 to Table A.4.130 The hardware implementation of an n-bit adder131 is implemented using one half-adder and n 1132 full-adders . A half-adder circuit is made up of 1133 XOR gate and 1 AND gate , while the full-adder circuit requires 2 XOR gates , 2 AND gates and 1 OR134 gate . Therefore , the cost of an n bit addition is 5n 3.135 There are n2 AND gates for n-bit element wise multiplications . A common architecture usually136 include ( n 1 ) n-bit adders besides the n2 AND gates . One n-bit adders is composed of one137 half-adder and n 1 full-adders . Hence the cost of multiplication is 6n2 8n+ 3.138 In the case of squaring , there are less AND gates representing element-wise multiplication . We139 consider two different cases : i ) if n is even the cost of squaring is 3n2 92n ii ) if n is odd , the cost140 of squaring is 3n2 92n+ 3 2 ,141 5 Training EuclidNets142 Training EuclidNets are much easier compared with other competitors such as AdderNets . This143 makes EuclidNet attractive for complex tasks such as image segmentation , and object detection144 where training compressed networks are challenging and causes large accuracy drop . However,145 EuclidNets are more expensive than AdderNets on floating points , but their quantization behavior146 unlike AdderNets resembles traditional convolution to a great extent . In another words EuclidNets147 are easiy to quantize.148 While training a network , it is more appropriate to use the identity149 Seuclid ( x , w ) = x 2 2 w 2 2 + xw , ( 6 ) and use this equation while training EuclidNets on GPUs which are optimized for inner product.150 Therefore training EuclidNets doesn ’ t require additional CUDA core [ 35 ] implementation unlike151 AdderNets . The official implementation of AdderNet [ 9 ] reflects order of 20⇥ slower training than152 the traditional convolution on Pytorch . This is specially problematic for large networks and complex153 tasks that even traditional convolution training takes few days or even weeks . EuclidNet training154 is 2⇥ in the worst case and their implementation is natural in deep learning frameworks such as155 PyTorch and Tensorflow.156 A common method in training neural networks is fine-tuning , initializing with weights trained on157 different data but with a similar nature . Here , we introduce the idea of using a weight initialization158 from a model trained on a related similarity.159 Rather than training from scratch , we wish to fine-tune EuclidNet starting from accurate CNN weights.160 This is achieved by an “ architecture homotopy '' where we change hyperparameters to convert a regular161 convolution to an Euclid operation162 S ( x , w ; k ) = xw k x 2 + w2 2 , with k = 0 + 1 0 n · k , ( 7 ) where n is the total number of epochs and 0 < 0 < 1 is the initial transition phase . Note that163 S ( x , w , 0 ) = Sconv ( x , w ) and S ( x , w , 1 ) = Seuclid ( x , w ) and equation 7 is the convex combination164 of the two similarities . One may interpret k as a schedule for the homotopy parameter , similar to165 how a schedule is defined for the learning rate in training a deep network . We found that a linear166 schedule above is effective empirically.167 Transformations like ( 7 ) are commonly used in scientific computing [ 3 ] . The idea of using homotopy168 in training neural networks can be traced back to [ 13 ] . Recently , homotopy was used in deep learning169 in the context of activation functions [ 37 , 8 , 33 , 18 ] , loss functions [ 20 ] , compression [ 10 ] and transfer170 learning [ 6 ] . Here , we use homotopy in the context of transforming network operations.171 Fine-tuning method in ( 7 ) is inspired by continuation methods in partial differential equations.172 Assume S is a solution for a differential equation with the initial condition S ( x , 0 ) = S0 ( x ) . In173 certain situations , solving this differential equation for S ( x , t ) and then evaluating at t = 1 might be174 simpler than solving directly for S1 . One may think of this homotopy method as an evolving neural175 network over time . At time zero the neural network consists of regular convolutional layers , but at176 time one transforms to Euclidean layers.177 The homotopy method can be interpreted as a sort of of knowledge distillation . Whereas knowledge178 distillation methods tries to match a student network to a teacher network , the homotopy can be seen179 as a slow transformation from the teacher network into a student network . Figure 3 shows a scheme180 of the idea . Curiously , problems that have been solved with homotopic approaches have also been181 tackled by knowledge distillation . For example , removing blocks or layers from a network [ 24 , 10 ] 182 along with transfer learning [ 45 , 6 ] .183 6 Experiments184 We consider try our proposed method on image classification task . Future work could be extended to185 other domains of application such as natural language and speech.186 6.1 CIFAR10187 First , we consider the CIFAR10 dataset , consisting of 32 ⇥ 32 RGB images with 10 possible188 classifications [ 29 ] . We normalize and augment the dataset with random crop and random horizontal189 flip . We consider two ResNet models [ 22 ] , ResNet-20 and ResNet-32.190 We train EuclidNet using the optimizer from [ 9 ] , which we will refer to as AdderSGD , to evaluate191 EuclidNet under a similar setup . We use initial learning rate 0.1 with cosine decay , momentum 0.9192 and weight decay 5⇥ 10 4 . We follow [ 9 ] in setting the learning-rate scaling parameter ⌘ . However,193 we use a batch-size of 128 for memory reasons . For traditional convlution network , we use the same194 hyper-parameters with stochastic gradient descent optimizer.195 In Table 3 we provide the details of classification accuracy . We consider two different weight196 initialization for EuclidNets . First , we initialize randomly and second , we initialize from weights197 pre-trained on a convolutional network . The accuracy for EuclidNets is approximately the same as for198 a standard ResNet . We see that for CIFAR10 training from scratch achieves even a higher accuracy,199 while initializing with convolution network and using linear Homotopy training improves it even200 further.201 During training , EuclidNets are unstable , despite careful choice of the optimizer . In Figure 4202 we compare with training the corresponding convolutional network . Fine-tuning directly from203 convolutional weights is more stable than training from scratch as expected . However , accuracy is204 lower but the convergence is faster when we use homotopy training and the accuracy is improved.205 Pre-trained convolution weights are commonly available in the most of neural compression tasks , so206 initializing EuclidNets with pre-trained convolution is more natural and preferable.207 EuclidNets are not only faster to train compared208 with other competitors , but also stand superior in209 terms of accuracy . AdderNet performs slightly210 worse but is much slower to train . The accu-211 racy is significantly lower for the synapse and212 the multiplication-free operator . In Table 4 we213 record top-1 accuracy obtained in which Adder-214 Net results are borrowed from [ 44 ] , that use215 knowledge distillation to close the gap with the216 full precision but still falls short compared with217 EuclidNet.218 Training a quantized Seuclid is very similar sim-219 ilar to convolution . This allows a wider use of220 such networks for lower resource devices . Quan-221 tization of the Euclid model to 8bits keeps accu-222 racy drop within the range of one percent [ 43 ] 223 similar to traditional convolution so they are224 like convolution when run on lower bits . Table 1225 shows 8-bit quantization of EuclidNet where the226 accuracy drop remains negligible . Similar to tra-227 ditional convolution , EuclidNets on CIFAR100228 exhibit a larger accuracy drop compared to CI-229 FAR10 , probably due to the complexity of the230 classification problem.231 6.2 ImageNet232 Next , we consider EuclidNet classifier built on ImageNet , a more challenging task ImageNet [ 16 ] .233 We train our baseline with standard augmentations of random resized crop and horizontal flip and234 normalization . We consider ResNet-18 and ResNet-50 models . Hyper-parameters tuning follows235 Section 6.1.236 Table 5 shows top-1 and top-5 classification accuracies . The accuracy from while EuclidNet is trained237 from scratch is lower , showing the importance of homotopy training . We believe that the accuracy238 drop with no homotopy is the difficulty of tuning training hyper-parameters for a large dataset such as239 ImageNet . Even though hyper-parameters that achieve equivalent accuracy from random initialization240 exist , they are too difficult to find . It is much easier to use the existing hyperparameters of traditional241 convolution , and transfer the geometry through homotopy training.242 7 Conclusion243 Euclid networks are obtained from typical neural models by replacing multiplication in convolutional244 layers by the Euclidean similarity . They are designed to be implemented on a custom designed low245 precision chipset , with the idea that subtraction and squaring can be implemented using approximately246 half the logic gates , compared to multiplication.247 While other efficient architectures can be difficult to train in low precision , EuclideNets are easily248 trained in low precisions . EuclidNets can be initialized with weights trained on the correspondent249 ConvNet to save training time , so on may regard them as a fine tuning convolutiuonal networks for a250 cheaper inference . The homotopy method further improves training in such scenarios and training251 using this method sometimes surpass regular convolution accuracy . Future work may focus on252 developing hardware that can realize the expected inference time losses and try similar experiments253 on down stream vision tasks like object detection and segmentation.254 7.1 Limitations255 While gate counts provide a fundamental method for assessing the cost of a chip , they are a crude256 estimate , and the real costs ( in terms of power usage , inference time , and memory ) of a chipset and257 architecture combination are much more complex to estimate . True final costs can require a hardware258 simulator or implementation . At the same time , the gate count provides a first approximation to the259 cost , and the fact that we can train and match accuracy in eight bit precision is promising.260 7.2 Societal Impact261 Deep Neural Network inference is costly in terms of power usage . If we can design and implement262 efficient architectures , this will reduce the societal cost of running these models on edge devices.263 References264 [ 1 ] Arman Afrasiyabi , Diaa Badawi , Baris Nasir , Ozan Yildi , Fatios T Yarman Vural , and A Enis Çetin.265 Non-euclidean vector product for neural networks . In 2018 IEEE International Conference on Acoustics,266 Speech and Signal Processing ( ICASSP ) , pages 6862–6866 . IEEE , 2018.267 [ 2 ] C. E. Akbaş , A. Bozkurt , A. E. Çetin , R. Çetin-Atalay , and A. Üner . Multiplication-free neural networks.268 In 2015 23nd Signal Processing and Communications Applications Conference ( SIU ) , pages 2416–2418,269 2015.270 [ 3 ] Eugene L Allgower and Kurt Georg . Introduction to numerical continuation methods . SIAM , 2003.271 [ 4 ] Diaa Badawi , Ece Akhan , Ma ’ en Mallah , Ayşegül Üner , Rengül Çetin-Atalay , and A Enis Çetin . Multipli-272 cation free neural network for cancer stem cell detection in h-and-e stained liver images . In Compressive273 Sensing VI : From Diverse Modalities to Big Data Analytics , volume 10211 , page 102110C . International274 Society for Optics and Photonics , 2017.275 [ 5 ] Shumeet Baluja , David Marwood , Michele Covell , and Nick Johnston . No multiplication ? no floating276 point ? no problem ! training networks for efficient inference . arXiv preprint arXiv:1809.09244 , 2018.277 [ 6 ] Yoshua Bengio , Jérôme Louradour , Ronan Collobert , and Jason Weston . Curriculum learning . In278 Proceedings of the 26th annual international conference on machine learning , pages 41–48 , 2009.279 [ 7 ] Hadjer Benmeziane , Kaoutar El Maghraoui , Hamza Ouarnoughi , Smail Niar , Martin Wistuba , and280 Naigang Wang . A comprehensive survey on hardware-aware neural architecture search . arXiv preprint281 arXiv:2101.09336 , 2021.282 [ 8 ] Zhangjie Cao , Mingsheng Long , Jianmin Wang , and Philip S Yu . Hashnet : Deep learning to hash by283 continuation . In Proceedings of the IEEE international conference on computer vision , pages 5608–5617,284 2017.285 [ 9 ] Hanting Chen , Yunhe Wang , Chunjing Xu , Boxin Shi , Chao Xu , Qi Tian , and Chang Xu . Addernet : Do we286 really need multiplications in deep learning ? In Proceedings of the IEEE/CVF Conference on Computer287 Vision and Pattern Recognition , pages 1468–1477 , 2020.288 [ 10 ] Qipin Chen and Wenrui Hao . An efficient homotopy training algorithm for neural networks . 2019.289 [ 11 ] Yu Cheng , Duo Wang , Pan Zhou , and Tao Zhang . A survey of model compression and acceleration for290 deep neural networks . arXiv preprint arXiv:1710.09282 , 2017.291 [ 12 ] Yu Cheng , Duo Wang , Pan Zhou , and Tao Zhang . Model compression and acceleration for deep neural292 networks : The principles , progress , and challenges . IEEE Signal Processing Magazine , 35 ( 1 ) :126–136,293 2018.294 [ 13 ] J Chow , L Udpa , and SS Udpa . Homotopy continuation methods for neural networks . In 1991. , IEEE295 International Sympoisum on Circuits and Systems , pages 2483–2486 . IEEE , 1991.296 [ 14 ] Michele Covell , David Marwood , Shumeet Baluja , and Nick Johnston . Table-based neural units : Fully297 quantizing networks for multiply-free inference . arXiv preprint arXiv:1906.04798 , 2019.298 [ 15 ] Jennifer L Davidson and Gerhard X Ritter . Theory of morphological neural networks . In Digital Optical299 Computing II , volume 1215 , pages 378–388 . International Society for Optics and Photonics , 1990.300 [ 16 ] J. Deng , W. Dong , R. Socher , L.-J . Li , K. Li , and L. Fei-Fei . ImageNet : A Large-Scale Hierarchical Image301 Database . In CVPR09 , 2009.302 [ 17 ] Radu Dogaru and Leon O Chua . The comparative synapse : A multiplication free approach to neuro-303 fuzzy classifiers . IEEE Transactions on Circuits and Systems I : Fundamental Theory and Applications,304 46 ( 11 ) :1366–1371 , 1999.305 [ 18 ] Farnoush Farhadi , Vahid Nia , and Andrea Lodi . Activation adaptation in neural networks . In Proceedings of306 the 9th International Conference on Pattern Recognition Applications and Methods - Volume 1 : ICPRAM , ,307 pages 249–257 . INSTICC , SciTePress , 2020.308 [ 19 ] Jonathan Frankle and Michael Carbin . The lottery ticket hypothesis : Finding sparse , trainable neural309 networks . arXiv preprint arXiv:1803.03635 , 2018.310 [ 20 ] Caglar Gulcehre , Marcin Moczulski , Francesco Visin , and Yoshua Bengio . Mollifying networks . arXiv311 preprint arXiv:1608.04980 , 2016.312 [ 21 ] Yunhui Guo . A survey on methods and theories of quantized neural networks . arXiv preprint313 arXiv:1808.04752 , 2018.314 [ 22 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition.315 corr abs/1512.03385 ( 2015 ) , 2015.316 [ 23 ] John L Hennessy and David A Patterson . Computer architecture : a quantitative approach . Elsevier , 2011.317 [ 24 ] Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . Distilling the knowledge in a neural network . arXiv318 preprint arXiv:1503.02531 , 2015.319 [ 25 ] Andrew G Howard , Menglong Zhu , Bo Chen , Dmitry Kalenichenko , Weijun Wang , Tobias Weyand , Marco320 Andreetto , and Hartwig Adam . Mobilenets : Efficient convolutional neural networks for mobile vision321 applications . arXiv preprint arXiv:1704.04861 , 2017.322 [ 26 ] Itay Hubara , Matthieu Courbariaux , Daniel Soudry , Ran El-Yaniv , and Yoshua Bengio . Binarized neural323 networks . In D. Lee , M. Sugiyama , U. Luxburg , I. Guyon , and R. Garnett , editors , Advances in Neural324 Information Processing Systems , volume 29 . Curran Associates , Inc. , 2016.325 [ 27 ] Forrest N Iandola , Song Han , Matthew W Moskewicz , Khalid Ashraf , William J Dally , and Kurt Keutzer.326 Squeezenet : Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size . arXiv preprint327 arXiv:1602.07360 , 2016.328 [ 28 ] Martin Kersner . Convolutional network without multiplication operation , Mar 2019.329 [ 29 ] Alex Krizhevsky , Geoffrey Hinton , et al . Learning multiple layers of features from tiny images . 2009.330 [ 30 ] Elena Limonova , Daniil Alfonso , Dmitry Nikolaev , and Vladimir V Arlazarov . Resnet-like architecture331 with low hardware requirements . arXiv preprint arXiv:2009.07190 , 2020.332 [ 31 ] Elena Limonova , Daniil Matveev , Dmitry Nikolaev , and Vladimir V Arlazarov . Bipolar morphological333 neural networks : convolution without multiplication . In Twelfth International Conference on Machine334 Vision ( ICMV 2019 ) , volume 11433 , page 114333J . International Society for Optics and Photonics , 2020.335 [ 32 ] Maen Mallah . Multiplication free neural networks . PhD thesis , Bilkent University , 2018.336 [ 33 ] Hossein Mobahi . Training recurrent neural networks by diffusion . arXiv preprint arXiv:1601.04114 , 2016.337 [ 34 ] Ranjan Mondal , Sanchayan Santra , and Bhabatosh Chanda . Dense morphological network : An universal338 function approximator , 2019.339 [ 35 ] NVIDIA , Péter Vingelmann , and Frank H.P . Fitzek . Cuda , release : 10.2.89 , 2020.340 [ 36 ] Hongyi Pan , Diaa Badawi , Xi Zhang , and Ahmet Enis Cetin . Additive neural network for forest fire341 detection . Signal , Image and Video Processing , pages 1–8 , 2019.342 [ 37 ] Harsh Nilesh Pathak and Randy Paffenroth . Parameter continuation methods for the optimization of deep343 neural networks . In 2019 18th IEEE International Conference On Machine Learning And Applications344 ( ICMLA ) , pages 1637–1643 . IEEE , 2019.345 [ 38 ] Mohammad Samragh Razlighi , Mohsen Imani , Farinaz Koushanfar , and Tajana Rosing . Looknn : Neural346 network with no multiplication . In Design , Automation & Test in Europe Conference & Exhibition ( DATE ) ,347 2017 , pages 1775–1780 . IEEE , 2017.348 [ 39 ] Russell Reed . Pruning algorithms-a survey . IEEE transactions on Neural Networks , 4 ( 5 ) :740–747 , 1993.349 [ 40 ] Gerhard X Ritter and Peter Sussner . An introduction to morphological neural networks . In Proceedings of350 13th International Conference on Pattern Recognition , volume 4 , pages 709–717 . IEEE , 1996.351 [ 41 ] Mingxing Tan and Quoc V Le . Efficientnet : Rethinking model scaling for convolutional neural networks.352 arXiv preprint arXiv:1905.11946 , 2019.353 [ 42 ] Carole-Jean Wu , David Brooks , Kevin Chen , Douglas Chen , Sy Choudhury , Marat Dukhan , Kim Hazel-354 wood , Eldad Isaac , Yangqing Jia , Bill Jia , et al . Machine learning at facebook : Understanding inference at355 the edge . In 2019 IEEE International Symposium on High Performance Computer Architecture ( HPCA ) ,356 pages 331–344 . IEEE , 2019.357 [ 43 ] Hao Wu , Patrick Judd , Xiaojie Zhang , Mikhail Isaev , and Paulius Micikevicius . Integer quantization for358 deep learning inference : Principles and empirical evaluation , 2020.359 [ 44 ] Yixing Xu , Chang Xu , Xinghao Chen , Wei Zhang , Chunjing Xu , and Yunhe Wang . Kernel based360 progressive distillation for adder neural networks . arXiv preprint arXiv:2009.13044 , 2020.361 [ 45 ] Junho Yim , Donggyu Joo , Jihoon Bae , and Junmo Kim . A gift from knowledge distillation : Fast362 optimization , network minimization and transfer learning . In Proceedings of the IEEE Conference on363 Computer Vision and Pattern Recognition , pages 4133–4141 , 2017.364 [ 46 ] Haoran You , Xiaohan Chen , Yongan Zhang , Chaojian Li , Sicheng Li , Zihao Liu , Zhangyang Wang,365 and Yingyan Lin . Shiftaddnet : A hardware-inspired deep network . In Hugo Larochelle , Marc ’ Aurelio366 Ranzato , Raia Hadsell , Maria-Florina Balcan , and Hsuan-Tien Lin , editors , Advances in Neural Information367 Processing Systems 33 : Annual Conference on Neural Information Processing Systems 2020 , NeurIPS368 2020 , December 6-12 , 2020 , virtual , 2020.369 [ 47 ] Xiangyu Zhang , Xinyu Zhou , Mengxiao Lin , and Jian Sun . Shufflenet : An extremely efficient convolutional370 neural network for mobile devices . In Proceedings of the IEEE conference on computer vision and pattern371 recognition , pages 6848–6856 , 2018.372 Checklist373 1 . For all authors ... 374 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s375 contributions and scope ? [ Yes ] 376 ( b ) Did you describe the limitations of your work ? [ Yes ] , see subsection 7.1.377 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] , see378 subsection 7.2.379 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to380 them ? [ Yes ] 381 2 . If you are including theoretical results ... 382 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 383 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] 384 3 . If you ran experiments ... 385 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-386 mental results ( either in the supplemental material or as a URL ) ? [ Yes ] 387 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they388 were chosen ) ? [ Yes ] 389 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running exper-390 iments multiple times ) ? [ No ] . It was too costly to train multiple times , we just ran391 once.392 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type393 of GPUs , internal cluster , or cloud provider ) ? [ No ] , but we gave standard training394 details in section 6.395 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 396 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 397 ( b ) Did you mention the license of the assets ? [ N/A ] 398 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 399 400 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re401 using/curating ? [ N/A ] 402 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable403 information or offensive content ? [ N/A ] 404 5 . If you used crowdsourcing or conducted research with human subjects ... 405 ( a ) Did you include the full text of instructions given to participants and screenshots , if406 applicable ? [ N/A ] 407 ( b ) Did you describe any potential participant risks , with links to Institutional Review408 Board ( IRB ) approvals , if applicable ? [ N/A ] 409 ( c ) Did you include the estimated hourly wage paid to participants and the total amount410 spent on participant compensation ? [ N/A ] 411

Summary: This work proposes EuclidNet, which replaces the multiplication in DNN with squared differences with less logic gates. It is proved to be as expressive as convolution networks and able to be trained in GPUs with at most 2x more time. The experiments are conducted on ResNet@CIFAR-10 and ImageNet.