EuclidNets: Combining hardware and architecture design for Efficient Inference and Training
Authors: Mariana Oliveira Prazeres
Xinlin Li
Vahid Partovi Nia
Adam M Oberman
Venue: NeurIPS 2021 Submitted (2021)
Keywords: deep Neural Networks
hardware aware architecture
compressed models

Abstract: In order to deploy deep neural networks on edge devices, compressed (resource efficient) networks need to be developed.  While established compression methods, such as quantization, pruning, and architecture search are designed for conventional hardware,  further gains are possible if compressed architectures are coupled with novel hardware designs.   In this work, we propose EuclidNet, a compressed network designed to be implemented on hardware which replaces multiplication, $wx$, with squared difference $(x-w)^2$.  EuclidNet allows for a low precision hardware implementation which is about twice as efficient (in term of logic gate counts) as the comparable conventional hardware, with acceptably small loss of accuracy.   Moveover, the network can be trained and quantized using standard methods, without requiring additional training time.  Codes and pre-trained models are available at \url{http://github.com/anonymous/}.

EuclidNets:
combining hardware and architecture design for efficient training and inference
Anonymous Author(s)
Affiliation
Address
email
Abstract
In order to deploy deep neural networks on edge devices, compressed (resource efficient) networks need be developed. While established compression methods, such as quantization, pruning, and architecture search are designed for conventional hardware, further gains are possible if compressed architectures are coupled with novel hardware designs. In this work, we propose EuclidNet, compressed network designed be implemented on hardware which replaces multiplication, wz, with squared difference (z w)². EuclidNet allows for low precision hardware implementation which is about twice as efficient (in term of logic gate counts) as the comparable conventional hardware, with acceptably small loss of accuracy. Moveover, the network can be trained and quantized using standard methods, without requiring additional training time. Codes and pre-trained models are available http://ggthhtt com/ anonymous/
10
12
Introduction
14
While the majority of deep neural networks are designed to be implemented on GPUs, they are increasingly being deployed on edge devices, such as mobile phones. These edge devices require compressed (more efficient), hardware aware architectures, due to memory and power constraints Α.. which seeks to compress the architecture for a given hardware design (e.g. GPU or lower precision chips). However, special- in mind. This leads to new problem formulation which we study here: design an efficient hardware architecture which allows networks to be trained on GPUs, then implemented on the hardware.
16
18
hardware is being designed with neural network inference
19
20
The combined problem of hardware and network design is complex, and the precise measurement of efficiency is both device and problem specific, taking into account latency, memory, energy consumption. Here we deliberately oversimplify the problem in order to make it tractable, by addressing a fundamental element of hardware cost. As a coarse surrogate efficiency, we use the number of logic gates required implement an arithmetic operation chip. While this is very
22
24
26 coarse, and full costs will depend on other aspects of hardware implementation, it nevertheless represents a fundamental unit of cost in hardware design 23
28 In a standard architecture, weights are multiplied by inputs, so the fundamental operation is multi- plication Scome (ær, w) war. In our work, we replace multiplication with the FuclidNet operator,
30
1
Seuclid (I, w)
w|?
31 which combines difference with squaring operator. We will refer to the family of networks that use
32 ① as EuclidNets. EuclidNets are compromise between standard architecture, and AdderNets ( which remove multiplication entirely, but at the cost of a significant loss of accuracy well as 34 difficulty training. Replacing multiplication with squaring is about half the cost (on chip), depending
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

(right).
Figure Feature representation of traditional convolution with S(x, w) aw (left), AdderNet Szz, w) = -I w| (middle), EuclidNet (xx w) w|2
35 on the number of bits used to represent the integer. The feature representation of each of the
36 architectures is illustrated in Figure EuclidNets can be implemented on 8-bit precision without loss of accuracy, see Table
37
38
The squaring operator is cheaper (in terms of logic gates) than multiplication and can reduced tiny look up table if run integer values. (5) D4) prove replacing look up table replace actual float computing, but results in practice do not translate to inference speed-up (28] Works such LookNN in (38 take the first step in designing hardware for look up table On low precision chip, we can compute Seuclid for about half the cost as Sconv, because hardware efficiencies 42
39
40
43
for squaring two fixed precision integer more than offsets the additional cost of difference. At the same time, the network does not lose expressivity, as explained below. To summarize, we make the following contributions
44
45
46
We design an architecture based on replacing the multiplication Scony (x. w) = w. by the squared difference (I). Quantized networks using this operation require about half the cost (measured by gate operators) on a custom chipset.
48
49
These networks are just as expressive as convolutional networks. In practice, they have comparable accuracy (drop of less than 1 percent on ImageNet on ResNet50 going from full precision convolutional to 8-bit Euclid).
50
51
In contrast to other network compression techniques, networks on GPUs without additional cost or difficulty.
train and quantize these
53
Table 1: Euclid-Net Accuracy with full precision and 8-bit quantization: Results ResNet-20 with Euclidian similarity for CIFARIO and CIFAR 100, and results on ResNet-18 8 for ImageNet. Euclid-Net achieves comparable or better accuracy with 8-bit precision, compared to the standard full precision convolutional network.
Top-1 accuracy
Network
Quantization Full precision 8-bit Full precision 8-bit
Chip Efficiency
CIFAR10
CIFAR
ImageNet
X
92.97
68.14
69.56
Sconv
92.07
68.02
69.59
X
93.32
68,84
69.69
Souclid
/
93.30
68.78
68.59
Full precision 8-bit
X
91.84
67.60
67.0
Sudder
91.78
67.60
68.8
BNN
1-bit
84.87
54.14
51.2
2
Context and related work
55 Neural compression comes the cost loss of accuracy, and may also increase training time (to greater extent quantized networks) 19 [2]. Part of the drop in accuracy comes simply from

---

57
decreasing model size, which is required for IoT and edge devices 42 Some of the most common neural compression methods include pruning [9 quantization 211. knowledge distillation A and efficient design 27
58
Here we focus on a small, unorganized sub-field of compression, that optimizes mathematical operations the network. This approach can be combined successfully
25 47
60
61
with common other compression methods like quantization 4
62
The most natural approach low bit quantization 211 The inference gains improves with lowering bit size, at the cost of accuracy drop and longer training. In the extreme case of binary networks, operations have negligible cost at inference but exhibits considerable accuracy drop
68
64
65
Knowledge distillation 24 consists of transferring information form a larger teacher network to a smaller student network. The idea is easily extended by thinking of information transfer between different similarity measures, which [ explore in the context of AdderNets. Knowledge distillation uncommon training procedure and requires extra implementation effort. EuclidNet keeps the accuracy without knowledge distillation. We suggest a straightforward training using a smooth transition between common convlotution and Euclid operation.
67
69
70
3
Network architecture and similarity operators
71
Consider an intermediate layer of neural network with input x E xxxWxx.. and output y E Hxx Cous where H, W are the dimensions of the input feature, and Gm. Cout the num- ber of input and output channels, respectively. For standard convolutional network, represent the transformation from input to output via weights w E RRdxdxon Coont as
72
73
75
m+dn4d Cin
Umnl = ΣΣΣ
WijkWijki
I= K=
Setting d = recovers the fully-connected layer. We can abstract the multiplication of the weights Wijki by ziyki in the equation above by using similarity measure S RxR R - R. The convolutional
layer corresponds
Scone (I. w) = -...
76 In our work, we replace Scony with Seuclid. given by 4 A number of works have also replaced the multiplication operator in neural network. The most relevant work is the AdderNet of [9, which instead uses
78
Sadder (Ir, w) = -1 w|
replacing multiplication by the absolute value of the difference. This operation can be implemented very efficiently on a custom chipset: subtraction and absolute value of a different n-bit integers 80
cost order gate operations, compared to order n² for multiplication Scone (I, w) zw. However, AdderNet comes with a significant loss in accuracy, and is difficult to train.
82
83
3.1
Other Measures of similarity neural network architectures
The idea of replacing multiplication operations to save resources within the context of neural networks dates back to 1990s. Equally motivated by computational speed-up and hardware requirement minimization, 17 define perceptrons that use the synapse similarity,
85
Skynapse (z. w) sign(x) sign(w) min([2], [w]),
(4)
87 which is cheaper than multiplication.
Although has not been experimented with in modern models and datasets, variation, the multiplication-Fis operator,
? introduced slight
89
Smio (x, w) sign(z) sign(w) (|ar + lw])).
(5)
Note that both and (5) induce the 1n-nomm. 32 explains that the updated design choice allows
91
contributions from both operands x and w. 1 studies the similarity in image classification on
CIFARIO. Other applications of 5 include 4 36
82
33 4d
further combines this similarity with a bit-shift, and claims an improved accuracy with negligible added cost. However, the plotted results for AdderNet appear lower than those reported in "
3

---

95
Another follow-up work uses knowledge distillation to further improve the accuracy of AdderNets
96
4
87
Instead of simply replacing the similarity on the summation, there is also the possibility to replace the full expression (2). 1E 15ll approximate the activation of given layer with exponential term. Unfortunately, it only leads to speed-up in certain cases and, in particular, it does not improve CPU inference time. Reported accuracy on benchmark problems also lower than the typical baseline.
99
100
101
In a recent work,
34 used three layer morphological neural networks for image classification.
Morphological neural networks were introduced in 1990s by [I5 40 and use the notion of erosion and dilation to replace 2
103
Erosion (z, w) min (rr myy) min(a, wy),
Dilation(r, w) = max Slzj, wy) max(r, + wy).
The authors propose two methods of stacking layers to expand networks, but admit the possibility of over-fitie fitting and difficult training issues, casting doubt on scalability the method.
105
Theoretical Results for EuclidNets
108
107
4.1
Expressivity of the EuclidNet network
108
Networks using the EuclidNet operation as just as expressive as those using multiplication, thanks to the polarization identity,
109
Sconv (z, w) = Seuclid (I, w) Seuclia (a:, 0) Seuctia (0, w)
110
which means that any multiplication operation can be expressed using only Euclid operations.
111
4.2
Logic Gate Cost for EuclidNet compared to ConvNet (multipication)
112
The above similarity may not come across immediately as an improved choice on the cost of convolutions. requires personalized hardware to obtain gains in inference speed like the other similarities. For example, in typical architecture, the cost of addition is very close to multiplication, and squaring is usually not considered distinctly from multiplication BC Table III]. Hence, first we discuss what these gains are theoretically. As for training, unlike other competitors such as AdderNet that embodies considerable slow training, we implement the Euclid similarity in way that is only slightly slower than Scorv-
113
114
115
116
117
119
Here we provide brief theoretical analysis of basic binary operations on custom hardware that optimized for model inference. Assuming equal cost between AND, XOR and OR gates, we first compute the cost of gate-level integer operations, defined in Appendix A. See Fig-
120
6000
Sconv
122
5000
Seuclid odd n
123
4000
Seuclid
125
ure 2
3000
126
The following formula gives the gate count of bitit operations:
2000
1000
Scouv = 6n² 8n + 3
Senclid 3n2 n2/22
10
15
20
128
(with a minor modification to the second for- mula to 3n² + n/22 3/2 when n is odd), refer
Figure 2: Comparison of the number of logic gates (y-axis) as a function of the number of bits (æ-axis) FuclidNet compared with the standard ConvNet.
130
to Table A.4
The hardware implementation of an n-bit adder implemented using one half-adder and n full-adders. A half-adder circuit is made up of
132
194
XOR gate and AND gate, while the full-adder circuit requires 2 XOR gates, 2 AND gates and OR gate. Therefore, the cost of bit addition is 5n 3.

---

Table 2: Time (seconds) and maximum training batch-size that can fit in a signle GPU Tesla VI00- SXM2-32GB, during ImageNet training. parenthesis the slowdown with respect to the S Scone baseline. We do not show times for AdderNet, which is much slower than both, because it is not implemented in CUDA
Maximum Batch-size
Time per step Training
Model
Method
power of integer
Testing
Scouv
1024
1439
0.149
0.066
18
Seuclid
512
869 (1.7x)
0.157 (1.1x)
0.133 (2x)
Scone
256
371
0.182
0.145
ResNet-50
Seuelid
128
248 (1.5x)
0.274 (1.5x)
0.160 (1.1x)
There are n.? AND gates for n-bit element wise multiplications. A common architecture usually include (n 1) r--bit adders besides the n² AND gates, One n-bit adders is composed of one half-adder and n full-adders. Hence the cost of multiplication is 6n 8n + 3.
136
137
139
In the case of squaring, there are less AND gates representing element wise multiplication. We consider two different cases: i) if n is even the cost of squaring is 3n n ii) if n is odd, the cost of squaring is 3n² n
140
142
5
Training EuclidNets
143
Training EuclidNets are much easier compared with other competitors such AdderNets. This makes EuclidNet attractive for complex tasks such as image segmentation, and object detection where training compressed networks are challenging and causes large accuracy drop. However, EuclidNets are more expensive than AdderNets on floating points, but their quantization behavior unlike AdderNets resembles traditional convolution to great extent. another words EuclidNets are easiy to quantize.
144
145
146
147
149
While training network, it is more appropriate to use the identity
Seuchad (I, w) = x²
2 + rw,
(6)
150
and use this equation while training EuclidNets on GPUs which are optimized for inner product. Therefore training EuclidNets doesn't 'tt require additional CUDA core (35] implementation unlike AdderNets. The official implementation of AdderNet ( reflects order of 20 slower training than the traditional convolution on Pytorch. This is specially problematic for large networks and complex tasks that even traditional convolution training takes few days or even weeks. EuclidNet training 2x in the worst case and their implementation natural in deep learning frameworks such as Porcch and Tensorflow.
152
153
154
155
156
A common method in training neural networks is fine-tuning, initializing with weights trained on different data but with a similar nature. Here, we introduce the idea of using a weight initialization from model trained on a related similarity.
158
159
160
Rather than training from scratch, we wish to fine-tune EuclidNet starting from accurate CNN weights. This is achieved by "architecture homotopy" where change hyperparameters to convert regular convolution to an Euclid operation
162
S(x, wi Avo) = xw A 2 2
1 Ao
with A = Ao
ki
where is the total number of epochs and 0 < A 1 is the initial transition phase. Note that S(æ, w. 0) = Sconv (zz w) and S(x, w, 1) = Souelia (I, w) and equation is the convex combination of the two similarities. One may interpret A as a schedule for the homotopy parameter, similar to how a schedule is defined for the learning rate in training a deep network. We found that a linear schedule above is effective empirically.
164
166
188
Transformations like 7 are commonly used in scientific computing 3 The idea of using homotopy
in training neural networks can be traced back to 13] Recently, homotopy was used in deep learning
170
the context of activation functions
33 X K 18 loss functions 20), compression [TO) and transfer
learning [6]. Here, we use homotopy in the context of transforming network operations.
5

---

Teacher Network
Hamsstpy Network
Initial Block
Block So()
Block 3 5n()
Final Block
Hometapy Natwork =
Initial Block
Block S(.e)
Block 3 S(+e))
Final Block
Hamatopy Network a 2c
Initial Block
Block S(- 2c)
Block 3 2r)
Final Block
Student Network
Homotapy Network
Initial Block
Block $.
Block 3 5,(0)
Final
Block
2 =
Training schema of EuclidNet using Homotopy, i.e. transitioning from traditional convolu- tion ((r, w) = aw towards EuclidNet S(x, w) = - w/2 through equation
Figure 3:
172 Fine-tuning method in is inspired by continuation methods in partial differential equations.
173 Assume 5 is a solution for differential equation with the initial condition S(x, 0) = So(x). In certain situations, solving this differential equation for S(x, t) and then evaluating at might be simpler than solving directly for Sj. One may think of this homotopy method an evolving neural network over time. At time zero the neural network consists of regular convolutional layers, but at time one transforms to Euclidean layers.
174
176
178
The homotopy method can be interpreted distillation methods tries to match a student network to a teacher network, the homotopy can be seen as slow transformation from the teacher network into student network. Figure 3 shows scheme of the idea. Curiously, problems that have been solved with homotopic approaches have also been tackled by knowledge distillation. For example, removing blocks or layers from network 22 10
sort of of knowledge distillation. Whereas knowledge
179
180
182
183
along with transfer learning 45 6
6 Experiments
185
We consider try our proposed method on image classification task. Future work could be extended to other domains of application such as natural language and speech.
186
187
6.1
CIFARIO
188
First, we consider the CIFARIO dataset, consisting of 32 X 32 RGB images with 10 possible classifications 29
189
We normalize and augment the dataset with random crop and random horizontal [2 ResNet-20 and ResNet-32.
flip. We consider two ResNet models
We train EuclidNet using the optimizer from 9. which will refer AdderSGD, evaluate EuclidNet under similar setup. We use initial learning rate 0.1 with cosine decay, momentum 0.9 and weight decay 5 x 10-4 We follow ( in setting the learning-rate scaling parameter 7/. However, we use batch-size of 128 for memory reasons. For traditional convlution network, we use the same haper-parameters with stochastic gradient descent optimizer.
192
183
194
195
196
In Table we provide the details of classification accuracy. We consider two different weight initialization for EuclidNets. First, we initialize randomly and second, we initialize from weights pre-trane traited on a convolutional network. The accuracy for EuclidNets is approximately the same as for standard ResNet. We see that for CIFARI0 training from scratch achieves while initializing with convolution network and using linear Homotopy training improves even further.
198
higher accuracy,
200
202
During training, FuclidNets are unstable, despite careful choice of the optimizer. In Figure compare with training the corresponding convolutional network. Fine-tuning directly from

---

Table 3: Results on CIFARIO. The initial learning rate is adjusted for non-random initialization.
Top-1 accuracy
Model
Similarity
Initialization
Homotopy
Epochs
CIFARIO
CIFAR 100
Sconv
Random Random
None
400
92.97
69.29
None
450
93.00
68.84
ResNet-20
Seuclid
None
100
90.45
64.62
Conv
Linear
100
93.32
68.84
Sconv
Random
None
400
93.93
71.07
Random
None
450
93.28
71.22
ResNet-32
Seucld
None
150
91.28
66.58
Conv
Linear
100
92.62
68.42
Table 4: Full precision results on ResNet-20 for CIFARIO for different multiplicatton--fee similarities.
Similarity
Scony 92.97
Seuclid 93.00
Sadder 91.84
Sufo 82.05
Saynapse
Accuracy
73.08
204
convolutional weights is more stable than training from scratch as expected. However, accuracy is lower but the convergence is faster when we use homotopy training and the accuracy is improved. Pre-trained convolution weights are commonly available in the most of neural compression tasks, so initializing EuclidNets with pre-trained convolution is more natural and preferable.
205
207
208
EuclidNets are not only faster to train compared with other competitors, but also stand superior in terms of accuracy. AdderNet performs slightly
209
ResNet18 -Cfar10
210
211
worse but much slower to train. The accu-
racy is significantly lower for the synapse and the multiplication-free operator. Table we record t accuracy obtained in which Adder- Net results are borrowed from [e), that use 215
213
0.8
214
WP hw the
knowledge distillation to close the gap with the
toxing
217
full precision but still falls short compared with EuclidNet.
Suw
Randorr
219
Training a quantized Seuclid is very similar sim- ilar to convolution. This allows wider use of such networks for lower resource devices. Quan- tization of the Euclid model to 8bits keeps accu-
0.
Sucto Carv
Cany Homolopp
221
350
222
223
racy drop within the range one percent [43 similar to traditional convolution
Figure Evolution of testing accuracy during training of ResNet-20 on CIFARIO, initialized with random weights, or initialized from convo- lution pre-trained network. Initializing from pre- trained convolution network speeds up the conver- gence. EuclidNet is harder to train compared with convolution network when both initialized from random weights.
they are like convolution when run on lower bits. Table shows 8-bit quantization of EuclidNet where the accuracy drop remains negligible. Similar to tra- ditional convolution, EuclidNets CIFAR100
225
227
229
cxhibit a larger accuracy drop compared to CI-
FAR 10, probably due to the complexity of the classification problem.
231
232
6.2
ImageNet
233
Next, we consider EuclidNet classifier built on ImageNet, a more challenging task ImageNet [[ We train baseline with standard augmentations of random resized crop and horizontal flip and normalization. We consider ResNet-18 and ResNet-50 models. Hyper-parameters tuning follows
235
Section 6.
237
Table shows 1 and top-5 classification accuracies. The accuracy from while EuclidNet is trained from scratch is lower, showing the importance of homotopy training. We believe that the accuracy
239
drop with no homotopy is the difficulty of tuning training hyper-parameter for large dataset such as ImageNet. Even though typer-parameed that achieve equivalent accuracy from random initialization
7

---

exist, they are too difficult to find. It is much easier to use the existing hyperparameters of traditional convolution, and transfer the geometry through homotopy training.
242
Table 5: Full precision results on ImageNet. Best result for each model is in bold.
Model
Similarity
Initialization
Homotopy
Epochs
Top-1 Accuracy
Top-5 Accuracy
Scony
Random
None
90
69.56
89.09
Random
None
90
64.93
86.46
None
90
68.52
88.79
ResNet-18
Seucli
10
65.36
86.71
Conv
Linear
60
69.21
89.13
90
69.69
89.38
Scony
Random
None
90
75.49
92.51
Random
None
37.89
63.99
None
90
75.12
92.50
ResNet-50
Seuclid
70.66
90.10
Conv
Linear
60
74.93
92.52
90
75.64
92.86
243 7
Conclusion
Euclid networks obtained from typical neural models replacing multiplication in convolutional layers by the Euclidean similarity. They are designed to be implemented on a custom designed low precision chipset, with the idea that subtraction and squaring can be implemented using approximately half the logic gates, compared to multiplication.
245
247
W Whlle other efficient architectures can be difficult to train in low precision, EuclideNets are easily trained low precisions. EuclidNets can be initialized with weights trained on the correspondent ConvNet to save training time, may regard them fine tuning convolutiuonal networks for cheaper inference. The homotopy method further improves training in such scenarios and training using this method sometimes surpass regular convolution accuracy. Future work may focus on developing hardware that can realize the expected inference time losses and try similar experiments on down stream vision tasks like object detection and segmentation.
249
251
253
255
7.1
Limitations
While gate counts provide fundamental method for assessing the cost of chip, they are crude estimate, and the real costs (in terms of power usage, inference time, and memory) of a chipset and architecture combination
257
much more complex estimate. True final costs can require hardware simulator or implementation. At the same time, the gate count provides a first approximation to the cost, and the fact that we can train and match accuracy in eight bit precision is promising.
259
280
261
7.2
Societal Impact
262
Deep Neural Network inference is costly in terms of power usage. If we can design and implement efficient architectures, this will reduce the societal cost of running these models on edge devices.
263
References
285
[1] Arman Afrasiyabi, Diaa Badawi, Baris Nasir, Ozan Yildi, Fatios T. Yarman Vural, and Enis Çetin. Non-euclidean vector product for neural networks. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6862-6866. IEEE, 2018.
288
[2] C. E. Akbaş, A. Bozkurt, A. E. Çetin, R. Atala, and A. Üner. Multiplication-Fis free neural networks. In 2015 23nd Signal Processing and Communications Applications Conference (SIU), pages 2416 2418, 2015.
270
[3] Eugene L Allgower and Kurt Georg. Introduction numerical continuation methods, SIAM, 2003.

---

272
[4] Diaa Badawi, Ece Akhan, Ma' en Mallah, Ayşegül Üner, Rengül Atainn,tall,, and A Enis Çetin. Multipli- cation free neural network for cancer stem cell detection in h-and-e stained liver images. In Compressive Sensing VI: From Diverse Modalities to Big Data Analytics, volume 10211, page 1021 International Society for Optics and Photonics, 2017.
274
275
276
[5] Shumeet Baluja, David Marwood, Michele Covell, and Nick Johnston. No multiplication? no floating point? no problem! training networks for efficient inference. arXiv preprint (afe::880002444 2018.
277
278
[6]
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In P Proceedings of the 26th annual international conference machine learning, pages 41 2009.
279
280
[7] Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar Martin Wistuba, and Naigang Wang. A comprehensive survey hardware-aware neural architecture search. arXiv preprint arXiv:2101.0 09300....93..... 2021.
282
283
[8] Zhangjic Cao, Mingsheng Long, Jianmin Wang, and Philip Yu. Hashnet: Deep learinng to hash by continuation. In Proceedings of the IEEE international conference on computer vision, pages 5608-5617, 5617, 2017.
284
285
286
[9] Hanting Chen, Yunhe Wang, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, and Chang Xu. Addemnet: Do we really need multiplications in deep learning? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1468-1477, 2020.
287
[10] Qipin Chen and Wenrui Hao. An efficient homotopy training algorithm for neural networks. 2019.
[11] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint arxiv:/710.0000
281
2017.
[12] Yu Cheng, Duo Wang. Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35(1):126-136, 2018.
293
294
295
[13] Chow, L Udpa, and SS Udpa. Homotopy continuation methods for neural networks. In 1991., IEEE International Sympoisum on Circuits and Syxtems, pages 2483-2486. IEEE, 1991.
296
297
[14] Michele Covell, David Marwood, Shumeet Baluja, and Nick Johnston. Table-based neural units: Fully quantizing networks for multiply free inference. arXiv preprint 17X1v:1906.04798, 2019.
[15] Jennifer I. Davidson and Gerhard X Ritter. Thcory of morphological neural networks. In Digital Optical Computing II, volume 1215, pages 378-388, International Society for Optics and Photonics, 1990.
[16] J. Deng, W. Dong, R. Sacher, L..J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. CVPR09, 2009.
302
303
[17] Radu Dogaru and Leon O Chua. The comparative synapse: A multiplication free approach to neuro- fuzzy classifiers. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 46(1 I): 1166::366113111 1371, 1999.
304
305
306
[18] Farnoush Farhadi, Vahid Nia, and Andrea Lodi, Activation adaptation in neural networks. In Proceedings of the 9th International Conference Pattern Recognition Applications and Methods Volume I: ICPRAM,, pages 249-257. INSTICC, SciTePress, 2020.
307
308
[19] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1rXiv:1803.0338 00.0003335, 2018
310
311
[20] Caglar Gulcehre, Marcin Moczulski, Francesco Visin, and Yoshua Bengio. Mollifying networks. arXiv
312
preprint orXiv:66088 04980, 2016.
313
[21] Yunhui Guo. A survey on methods and theories of quantized neural networks. arXiv preprint arXiv:1808.00 04752, 2018.
314
315
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. corr abs/1512.03385 (2015), 2015.
318
317
[23] John L Hennessy and David A Patterson. Computer architecture: quantitative appreach. Elsevier, 2011. I.
318
[24] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 2015. 2015.

---

320
[25] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint urXiv: 70000444..1224 2017.
322
323
[26] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- La-iiiiv, and Yoshua Bengio, Binarized neural networks. In D. Lee. M. Sugiyama, U. Luxburg. I. Guyon, and R. Gamett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
325
328
[27] Forrest N landola. Song Han, Matthew W Moskewicz. Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters ands 0.5 mb model size. arXiv preprint arXiv:/602.07:66/ 2016.
328
329
[28]
Martin Kersner. Convolutional network without multiplication operation, Mar 2019.
330
[29]
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009,
331
[30] Elena Limonova, Daniil Alfonso. Dmitry Nikolaev, and Vladimir V Arlazarov.. Resnet-like architecture with low hardware requirements. arXiv preprint 2rXi::200 2020.
333
[31] Elena Limonova, Daniil Matveev, Dmitry Nikolaev, and Vladimir V Arlazarov. Bipolar morphological neural networks: convolution without multiplication. In Twelfih International Conference on Machine Vision (ICMV 2019), volume 1 1433, page 114333J. International Society for Opties and Photonics, 2020.
334
336
[32]
Maen Mallah. Multiplication free neural networks. PhD thesis, Bilkent University, 2018.
[33] Hossein Mobahi. Training recurrent neural networks by diffusion. arXiv preprint 1601001601..4114 2016.
338
[34] Ranjan Mondal, Sanchayan Santra, and Bhabatosh Chanda. Dense morphological network: An universal
339
function approximator, 2019.
340
[35] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 10.2.89, 2020.
341
[36] Hongyi Pan, Diaa Badawi, Xi Zhang, and Ahmet Enis Cetin. Additive neural network for forest fire detection. Signal, Image and Video Processing, pages 1-8, 2019.
343
[37] Harsh Nilesh Pathak and Randy Paffenroth. Parameter continuation methods for the optimization of deep neural networks. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1643. IEEE, 2019.
344
346
[38] Mohammad Samragh Razlighi, Mohsen Imani, Farinaz Koushanfar, and Tajana Rosing. Looknn: Neural network with no multiplication. In Design, Automation & Test in Europe Conference & Exhibition (DATE),
347
2017, pages 1775-1780. IEEE, 2017.
349
[39]
Russell Reed. Pruning algorithms-a survey. IEEE transactions on Neural Nenworks, 4(5):740-747, 1993.
[40] Gerhard X Ritter and Peter Sussner. An introduction to morphological neural networks. In Proceedings of 13th International Conference on Pattern Recognition, volume 4, pages 709-717. TEFE, 1996.
352
[41] Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905 / /1946, 2019.
354
[42] Carole-Jcan Wu, David Brooks, Kevin Chen, Douglas Chen, Sy Choudhury, Marat Dukhan, Kim Hazel- wood, Eldad Isaac, Yangqing Jia, Bill Jia, et al. Machine learning at facebook: Understanding inference at the edge. In 2019 IEEE International Symposium on High Performance Computer Architecture (HPPA),, pages 331-344. IEEE, 2019.
355
356
358
[43] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isacv, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation, 2020.
359
360
[44] Vixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing Xu, and Yunhe Wang. Kernel based progressive distillation for adder neural networks. arXiv preprint arXiv:2009.1044: 2020.
381
362
[45] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages (133-4141,200)
363
364
[46] Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspirer deep network. Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020.
388
367
388
10

---

370
[47] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848-6856, 2018.
372
Checklist
374
For all authors...
375
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
378
contributions and scope? [Yes]
377
(b) Did you describe the limitations of your work? [Yes], see subsection 7.1
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
379
Subsection 7.2
380
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
381
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
385
3. If you ran experiments..
386
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
387
388
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
389
(c) Did you report error bars (e.g., with respect to the random seed after rumning exper- iments multiple times)? [No]. It was too costly to train multiple times, we just ran once.
391
393
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs. internal cluster, or cloud provider)? [No]. but we gave standard training details in section 6
394
395
396
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
389
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
400
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
402
403
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
404
5. If you used crowdsoureing conducted research with human subjects...
406
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
407
408
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
410
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
11


Reviews:
[{"review_id": "nSaJ0gRdzXz", "reviewer": "Reviewer_yKQG", "summary": "The authors propose to replace the traditional convolution operation with squared differences in vision models. Compared to convolution, squared differences can be implemented with fewer logic gates, thus increasing energy efficiency and potentially improving the performance of inference.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a new category of vision model, namely EuclidNets, which uses squared difference as the computational kernel. The intuition behind Euclidean nets is clear, since any multiplicative operation can be expressed in terms of Euclidean operations only. The results of the empirical evaluation are also promising, as EuclidNets can achieve similar accuracy as ordinary convolutional neural networks. The idea is sound to me. I believe this paper clearly articulates a contribution to the literature. However, the paper should be improved in the following aspects:\n1) The organization and writing of the paper needs to be improved. For example, Table 1 is an important table containing the main results of EuclidNets and a comparison with the baseline and related work. However, the experimental setting is not presented, which makes it difficult for the audience to know whether the figures in the table are directly comparable. I suggest to move the paper to the experimental section with more detailed explanation.\n2) Theoretical gate count estimation. I appreciate the theoretical gate count estimation in the paper and appendix. However, in terms of the actual circuit implementation, the logical synthesis tool will optimize the circuit, which may change the results. I suggest to obtain the gate count using either commercial or open source logical tools. In addition to the number of gates, the authors can also report the circuit depth or delay of the critical path in convolution, addition, and squared difference circuits.\n3) Fair comparison with AdderNet. Both Table 1, 3, 4 contain results of AdderNet with and without knowledge distillation. However, the numbers reported in this paper do not match with numbers reported in AdderNet paper and github repo [1].  For example, the github repo of AdderNet claims 92.96% and 68.8% top-1 accuracy with knowledge distillation on ResNet-20 for CIFAR-10 and ResNet-18 for ImageNet, respectively. I think it should be more prudent to specify whether the results are obtained by reference to a paper, some open source implementation, or self-implementation. More importantly, the current comparison with AdderNet is not fair, because addition requires much fewer logic gates than squared differences. The authors should come up with a better way to compare the two and thus demonstrate the advantages of EuclidNets.\n\n[1] https://github.com/huawei-noah/AdderNet/blob/e69b13dff4f64f859ff1d32f6d2ec2ecae2a2c1b/README.md\n\nNitpicks: \n1) Title of Figure 4 ResNet 18 - Cifar-10\n3) Line 172: Fine-tuning method in (7) is inspired by continuation methods in partial differential equations\n2) Line 219: Training a quantized Seuclid is very similar similar to convolution\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "BDSqh4cKcvE", "reviewer": "Reviewer_TQtz", "summary": "==+== A. Paper summary ==+== \n\nThis paper presents EuclidNet, a method to replace multiplication in traditional ConvNet with l2 or Euclid distance. Results show that fine-tuning from a pretrained CNN to a EuclidNet can achieve similar or better results compared to traditional ConvNet and Addernet.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "==+== B. Strengths \n\n-- The idea of the paper is good and thought-provoking. Equation 4.1 indicates that EulidNet lies in the middle between quantized and full-precision models.\n\n-- Result shows the accuracy/fine-tuning cost is low.\n\n==+== C. Weakness \n\n--- The presentation can be improved a lot.\n\n--  The evaluation is not thorough.\n\n--  Training from scratch is unstable.\n\nQuestions:\n\nDo we need two squaring hardware to handle odd/even numbers? Figure 2 is confusing. And if we double the size of logic gates for squaring, it would show larger cost than multiplication. \n\nI can see one downside of EulidNet is that it cannot exploit sparsity like traditional ConvNet. In my opinion, your work is orthogonal to other works dealing with model sparsity; and you should compare your method with them. \n\nFigure 3 is confusing, are you searching the best $\\lambda$ for fine-tuning using linear grid search? If that is the case, wouldn’t this greatly increase your fine-tuning time?\n\nThe evaluation section is also confusing and a lot of details are missing. For example, I don’t know if Table 3 is for quantized models or full-precision models. If it is for a full-precision model, I suggest you add a quantization baseline to show your method is better than directly quantizing a ConvNet.\n\nRegarding the hardware cost, I suggest the author find an efficient hardware implementation of squaring. You can synthesize a piece of hardware (FP squaring v.s. FP multiplication) to see the simulated runtime cycles. As traditional metric (FLOPs) is not working in your scenario, it is also better to come up with a new metric (e.g., cycles) to evaluate your runtime cost against other baselines.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "4_lRI2kf2y2", "reviewer": "Reviewer_2baF", "summary": "This work proposes EuclidNet, which replaces the multiplication in DNN with squared differences with less logic gates.\nIt is proved to be as expressive as convolution networks and able to be trained in GPUs with at most 2x more time.\nThe experiments are conducted on ResNet@CIFAR-10 and ImageNet.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "> + Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nReplacing multiplication with squared differences is relatively new and the analysis on the expressivity of the EuclidNet in Section 4.1 and the discussion on the logic gate cost in Section 4.2 also help on this point.\n\n> + Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\n1. The comparison with AdderNet: It seems that EuclidNet cost more logic gates than AdderNet but is not always better than AdderNet in terms of accuracy. In Table 1, AdderNet with 8-bit quantization achieves higher accuracy than EuclidNet on ImageNet, i.e., does it mean EuclidNet will perform worse in bigger models@bigger datasets? There is no other comparison with AdderNet and its corresponding variants [1,2] in Section 6, it would be better to add them.\n2. Limited tasks: Currently all the tasks are conducted in Image Classification tasks, if the the transferability of the network to other tasks is verified to be valid, it will make this a more solid work.\n3. Confusing experiments settings: In line 196-197, \"We consider two different weight initialization for EuclidNets. First, we initialize randomly and second, we initialize from weights197 pre-trained on a convolutional network.\" However, in Table 3, all initialization function for EuclidNets are \"Conv\", so not sure if there is a real fair comparison with standard Conv under the random initialization setting. If all the performance of EuclidNets in Section 6 needs a pre-trained standard CNN, it would be better to at least add an ablation study to fine-tune standard CNN with the same training cost.\n\n> + Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nThe submission is clearly written and well organized. \n\nFor the reproducibility, the code seems to be not in the Appendix and code link the in submission seems to be invalid. It would be better if the authors can share their implementations in GPU to verify their claim about \"EuclidNet training is 2x n the worst case and their implementation is natural in deep learning frameworks such as PyTorch and Tensorflow\".\n\n> + Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\nBecause the comparison with AdderNet is not well illustrated and analyzed, I think the significance is slightly lower than the average of NeurIPS accepted papers.\n\n[1] Kernel Based Progressive Distillation for Adder Neural Networks. Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing XU, Yunhe Wang. NeurIPS, 2020. \n\n[2] ShiftAddNet: A Hardware-Inspired Deep Network. Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, Yingyan Lin. NeurIPS, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors clearly addressed the limitations of their work in Section 7.1", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "220-BGMIoJ", "reviewer": "Reviewer_vThL", "summary": "EuclidNets propose a new multiplication free method to conduct convolution in neural networks, which only involves subtraction and squaring, and achieves improved accuracy as compared with other baseline multiplication free methods while not sacrificing too much of the hardware efficiency, in terms of the number of logic gates used.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nOverall this paper is \n1. Clear and illustrative; provides the good reasoning and intuitive demonstration of why the proposed method is effective\n2. propose a new and interesting direction which still leaves much room for future work if more hardware flexibility is considered\n\nWeakness: \nHowever, there are a few concerns which would be great if get addressed: \n1. The accuracy improvements seems to be marginal as compared with the baselines; it would be good if the authors could conduct more trials of the experiments and provide the variance of the converged accuracy\n2. The evaluated network variety seems to be limited. Currently only ResNet series are considered. It would be more informative if more networks or even tasks are considered to show the methods general effectiveness.\n3. The hardware cost, in terms of the logic gates, seems still to be at the same magnitude against the multiplication network, through theatrical calculation. End2end (if possible) latency/energy/area consumption of the proposed method against the multiplication method would be good.\n4. More theoretical proof of the EuclidNets' superiority over the baselines such as AdderNet in terms of converged optima or rate would be better. Otherwise, the method seems a little too empirical.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It's addressed", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The discussion phase greatly helped clarify several concerns raised by reviewers around some experimental comparisons, particularly around the use of knowledge distillation in some of the experimental results reported from other papers. One sigficant outstanding issue that came up during the discussion phase is around the complexity of EuclidNets and whether gate counts are appropriate estimates of complexity or not. Specifically, one of the reviewers noted that the matrix multiplication unit in TPUs consumes only a small fraction (30%) of the area, thus reducing the impact that EuclidNets would have compared to that platform. Other reviewers noted that comparisons to sparse training methods (fp16, bp16) are necessary. Overall, the lack of measures of complexity other than gate counts makes assessing the contribution of the method particularly challenging, particularly given that hardware implementations of AdderNet, albeit recent, do exist.
Decision: Reject