Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes EuclidNet, which replaces the multiplication in DNN with squared differences with less logic gates.
It is proved to be as expressive as convolution networks and able to be trained in GPUs with at most 2x more time.
The experiments are conducted on ResNet@CIFAR-10 and ImageNet.
main_review: > + Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?

Replacing multiplication with squared differences is relatively new and the analysis on the expressivity of the EuclidNet in Section 4.1 and the discussion on the logic gate cost in Section 4.2 also help on this point.

> + Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

1. The comparison with AdderNet: It seems that EuclidNet cost more logic gates than AdderNet but is not always better than AdderNet in terms of accuracy. In Table 1, AdderNet with 8-bit quantization achieves higher accuracy than EuclidNet on ImageNet, i.e., does it mean EuclidNet will perform worse in bigger models@bigger datasets? There is no other comparison with AdderNet and its corresponding variants [1,2] in Section 6, it would be better to add them.
2. Limited tasks: Currently all the tasks are conducted in Image Classification tasks, if the the transferability of the network to other tasks is verified to be valid, it will make this a more solid work.
3. Confusing experiments settings: In line 196-197, "We consider two different weight initialization for EuclidNets. First, we initialize randomly and second, we initialize from weights197 pre-trained on a convolutional network." However, in Table 3, all initialization function for EuclidNets are "Conv", so not sure if there is a real fair comparison with standard Conv under the random initialization setting. If all the performance of EuclidNets in Section 6 needs a pre-trained standard CNN, it would be better to at least add an ablation study to fine-tune standard CNN with the same training cost.

> + Clarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)

The submission is clearly written and well organized. 

For the reproducibility, the code seems to be not in the Appendix and code link the in submission seems to be invalid. It would be better if the authors can share their implementations in GPU to verify their claim about "EuclidNet training is 2x n the worst case and their implementation is natural in deep learning frameworks such as PyTorch and Tensorflow".

> + Significance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?

Because the comparison with AdderNet is not well illustrated and analyzed, I think the significance is slightly lower than the average of NeurIPS accepted papers.

[1] Kernel Based Progressive Distillation for Adder Neural Networks. Yixing Xu, Chang Xu, Xinghao Chen, Wei Zhang, Chunjing XU, Yunhe Wang. NeurIPS, 2020. 

[2] ShiftAddNet: A Hardware-Inspired Deep Network. Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, Yingyan Lin. NeurIPS, 2020.
limitations_and_societal_impact: The authors clearly addressed the limitations of their work in Section 7.1
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: EuclidNets propose a new multiplication free method to conduct convolution in neural networks, which only involves subtraction and squaring, and achieves improved accuracy as compared with other baseline multiplication free methods while not sacrificing too much of the hardware efficiency, in terms of the number of logic gates used.
main_review: Strength:
Overall this paper is 
1. Clear and illustrative; provides the good reasoning and intuitive demonstration of why the proposed method is effective
2. propose a new and interesting direction which still leaves much room for future work if more hardware flexibility is considered

Weakness: 
However, there are a few concerns which would be great if get addressed: 
1. The accuracy improvements seems to be marginal as compared with the baselines; it would be good if the authors could conduct more trials of the experiments and provide the variance of the converged accuracy
2. The evaluated network variety seems to be limited. Currently only ResNet series are considered. It would be more informative if more networks or even tasks are considered to show the methods general effectiveness.
3. The hardware cost, in terms of the logic gates, seems still to be at the same magnitude against the multiplication network, through theatrical calculation. End2end (if possible) latency/energy/area consumption of the proposed method against the multiplication method would be good.
4. More theoretical proof of the EuclidNets' superiority over the baselines such as AdderNet in terms of converged optima or rate would be better. Otherwise, the method seems a little too empirical.
limitations_and_societal_impact: It's addressed
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose to replace the traditional convolution operation with squared differences in vision models. Compared to convolution, squared differences can be implemented with fewer logic gates, thus increasing energy efficiency and potentially improving the performance of inference.
main_review: The paper presents a new category of vision model, namely EuclidNets, which uses squared difference as the computational kernel. The intuition behind Euclidean nets is clear, since any multiplicative operation can be expressed in terms of Euclidean operations only. The results of the empirical evaluation are also promising, as EuclidNets can achieve similar accuracy as ordinary convolutional neural networks. The idea is sound to me. I believe this paper clearly articulates a contribution to the literature. However, the paper should be improved in the following aspects:
1) The organization and writing of the paper needs to be improved. For example, Table 1 is an important table containing the main results of EuclidNets and a comparison with the baseline and related work. However, the experimental setting is not presented, which makes it difficult for the audience to know whether the figures in the table are directly comparable. I suggest to move the paper to the experimental section with more detailed explanation.
2) Theoretical gate count estimation. I appreciate the theoretical gate count estimation in the paper and appendix. However, in terms of the actual circuit implementation, the logical synthesis tool will optimize the circuit, which may change the results. I suggest to obtain the gate count using either commercial or open source logical tools. In addition to the number of gates, the authors can also report the circuit depth or delay of the critical path in convolution, addition, and squared difference circuits.
3) Fair comparison with AdderNet. Both Table 1, 3, 4 contain results of AdderNet with and without knowledge distillation. However, the numbers reported in this paper do not match with numbers reported in AdderNet paper and github repo [1].  For example, the github repo of AdderNet claims 92.96% and 68.8% top-1 accuracy with knowledge distillation on ResNet-20 for CIFAR-10 and ResNet-18 for ImageNet, respectively. I think it should be more prudent to specify whether the results are obtained by reference to a paper, some open source implementation, or self-implementation. More importantly, the current comparison with AdderNet is not fair, because addition requires much fewer logic gates than squared differences. The authors should come up with a better way to compare the two and thus demonstrate the advantages of EuclidNets.

[1] https://github.com/huawei-noah/AdderNet/blob/e69b13dff4f64f859ff1d32f6d2ec2ecae2a2c1b/README.md

Nitpicks: 
1) Title of Figure 4 ResNet 18 - Cifar-10
3) Line 172: Fine-tuning method in (7) is inspired by continuation methods in partial differential equations
2) Line 219: Training a quantized Seuclid is very similar similar to convolution

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: ==+== A. Paper summary ==+== 

This paper presents EuclidNet, a method to replace multiplication in traditional ConvNet with l2 or Euclid distance. Results show that fine-tuning from a pretrained CNN to a EuclidNet can achieve similar or better results compared to traditional ConvNet and Addernet.
main_review: ==+== B. Strengths 

-- The idea of the paper is good and thought-provoking. Equation 4.1 indicates that EulidNet lies in the middle between quantized and full-precision models.

-- Result shows the accuracy/fine-tuning cost is low.

==+== C. Weakness 

--- The presentation can be improved a lot.

--  The evaluation is not thorough.

--  Training from scratch is unstable.

Questions:

Do we need two squaring hardware to handle odd/even numbers? Figure 2 is confusing. And if we double the size of logic gates for squaring, it would show larger cost than multiplication. 

I can see one downside of EulidNet is that it cannot exploit sparsity like traditional ConvNet. In my opinion, your work is orthogonal to other works dealing with model sparsity; and you should compare your method with them. 

Figure 3 is confusing, are you searching the best $\lambda$ for fine-tuning using linear grid search? If that is the case, wouldn’t this greatly increase your fine-tuning time?

The evaluation section is also confusing and a lot of details are missing. For example, I don’t know if Table 3 is for quantized models or full-precision models. If it is for a full-precision model, I suggest you add a quantization baseline to show your method is better than directly quantizing a ConvNet.

Regarding the hardware cost, I suggest the author find an efficient hardware implementation of squaring. You can synthesize a piece of hardware (FP squaring v.s. FP multiplication) to see the simulated runtime cycles. As traditional metric (FLOPs) is not working in your scenario, it is also better to come up with a new metric (e.g., cycles) to evaluate your runtime cost against other baselines.
limitations_and_societal_impact: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes.
review_title: null

Meta-review: The discussion phase greatly helped clarify several concerns raised by reviewers around some experimental comparisons, particularly around the use of knowledge distillation in some of the experimental results reported from other papers. One sigficant outstanding issue that came up during the discussion phase is around the complexity of EuclidNets and whether gate counts are appropriate estimates of complexity or not. Specifically, one of the reviewers noted that the matrix multiplication unit in TPUs consumes only a small fraction (30%) of the area, thus reducing the impact that EuclidNets would have compared to that platform. Other reviewers noted that comparisons to sparse training methods (fp16, bp16) are necessary. Overall, the lack of measures of complexity other than gate counts makes assessing the contribution of the method particularly challenging, particularly given that hardware implementations of AdderNet, albeit recent, do exist.
Decision: Reject