Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I started reading the paper with high hopes. The abstract and the introduction were set up nicely and I was quite intrigued to see a theoretical analysis and practical implementation of a neural network using analogue hardware. However, as I read through the paper carefully multiple times, I realized that the expository opening description fails to live up to the standard it sets. 

To be more specific, I did not enjoy the elaborate description of noisy analogue conductances as the noise models analyzed in the paper are not tested on any of such devices. The noise model introduced is fairly simplistic and arguably, the real-world systems are much more complex compared to such simplistic assumptions. The authors could have presented the paper as an analysis of knowledge distillation in neural network training. Even if the paper were presented that way, I would have doubted its chance of acceptance due to the incrementality of the theoretical contribution.

All in all, I believe this is a very promising direction to invest, but the paper is not quite ready for ICLR.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The manuscript illustrates how a noisy neural network can reduce the learning capacity. To mitigate this loss, the authors propose a method that combines the method of "noise injection and "knowledge distillation". However, from a conceptual point of view,  their contribution (i.e. (10) in Section 5,) is unclear to me. Specifically,  the authors are not precise about how do they merge the aforementioned previous ideas and come up with the new loss function (10). 

Minor comment: Please correct (7).



review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The authors of the manuscript study how inherent noise in the analog neural networks affects its accuracy. This is a very important topic as neural network based inference becomes uobiqutous and is required to run with very low power consumption and latency.

The manuscipt considers a system where the values of the neural network weights and biases are experiencing i.i.d Gaussian noise, which is a pretty good assumptions. However, in heavy use the system may warm up, and then there could be an effect that is correlated accross different weights. The noise model used would not be able to ensure proper inference in these conditions. I would like to see a discussion on the effect of correlations in the injected noise.

The mutual information is considered and evaluated for a the "noisy" and "clean" versions and the result is according to expectations. The some degree, I do not see this part very valuable, as it does not bring any particular insights on the analog neural network operation. Rather I woudl like to see how, the analog performance under noise scales as the neural has  more layers.  Also, the noise behavior of an analog RNN woudl be very interesting.

The authors have detected that even, if the neural network trainded without noise is not robust when the weigths fluctuate, the trained network is a good starting point for transfer learning. To some degree I do not find this to be a very inventive step as transfer learning has shown to able to cross much larger training data set alterations.

Good solid work, but lacking non-obvious results, and I do not see manuscript adressing the the harder challenges. However, the quantified results may have a notable practical importance.



review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: * Summary *
The article on "Noisy Machines" addresses the issue of implementing deep neural network inference on a noisy hardware computing substrate, e.g. analog accelerators. This is an important topic because analog devices allow fast and energy efficient inference, which is crucial for inference at the edge. Because of their analog nature such devices suffer from noisy computations, and in this article the case of noisy weights is studied. 

The main contributions of this article are the following:
- an analysis of the performance loss in noisy networks by means of information theory and empirical results
- the idea of combining noise injection during training with knowledge distillation
- experimental evidence for a LeNet5 on MNIST, CIFAR10, and ResNet-50 on ImageNet

It has been shown in the literature that noise injection during training is an effective way to increase the noise robustness of neural networks. Relevant literature in this domain is cited. The novelty of the approach is to combine noise injection with distillation, by using the noise-free network as a teacher for the noisy network, which is initialized with the weights of the teacher. This is a novel variant of distillation and sounds like a simple to implement trick with beneficial results for increasing noise resiliency of networks. It is also proposed and shown that the method works for quantized networks.

The experimental results show that the combination of distillation and noise injection outperforms pure noise injection on all networks, as well as noisy inference without retraining. The effect is even more pronounced for quantized networks.

* Evaluation *
Overall I like this paper and think it is suitable to accept for ICLR, because it addresses an important practical problem of implementing deep networks on efficient hardware. The paper is well written and simple to understand and should be easy to implement (it would really help here providing code for the examples though). To the best of my knowledge I have not seen precisely this combination of noise injection and distillation, although there is a lot of literature about each individual approach. I appreciate that the authors made an effort to not just show empirical results but also motivate their findings by theory, although the argumentation stays a bit superficial.

What I am mainly missing are two points:
1. The assumed noise model of i.i.D. Gaussian weights is the simplest possible, and might deviate quite a bit in actual analog hardware. I would have liked to see a noise model that is derived from actual hardware observations, or maybe even a prototype implementation in hardware, such as was done e.g. in Binas et al. 2016. At the very least I would suggest to test the model on other noise models, including temporally changing noise levels, which could be a realistic scenario due to temperature fluctuations or other events.

2. The experimental results focus on MNIST, CIFAR10, and later briefly on ImageNet. While the results are quite convincing on MNIST and CIFAR, these are easier datasets with usually well separable classes, so the effect of noisy inference might not be as pronounced, as in datasets with more confusion even in the clean case. In the case of ImageNet (Table 1) it looks like the difference to pure noise injection is not as big as it was in the CIFAR case, but here also only lower noise levels were tested. I would recommend testing also the same noise range as for CIFAR to understand whether distillation always shows the desired benefits, or if this is a diminishing effect for larger networks. Overall it would help to understand how the effect scales with network depth, e.g. by comparing the information loss for different ResNet depths.

I'm giving weak accept and would change to accept if there could be clarification on how the approach scales to different network architectures and noise models closer to actual hardware. I also recommend publishing some example code for this approach.
review_title: Official Blind Review #1

Meta-review: This paper argues that NNs deployed to hardware needs to robust to additive noise and introduces two methods to achieve this.

The reviewers liked aspects of the paper and the paper is borderline. However, all in all sufficient reservations were raised to put the paper below the threshold. The criticism was constructive and can be used in an updated version submitted to next conference.

Rejection is recommended.
Decision: Reject