Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: Taking inspiration from the theory of predictive coding in neuroscience, the authors implement CNNs with layerwise predictive activity, as a form of unsupervised training. Experiments show that this approach is helpful in case of image corruption.
main_review: The paper is generally well written and easy to follow.  The illustrations are effective. The section dedicated to the description with code snapshots of the developed library is unnecessary in the paper and could be relegated to supplementary material or link documentation. I would use the space available to provide more references to the state of the art and describe the proposed approach in more detail.

The idea of predictive coding applied to neural network layers is not new. The authors should expand the related works section to be more inclusive of more recent work. 
limitations_and_societal_impact: The novelty of the work is very limited. Since the model is not new, they only show that the application brings advantages in the case of corrupted images with perturbations. 

The experimental part is also limited. The authors only evaluate one type of image noise. It would have been interesting to see if this theory could contribute to improving robustness against adversarial attacks.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a new algorithm to implement a popular neuroscience theory, i.e. predictive coding, in deep neural networks (DNN). The feedforward connection weights are pretrained for object recognition and the feedback connection weights are trained for image reconstruction, and then predictive coding introduces recurrent computations with the trained DNNs. The authors use two image recognition DNNs (VGG16 and EfficientNetB0) in experiments. They found that the recurrent computation brings improvement in object recognition and image/feature reconstruction over time given noisy inputs. This property of predictive coding improves DNN robustness to noisy examples and adversarial attacks. The authors also provide a python package to allow community to explore the predictive coding algorithm.
main_review: Predictive coding is an important theory in neuroscience and recently some researchers have explored several implementations with deep neural network to help understand the theory. How neurons recurrently process information is still unknown in predictive coding theory. This paper proposed a new implementation (though have some similarity to previous works) and did extensive experiments and found desirable properties of recurrent computation over time. This is very interesting and valuable to both neuroscience and deep learning communities. The writing is clear and easy to understand.

The feedforward connection weights in DNN are trained with forward propagation for object recognition (pretrained) and the feedback connect weights are trained with reconstruction of lower layer features. No recurrent computation is involved in training the connection weights, which may not be the case in the brain. It would be good if the author can discuss this in the perspectives of both neuroscience and machine learning.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper implemented the predictive coding dynamics into deep neural nets, and observed the increase in robustness against various corruptions. The authors also provided a pytorch-based package for easy implementation of the proposed dynamics into deep neural nets. 
main_review: The predictive coding principle has gained increasing attention in the machine learning community. Despite not being the first to incorporate the predictive coding dynamics into neural networks, this paper did a good job in formulating the predictive coding principle in a clean way and they were able to test out the ideas on large scale datasets like Imagenet. The paper was well written and provided enough details to reproduce their results. In general, I enjoyed reading this paper. 
However, I do have some comments and questions regarding the proposed mechanism as well:
1) In Eq. 2, the feedforward representations were updated to reduce the reconstruction error in the lower level (the alpha term). However, if we believe that the feedforward representations for noisy images are far away from that of the clean images and will cause wrong classification, why reducing the distance between the feedback representation and this "wrong" forward representation helps robustness? I understand that experimentally we see some positive results, but it would be good to comment on this point. 
Also, it may be good to take a pretrained robust feedfoward network (for example, network trained with data augmentation techniques or adversarial training), and see whether the proposed mechanism can still improve the robustness.
2) In the paper, the authors only reported relative improvements, but I am curious to see the absolute value of robustness accuracy. Since the authors take pretrained networks, I am wondering if similar robustness improvements can also be achieved by training the forward model differently (for example, using data augmentation or adversarial training). This is related to the last point I mentioned in 1). It would be good to show whether feedback can further improve robustly trained feedforward models.
3) Since the authors introduced more weights in the model (the feedback path), could the robustness improvement be a result of using larger model? It would be good to take a deeper or wider feedforward model to match the number of parameters in the feedback model and see if the improvement is still there. 
4) In the pseudo code, it looks like the authors loop through all PCoder modules first, and then move on to the next time step. How would this compared with performing several update steps first until one PCoder is convergent and then move on to the PCoder in the next level?
5) For the adversarial robustness experiment, I was wondering how the authors perform the attack? Are they performing end-to-end attack considering the predictive coding dynamics? If not, this should be evaluated. In addition, when performing end-to-end attack, adversarial attack may suffer from gradient obfuscation problem, so it would also be good to add black-box attack evaluation.
6) How sensitive is the method's performance with respective to the coefficient \alpha in Eq. 2? In Table 2 and Table 3 in appendix, \alpha is fixed to be 0.01. But this term is an very important term for the predictive coding principle. Also the authors scaled the gradient based on feature size, so it seems that this is a sensitive hyper-parameter. 

limitations_and_societal_impact: The authors have provided sufficient discussion on this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors draw inspiration from neuroscience and propose augmenting feedforward neural networks with recurrent dynamics to improve their adversarial and corruption robustness. They provide a PyTorch package that can augment any convolutional neural network with recurrence. Their empirical tests show that recurrence mostly helps with robustness.
main_review: *Review summary:*

The proposed method is well explained and the provided recurrence software package could be a valuable contribution to the community. The paper could be improved by reporting more interpretable metrics and addressing inconsistencies in the results. Additionally, the literature review needs to be improved and the novelty made more explicit. I do not believe the paper is good enough for publication in its current form, but am ready to revise my score if my concerns are adequately addressed.

*Strengths:*

The paper is well written in terms of grammar and fluency. The introductory paragraphs provide good motivation for exploring recurrence in DNNs, and the model description is clear and concise. The concluding and broader impact statements are also well argued. The breadth of experiments (denoising, corruption robustness, and adversarial robustness) is helpful for providing a holistic understanding for how recurrence influences resilience to perturbations. While I didn’t look at the code itself, I believe that the provided codebase could be helpful for important future work in this space. I also appreciate the detailed comparisons to [22] and [24] in the Appendix.

*Weaknesses:*

This is a fairly long list that is not in order of importance (sorry). However, I hope many of these issues can be easily resolved by the authors.

a) I recommend additional background discussion on related prior work. There is a wealth of research attempting to use predictive coding ideas to improve DNN robustness. If we only focus on recurrence then the scope broadens further {2, 3, 8}. Although note that {8} heavily draws on the concept of analysis-by-synthesis, which is also a core idea in predictive coding. A more related example would be sparse coding, which is a predictive coding model that has been used for denoising for decades {6, 7}. Recently, several papers have also demonstrated adversarial robustness with sparse coding {1, 4, 5}. I have provided some examples here, but I would encourage the authors to look at citations of and within them to inform a proper discussion on the context of their work such that readers can properly assess the novelty of their ideas. An additional (minor) issue is that the existing references should also be updated. Many papers cited as arxiv preprints are now published in peer-reviewed venues.

b) It would be easier to compare the networks (VGG vs EfficientNet) if the authors used shared vertical axes in adjacent plots. This recommendation applies to almost every subfigure. The study’s goal is clearly not to compare VGG against EfficientNet, but understanding the relative differences between baselines can give better context for the change in performance when recurrence is added.

c) It is not clear to me what motivated the stopping criteria for the recurrence in any given experiment. The authors appear to arbitrarily stop the model at 4, 8, 10, 15, and 20 time steps. Why is this? I would like the authors to extend the time steps as long as necessary (given computational constraints) to give a complete picture of how recurrence influences the representations. Presumably all of the curves will eventually converge to a steady state -- when is that? If not, why? In a few instances (e.g. figure 4 for eps=0.003 and figure 3(b) for PEfficentNet), the performance does a reversal, which should at least be discussed.

d) When interpreting the gaussian noise experiment results (figure 2) the authors state that “Both networks demonstrate significant accuracy improvement across timesteps under noisy conditions, while maintaining a performance close to the feedforward level for clean images.” I do not believe this statement is supported by the data. For one, the use of the word “significant” implies that they did some sort of significance test, which they did not as far as I can tell. Perhaps more importantly is that for certain settings (e.g. the blue and orange curves in figure 2(a)) the recurrence appears to hurt performance. Due to the axis rescaling (see point b above), it is hard to see whether these differences matter. Finally, they should note what data that accuracy was computed on -- I assume one of the ImageNet test sets?

e) I do not understand the metric used in figure 2(b). I believe a normalized MSE distance of 1.0 indicates that the reconstruction quality with recurrence matches that without. The authors should be explicit about this. This metric is not as helpful to understand as reporting the raw MSE values (this is a theme across my suggestions). I find it interesting that there is not a consistent correlation or trend when comparing MSE against noise level. In fact, the PVGG net and PEfficientNet appear to have opposite trends, where the noise free case has the highest score for PVGG and almost lowest for PEfficientNet. There is a similar inconsistency (and similar metric confusion) in figure 2(d), where there is not a monotonic relationship between layer and distance, as I would expect. I would appreciate it if the authors would comment on this. 

f) I think it would be helpful if the authors used the same images for the two networks in figure2(C) so that readers can understand the relative difference in the baselines used.

g) In terms of the corruption experiments, given that the clean accuracy has changed I think it makes more sense to use the relative CE score from [8] than the regular CE. Also, the authors do not say if they are using the same AlexNet baseline as was used in [8]. At timestep 0 they always report CE of 1.0, which leads me to guess that they are using the standard VGG or EfficientNet score as the denominator. At a minimum the authors should be explicit about how they are computing the metric instead of deferring to the citation (unless they are indeed computing the score exactly as was done in [8]). Ideally they should use AlexNet as their denominator so that the reader can properly compare their performance with other works in the field. I appreciate that the authors intended to focus on the added value of feedback, instead of achieving SOTA on some benchmark. However, it is still helpful to understand the added value if we have a good relative context.

h) The authors alternate between showing individual CE scores without hyperparameter tuning in figure 3.(a) and mCE with tuning in figure 3.(b). But in the appendix figure 6 we can see that there are tuned CE scores that perform worse with recurrence (e.g. zoom & fog for VGG). To clarify the results I recommend that the authors provide more information in the main paper. Specifically, they should acknowledge negative CE results from the tuned/untuned models in the main paper. And if the goal is to compare the results “with tuning” against “without tuning”, then they should use the same metric in both scenarios. 

i) The robustness metrics used are also not standard in that field. The appendix figures 8, 9, and 10 are much more in line with what is typically reported. I don’t see the added benefit of the presentation in figure 4 when we can easily see the “baseline corrected” score by comparing time step 0 with those that are > 0 (this argument applies to many of the above points as well). Again, even though the authors wish to stress a relative comparison it is important that they report their results in such a way that it can be viewed in the context of other work. As a point of comparison, the results reported in {9} also used FoolBox and are more aligned to typical setup and formatting.

j) I’m not sure if the concerns raised in {4, 8} about backpropagating through recurrence are relevant here, but it would be helpful if the authors would explicitly say that the white-box attacks did not suffer from gradient obfuscation issues (see {9}). Even better would be if they additionally used gradient-free attacks as in {8}.

{1} Sulam et al., Adversarial Robustness of Supervised Sparse Coding, https://proceedings.neurips.cc/paper/2020/hash/170f6aa36530c364b77ddf83a84e7351-Abstract.html

{2} Krotov et al., Dense associative memory is robust to adversarial inputs, https://direct.mit.edu/neco/article-abstract/30/12/3151/8426/Dense-Associative-Memory-Is-Robust-to-Adversarial?redirectedFrom=fulltext

{3} Frosst et al., DARCC: Detecting Adversaries by Reconstruction from Class Conditional Capsules, ​​https://arxiv.org/abs/1811.06969

{4} Paiton et al., Selectivity and robustness of sparse coding networks, https://jov.arvojournals.org/article.aspx?articleid=2772000

{5} Kim et al., Modeling Biological Immunity to Adversarial Examples, https://openaccess.thecvf.com/content_CVPR_2020/html/Kim_Modeling_Biological_Immunity_to_Adversarial_Examples_CVPR_2020_paper.html

{6} Lu et al., Sparse coding for image denoising using spike and slab prior, https://www.sciencedirect.com/science/article/abs/pii/S0925231212007643?via%3Dihub

{7} Oja et al., Image Feature Extraction and Denoising by Sparse Coding, https://link.springer.com/article/10.1007/s100440050021

{8} Schott et al., Towards the first adversarially robust neural network model on MNIST, https://openreview.net/forum?id=S1EHOsC9tX

{9} Athalye et al. Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples http://proceedings.mlr.press/v80/athalye18a.html
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper introduces a framework for incorporating recurrent feedback connections based on predictive coding principles into feed-forward networks. Overall, the reviewers expressed excitement for the integration of neuroscience with ML and thought that the evaluations were convincing. However, there was initially high spread in reviewer scores due to concern over the novelty of the model and approach. After the detailed author responses and a thorough discussion, the consensus moved towards acceptance and two reviewers increased their scores.  
Decision: Accept (Poster)