Venue: NeurIPS 2020 Workshop SVRHM

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper seeks to understand the role of recurrence/feedback in neural system. While literature suggests that adding recurrence provides a significant boost to network performance for certain tasks, this paper aims to demonstrate that equivalent or better quality can also be achieved by a feed-forward network. This is a useful result that indicates that more analysis is necessary to fully characterize the necessity and utility of recurrence in neural networks.

The authors utilize the Pathfinder and cABC datasets for their experiment, which have been previously used to demonstrate the superiority of recurrence. They present results on variants of a ResNet18 model as the feed-forward network, finding that it is capable of beating the accuracy of prior state of art.

This work certainly causes the reader to question whether recurrence (with its training and representational complexity) is better than a carefully designed feed-forward architecture. The result is technically sound and supports the conclusion for the chosen architecture and datasets. Hence, I vote for acceptance.

However, a deeper dive into why residual networks perform unexpectedly good at these tasks would have helped the reader further. E.g.

* If we removed the residual connections, how much bigger would the network have to be for identical accuracy?
* Can recurrence help realize an even smaller ResNet?

Essentially, I am curious whether the improvements due to choosing a ResNet and due to using recurrence are orthogonal and additive. That would help shed some more light on the nature of these results. Regardless, it would be great to see this work published.
review_title: Great experiment and results, more discussion would be even better.

final_score: rating: 3: Clear rejection
review_content: Summary:
This is a re-examination of recent works that have described a limited ability of feedforward models (ResNets as an example) to solve tasks which are designed to evoke perceptual grouping routines. They perform experiments over a range of training-set sizes not reported in the papers that introduced these tasks, and find the number of examples needed for successfully training ResNets. This number is much greater than what the feedback networks reported in those papers require. These results are paired with replications from the original papers, along with novel results on the stability of training for ResNets.

Strengths:

The authors replicated the results in Kim et al., 2020, which is great to see! Especially the dissociation between the importance of horizontal and top-down connections on Pathfinder vs. cABC.

Weaknesses:

The authors' argue a point about the ability of ResNets to learn Pathfinder and cABC. This is not disputed in Kim et al., 2020. In fact, in Figure S8 they report results from different ResNet parameterizations on Pathfinder and cABC,  much like the authors do here. Kim and colleagues show that *you can* change the architecture to do better, but that change doesn't hold for both tasks.

But the most critical point is that Kim and colleagues argue that feedback connections are important for *sample efficiency*. This is reinforced by the findings in the current paper. The authors do not describe a ResNet that can trace paths as efficiently as feedback networks.

In order to demonstrate that a ResNet can trace paths as efficiently or better than feedback networks, the authors must present a ResNet which can achieve comparable or better performance on the same or fewer training examples. The authors fail to demonstrate this.


review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper demonstrates that benchmarks constructed to motivate the use of recurrent vision architectures can actually be solved by standard ResNets, with similar parameter- and sample-efficiency.

Strengths:

The submission provides compelling evidence both that Pathfinder and cABC tasks are too easy to be worth studying and that their use to motivate exotic architectures is misguided. This seems like a valuable contribution.

The work not only demonstrates that off-the-shelf ResNets can work, but also shows that they can be tuned to achieve similar parameter- and sample-efficiency.

Weaknesses:

The paper that introduced the Pathfinder dataset (ref. 1) evaluates ResNet-18 and claims that it achieves high accuracy at path lengths of 6 and 9 but fails at path length 14. I’m inclined to believe the results presented here, because I think it’s easier to make models fail than to make them work. Nonetheless, it would strengthen the work to know exactly how the training setup differs from those of refs. 1 and 2 of the submission that makes the ResNets here succeed whereas that previous work failed.

It’s unclear to me whether results regarding training time and stability in Figure 3, Table 2, and Figure 4 are meaningful. These findings are potentially sensitive to hyperparameters, but the authors do not appear tune the hyperparameters for different model or batch sizes. The authors seem to be surprised on L141 that tuning hyperparameters improves accuracy, but the value of hyperparameter tuning in deep learning, and particularly of tuning the learning rate, is well-established (see e.g. section 11.4 of the Goodfellow et al. deep learning book [1]). In addition, I wonder how much of the instability reported by the authors is related to the use of Adam with default values of the hyperparameters. Although refs 1 and 2 use Adam, most work involving ResNets uses SGD + momentum as in He et al. [2].

In Section 2.4.2, I'm not sure that the use of augmentation or pretraining is fair, since those methods effectively increase the training set size and could also improve sample efficiency of the baseline recurrent model. Nonetheless, the authors seem to achieve similar performance at the same number of training samples on Pathfinder simply by tuning the learning rate. On cABC, there is still a bit of a gap.

[1] Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1, p. 2). Cambridge: MIT press.
[2] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
review_title: Interesting and important reappraisal of the performance of ResNets on Pathfinder and cABC benchmarks

Decision: Accept (Poster)