Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a recurrent convolutional architecture (LocRNN) to improve extrapolation in visual tasks. The novelty of the architecture is presented as being long-range lateral connections between neurons from the same layer. The architecture is tested on two tasks: Mazes (route segmentation) and PathFinder (curve tracing). Both of these have multiple difficulty levels to test on. 

LocRNN architecture is based on an ODE-based model of lateral connections between cortical columns in the primate brain. The architecture uses a long range neural population and a short range neural population, each of which has gated updates and lateral connections defined by convolutional kernels within and between the neural populations. 

This is added to residual network architecture, and compared with GRU and ResNets themselves. LocRNN and GRU outperform ResNets (feedforward only) on both tasks. To show extrapolation, experiments are conducted by training on one difficulty level and testing on the rest - ConvGRU and LocRNN both do well in extrapolation, and ConvGRU (specialized to sequential processing) does the best. 

Finally, the paper claims that extrapolation ability (as shown by the existing recurrent architectures) trades off with task performance because performant algorithms lack stability in the hidden space. 
strength_and_weaknesses: **Strengths**
- Architecture is intuitive and appears to be a simple but effective model and discretization of the human neural "long lateral connections"
- Paper is well-scoped - it raises certain questions and answers them effectively with experiments that tie into each other
- Experiments are well-thought out and support claims
- Results are significant, particularly the extrapolation result that shows good performance on 
- Generality of approach, compared to more specialized architectures, is exciting

**Weaknesses**
- Experimental suites are somewhat limited, could benefit from more tasks
- Not enough results are presented in the main paper - we are told that LocRNN has better task performance than others, but there are no tables to highlight the statistics and Figure 2 doesn't clearly demonstrate this relative to GRU (or at least, I didn't notice) 
- The tradeoff section is unconvincing - it's a brief paragraph relying on a visualization, and even if it had more concrete evidence, the connection between the statistics presented and hidden state stability is unclear. Ultimately, we have three data points, not an empirically established or theoretically proven trend (as far as the paper goes), though connections could be established.
clarity,_quality,_novelty_and_reproducibility: **Clarity**
- Results section is claims-driven, which is good, but could overall use more signposting. It does help to have certain sentences bolded.
- Figures, particularly 2 and 3, are whitespace-heavy and lack clear highlights of important trends
- Some analysis isn't fully fleshed out. For example, the connection between Fig 3B (connection between validation accuracy as a function of recurrent iterations) and stability seems fair, but I would benefit from clearer explanation of how to interpret the graphical results and what specifically they mean for stability

**Quality**
- Main issue is need for more experimentation
- Quality of experiment execution seems high 

**Originality**
- This paper dos not compare to Nayebi et al. 2021 (ConvRNN), which also uses long connections (though not lateral). It is the most similar work I know of and it would help for the authors to address it
summary_of_the_review: I am recommending a weak acceptance because I do not see any major flaws in the paper and the general agent performance results support the main claims they are attached to. I am not recommending strong acceptance because important and very similar convolutional recurrent networks prior work has not been compared to, and the scope of the experiments and method is somewhat limited. Furthermore, the tradeoff is addressed briefly despite being a main contribution and a compelling claim, and the paper would benefit from doing more for that claim.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a new gated convolutional recurrent architecture consisting of separate neural populations with long-range and short-range interactions. This structure is vaguely motivated by neuroanatomical cortical observations that revealed that neurons in the primary visual cortex of mammals are strongly recurrently coupled and this recurrence tends to be subdivided in long-range excitatory inputs and inhibitory inputs from local interneurons. In particular, these two pathways were implemented through convolutional operations with different kernel sizes, which are then mixed and combined with gating operations.
This novel recurrent architecture is then empirically compared against feed-forward vision architectures (ResNets) and recurrent vision architectures (ConvGRU and a recurrent version of ResNet obtained by tying weights across layers) on two previously proposed path integration tasks: Mazes (that asks to output a segment connecting two points in a 2d maze) and PathFinder (a classification tasks consisting on determining whether two disks on a image are connected or not by a curved segment among other distracting segments).
The proposed architecture performs better on these tasks than the competitors at most difficulty levels, although it is interestingly revealed that a version of ConvGRU is better at extrapolating on a difficult level after being trained on a different difficulty level. Related to this, ConvGRU is also shown to be more robust to varying the number of recurrent steps. 
strength_and_weaknesses: The paper has some very interesting elements, starting from the biological and fundamental science motivation. However, it still needs some work in terms of providing mechanistic and algorithmic understanding on how the cortically motivated recurrent motif accomplishes what it accomplishes, and establishing the relevance of the specific path integration tasks in the context of the cortical substrate that the paper purportedly examines. More in detail, here are some of the strengths and weaknesses of the paper.

Strengths:
- The paper is trying to connect known fact of the anatomy and functionalities of primary visual cortex of the brain to the engineering of deep learning. In particular, the fact that the connectivity of the cortex is highly recurrent and it seems to be organized in terms of clearly defined short- and long-range connections. This sort of investigation seems very promising as it may help elucidate the algorithmic properties of neuro-physiological and anatomical structures.
- The paper investigate relevant metrics beyond accuracy, such as robustness as a function of recurrent steps, and generalization across task difficulty.

Weaknesses:
- The paper is in part motivated by the statement that feedforward networks are "strictly restricted" and "cannot dynamically change their computational graph". On the other hand, conditional modules like conditional batchnorm or context-dependent representations like those provided by transformers strongly contradict this claim. In addition, a recurrent architecture when unfolded through time can be thought of as a feedforward network with tied weight (as the paper also recognizes when examining the R-ResNet architecture). It is then not clear what exactly this claim regarding the limitations of feedforward architectures alludes to exactly, since it would also pertain to RNNs (being a special case of feedforward architecture). It would be beneficial to re-target this claim and re-contextualize in light of these two objections.
- The paper proposes a new recurrent module without however providing any quantitative analysis of its functioning. It would be interesting if the paper would also include an analysis of the stability of module examining the vanishing and exploding gradient problems. 
- The paper is highly focused on a very restricted and idiosyncratic type of task (path finding) which isn't  clearly motivated in terms of being of wide enough interest for the machine learning and deep learning communities at the moment. This could be mitigated by arguing that these tasks are at least a good fit for the cortical-type of architectures that are then proposed. In other words, what is the evidence that these tasks are paradigmatic of the type of tasks that the visual cortex and its architecture are solving? One would assume that the visual cortex is rather engaged in visual perceptual tasks, rather than planning and path finding. 
- The objection for the type of task that is being investigated could arguably also apply to the use of ReNets as baseline models, since they have been built to solve visual perceptual tasks, while, based on the fact that Dijkstra's algorithm can be used to solve shortest path problems, one would assume that algorithms for path finding have more to do with implementing dynamic programming.
- The paper demonstrate that the novel proposed RNN module displays some interesting properties in how it seem to be tackling the path finding tasks differently from other RNNs and feeforward architectures. It is however not clear from the paper what computational or algorithmic features distinguish the different ways of tackling this class of problems. Presumably, the differences in performance have to do with the different inductive biases that different architectures have for solving this problem in the specific way that it is formulated (as visual classification tasks).
clarity,_quality,_novelty_and_reproducibility: The paper would benefit from a more in-depth presentation and analysis of the LocRNN module, and how it relates to other RNN modules and it's computational properties (stability, vanishing and exploding gradient, etc).
summary_of_the_review: The paper starts with a interesting motivation, understanding the computational properties that the anatomy of cortex (high recurrence, separate local and long-range connection) provides to a RNN module. However, the paper doesn´t analyze the RNN module that is derived to incorporate these properties in a deep enough quantitative way. Moreover, the derived architecture is only benchmarked on a very restrictive set of tasks (path finding) that are arguably only a weak match with respect to the biological motivation (understanding cortex) and the conv-nets baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a recurrent neural network architecture suited for task extrapolation, in which the level of computation required to solve a problem can be dynamically altered depending on the task's difficulty. The architecture is biologically-motivated, and the authors demonstrate performance on the Mazes and Pathfinder tasks. Experiments reveal that the proposed method, LocRNN, is successfully able to perform task extrapolation and outperforms other baselines. The authors also find an intriguing tradeoff between performance on task extrapolation and stability.
strength_and_weaknesses: **Strengths**
The authors investigate a potential solution to the problem of task extrapolation, which is an important area, and moreover take inspiration from research on visual routines. The proposed LocRNN method is well explained and has a good biological motivation. In the experiments, a number of good baselines are compared. Analysis of the experiments is strong; in particular, the tradeoff between extrapolation and stability is a nice addition to this section.

**Weaknesses**
In my view, the main weakness with this paper is with the empirical results. First, the Mazes and Pathfinder tasks may be a little too simple. If the goal of this paper is to propose an RNN solution to general visual task learning, then it would be better to experiment with more complex tasks. Otherwise, the significance of the paper is unfortunately limited.

Also, it is unclear how much better LocRNN performs compared to the baselines. For example, in Figure 2 it appears that ResNet-18 performs similarly to LocRNN on Pathfinder. Also, as the authors point out, on Mazes, ConvGRU outperforms LocRNN on task extrapolation to large mazes. I may be missing a key result, but I unfortunately don't see where LocRNN significantly outperforms the baselines.

As a more minor point, I'm unsure in section 5.3 if the instability of LocRNN is what causes better performance. The authors propose a potential explanation for why this may be the case, but it would help to have an experiment justifying this explanation. For example, how would ConvGRU perform with ReLU activation?

Finally, it would be nice to include some ablation experiments on LocRNN. What aspect of LocRNN makes it effective compared to the baselines (e.g. is it the gating mechanism or the division into two neural populations)?
clarity,_quality,_novelty_and_reproducibility: **Quality**
The proposed method is not theoretically motivated, but this is acceptable if there strong empirical results, particularly since the method is biologically motivated. However, there are unfortunately some weaknesses with the empirical results as noted above. Nevertheless, the proposed direction is interesting, and the paper could be a strong contribution if the authors can convingly show that LocRNN strongly outperforms baselines on challenging benchmarks.

**Clarity**
The paper is clear for the most part. The bolded sentences and paragraph headings in sections 4 and 5 are helpful. However, there are a few minor areas where the clarity could be improved:
1. It may help to illustrate the architecture of LocRNN (particularly the receptive fields).
2. It may be better to define LN right after eqns 1 and 2.
3. In the "LocRNN shows strong extrapolation on the harder PathFinder challenge" paragraph, Fig 5.2 doesn't appear to exist.

**Originality**
The proposed method appears novel; to my knowledge, there are no similar methods applied to visual task extrapolation.
summary_of_the_review: Overall, the paper proposes an interesting, biologically-motivated solution to the problem of visual task extrapolation. Unfortunately, the empirical results are a little weak which limits the significance of the paper in its current form. If the experiments can be improved, the paper would be a much stronger contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A novel recurrent architecture is introduced with the express goal of learning models that can generalize to examples that are more/less difficult than the training data. On two benchmark datasets with test sets of several difficulties, this new architecture beats existing models on out of distribution performance.
strength_and_weaknesses: Strengths:
- The paper flows well.
- The experiments are thorough and illuminating.
- The writing is mostly clear.
- The architecture is meaningfully different from existing techniques, and shows a real benefit over those methods.

Weaknesses:
- I don’t understand the point about convergence and computational budget. Are there models that didn't train successfully? What was tried here? More specifically, in the footnote, it sounds like these models that don't train are paired down, lighter weight versions than were originally proposed -- do the full-strength models train? How is pairing down the model size (and computation budget) justified if these networks don't train?
- The results that aim to show that LocRNN is better on mazes is not so compelling. In Figures 2 and 3 ConvGRU looks superior on Mazes. The text and the captions seem to minimize this, so at best the difference is unclear to the reader. More information and clearer claims around this are needed. 
- Code is missing. I followed the link to the anonymized GitHub repository on October 17th and it was empty. Particularly, when it is stated that code is available, I take issue with this. I understand the criteria for submission do not require code, but I don't think there is enough information to reliably reproduce the results here *and* the authors state that code is available. This needs to be addressed.

Two minor issues (not affecting score): 
- There is a bad reference to a "Fig 5.2" in the second paragraph on page 8. 
- The second sentence of section 5.3 is unclear: “LocRNN is the best performing architecture that outperforms all
compared baselines in terms of raw performance, whereas ConvGRU and hConvGRU which are much inferior to LocRNN on the PathFinder challenge reach top extrapolation performance.”

clarity,_quality,_novelty_and_reproducibility: The writing is clear, except for a couple minor points above. The overall flow of the paper is sensible, and the the choice of datasets as well as models to compare with is good. The quality is high -- the experimental results support the main claims. As far as I know, the work is an original contribution building on other important works in the field and prompting even more research into recurrence and task extrapolation.
summary_of_the_review: I find the paper solid. There are a couple weaknesses and a couple minor points that can be addressed to improve this paper. I look forward to the authors' response. Should my concerns be adequately addressed, I'll be happy to raise my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a novel recurrent neural network model (LocRNN) that has biological motivation and adherence, such as long-range and short-range interactions. The model is tested on task extrapolation, in which the level of computation required to solve a problem can be dynamically altered depending on the task's difficulty. Specifically, the model is tested on the Mazes and Pathfinder benchmark tasks (both related to the contour integration and curve tracing capabilities found in visual brain areas) with several levels of difficulty, and somewhat outperforms other baselines. The experiments support the claims that the proposed model is able to perform task extrapolation (training on one difficulty level and testing on others).

-- STRENGTHS --

1) Technical novelty of proposed method, especially with fair amount of proper biological motivation

2) Generally robust evaluation and clear results, especially after revision by authors in response to initial reviews.


-- WEAKNESSES --

(A good number of discussion points about weaknesses on OpenReview were more about points needing clarification, and they were pretty technical or specific. Also, the authors managed to address a good number of reviewer concerns, and the reviewers upgraded their scores. Hence, here I focus on key big-picture weaknesses, which I thought were somehow not adequately covered by the reviews.)

Overall, the claimed contributions, while consistent with the results, are either weak contributions or are only weakly supported. One example is the title, which makes a very general statement about recurrence enabling task extrapolation. While the proposed model may indeed show this, the phrasing of the title suggests that all or any form of recurrence could enable task extrapolation, which certainly has not been shown. More specific examples are as follows.

1) Claimed contribution #1, that the paper shows the advantage of recurrent over feedforward processing on task extrapolation, is only weakly supported. The paper ultimately only tests a small number of models on two datasets. No doubt any single paper can only perform a limited number of experiments, but then surely the claim should then be scoped down accordingly. As a reviewer pointed out, the claim would be much more convincing if theoretical proofs or detailed mechanistic explanations were provided.

2) As stated in the paper, claimed contribution #2 is simply that of the LocRNN model, which purportedly introduces a new type of connection, which is hypothesized to enable learning of visual routines. This is a very weakly worded contribution. While one could counter-argue that this is simply a matter of writing, my view is that this is symptomatic of the paper overall, which is not clear on what exactly is the "story" it is trying to tell.

3) Claimed contribution #3 is that the paper *posits* "the potential tradeoff between task performance ... vs stability...". This is a very weak thing to claim as a contribution. To just posit (put forward) an idea that is potentially true, based simply on "an instance of this tradeoff we observe" is not really a proper contribution in my view, but rather an observation more suitable as an interesting point for the discussion section.


-- OVERALL --

Overall, this is not a bad paper by any means. The model is fairly novel and interesting, the results are positive, and I personally like the proper treatment given to the biologically-inspired approach. However, the reviewers are all lukewarm at best about this paper, with hardly any real excitement or enthusiasm even when describing the strengths. These are reflected in both their qualitative and quantitative feedback.

In making my recommendation (to reject), I took a big-picture view of the paper: what is the real significance and contribution of this work, as written by the authors? The authors chose to explicitly claim scientific-type contributions, and wrote the abstract, intro and discusson as such (in a more typically scientific style rather than AI/ML style), even though the body of the paper is a pretty typical dataset-model-evaluation-analysis one. I found the paper to be very weak in terms of articulating accurately and precisely how the community would benefit from this work, with vague or hedged phrases such as "our work encourages further study", "we show promising evidence", etc.

I find the work to be a tad too preliminary for acceptance in its current form. However, the core of the paper holds potential for some really interesting contributions (e.g. "we show that a recurrent network is able to flexibly adapt its computational budget during inference"), and I would encourage the authors to continue this line of work and keep improving it.


(Note: the authors are not wrong to point out that reviewer j482's score of 3 may not be commensurate with the content of their review, as well as their subsequent response to the rebuttal. I have factored this into my recommendation).


justification_for_why_not_higher_score: 1) Overly general claims that are not well-supported by results
2) Weak contributions, as least in terms of articulation
justification_for_why_not_lower_score: n/a
Decision: Reject