Cortically motivated recurrence enables task extrapolation
Authors: Vijay Veerabadran
Yuan Tang
Ritik Raina
Virginia R. de Sa
Venue: Submitted to ICLR 2023 (2023)
Keywords: cognitive science
recurrent neural networks
task extrapolation
out of distribution generalization
visual routines
path integration

Abstract: Feedforward deep neural networks have become the standard class of models in the field of computer vision. Yet, they possess a striking difference relative to their biological counterparts which predominantly perform “recurrent” computations. Why do biological neurons evolve to employ recurrence pervasively? In this paper, we show that a recurrent network is able to flexibly adapt its computational budget during inference and generalize within-task across difficulties. Simultaneously in this study, we contribute a recurrent module we call LocRNN that is designed based on a prior computational model of local recurrent intracortical connections in primates to support such dynamic task extrapolation. LocRNN learns highly accurate solutions to the challenging visual reasoning problems of Mazes and PathFinder that we use here. More importantly, it is able to flexibly use less or more recurrent iterations during inference to zero-shot generalize to less- and more difficult instantiations of each task without requiring extra training data, a potential functional advantage of recurrence that biological visual systems capitalize on. Feedforward networks on the other hand with their fixed computational graphs only partially exhibit this trend, potentially owing to image-level similarities across difficulties. We also posit an intriguing tradeoff between recurrent networks’ representational capacity and their stability in the recurrent state space. Our work encourages further study of the role of recurrence in deep learning models – especially from the context of out-of-distribution generalization & task extrapolation – and their properties of task performance and stability.

Under review as a conference paper at ICLR 2023
CORTICALLY MOTIVATED RECURRENCE TASK EXTRAPOLATION
ENABLES
Anonymous authors Paper under double-blind review
ABSTRACT
Feedforward deep neural networks have become the standard class of models the field of computer vision. Yet, they possess striking difference relative to their biological counterparts which predominantly perform "recurrent" computa- tions. Why do biological neurons evolve employ recurrence pervasively? In this paper, we show that recurrent network is able to flexibly adapt its com- putational budget during inference and generalize within-task across difficulties. Simultancously in this study, we contribute a recurrent module we call LocRNN that is designed based on prior computational model of local recurrent intracorti- cal connections in primates to support such dynamic task extrapolation. LocRNN learns highly accurate solutions the challenging visual reasoning problems of Mazes and PathFinder that we use here. More importantly, it is able to flexibly use less more recurrent iterations during inference zero-shot generalize less- and more difficult instantiations of each task without requiring extra training data, a potential functional advantage of recurrence that biological visual systems capitalize on. Feedforward networks on the other hand with their fixed compu- tational graphs only partially exhibit this trend, potentially owing to image-level similarities across difficulties. We also posit an intriguing tradeoff between recur- rent networks' representational capacity and their stability in the recurrent state space. Our work encourages further study of the role of recurrence in deep learn- ing models especially from the context of aut-of-distributin generalization task extrapolation and their properties of task performance and stability.
INTRODUCTION
Deep learning based models for computer vision have recently matched and even surpassed human- level performance on various semantic tasks (Dosovitskiy al., (200 Vaswani et al. [022]] He et al.] 2021 Liu et al. 2022). While the gap between human and machine task performance has been di-
minishing with more successful deep learning architectures, differences their architectures and in
critical behaviors such adversarial vulnerability (Athalye al. 2018), texture bias (Geirhos al. 2018). lack of robustness to perceptual distortions (Hendrycks Dietterich] 2019 Geirhos et al, 2019). etc. have increased significantlly We interested in such stark architectural difference between artificial and biological vision that we believe underlies the above-mentionel critical behav- ioral differences, i.e., recurrent neural processing. While biological neurons predominantly process input stimuli with recurrence, existing high-performing deep learning architectures are largely feed- forward in nature. In this work, we argue for further incorporation of recurrent processing future deep learning architectures we compare matched recurrent and feedforward networks to show how the former are capable of extrapolating representations learned on task unseen difficulty levels without extra training data; feedforward networks are strictly restricted in this front as they cannot dynamically change their computational graph.
Ullman 1984) introduced a particularly important research direction in visual cognition that is of fundamental importance to understanding the human ability to extrapolate learned representations within-task across difficulties. Ullman hypothesized that all visual tasks we perform are supported by combinations of a small set of key elemental operations that are applied in a sequential man- ner (analogous to recurrent processing) like an instruction set. Instances of varying difficulty levels of a task can be solved by dynamically piecing together shorter or longer sequences of operations corresponding to that task. This approach of decomposing tasks into sequence of elemental opera-

---

Under review as a conference paper at ICLR 2023
tions (aka visual routines) avoids the intractable need for vast amounts of training data representing every instantiation of a task and is hypothesized to support our human visual system's 's systematic generalization ability. Examples of such elemental operations that compose visual routines include incremental contour grouping, curve tracing, etc. and a review of these operations along with physi- ological evidence can be found in Roelfsema al. (2000). Can we develop artificial neural networks that also learn visual routines for any task when constrained to deliberately use specialized recurrent operations?
In this work we show promising evidence for employing recurrent architectures to learn such general solutions visual tasks and exhibit task extrapolation to various difficulty levels. We note that such extrapolation is one kind of out-of out-of-distribution generalization that standard feedforward deep learning models struggle to perform. For this study, we perform experiments using two challenging synthetic visual tasks, namely Mazes and PathFinder (discussed in Sec. 3p. We make the following contributions as part of this work:
1. We show the advantage of recurrent processing over feedforward processing on task extrap- olation. We show evidence for strong visual task extrapolation using specialized recurrent- convolutional architectures including our proposed recurrent architecture, LocRNN on chal- lenging visual reasoning problems.
2. We contribute LocRNN, biologically inspired recurrent convolutional architecture that is de- veloped based on Li (1998, a prior computational ncuroscience model of recurrent processing in primate area V1. LocRNN introduces connection type that is missing from commonly used deep neural networks
long-range lateral connections that connect neurons within the same layer in cortex [Bosking et al. (1997). We hypothesize such lateral recurrent connections enable the learning and sequencing of elemental visual routines.
3. Comparing task performance alongside extrapolation performance of various recurrent archi- tectures, we posit the potential tradeoff between task performance - the ability of recurrent networks learn sophisticated iterative functions of their input excel on downstream tasks vs stability in the state space - local smoothness of the trajectory of recurrent states through time. We show em- pirical evidence for an instance of this tradeoff we observe and identify an important open problem which must be solved to establish high-performing and stable recurrent architectures for vision.
We combine cognitive insights from Ullman routines (Ullman 1984) with model of cortical re- currence (Li et al., 2006) from computational neuroscience to improve -tate-of-the-art recurrent neural network based machine learning architectures. This unique synergy results in demonstrating the superior ability of recurrent networks for task extrapolation by flexibly adapting their test-time computational budget, feat not possible for feedforward architectures. Our work encourages future work to further explore the role of recurrence in the design of deep learning architectures that behave like humans and generalize out-of distiiibbbt
2 RELATED WORK
As mentioned our work is highly relevant to the visual routines literature introduced by Ullman (1984) and further reviewed elaborately in RRelfssmma et al. (2000). The core idea of visual routines that make it relevant to our studied question of task extrapolation is the flexible sequencing of ele- mental operations resulting in dynamic computational graph. This idea has been studied by prior research on using recurrent neural networks to solve sequential problems. On a related note there have been several attempts to learn how much recurrent computation to use for given input sample on a given task (Graves) 2016 Saxton et al. 2019).
The most relevant to our work is (Schwarzschild et al. 2021) where the authors evaluate the appli- cation of recurrent neural networks to generalize from easier to harder problems. Our work extends and differs from their intriguing study of recurrence in task extrapolation in the following ways: 1) While their work explores sequential task extrapolation in general with abstract problems such as Prefix Sum and solving Chess Puzzles, our work extends it particularly focus extrapolation in visual task learning. Hence their maze segmentation task is of relevance to us and we use the same for evaluation (while also re- re-implementing models used in their study). In addition, present eval- uation on the Pathfinder challenge (Linsley et al. (2018). a relatively significantly more challenging and large-scale visual reasoning task, the design of which dates back to Policoeur al. (1986).
2

---

Under review as a conference paper at ICLR 2023
2) Schwarzschild et al. (2021) implement only a simple and weak form of recurrence realized by weight-tying, i.c., they only evaluate ResNets with weight sharing across residual blocks. addi- tion to such recurrent ResNets,
present analyses with highly sophisticated recurrent architectures specialized for recurrent image processing. 3) We introduce LocRNN, a high performing recurrent architecture hased on prior computational models of cortical recurrence that is inductively biased to learn sequential visual routines while trained on suitable downstream tasks.
On the design of recurrent architectures, our work is loosely relevant [igen et al. (2013), Pin- heiro & Collobert (2014) and Liao & Poggio (016) which discuss the role of weight sharing in feedforward networks produce recurrent processing. We are interested. however, in designing new specialized recurrent architectures that play role both in human and machine vision. [Linsley et al. 2018) developed one such recurrent architecture called hCConGRU that we also evaluate for comparison. Relative to hConvGRU (Linsley et al., 2018), LocRNN is a relatively computationaly simpler yet more perh-perforiing model (as highlighted by matching comparisons in Sec. D..
3 DATASETS
For evaluating the ability of various models in exhibiting task extrapolation, curate two challeng- ing visual tasks, Mazes and PathFinder, with instances at parametric difficulty levels. Both tasks involve the visual routines of marking and curve tracing (Ullman] 1984). These datasets are in- spired by prior visual psychophysics research where such tasks were used abundantly to estimate the cognitive and neural underpinnings of sequential visual processing, such incremental grouping. structure and preference of lateral connections, etc. (Jolicoeur & Ingleton| [991 Ullman 1984 Li
al. 2006 Roelfsema 2006). In the following subsections we describe the specilics of our tasks.
3.1 MAZES CHALLEENGE ROUTE SEGMENTATION
Task description Human beings are adept at solving mazes, a task that requires application of a similar serial grouping operation like PathFinder in order to discover connectivity from a starting point to the final destination of the maze amidst blocking walls. For evaluating model performance on solving mazes of varying difficulty, developed by |Schwarzschild et al. (202T). They implemented this Mazes challenge as a binary segmentation problem where models take N N images of square-shaped mazes as input with three channels (RGB) with the start position, end position, permissible regions and impermissible regions marked in the image. The output produced by models is a binary segmentation of the route discovered by the model from the start to the end position.
use the publicly available version of the Mazes challenge
Difficulty levels: The Mazes challenge has been designed at 3 difficulty levels, each difficulty level is parameterized by the size of the square grid that the maze is designed into. The grid size corre- sponding to
mazes 9x 11x11/ 13x 13 (see example images in Fig. in Appendix). This dataset consists of 50,000 training images and 10,000 testing images with the following spatial resolution: Small mazes - 24 x 24 pixels, Medium mazes - 28 x 28 pixels and Large mazes- - 32 x 32.
Evaluation criteria:
Although Mazes a segmentation challenge and hence, one could potentially consider partially correct routes during evaluation (for example with average of per-pixel accuracy). However this is less strict than giving each image a single binary score reflecting if all pixels are labeled correctly. Evaluation criteria for mazes is hence the total % of test-set mazes completely accurately solved at a given difficulty level.
3.2 PATHFINDER CHALLENGE CURVE TRACING
Task description While Mazes is an interesting challenge examined in Schwarzschild et al. (2021) and Schwarzschild et al. 2022), we consider it much smaller scale (e.g. MNIST/ / CIFAR- 10-scale) than the PathFinder challenge. In the PathFinder task introduced by Linsley et al. (2018). models are trained to identify whether two circular disks in an input stimulus form the two ends of a locally connected path made up of small "segments". Each image consists of two main connected paths Po and P1 made up of locally co-cireular segments Each image also contains two circular disks each of which is placed at one of the 4 end points of Po and P,. Images that contain disk on both
3

---

Under review as a conference paper at ICLR 2023
ends of the same path are classified positive, and those containing disk endpoints of different paths are classified as negative.
Difficulty levels: Pathfinder is designed at 3 difficulty levels parameterized by the length (number of segments) of the paths Po and P1 mentioned above. The easiest version of pathlinder uses paths that are 9 segments long (PathFinder-9), while the medium and hard versions contain paths that 14 (PathFinder- 14) and 18 (PathFinder-18) segments long respectively (see example images in Fig. i Appendix). This dataset consists of a total of 800,000 RGB images at each difficulty level, each with a spatial resolution of 150 x 150 pixels. There is an equal number of positive and negative instances at each difficulty level. We use 700,000 images for training and 100,000 images as the test set at each difficulty level. For evaluating zero-shot task extrapolation on PathFinder in Sec. [.2 we used a random sample of 20,000 images from PathFinder-9 and PathFinder-I1 18 test sets respectively.
Evaluation criteria Since this is classification challenge, we use accuracy, i.e. % correct on test- images as the evaluation metric to rank model performance on PathFinder. Model architectures will receive three-channel input image and process it via a stack of standard convolution/recurrent convolution layers followed by a classification readout which outputs two scores corresponding to the two classes. Since this is a binary classification challenge with balanced classes, chance performance is 50% when models are making random predictions.
4 MODEL ARCHITECTURES AND TRAINING
4.1 FORMULATION OF LOCRNN
We note that prior work has explored the development of recurrent architectures tailored for vision. Here, we introduce a similar but highly expressive (as demonstrated by Fig. [ recurrent architec- ture designed ground-up based on a computational model of iterative contour processing in primate vision from Li 1998). This model is an ODE-based computational model of interactions between cortical columns in primate area VI mediated by "lateral connections" local a cortical area. Anal- ogous to the cortical area VI is a set of activation maps in our deep networks, say. output by the first convolution layer, comprised of neurons firing in response to a variety of low-level stimulus features such as oriented bars and gradients. Distant neurons in these activation maps do not interact as interactions are restricted to the small receptive field sizes of convolutions. LocRNN symbolizes an RNN enabling long-range local interactions iterative interactions between distant neurons that are local to a set of activation maps in a given layer of a deep network.
By discretizing the continuous-form ODE dynamics of processing units from Li 1998), we arrived at an interpretable and powerful set of dynamics for "LocRNN". The effective receptive field of LocRNN output neurons increases linearly with recurrent timesteps.
LocRNN's hidden state is composed of two neural populations, Le and S, at timestep respec- tively. L, receives longer-range lateral connections, i.c., presynaptic neurons connecting to L, cover a wider range of image space than those connecting to St. These two populations are analogous to the x (excitatory) and y (inhibitory) neurons in Li (1998). However, our implementation tain this strict excilatory/nhibioory profile by restricting their recurrent weights to be strictly posi- tive/negative did not converge in stable manner. On the other hand, retention of two populations of neurons performed better than replacing them with single uniform population. Initially, both Lo and So populations are set tensor of zeros with the same shape as the input X. I and S update gates G and G (same shape as L and S) are computed as functions of the input and current hidden states L,- and S using 1x1 convolutions U...
G = (LL((Uxx * X) + LL((Un * LL-1))
(1)
G = =LLNUU Ex X) + LN(( St-1)
(2)
Each of the types of lateral connections namely 1) L + 2) L S,4) L neurons are modeled by convolution kernels Wn. Wa. W and Wts respectively. W11 and W modeling long-range lateral interactions are of shape d x d x hy x wi where d is the dimensionality of the hidden state and h; and wy represent the kernel spatial size. To be precise, each neuron at location (i.j. j. m) in L (spatial location i.j and channel m) receives lateral connections from a window of neurons at locations (Ir, y, n), x [i hr/ + hr/2). y E [j w1/2. j wr/2), n E [0, k) in L
4

---

Under review as a conference paper at ICLR 2023
and S mediated by Wi and Wt respectively. Similarly, Was Wal are of shape hg and wa such that (h,, wy) > (he: wg).
L 7( [ixx Ix X Wn Lt-1 + Wt St-1)
(3)
S, 7( (Wex ex X Wst L W ss SS-1)
(4)
Once the long-ran range lateral influences are computed and stored in L, and St, these are mixed with the previous hidden states using the gates computed in Eq. [ These hidden states are then passed on to subsequent recurrent iterations where even longer-range interactions occur (as time increases).
Lt <(LN(G) @ La + (1- G.-) @ Lt-1))
(5)
S. = K ( o + (1 ) St-1)
(6)
In the above equations, LN() stands for Layer Normalization Ba ct al. 2016), and the nonlinearitics y and are both set to ReLU.
stands
4.2 MODEL FAMILIES EVALUATED ON TASK EXTRAPOLATION
As we have noted in Related Work, prior work has examined the relationship between recurrent networks. standard ResNets, and ResNets with weight sharing. Owing to commonalities between these classes of architectures, they are most relevant to studying task extrapolation; we design ResNets, Recurrent ResNets which refer to as R-ResNets, and recurrent networks of varying effective depths. We describe details of our specific implementation (Anonymized code available at hwwp:///hoo.ommaaeeaaaaaaaaataaa
of these architectures in this section.
Specialized recurrent networks: We evaluated the task extrapolation ability of the following specialized recurrent architectures. We evaluated the LocRNN model described in detail in Sec. 4.1 In addition, we also evaluated the hConvGRU (Linsley et al.) 2018). ConvRNN (Nayebi et al.) 2022) and CORNet-S (Kubilius et al.) 2018) all of which are relevant convolutional recurrent neural networks. The Gated Recurrent Unit is quite relevant to both these architectures and utilizes a similar gating strategy as LocRNN and hConyGRU. We evalute CRRVGRU as well in the subsequent section on PathFinder, Mazes and on task extrapolation.
Residual Networks (ResNets): ResNets are not only high performing feature extractors for various downstream visual processing tasks, they also attain high scores on the Brain-Score leaderboard (Schrimpf Kubilius 2018) that measures representational similarity to biological vision. this work we implement three ResNet architectues on the Mazes challenge and two ResNet architectures on the PathFinder challenge. On Mazes, we used 1) ResNet-12, 12, 2) ResNet-20 and 3) ResNet- 36 architectures while on PathFinder, we used 1) ResNet-10 and 2) ResNet-18 architectures; implemented all these ResNet architectures to use the same width (# of output channels) at each residual block. This width parameter set to 64 for ResNets evaluated Mazes and to 54 for ResNets evaluated on PathFinder.
Recurrent Residual Networks (R-ResNets): We implemented R-ResNets by enabling weight- sharing across the stack of residual blocks and note that this is a much lighter weaker notion of recurrence compared specialized recurrent architectures. This is reflected in the raw performance difference and extrapolation performance difference in comparison to specialized recurrent archi- tectures. R-ResNets are designed with an input convolution layer and output readout layers that together form 4 layers of processing. Each recurrent block that extracts features for PathFinder and Mazes is also 4 layers deep. Hence, the total number of layers in R-ResNets is multiple of To enable fair comparison among recurrent networks used the same number of recurrent iterations. kernel size and roughly matched the parameter count for all three recurrent networks on both tasks separately.
5
RESULTS
5.1 LOCRNN EXCELS BOTH AT MAZES AND PATHFINDER
Before testing for length generalization of our selected models (discussed in Sec. 4 ated models' raw ability to integrate long-range spatial dependencies on Pathfinder and Maze chal- lenges. We systematically tested this ability by varying the difficulty of Pathfinder across 3 levels
we first evalu-
5

---

Under review as a conference paper at ICLR 2023
parameterized by path length (9, 14, and 18), and the difficulty of Mazes across levels of difficulty parameterized by the maze's ' grid size (9x9, 1 11, and 13 13 respectively). For each of the datasets (3 Pathfinder and 3 Maze respectively),
used overoaallg training and testing splits of the full datasets to train models and evaluate their performance during test-time.
Evaluation on Mazes (small, medium, large)
LocRNN
hConvōRU
ConvGRU
CORNet-S
ConvANN
ResNet-12
ResNet-20
ResNet-36
Rodie 23
aucuracy (%)
Validation
Evaluation on PathFinder (-9, -14, -18)
LocRNN
hConvGRU
ConvGRU
CORNet-S
ConvRNN
ResNet-10
ResNet-18
R- 18
100
300
100
100
130
100
300
Validation
92
90
90
50
90
accuracy
8
50
30
80
90
R
70
70
70
70
70
34
14
14
Figure 1: (Top) Model performance comparison on Mazes. All recurrent networks outperform the ResNet-based feedforward baselines. We observe trend that increasing depth improves the perfor- mance of ResNets on Mazes. (Bottom) Model performance comparison on PathFinder. Differences between the recurrent networks becomes clearer on the more difficult task of PathFinder.
We created non-overlapping training and testing sets of Pathfinder and Maze datasets that were used to train the models and evaluate their test accuracy. All models used the same training and testing image sets.
Results from Fig. (Top) shows that all recurrent networks outperform their BesNet--based feedfor- ward baselines on Mazes (including the deepest ResNet-36 network) despite using many fewer free parameters. We replicate the trend observed by /Schwarzschild et al. (2022): ResNets and R-ResNets (their recurrent counterparts) at matched effective depths are quite similar in their performance on the 3 Maze challenges. However we show that both ResNets and R-ResNets are considerably different from all specialized recurrent architectures (LocRNN, hConyGRU, ConvGRU) which are all more accurate and sample efficient compared to the ResNets. We would like emphasize that this difference is sigmificant and argues for improved effectiveness of specialized recurrent ar- chitectures representing and reasoning with spatial context via integration of long-range spatial dependencies. On Mazes, we find that by narrow margin of 1.34 %, ConyGRU outperforms LocRNN and the best performing model on Mazes.
On Finder challenge, LocRNN is the best performing architecture across all difficulty lev- els of the challenge including the most difficult PathFinder-188 18. Results on PathFinder show key dif- ferences between the recurrent architectures. Simple weight-tying based R-ResNet-18 18 is unable to learn even the easiest version of the challenge. On the other hand there exist significant differences between the sophisticated recurrent architectures LocRNN, ConvGRU, hConvGRU, CORNet-S (Kubilius et al.) 2018) and ConvRNN (Nayebi et al.] 2018). hConyGRU learns the easy version of PathFinder (PathFinder-9) but fails converge PathFinder-14 14 and PathFinder-18 | Over- all in this evaluation the LocRNN model shines with top performance on all 3 difficulty levels of both challenges either closely matching or outperforming feedforward ResNet- baselines with or- ders of magnitude more parameters. LocRNN's performance is followed by ResNet-18 18 and then CORNet-S which achieves better than chance performance (yet much inferior to LocRNN) on PathFinder-14 14 and PathFinder-188 18. Importantly, ConvGRU and hConvGRU, the only models to match/outperform LocRNN on Mazes are significantly worse than LocRNN on Pithernder with ConvGRU being at chance on all difficulty levels and hConvGRU being at chance on all but the casiest difficulty level.
'Training hConvGRU using code made available by Linsley et al. 2018 (using larger recurrent convolu- tional kernels and an order more GFLOPS than what we used here) in an architecture significantly heavier in GPU costs was able to solve PathFinder- 14. However this training setting is too expensive and and we report results on modest hyperparameter settings lightweight architectures used by all models.

---

Under review as a conference paper at ICLR 2023
5.2 RECURRENT NETWORKS EXHIBIT TASK EXTRAPOLATION
While models are trained on one difficulty level of a task, we define task extrapolation as the ability to dynamically reduce (and increase) computation to generalize zero-shot to easier (and harder) dif- ficulty levels of the task without extra training data; this is impossible to achieve with feedforward networks aren't It intrinsically suited to dynamically vary their computational budget during inference (for e.g. recurrent segmentation at variable computational budgets (McIntosh et al. 2016)). However, as soon as feedforward networks use weight-lying, we refer to them as recurrent networks unrolling their temporal computation via depth and can hence equip variable computational bud- get during inference. We test the ability of various recurrent architectures from Sec. 5.1 in task extrapolation by altering their recurrent timesteps during inference.
For instance, the accuracy of I LocRNN tested on Small Mazes while it was trained on Medium Mazes measures the extrapolation of LocRNN an easier Maze task (Medium - Small). Although it may seem trivial for biological intelligence to solve easier problems after learning a hard problem. it need not necessarily be the case for neural networks which have no exposure to instances from an easier challenge. Hence we evaluate generalization to both easier and harder problems.
We chose those recurrent architectures with high performance on the medium Mazes and PathFinder challenges for their respective task extrapolation experiments. Remaining models with poor or chance performance on these tasks are not fit for the evaluation of extrapolation as their within- distribution performance is itself lacking. We reused the following architectures based on the above criteria for evaluating task extrapolation - a) LocRNN, b) ConvGRU, and c) R- R-ResNet-36 trained on Medium Mazes and only LocRNN trained on PathFinder-14 (since other recurrent network was able to converge to accurate solutions on PathFinder-14), Per-timestep batch normalization layers in hConvGRU prevent us from performing inference with this model at arbitrary number of itera- tions aside from what was used during training. hence we could not evaluate its task extrapolation performance other than with the same number of steps it trained on (this is discussed below). We evaluated the zero-shot generalization performance of the above architectures on easier and harder variants of Mazes and PathFinder with & < T (for easier) and t > T (for harder) recurrent steps where T denotes the number of recurrent steps used during training.
Exalusting task octrapolation on Masos
CORNER
Pollower lidel
M
Concal)
Figure 2: (Top) Task extrapolation performance of models to easier (Small) maze by decreasing number of recurrent steps as we move horizontally in each plot from the right (timesteps used during training) to the left. (Bottom) Task extrapolation performance of models to harder (Large) maze by increasing number of recurrent steps as we move horizontally in each plot from the left (timesteps used during training) to the right. Dotted blue line shows the zero-shot generalization performance of the feedforward ResNet-36 model to small and large mazes. ResNet-36 is significantly worse than all recurrent models in generalizing to the harder Maze. Number of recurrent timesteps used during training is circled on each plot's x-axis.
All RNNs adapt recurrence to show extrapolation Each plot in Fig. 2.A shows the extrapolation performance of different models on Mazes. From these plots we see that recurrent networks in general are able to gain more performance on easier and harder mazes simply by using fewer and more recurrent steps than what they were trained with (highlighted with circles the x-axis).
7

---

Under review as a conference paper at ICLR 2023
Specialized recurrent architectures show stronger extrapolation: As we've noted before, R- ResNet uses a weak form of recurrence by using weight-tying across layers of a feedforward net- work, this difference reflects the model's inferior generalization performance relative to special- ized recurrent architectures we evaluated. Fig. 2 shows that ConyGRU designed particularly for sequential processing is overall the best-performing model (95.6%) on extrapolating large mazes. On small mazes, all recurrent architectures have matching extrapolation performance and are still better than R-ResNet. The architectures CORNet-S and hConvGRU use learnable BatchNorm pa- rameters at each timestep. preventing us from running them for more than the number of timesteps they were trained with. However, we use the following workaround: we perform soft restoration of learned weights; up until the number of timesteps used during training the per-timestep learned BatchNorm parameters are restored from the trained model. During inference on greater number of timesteps than what was used during training, we initialize these learnable BatchNorm parameters at random.
LocRNN shows strong extrapolation on the harder PathFinder challenge We show extrapolation to easier (PathFinder-9) and harder (PathFinder-18) PathFinder problems in Fig. B A. LocRNN shows strong extrapolation performance to the easier and harder PathFinder challenge images. While generalizing to PathFinder-9, LocRNN grows from chance performance at 12 steps recurrence (used during training) to achieving > 90% accuracy on reduced computational budget (8 steps). On the harder PathFinder- 18 problem, while it is at chance when run for 12 recurrent steps (used during training). increasing recurrence to 14 steps allows the model reach a nontrivial accuracy of 62% matching the much more parameter heavy feedforward ResNet's generalization performance.
5.3 ABLATION STUDY OF LOCRNN
Interneurons are crucial for good performance: We found strong evidence suggesting that in- terneurons are crucial to LocRNN's high performance. First, we retained L neurons and removed S neurons (which are interneurons in LocRNN mentioned above) in LocRNN (-S neurons in 3). This ablation converges to a stable solution in only I out of 3 random seeds (also with a 38% drop in sample efficiency). Next, match the number of parameters, of L. neurons to 64 channels (-S neurons + 2x L neurons in Fig BB). While there was a slight im- provement (2 out of seeds converged). the sample efficiency of the models that converged very low compared to the full model (25% drop in sample efficiency). In an interesting third abla- tion experiment, we retained both L and neurons and sent both neuron populations upward for classification. This explicitly tests whether the interneuron property helps improve expressivity. Again, only one of the 3 seeds converged an accurate solution and with very poor sample effi ciency, highlighting that the property of S neurons being interneurons is of utmost importance for LocRNN's 's high performance.
increased the width
still
ReLU is important for good performance: Having long range of linear (non-saturating) oper- ating mode in the activation function is critical to LocRNN's performance. Replacing ReLUU with tanh completely breaks LocRNN's performance. However, replacing ReLU with clipped ReLU (clipped at 6) did not cause any drop in performance. Also, in agreement with this, when we re- place tanh in ConvGRU with ReLU, achieve a significant gain in ConyGRU's PathFinder- 14 performance (+10.4%)
Gating and LayerNorm contribute to improving performance and sample efficiency: We find that LayerNorm affects performance the least among all components. Even in the absence of LN, LocRNN achieves 80% performance on PathFinder. However, L.N seems to improve sample efficiency significantly, similar to gating; the absence of which hurts both performance and sample efficiency.
5.4 A TRADEOFF BETWEEN TASK-OPTIMIZATION AND EXTRAPOLATION
Our observation comparing the performance of recurrent architectures in Fig. a Fig. A high- light an interesting conundrum. While LocRNN is the only recurrent architecture to achieve near- perfect accuracy on both Mazes and PathFinder, ConvGRU (which is much inferior to LocRNN on the PathFinder challenge) produces the best extrapolation performance an intriguing tradeoff between task performance optimization and stability in hidden state space. ConvGRU is inductively biased by design to be more stable due the tanh nonlinearity used in
Mazes. This suggests
8

---

Under review as a conference paper at ICLR 2023
(A)
Evalusting extrapolation
(B)
Ablation of LooRNN
(Hard eu Easy extrapoliaton))
(Rae on Hard acrapelatinn
Valtttton caneet
300
(iena under vall curvel
offieeny
LockdN
FrndFooncal Rackia
Vatobotion perhmmaace
80
15
Profforrod Rocles
LecEAN
70
70
Anda undde
1
04.
12
14
Full modier
Di peeErIN
Gletngg
LayerHom
Pudl montel
dith pecrrnet
Gasing
LayerHom
-Reui
Extropoittoon with fower recuerved (tarations mutier prodken
Entrpoelation with more baratiooe hurder prodium
Figure 3: (A) Task extrapolation performance of LocRNN from Pathfinder- 14 to Pathfinder-9 and PathFinder-188 18. ConvGRU and R-ResNet not shown here as they were unable to converge on PathFinder within the provided training budget. The number of recurrent steps used during training highlighted with a circle on the x-axis of each plot. (B) Validation accuracy and sample efficiency of various ablations of LocRNN.
its output layer (a saturating nonlinearity that has bounded outputs) whereas LocRNN uses a ReLU nonlinearity with unbounded outputs. This could be the underlying reason why LocRNN is unstable yet high performing model. In fact, replacing the tanh in ConvGRU with ReLU enhances the model's performance from chance to 90.05% and 60% on PathFinder-9 and PathFinder- 14 respec- tively.
The high task performance of LocRNN potentially comes at the cost of reduced stability in the hidden state space with non-smooth dynamics around the number of training recurrent steps. This is evident from our visualization of the validation accuracy of LocRNN and ConvGRU as a function of the number of recurrent steps in Fig. (in Appendix). Our future work explores techniques to improve the stability of LocRNN in order to unlock class of recurrent neural networks that can be optimized on any given task to produce high performance well as learn stable state space representations as a function of recurrent iterations.
DISCUSSION
Extrapolating to new easier and harder instantiations of a task is a kind of out-of-distributie gen- eralization where agents learn to dynamically adapt their learned algorithms solve new easier / harder versions of a problem. [Ulman (9944) visual routines provide a conceptual framework to accomplish the development and deployment of such flexible algorithms based on the recurrent (se- quential) application of a set of elemental operations within human cognition. We combine this conceptual framework with Li] (1998''s computational model of recurrence in primate early vision and a state of the art recurrent neural architecture to demonstrate that neural networks can also ex- hibit task extrapolation which is a trademark characteristic of human cognition. As part of this work introduced high-performing recurrent architecture called LocRNN that achieves excellent curacy and sample efficiency on two challenging visual spatial reasoning benchmarks, Mazes and PathFinder (best performing model on this task). An ablation study of LocRNN reveals the individ- ual contributions of each of its components and provides mechanistic understanding of LocRNN's high performance. LocRNN and other recurrent architectures exhibited task extrapolation on both the above tasks in contrast to feedforward networks that are restricted by their fixed computational budget during inference which cannot accomplish such flexible generalization. We believe that the abundance of recurrent processing in biological neurons underlies human ability to exhibit task ex- trapolation. Upon comparing the performance of LocRNN with other recurrent architectures, we develop the key insight of a potential trade-off between stability and task performance in recurrent architectures. Our future work will examine the development of new class of stable recurrent architectures that, while using unbounded activation functions such ReLU (that contribute high expressivity). are also jointly designed/optimizee strong task extrapolation behavior. Our work argues that recurrent operations are fundamental to bi- ological and artificial vision and inspires future work studying the utility of recurrent architectures in the performance and disttriiiitiittti generalizability of deep learning models.
to produce stable recurrent activations which show

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pp. 284-293 PMLR, 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: 1XXiv:607.004553 2016.
William H Bosking, Ying Zhang. Brett Schofield, and David Fitzpatrick. Orientation selectivity and the arrangement of horizontal connections in tree shrew striate cortex. Journal of neuroscience, 1766)2112-2127, 1997.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint Ntt::2000.119220 2020.
David Eigen, Jason Rolfe, Rob Fergus, and Yann LeCun. Understanding deep architectures using recursive convolutional network. arXiv preprint arXiv: 13X.::3112.1477 2013.
R Geirhos, CR Medina Temme, Rauber, HH Schüit,, M Bethge, and FA Wichmann. Generalisation in humans and deep neural networks. In Thirty-second Annual Conference on Neural Information Processing Systems 2018 (NeurIPS 2018), pp. 7549-7561. Curran, 2019.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and
Wieland Brendel. Imagenct-trained cnns biased towards texture; increasing shape bias im- proves accuracy and robustness. arXiv preprint arXiv:1811.12331, 2018
Alex Graves. Adaptive computation time for recurrent neural networks.
arXiv preprint
arXiv:1603.08983 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2 11.0677, 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
Pierre Jolicoeur and Margaret Ingleton. Size invariance curve tracing. Memory & Cognition, 19 (1):21-36, 1991.
Pierre Jolicoeur, Shimon Ullman, and Marilynn Mackay. Curve tracing: A possible basic operation in the perception of spatial relations. Memory & Cognition, 14(2):129-140. 1986.
Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James Di- Carlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, pp. 408385, 2018.
Wu Li, Valentin Piëch, and Charles D Gilbert. Contour saliency in primary visual cortex. Neuron, 50(6):551-966 2006.
Zhaoping Li. A neural model of contour integration in the primary visual cortex. Neural computa- tion, 10(4):903-945 9400, 1998.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.036400 2016.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in Neural Information Processing Systems, pp. 152-164, 2018.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11976-1 1986, 2022.
10

---

Under review as a conference paper at ICLR 2023
Lane McIntosh, Niru Maheswaranathan, Aran Nayebi, Surya Ganguli, and Stephen Baccus. Deep learning models of the retinal response to natural scenes. In Advances in neural information processing systems, pp. 1369-1377, 2016.
Aran Nayebi, Daniel Bear, Jonas Kubilius, Kohitij Kar, Surya Ganguli, David Sussillo, James J. DiCarlo, and Daniel L. K. Yamins. Task-Driven Convolutional Recurrent Models of the Visual System. Advances in Neural Information Processing Systems, 1.B8-eccmmber:529-5555 2018. URLpttp://azxtt orq/ abs 00053
Aran Nayebi, Javier Sagastuy-Brena, Daniel M Bear, Kohitij Kar. Jonas Kubilius, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel LK Yamins. Recurrent connections the primate ventral visual stream mediate tradeoff between task performance and network size during core object recognition. bioRxiv, pp. 2021-02, 2022.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Pedro Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. In International conference on machine learning, pp. 82-90. PMLR, 2014.
Pieter R. Roelfsema. CORTICAL ALGORITHMS FOR PERCEPTUAL GROUPING, Annual Re- view of Neuroscience, 29(1):003-277, 2006. ISSN 0147-006X. doi: 10.1 40.11/////nu 051605. 111699 12939. URL http: www.aaaaa..eeesss.ooooo.oooo. 1ttp://0...114///murreaaa
neuro. 29, 051605 12939
Pieter Roelfsema, Victor AF Lamme, and Henk Spekreijse. The implementation of visual routines. Vision research, 40(10-12):1338 12)::1::::1111 1411, 2000.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities neural models. arXiv preprint arXiv: TrXiv:1944001577 2019.
Martin Schrimpf and Jonas Kubilius. Brain-score: Which artificial neural network for object recognition most brain-like? bioRxiv preprint, 2018. doi: 10.1101/407007. URL https:
www..bior.... wtt.////oooooo.oooo htrg//oooen///ttn 11 1101/ 10000/147700 /407007v1
Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems, 34:6695-6706, 2021.
Avi Schwarzschild, Arjun Gupta, Micah Goldblum, and Tom Goldstein. The uncanny similarity of recurrence and depth. 10th International Conference on Learning Representations, ICLR 2022, 2022.
Shimon Ullman. Visual routines. Cognition, 18(1-3):97-159, 12 1984. ISSN 00100277. doi: 00110160010022277449000223
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12894- 12904, 2021.

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 TRAINING DETAILS
All experiments were performed using PyTorch (Paszke al. 2017). On Mazes, trained models using per-pixel binary cross entropy with a minibatch size of 256 images and a learning rate schedule starting with warmup followed by learning rate decay as indicated in Schwarzschild al. (202I) for 50 total epochs of training. On PathFinder-14, we used binary cross-entropy to train models with minibatch size of 256 images and a constant learning rate of le-4 for all models for total of epochs of training (LocRNN often learns to solve the task in less than epoch and is the most sample efficient model). For models that did not converge, we varied the learning rate in steps of 0.1 lesser and greater than le-4 but still confirm that they do not converge. Models are quite robust the choice of learning rate on both PathFinder and Mazes.
A.2 PER-TIMESTEP PREDICTIONS FROM LOCRNN ON MEDIUM MAZES
Visualizing predictions of LoCRNN at subsequent timetesps of recurrence
I < T train
T train
Ind
Eel2
Imih
Trik
countoun
300
5
FUJ
Figure 4: Visualizing predictions of LocRNN (trained medium mazes) for Medium Maze image samples at subsequent timesteps shows that the model is most stable near the number of recurrent steps used during training. As increase / T used during train, the predictions of maze solution become blurred and the overall activity of the prediction grows without bounds.
A.3 EXAMPLE IMAGES FROM PATHFINDER AND MAZES
Example images from Mazes
Example images from PathFinder
ITU
Small mazes
Medium mazes (11x11)
Large
PathFinder-9
PathFinder-14
PathFinder-18
mazes (13x13)
(9x9)
Figure 5: (Left) Example images from Mazes three difficult levels (small, medium, large); (Right) Example images from PathFinder at three difficulty levels (PathFinder-9, PathFinder- 14, PathFinder- 18). Scales for different difficulties within Mazes are as shown; Pathfinder images have five times the resolution of Large Mazes.
12

---

Under review as a conference paper at ICLR 2023
A.4
INPUT AND OUTPUT FORMAT FOR PATHFINDER AND MAZES
Model output (Maze solution segmentation)
Input image
(Large
TT
Input Imagee (PathFinder-16 14)
Model output (Binary classification scores)
[00001, 0.99 [0.82, 0.18]]
Y=o
Figure 6: (Top) Example images from Large Mazes processed by model to produce the solution as a segmentation prediction. (Bottom) Example input images from PathFinder- 14 processed by classifier to produce binary classification output.
A.5 MODEL STABILITY ON MEDIUM MAZES
Measuring model stability on Medium mazes
100
Validation accuracy (%)
80
LocRNN
60
LocRNN
Feedforward ResNet
40
23 .01
11
13
13
19
Number of ilerations
Validation accuracy (%))
301-
ConvGRU
60-
40-
20-
ConvGRU
Feodforward ResNet
11
17 19 23 25
Number of iterations
Figure 7: Visualizing validation accuracy of LocRNN (top) and ConvGRU (bottom) function of the recurrent iterations. Number of iterations used during training is highlighted with circle on the horizontal axis. We observe that LocRNN is unstable when heavily increase the number of recurrent iterations in contrast to the more stable ConvGRU,
13

---

Under review as a conference paper at ICLR 2023
A.6
MODEL ARCHITECTURE DETAILS (LAYER WISE CONFIGURATION)
Layer
Layer type
Kernel size
N-out
Pooling
Input
Output
LocRNN-
Maze
Conv2D
3*3
64
None
[H. W.3]
[H. W.64]
divnorm
3*3
2
DaleR
Exc 5*5
Inh
64
None
[H,W.64]
[H. 6.6]
3*3
3
Conv2D
3*3
32
None
[H.W.64]
[32
Conv2D
3*3
None
[3,W.32]
[H.W.8]
5
Conv2D
3*3
None
[H,W88]
[H.W22]
hGRU-
Maze
Conv21)
3*3
64
None
[H.W.3]
[H. 64
2
&ConvGRU
5*5
64
None
[H.
[H. HH.W.64
Conv2D
3*3
32
None
[. H,6.
[.3)
4
Conv2D
3#3
8
None
H. W,32
[H.
5
Conv2D
3*3
None
H.W.8]
[H. W.2.
GRU-Maze
Conv2l)
3*3
64
None
[H,W.3]
[H,
2
ConvGRU
5*5
64
None
[H,,,64]
6.6
Conv2D
3*3
32
None
[. W.64]
[3.2332]
4
Conv2D
3#3
8
None
H. W,32]
[H,W88]
5
Conv2l)
3#3
None
H,W.8]
[H,W22]
Layer ResNet- 12-
Layer type
Kernel size
N-out
Pooling
Input
Output
Maze
Conv2D
3#3
None
[H.W.3]
[H. 6,64
4*Block
Conv2D
3#3
64
None
[,,W,64]
[H. 64]
Cony2D
3#3
64
None
[H,W,64]
[4,
shortcut
3
Conv2D
3#3
32
None
[H,W,44]
[H,W332]
Conv2l)
3#3
None
H,,W322
[H,
Conv2D
3*3
None
[..W88]
[H. W.2]
ResNet-20-
Maze
Conv2D
3#3
64
None
[H,W.3]
[..W.64]
BBBlock
Conv2D
343
64
None
H.W.64
[4.
Conv2l)
3#3
None
[,,W,64]
[H. 6.,W
shortcut
3
Conv2D
3*3
32
None
[H,W,64]
[H. (.32
4
Conv2D
3#3
None
[H,W.32]
[H. [.8)
5
Conv2D
3#3
None
[H,W88]
[.,
Nes-6t-36-
Maze
Conv2D
3#3
64
None
H.W.3]
[4.
16oBlock
Conv2D
3#3
None
H.W,64]
6.64
Conv2D
3#3
64
None
[H,W644]
[4
shortcut
3
Conv2D
3*3
32
None
[H,W,64]
[H.W332]
4
Conv2D
3*3
8
None
H,WW32)
[H. W,8
5
Conv2D
3#3
None
H.W.8]
[H. W.2]
14

---

Under review as a conference paper at ICLR 2023
Layer
Layer type
Kernel size
N-out
Pooling
Input
Output
R-ResNet-12 12-
Maze
Conv2D
3*3
None
[3,W3]]
[..W.64]
4ltck*Block
Conv2D
3
64
None
[H,W,64]
[,,W.64]
Conv2D
3
64
None
[H.
[6.
shortcut
3
Conv2D
3*3
32
None
[H,W,64]
[2,
Conv2D
3*3
8
None
H. H,3,
[..W8]]
5
Conv2D
3*3
2
None
H,W,8]
[H.W2]]
R-ResNet-20-
Maze
Conv2D
3*3
64
None
[H,W.3]
[,,W,64]
Siter*Block Block
Conv2D
3*3
None
[66
[H.W 64]
Conv2D
3*3
64
None
[H,W,64]
[4.
shortcut
Conv2D
3
32
None
[H,W,64]
[H.W33]
4
Conv2D
3
8
None
[,,,332]
[H.W8]]
5
Conv2D
3#3
2
None
[H, H,8,88
[22
R- ReseNet--3-
Maze
Conv2D
3*3
64
None
[H,W3]]
[6.W
Blockr+Boock
Conv2D
3*3
64
None
H,6,
[.. 64]
Conv2D
3#3
64
None
H.
[,6]
shorteut
3
Conv2D
3*3
32
None
[H,W,64]
[2.W32]
Conv2D
3#3
8
None
[H,W,32]
[H.W88]
Conv2D
3*3
None
W.8.
[2222]
Layer
Layer type
Kernel size
N-out
Pooling
Input
Output
LocRNN-
Pathfinder
Gabor Filter
3#3
32
Max
[H.W.3]
[H/4, W/4,32]
divnorm 5*5
DaleRNNLayer
Exc 9-9
Inh
32
None
[H/4, 3/4,3]
[H/4, W74,32
5*5
3
Conv2D
1
Max
H/4,W/4.32)
22]
Linear
None
[2]
[2]
hGRU-
Pathfinder
Gabor Filter
3*3
32
Max
[H.W.3]
[H/4, 3/4.32]
2
hConvGRU
9#9
38
None
H//.W/4,333
HH/4, W74.32]
3
Conv2D
1*1
Max
H/4, W/4,32)
[2
4
Linear
None
2
[2]
GRU-
Pathfinder
1
Gabor Filter
3*3
32
Max
[H,W.3]
1H/4,W/4.3)
2
ConyGRU
9*9
24
None
H/4.W/4432]
H/4,3/4.321
Conv2D
1*1
Max
1/4.3/2
2
4
Linear
None
22
[]
15


Reviews:
[{"review_id": "_tJf6pv-ivR", "reviewer": "Reviewer_3CSc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A novel recurrent architecture is introduced with the express goal of learning models that can generalize to examples that are more/less difficult than the training data. On two benchmark datasets with test sets of several difficulties, this new architecture beats existing models on out of distribution performance.", "summary_of_the_review": "I find the paper solid. There are a couple weaknesses and a couple minor points that can be addressed to improve this paper. I look forward to the authors' response. Should my concerns be adequately addressed, I'll be happy to raise my score.", "strength_and_weaknesses": "Strengths:\n- The paper flows well.\n- The experiments are thorough and illuminating.\n- The writing is mostly clear.\n- The architecture is meaningfully different from existing techniques, and shows a real benefit over those methods.\n\nWeaknesses:\n- I don’t understand the point about convergence and computational budget. Are there models that didn't train successfully? What was tried here? More specifically, in the footnote, it sounds like these models that don't train are paired down, lighter weight versions than were originally proposed -- do the full-strength models train? How is pairing down the model size (and computation budget) justified if these networks don't train?\n- The results that aim to show that LocRNN is better on mazes is not so compelling. In Figures 2 and 3 ConvGRU looks superior on Mazes. The text and the captions seem to minimize this, so at best the difference is unclear to the reader. More information and clearer claims around this are needed. \n- Code is missing. I followed the link to the anonymized GitHub repository on October 17th and it was empty. Particularly, when it is stated that code is available, I take issue with this. I understand the criteria for submission do not require code, but I don't think there is enough information to reliably reproduce the results here *and* the authors state that code is available. This needs to be addressed.\n\nTwo minor issues (not affecting score): \n- There is a bad reference to a \"Fig 5.2\" in the second paragraph on page 8. \n- The second sentence of section 5.3 is unclear: “LocRNN is the best performing architecture that outperforms all\ncompared baselines in terms of raw performance, whereas ConvGRU and hConvGRU which are much inferior to LocRNN on the PathFinder challenge reach top extrapolation performance.”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_O4w3l_UbiZ", "reviewer": "Reviewer_vnpB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a recurrent neural network architecture suited for task extrapolation, in which the level of computation required to solve a problem can be dynamically altered depending on the task's difficulty. The architecture is biologically-motivated, and the authors demonstrate performance on the Mazes and Pathfinder tasks. Experiments reveal that the proposed method, LocRNN, is successfully able to perform task extrapolation and outperforms other baselines. The authors also find an intriguing tradeoff between performance on task extrapolation and stability.", "summary_of_the_review": "Overall, the paper proposes an interesting, biologically-motivated solution to the problem of visual task extrapolation. Unfortunately, the empirical results are a little weak which limits the significance of the paper in its current form. If the experiments can be improved, the paper would be a much stronger contribution.", "strength_and_weaknesses": "**Strengths**\nThe authors investigate a potential solution to the problem of task extrapolation, which is an important area, and moreover take inspiration from research on visual routines. The proposed LocRNN method is well explained and has a good biological motivation. In the experiments, a number of good baselines are compared. Analysis of the experiments is strong; in particular, the tradeoff between extrapolation and stability is a nice addition to this section.\n\n**Weaknesses**\nIn my view, the main weakness with this paper is with the empirical results. First, the Mazes and Pathfinder tasks may be a little too simple. If the goal of this paper is to propose an RNN solution to general visual task learning, then it would be better to experiment with more complex tasks. Otherwise, the significance of the paper is unfortunately limited.\n\nAlso, it is unclear how much better LocRNN performs compared to the baselines. For example, in Figure 2 it appears that ResNet-18 performs similarly to LocRNN on Pathfinder. Also, as the authors point out, on Mazes, ConvGRU outperforms LocRNN on task extrapolation to large mazes. I may be missing a key result, but I unfortunately don't see where LocRNN significantly outperforms the baselines.\n\nAs a more minor point, I'm unsure in section 5.3 if the instability of LocRNN is what causes better performance. The authors propose a potential explanation for why this may be the case, but it would help to have an experiment justifying this explanation. For example, how would ConvGRU perform with ReLU activation?\n\nFinally, it would be nice to include some ablation experiments on LocRNN. What aspect of LocRNN makes it effective compared to the baselines (e.g. is it the gating mechanism or the division into two neural populations)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Gd263HODUSj", "reviewer": "Reviewer_j482", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new gated convolutional recurrent architecture consisting of separate neural populations with long-range and short-range interactions. This structure is vaguely motivated by neuroanatomical cortical observations that revealed that neurons in the primary visual cortex of mammals are strongly recurrently coupled and this recurrence tends to be subdivided in long-range excitatory inputs and inhibitory inputs from local interneurons. In particular, these two pathways were implemented through convolutional operations with different kernel sizes, which are then mixed and combined with gating operations.\nThis novel recurrent architecture is then empirically compared against feed-forward vision architectures (ResNets) and recurrent vision architectures (ConvGRU and a recurrent version of ResNet obtained by tying weights across layers) on two previously proposed path integration tasks: Mazes (that asks to output a segment connecting two points in a 2d maze) and PathFinder (a classification tasks consisting on determining whether two disks on a image are connected or not by a curved segment among other distracting segments).\nThe proposed architecture performs better on these tasks than the competitors at most difficulty levels, although it is interestingly revealed that a version of ConvGRU is better at extrapolating on a difficult level after being trained on a different difficulty level. Related to this, ConvGRU is also shown to be more robust to varying the number of recurrent steps. ", "summary_of_the_review": "The paper starts with a interesting motivation, understanding the computational properties that the anatomy of cortex (high recurrence, separate local and long-range connection) provides to a RNN module. However, the paper doesn´t analyze the RNN module that is derived to incorporate these properties in a deep enough quantitative way. Moreover, the derived architecture is only benchmarked on a very restrictive set of tasks (path finding) that are arguably only a weak match with respect to the biological motivation (understanding cortex) and the conv-nets baselines.", "strength_and_weaknesses": "The paper has some very interesting elements, starting from the biological and fundamental science motivation. However, it still needs some work in terms of providing mechanistic and algorithmic understanding on how the cortically motivated recurrent motif accomplishes what it accomplishes, and establishing the relevance of the specific path integration tasks in the context of the cortical substrate that the paper purportedly examines. More in detail, here are some of the strengths and weaknesses of the paper.\n\nStrengths:\n- The paper is trying to connect known fact of the anatomy and functionalities of primary visual cortex of the brain to the engineering of deep learning. In particular, the fact that the connectivity of the cortex is highly recurrent and it seems to be organized in terms of clearly defined short- and long-range connections. This sort of investigation seems very promising as it may help elucidate the algorithmic properties of neuro-physiological and anatomical structures.\n- The paper investigate relevant metrics beyond accuracy, such as robustness as a function of recurrent steps, and generalization across task difficulty.\n\nWeaknesses:\n- The paper is in part motivated by the statement that feedforward networks are \"strictly restricted\" and \"cannot dynamically change their computational graph\". On the other hand, conditional modules like conditional batchnorm or context-dependent representations like those provided by transformers strongly contradict this claim. In addition, a recurrent architecture when unfolded through time can be thought of as a feedforward network with tied weight (as the paper also recognizes when examining the R-ResNet architecture). It is then not clear what exactly this claim regarding the limitations of feedforward architectures alludes to exactly, since it would also pertain to RNNs (being a special case of feedforward architecture). It would be beneficial to re-target this claim and re-contextualize in light of these two objections.\n- The paper proposes a new recurrent module without however providing any quantitative analysis of its functioning. It would be interesting if the paper would also include an analysis of the stability of module examining the vanishing and exploding gradient problems. \n- The paper is highly focused on a very restricted and idiosyncratic type of task (path finding) which isn't  clearly motivated in terms of being of wide enough interest for the machine learning and deep learning communities at the moment. This could be mitigated by arguing that these tasks are at least a good fit for the cortical-type of architectures that are then proposed. In other words, what is the evidence that these tasks are paradigmatic of the type of tasks that the visual cortex and its architecture are solving? One would assume that the visual cortex is rather engaged in visual perceptual tasks, rather than planning and path finding. \n- The objection for the type of task that is being investigated could arguably also apply to the use of ReNets as baseline models, since they have been built to solve visual perceptual tasks, while, based on the fact that Dijkstra's algorithm can be used to solve shortest path problems, one would assume that algorithms for path finding have more to do with implementing dynamic programming.\n- The paper demonstrate that the novel proposed RNN module displays some interesting properties in how it seem to be tackling the path finding tasks differently from other RNNs and feeforward architectures. It is however not clear from the paper what computational or algorithmic features distinguish the different ways of tackling this class of problems. Presumably, the differences in performance have to do with the different inductive biases that different architectures have for solving this problem in the specific way that it is formulated (as visual classification tasks).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CgVU_6euJ4", "reviewer": "Reviewer_i8Ub", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a recurrent convolutional architecture (LocRNN) to improve extrapolation in visual tasks. The novelty of the architecture is presented as being long-range lateral connections between neurons from the same layer. The architecture is tested on two tasks: Mazes (route segmentation) and PathFinder (curve tracing). Both of these have multiple difficulty levels to test on. \n\nLocRNN architecture is based on an ODE-based model of lateral connections between cortical columns in the primate brain. The architecture uses a long range neural population and a short range neural population, each of which has gated updates and lateral connections defined by convolutional kernels within and between the neural populations. \n\nThis is added to residual network architecture, and compared with GRU and ResNets themselves. LocRNN and GRU outperform ResNets (feedforward only) on both tasks. To show extrapolation, experiments are conducted by training on one difficulty level and testing on the rest - ConvGRU and LocRNN both do well in extrapolation, and ConvGRU (specialized to sequential processing) does the best. \n\nFinally, the paper claims that extrapolation ability (as shown by the existing recurrent architectures) trades off with task performance because performant algorithms lack stability in the hidden space. ", "summary_of_the_review": "I am recommending a weak acceptance because I do not see any major flaws in the paper and the general agent performance results support the main claims they are attached to. I am not recommending strong acceptance because important and very similar convolutional recurrent networks prior work has not been compared to, and the scope of the experiments and method is somewhat limited. Furthermore, the tradeoff is addressed briefly despite being a main contribution and a compelling claim, and the paper would benefit from doing more for that claim.", "strength_and_weaknesses": "**Strengths**\n- Architecture is intuitive and appears to be a simple but effective model and discretization of the human neural \"long lateral connections\"\n- Paper is well-scoped - it raises certain questions and answers them effectively with experiments that tie into each other\n- Experiments are well-thought out and support claims\n- Results are significant, particularly the extrapolation result that shows good performance on \n- Generality of approach, compared to more specialized architectures, is exciting\n\n**Weaknesses**\n- Experimental suites are somewhat limited, could benefit from more tasks\n- Not enough results are presented in the main paper - we are told that LocRNN has better task performance than others, but there are no tables to highlight the statistics and Figure 2 doesn't clearly demonstrate this relative to GRU (or at least, I didn't notice) \n- The tradeoff section is unconvincing - it's a brief paragraph relying on a visualization, and even if it had more concrete evidence, the connection between the statistics presented and hidden state stability is unclear. Ultimately, we have three data points, not an empirically established or theoretically proven trend (as far as the paper goes), though connections could be established.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject