Title: Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels
Venue: NeurIPS 2023
Accepted: yes
Keywords: cognitive science

recurrent neural networks

adaptive computation time

visual reasoning
Abstract: Humans solving algorithmic (or) reasoning problems typically exhibit solution times that grow as a function of problem difficulty. 
Adaptive recurrent neural networks have been shown to exhibit this property for various language-processing tasks. However, little work has been performed to assess whether such adaptive computation can also enable vision models to extrapolate solutions beyond their training distribution's difficulty level, with prior work focusing on very simple tasks. In this study, we investigate a critical functional role of such adaptive processing using recurrent neural networks: to dynamically scale computational resources conditional on input requirements that allow for zero-shot generalization to novel difficulty levels not seen during training using two challenging visual reasoning tasks: PathFinder and Mazes. We combine convolutional recurrent neural networks (ConvRNNs) with a learnable halting mechanism based on Graves (2016). We explore various implementations of such adaptive ConvRNNs (AdRNNs) ranging from tying weights across layers to more sophisticated biologically inspired recurrent networks that possess lateral connections and gating. We show that 1) AdRNNs learn to dynamically halt processing early (or late) to solve easier (or harder) problems, 2) these RNNs zero-shot generalize to more difficult problem settings not shown during training by dynamically increasing the number of recurrent iterations at test time. Our study provides modeling evidence supporting the hypothesis that recurrent processing enables the functional advantage of adaptively allocating compute resources conditional on input requirements and hence allowing generalization to harder difficulty levels of a visual reasoning problem without training.

Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels

Abstract

Humans solving algorithmic (or) reasoning problems typically exhibit solution times that grow as a function of problem difficulty. Adaptive recurrent neural networks have been shown to exhibit this property for various language-processing tasks. However, little work has been performed to assess whether such adaptive computation can also enable vision models to extrapolate solutions beyond their training distribution's difficulty level, with prior work focusing on very simple tasks. In this study, we investigate a critical functional role of such adaptive processing using recurrent neural networks: to dynamically scale computational resources conditional on input requirements that allow for zero-shot generalization to novel difficulty levels not seen during training using two challenging visual reasoning tasks: PathFinder and Mazes. We combine convolutional recurrent neural networks (ConvRNNs) with a learnable halting mechanism based on (Graves, 2016). We explore various implementations of such adaptive ConvRNNs (AdRNNs) ranging from tying weights across layers to more sophisticated biologically inspired recurrent networks that possess lateral connections and gating. We show that 1) AdRNNs learn to dynamically halt processing early (or late) to solve easier (or harder) problems, 2) these RNNs zero-shot generalize to more difficult problem settings not shown during training by dynamically increasing the number of recurrent iterations at test time. Our study provides modeling evidence supporting the hypothesis that recurrent processing enables the functional advantage of adaptively allocating compute resources conditional on input requirements and hence allowing generalization to harder difficulty levels of a visual reasoning problem without training.

1 Introduction

Recurrent Neural Networks (RNNs) have emerged as a powerful tool for solving machine reasoning tasks, demonstrating impressive performance across a variety of tasks that require processing of sequential inputs. While recurrent processing is valuable for time-varying inputs, it can also be useful for static inputs as recurrent networks have the ability to scale computation to varying difficulty levels by varying the number of recurrent iterations. This can be helpful when different problems from the same task family exhibit significant variations in complexity (consider mazes, for example). However, conventional RNNs typically struggle to automatically generalize across different difficulty levels, requiring retraining, fine-tuning, or human intervention to pick the number of recurrent iterations needed. This limitation hampers their practical utility in real-world scenarios, since the span of difficulty levels seen during training is finite.

Human reasoning, in contrast, is characterized by a highly adaptible use of computation - arbitrarily more computation can be used for more difficult tasks. Despite the evidence for such scaling in human computing there is limited work in the literature analyzing Adaptive RNNs (AdRNNs) that, on a sample-by-sample basis, automatically decide when to stop. Graves introduced Adaptive Computation Time (ACT), a mechanism to automatically stop computation, by learning to generate a scalar halting probability (Graves, 2016). However, similar to subsequent work (Banino et al., 2021), it was evaluated on either simple tasks such as parity checking, or language tasks. Few studies examine the training and evaluation of these AdRNNs on visual reasoning tasks, which pose unique challenges due to the redundant and high-dimensional nature of visual data. Furthermore, extrapolating to harder instances within-task has not been sufficiently studied, with Banino et al. (2021) evaluating such zero-shot performance only on the simple parity task.

In this paper, we study the problem of computation scaling in recurrent vision RNNs, with an emphasis on zero-shot extrapolation to harder/larger problems within the same task. The ability to handle unseen difficulty levels without fine-tuning, or human intervention, enables more robust and adaptable computer vision systems, which are crucial for real-world applications. We investigate the effectiveness of AdRNNs on two challenging, publically available, visual reasoning tasks based on curve tracing and route segmentation, namely PathFinder (introduced by Linsley et al. (2018)) and Mazes (introduced by Schwarzschild et al. (2021)).

The contributions of this work are as follows:

We combine Convolutional RNNs with an adaptive computation method of Graves (2016) producing Adaptive ConvRNNs (AdRNNs) that are capable of learning a downstream task simultaneously while also learning to scale their computational steps as per input image/task requirements.

We introduce LocRNN, a high performing recurrent architecture inspired by prior computational models of recurrence in biological vision.

We show that AdRNNs learn to dynamically halt processing early (or late) to solve easier (or harder) problems when the train- and test-difficulty levels are matched on complex visual reasoning problems inspired by stimuli used in prior cognitive science research.

During test time we introduce a previously unseen harder difficulty level. We evaluate AdRNNs on these new difficulty levels and show that they zero-shot generalize to more difficult problem settings not shown during training by dynamically increasing the number of recurrent iterations at test time well beyond the number of recurrent steps used during training.

2 Related Work

Our work is relevant to the visual routines literature introduced by Ullman (1984) and further reviewed elaborately by Roelfsema et al. (2000). The core idea of visual routines that make it relevant to our studied question of task extrapolation is the flexible sequencing of elemental operations resulting in a dynamic computational graph, which make RNNs a natural approach to solve such tasks.

Prior research has proposed mechanisms for RNNs that learn the amount of recurrent computational steps required. Following are two particularly relevant attempts in prior literature in this area: Graves (2016) developed the Adaptive Computation Time (ACT) method to train RNNs on Natural Language Processing tasks with a sigmoidal halting unit that determines when to halt recurrent processing, and Banino et al. (2021) extend upon this pioneering work by taking a probabilistic approach to halting and introducing a geometric prior-based specification of computational budget. However, neither of these studies applied adaptive computation to vision RNNs (that are significantly trickier to optimize). Also the prior studies did not attempt to examine computation scaling under zero-shot generalization to harder difficulty levels. Additionally, Eyzaguirre and Soto (2020) developed a version of ACT applied to visual reasoning problems, however, their observations on the CLEVR dataset (in which the issues related to object diversity are highlighted by Kim et al. (2018)) do not explore generalization to new difficulty levels which is central to our results.

In terms of generalization from easy to hard vision problems, our work is relevant to (Schwarzschild et al., 2021) and (Bansal et al., 2022) where the authors evaluate the application of recurrent neural networks to generalize from easier to harder problems. Our work extends and differs from their

intriguing study of recurrence in task extrapolation in the following ways: 1) For recurrent networks used in their study, human intervention is required to specify the number of recurrent computational steps during the testing phase. 2) Their work explores sequential task extrapolation in general with abstract problems such as Prefix Sum and solving Chess Puzzles while our work extends it to particularly focus on extrapolation in visual task learning. 3) We present evaluation on the Pathfinder challenge (Linsley et al., 2018), a relatively more large-scale visual reasoning task shown to be very challenging (Tay et al., 2020), the design of which dates back to (Jolicoeur et al., 1986). 4) Schwarzschild et al. (2021) and Bansal et al. (2022) implement only the most straightforward form of recurrence realized by weight-tying, i.e., they only evaluate ResNets with weight sharing across residual blocks (albeit with a novel training scheme in the follow up study (Bansal et al., 2022)). In addition to such recurrent ResNets, we present analyses with highly sophisticated recurrent architectures specialized for recurrent image processing. 5) We introduce LocRNN, a high performing recurrent architecture based on prior computational models of cortical recurrence.

On the design of recurrent architectures, our work is loosely related to (Eigen et al., 2013), (Pinheiro and Collobert, 2014), (Veerabadran and de Sa, 2020) and (Liao and Poggio, 2016) which discuss the role of weight sharing in feedforward networks to produce recurrent processing. We are interested, however, in designing new specialized recurrent architectures that play a role both in human and machine vision. While there exist more such recurrent architectures informed by neuroscience such as Linsley et al. (2018); Nayebi et al. (2022), we find these architectures to be quite difficult to interpret and unstable to train (based on in-difficulty performance evaluation included in the Supplementary). Our proposed LocRNN architecture in comparison is an elegant and easy-to-interpret implementation of recurrence that is stable to train. Our work is also relevant to prior work on modeling speedaccuracy tradeoffs observed during visual perception (Spoerer et al., 2020).

3 Datasets

For evaluating the ability of various models in exhibiting task extrapolation, we curate two challenging visual reasoning tasks, Mazes and PathFinder, with instances at multiple parametric difficulty levels. Both tasks involve the visual routines of marking and curve tracing (Ullman, 1984). These datasets are inspired by prior visual psychophysics research where such tasks were used abundantly to estimate the cognitive and neural underpinnings of sequential visual processing, such as incremental grouping, structure and preference of lateral connections, etc. (Jolicoeur and Ingleton, 1991; Ullman, 1984; Li et al., 2006; Roelfsema, 2006). In the following subsections we describe the specifics of our tasks.

3.1 PathFinder challenge - curve tracing

(a)PathFinder-14

(b)PathFinder-18

(c)PathFinder-24

96x6()

(b)15x15

(c)19x19 Maze

(d)25x25Maze

Maze

Task description: In the PathFinder task introduced by Linsley et al. (2018), models are trained to identify whether two circular disks in an input stimulus form the two ends of a locally connected path made up of small 'segments'. Each image consists of two main long connected paths P 0 and P 1 made up of locally aligned segments as well as shorter distractor paths. Each image also contains two circular disks which are placed at two of the 4 possible endpoints of P 0 and P 1 . Images that contain a disk on both ends of the same path are classified as positive, and those containing a disk on endpoints of different paths are classified as negative. Examples are shown in Figure 1.

Difficulty levels: Pathfinder is designed at different difficulty levels parameterized by the length (number of segments) of the paths P 0 and P 1 mentioned above. The easiest version uses paths that are 9 segments long (PathFinder-9), while the medium and hard versions contain paths that are

14 (PathFinder-14) and 18 (PathFinder-18) segments long respectively (see example images in the Appendix). This dataset consists of a total of 800,000 RGB images at each difficulty level, each with a spatial resolution of 160 × 160 pixels. There are an equal number of positive and negative instances at each difficulty level. We use 700,000 images for training and 100,000 images as the test set from each difficulty level. We combined these datasets to create a more challenging dataset with varying levels of difficulty, which we refer to as PathFinder-Mixed. To evaluate the zero-shot difficulty extrapolation on PathFinder in Sec. 5.2 we generated 100,000 images each with contour lengths 21 and 24 respectively; we call these PathFinder-21 and PathFinder-24.

Evaluation criteria: Since this is a classification challenge, we use accuracy, i.e. % correct on test-images as the evaluation metric to rank model performance on PathFinder. Model architectures receive an input image and process it via a stack of standard convolution/recurrent-convolution layers followed by a classification two-class readout. Since this is a binary classification challenge with balanced classes, chance performance is 50% for random predictions.

3.2 Mazes challenge - route segmentation

Task description: Human beings are adept at solving mazes, a task that requires application of a similar serial grouping operation like PathFinder in order to discover connectivity from a starting point to the final destination of the maze amidst blocking walls. For evaluating model performance on solving mazes of varying difficulty, we use the publicly available version of the Mazes challenge developed by Schwarzschild et al. (2021). They implemented this Mazes challenge as a binary segmentation problem where models take N × N images of square-shaped mazes as input with three channels (RGB) with the start position, end position, permissible regions and impermissible regions marked in the image. The output produced by models is a binary segmentation of the route discovered by the model from the start to the end position.

Difficulty levels: The Mazes challenge has been designed at several difficulty levels, each difficulty level is parameterized by the size of the square grid that the maze is designed into. We use maze datasets of grid sizes 9 × 9 and 15 × 15 for training. Each dataset consists of 50,000 training images and 10,000 test images that are guaranteed to not overlap. The spatial resolution of 9 × 9 maze images is 24 × 24 pixels and that of 15 × 15 maze images is 36 × 36 pixels. Similar to PathFinder-Mixed, we combine 9 × 9 and 15 × 15 grid sizes to form Mazes-Mixed. We also constructed larger mazes with grid sizes 19 × 19 (with spatial resolution 44 × 44 pixels) and 25 × 25 (with spatial resolution 56 × 56 pixels) to evaluate a model's ability to extrapolate to difficulties not seen during training. As above with PathFinder's extrapolation evaluation, these novel difficulty mazes were only used during testing and were not in any form used in the training or hyperparameter optimization process.

Evaluation criteria: Mazes is a segmentation challenge and hence, one could potentially consider partially correct routes during evaluation (for example with average of per-pixel accuracy). However this is less strict than giving each image a single binary score reflecting if all pixels are labeled correctly. Evaluation criteria for mazes is hence the total %of test-set mazes completely accurately solved at a given difficulty level.

4 Model architectures and training

4.1 Implementations of adaptive computations evaluated on task extrapolation

In this section, we describe our choice of recurrent architecture designs that we use to study the behavior of adaptive computation and task extrapolation in deep learning. We process images (denoted as X ∈ R c,h,w ) in three stages that are common to all models we evaluate in this study. These three stages are as follows: (1) an input convolution layer (denoted as input ( . ) ) that operates on the image directly.

The output from this stage h 0 ∈ R d,h,w is fed as input to the following recurrent block in (2).

where r ( . ) is the recurrent block. t train is a training hyperparameter indicating the maximum number of timesteps for unrolling r ( . ) during training. This block consists of a sequence of convolution

layers applied in an iterative manner for any arbitrary number of timesteps. This is the sub-part of our model that is capable of performing adaptive computations. (The specific architecture of these convolution blocks constitute different implementations of recurrent operations described further below.) (3) a readout layer containing a block of convolution and pooling operations that produce the desired output from our network.

While we keep the input and readout layer architectures the same for all models we evaluate, their intermediate recurrent blocks have different architectures (corresponding to their respective recurrent cells). We explore three different implementations of recurrent computations which are used in the second, recurrent block of models. Our first choice as the intermediate feature processing block consists of a residual network with weight tying across all layers; we refer to this model as R-ResNet30. Next, we study the performance of the following specialized convolutional recurrent units from prior work: horizontal convolutional GRU (hConvGRU) and its stable variant (Linsley et al., 2018, 2020) and a convolutional Gated Recurrent Unit (ConvGRU) (Ballas et al., 2015) with LayerNorm (ConvGRU does not converge in the absence of LayerNorm). Third, we design a novel recurrent cell based on prior computational models of cortical recurrent processing (Li et al., 2006); this model equips the biologically inspired design choices of long-range lateral interactions, gating, and a separate population of interneurons. This model is referred to as LocRNN and is described in the following Sec. 4.3. Importantly all models are matched for trainable parameters.

4.2 Combining ConvRNNs with Adaptive Computation Time (ACT)

The central theme of our work is to show that RNNs can flexibly adapt (or scale) their computation according to input requirements. We achieve this ability by combining ConvRNNs with an adaptive computation mechanism based on Graves (2016) called Adaptive Computation Time (ACT). A difference between our work and ACT is that our visual reasoning task involves static inputs (i.e. sequence of length 1) whereas Graves (2016) deals with variable-length sequences and learns adaptive processing of each token. Owing to this difference, our halting mechanism is similar to ACT applied to a 1-token input sequence.

The key idea of ACT is to introduce a separate 'halting mechanism' that learns to control the number of recurrent computation steps dynamically, conditioned on each input example's processing. In addition to producing the next recurrent state, an RNN equipped with ACT also produces a scalar value called the 'halting score' for each computation step. In addition to the next hidden state computed using Equation 2, we generate a scalar halting score p t at each step using a learnt convolution layer ( halt_conv ( . ) ) that is shared across timesteps:

The RNN treats the cumulative sum of the halting scores up to timestep t ( P t described below) as a quantity used to determine whether processing is terminated at that timestep.

The RNN keeps track of the accumulated halting scores and checks if the cumulative sum of the halting scores at each step reaches a predefined threshold (1 -ϵ ) . If the threshold is reached, the computation stops (i.e., p t = 0 ∀ t > t halt ) where the cumulative halting score threshold is reached at timestep t halt = min { t : P t > = (1 -ϵ ) } . The adaptive hidden state is then computed as a weighted average of the hidden states up to t halt , scaled by the halting scores at each time step. The readout is then applied to this final adaptive hidden state to produce the ACT task prediction, ˆ y act :

The abovementioned ConvRNN combined with ACT is trained to optimize both the downstream task loss ( || y -ˆ y act || p ) and an auxiliary 'ponder cost' corresponding to the halting mechanism

that encourages models to use the fewest number of recurrent step to process an input example. Ponder cost is computed as the cumulative halting score until timestep t halt -1 , maximizing this term encourages (or minimizing its negative as shown below) completing the task as quickly (with as few recurrent steps) as possible. The following final objective function is optimized to train the full model containing ConvRNNs with ACT where τ is a hyperparameter.

4.3 Formulation of LocRNN

We note that prior work has explored the development of recurrent architectures tailored for vision. Here, we introduce a similar but highly expressive recurrent architecture designed based on a computational model of iterative contour processing in primate vision from Li (1998). This model is an ODE-based computational model of interactions between cortical columns in primate area V1 mediated by 'lateral connections' local to a cortical area.

→

ho

LocRNNarchitecture

Tonextlayer

Lt activations showing contour tracing in PF-14

K

Input

t=1

t=3

t=5

t=7

L>

一

→ho

ConvGRUarchitectureTonextlayer

Y

FSt

X

>Lt

个

（1-

St-1

K

→-Lβ

By discretizing the continuous-form ODE dynamics of processing units from Li (1998), we arrived at an interpretable and powerful set of dynamics for 'LocRNN'. As in ConvGRU, the effective receptive field of LocRNN output neurons increases linearly with recurrent timesteps.

Unlike ConvGRU, the LocRNN's hidden state is composed of two neural populations, L and S ; the activity in these populations are referred to as L t and S t at timestep t respectively. These two populations are motivated by the x (excitatory) and y (inhibitory) neurons in Li (1998). However we found that restricting the signs of their weights (to reflect exclusively excitatory and inhibitory connections) led to less stable behavior. On the other hand, retaining the interneuron property (local computation not projecting out for downstream processing) of the S population performed better than using a single uniform population. The following equations illustrate the working of ACT and how the readout is applied in LocRNN.

Initially, both L 0 and S 0 populations are set to a tensor of zeros with the same shape as h 0 , a 4 d tensor of shape ( batch_size × channels × height × width ).

L and S update gates G L t and G S t (same shape as L and S ) are computed as functions of the input and current hidden states L t -1 and S t -1 using 1x1 convolutions U ∗ .

Each of the 4 types of lateral connections are modeled by convolution kernels W L → L , W L → S , W S → S , and W S → L respectively of shape d × d × k × k where d is the dimensionality of the hidden state and k represents the kernel spatial size.

Once the long-range lateral influences are computed and stored in L ˜ t and S ˜ t , these are mixed with the previous hidden states using the gates computed in Eq. 10 and 11. These hidden states are then passed on to subsequent recurrent iterations where even longer-range interactions occur (as time increases).

In the above equations, LN () stands for Layer Normalization (Ba et al., 2016), and the nonlinearities γ and κ are both set to ReLU. One of the differences between LocRNN and ConvGRU (which we also evaluate) is the presence of the interneuron S population in the former.

5 Results

5.1 Adaptive RNNs scale their computation as a function of input difficulty

In this section, we report our observations from training AdRNNs on a mixture of difficulty levels from PathFinder and Mazes & evaluating them on a held-out set of images from the same mixture of difficulty levels of PathFinder and Mazes. We created one training dataset for each of these challenges by combining input samples from multiple 'easy' difficulty levels. Combining input samples from multiple difficulty levels increases the diversity of computational requirements during training and helps in developing models that do not degenerate to using the same computational steps for all input samples.

As described in Section 3, we created a PathFinder training set of images and corresponding labels by sampling an equal number of images from three difficulty levels: PathFinder-9, PathFinder-14, and PathFinder-18. Similarly, we created a Mazes training set of images and ground-truth segmentation labels by sampling an equal number of images from two difficulty levels: 9x9 mazes and 15x15 mazes.

On held-out test sets that matched the difficulty level of the above-described training data, we tested whether models were able to scale their recurrent computational steps as a function of the input difficulty level. We show the results from this evaluation in Table. 1. We observed that both the sophisticated AdRNNs tested (ConvGRU and LocRNN) were able to generalize to the held-out set. We also observed that variants of horizontal GRU (Linsley et al., 2018) and Linsley et al. (2020) models generalized to the (within-difficulty) held-out set without using ACT. The simpler recurrent network without gating, weight-tied R-ResNet-30, was unable to learn on PathFinder and on Mazes highlighting the importance of specialized operations such as gating and backpropagation through time. The presence of skip connections between R-ResNet-30's recurrent blocks could still not match the expressivity of the specialized RNNs.

If ACT is working as expected, we must observe that AdRNNs that learn the task dynamically use less compute to solve easy examples and more compute (more recurrent iterations) to solve harder examples. To check for this trend, we analyzed the number of steps chosen by the model before halting for each example from the validation sets of PathFinder and Mazes; with varying contour lengths and maze sizes respectively. These results are shown by the cool colors in Figure 3 for PathFinder and in the Supplementary for Mazes. As is clearly observable from the trend in these Figures, examples that we consider as harder (longer contours in PathFinder and larger mazes) are assigned a higher number of recurrent computation steps by ACT than easy examples. Hence we show that AdRNNs obtained by combining ConvRNNs with ACT training are capable of learning both PathFinder and Mazes in addition to learning to adapt their recurrent computational steps as a function of input example difficulty. To the best of our knowledge, we are the first to show the above result for visual tasks inspired by stimuli used in prior cognitive science research.

Distribution of halting steps: LocRNN

Distribution of halting steps: ConvGRU

100000

PF9

80000

PF9

PF14

PF14

80000

PF18

PF18

Frequency

PF21

Frequency

60000

PF21

60000

PF24

PF24

40000

40000

20000

20000

0

0

5

6

7

8

9

10

3

4

5

6

7

8

Halting step

Haltingstep

Performance on PF-9,PF-14 and PF-18 (combined)

Performance on held out PathFinder-21

Performance on held out PathFinder-24

1.0

1.0

1.0

Validation performance

6'0

Validation performance

0.9

Validation performance

0.9

ConvGRU

ConvGRU

ConvGRU

0.8

R-Resnet-30

0.8

R-Resnet-30

0.8

R-Resnet-30

Resnet-30

Resnet-30

Resnet-30

0.7

ConvGRU (ACT)

0.7

ConvGRU (ACT)

0.7

ConvGRU (ACT)

9°0

0.6 -

0.6

0.5

0.5

0.5

34567891011121314151617181920

2

345

67891011121314151617181920

123456

7891011121314151617181920

Inference RNN iterations

Inference RNN iterations

Inference RNN iterations

5.2 Adaptive RNNs generalize to novel difficulty levels by scaling their computation

In typical circumstances where RNNs are expected to solve instances that are more difficult, human intervention in the form of specification of the number of recurrent iterations is commonplace (Schwarzschild et al., 2022). This approach is both laborious and expensive to perform as it requires training and/or evaluation on new test sets with various settings of the number of recurrent timesteps. Additionally, this process needs to be repeated at every occasion of new incoming test data that is potentially of increased difficulty. In the previous section, we showed the ability of AdRNNs in learning to solve PathFinder and Mazes while also demonstrating that they assign computation as a function of input difficulty on a test-set with matched task difficulty level as the training set. Here we are interested in testing whether AdRNNs are able to extrapolate (by using more recurrent computational steps compared to training) in complex reasoning tasks. We constructed two additional datasets each in the PathFinder and Mazes families to test these adaptive models' ability to extrapolate, as mentioned in Section 3. While AdRNNs are trained on the training difficulty levels for a maximum of t train iterations, we evaluated them on these new (more difficult) datasets at inference with a maximum of t inference ≥ t train recurrent iterations where we refer to AdRNNs as operating in their extrapolation phase. We report the results from this evaluation in Table. 2 and in Fig. 4.

Table 2: Accuracies (Mean % ± SEM) ↑ on extrapolation datasets. Chance performance is 50% for pathfinder and 0% for Mazes.

∗ hConvGRU training converged only on 1 out of 3 seeds (which were chosen randomly for all models here) on PathFinder, corresponding results above show this converged model's performance.

Adaptive RNNs generalize to novel difficulty levels by scaling their computation Our evaluation of AdRNNs trained with ACT on novel harder difficulty levels shows that as expected, AdRNNs have learned to use the optimal number of recurrent computational steps required for achieving strong generalization to the novel difficulty levels on a per-image level. On PathFinder, as seen in Figure 3, most instances from PathFinder-21 and PathFinder-24 take 9 steps in LocRNN and up to 7 steps in ConvGRU, higher than the number of steps used for easy training difficulty examples. In Fig 5 we visualize the relationship between mazes' difficulty and the number of recurrent iterations used by ACT (on a per-instance level). We observe that maze difficulty (length of the ground-truth route in pixels) and the number of ACT iterations are strongly positively correlated. Notably, AdRNNs trained with ACT choose to make t halt during inference on longer mazes greater than t halt used on the shorter training mazes.

30

100

XK

Xx

08

×

X

25

XXXX

XOX

X

XXXKXX

XXXX

Halting step

Halting step

20

XXX.

，

XXXX

60

双

XX

x

15

XXXX

X

40

10

X

+

美

X

20

LocRNN (19x19 mazes)

5

>

?

ConvGRU (19x19 mazes)

×

LocRNN (25x25 mazes)

美X

ConvGRU (25x25 mazes)

0

200

400

600

800

1000

0

200

400

600

800

1000

Mazesolutionlength

Mazesolution length

5.3 AdRNNs outperform halting in hConvGRU based on stability of hidden-state dynamics

Using stability in hidden-state space as an alternative way to perform halting during extrapolation, we evaluate hConvGRU on extrapolation. For this method, we halted processing in hConvGRU when the mean absolute difference between two subsequent states ( || h t -h t -1 || ) reduced to less than a tenth of the difference between the first two states (heuristic for stability). This method referred as hConvGRU (stable halting) in Tables 1 and 2 shows differences in generalization across our datasets. AdRNNs perform better than hConvGRU (stable halting) on both datasets ; especially, stable halting completely fails to show generalization to PathFinder-21 and -24. We hypothesize this method to work suboptimally for the following reasons:

Stability-based halting requires defining a hand-engineered heuristic on how much change in the output is considered small enough to halt. In typical segmentation tasks like Mazes, the network output for the initial few timesteps is highly stable in making nonsensical

predictions (predicting all pixels as the negative class) and thus, heuristics need to identify an inflection point in the output trajectory where meaningful predictions start to emerge and stabilize. In the absence of ground truth information, one cannot pick a heuristic that generalizes to unseen data.

Learnable halting makes less assumptions about the hidden state's properties, and hence doesn't enforce hard constraints such as stability to be satisfied by training. Some, but not all RNNs, have stable hidden states wherein the network response stops changing after reaching an attractor. Linsley et al. (2020) argue that RNNs that are expressive have an intrinsic inability to learn stable hidden states. For RNNs to be stable, their hidden state transformation needs to model a contractive mapping (Miller and Hardt, 2018; Pascanu et al., 2013). That is, the recurrent transition function F satisfies the following inequality: || F ( h t ) -F ( h t -1 ) || p < λ || h t -h t -1 || p . RNNs with stable hidden states that satisfy the above inequality are quite difficult to train on challenging problems in practice. Even when stable models perform comparably to unstable models as in (Miller and Hardt, 2018), the authors show unstable models' advantages such as performance improvements in the short-time horizon and lesser vanishing gradient issues.

6 Conclusion

The advantage of using recurrent networks for processing static inputs adaptively, particularly in order to zero-shot generalize to new difficulty levels is understudied. In this work, we show that deep convolutional networks with intermediate recurrent blocks (ConvRNNs operating on image features) can be combined with an adaptive computation technique to learn to dynamically process different input examples based on a per-instance difficulty level. We combine ConvRNNs with a learnable halting mechanism that is based on Graves (2016) to produce AdRNNs. We evaluated diverse implementations of recurrence in increasing level of sophistication/complexity, R-ResNet-30 with weight-tying, ConvGRU and hConvGRU with gating, and LocRNN with two separate populations of horizontally connected units (one of which are interneurons) and gating on two challenging visual reasoning problems, PathFinder and Mazes, which are generated at various levels of difficulty. First, we showed that only the specialized RNNs (hConvGRU) and AdRNNs (LocRNN and ConvGRU) are capable of learning PathFinder and Mazes. Second, these AdRNNs trained with ACT are learning to dynamically use less (or more) recurrent computational steps for easy (or hard) PathFinder and Maze problems respectively as discussed in Section. 5.1. More interestingly when the difficulty level of the test set was increased relative to training difficulty levels, AdRNNs generalized to these harder instances in a zero-shot manner by allocating more recurrent computation than was ever used during training. They also outperform stability heuristics-based adaptive recurrent networks. Our work empirically shows this hypothesized advantage of using adaptive recurrent processing on static-image tasks for the first time to the best of our knowledge.

In addition to adaptively scaling computation according to the need, the AdRNNs can also solve the problems more efficiently, choosing to stop at earlier times than the non-adaptively trained RNNs (compare halting times in Fig 3 with performance curves in Fig 4) with human arbitrarily chosen training iterations. Thus, just as computation can be scaled to unseen problem difficulties without human intervention, the training iterations required can be (better) discovered automatically.

7 Limitations and Future Work

The current work has been only applied to static images, but recurrent networks are able to process time-varying input; future work will explore our models on video input. In addition, there appears to be a benefit of the LocRNN model over ConvGRU that deserves further study to determine the mechanism behind the benefit of the interneuron population.

For future work, we would like to further explore the benefits of training only on very small problem instances with the ACT mechanism automatically allowing for harder instances. This could greatly reduce latency and energy consumption both during training and inference, an increasingly relevant concern as the size, memory footprint, training time and inference latency of models grow drastically. Finally, we wish to explore how this approach might link to the human developmental literature, similar to earlier work by Elman (1993).

8 Acknowledgments & Funding disclosure

We thank Garrison W. Cottrell, Michael C. Mozer, Michael L. Iuzzolino, Pradeep Shenoy and Saurabh Singh for their feedback and insights on this work. This work is funded by: (1) NSF CRCNS Award No: 2208362, (2) Sony Faculty Innovation Award, and (3) Kavli Symposium Inspired Proposal and support from Social Sciences Computing Facility and Department of Cognitive Science at UC San Diego.

References

L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Ballas, L. Yao, C. Pal, and A. Courville. Delving deeper into convolutional networks for learning video representations. arXiv preprint arXiv:1511.06432 , 2015.

Banino, J. Balaguer, and C. Blundell. Pondernet: Learning to ponder. arXiv preprint arXiv:2107.05407 , 2021.

Bansal, A. Schwarzschild, E. Borgnia, Z. Emam, F. Huang, M. Goldblum, and T. Goldstein. Endto-end algorithm synthesis with recurrent networks: Logical extrapolation without overthinking. arXiv preprint arXiv:2202.05826 , 2022.

Eigen, J. Rolfe, R. Fergus, and Y. LeCun. Understanding deep architectures using a recursive convolutional network. arXiv preprint arXiv:1312.1847 , 2013.

L. Elman. Learning and development in neural networks: The importance of starting small. Cognition , 48(1):71-99, 1993.

Eyzaguirre and A. Soto. Differentiable adaptive computation time for visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12817-12825, 2020.

Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983 , 2016.

Iuzzolino, M. C. Mozer, and S. Bengio. Improving anytime prediction with parallel cascaded networks and a temporal-difference loss. Advances in Neural Information Processing Systems , 34: 27631-27644, 2021.

Jolicoeur and M. Ingleton. Size invariance in curve tracing. Memory & Cognition , 19(1):21-36, 1991.

Jolicoeur, S. Ullman, and M. Mackay. Curve tracing: A possible basic operation in the perception of spatial relations. Memory & Cognition , 14(2):129-140, 1986.

Kim, M. Ricci, and T. Serre. Not-so-clevr: learning same-different relations strains feedforward neural networks. Interface focus , 8(4):20180011, 2018.

Li, V. Piëch, and C. D. Gilbert. Contour saliency in primary visual cortex. Neuron , 50(6):951-962, 2006.

Li. A neural model of contour integration in the primary visual cortex. Neural computation , 10(4): 903-940, 1998.

Liao and T. Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint arXiv:1604.03640 , 2016.

Linsley, J. Kim, V. Veerabadran, C. Windolf, and T. Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. In Advances in Neural Information Processing Systems , pages 152-164, 2018.

Linsley, A. Karkada Ashok, L. N. Govindarajan, R. Liu, and T. Serre. Stable and expressive recurrent vision models. Advances in Neural Information Processing Systems , 33:10456-10467, 2020.

Miller and M. Hardt. Stable recurrent models. arXiv preprint arXiv:1805.10369 , 2018.

Nayebi, J. Sagastuy-Brena, D. M. Bear, K. Kar, J. Kubilius, S. Ganguli, D. Sussillo, J. J. DiCarlo, and D. L. Yamins. Recurrent connections in the primate ventral visual stream mediate a tradeoff between task performance and network size during core object recognition. bioRxiv , pages 2021-02, 2022.

Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013.

Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.

Pinheiro and R. Collobert. Recurrent convolutional neural networks for scene labeling. In International conference on machine learning , pages 82-90. PMLR, 2014.

R. Roelfsema. CORTICAL ALGORITHMS FOR PERCEPTUAL GROUPING. Annual Review of Neuroscience , 29(1):203-227, 7 2006. ISSN 0147-006X. doi: 10.1146/annurev.neuro. 29.051605.112939. URL http://www.annualreviews.org/doi/10.1146/annurev.neuro. 29.051605.112939 .

R. Roelfsema, V. A. Lamme, and H. Spekreijse. The implementation of visual routines. Vision research , 40(10-12):1385-1411, 2000.

Schwarzschild, E. Borgnia, A. Gupta, F. Huang, U. Vishkin, M. Goldblum, and T. Goldstein. Can you learn an algorithm? generalizing from easy to hard problems with recurrent networks. Advances in Neural Information Processing Systems , 34:6695-6706, 2021.

Schwarzschild, A. Gupta, M. Goldblum, and T. Goldstein. The uncanny similarity of recurrence and depth. 10th International Conference on Learning Representations, ICLR 2022 , 2022.

J. Spoerer, T. C. Kietzmann, J. Mehrer, I. Charest, and N. Kriegeskorte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS computational biology , 16(10):e1008215, 2020.

Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006 , 2020.

Ullman. Visual routines. Cognition , 18(1-3):97-159, 12 1984. ISSN 00100277. doi: 10.1016/ 0010-0277(84)90023-4.

Veerabadran and V. R. de Sa. Learning compact generalizable neural representations supporting perceptual grouping. arXiv preprint arXiv:2006.11716 , 2020.

Supplementary

A Training and implementation details

All architectures we evaluated within a task were matched in terms of the number of parameters. The input convolutional layer's kernel size is 7 × 7. Number of channels used by the model remains unchanged across layers and is determined per-model for matching overall number of trainable parameters across models. For PathFinder, we fix the number of channels to be 32 for LocRNN, 21 for hConvGRU and ConvGRU, and 64 for ResNet-30 with a filter size of 9x9 in the intermediate recurrent layers. For Mazes, we fix the number of kernels (d) to be 128 for LocRNN, ConvGRU, hConvGRU, and 100 for ResNet-30 & R-ResNet-30 and the kernel size is fixed to be 5x5. Our readout layers for classification (PathFinder) contain a global average pooling layer followed by a fully-connected layer with output dimensionality of 1 producing the classification logit. We apply binary cross-entropy loss on the logit to train models on PathFinder. On Mazes, we use a 1 × 1 convolution with 1 output channel to produce a binary segmentation map. We use pixel-wise binary cross-entropy to train models on Mazes. These readout layers are used uniformly for all architectures evaluated.

On Mazes training minibatch size is set to 64 images (and inference batch size of 50 images) and a learning rate schedule starting with warmup followed by step learning rate decay as indicated in Schwarzschild et al. (2021) for 50 total epochs of training. On PathFinder, we set the training minibatch size to 256 images and a constant learning rate of 1e-4 for all models for a total of 20 epochs of training. All models were trained on NVIDIA RTX A6000 GPUs and implemented using PyTorch.(Paszke et al., 2017).

B Instability of other baseline ConvRNNs

ConvRNN training is often faced with instability issues that lead to sensitivity with respect to random seeds or lack of convergence of models on downstream tasks. We tested a suite of ConvRNNs previously introduced that are similar to LocRNN on three difficulty levels of PathFinder (in-difficulty evaluation, i.e., training and testing on each difficulty level independently). This evaluation highlighted the above issue especially on the difficult levels of PathFinder where LocRNN was the only model which could converge to stable solutions across different random seeds unlike the other networks which performed at chance as shown below in Fig. 6.

PathFinder-9

PathFinder-14

PathFinder-18

100

Validation accuracy (%)

75

50

25

0

hConvGRU

CORNet-S

ConvRNN

C Input and output format for PathFinder and Mazes

Each example maze is an n × n RGB matrix, with colored squares indicating the start (green) and end (red) positions in the maze. The output is a binary matrix of size n × n with the segmented path indicating the maze solution. An example is shown in Figure 7 (top).

Each PathFinder example is an n × n binary matrix as shown in Figure 7 (bottom). The output for one sample is a pair of probabilities denoting which class the sample belongs to (negative, meaning

the disks are at the end of disconnected paths, or positive, meaning the disks are connected through the contour).

Model output

Input image

(Maze solution

(Large mazes)

segmentation)

Input images

(PathFinder-14)

Model output

= 1

(Binary classification

scores)

[[0.01,0.99],

[0.82, 0.18]]

II

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a combination of Convolutional Recurrent Neural Networks (ConvRNNs) with a learnable termination mechanism from Adaptive Recurrent Neural Networks (AdRNNs), with the purpose of solving complex, variable-difficulty vision tasks. The paper adapts to the purpose a ConvGRU architecture, or alternatively a “LocRNN” architecture based on a computational model of iterative contour processing in primate vision. Both architectures, together with a ResNet baseline, are tested on the “Pathfinder” and “Mazes” tasks, with a variety of task difficulties. Importantly, the authors show that the AdRNNs architecture are able to generalize to task difficulties greater than those seen during training, requiring a number of computation steps beyond those used in training.\n\nSTRENGTHS:\nThe paper consists of an interesting and timely investigation into the hot topic of adaptive computation for problem solving.\n\nWhile the authors are not the first to work on this, they appear to be the first to have demonstrated that RNN models can be effectively applied for vision tasks with adaptive difficulty, and generalize to unseen difficulty levels.\n\nThe work is clearly motivated, and the overall story and contribution is immediately apparent.\n\nWEAKNESSES:\nI believe that the following points need to be addressed for my assessment of the paper to be changed more positively:\n* **[a]** Overall, the main contribution of the paper is the introduction of LocRNN, as all results, including the one on generalization to unseen difficulty levels for tasks, are downstream of LocRNN being a better architecture than alternatives. However, Section 4.3 appears crowded and hard to parse. I believe that the inclusion of figures illustrating the architectures and describing the differences between ConvRNN and LocRNN would greatly improve the presentation. Moreover, not much effort is spent intuitively justifying the reasons behind the improved performance of LocRNN over other ConvRNNs.\n* **[b]** In Line 197, it is mentioned that the computational models of cortical recurrent processing are described in detail in Supp. Info, but there is no such section in the supplementary material. Please include this section, to again better explain the intuition behind the introduction of LocRNN.\n* **[c]** There are small formatting issues. In Line 40, citation names are used without brackets even when they don’t naturally flow within the sentence.\n\nQUESTIONS:\nWhat is the computational model of cortical processing relevant to the introduction of LocRNN? What is the intuition behind LocRNN’s improved performance and stability over other ConvRNNs?\n\nLIMITATIONS:\nThe authors seem to have fairly acknowledged the limitations of their work in their “Limitations” section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors combine Adaptive computation time (ACT) with convolutional recurrent neural networks to solve two tasks with which their generalization properties can be studied. These adaptive-timestep RNNs were found to halt quicker for easier problems while taking more steps for harder ones. This also leads to generalization to an unseen difficulty level.\n\nSTRENGTHS:\nThe paper is well written and the results are convincing for the claim. \n* Pathfinder task is known to be challenging (Tay et al., 2020) so high accuracy and generalization on that task is very interesting. \n* Adaptive choosing of timesteps within a dataset is important & valuable for neuroscience and cognitive science research since it allows for identification of harder and easier stimuli. \n* Inference-time timesteps being more than training is a strong indicator that the RNNs are generalizing.\n\nWEAKNESSES:\nWhile I found the work interesting, I found the paper to be severely lacking in terms of novelty. The paper also misses some related works ([1], [2]). \n \n*  As mentioned in the paper, Bansal et al., (2022) have studied this with the maze task. Authors say \"recurrent networks used in their study are not adaptive, human intervention is required to specify the number of recurrent computational steps by brute force during the testing phase\" which I do not agree with - since their RNN stops changing its response after a while, that point can be used as to decide when to stop. \n* [2] has studied this with Pathfinder task - even finding generalization to difficult levels. \n\nBetween Graves et al. (2016), Bansal et al. (2022) and [2] I do not see the novelty of this work. \n\n\n\n\n\n\n1. Spoerer CJ, Kietzmann TC, Mehrer J, Charest I, Kriegeskorte N (2020) Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLOS Computational Biology 16(10): e1008215. https://doi.org/10.1371/journal.pcbi.1008215\n2. Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, Thomas Serre (2020) Stable and expressive recurrent vision models. NeurIPS 2020\n\nQUESTIONS:\n1. I am curious to hear why the authors say RNNs used in Bansal et al., (2022) \"are not adaptive, human intervention is required to specify the number of recurrent computational steps by brute force during the testing phase\" since the RNN can be run till response stops changing. \n\n2. What properties of the images makes the RNNs stop early vs late? For example, in Fig 4 some 19x19 mazes seem to need the timesteps as 25x25 mazes. Have you seen what causes this? Same for the other task. Can this lead to some interpretation/understanding of the RNNs and the solution they are implementing?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript presents adaptive recurrent networks for processing of static images. The proposed approach augments convolutional recurrent neural networks with the adaptive computation time mechanism, in which the RNN at each step computes an additional halting unit, the value of which is used to determine when to stop the iterative computation. This is shown to yield recurrent vision models that adapt their computational budget to scale with the difficulty level of tasks that require serial grouping operations, such as determining whether two points are located on opposite ends of the same path or segmenting the solution route of a maze. The experimental section also shows that adaptive recurrent networks with gating mechanisms successfully scale to solve the tasks at higher difficulty levels than seen during training.\n\nSTRENGTHS:\nThe proposed approach is novel and well-motivated. The experimental results confirm that the proposed models are able to extrapolate the learned knowledge beyond the training distribution to solve more difficult versions of the two tasks. This is quite impressive and I'm not aware of similar studies.\n\nWEAKNESSES:\nThe write-up is a bit difficult to follow:\n\n1. It seems to me that some important details of the original ACT [1] halting mechanism are omitted in this manuscript or the proposed mechanism is a modified version, which should be clearly stated. I might be mistaken and only the formulation or notation is different. If so, please let me know.\n2. Both $p_t$ and $P_{t'}$ are not really probabilities, as I understand it. You could maybe simply call $P_{t'}$ a quantity that, if it reaches a predefined threshold, halts computation. \n3. $\\phi$ in Equation 4 is not defined\n4. It should be stated right after the first occurrence that LN stands for layer normalization.\n\nI think some details for reproducibility are missing or unclear:\n\n5. The sentence in line 188 states that *four* choices are explored, then in 191 it is stated that *three* different recurrent implementations are explored. As far as I understood, the first of the four choices is a non-recurrent ResNet-30. It is not clearly explained how this model is implemented as Equation 2 only covers recurrent architectures.\n6. The exact architectures including hyperparameters (e.g. number of kernels $d$ and their sizes) of the recurrent models are not provided.\n7. Why is R-ResNet-30 equipped with ACT in Table 1, but not in Table2?\n\nI think the experimental setup should take the following into consideration:\n\n8. For completeness, I think is would make sense to add a few more non-recurrent baselines. It seems intuitive that the considered tasks require multi-step reasoning, but it'd be important to show this empirically, e.g. by using a small vision transformer. I would be surprised if a transformer would completely fail on the task, given the fairly large number of training samples. I can, however, imagine that the proposed architecture performs much better on unseen difficulty levels. Showing this empirically would help in establishing AdRNNs as an important alternatives to current vision models.\n9. Accuracies are not reported as mean and standard deviation over multiple runs with different random seeds. This is very problematic, since with some tasks and architectures, single runs may underperform or not converge to any meaningful solution at all, especially with binary success measures (such as exactly correct maze segmentation). Aggregating metrics over several runs (e.g. 5) would help the reader interpret how significant a different in performance really is (e.g. is it more than 1 standard deviation?)\n\nRegarding references:\n\n10. I think the correct citation for *convolutional* GRUs would be [2]\n\n## References\n1. Graves, Alex. \"Adaptive computation time for recurrent neural networks.\" arXiv preprint arXiv:1603.08983 (2016).\n2. Ballas, Nicolas, et al. \"Delving deeper into convolutional networks for learning video representations.\" arXiv preprint arXiv:1511.06432 (2015).\n\nQUESTIONS:\nOverall, I think the proposed method is novel and has potential for impact, but given the issues mentioned in the weaknesses section, some more work is necessary for publication. I will consider to increase the score significantly should the issues be addressed during the author response period.\n\nI have some minor comments regarding the write-up:\nLine 63: starts upper-case after \"and\" in the previous item\nLine 100 and 105: Shouldn't it be \"4)\" and \"5)\" instead of \"2)\" and \"3)\"? Either way, it is a bit confusing.\nEquation 2: I'd explicitly state that $\\mathbf{r}$ is the recurrent block and how $h_{-1}$ is defined. Also, I think it'd be good to explicitly state what $t$ and $t_\\text{train}$ correspond to.\nLine 197: \"in Supp. Info\" should maybe be \"in the Appendix\" or \"in the supplementary material\"\nLine 266: \"While ... whereas\" (\"while\" or \"whereas\" should be removed)\nLine 340-343: I'd avoid nesting parentheses.\n\n## Acknowledgement of rebuttals\nI have read the rebuttal and provided follow-up questions. The rebuttal addressed my concerns, except the addition of a strong non-recurrent baseline. I have accordingly increased the score slightly.\n\nLIMITATIONS:\nThe authors state some limitations of the approach and propose directions for future research.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe primary subject of this paper is fostering computational efficiency in RNNs solving vision tasks by flexibly adapting the number of computing steps depending on the difficulty of the input. This is achieved through the addition of ACT (Adaptive Computation Time introduced by Graves, 2016) to recurrent vision models. The paper evaluates 4 models (3 with recurrence) on 2 visual reasoning tasks popular in the literature for studying RNN behavior (PathFinder and Mazes). One model is newly introduced here and given the name LocRNN. The key result put forward in the paper is that ACT-trained RNNs will indeed adaptively carry out more time steps before halting computation for difficult inputs (longer paths or larger mazes) than for easy inputs. Finally, another finding is that ACT-trained RNNs generalize to more difficult task conditions than those encountered during training.\n\nSTRENGTHS:\nThe paper touches upon a significant and topical challenge in deep learning: how to avoid ‘wasting’ computations on easy inputs while still solving the hard ones to achieve computational efficiency. Efforts in that direction are valuable from an ecological perspective, among other perspectives.\n\nAlthough the work doesn’t address this directly, mapping out which inputs are hard versus easy for a model can help acquire insights into the strategies a model has learned to solve a (visual) task. It is also of relevance for cognitive computational neuroscience and human-modal comparisons.\n\nThe paper reads very smoothly. The questions are formulated clearly, and so are the methods. It is overall well written.\n\nWEAKNESSES:\nIt is good to see the prediction confirmed. Still, the finding that ACT will flexibly halt computation early for easier inputs does not tremendously advance the field, considering it’s what Graves (2016) already reported. With an ACT-like method (“DACT”) Eyzaguirre and Soto (2020) have already found this in visual reasoning tasks too. It’s worth a citation.\n\nSimilarly, there have been other thorough demonstrations that vision RNNs can generalize to more difficult task conditions, even on the very same Pathfinder task. I encourage the authors to check out and cite Linsley et al. (2020) \"Stable and expressive recurrent vision models\". In particular, Fig. 3 in this current paper is explored in detail and analyzed in the Linsley (2020) article. \n\nThe introduction of LocRNN then becomes the most novel aspect of the work, but there is not a lot of in-depth discussion in the main text about how it differs from prior models informed by neuroscience. At the very least, one would require a thorough comparison between the LocRNN and the model presented in Linsley et al. (2018) regarding their performance metrics and architectural similarities. \n\nFinally, can the authors comment on whether this approach would work in other scenarios of visual difficulty apart from simply spatial extent? The notion of visual complexity involves a host of other factors which a non-hierarchical model would fail to account for, and which needs discussion. \n\nTypos and minor fixes:\n\nSome of the citations appear to be in the wrong format (e.g., L40)\n\nL. 322: Fig.4 → Fig. 3?\n\nFig. 3: in a revised version, probably good to discuss these results in the main text more, specifically with regards to the non-ACT controls.\n\nL 328: Fig. 4 has no panel c.\n\nQUESTIONS:\nPlease refer to the weaknesses section above.\n\nLIMITATIONS:\nPlease refer to the weaknesses section above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors show that using recurrent networks for adaptively processing static inputs for a variable number of iterations allows for zero-shot generalization to more difficult problems by simply unrolling the model predictions for more time steps at inference. Authors propose LocRNN and show model performance on the pathfinder and maze datasets, highlighting that their model uses a greater number of iterations for harder problems and can generalize to harder problems than those seen during training.\n\nSTRENGTHS:\nClarity of Writing. This was by far my favorite paper to read of my reviewing batch because of the clarity of writing. I was able to clearly understand the premise and motivation and follow along with the experiment setup and evaluation.  The insight that recurrent networks can be used to scale up or down the amount of compute depending on the difficulty of the problem is well articulated.\nCompelling Results. Authors show that LocRNN beats competitive baselines on two standardized benchmarks. Impressively, Table 2 shows that LocRNN beats other baselines on extrapolation datasets by a wide margin.\n\nWEAKNESSES:\nMotivation. Although I agree with the premise of the paper, I want to push back on using the biological inspiration of variable compute for neural networks. Although humans naturally can scale to harder and easier problems, state-of-the-art vision networks do not do this. Although authors cite efficiency as a reason for favoring RNNs for variable compute problems, no experimental evidence is provided.\n\nLimited Baselines. Similarly, although the focus of this paper is on adaptive compute, two strong baselines would be LocRNN trained with only one computational step and LocRNN trained with N computational steps (where N is some fixed number of maximum steps allowed). Given the significant engineering that went into the network architecture of LocRNN, the ResNet-30 baseline does not seem convincing to show the necessity of adaptive compute. \n\nDatasets. Although authors evaluate on two standard benchmarks, I would argue that these toy datasets do not represent real tasks. It would be more convincing to compete LocRNN against a state-of-the-art model on a more traditional computer vision benchmark (e.g. ImageNet, COCO, etc.) One place I think this approach could shine is in object detection. Given an image with a single near-field object, we should not need to spend a lot of compute. Similarly, given an image with many far-field small objects, we should likely spend more compute.\n\nQUESTIONS:\nIntermediate Predictions. How do we know that the iteration that the model stops at is optimal? Can we evaluate the model prediction at intermediate steps and evaluate the model prediction beyond when it naturally stops?\nArchitecture Diagram (Sec. 4.3). Providing an architecture diagram for LocRNN would help improve the clarity of Section 4.3. Please add this to the appendix.\n\nLIMITATIONS:\nAuthors highlight that their work explore this problem of adaptive compute on static images and will work on videos in future work. However, I would encourage authors to explore more complex problems within static image domains (e.g. classification, detection, segmentation of natural images instead of toy-like environments).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}