Title: Computing a human-like reaction time metric from stable recurrent vision models
Venue: NeurIPS 2023
Accepted: yes
Keywords: alignment

RNNs

reaction times

equilibrium dynamics

perceptual grouping

decision making
Abstract: The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to ``align” these models with humans. One dimension of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decision-making has been challenging. Here, we sketch a general-purpose methodology to construct computational accounts of reaction times from a stimulus-computable, task-optimized model. Specifically, we introduce a novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent vision models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision-making tasks spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: https://serre-lab.github.io/rnn_rts_site/.

Computing a human-like reaction time metric from stable recurrent vision models

Abstract

The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to 'align' these models with humans. One dimension of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decisionmaking has been challenging. Here, we sketch a general-purpose methodology to construct computational accounts of reaction times from a stimulus-computable, task-optimized model. Specifically, we introduce a novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent vision models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision-making tasks spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: https://serre-lab.github.io/rnn_rts_site/ .

1 Introduction

The symbiosis between visual neuroscience and machine learning has created a distinctive and unprecedented phase in the field of vision. On the one hand, formidable advances in machine learning have supported the development of computational models of visual cognition that rival, and in some cases surpass, human-level performance on naturalistic tasks [1-5]. On the other hand, incorporating computational and neurophysiological constraints from biological visual systems has yielded significant benefits for AI models in terms of their performance [6, 7], efficiency [8], and robustness [9-11]. Additionally, performant computational vision models that operate directly on high-dimensional sensory inputs have shown great promise for scientific discovery through the generation of testable hypotheses for neuroscience [12, 13]. At the core of this synergistic loop lies the need for reliably quantifying the degree of 'alignment' between models and the primate visual system, which can subsequently serve as a control knob for closed-loop hypothesis testing.

The notion of 'alignment' is subjective and can be operationalized at varying levels of analysis [14]. One of the most prominent alignment techniques is to quantify the degree to which model feature

representations predict time-averaged firing rates of single neurons across various stimuli [11, 15-17]. In contrast, computational-level alignment analyses typically use axiomatic attribution methods [18] to extract minimal sub-circuits from larger models and probe them to gain mechanistic insights [12]. A third popular approach to alignment is at a higher behavioral level, where techniques focus on understanding the extent to which models and primates utilize similar 'visual strategies'. Top-down attention masks, diagnostic of human observers' visual strategies, are a commonly employed axis of comparison in this type of model alignment [7, 19]. At the same time, other behavioral measures such as object confusion patterns [20, 21] or human-derived pairwise image similarity ratings [22] are also widely considered effective quantities that can form the basis of alignment.

Each of these techniques has a common tenet: their purview is limited to measures of alignment that are global, time-independent summaries of neural or behavioral states. While this may be sufficient for task paradigms primarily involving feedforward processing, a large part of visual cognition is highly dynamic [23-25]. Understanding visual computations, thus, necessarily involves characterizing and modeling the time-varying properties of neural responses to complex and varied inputs. Why, then, is there a dearth of alignment methods focusing directly on the level of visual processing dynamics? We posit a two-fold bottleneck. First, there is a need for more performant models that exhibit and mimic visual cortical dynamics at scale. Second, we lack the mathematical tools to extract meaningful dynamical signatures from such models and compare them with biological systems. While recent progress in biologically-inspired machine learning has started to open up solutions for the former [8, 19, 26-28], the latter remains a wide-open research area.

Contributions. In this study, we consider this temporal alignment perspective. Recent advances in training biologically motivated large-scale convolutional recurrent neural network models (cRNNs henceforth) on visual cognition tasks [8] have provided candidate models to study the temporal alignment problem. A cRNN's inherent design affords access to its time-varying dynamical properties that can be explicitly analyzed. Throughout this paper, we specifically focus on extracting a metric that captures a cRNN's dynamical signature and on quantifying the ability of this metric to explain primate decision time courses in the form of reaction times (RTs) in concomitant visual psychophysics tasks.

We introduce a novel computational framework to train, analyze, and interpret the behavior of cRNNs on visual cognitive tasks of choice. Our framework triangulates insights from an attractor dynamics-based training routine [8] and evidential learning theory [29] to support stable and expressive evidence accumulation strategies.

We derive a stimulus-computable, task- and model-agnostic metric to characterize evidence accumulation in cRNNs. Our metric does not require extra supervision and purely leverages internal cRNN activity dynamics.

We comprehensively demonstrate the efficacy of our metric in capturing stimulus-dependent primate decision time courses in the form of reaction times (RTs) in four disparate visual cognitive challenges that include incremental grouping, mental simulation, and scene categorization. To the best of our knowledge, this is the first demonstration of qualitative temporal alignments between models and primates across task paradigms.

2 Related Work

Cognitive process models. The speed-accuracy tradeoff in human behavior is ubiquitous in cognitive science [31] and has garnered much attention from the modeling perspective. The most prominent of them is the Drift-Diffusion Model (DDM) family, which specifies RTs by determining when the amount of accumulated evidence for a behavioral choice from a noisy process reaches a threshold value [32, 33]. Though such models have had phenomenal successes in aligning neural and behavioral data [34], they are typically not stimulus-computable or rely on handcrafted heuristics [35]. Moreover, such models are usually directly fitted to RT data [35, 36], unlike our approach, where RT is a derived quantity without any additional supervision.

Computing reaction-time proxies from neural network models. Several notable efforts have attempted to derive RT measures from neural network models to serve as a proxy for 'computational cost" [26, 37-42]. Spoerer et al. [26] introduced the notion of energy demand in their objective function that motivated parsimony regarding the number of computations the cRNN executes. Bansal

cRNN with EDL

Low uncertainty

High uncertainty

D(p|

α

)

0

1

p(Yes)

ϵ

1

0

p(No)

cRNN

Readout

h

0

α

t

ϵ

D(p|

α

)

0

1

p(Yes)

1

0

p(No)

b

a

0

T

0

0.1

0.2

0.3

0.4

0.5

0.6

Time (t)

Uncertainty (

ϵ

)

ξ

cRNN

et al. [40] discussed 'algorithmic extrapolation" where their networks solved problems of greater difficulty than those in their training set by performing additional computations. Graves [37] proposed the use of a 'halting unit' whose activation level determined the probability of continuing model computations. Finally, Figurnov et al. [43], motivated by [37], adaptively determined the spatial extent and number of processing layers in a deep convolutional network. In a broader sense, 'conditional computing' approaches aim to learn dropout-like policies [44, 45] to train neural networks that can balance parsimony and performance. While the approaches presented in [37, 43-45] are impressive advances, they neither operate explicitly with cRNNs nor yield RT values for downstream comparisons on visual cognitive tasks. The elegant approach of [26] suffers from the primary drawback that it relies on the backpropagation through time (BPTT) algorithm. BPTT imposes high memory demands, limiting the number of timesteps a cRNN can perform and therefore condemning any derived measure to be coarse. Furthermore, vanilla BPTT forces the dynamics to be constant (i.e., the cRNN arrives at a solution at t = T for any input) making it non-stimulus-adaptive [8].

Probabilistic interpretation of model uncertainty. Decision certainty and computational parsimony are duals of each other. In other words, easier tasks require fewer computational resources, with models typically being most certain about their predictions, while the opposite holds for harder tasks. Modeling decision uncertainties provide another effective angle to capture the 'computational demand' of a stimulus condition. While deterministic neural networks suffer from ill-calibrated prediction confidence in Softmax outputs, Bayesian neural networks derive prediction confidence from their weight uncertainties but are generally not performant. Inspired by the Dempster-Shafer theory of evidence [29], Sensoy et al. [30] propose a formulation that treats model outputs as parameters of a Dirichlet distribution that signifies the total amount of evidence accumulated for each class. Although influential, this body of work, called Evidential Deep Learning (EDL), has never been linked to RT measures or applied to visual cognitive tasks. We refer the interested reader to Ulmer et al. [46] for a comprehensive survey of this literature. In this paper, we specifically leverage EDL in conjunction with attractor dynamics to support the derivation of an RT proxy.

3 General Methods

3.1 Models

cRNN. The model of interest chosen in this work is the horizontal gated recurrent unit (hGRU; [19]), a convolutional recurrent neural network model that we will canonically refer to as cRNN throughout the rest of the paper. In principle, our framework applies to other cRNN architectures as well. The hGRU is inspired by the anatomical and neurophysiological properties of the primate

a

TASK

b

Are the dots on the same object?

4

t=small

t→

PC 2 (5%)

t=T

2

0

2

t=T

4

t→∞

No

Yes

-10

0

10

PC 1 (83%)

C

Humans

Model

EcRNN

780

(*)

n.s.

20

(sw)

Mean EcRNN

25

760

15

Fix

20

D

B

RT

740

*米

Mean R

10

15

D

720

***

10

5

700

5

0

0

ABCD

A

B（

C

D

visual cortex. Architectural facets of the hGRU include an hypercolumnar organization, distinct excitatory and inhibitory subpopulations, and local- and long-range feedback mechanisms, rendering it overcomplete and an ideal candidate cRNN for our purposes. The hGRU and its extensions have previously demonstrated proficiency in learning long-range spatial dependency challenges, including naturalistic ones [8, 19, 48]. For implementational details, refer to SI §A.

Control models. To further motivate our cRNN choice, we refer to its ability to systematically generalize to more challenging task settings after being trained on less challenging ones [8]. Inference-time flexibility and performance are critical desiderata, given that our goal is to probe the computational demands of a task via stimulus manipulations. We first demonstrate that the cRNN's performance remained well above chance on exceedingly challenging versions of the first task we consider (see Section 4.1) after training on a simplified version (see SI §B for full experimental details). We trained pure feedforward convolutional neural networks (CNNs) as control models under identical training

data distributions. The CNNs struggled to generalize effectively, as did a naive control RNN model (Fig. S1). Because CNNs lack explicit temporal dynamics and our recurrent control models failed our generalization test, we limit the experimental results presented in this paper to our cRNN. For completeness, we remark that a Vision Transformer [49] passed the generalization test. However, we exclude these models from our analyses for two reasons. First, as such, Vision Transformers lack biological analogs. Second, our focus in this work is to derive RT measures from model dynamics, which transformers do not exhibit.

3.2 Training Framework

General remarks. All model runs reported herein were done from scratch using only the ground truth class labels for supervision. Any alignment between the cRNN model and humans would have had to emerge as a byproduct of task constraints. We emphasize that no human RT data were involved in our training routines. For the cRNNs, we computed a task loss only at the network's end state to preserve the ability to remain flexible during inference. The cRNNs were trained in a data-parallel manner on 4 Nvidia RTX GPUs (Titan/3090/A5000) with 24GB of memory each. Each training run took approximately 48 hours, except those in Section 7 (2 hours), which were faster due to the smaller training dataset size. They were furthermore trained with Adam [50], a learning rate of 1 e -3 , and γ = 100 for the C-RBP penalty ([8]; also see SI §A). We use T = 40 recurrent timesteps during the training phase, with an exception for the task in Section 6, where T = 80 and γ = 1000 .

Stabilizing cRNNs. We train our cRNN model with the contractor recurrent back-propagation algorithm (C-RBP; [8]). C-RBP imposes an attractor-dynamic constraint on the cRNN's computational trajectory, imparting several benefits over the standard BPTT algorithm. C-RBP is significantly more memory efficient and affords arbitrary precision in the cRNN temporal dynamics (i.e., T can be high). C-RBP also allows the cRNN to learn stable and expressive solutions. We can observe this from the cRNN's latent trajectories, as visualized in Fig. 2b, where extrapolation in time (beyond T ) is non-catastrophic. We argue that model instability is why BPTT-trained controls failed the generalization test (Fig. S1). Furthermore, stable solutions imparted by C-RBP allow our cRNNs to dynamically adapt their number of processing steps in a stimulus-dependent manner. That is, we pose that a C-RBP-trained cRNN can utilize its computational budget differently depending on the input (see Fig. S2 for an illustration). Finally, we note that our strategy to achieve stable recurrent solutions is in contrast to that employed by Bansal et al. [40], wherein cRNNs were stabilized through architectural choices along with an incremental training process.

Evidential loss. We augment our cRNN training objective with an evidential deep learning loss (EDL; [30]). In contrast to the classic cross-entropy loss, EDL considers cRNN outputs to be the parameters of a belief distribution rather than point estimates of class probabilities (Fig. 1a). We formalize cRNN beliefs as a Dirichlet distribution. Strong evidence in favor of one class causes the Dirichlet distribution to be peaked, reflecting low uncertainty ( ϵ ). In contrast, when there is a lack of overall evidence, the Dirichlet distribution becomes more uniform, thereby increasing ϵ . For a complete mathematical specification of this loss formulation, refer to SI §A.

3.3 RT Metric ( ξ cRNN )

We derive a novel metric to quantify the differential computational demands imposed on the cRNN by stimulus manipulations and capture the essence of model decision time courses for alignment. Modeling uncertainty ( ϵ ) explicitly allows us to track its evolution over time from the cRNN (Fig. 1b). We formulate our RT metric ( ξ cRNN ) as the area under this uncertainty curve. Precisely, ξ cRNN = ∫ T 0 ϵ ( t ) dt (Fig. 1b). Intuitively, high (low) ξ cRNN can be understood as the cRNN spending a large (small) amount of time in an uncertain regime. Our metric affords direct comparisons to human RTs. In the following sections, we apply this metric to cRNNs trained on a host of visual cognitive tasks and study their alignment to human reaction time data.

ξ

cRNN

5

10

15

20

25

Fix

Fix

Fix

Fix

4 Incremental Grouping

4.1 Task, Stimuli, and Visualizations

Inspired by an incremental perceptual grouping task in humans [47], we propose an equivalent challenge for the cRNN where the model is required to determine whether two dots are on the same object (Fig. 2a). The stimuli consist of white outlines of natural shapes on a black background ( 150 × 150 px). We created 340 K ( 15 K ) training (validation) outline stimuli from MS COCO images [51]. The dots were placed semi-randomly such that there was no difference in the distribution of dot distances between the two stimulus classes.

We additionally introduce two novel types of visualizations to probe cRNN visual strategies. First, we track cRNN dynamics by looking at its latent activity h t at every time step t , averaged across the channel dimension and smoothed by averaging h t and h t + 1 . To facilitate interpretation, we provide these videos for the reader (see SI §D). Second, we compute a 'spatial uncertainty' (SU) map for each outline stimulus by keeping one dot fixed and varying the position of the other. Each spatial coordinate has a value equal to ξ cRNN for that corresponding dot configuration. Examples are shown in Fig. 2c (right) and Fig. 3.

4.2 Results

Despite its lightweight architecture, our cRNN solved the incremental grouping task, achieving 98% validation accuracy. Visualizing the cRNN latent activities revealed that the model learned a cognitively-viable filling-in strategy. Network activity originated at each of the dot locations and gradually spread, eventually filling up the whole object(s) (Fig. 2b). We note that this behavior is an emergent property from purely the classification task constraint instead of direct supervision for any form of segmentation. We then sought to ask if ξ cRNN effectively captures these visual decision-making dynamics. In the following sections, we report results detailing the relationship between ξ cRNN and a host of complex, topological stimulus properties.

ξ cRNN captures the serial spread of attention in our models. Human visual strategies in solving the incremental grouping task are thought to involve a serial spread of attentional resources, resulting in longer RTs for higher distances [47]. To test this effect in our model, we took a subset of our validation outlines and repositioned the dots to be in the center of an object, 14 px apart. We then systematically kept moving the dots 14 px more apart until reaching the object boundary. A linear mixed-effects regression analysis revealed a significant positive effect of this manipulation on ξ cRNN , b = 1 . 46 , SE = 0 . 11 , z = 13 . 58 , p < . 001 (Fig. S3, Fig. S4). The SU maps (Fig. 2c right, Fig. 3) exhibit this distance effect too. Placing the second dot further away from the fixed dot (indicated in white) yielded higher ξ cRNN values.

ξ cRNN recapitulates the spatial anisotropy in human RT data. Euclidean distance only paints a partial picture. Human RTs are affected by several other factors, such as topological distance, the proximity of the dots to boundaries, and the need to spread through narrow object regions, all amounting to a well-documented spatial anisotropy in the spread of visual attention [47, 52]. While anisotropy is evident in the SU maps from our cRNN (Fig. 2c right, Fig. 3), we also formally assess the alignment between humans and the cRNN in this regard, using human RT data from [47].

0

10

20

30

0

1

2

3

4

5

6

ξ

cRNN

Human RT (s)

r

= .31,

p

< .001

First, Jeurissen et al. [47] quantified their stimuli on an alternative dot distance measure that went beyond Euclidean distance to reflect the other aforementioned factors and was more predictive of human RT (see SI §F). We found this measure to be a better predictor of ξ cRNN ( r = . 80 ) as well (versus r = . 45 for Euclidean distance), t (63) = 4 . 68 , p < . 001 . Second, the stimulus set featured distinct experimental conditions, varying in expected difficulty based on the position of the dots (Fig. 2c left). As shown in Fig. 2c (middle), the effects of this manipulation on ξ cRNN were qualitatively aligned with those on human RT. A one-way repeated measures ANOVA across 22 outline stimuli revealed a significant effect of 'Condition' on ξ cRNN , F (3 , 63) = 67 . 65 , p < . 001 , as well as on the average RTs from [47], F (3 , 63) = 16 . 95 , p < . 001 . Furthermore, Fig. 2c (middle) presents the results of planned contrasts (paired t-tests). ξ cRNN was significantly higher for B compared to A, Cohen ′ s d = 2 . 00 , t (21) = 9 . 39 , p < . 001 , but also for the more difficult Condition C compared to B, d = 0 . 73 , t (21) = 3 . 42 , p = . 003 . Third, we observed a significant overall correlation between human RT and ξ cRNN on Yes-stimuli, r = . 31 , p = . 01 . Fourth, on a novel test set of our own, paired t-tests showed that ξ cRNN was significantly higher when two dots on the same object were placed in narrow versus wide object regions, d = 0 . 70 , t (22) = 3 . 34 , p = . 003 , and when the path was curved (i.e., longer topological distance) versus when it was not, d = 0 . 98 , t (22) = 4 . 71 , p < . 001 (see SI §G, Fig. S5). Finally, SI §G also describes an experiment in which we varied the distance between one dot and the object boundary while keeping the distance between the dots themselves constant. We found that our model exhibits higher values near the boundaries, which is strikingly similar to human behavioral results presented in [47].

Ambiguous stimuli. As a final way of gaining insight into the cRNN's task behavior, we looked at those stimuli in the validation set that elicited the highest ξ cRNN values. We observed that some were characterized by uncertainty curves that remain high, even at t = T , suggesting the model deems them ambiguous. In a test set we designed to be ambiguous (it features occlusions), we frequently observed such non-decreasing uncertainty curves too (see SI §H). We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.

Taken together, these results suggest that our metric is able to capture the dynamical signatures of object-based attention in our cRNNs and provides a means to quantify alignment with human visual strategies.

5 Visual Simulation Planko

5.1 Task and Stimuli

The Planko task (and corresponding dataset) was introduced in [53] and features stimuli ( 64 × 64 px; 80 K training, 5 K validation) displaying a ball about to fall down and eventually land in one of two baskets ('left'/'right'). The outcome depends on the position and orientation of a set of planks placed in between the ball and the baskets (Fig. 4a). Humans are thought to solve this task through mental

b

c

d

a

Is there a path to

connect the squares?

No

Yes

TASK

0

T

0

1

Predicts "No"

Predicts "Yes"

Time (t)

Uncertainty (

𝜖

)

0

T

0

1

Time (t)

Uncertainty (

𝜖

)

0

20 40 60 80

0

10

20

30

40

Path length

ξ

cRNN

r

= .84,

p

< .001

h

t

→

∞

h

t

→

∞

h

t=small

h

t=small

visual simulation [53]. Notably, feedforward CNNs struggle to solve the task [54] and when they do, they appear to adopt non-aligned strategies to humans [53]. In contrast, cRNNs show evidence of simulation-like behavior on a variant of the task [54]. With our metric, we are now able to directly compare human (collected by [53]) and cRNN RTs. To this end, we trained a cRNN from scratch on Planko (see Section 3.2 and SI §A for details on training).

5.2 Results

The stimuli displayed in Fig. 4 illustrate the challenge of the Planko task. Our cRNN achieves a performance of 84% on the validation set, well within the confidence interval of human accuracy [53]. Stimuli presented to human participants in the original study ( N = 200 ) were used here as a held-out set. We observed a significant correlation between model ξ cRNN values and human RTs, r = . 31 , p < . 001) (Fig. 4b). This result indicates that stimuli that were harder for humans tended to be computationally challenging for the cRNN (see examples in Fig. 4c).

What makes a stimulus challenging? Variance in human RTs is known to be partially explained by the statistics of the planks. For instance, stimuli for which the outcome is more likely to change as a consequence of perturbing the plank configurations by a small amount were deemed 'uncertain'. Indeed, participants responded slower to these stimuli [53]. We were able to replicate this result in our cRNN, r = . 31 , p < . 001 . These results showcase how our methodology can be used to make stimulus-dependent RT predictions. This could help shed light on, for example, the remarkable yet under-examined ability of humans to flexibly adopt different strategies in a Planko task depending on the stimulus demand.

6 Mazes

6.1 Task and Stimuli

We devised a maze task where the goal is to tell whether two cued locations are connected by a path (Fig. 5a). To build our training ( N = 34 . 5 K ) and validation ( N = 14 K ) set, we started from mazes provided by [40] but modified them to make them suitable for classification (the originals did not feature any disconnected paths). The cues were placed such that the two classes ('yes'/'no') did not differ in Euclidean cue distance. Each maze was 144 × 144 px.

6.2 Results

Here too, we observed that the uncertainty curves and derived ξ cRNN values differed considerably depending on the input, even if nearly all mazes were classified correctly at t = T ( 99% correct

20

40

60

80

100

10

12

14

16

18

20

0.35

0.36

0.37

0.38

0.39

Discriminability (%)

ξ

cRNN

Human RT (s)

Natural or manmade?

a

b

TASK

Natural

Manmade

on validation mazes). For the Yes-mazes contained in a random subset of the validation set, we found that the model spent more time being uncertain (higher ξ cRNN ) when the length of the path between the two cued locations was longer, r = . 84 , p < . 001 (Fig. 5b,d). Moreover, path length was a significantly better predictor than simple Euclidean distance ( r = . 39 ), t (234) = 5 . 81 , p < . 001 . Similarly, human RTs on maze tasks have been shown to increase with path length [55]. When the cued locations were on two distinct maze segments (No-mazes), ξ cRNN was significantly predicted by the mean of both segment lengths, r = . 56 , p < . 001 . The kind of visual strategy that produced such distance effects can be understood by inspecting the visualizations of the latent activities (Fig. 5c). The model gradually spread activity along the maze segment(s) containing the cues. We strongly encourage the reader to watch our videos (see SI §I) to better appreciate this spread in conjunction with the evolution of uncertainty over time. Finally, we did not find evidence for an effect on ξ cRNN of the number of T-junctions on the path, which are known to predict human RT [56]. We speculate that this potential misalignment, worth investigating in future work, could be due to the model not being restricted in how many possible paths it can explore at once.

7 Scene Categorization

7.1 Task and Stimuli

To showcase how our general approach can also be applied to study higher-level visual tasks, we consider a natural versus manmade scene categorization task as our final challenge (Fig. 6). Human RT data from a rapid-scene categorization version of this task were already available from [57]. We selected images from the SUN database [58] querying the same classes as in [57], but excluding the images shown to human participants, and performed a stratified train ( N = 8 . 8 K ) and validation ( N = 979 ) split. The image preprocessing pipeline entailed cropping the image to be square, resizing to 150 × 150 px, converting to grayscale, and random horizontal flipping. In addition, the images were matched for overall brightness.

7.2 Results

The model was able to correctly classify 83% of the validation images and 73% of the held-out test images from [57]. The errors can be understood, at least partly, in light of the less-thanclear-cut labels (e.g., an islet with a house on top, a building behind many trees). Each of the test images had previously been assigned a discriminability score [57], based on the average error rate of logistic classifiers trained on 'gist' features [59]. Human participants were reported to be slower with decreasing discriminability [57]. Here, we show a similar trend in the model (Fig. 6b), with a linear regression analysis revealing a significant, negative discriminability slope, b = -0 . 47 , SE = 0 . 03 , t = -16 . 45 , p < . 001 . Directly comparing ξ cRNN to human RT, we found them to be significantly correlated, r = . 19 , p < . 001 . While our cRNN model of choice here does not perfectly account for empirical patterns in human reaction times, specifically in their non-monotonic reaction time readouts for the low-discriminability stimuli, they provide a promising

start. Our framework is model-agnostic (e.g., see SI §J for results on a convLSTM) and thus can be used to test hypotheses on architectural improvements.

8 Discussion

We are in the midst of a paradigm shift in visual neuroscience. The continued development of computational models that rival human behavioral performance has called for tools to quantify how well the internal mechanisms of such models compare to biological truths. These efforts have demonstrably been beneficial for both the AI and neuroscience ecosystems.

In this work, we push this frontier by introducing a framework to characterize the evidence accumulation strategies of stable recurrent vision models. Our experiments thus far show the potential to generate novel hypotheses of interest to visual neuroscientists. The most direct application of our framework for experimental design is in taxonomizing stimuli in terms of the putative computational demand based on our metric toward understanding neural processes at varying timescales. Furthermore, the value proposition of this framework includes the ability to adjudicate between models based on the consistency of their latent dynamics with human reaction time data. For example, our metric from a cRNN trained on the maze task is uncorrelated with the number of T-junctions in the stimulus, indicating a potential divergence between serial and parallel exploration strategies (SI §I). It is worth noting that distinguishing between serial and parallel attention models has been of long-standing interest [60]. Similarly, testing a zoo of hierarchical models on the scene categorization task while keeping an eye on the degree of alignment with our metric can potentially lend novel mechanistic insights and help delineate bottom-up, lateral, and top-down processes [61]. We are also enthusiastic about drawing theoretical parallels between our metric summarizing the cRNN's evidence accumulation strategy and the complexity of natural data statistics toward understanding the sample efficiency of learning in models and primates.

Limitations and an outlook for the future. Within the scope of this work, we have considered four disparate visual cognition tasks to showcase how our approach offers the means to study the temporal alignment between recurrent vision models and the human visual system. Of course, in the face of the myriad of tasks that cognitive scientists have used over centuries, our selection still needs to be improved. Our results do not directly speak to the diversity in response types, stimulus types, or naturalism, to name a few. However, our approach is general-purpose enough to adapt to newer tasks as long as they are cast as a classification problem. We believe this should cover a significant share of cognitive science and AI tasks. Extending the approach beyond choice tasks would require a considerable revision of how to make the model's uncertainty explicit. Another limitation is that there could easily be other cRNNs that align with human RTs better. While we motivate the choice for the cRNN in Section 3.1, our main contribution here is to introduce the overall framework integrating stable recurrent vision models with evidential learning theory to support the derivation of a novel RT metric. A comprehensive comparison of different models, apart from those in SI §B, was hence beyond this work's scope. However, we are confident our framework paves the way to build a taxonomy of models in the future as well as generate testable hypotheses for neuroscience.

Broader impacts. With AI vision models becoming ubiquitous in our daily lives, the call to make them fairer, transparent, and robust is growing louder. Our framework facilitates this by allowing researchers to peek into the inner working of these AI models for transparency and thinking about their match to biological systems, which are arguably more robust. From a neuroscience perspective, computational models of RT are gaining prominence in computational psychiatry and hold the key to understanding many neurobiological disorders [62, 63]. We generally do not anticipate any negative impact on society in either scenario.

Acknowledgments

The authors would like to explicitly thank Jeurissen et al. [47] and Sofer et al. [57] for generously sharing their stimuli and human reaction time data, and Bansal et al. [40] for their maze stimuli. We are also grateful to Drew Linsley for his helpful comments and feedback. This work was supported by ONR (N00014-19-1-2029), and NSF (IIS-1912280). Additional support provided by the Carney

Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the computing hardware resources supported by NIH Office of the Director grant S10OD025181.

References

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM , 60(6):84-90, 2017.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations , 2015.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Grace W Lindsay and Kenneth D Miller. How biological attention mechanisms improve task performance in a large-scale visual system model. ELife , 7:e38105, 2018.

Thomas Fel, Ivan F Rodriguez, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 9432-9446. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/ paper/2022/file/3d681cc4487b97c08e5aa67224dd74f2-Paper-Conference.pdf .

Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 10456-10467. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/766d856ef1a6b02f93d894415e6bfa0e-Paper.pdf .

Aran Nayebi and Surya Ganguli. Biologically inspired protection of deep networks from adversarial attacks. arXiv preprint arXiv:1703.09202 , 2017.

Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems , 33:13073-13087, 2020.

Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Baldwin Geary, Michael Ferguson, David Daniel Cox, and James J. DiCarlo. Aligning model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robustness. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=SMYdcXjJh1q .

Hidenori Tanaka, Aran Nayebi, Niru Maheswaranathan, Lane McIntosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction. Advances in neural information processing systems , 32, 2019.

Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences , 119(43):e2200800119, 2022.

David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT press, 2010.

Daniel L Yamins, Ha Hong, Charles Cadieu, and James J DiCarlo. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque it and human ventral stream. Advances in neural information processing systems , 26, 2013.

Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron , 108(3):413-423, 2020.

Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology , 10(11):e1003915, 2014.

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning , pages 3319-3328. PMLR, 2017.

Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning longrange spatial dependencies with horizontal gated recurrent units. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/ paper/2018/file/ec8956637a99787bd197eacd77acce5e-Paper.pdf .

Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. Journal of Neuroscience , 38(33):7255-7269, 2018.

Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems , 34:23885-23899, 2021.

Joshua C Peterson, Joshua T Abbott, and Thomas L Griffiths. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive science , 42(8):2648-2669, 2018.

Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl J Friston. Canonical microcircuits for predictive coding. Neuron , 76(4):695-711, 2012.

Maxime Cauchoix, Gladys Barragan-Jason, Thomas Serre, and Emmanuel J Barbeau. The neural dynamics of face detection in the wild revealed by mvpa. Journal of Neuroscience , 34(3):846-854, 2014.

Andre Moraes Bastos, Julien Vezoli, Conrado Arturo Bosman, Jan-Mathijs Schoffelen, Robert Oostenveld, Jarrod Robert Dowdall, Peter De Weerd, Henry Kennedy, and Pascal Fries. Visual areas exert feedforward and feedback influences through distinct frequency channels. Neuron , 85(2):390-401, 2015.

Courtney J Spoerer, Tim C Kietzmann, Johannes Mehrer, Ian Charest, and Nikolaus Kriegeskorte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS computational biology , 16(10):e1008215, 2020.

Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel LK Yamins, and James J DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv , page 408385, 2018.

Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical to the ventral stream's execution of core object recognition behavior. Nature Neuroscience , 22(6):974-983, April 2019. doi: 10.1038/s41593-019-0392-5. URL https: //doi.org/10.1038/s41593-019-0392-5 .

Ronald R Yager and Liping Liu. Classic works of the Dempster-Shafer theory of belief functions , volume 219. Springer, 2008.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Richard P Heitz. The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in neuroscience , 8:150, 2014.

Roger Ratcliff and Gail McKoon. The diffusion decision model: theory and data for two-choice decision tasks. Neural computation , 20(4):873-922, 2008.

Alexander Fengler, Lakshmi N Govindarajan, Tony Chen, and Michael J Frank. Likelihood approximation networks (lans) for fast inference of simulation models in cognitive neuroscience. Elife , 10:e65074, 2021.

James F Cavanagh and Michael J Frank. Frontal theta as a mechanism for cognitive control. Trends in cognitive sciences , 18(8):414-421, 2014.

Budmonde Duinkharjav, Praneeth Chakravarthula, Rachel Brown, Anjul Patney, and Qi Sun. Image features influence reaction time. ACM Transactions on Graphics , 41(4):1-15, July 2022. doi: 10.1145/ 3528223.3530055. URL https://doi.org/10.1145/3528223.3530055 .

Amin Mirzaei, Seyed-Mahdi Khaligh-Razavi, Masoud Ghodrati, Sajjad Zabbah, and Reza Ebrahimpour. Predicting the human reaction time based on natural image statistics in a rapid categorization task. Vision Research , 81:36-44, 2013. ISSN 0042-6989. doi: https://doi.org/10.1016/j.visres.2013.02.003. URL https://www.sciencedirect.com/science/article/pii/S0042698913000230 .

Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983 , 2016.

Omkar Kumbhar, Elena Sizikova, Najib J. Majaj, and Denis G. Pelli. Anytime prediction as a model of human reaction time. CoRR , abs/2011.12859, 2020. URL https://arxiv.org/abs/2011.12859 .

Razvan Pascanu, Yujia Li, Oriol Vinyals, Nicolas Heess, Lars Buesing, Sébastien Racanière, David P. Reichert, Theophane Weber, Daan Wierstra, and Peter W. Battaglia. Learning model-based planning from scratch. CoRR , abs/1707.06170, 2017. URL http://arxiv.org/abs/1707.06170 .

Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=PPjSKy40XUB .

Kristopher T. Jensen, Guillaume Hennequin, and Marcelo G. Mattar. A recurrent network model of planning explains hippocampal replay and human behavior. bioRxiv , 2023. doi: 10.1101/2023.01.16.523429. URL https://www.biorxiv.org/content/early/2023/01/19/2023.01.16.523429 .

Joshua I Gold and Michael N Shadlen. The neural basis of decision making. Annu. Rev. Neurosci. , 30: 535-574, 2007.

Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1039-1048, 2017.

Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297 , 2015.

Ludovic Denoyer and Patrick Gallinari. Deep sequential neural network. arXiv preprint arXiv:1410.0510 , 2014.

Dennis Thomas Ulmer, Christian Hardmeier, and Jes Frellsen. Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation. Transactions on Machine Learning Research , 2023.

Danique Jeurissen, Matthew W Self, and Pieter R Roelfsema. Serial grouping of 2d-image regions with object-based attention in humans. eLife , 5:e14320, 2016. ISSN 2050-084X. doi: 10.7554/eLife.14320.

Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi Narasimhan Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-recognition in humans and machines. In Proceedings of the 35th International Conference on Neural Information Processing Systems , 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum? id=YicbFdNTTy .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412. 6980 .

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, The European Conference on Computer Vision (ECCV) 2014 , pages 740-755. Springer International Publishing, 2014. ISBN 978-3-319-10602-1.

Arezoo Pooresmaeili and Pieter R. Roelfsema. A growth-cone model for the spread of object-based attention during contour grouping. Current Biology , 24(24):2869-2877, 2014. doi: 10.1016/j.cub.2014.10.007. URL https://doi.org/10.1016/j.cub.2014.10.007 .

Aarit Ahuja and David L. Sheinberg. Behavioral and oculomotor evidence for visual simulation of object movement. Journal of Vision , 19(6):13, June 2019. doi: 10.1167/19.6.13. URL https://doi.org/10. 1167/19.6.13 .

Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Drew Linsley, David Sheinberg, and Thomas Serre. The emergence of visual simulation in task-optimized recurrent neural networks. In SVRHM 2022 Workshop @ NeurIPS , 2022. URL https://openreview.net/forum?id=qYLp6nNU7m-.

David A. Crowe, Bruno B. Averbeck, Matthew V. Chafee, John H. Anderson, and Apostolos P. Georgopoulos. Mental maze solving. Journal of Cognitive Neuroscience , 12(5):813-827, September 2000. doi: 10.1162/089892900562426. URL https://doi.org/10.1162/089892900562426 .

Florian Kadner, Hannah Willkomm, Inga Ibs, and Constantin A. Rothkopf. Finding your way out: Planning strategies in human maze-solving behavior. February 2023. doi: 10.31234/osf.io/nme9j. URL https://doi.org/10.31234/osf.io/nme9j .

Imri Sofer, Sébastien M. Crouzet, and Thomas Serre. Explaining the timing of natural scene understanding with a computational model of perceptual categorization. PLOS Computational Biology , 11(9):e1004456, September 2015. doi: 10.1371/journal.pcbi.1004456. URL https://doi.org/10.1371/journal. pcbi.1004456 .

Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Largescale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 3485-3492, 2010. doi: 10.1109/CVPR.2010.5539970.

Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision , 42(3):145-175, 2001. doi: 10.1023/a:1011139631724. URL https://doi.org/10.1023/a:1011139631724 .

Rani Moran, Michael Zehetleitner, Heinrich René Liesefeld, Hermann J Müller, and Marius Usher. Serial vs. parallel models of attention in visual search: accounting for benchmark rt-distributions. Psychonomic bulletin & review , 23(5):1300-1315, 2016.

Thomas Serre, Aude Oliva, and Tomaso Poggio. A feedforward architecture accounts for rapid categorization. Proceedings of the national academy of sciences , 104(15):6424-6429, 2007.

Quentin JM Huys, Tiago V Maia, and Michael J Frank. Computational psychiatry as a bridge from neuroscience to clinical applications. Nature neuroscience , 19(3):404-413, 2016.

Peter F Hitchcock, Eiko I Fried, and Michael J Frank. Computational psychiatry needs time and context. Annual review of psychology , 73:243-270, 2022.

Tim Cooijmans, Nicolas Ballas, César Laurent, Ça˘ glar Gülçehre, and Aaron Courville. Recurrent batch normalization. In International Conference on Learning Representations , 2017.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper_files/paper/2012/file/ c399862d3b9d6b76c8436e924a68c45b-Paper.pdf .

Siqi Yang. Pytorch-alexnet. https://github.com/YOUSIKI/PyTorch-AlexNet , 2020.

H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics , 22: 400-407, 1951.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.1556 .

Daniel Falbel. torchvision: Models, Datasets and Transformations for Images , 2023. https://torchvision.mlverse.org, https://github.com/mlverse/torchvision.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun WOO. Convolutional lstm network: A machine learning approach for precipitation nowcasting. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper_files/ paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-Paper.pdf .

Supplementary Information

A Implementation Details

A.1 Model Implementation

We instantiate our cRNN as a canonical recurrent convolutional neural network layer inspired by the primate visual cortex's anatomical and neurophysiological properties. This model, termed hGRU and first introduced in Linsley et al. [19], implements an hypercolumnar organization with distinct excitatory and inhibitory subpopulations and local- and long-range feedback mechanisms. For the sake of completeness, we outline the governing equations of the model below.

First compute suppressive interactions:

Then compute the facilitation terms followed by the recurrent update:

G F = sigmoid ( U F ∗ S ) # Facilitation gate C F = BN ( W F ∗ S ) # Facilitation term ˜ H = [ ν ( C F + S ) + ω ( C F ∗ S ) ] + # Modulation of S through additive and multiplicative facilitation H [ t ] = (1 -G F ) ⊙ H [ t -1] + G F ⊙ ˜ H # State update

In these equations, H , Z ∈ R X × Y × C represent the cRNN's latent state and bottom-up feedforward drive, respectively, with height/width/channels X,Y,C . W S , W F ∈ R E × E × C × C are learnable kernels controlling the suppressive and facilitatory interactions and E is the spatial width of the lateral interactions. For all our experiments, we set E = 5 , except for Planko where E = 3 . Similarly, U S , U F ∈ R 1 × 1 × C × C are learnable kernels controlling the gates. Model dynamics are discretized, with t ∈ [1 ...T ] representing the timesteps of processing. We use softplus as our rectifying nonlinearity (denoted by [ · ] + ). We also use Batch Normalization ( BN ) in the model to control exploding/vanishing gradients [64]. BN kernels are shared across timesteps of processing.

A.2 Training Implementation Details

Leveraging equilibrium dynamics for training. Training cRNNs has traditionally posed a problem due to the severe memory bottlenecks imposed by the backpropagation through time (BPTT) algorithm. To remedy this, Linsley et al. [8] introduced a variant of the classic recurrent backpropagation algorithm (RBP) that leveraged equilibrium dynamics to compute model gradients with O (1) memory complexity. We outline the high-level idea here and refer the reader to [8] for more details.

Our discretized cRNN dynamics can be written as follows,

where f ( . ) is the model architecture described, H [ t ] is the cRNN's state at timestep t , and X is the bottom-up sensory input. For convenience, we use W to denote all the trainable parameters of f ( . ) . If f ( . ) is a contraction mapping, then H [ t ] → H ∗ as t →∞ . One can define an implicit function as follows to aid in the gradient computation: ψ ( H ; X , W ) = H -f ( H , X ; W ) . By construction, we can observe that ψ ( H ∗ ; X , W ) = 0 and by extension,

Following a sequence of algebraic manipulations we can compute the gradient of the equilibrium state H ∗ w.r.t. parameters W as follows.

where J f, H ∗ is the Jacobian of the cRNN f ( . ) evaluated at H ∗ . The ability to compute this gradient, however, relies on ( I -J f, H ∗ ) being invertible. To guarantee this, Linsley et al. [8] introduced the Lipschitz Coefficient Penalty (LCP) as L LCP ( W ) = ∥ ( 1 · J f, H ∗ -λ ) + ∥ 2 which controls the degree of contraction in f ( . ) and by extension the invertibility of the term discussed above. Moreover, this regularization can be added to any task loss which we exploit.

Evidential learning formulation. To support our goal of characterizing evidence accumulation strategies in our cRNN, we parameterize our model readout to explicitly express uncertainty. Specifically, we follow the evidential learning strategy introduced in Sensoy et al. [30] to interpret cRNN readouts as the parameters of the Dirichlet density on the class predictors, α = < α 1 , ..., α K > , where K is the total number of classes in the classification problem of interest. As in [30], our evidential loss comprises of two components, one that supports belief accumulation for the 'correct' class ( L risk ) and a second that ensures that the beliefs for all the other 'wrong' classes have uniform mass ( L balance ).

In the above, y denotes a one-hot ground truth label for a given sample with y j = 1 if the sample belongs to class j , and 0 otherwise. ˆ p j is from the model that denotes the expected probability of the j -th singleton under the corresponding Dirichlet such that ˆ p j = α j S . S = ∑ K j =1 α j resembles the 'total evidence' accumulated by the model (or the Dirichlet strength). D KL is the Kullback-Leibler divergence measure. ˆ α are the Dirichlet parameters of just the 'misleading' evidence given by ˆ α = y +(1 -y ) α . We define the instantaneous uncertainty as ϵ = K S .

Finally, ρ is an annealing term that is used to balance the relative importance of L risk and L balance . In practice, we use the following increment schedule: ρ epoch = min [ 1 , epoch τ ] , where τ is a hyperparameter. For a complete derivation of the loss formula, we refer the reader to [30]. We choose τ = 16 for incremental grouping, τ = 10 for Planko, τ = 20 for the maze task, and τ = 16 for scene categorization.

Training objective. Our final training objective is thus L ( W ) = L EDL ( W ) + γ L LCP ( W ) .

B Generalization Experiments

B.1 Task and Stimuli

In the main text, we identified the flexibility to generalize to more complex task settings at inference as a critical desideratum for obtaining a human-like RT metric from any model. We tested generalizability in the cRNN as well as in a set of control models (see further) using the incremental grouping task described in the main text (dots on the same object 'yes'/'no'?). We trained on an easy version of the task by only selecting training stimuli for which the Euclidean dot distance was at most 44 px (50 th percentile). We refer to this reduced training set as 'short training stimuli' ( N = 170 K ). For evaluation, we created two subsets of the validation set, one easy and one hard. The easy set ('short validation stimuli', N = 7 . 7 K ) consisted of those validation stimuli for which the dots were at most 44 px apart. The hard set ('long validation stimuli', N = 1 . 5 K ) consisted of those stimuli for which the dots were more than 82 px apart (90 th percentile). All thresholds were computed across the training stimuli in the original set.

B.2 Models and Training

All models listed below were trained from scratch with a batch size of 128 and a standard cross entropy loss. Training was done in a data-parallel manner on four Nvidia RTX GPUs (Titan/3090/A5000) with 24GB of memory each.

AlexNet-BN . This model was included as a pure feedforward control. We used the PyTorch implementation for AlexNet [65] with BN provided in [66]. The optimizer was stochastic gradient descent (SGD) [67] and the learning rate was 1 e -2 . We report the results for the best epoch out of 59 (based on performance on the short validation stimuli) for three individual runs.

VGG-19-BN . This model was included as an additional pure feedforward control. We trained a variant of the original VGG architecture [68] referred to as 'vgg19_bn' in the torchvision model zoo [69]. We used an SGD [67] optimizer and a learning rate of 1 e -2 . We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs. Interestingly, in two additional runs with the same hyperparameters, VGG-19-BN was unsuccessful in solving even the easy version of the task and performed at chance level.

hGRU-BPTT . This is a recurrent control model with the same architecture (see SI §A) as the model studied in the main paper (referred to there as cRNN), but trained with the classic BPTT. Furthermore, we found that in order to learn the easy version of the task with BPTT, we had to replace BN with group normalization. The optimizer was Adam [50], lr = 1 e -3 , and T = 15 . Our hypothesis was that this control model would fail the generalization test and perform near chance level on the harder long validation stimuli. We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.

hGRU-CRBP . This model shares the same recurrent architecture (see SI §A) and training algorithm algorithm (C-RBP) as the cRNN used in the main paper. We trained the hGRUCRBP models with an Adam [50] optimizer, lr = 1 e -3 , T = 40 , and γ = 100 . Our hypothesis was that this model would still perform well above chance level on the harder long validation stimuli, in contrast to both the feedforward controls and the RNN control without C-RBP. This hypothesis was motivated by findings in [8]. We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.

ConvLSTM . This is a recurrent control model with the LSTM architecture [70] that shares the C-RBP training algorithm used to train the cRNNs in the main text, but lacks the anatomical and neurophysiological constraints. We trained the ConvLSTM models [71] with Adam [50], lr = 1 e -3 , T = 40 , and a γ of either 100 or 1000 . We analyzed the generalization performance of the runs that reached at least 90% accuracy on the short validation stimuli.

ViT-B-16 . ViT-B-16 was included as a control model from the Vision Transformer family [49]. We used the implementation included in the torchvision zoo [69] as 'vit_b_16'. To

train the model, we used an Adam [50] optimizer, a patch size of 10 , a learning rate of 1 e -2 , a dropout rate of . 10 , and an attention dropout rate of . 10 . We report the results for the best epoch out of 299 (based on performance on the short validation stimuli).

B.3 Results

Fig. S1 summarizes the results of the generalization experiments. The models that share the same architecture and training algorithm (hGRU-CRBP) as the cRNNs studied in the main text performed well above chance on the long validation stimuli. Despite having high accuracy on the short validation stimuli (i.e., those with dot distances within the range they were trained for), the control models performed near or at chance on the harder long validation stimuli. The only exception to this finding was ViT-B-16.

AlexNet-BN (1)

AlexNet-BN (2)

AlexNet-BN (3)

VGG-19-BN (1)

VGG-19-BN (2)

VGG-19-BN (3)

hGRU-BPTT (1)

hGRU-BPTT (2)

hGRU-BPTT (3)

hGRU-CRBP (1)

hGRU-CRBP (2)

hGRU-CRBP (3)

ConvLSTM (1)

ConvLSTM (2)

ViT-B-16 (1)

0.5

0.6

0.7

0.8

0.9

1

Test:

Short val. stimuli

Long val. stimuli

Test accuracy

C Dynamically Expendable Computational Budget

a

b

ImageIstabilizesearly

ImageIIstabilizeslate

Img.I1

→0

8

8

8

(5%)

size

6

size

6

t=small

Time

2

step

*

t=T

Step

4

Step

4

size

D

Img.I

2

2

t=small

0

15

0

15

30

45

t*

PC 1 (83%)

Time (t)

Time (t)

D Videos

For the incremental grouping task (Section 4 in the main text) and the maze task (Section 6 in the main text), the cRNN learned a filling-in strategy that was evident from inspecting the latent activity h t over time steps ( t ). The gradual spread of activity can be appreciated best in video-format. Examples of such videos can be found on our project page: https://serre-lab.github.io/rnn_rts_site.

Each video consists of three panels. The left panel shows the input to the model. The middle panel shows the uncertainty curve, with a marker added at every time step to indicate the progression of time. The right panel shows a visualization of the latent activity over time. Each frame shows h t averaged over the channel dimension and smoothed by additionally averaging between h t and h t + 1 . With each frame in the video, one can observe the spread of activity and the corresponding change in uncertainty.

E Distance Experiment

When humans perform the incremental grouping task (Section 4 in the main text), their RT tends to increase with increasing distances between the dots. This effect has been attributed to a serial spread of attentional resources [47, 52]. To test whether the effect was also present in the cRNN we trained, we created a dedicated test set, derived from a random subset of the validation outlines ( N = 150 ). As explained in the main text, we repositioned the dots to be in the center of an object and 14 px apart in Euclidean distance. We then systemically moved them 14 px further apart until they hit an object boundary, thus creating different distance conditions for a given outline (see Fig. S3 for an example). This procedure resulted in a total of 1852 stimuli. Fig. S3 demonstrates how the distance manipulation affected the uncertainty curves for one example outline. The cRNN spent more time being uncertain if the dots were placed further apart, which is captured in our ξ cRNN metric. Analyzing the relation between distance and ξ cRNN across the whole test set using a linear mixed-effects model, we found a significant positive effect, b = 1 . 46 , SE = 0 . 11 , z = 13 . 58 , p < . 001 (Fig. S4).

n

5nl

30

E

CRNN

=3.06

=

：4.28

=6.73

E

CRNN

：9.46

= 17.92

：20.68

14

28

42

56

70

84

98 112 126 140 154

4

6

8

10

12

Euclidean dot distance (px)

ξ

cRNN

F Non-Euclidean Distance Metric

Jeurissen et al. [47] describe a growth cone distance metric that is sensitive to the topology of the object, proximity of boundaries, and narrowness. The authors construct a series of non-overlapping, varying-sized discs whose centers trace a curve (originating at the fixation location and ending at the cue location). The radii are such that a disc is the maximum inscribed disc at its location. Disks never cross object boundaries. The final distance metric is defined as the total number of discs that constitute this curve

G Spatial Anisotropy in Incremental Grouping - Additional Experiments

As a further test of how factors beyond Euclidean distance might affect ξ cRNN in the incremental grouping task, we created two sets of stimuli (Fig. S7, Fig. S8), each derived from the same 23 novel outlines by placing dots in different configurations. Following the example of [47], each stimulus featured two adjacent objects of the same object class. In one set (Fig. S7), we created two conditions by placing one of the dots (or both) in a narrow region of an object in one condition, and moving the pair of dots to a wider region in the other condition. The Euclidean dot distance, as well as the topological distance were matched between both conditions. In the other stimulus set (Fig. S8), we used the same outlines but with different dot conditions. Here, one condition had the dots placed in a such a way that they could be connected by a straight path. In the other, the path was curved making that the higher topological distance condition. The Euclidean dot distance, on the other hand, was still matched. Fig. S5 visualizes the effect of both these manipulations on ξ cRNN . As reported in the main text, paired t-tests showed that ξ cRNN was significantly higher in the narrow versus wide condition, d = 0 . 70 , t (22) = 3 . 34 , p = . 003 , and in the curved versus straight condition, d = 0 . 98 , t (22) = 4 . 71 , p < . 001 .

Wide

Straight

path

Narrow

Curved

path

0

5

10

15

20

25

30

**

***

ξ

cRNN

Finally, in a last follow-up experiment, we varied the distance between one dot and an object boundary while keeping the Euclidean distance between the dots themselves constant (Fig. S6). We found that our model exhibits higher values near the boundaries (Fig. S6b,c). This is strikingly similar to human behavior presented in prior literature ([47], Fig. 3d).

a

b

d

Humans

Model

25

ECRNN

0.7

Class

20

Yes

25

S

0.65

cRNN

15

No

20

Cue18

15

0.6

WS

10

10

Fix

5

Fix

5

0.55

Proximity to boundary

Dot distance (Eucl.)

CueLocation

0

1

6

CueLocation

12

18

24

Narrow

Narrow

Narrow

Wide

Wide

Wide

S

h.

h.

Curved

Curved

Straight

Straight

Curved

Straight

.

H Ambiguous Grouping Stimuli and Non-zero Uncertainties

Here, we take a closer look at which stimuli elicit very high ξ cRNN values. Looking at our validation set first, we observed that some highξ cRNN stimuli are characterized by uncertainty curves that remain high, even at t = T , suggesting the model deems them ambiguous (see Fig. S9 for an example).

Uncertainty (e)

0

0

Time (t)

）T

Furthermore, in a test set we designed to be ambiguous (it features occlusions), we frequently observed such non-decreasing uncertainty curves too (see Fig. S10). We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.

e

b

证

Uncertainty

30

25

EcRNN

20

0

0

Time (t)

T

15

Uncertainty (e)

10

5

0

ControlOccluded

Control

Occluded

Control

Occluded

Control

Predicts"No"

Predicts"Yes"

0

0

T

Time (t)

I Additional Maze Analyses

Human participants have been found to take more time to solve a maze when the solution path features more junctions [56]. To study whether the cRNN we trained exhibited a similar slow-down, we computed two measures for the Yes-mazes (Fig. S11). One was the number of T-junctions encountered on the shortest path between the two cues. The other was the total number of T-junctions on the maze segment containing the cues. Linear regression analyses that statistically controlled for path length by including it as a predictor, did not yield evidence for an effect of the first measure, b = -0 . 23 , SE = 0 . 29 , t (234) = -0 . 79 , p = . 43 or the latter, b = 0 . 28 , SE = 0 . 18 , t (234) = 1 . 56 , p = . 12 . We treat this finding as an important example for how our metric can help identify potential mismatches between human- and model-behavior and ultimately guide the development of better aligned models.

a

b

J ConvLSTM Results on Scene Categorization

a

TASK

b

Naturalormanmade?

20

0.39

0.38

S

EcRNN

15

R

0.37

n

10

w

←·ConvLSTM

0.36

工

→cRNN (hGRU)

5

0.35

20

40

60

80100

Natural

Manmade

Discriminability (%)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new measure of visual artificial network computation 'time', $\\xi_{cRNN}$, as the time-averaged uncertainty of a convolutional RNN trained with an evidential deep learning loss (Sensoy et al.). The authors proceed to analyse the dynamics of this network as it solves a range of classification tasks. Importantly, beyond the network learning to solve the tasks, $\\xi_{cRNN}$ is generally well correlated with human reaction times, with which it also shares several qualitative features across tasks.\n\nSTRENGTHS:\nDeep network models of the visual system have often focused purely on classification accuracy and neural representations when comparing to biological data. However, this neglects an important aspect of human behaviour, namely the time it takes to arrive at a decision. The present paper tackles this important question using recent ideas from the machine learning literature and represents an interesting step in the direction of capturing the temporal variability in human behaviour.\nThe authors analyse an impressive breadth of tasks and behavioural data and include many interesting analyses, both qualitative and quantitative.\n\nWEAKNESSES:\nThe major weakness of the submission is that while there are strong _correlations_ between $\\xi_{cRNN}$ and human reaction times, there is less evidence that $\\xi_{cRNN}$ is mechanistically similar to human reaction times, since it is fundamentally a measure of uncertainty rather than computation time. In particular, human reaction times generally involve a tradeoff between computation/evidence accumulation and decision making (as in common drift diffusion models). On the contrary, the cRNN has a fixed computational budget and has no need or even capacity for evaluating this tradeoff. This is in contrast to a few previous deep learning models in the literature that are capable of explicitly trading off computation and actions (e.g. Graves et al., Pascanu et al., and refs by the authors in the ML literature, and Jensen et al. in the Neuro/Cogsci literature). These considerations are important because e.g. task difficulty is likely to correlate with both uncertainty and reaction time, and this raises the question of whether a model of one is automatically a model of the other.\n\nIt might the interesting to compare the current model to alternative models that more explicitly have adaptive/variable computation time, such as a cRNN that computes until $\\epsilon$ reaches a certain threshold (akin to classical drift diffusion models).\n\nAs the authors mention in L326, another potential weakness is that the present approach does not easily generalize beyond classification tasks, which form a small subset of the types of problems humans and animals are faced with during natural behaviour.\n\n_References:_ \\\nGraves et al.: \"Adaptive computation time for recurrent neural networks\", arXiv 2016.\\\nPascanu et al.: \"Learning model-based planning from scratch\", arXiv 2017.\\\nJensen et al.: \"A recurrent network model of planning explains hippocampal replay and human behavior\", bioRxiv 2023.\\\nBansal et al.: \"End-to-end Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking\", NeurIPS 2022.\n\nQUESTIONS:\nL106: what does 'without loss of generality' mean here?\n\nL123 \"our control models lack inference-time flexibility.\" What does this mean?\n\nL144: It might be worth comparing to or discussing recent work by Bansal et al. (NeurIPS 2022), which develops a way of stably training convolutional RNNs with long computation times that generalize from small to large problem settings.\n\nL156 \"for a complete mathematical specification of this loss, refer to SI 1\". $\\epsilon$ does not seem to be defined in either the main text or SI beyond a qualitative description as 'uncertainty'. It would be good to provide a precise mathematical definition somewhere.\n\nL204: it would be good to briefly explain how this distance measure by Jeurissen et al. is defined (or at least refer to SI where it can be described in more detail).\n\nFigure 3b and 5c: it's not entirely clear to me how these 'activity maps' are computed?\n\nFigure 7b: it might be worth talking a bit more about the differences between the human and model data here instead of doing linear regression and calling it a day. The cRNN data is decidedly not linear and exhibits something resembling a plateau followed by a sharp drop at high discriminability. This seems rather different from the monotonic decrease in human RT, and the authors could perhaps speculate on or discuss this difference.\n\nSupplementary material: it would be worth having a separate folder with just a few example videos of each condition in addition to the larger set of videos currently provided.\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this work propose a combination of model output uncertainty predictions along with stable recurrent vision architectures in order to derive a proxy for models' reaction time to process input static images. The authors use the previously published horizontal GRU architecture combined with stable RNN training methods (contractive RBP) in their experiments. The proposed work is technically sound with extensive evaluation on 4 diverse datasets (some inspired by prior work in visual psychology) where the computed proxy model reaction time ($\\xi$) trends correlate positively with that of humans performing the same task. Overall, this paper contributes a new way to estimate output uncertainty of recurrent convolutional networks and to evaluate similarity between model & human perception by taking into account temporal dynamics of processing static inputs.\n\nSTRENGTHS:\n+ The authors make several key contributions: (1) combining Evidential Deep Learning-based output uncertainty prediction with convolutional RNNs and using the AUC of the output uncertainties over time as a measure of the model's reaction time; (2) training stable recurrent vision models using the above EDL-based readout + objective function in order to obtain models whose temporal dynamics of processing visual inputs matches that of humans performing the same task.\n+ The experiments performed are thorough on all datasets, it is clear that the positive correlation between model and human reaction times is present on the tasks evaluated.\n+ If the authors were to release the code for some of these datasets (especially the incremental grouping task in Fig 2), this would make the contribution even more valuable as it would encourage further exploration of these understudied grouping problems\n+ Clear presentation; The authors have exactly stated their contributions and have written the paper with good clarity and detail. The figures are intuitive and appreciate the video presentations showing model activations and uncertainty through time in the Supplementary.\n\nWEAKNESSES:\n- Choice of tasks: While the presented 4 visual cognition tasks are interesting and relevant to cognitive scientists, only the scene recognition task concerns real-world stimuli and natural vision. It would be great if the authors found any interesting patterns of reaction time on natural images or more naturalistic tasks than the ones shown here.\n- Advantages of similarity in reaction time: It is very interesting that the authors are presenting recurrent networks that are correlated with humans in terms of reaction time (and the benefits as a model of biological visual processing are clear), but does this similarity translate to any significant advantages for machine learning? Are models with better RT correlation with humans also well calibrated, or more robust compared to others? In summary, inter-model comparison based on how well they rank on RT similarity with humans would be an interesting direction that the authors don't comment about here.\n\nQUESTIONS:\nPlease refer to my review above for questions.\n\nLIMITATIONS:\nThe authors have adequately addressed limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of the paper present a novel concept of crafting a metric for analyzing the temporal alignment between Recurrent Neural Networks (RNNs) and human behavior. The innovative approach, rooted in the estimate of task uncertainty via the application of the Derelict distribution, is both interested and pertinent. In an era where understanding the interface between machine learning models, specifically RNNs, and human behavior is of utmost importance, the authors' research is timely and bears the potential for broad impact. A commendable aspect of this study is the diverse range of tasks that the authors have attempted to cover. Their inclusive approach elevates the practical relevance of their research, thus ensuring the applicability of their findings across multiple contexts. However, there are two crucial areas that the authors need to address to improve the overall quality and comprehension of the manuscript. Firstly, in the same/different dot object task, the authors discuss a correlation between the task's difficulty, reaction time, and dot distance. It is important to note that human reaction time also ties strongly with factors such as the object's topology and occlusion. The authors assert that their network and measurements reflect human reaction time concerning these more intricate features. Nevertheless, the demonstration of this claim in the manuscript is not lucid enough. To solidify their claim, it is crucial for the authors to condition the different stimulus conditions on the distance between dots. They should explicitly illustrate how elements like the narrowness of the object outline and occlusion correlate with human reaction time and their RNN metric. Secondly, the specificity of the proposed metric to the particular cortically inspired architecture employed in this research should be clarified. It would be beneficial to understand if this metric can be generalized to other architectures. For instance, would it be feasible to use this metric if one were to implement a convolutional LSTM or another vanilla RNN? The authors' response to this concern could significantly influence the wider application of their proposed metric.  In conclusion, while the paper is insightful and tackles a highly relevant topic, it is recommended that the authors address the concerns raised to improve the robustness and clarity of their research. By doing so, the potential impact of the manuscript can be further augmented, providing a more substantial contribution to the field.\n\nSTRENGTHS:\nA relevant and timely paper extending the comparison between human behavior and ANNs.\n\nWEAKNESSES:\nNot clear how well they demonstrate that their metric predicts human reaction time between the more straightforward relationship of dots distance, task difficulty and reaction time. In other words do they predict reaction time and object occlusion, and more complex topology of objects (narrowness of segmentation boundaries etc). \n\nDoes their metric extend to other architectures beyond the specific one they use here.\n\nQUESTIONS:\nPerform analysis to condition on dot distance to demonstrate that other more intricate relationships predict human reaction time.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study introduces a metric for evaluating the alignment between model and human behavior wrt task complexity reflected in reaction times. The metric is easy to compute and shows qualitative correspondence with human RTs in different tasks.\n\nSTRENGTHS:\nThe paper is well written and easy to understand.\n\nThe study attempts to go beyond choice responses to establish correspondence between model and behavior. For image-computable models, this is both novel and important.\n\nThe idea of using model uncertainty as a proxy for RT is intuitive and appealing.\n\nWEAKNESSES:\nThe models are trained using an algorithm that imposes attractor dynamics which causes instantaneous  uncertainty to become zero. This is critical for their proposed metric to remain bounded. But this is a bit odd because subjective uncertainty should be non-zero for difficult tasks even if given a long time. This raises the question of whether the metric is meaningful for other tasks e.g. with ambiguous stimuli.\n\nThe authors make a convincing case for the utility of this metric for evaluating models. But the metric itself is artificial and might not have a correlate in neural activity. An alternative (and more brain-like) but admittedly more challenging approach would be to  learn a policy that chooses left/right/wait at each moment (e.g. via RL) based on the latent representation such that you can directly get a reaction time from the model. This might even work with BPTT.\n\nQUESTIONS:\nSentence in lines 88-90: I don't follow. Please unpack.\n\nLines 38-39. This assertion is a bit strong. There have been some preliminary attempts to explain human RTs using image-computable models so it would be good to cite them.\n\nLIMITATIONS:\nAs stated above, would be useful to give some examples of the challenges which may arise when applying this approach to other types of tasks.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}