Computing a human-like reaction time metric from stable recurrent vision models
Authors: Lore Goetschalckx
Lakshmi Narasimhan Govindarajan
Alekh Karkada Ashok
Aarit Ahuja
David Sheinberg
Thomas Serre
Venue: NeurIPS 2023 spotlight (2023)
Keywords: alignment
RNNs
reaction times
equilibrium dynamics
perceptual grouping
decision making

Abstract: The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to ``align” these models with humans. One dimension of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decision-making has been challenging. Here, we sketch a general-purpose methodology to construct computational accounts of reaction times from a stimulus-computable, task-optimized model. Specifically, we introduce a novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent vision models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision-making tasks spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: https://serre-lab.github.io/rnn_rts_site/.

Computing a human-like reaction time metric from stable recurrent vision models
Lore Goetschalckx* 1,2 Lakshmi N. Govindarajan"S Alekh K. Ashok 12, Aarit Ahuja", David L. Sheinberg', and Thomas Serre 1.2
Depaarmeent Cognitive, Linguistic, Psychological Sciences, Brown University, Providence, RI 02912 "Camey Institute for Brain Science, Brown University, Providence, RI 02912 alekh karkaaaaaasaaaaaaaaaaaa
(eommasserreebbbrooot edu Integrative Computational Neuroscience (ICoN) Center, MIT, Cambridge, MA 02319
lakshmin@mit, edu
"Nurroscience Department, Brown University, Providence, RI 02912 serpitt_ahjjaa,aee hww.daaddddddeeeeeeeeeeeeeeeeeee ebrow...dd
Abstract
The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to "align" these models with humans. One dimen- sion of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decision- making has been challenging. Here, we sketch ceneral-purpose methodology to construct computational accounts of reaction times from stimulus-computahby task-optimized model. Specifically, we introduce novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent sion models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision- spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: httts://serree: lab. gi thub. io/ Seennnntt__t ssste//
tasks
Introduction
The symbiosis between visual neuroscience and machine learning has created a distinctive and unprecedented phase in the field of vision. On the one hand, formidable advances in machine learning have supported the development of computational models of visual cognition that rival, and in some cases surpass, human-level performance on naturalistic tasks 5 On the other hand, incorporating computational and neurophysiological constraints from biological visual systems has yielded significant benefits for AI models in terms of their performance 6 efficiency [8] and robustness [9 Additionally, performant computational vision models that operate directly on high-dimensional sensory inputs have shown great promise for scientific discovery through the generation of testable hypotheses for neuroscience [1] [3]. At the core of this synergistic loop lies the need for reliably quantifying the degree of "alignment" between models and the primate visual system, which can subsequently serve control knob for closed-loop hypothesis testing.
The notion of "alignment" is subjective and can be operationalized varying levels of analysis 14 One of the most prominent alignment techniques is to quantify the degree to which model feature
'These authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

representations predict time-averaged firing rates of single neurons across various stimuli [1] 17 In contrast, computational-lewv alignment analyses typically use axiomatic attribution methods [8 to extract minimal sub-circuits from larger models and probe them to gain mechanistic insights [22 third popular approach to alignment at higher behavioral level, where techniques focus on understanding the extent to which models and primates utilize similar "visual strategies". Top-ddwn attention masks, diagnostic of human observers' visual strategies, are a commonly employed axis of comparison in this type of model alignment 7 [9I. At the same time, other behavioral measures such as object confusion patterns 200 20 or human-derived pairwise image similarity ratings [2] are also widely considered effective quantities that can form the basis of alignment.
Each of these techniques has common tenet: their purview limited to measures of alignment that are global, time -ndeeeddeennd summaries of neural or behavioral states. While this may be sufficient for task paradigms primarily involving feedforward processing, a large part of visual cognition is highly dynamic [23 Understanding visual computations, thus, necessarily involves characterizing and modeling the time-varying properties of neural responses to complex and varied inputs. Why, then, is there dearth of alignment methods focusing directly on the level of visual processing dynamics? We posit a two-fold bottleneck. First, there is a need for more performant models that exhibit and mimic visual cortical dynamics at scale. Second, we lack the mathematical tools to extract meaningful dynamical signatures from such models and compare them with biological systems. While recent progress in biologically- inspired machine learning has started to open up solutions for the former g 19 26 28], the latter remains a wide-open research area.
Contributions. In this study, we consider this temporal alignment perspective.
Recent advances in training biologically motivated large-scale convolutional recurrent neural network models (cRNNs henceforth) on visual cognition tasks [8] have provided candidate models to study the temporal alignment problem. A cRNN's inherent design affords access to its time-varying dynamical properties that can be explicitly analyzed. Throughout this paper, we specifically focus extracting metric that captures cRNN's dynamical signature and on quantifying the ability of this metric to explain primate decision time courses in the form of reaction times (RTs) in concomitant visual psychophysics tasks.
We introduce novel computational framework train, analyze, and interpret the behavior of cRNNs on visual cogmitive tasks of choice. Our framework triangulates insights from an attractor dynamics-based training routine [8] and evidential learning theory (29 support stable and expressive evidence accumulation strategies.
We derive timulus-computabbe task- and model-agnostic metric to characterize evidence accu- mulation in cRNNs. Our metric does not require extra supervision and purely leverages internal cRNN activity dynamics.
We comprehensively demonstrate the efficacy of our metric in capturing himulus-dependen primate decision time courses in the form of reaction times (RTs) in four disparate visual cognitive challenges that include incremental grouping, mental simulation, and scene categorization. To the best of our knowledge, this is the first demonstration of qualitative temporal alignments between models and primates across task paradigms.
2 Related Work
Cognitive process models. The speed-accuracy tradeoff in human behavior is ubiquitous in cognitive science (31] and has garnered much attention from the modeling perspective. The most prominent of them is the Drift-Diffusion Model (DDM) family, which specifies RTs by determining when the amount of accumulated evidence for behavioral choice from noisy process reaches threshold value 33 Though such models have had phenomenal successes in aligning neural and behavioral data 344 they are typically not stimulus-computale
or rely on handcrafted heuristics [35]. Moreover,
such models are usually directly fitted to RT data 35 36 quantity without any additional supervision.
unlike our approach, where RT is a derived
Computing reaction-time proxies from neural network models. Several notable efforts have attempted to derive RT measures from neural network models to serve as proxy for "computational cost" 26 42). Spoerer et al. (66 introduced the notion of energy demand in their objective function that motivated parsimony regarding the number of computations the cRNN executes. Bansal
2

---

CRNN with EDL
b
0.6
CRNN
Readout
0.5
ERRNN
hg
Uncertainty
0.4
High uncertainty
Low uncertainty
P(No)
0
P(No)
0
0.3
0.2
D(plo)
D(plo)
0.1
0
P(Yes)
P(Yes)
1
Time (t)
Figure 1: Computing reaction time metric from a recurrent vision model. a. A schematic representation of training cRNN with evidential deep learning (EDL; (30). Model outputs are interpreted as parameters (a) of a Dirichlet distribution over class probability estimates, with higher values (symbolized here by darker gray) reflecting more generated evidence in favor of the corre- sponding class. In this framework, the width of the distribution signals the model's uncertainty (e) about its predictions. Panel a) used in the task studied in Section 4 The metric, denoted {aRNN, is defined as the area under the uncertainty curve, i.e., the evolution of uncertainty (c) over time.
Visualization of our metric, EARNN, computed for an example stimulus (see
ct al. [40] discussed "algorithmic extrapolation" where their networks solved problems of greater difficulty than those in their training set by performing additional computations. Graves (37) proposed the use of a "halting unit" whose activation level determined the probability of continuing model computations. Finally, Figurno et al. 43 motivated by [37, adaptively determined the spatial extent and number of processing layers in deep convolutional network. In a broader sense, "conditional computing" approaches aim to learn dropout-like policies 44 45 to train neural networks that can balance parsimony and performance. While the approaches presented in [37 43 are impressive advances, they neither operate explicitly with cRNNs nor yield RT values for downstream comparisons on visual cognitive tasks. The elegant approach of [26) suffers from the primary drawback that it relies on the backpropagation through time (BPTT) algorithm. BPTT imposes high memory demands, limiting the number of timesteps a cRNN can perform and therefore condemning any derived measure be coarse. Furthermore, vanilla BPTT forces the dynamics be constant (i.e., the cRNN arrives at a solution at t T for any input) making it non-stimulus-adappiee [8]
Probabilistic interpretation of model uncertainty. Decision certainty and computational parsimony are duals of each other. In other words, easier tasks require fewer computational resources, with models typically being most certain about their predictions, while the opposite holds for harder tasks. Modeling decision uncertainties provide another effective angle to capture the "computational demand" of a stimulus condition. While deterministic neural networks suffer from ill-calibrated prediction confidence in Softmax outputs, Bayesian neural networks derive prediction confidence from their weight uncertainties but are generally not performant. Inspired by the Dempster-Shane Shafer theory of evidence (2)], Sensoy et al. (30) propose formulation that treats model outputs as parameters of Dirichlet distribution that signifies the total amount of evidence accumulated for each class, Although influential, this body of work, called Evidential Deep Learning (EDL), has never been linked to RT measures or applied to visual cognitive tasks. We refer the interested reader to Ulmer al. [46) for comprehensive survey of this literature. In this paper, specifically leverage EDL in conjunction with attractor dynamics to support the derivation of an RT proxy.
General Methods
3.1 Models
The model of interest chosen this work the horizontal gated recurrent unit (hGRU; 19). convolutional recurrent neural network model that we will canonically refer to cRNN throughout the rest of the paper. In principle, our framework applies to other cRNN architectures well. The hGRU is inspired by the anatomical and neurophysiological properties of the primate
cRNN.
3

---

a
TASK
b
Are the dots on the same object?
Avg latent trajectory Yes-class
t=small
u
W
T=]
PC 2 (5%)
-2
Avg latent trajectory No-class
No
Yes
-10
10
PC (83%)
Humans
Model
ECRNN
Mean RT (ms)
Mean ECRNN
25
760
15
20
740
10
15
720
10
700
A
C
D
A A
Figure Human versus cRNN temporal alignment on an incremental grouping task. a. Description of the task (inspired by cognitive neuroscience studies (47). b. Visualization of the cRNN dynamics. The two lines represent the average latent trajectories across 1K validation stimuli labeled "yes" and "no" respectively. Marker size indicates average uncertainty across the same stimuli. Evident from the graph that the two trajectories start to diverge after some initial time passes, clearly separating the two classes. Owing to the C-RBP training algorithm and attesting to the dynamics approaching an equilibrium, the step sizes become increasingly small over time. We also include snapshots of the latent activity in the cRNN for example stimuli (one "yes", one "no"; see Panel a) along the respective trajectory. Each snapshot represents h, at time step t, averaged across the channel dimension and smoothed by averaging h, and h Notice the spread of activity over time. The cRNN gradually fills the segment containing the dots. The strategy can be appreciated even better in video format (see SI $D) Comparison against data from the experiment with human participants in [47). The position of one dot (white. labeled "Fix") was kept constant, while the position of the other was manipulated to create experimental conditions of varying difficulty. These manipulations have qualitatively similar effects on the cRNN as they do on human behavior. Error bars represent the standard error the mean across stimuli. The significance levels for shown contrasts were adjusted downward from .1 (*), .05*, .01 **, and .001 using Bonferroni. The spatial uncertainty map shown on the right visualizes the spatial anisotropy observed in the model for the same example stimulus shown on the left.
visual cortex. Architectural facets of the hGRU include an hypercolumnar organization, distinct excitatory and inhibitory subpopulations, and local- and long-range feedback mechanisms, rendering it overcomplete and an ideal candidate cRNN for our purposes, The hGRU and its extensions have previously demonstrated proficiency in learning long- long-range spatial dependency challenges, including For implementational details, refer to SI $A
naturalistic ones 8 19 48
Control models. To further motivate our cRNN choice, we refer to its ability to systematically gener- alize more challenging task settings after being trained less challenging ones (8). Inference-tine flexibility and performance are critical desiderata, given that our goal is to probe the computational demands of a task via stimulus manipulations. We first demonstrate that the cRNN's performance remained well above chance on exceedingly challenging versions of the first task we consider (see Section D) after training on a simplified version (see SI B for full experimental details). We trained pure feedforward convolutional neural networks (CNNs) as control models under identical training

---

data distributions. The CNNs struggled to generalize effectively, as did naive control RNN model (Fig. [ST). Because CNNs lack explicit temporal dynamics and our recurrent control models failed our generalization test, we limit the experimental results presented in this paper to our cRNN. For completeness, we remark that a Vision Transformer [49] passed the generalization test. However, we exclude these models from our analyses for two reasons. First, as such, Vision Transformers lack biological analogs. Second, our focus in this work is to derive RT measures from model dynamics, which transformers do not exhibit.
3.2 Training Framework
General remarks. All model runs reported herein were done from scratch using only the ground truth class labels for supervision. Any alignment between the cRNN model and humans would have had to emerge as a byproduct of task constraints. We emphasize that no human RT data were involved in our training routines. For the cRNNs, we computed a task loss only at the network's ' end state to preserve the ability to remain flexible during inference. The cRNNs were trained in data-parall aata-parallel manner 4 Nvidia RTX GPUs (Titan/3000/A5000 with 24GB of memory each. Each training run took approximately 48 hours, except those in Section (2 hours), which were faster due to the smaller training dataset size. They were furthermore trained with Adam [50], learning rate of 1e 3, and y = 100 for the C-RBP penalty (8); also see SI РA We use = 40 recurrent timesteps during the training phase, with an exception for the task in Section 6 where T 80 and y 1000.
Stabilizing cRNNS. We train our cRNN model with the contractor recurrent back- propagation algorithm (C-RBP: (8). C-RBP imposes an attractordyynamic constraint on the cRNN's s computa- tional trajectory, imparting several benefits over the standard BPTT algorithm. C-RBP is significantly more memory efficient and affords arbitrary precision in the cRNN temporal dynamies (i.e., T can be high). C-RBP also allows the cRNN to learn stable and expressive solutions. We can observe this from the cRNN's latent trajectories, as visualized in Fig. 2b, where extrapolation in time (beyond T') is non-catastrophic. We argue that model instability why BPTT-trained controls failed the generalization test (Fig. (ST). Furthermore, stable solutions imparted by C-RBP allow our cRNNs to dynamically adapt their number of processing steps in ciimulus-dependee manner. That is, pose that a ERBP-ttrinne cRNN can utilize its computational budget differently depending on the input (see Fig. S22 for illustration). Finally, we note that our strategy to achieve stable recurrent solutions is in contrast to that employed by Bansal et al. [40), wherein cRNNS were stabilized through architectural choices along with an incremental training process.
Evidential loss. We augment our cRNN training objective with an evidential deep learning loss (EDL:; (0)) contrast to the classic cross-entropy loss, EDL considers cRNN outputs to be the parameters of a belief distribution rather than point estimates of class probabilities (Fig. (p). We formalize cRNN beliefs as a Dirichlet distribution. Strong evidence in favor of one class causes the Dirichlet distribution to be peaked, reflecting low uncertainty (e). In contrast, when there is lack of overall evidence, the Dirichlet distribution becomes more uniform, thereby increasing c. For a complete mathematical specification of this loss formulation, refer to SI SA
3.3 RT Metric (EERNN)
We derive a novel metric to quantify the differential computational demands imposed on the cRNN by stimulus manipulations and capture the essence of model decision time courses for alignment. Modeling uncertainty (e) explicitly allows us to track its evolution over time from the cRNN (Fig. [[)). We formulate our RT metric NE as the area under this uncertainty curve. Preciscly, ECRNN
Se" e (1) di (Fig. (p). Intuitively, high (low) ERRNN can understood as the cRNN spending large (small) amount of time in an uncertain regime. Our metric affords direct comparisons to human RT'.
In the following sections, we apply this metric to cRNNs trained on host of visual cognitive tasks and study their alignment to human reaction time data.
5

---

&cRNN
25
FIx
20
Fix
Fix fo
10
FIX
5
Figure 3: ECRNN is spatially anisotropic. We present spatial uncertainty (SU) maps visualize the impact given fixed dot position will have on model RT' when the position of the second dot is varied. Higher (lower) values of ÊCRNN represent slower (faster) model responses.
4 Incremental Grouping
4.1 Task, Stimuli, and Visualizations
we propose an equivalent challenge for the cRNN where the model is required to determine whether two dots are on the same object (Fig. 2p). The stimuli consist of white outlines of natural shapes on a black background (150 x 150 px). We created 340K (15K) training (validation) outline stimuli from MS COCO images [5T] The dots were placed remi-randomly such that there was difference in the distribution of dot distances between the two stimulus classes.
Inspired by an incremental perceptual grouping task in humans 47
We additionally introduce two novel types of visualizations to probe cRNN visual strategies. First, we track cRNN dynamics by looking at its latent activity h, at every time step t. averaged across the channel dimension and smoothed by averaging h, and h To facilitate interpretation, we provide these videos for the reader (see SI $D). Second, we compute "spatial uncertainty" (SU) map for each outline stimulus by keeping one dot fixed and varying the position of the other. Each spatial coordinate has a value equal to &ANNN for that corresponding dot configuration. Examples are shown in Fig. 2 (right) and Fig.
4.2 Results
Despite its lightweight architecture, our cRNN solved the incremental grouping task, achieving 98% validation accuracy. Visualizing the cRNN latent activities revealed that the model learned cognitively-viabl filling-in strategy. Network activity originated at each of the dot locations and gradually spread, eventually filling up the whole object(s) (Fig. Σp). We note that this behavior is an emergent property from purely the classification task constraint instead of direct supervision for any form of segmentation. We then sought to ask if EERNN effectively captures these visual decision-makinn dynamics. In the following sections, we report results detailing the relationship between EARNN n and host of complex, topological stimulus properties.
ECRNN captures the serial spread of attention in our models. Human visual strategies in solving the incremental grouping task are thought to involve a serial spread of attentional resources, resulting in longer RT' for higher distances [47]. To test this effect in our model, we took subset of our validation outlines and repositioned the dots be in the center of an object, 14 px apart. We then systematically kept moving the dots 14 px more apart until reaching the object boundary. A linear mixed-effects regression analysis revealed a significant positive effect of this manipulation on &CRNN. b = 1.46, F 0.11, 13.58, .001 (Fig. S3 Fig. S4). The SU maps (Fig. [ right, Fig. 3 exhibit this distance effect too. Placing the second dot further away from the fixed dot (indicated in white) yielded higher ÊaNNN values.
EERNN recapitulates the spatial anisotropy in human RT data. Euclidean distance only paints partial picture. Human RTs are affected by several other factors, such as topological distance, the proximity the dots to boundaries, and the need to spread through narrow object regions, all
amounting to a well-documented spatial anisotropy in the spread of visual attention 47 52 While anisotropy is evident in the SU maps from our cRNN (Fig. 2 right, Fig. 33,, we also formally assess the alignment between humans and the cRNN in this regard, using human RT data from [47]

---

TASK In which basket (L) will
Human RT (s)
the ball (+) land?
Laws
p
0010
BEANN
10 20
30
Left
Right
ECANN
Figure 4: ECRNN from our model recapitulates human RT' the Planko task. a. The stimuli
consist of a ball on top. followed by ten randomly placed planks and finally two baskets in the bottom. Participants are shown these static stimuli and are asked to guess if the ball will end up in the left or the right basket when released and allowed to bounce around on the planks. b. Human RTs (collected in (53] are predicted by N.NNN. The five colored symbols reflect the data points obtained from the stimuli shown in the next panel. Five example stimuli of increasing EARNN.
First, Jeurissen et al. [47] quantified their stimuli on an alternative dot distance measure that went beyond Euclidean distance to reflect the other aforementioned factors and was more predictive of human RT (see SI $ We found this measure to be better predictor of &ARNN (r .80) as well (versus
.45 for Euclidean distance), 1633 4.68, p .001.. Second, the stimulus set featured distinct experimental conditions, varying in expected difficulty based on the position of the dots (Fig. 2 left). As shown Fig. 2 (middle), the effects of this manipulation on ECRNN were qualitatively aligned with those on human RT. A one-way repeated measures ANOVA across 22 outline stimuli revealed a sigmificant effect of "Condition" on CCRNN, F(, 63) 67.65, p < .001, as well as on the average RTs from [47] F33 63) = 16.95, .001. Furthermore, Fig, Q (middle) presents the results of planned contrasts (paired t-tests). CCRNN was significantly higher for B compared to A, Cohen's s d = 2.00, t(21) 9.399. < .001, but also for the more difficult Condition compared to B. = 0.73, t(21) 3.42. p .003. Third, we observed a significant overall correlation between human RT and &RRNN on Yes-stimuli, set of our own, paired t-tests showed that NARNN was significantly higher when two dots on the same object were placed in narrow versus wide object regions, d = 0.70, ((22) = 3.34, p .003, and when the path was curved (i.c., longer topological distance) versus when was not, d = 0.98, f(22) = 4.71, p .001 (see SI $G Fig. S5) Finally, SI @G describes experiment in which we varied the distance between one dot and the object boundary while keeping the distance between the dots themselves constant. We found that model exhibits higher values near the boundaries, which is strikingly similar to human behavioral results presented in [7]]
.31, .01. Fourth,
novel test
Ambiguous stimuli. As a final way of gaining insight into the cRNN's task behavior, we looked at those stimuli in the validation set that elicited the highest ECRNN v values. We observed that some were characterized by uncertainty curves that remain high, even at t = T, suggesting the model deems them ambiguous. In test set designed to be ambiguous (it features occlusions),
frequently observed such don-decraa uncertainty curves too (see SI H We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.
Taken together, these results suggest that our metric is able to capture the dynamical signatures of object-based attention in our cRNNs and provides a means to quantify alignment with human visual strategies.
5 Visual Simulation Planko
5.1 Task and Stimuli
The Planko task (and corresponding dataset) was introduced 53 and features stimuli (64 x 64 px; 80K training, 5K validation) displaying a ball about to fall down and eventually land in one of two baskets ("left"""rightt). The outcome depends on the position and orientation of of planks placed in between the ball and the baskets (Fig. 44). Humans are thought to solve this task through mental
7

---

b
TASK
he-small ha-
Uncertainty (e)
Predicts "No"
Is there path to connect the squares?
40
Predicts "Yes"
30
EcRNN
(t)
hresmall he-.::
Uncertainty (e)
No
Yes
r= .84, p .001
20 40 60 80
length
Time (t)
Figure 5: cRNN learns filling-in strategy to solve mazes. Task description. b. The cRNN model takes more time to solve mazes with longer path lengths, an effect previously found in human participants too. c. Latent activity visualizations for yes-mazes. The cRNN gradually fills the segment containing the cues. The strategy can be appreciated even better in the vidcos supplied in the SI. d. Uncertainty curves for the two inputs shown in Panel c. The cRNN remains uncertain for much longer for the maze featuring the longest path. The uncertainty evolution is visualized dynamically in conjunction with the change in latent activity in the supplementary videos.
visual simulation 33 Notably, feedforward CNNs struggle to solve the task (54 and when they do, they appear to adopt non-aligned strategies to humans [53]. In contrast, cRNNs show evidence of simulation-liz like behavior on a variant of the task [54]. With our metric, we are now able to directly compare human (collected by [53] and cRNN RTs. To this end, we trained a cRNN from scratch on Planko (see Section 3.2 and SI A for details on training).
5.2 Results
The stimuli displayed in Fig. 1g..4in illustrreee the challenge of the Planko task. Our cRNN achieves performance of 84% on the validation set, well within the confidence interval of human accuracy [53]. Stimuli presented
human participants in the original study (N 200) were used here as held-out set. We observed significant correlation between model &CRNN values and human RTs,
r = -311 p .001) (Fig. 4p). This result indicates that stimuli that were harder for humans tended to be computationally challenging for the cRNN (see examples in Fig. Fb..
What makes stimulus challenging? Variance in human RT' is known to be partially explained by the statistics of the planks. For instance, stimuli for which the outcome is more likely to change as consequence of perturbing the plank configurations by a small amount were deemed "uncertain". Indeed, participants responded slower to these stimuli (53]. We were able to replicate this result in cRNN, .31, p < .001. These results showcase how our methodology can be used make stimulus-dependent RT predictions. This could help shed light on. for example, the remarkable yet examind ability of humans to flexibly adopt different strategies Planko task depending on the stimulus demand.
6 Mazes
6.1
Task and Stimuli
We devised maze task where the goal is to tell whether two cued locations are connected by path (k).. To build our training (N 34.5K) and validation (N = 14K) set, we started from mazes provided by [40] but modified them make them suitable for classification (the originals did not feature any disconnected paths). The cues were placed such that the two classes (""yes"''no" did not differ in Euclidean distance. Each maze was 144 x 144 px.
6.2 Results
Here too, we observed that the uncertainty curves and derived EARNN values differed considerably depending the input, even if nearly all mazes were classified correctly at T (99% correct

---

TASK
b
Natural or manmade?
20
0.39
EcRNN
16-
0.37
Human
14
10
0.35
20 40
60
80
100
Natural
Manmade
Discriminability (%)
Figure 6: ECANN is predictive of the discriminability of visual scene. Task description. b. Human RTs (collected in (57]) decrease as function of discriminability. as do &CRNN- Highly discriminable scenes easier for both human participants and the cRNN model. Error bars represent the standard error of the mean.
on validation mazes). For the Yes-mazes contained in a random subset of the validation set, we found that the model spent more time being uncertain (higher &GRNNN) when the length of the path between the two cued locations was longer, r = .84, p < .001 (Fig. 5p.d). Moreover, path length was significantly better predictor than simple Euclidean distance (r .39), t(234) = 5.81, p < .001. Similarly, human RTs on maze tasks have been shown to increase with path length (55). When the cued locations were two distinct maze segments (No-mazes), EANNN the mean of both segment lengths, r .56, p < .001. The kind of visual strategy that produced such distance effects can be understood by inspecting the visualizations of the latent activities (Fig. 5pp). The model gradually spread activity along the maze segment(s) containing the cues. We strongly encourage the reader watch videos (see SI $ better appreciate this spread in conjunction with the evolution uncertainty over time. Finally, we did not find evidence for an effect on &ARNN of the number of T-junctions on the path, which are known to predict human RT (56). We speculate that this potential misalignment, worth investigating in future work, could be due to the model not being restricted in how many possible paths can explore at once.
significantly predicted by
Scene Categorization
7.1 Task and Stimuli
To showcase how our general approach can also be applied to study higher-level visual tasks, consider a natural versus manmade scene categorization task as our final challenge (Fig. (). Human RT data from a rapid-scene categorization version of this task were already available from [57] We selected images from the SUN database (58] querying the same classes as in [57]. but excluding the
images shown to human participants, and performed stratified train (N 8.8K) and validation
(N 979) split. The image preprocessing pipeline entailed cropping the image to be square, resizing to 150 x 150 px, converting to grayscale, and random horizontal flipping. In addition, the images were matched for overall brightness.
7.2 Results
The model was able to correctly classify 83% of the validation images and 73% of the held-out test images from [57]. The errors can be understood, least partly, in light of the less-than- clear-cut labels (c.g., an islet with a house on top, building behind many trees). Each of the test images had previously been assigned a discriminability score (57), based on the average error rate of logistic classifiers trained on "gist" features [59]. Human participants were reported to be slower with decreasing discriminability [57]. Here, we show similar trend in the model (Fig. (D), with a linear regression analysis revealing significant, negative discriminability slope, 0.47, SE 0.03, / = 16.45, .001. Directly comparing EARNN to human RT, we found them to be significantly correlated, here does not perfectly account for empirical patterns in human reaction times, specifically in their non-monotonic reaction time readouts for the low-discriminabilit stimuli, they provide promising
.19,p < .001. While our cRNN model of choice

---

start. Our framework is model-agnosten (e.g., see SI for results on convLSTM) and thus can be used test hypotheses on architectural improvements.
Discussion
We are in the midst of a paradigm shift in visual neuroscience. The continued development of computational models that rival human behavioral performance has called for tools to quantify how well the internal mechanisms of such models compare biological truths. These efforts have demonstrably been beneficial for both the Al and neuroscience ecosystems.
In this work, we push this frontier by introducing framework characterize the evidence accu- mulation strategies of stable recurrent vision models. Our experiments thus far show the potential generate novel hypotheses of interest visual neuroscientists. The most direct application of our framework for experimental design is in taxonomizing stimuli in terms of the putative compu- tational demand based on our metric toward understanding neural processes at varying timescales. Furthermore, the value proposition of this framework includes the ability to adjudicate between models based on the consistency of their latent dynamics with human reaction time data. For example, our metric from a cRNN trained on the maze task is uncorrelated with the number of T-junctions in the stimulus, indicating a potential divergence between serial and parallel exploration strategies (SI $T) It is worth noting that distinguishing between serial and parallel attention models has been of long-standing interest [60]. Similarly, testing zoo of hierarchical models on the scene categorization task while keeping an eye the degree of alignment with our metric can potentially lend novel mechanistic insights and help delineate bottom-up, lateral, and top-down processes [6]]. We are also enthusiastic about drawing theoretical parallels between our metric summarizing the cRNN's evidence accumulation strategy and the complexity of natural data statistics toward understanding the sample efficiency of learning in models and primates.
Limitations and an outlook for the future. Within the scope of this work, we have considered four disparate visual cognition tasks to showcase how our approach offers the means to study the temporal alignment between recurrent vision models and the human visual system. Of course, in the face of the myriad of tasks that cognitive scientists have used over centuries, our selection still needs to be improved. Our results do not directly speak to the diversity in response types, stimulus types, or naturalism, to name a few. However, our approach is general-purpose enough to adapt to newer tasks as long as they are cast as classification problem. We believe this should cover significant share of cognitive science and AI tasks. Extending the approach beyond choice tasks would require considerable revision of how to make the model's uncertainty explicit. Another limitation is that there could easily be other cRNNs that align with human RTs better. While we motivate the choice for the cRNN in Section 3.1 our main contribution here is to introduce the overall framework integrating stable recurrent vision models with evidential learning theory to support the derivation of novel RT metric. A comprehensive comparison of different models, apart from those in SI S was hence beyond this work's scope. However, we are confident our framework paves the way to build a taxonomy of models in the future as well generate testable hypotheses for neuroscience.
Broader impacts. With AI vision models becoming ubiquitous in our daily lives, the call make them fairer, transparent, and robust is growing louder. Our framework facilitates this by allowing researchers to peek into the inner working of these AI models for transparency and thinking about their match to biological systems, which are arguably more robust. From a neuroscience perspective, computational models of RT are gaining prominence in computational psychiatry and hold the key to understanding many neurobiological disorders [22 63 impact on society in either scenario.
We generally do not anticipate any negative
Acknowledgments
The authors would like to explicitly thank Jeurissen et al. [47 and Sofer et al. 57 for generously sharing their stimuli and human reaction time data, and Bansal et al. (40] for their maze stimuli. We are also grateful to Drew Linsley for his helpful comments and feedback. This work was supported by ONR (N00014-19) 1--11----2029) and NSF (IIS- 19S-191220 1912280). Additional support provided by the Carney
10

---

Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the computing hardware resources supported by NIH Office of the Director grant S100D025181.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[2] Christian Szegedy. Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan. Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[3] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1002 1004,00,,22 2015.
[6] Grace W Lindsay and Kenneth D Miller. How biological attention mechanisms improve task performance in large-scale visual system model. ELife, 7:s38105, 2018
[7] Thomas Fel. Ivan F Rodriguez. Drew Linsley, and Thomas Serre Harmonizing the object recognition strategies of deep neural networks with humans. S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 9432-9446, Curran Associates, Inc., 2022. URL https: /ttttp::/pproe httpp//reee...nnuuunrreeeeeeeeee Pappeaarrr----------------rr Conference. pdf
ww/ppper......... supper_fllee files/
[8] Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrett vision models. In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33. pages 10456-10467, Cur- ran Associates, Inc., 2020. URL https: https:///proceedingg mww.eiiiiiiinnnnnnnooeeeomee wwwrips.cc/paaeeeeepppeeeeeeeeee wttp:/aapaeeeeeeleeppeppeeeeeeee /itps//apsssse///2000000/ file/766d856ef 1aw.0100934444444666666000000000
.ddf
[9] Aran Nayebi and Surya Ganguli. Biologically inspired protection of deep networks from adversarial attacks. arXiv preprint arXiv:1703.09200. 2017.
[10] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems, 33: 33:13073-13087 2020.
[11] Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Baldwin Geary, Michael Ferguson, David Daniel Cox, and James J. DiCarlo. Aligming model and macaque inferior temporal cortex representations improves model-to-human behavioral alignment and adversarial robustness. In The Eleventh International Conference on Learning Representations, 2023. URL htttttt:///ppeeeeeeeeee wttpeeewwiiiiieeeeeeeeeeeeeeeeee
[12] Hidenori Tanaka. Aran Nayebi. Niru Maheswaranathan, Lane Mcintosh, Stephen Baccus, and Surya Ganguli. From deep learning to mechanistic understanding in neuroscience; the structure of retinal prediction. Advances in neural information processing systems, 32, 2019.
[13] Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(4)))2200000011 19, 2022.
[14] David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010.
[15] Daniel L Yamins, Ha Hong, Charles Cadieu, and James J DiCarlo. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque and human ventral stream. Advances in neural information processing systems, 26, 2013.
[16] Martin Schrimpf, Jonas Kubilius, Michael J Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Newron, 108(3):413 2020

---

[17] Beyed-Mahdi Khaligh-Razzi Razavi and Nikolaus Kriegeskorte. Deep supervised. but not unsupervised, models may explain it cortical representation. PLoS compulational biology, (0e11)ee003913 2014.
[18] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning. pages 3319-3328. PMI.R, 2017.
[19] Drew Linsley. Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long- range spatial dependencies with horizontal gated recurrent units. In S. Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URLhttps://prott Paper.
hew.ssenuri.pp... heurips.cc/papene wwppappr_p...pp shaper_filess files/
[20] Rishi Rajalingham, Elias B Issa, Pouya Bashivan, Kohitij Kar, Kailyn Schmidt, and James J DiCarlo. Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-ae deep artificial neural networks. Journal of Neuroxcience, 38(33):7255-726, 2018.
[21] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 34:23885-23899 2021.
[22] Joshua Peterson, Joshua T Abbott, and Thomas L Griffiths. Evaluating (and improving) the correspon- dence between deep neural networks and human representations. Cognitive science, 22(8):24482661 2018.
[23] Andre M Bastos, W Martin Usrey, Rick A Adams, George R Mangun, Pascal Fries, and Karl Friston. Canonical microcircuits for predictive coding. Neuron, 76(4):695-711, 2012.
[24] Maxime Cauchoix, Gladys Barragan-Jason, Thomas Serre, and Emmanuel Barbeau. The neural dynamics of face detection in the wild revealed by mvpa. Journal of Neuroscience, 34(3):846-88) 854, 2014.
[25] Andre Moraes Bastos, Julien Vezoli, Conrado Arturo Bosman, Jan-Mathijs Schoffelen, Robert Onstenveld, Jarrod Rohert Dowdall, Peter De Weerd, Henry Kennedy, and Pascal Fries. Visual areas exert feedforward and feedback influences through distinct frequency channels. Neuron, 8502):390-401, 2015.
[26] Courtney J Spocrer, Tim C Kietzmann, Johannes Mehrer, Ian Charest, and Nikolaus Kriegeskorte. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS computational biology, 16( (6(10)=e008215 2020.
[27] Jonas Kubilius, Martin Schrimpf. Aran Nayebi. Daniel Bear, Daniel LK Yamins. and James J DiCarlo. Cornet: Modeling the neural mechanisms of core object recognition. BioRxiv, page 408385, 2018.
[28] Kohitij Kar, Jonas Kubilius, Kailyn Schmidt, Elias B. Issa, and James J. DiCarlo. Evidence that recurrent circuits are critical the ventral stream's execution of core object recognition hehavior. Nature Neuroscience, 22(6):974-983, April 2019. doi:
URI. httpps
//doi meoii.org/100 103g//0..03///4440
[29] Ronald R Yager and Liping Liu. Classic works of the Dempster-Shafer theory of belief functions, volume 219. Springer, 2008
[30] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[31] Richard P Heitz. The speedaacuuaae tradcoff:: history, physiology, methodology, and behavior. Frontiers in neuroscience, 8:150. 2014.
[32] Roger Ratcliff and Gail McKoon. The diffusion decision model: theory and data for two-choice decision tasks. Neural computation, 20(4):873-222, 2008
[33] Alexander Fengler, Lakshmi N Govindarajan, Tony Chen, and Michael J Frank. Likelihood approximation networks (lans) for fast inference of simulation models in cognitive neuroscience. Elife. 10:e65074, 2021.
[34] James F Cavanagh and Michael Frank. Frontal theta as a mechanism for cognitive control. Trends in cognitive sciences, 18(8):114-21, 2014.
[35] Budmonde Duinkharjav, Praneeth Chakravarthula, Rachel Brown, Anjul Patney, and Qi Sun. Image features influence reaction time. ACM Transactions on Graphics, 41(4):1-15, July 2022. doi: 10.1/ 3528223. 3528223.3530055
URL httpp::/tt/oo htt::///oo..o/1///// 1145/3558228.3530000
12

---

[36] Amin Mirzaei, Seyed-Mahdli Mahddi Shaligh-Razavi Masoud Ghodrati, Sajjad Zabbah, and Reza Ebrahimpour: Predicting the human reaction time based on natural image statistics in a rapid categorization task. Vision Rexearch, 81:36-44, 2013. ISSN 0042-6989. doi: http:://do..ong/11.1016f.. wisree......i..s.....eee 20reess...1...000 3.02.003. URL
https: sstps://www.i http://www..cc.enccrree....... htt.:///c...eeatat..ca//////////
[37] Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint arXi::1003.08953 2016.
[38] Omkar Kumbhar, Elena Sizikova, Najib J. Majaj, and Denis G. Pelli. Anytime prediction as model of human reaction time. CoRR, abs/201 I. 12859, 2020. htttttttps///xxvi www.:/rgg//bb///eeeeeee 12859
[39] Razvan Pascanu, Yujia Li, Oriol Vinyals. Nicolas Heess, Lars Buesing. Sébastien Racanière, David P. Reichert, Theophane Weber, Daan Wierstra, and Peter W. Battaglia. Learning model-based planning from scratch. CoRR, 0ess/7777.06700 2017. URL httLllttpp/aatteie wttp://wwwww.//////////////// wrg/ 06170
[40] Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-end algorithm synthesis with recurrent networks: Extrapolation without overthinking. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. httpp/ppp///pppeeeeeeeeeeee/rere
SKY40XX
[41] Kristopher T. Jensen, Guillaume Hennequin, and Marcelo G. Mattar recurrent network model of planning explains hippocampal replay and human behavior. bioRxiv, 2023. doi: 10.1101/2023.0 2111000222.011...222222999 URI.
https: //ww: biorxiv. org/ content. /ttp///aaaa///////////////////// 2ttp///11119//033.1 011123.11..0 16. 523429
[42] Joshua I Gold and Michael N Shadlen. The neural basis of decision making. Annu. Rev. Neurosci. 30: 535-574, 2007.
[43] Michael Figurnov. Maxwell D Collins, Yukun Zhu. Li Zhang. Jonathan Huang. Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1039-1048, 2017.
[44] Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pincau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv: 1iXiv:5511.06297 1.06297, 2015.
[45] Ludovic Denoyer and Patrick Gallinari. Deep sequential neural network. arXiv preprint arXiv:1410.0510, 2014.
[46] Dennis Thomas Ulmer, Christian Hardmeier, and Jes Frellsen. Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation. Transactions on Machine Learning Research, 2023.
[47] Danique Jeurissen, Matthew W Self. and Pieter R Roelfsema. Serial grouping of 2d-image regions with abject-based attention in humans. el.ife, 5:e1 4320, 2016. ISSN 2050-084X. doi: 10.7554/cl life. 14320.
[48] Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi Narasimhan Govindarajan, Ennio Mingolla, and Thomas Serre. Tracking without re-rocognition in humans and machines. In Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021.
[49] Alexey Dosovitskiy, Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URI httppp:://peerreviie..etttteeeee ntw.:ewwwwee//oo/ id=YicbFaNTly
[50] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:://axxii org/ aba/ 1412
6980
[51] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCÓ: Common objects in context. In David Fleet. Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors. The European Conference on Computer Vision (ECCV) 2014, pages 740-755. Springer International Publishing, 2014, ISBN 978-3-319-10660
[52] Arezoo Pooresmaeili and Pieter R. Roelfsema. A growth-cone model for the spread of object-based based attention during contour grouping. Current Biology. 24(24):2869 2877, 2014. doi: 11111//..cb../000 10.000. URL http:://pp/oo org/ 1016/ cub. 20.4.10.00
13

---

[53] Aarit Ahuja and David L. Sheinberg, Behavioral and oculomotor evidence for visual simulation of object movement. Journal of Vision, 19(6):13, June 2019. doi: 10.1167/19.6.13, URL https:://di...rg////// 1167/19.6.13
[54] Alckh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Drew Linsley, David Sheinberg, and Thomas Serre. The emergence of visual simulation in task recurrent neural networks. In SVRHM 2022 Workshop @ NeurIPS, 2022. httphttttp://ppeeeee.eee wttp://www....eoooooo///////////
[55] David A. Crowe, Bruno B. Averbeck, Matthew V. Chafee, John IL Anderson, and Apostolos P. Geor- gopoulos. Mental maze solving. Journal of Cogmitive Neuroscience, 1265)813-827, September 2000. doi: 10.1 1000899999999990522226 URL HRLL/ttts::/// //doi org/
[56] Florian Kadner, Hannah Willkomm, Inga lbs, and Constantin A. Rothkopf. Finding your way out: Planning strategics in human maze-solving behavior. February 2023. doi: 10.3 00.31234/osf.o/mmee)) URL
https://doi. org/ 10. 31234/osf jo/nme9j
[57] Imri Sofer, Sébastien M. Crouzet, and Thomas Serre. Explaining the timing of natural scene understanding with a computational model of perceptual categorization. PLOS Computational Biology. 1004450004456 September 2015. doi: 10. 137 1...7////orall.pcbe 1004456. URL btttps://don org/ 10. 1371/ 1
pcbi 1004456
[58] Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to zoo. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010. doi: 10.1000/2...
[59] Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 42(3):145-15, 2001, doi: 10102//:0011119666 724. URL /ttpss//::///o http:/::///ooo//////////// 10. 1023/a: 1011139631724
[60] Rani Moran, Michael Zehetleitner, Heinrich René Liesefeld, Hermann J Müller, and Marius Usher. Serial vs. parallel models of attention in visual search: accounting for benchmark rt dt-distributions. Psychonomic bulletin & review, 13(55)1300-133 2016.
[61] Thomas Serre, Aude Oliva, and Tomaso Poggio. A feedforward archhttte accounts for rapid categoriza- tion. Proceedings of the national academy of sciences, 104(15)::424-6422, 2007.
[62] Quentin JM Huys, Tiago V Maia, and Michael Frank. Computational psychiatry as a bridge from neuroscience to clinical applications. Nature neuroscience, 19(3):404-411 2016.
[63] Peter F Hitchoock. Eiko I Fried, and Michael Frank. Computational psychiatry needs time and context. Annual review of psychology, 73:243-270, 2022.
[64] Tim Cooijmans, Nicolas Ballas, César Laurent, Çaglar Gülçehre, and Aaron Courville. Recurrent batch normalization. In International Conference on Learning Representations, 2017.
[65] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger, editors, Advances Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: htttps: htttoee:///.ooeiiigg.... 239822bbabbr444444444444444--333
.pdf
[66] Siqi Yang. Pytorch-alexnet. Inttps://githin hwt.://USSS////yyooooollleeeeee. 2020.
2020.
[67] H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22: 400 407, 1951.
[68] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Represen- tations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv. http:///arii..org//a////000 1556
[69] Daniel Falbel. torchvision:
Models, Datasets and Transformations for Images, 2023.
https//torhviisionmlvee..e.oo.
[70] Sepp Hochreiter and Jürgen Schmidhuher. Long short-term memory. Neural computation, 98):1735-)
1997.
[71] Xingjian SHI, Zhourong Chen, Hao Wang, Dit-Yan Yeung. Wai-kin Wong, and Wang-chun WOO. Convolu- tional Istm network: machine learning approach for precipitation nowcasting.
Cortes, N. Lawrence,
D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 28, Curran Associates, Inc., 2015, URL h/tp:///dioc/.......oooe .ettsieeuisssp..e wtt.iips.cpppppppe cctppperppp_e_c// files/ Paper. saper.pde pdf
14

---

Supplementary Information
A Implementation Details
A.1 Model Implementation
We instantiate cRNN canonical recurrent convolutional neural network layer inspired by the primate visual cortex's anatomical and neurophysiological properties. This model, termed hGRU and first introduced in Linsley et al. [19], implements hypercolumnar organization with distinet excitatory and inhibitory subpopulations and local- and long-range feedback mechanisms. For the sake of completeness, outline the governing equations of the model below.
First compute suppressive interactions:
G sUgmoid(U H[st 1])
# Compute suppression gate function of current state Applying the gate
C BN (W (H - 1] G9))
S Z (a 1| 1| C 1.]
Additive and multiplicative suppression Z
Then compute the facilitation terms followed by the recurrent update:
G² sU +S)
# Facilitation gate
C = BN(WF= =S)
# Facilitation term
((CC + +S)
# Modulation of S through additive and multiplicative facilitation
[[] = (1 G
+ G H
# State update
In these equations, Z E RXXIXC represent the eRNN's latent state and bottom-up feedforward drive, respectively, with heigh//width/channlls X, Y, C. WS W E IR Ex are learnable kernels controlling the suppressive and facilitatory interactions and E the spatial width of the lateral interactions. For all experiments, we set E 5, except for Planko where E 3. Similarly, U UF E |x are learnable kernels controlling the gates. Model dynamics are discretized, with E [1...]] representing the timesteps of processing. We use softplus rectifying nonlinearity (denoted by [.L.). We also use Batch Normalization (BN) the model to control apploding/vanishing gradients [64]. BN kernels are shared across timesteps of processing.
A.2 Training Implementation Details
Leveraging equilibrium dynamies for training. Training cRNNs has traditionally posed prob- lem due to the severe memory bottlenecks imposed by the backpropagation through time (BPTT) algorithm. To remedy this, Linsley et [ introduced variant of the classic recurrent backpropaga- tion algorithm (RBP) that leveraged equilibrium dynamics to compute model gradients with (11) memory complexity. We outline the high-level idea here and refer the reader to [ for more details.
Our discretized cRNN dynamics can be written as follows,
H fH - X,, W),
where f()) is the model architecture described, H(L) is the cRNN's state at timestep and X is the bottom-up sensory input. For convenience, we use W to denote all the trainable parameters of (-). If f-) is contraction mapping, then H() H" as f - x. One can define an implicit function as follows to aid the gradient computation: (HH X, W) = fHH W). By construction, we
can observe that (H"; X, W) 0 and by extension,
d X, W)
Ow
15

---

Following sequence of algebraic manipulations we can compute the gradient of the equilibrium state H w.r.t. parameters W as follows.
(I J.H.) Of XX; W) OW
ow
where JJ.H- is the Jacobian of the cRNN f(.) evaluated at H The ability to compute this gradient, however, relies (I-J being invertible. To guarantee this, Linslcy et al. [8] introduced the Lipschitz Coefficient Penalty (LCP) as CLer(W) = (1-Jj,HH- - A) l|2 which controls the degree
of contraction in f()) and by extension the invertibility of the term discussed above. Moreover, this regularization can be added to any task loss which we exploit.
Evidential learning formulation. To support our goal of characterizing evidence accumulation strategies in our cRNN, we parameterize our model readout to explicitly express uncertainty, Specifi- cally, we follow the evidential learning strategy introduced in Sensoy al. (30 to interpret cRNN readouts as the parameters of the Dirichlet density on the class predictors, a =< 011 O1....., OKK >, where K is the total number of classes in the classification problem of interest. As in (30), evidential loss comprises of two components, one that supports belief accumulation for the "correct" class (Lrisk) and second that ensures that the beliefs for all the other "wrong" classes have uniform mass (Lhaionce).
i=
(W) =
Σ C + pabannce (W)
j=K
p1 Pj)
Crisk (W) Σ
(u)
S+1
i=1
Lbalance (W) = DI (D(((ll)) D(p) 11,..,1>))
In the above, y denotes one-hot ground truth label for given sample with yj if the sample belongs to class j. and () otherwise. Pj is from the model that denotes the expected probability of the j-th singleton under the corresponding Dirichlet such that Pj TS. S ΣØ= aj resembles the "total evidence" accumulated by the model (or the Dirichlet strength). D is the Kullback-Leibler divergence measure. are the Dirichlet parameters of just the "misleading" evidence given by & y + (1 y)a. We define the instantaneous uncertainty as e- K
Finally, p an annealing term that is used to balance the relative importance of Crisk and Lbatance
In practice, we use the following increment schedule: Pepoch min 1, spach
where is
hyperparameter. For a complete derivation of the loss formula, we refer the reader to 30 We choose
== 16 for incremental grouping, 10 for Planko, 20 for the maze task, and 16 for scene categorization.
Training objective. Our final training objective is thus L(W) LEDI WLC)
16

---

B Generalization Experiments
B.1 Task and Stimuli
In the main text, we identified the flexibility to generalize to more complex task settings at inference critical desideratum for obtaining human-like RT metric from any model. We tested generaliz- ability in the cRNN as well as in set of control models (see further) using the incremental grouping task described in the main text (dots on the same object "yes"""no??), We trained on an easy version of the task by only selecting training stimuli for which the Euclidean dot distance was at most 44 px (50th percentile). We refer to this reduced training set as "short training stimuli" (NV = 170K). For evaluation, we created two subsets of the validation set, one easy and one hard. The easy set ("short validation stimuli", N 7.7K) consisted of those validation stimuli for which the dots were at most 44 px apart. The hard set ("long validation stimuli"" .N = 1.5K) consisted of those stimuli for which the dots were more than 82 px apart (90th percentile). All thresholds were computed across the training stimuli in the original set.
B.2 Models and Training
All models listed below were trained from scratch with a batch size of 128 and a standard cross entropy loss. Training was done in a data-parallel manner on four Nvidia RTX GPUs (Titan/3090//50000) with 24GB of memory each.
AlexNet-BN. This model was included as pure feedforward control. We used the PyTorch implementation for AlexNet (65] with BN provided in [66]. The optimizer was stochastic gradient descent (SGD) [67] and the learning rate was 1e - 2. We report the results for the best epoch out of 59 (based on performance on the short validation stimuli) for three individual runs.
This model was included as an additional pure feedforward control. We trained a variant of the original VGG architecture [68] referred to "vgg19_bn 19__n" in the torchvision model zoo [69] We used an SGD (67 optimizer and a learning rate of le - 2. the short validation stimuli) for three individual runs. Interestingly, in two additional runs with the same hyperparameters. VGG-19-BN was unsuccessful in solving even the easy version of the task and performed at chance level.
VGG-19-BN.
We report the results for the best epoch out of 32 (based performance
hGRU-BPTT. This is recurrent control model with the same architecture (see SI ς as the model studied in the main paper (referred to there as cRNN), but trained with the classic BPTT. Furthermore, we found that order to learn the easy version of the task with BPTT, we had to replace BN with group normalization. The optimizer was Adam [50), Ir = le 3. and T = 15. Our hypothesis was that this control model would fail the generalization test and perform near chance level on the harder long validation stimuli. We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.
This model shares the same recurrent architecture (see SI A and training algorithm algorithm (C-RBP) as the cRNN used in the main paper. We trained the hGRU- CRBP models with an Adam [50] optimizer, Ir = le - 3, T 40, and = 100. Our hypothesis was that this model would still perform well above chance level on the harder long validation stimuli, in contrast to both the feedforward controls and the RNN control without C-RBP. This hypothesis was motivated by findings in the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.
EGRU-CRBP.
[8] We report the results for
ConvLSTM. This is a recurrent control model with the LSTM architecture 770 that shares the C-RBP training algorithm used to train the cRNNs in the main text, but lacks the anatomical and neurophysiological constraints. We trained the ConvLSTM models (71) with Adam (50, Ir = le 3, T = 40, and a 7 of either 100 or 1000. We analyzed the generalization performance of the runs that reached least 90% accuracy on the short validation stimuli.
ViT-R-16. ViT-B-16 was included as a control model from the Vision Transformer family 49]. We used the implementation included in the torchvision zoo (69 "vit b 16". To
17

---

train the model, we used an Adam 50 optimizer, patch size of 10, learning rate of
2, a dropout of _10, and an attention dropout rate of .10. We report the results for the best epoch out of 299 (based on performance on the short validation stimuli).
B.3 Results
Fig. ST summarizes the results of the generalization experiments. The models that share the same architecture and training algorithm (hGRU-CROC CRBP) as the cRNNs studied in the main text performed well above chance on the long validation stimuli. Despite having high accuracy on the short validation stimuli (i.e., those with dot distances within the range they were trained for), the control models performed near or chance on the harder long validation stimuli. The only exception to this finding was ViT-B-16.
Train: Short training stimuli
Test:
Test accuracy
0.9
Short val. stimuli
0.8
Long val. stimuli
0.7
0.6
0.5
AlexiNet-BN (1)
AlexNet- BN (2)
Alextlet- BN (3)
VGG-19 19-1 (1)
VGG-19 196--9-90 BN (2)
VGG-19 19-8N (3)
hGRU-BPT BPFTT (1)
hGRU hGRU-BPPTT BPTT (2)
hGRU-8PTT BRTT (3)
hGRU-CRBP CR-PRRBP (1)
hGRU-
hGRU-CCBB -CBBP (2)
hGRU-CRBP CRRP (3)
COnVLSTM (2)
ConvL (2)
VIT-B-16 (1)
Figure S1: Control models fail to generalize. The graph summarizes the results from generalization experiments involving the incremental grouping task. To study generalization, all models first trained on an easy version of the task with training stimuli for which the distance between the dots was short (S 44 px). They were then evaluated validation stimuli with the same, short dot distances (transparent bars), as well as on validation stimuli with longer dot distances (>> 82 px; cross hatched bars). The longer distances constituted harder version of the task. hGRU-CRBP models, sharing the same architecture and contractor recurrent back proppprppagttion (C-RBP) algorithm as the cRNN models studied in the main text, still performed well above chance level (0.50) on the hard version. attesting to their generalizability. The control models, in contrast, struggled with the hard version of the task, with ViT-B-16 being a noteworthy exception.
18

---

Dynamically Expendable Computational Budget
Image stabilizes early
C
Image 11 stabilizes
T--00
8
PC 2 (5%)
tesmall
Time
step AiP
Step
Step
tesmall
-15
30
C+
PC (83%)
Time
Time (t)
Figure S2: Dynamic computational time. a. Low-dimensional projections of our model dynamics for two input conditions in the incremental grouping task (Section[4gg of the main text). For the easy input (Panel b), the model effectively "stops" computing at earlier time point (t"), evidenced by dramatically reduced step sizes in the latent space. For the harder input (Panel c), step sizes remain large (above the dotted line) for longer period. Overall this demonstrates that model dynamically
utilizes its compute budget a stimulus-dependent manner. b,c. Alternative visualization the step size function of time for the easy and hard stimuli, respectively. The dotted line here is arbitrarily chosen for visual comparison.
19

---

D
Videos
For the incremental grouping task (Section in the main text) and the maze task (Section 6 in the main text), the cRNN learned a filling-in strategy that was evident from inspecting the latent activity h, over time steps (f). The gradual spread of activity can be appreciated best video-format. Examples of such videos can be found on our project page: https://serre-lar lab. wat.eellbb.a........... github.io/rnn rts site
Each video consists of three panels. The left panel shows the input to the model. The middle panel shows the uncertainty curve, with a marker added at every time step to indicate the progression of time. The right panel shows visualization of the latent activity averaged over the channel dimension and smoothed by additionally averaging between h, and h With each frame in the video, one can observe the spread of activity and the corresponding change in uncertainty.
time. Each frame shows h,
20

---

E Distance Experiment
When humans perform the incremental grouping task (Section in the main text), their RT tends to increase with increasing distances between the dots. This effect has been attributed to a serial spread of attentional resources [7 52 To whether the effect was also present in the cRNN we trained, we created a dedicated test set, derived from random subset of the validation outlines (N 150). As explained in the main text, we repositioned dots to be in the center of an object and 14 px apart in Euclidean distance. We then systemically moved them 14 further apart until they hit an object boundary, thus creating different distance conditions for a given outline (see Fig. [33 for an example). This procedure resulted in total of 1852 stimuli. Fig. S3 demonstrates how the distance manipulation affected the uncertainty curves for one example outline. The cRNN spent more time being uncertain if the dots were placed further apart, which captured in our EeRNN metric. Analyzing the relation between distance and EARNN across the whole test set using a linear mixed-effects model, we found significant positive effect, 1.46, SE 0.11, 13.58. p .001 (Fig. (S4).
EORNY 3.06
ELANN 4,28
EcRNN 6.73
EceNN 9.46
Eanom 17.92
5ONNN 20.68
1.00
Predicts Yes'
Uncertainty
Prosicts
0.50
0.25
0.00
Time (t)
Time (t)
Time (t)
Time (t)
Time (t)
Time (c)
Figure S3: Manipulating dot distance in the incremental grouping task. The top row demonstrates
the manipulation of the dot distance for an example outline. The dots are colored orange here for
visualization purposes. The bottom row represents the corresponding uncertainty curves, with &eRNN annotated on top. With the dots further apart, the cRNN spends more time being uncertain, resulting in higher ELRNN.
12
10
6-
4
14
70 84
126 140 154
Euclidean dot distance (px)
Figure S4: Overall distance effect in the incremental grouping task. The graph shows the average
SCANN stimuli as we systematically moved the dots 14 px more apart. Error bars represent the standard error of the mean. The cRNN exhibits a clear distance effect, as did human RTs in 47
21

---

Non-Euclidean Distance Metric
Jeurissen al. 47 describe growth cone distance metric that is sensitive the topology of the object, proximity of boundaries, and narrowness. The authors construct series of non-overlapping. varying-sized discs whose centers trace a curve (originating at the fixation location and ending at the cue location). The radii are such that a disc is the maximum inscribed disc at its location. Disks never cross object boundaries. The final distance metric is defined as total number of discs that constitute this curve
22

---

G Spatial Anisotropy in Incremental Grouping Additional Experiments
As further test of how factors beyond Euclidean distance might affect CaNNN in the incremental grouping task, we created two sets of stimuli (Fig. SS] Fig. (8). each derived from the same 23 novel outlines by placing dots in different configurations. Following the example of 47 each stimulus featured two adjacent objects of the same object class. In one set (Fig. (S7), we created two conditions narrow region an object in one condition, and moving
by placing one of dots (or both) the pair of dots to a wider region in the other condition. The Euclidean dot distance, as well as the topological distance were matched between both conditions. In the other stimulus set (Fig. S8) we used the same outlines but with different dot conditions. Here, in such a way that they could be connected by a straight path. In the other, the path was curved making that the higher topological distance condition. The Euclidean dot distance, on the other hand, was still matched. Fig. S5 visualizes the effect of both these manipulations on EARNN. As reported in the main text, paired t-tests showed that EARN versus wide condition, d 0.98, t(22) 4.71, p .001.
condition had the dots placed
significantly higher in the narrow 0.70, t(22) = 3.34, p = .003, and in the curved versus straight condition,
30
25
20
15
10
5
Wide
Narrow
Straight Curved path
path
Figure S5: Effects beyond those Euclidean distance the incremental grouping task. We created two novel test sets (Fig. S7 Fig. S8) In one we manipulated whether the dots were placed in a wide versus narrow region of an object. In the other the path between the dots was either straight or curved, thereby manipulating topological distance. In both manipulations, Euclidean distance was kept constant. The boxplots show the effect of these manipulations on ECRNN. The star markers inside the boxes represent the condition mean. The asterisks top represent the p-values resulting .001.
from a paired t-test, ** < .01,
Finally, in last follow-up experiment, we varied the distance between one dot and an object boundary while keeping the Euclidean distance between the dots themselves constant (Fig. (66). found that our model exhibits higher values near the boundaries (Fig. [S.b.c). This is strikingly similar to human behavior presented in prior literature (47) Fig. 3d).
Model
0.7
Class
RRN
25
20
Vies
0.69
FANN
13
No
20
15
R
10
Fix
Fix
Det dblunaa (Buck)
Cun Location
Cue Location
12 18
24
Figure S6: Fixed Euclidean distance; varying distance to boundaries. Wedge stimuli presented to our model. b. We observe higher EARNN closer to boundaries, as was observed for human RT in 47). Data from [47] are shown the left here. Visualizing model RT predictions for fixed chosen radius d. The complete SU-map reveals a further disparity between the "yes" and "no" conditions.
23

---

Narrow
Wide
Narrow
Wide
Narrow
Wide
Figure S7: Novel outline stimuli for the incremental grouping task featuring a narrow versus
wide manipulation. Dots colored orange for visualization purposes.
24

---

Curved
Straight
Curved
Straight
Curved
Straight
Figure S8: Novel outline stimuli for the incremental grouping task featuring a topological distance manipulation. The dots were placed such that it was either straight or curved path (higher topological distance) connecting them, whereas the Fuclidean distance was the same. Dots are colored orange for visualization purposes.
25

---

H Ambiguous Grouping Stimuli and Non-zero Uncertainties
Here, we take closer look which stimuli elicit very high EERNN values. Looking at our validation set first, we observed that some high-CeRNN stimuli are characterized by uncertainty curves that remain high, even at t T, suggesting the model deems them ambiguous (see Fig. S9 for an example).
Uncertainty (E)
0
Time (t) T
Figure S9: High uncertainty example. Some challenging samples our validation set display non-zero uncertainty.
Furthermore, in test set we designed to be ambiguous (it features occlusions), we frequently observed such non-decreasing uncertainty curves too (see (Tg). We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.
Uncortainty (c)
b
30
25
20
Time
15
Uncortainty (c)
10
Control Occluded
Control
Occluded
Control
Occluded
Control
Time (0)
Figure S10: Testing model uncertainty on occluded stimuli. a. We created dataset (N=76) of occluded images, half of which have both dots the same side of the occluding object (control). The Fuclidean dot distance matched between the control and occluded condition. We performed zero-shot transfer with trained cRNN this set and observed that for the occluded samples, model uncertainty often remained at high (non-zero) values in the time limit. b. A paired t-test reveals that our metric significantly higher in the occluded condition compared the control condition: Cohen's 's = 2.85, ((7)) = 17.54, < .001.
26

---

I Additional Maze Analyses
Human participants have been found to take more time to solve a maze when the solution path features more junctions [56]. To study whether the cRNN we trained exhibited similar slow-down, we computed two measures for the Yes-mazes (Fig. (STT). One was the number of T-junctions encountered on the shortest path between the two cues. The other was the total number of T-junctions on maze segment containing the cues. Linear regression analyses that statistically controlled for path length by including it predictor, did not yield evidence for an effect of the first measure, b 00.23, SE 0.29, ((34) -0.79, = .43 or the latter, 1.56. p .12. We treat this finding important example for how our metric help identify potential mismatches between human- and model-behavior and ultimately guide the development of better aligned models.
0.28, SE 0.18, ((234)
b
Figure S11: Counting T-junctions in Yes-mazes. a. An example maze. b. Visualization of the T-junctions. The solution path connecting the cued locations for the same maze shown in the previous panel is indicated in blue here. We found no evidence for an effect of the number of T-junctions included in the solution path (orange squares), nor of the number of T-junctions included in entire maze segment containing the cues (orange + green squares) on EGRNN.
27

---

J
ConvLSTM Results on Scene Categorization
TASK
b
Natural or manmade?
20
0.39
EcRNN
0.37
Human
10
ConVLSTM
cRNN (hGRU)
0.35
40
60
100
Natural Manmade
Discriminability (%)
Figure S12: A convLSTM equipped with metric captures human RT patterns in categorization task. Within our framework, we are able to swap out various model (or architecture) choices without affecting the downstream formulation. Whereas choice of architecture in the main text was motivated by the demands that task like incremental grouping imposes, we show here that E can also be computed for other architectures, such convLSTM. The fact that our metric is model-agnostic makes it ideally suited for comparing and contrasting models in terms of their temporal alignment.
28


Reviews:
[{"review_id": "8D8us2tdA0", "reviewer": "Reviewer_yVHQ", "summary": "This study introduces a metric for evaluating the alignment between model and human behavior wrt task complexity reflected in reaction times. The metric is easy to compute and shows qualitative correspondence with human RTs in different tasks.", "questions": "Sentence in lines 88-90: I don't follow. Please unpack.\n\nLines 38-39. This assertion is a bit strong. There have been some preliminary attempts to explain human RTs using image-computable models so it would be good to cite them.", "limitations": "As stated above, would be useful to give some examples of the challenges which may arise when applying this approach to other types of tasks.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to understand.\n\nThe study attempts to go beyond choice responses to establish correspondence between model and behavior. For image-computable models, this is both novel and important.\n\nThe idea of using model uncertainty as a proxy for RT is intuitive and appealing.", "weaknesses": "The models are trained using an algorithm that imposes attractor dynamics which causes instantaneous  uncertainty to become zero. This is critical for their proposed metric to remain bounded. But this is a bit odd because subjective uncertainty should be non-zero for difficult tasks even if given a long time. This raises the question of whether the metric is meaningful for other tasks e.g. with ambiguous stimuli.\n\nThe authors make a convincing case for the utility of this metric for evaluating models. But the metric itself is artificial and might not have a correlate in neural activity. An alternative (and more brain-like) but admittedly more challenging approach would be to  learn a policy that chooses left/right/wait at each moment (e.g. via RL) based on the latent representation such that you can directly get a reaction time from the model. This might even work with BPTT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their overall positive feedback and insightful comments.\n\n>**Does instantaneous uncertainty have to go to zero? What happens in the case of ambiguous stimuli?**\n\nWe thank the reviewer for this observation and their suggestion to test the model on ambiguous stimuli. \n\nWe take this opportunity to clarify that while our training procedure promotes stable attractor states in the model's *latent* representation, this does not directly correspond to a model reaching an uncertainty (obtained from the *readout*) value of *zero*. Additionally, given that we operate within a discrete, finite-time horizon paradigm, our metric does indeed remain bounded. In **Fig. R2**, we present one such example where a challenging stimulus results in an attractor state with non-zero uncertainty.\n\nWe also curated a novel dataset consisting of \"occluded\" stimuli, a classic scenario of ambiguity (**Fig R1a**). We perform zero-shot transfer of our trained model to this novel test set and observe its uncertainty traces which do match the reviewer's intuition (**Fig. R1a**). A paired t-test reveals that our metric is significantly higher in the Occluded condition compared to the Control condition: $Cohen's\\ d =  2.85,  t(37) = 17.54, p < .001$ (**Fig R1b**). We will add this analysis as a part of the discussion in the manuscript. \n\n>**Biological plausibility of the metric**\n\nExcellent point! We agree that metrics mined from an RL-trained model are likely to have analogous biological correspondences (as opposed to models trained via supervision). However, this is a hypothesis that needs empirical confirmation. We are excited to pursue this line of research in future work.\n\n>**Sentence in lines 88-90: I don't follow. Please unpack.**\n\nWe apologize for the confusing wording. We will make the correction as follows:\n\nThe elegant approach of [26] suffers from the primary drawback that it relies on the backpropagation through time (BPTT) algorithm. BPTT imposes high memory demands, limiting the number of timesteps a cRNN can perform and therefore condemning any derived measure to be coarse. Furthermore, vanilla BPTT forces the dynamics to be constant (i.e., the cRNN arrives at a solution at t = T for any input) making it non-stimulus-adaptive [8].\n\n>**Lines 38-39. This assertion is a bit strong. There have been some preliminary attempts to explain human RTs using image-computable models so it would be good to cite them.**\n\nThanks for the comment. We will include the following references as prior art in the domain of explaining human RTs using image-computable models. We shall also highlight better reference [26] from the current main text.\n\nMirzaei et al. (2013) Predicting the human reaction time based on natural image statistics in a rapid categorization task. Vision Research, 81(5), 36-44.\n\nKumbhar et al. (2020) Anytime Prediction as a Model of Human Reaction Time. arXiv preprint arXiv:2011.12859.\n\nDuinkharjav et al. (2022) Image Features Influence Reaction Time: A Learned Probabilistic Perceptual Model for Saccade Latency. ACM Transactions on Graphics, 41(4), 1–15. \n\n"}, {"review_id": "92XaYn0Rff", "reviewer": "Reviewer_JuhY", "summary": "The authors of the paper present a novel concept of crafting a metric for analyzing the temporal alignment between Recurrent Neural Networks (RNNs) and human behavior. The innovative approach, rooted in the estimate of task uncertainty via the application of the Derelict distribution, is both interested and pertinent. In an era where understanding the interface between machine learning models, specifically RNNs, and human behavior is of utmost importance, the authors' research is timely and bears the potential for broad impact. A commendable aspect of this study is the diverse range of tasks that the authors have attempted to cover. Their inclusive approach elevates the practical relevance of their research, thus ensuring the applicability of their findings across multiple contexts. However, there are two crucial areas that the authors need to address to improve the overall quality and comprehension of the manuscript. Firstly, in the same/different dot object task, the authors discuss a correlation between the task's difficulty, reaction time, and dot distance. It is important to note that human reaction time also ties strongly with factors such as the object's topology and occlusion. The authors assert that their network and measurements reflect human reaction time concerning these more intricate features. Nevertheless, the demonstration of this claim in the manuscript is not lucid enough. To solidify their claim, it is crucial for the authors to condition the different stimulus conditions on the distance between dots. They should explicitly illustrate how elements like the narrowness of the object outline and occlusion correlate with human reaction time and their RNN metric. Secondly, the specificity of the proposed metric to the particular cortically inspired architecture employed in this research should be clarified. It would be beneficial to understand if this metric can be generalized to other architectures. For instance, would it be feasible to use this metric if one were to implement a convolutional LSTM or another vanilla RNN? The authors' response to this concern could significantly influence the wider application of their proposed metric.  In conclusion, while the paper is insightful and tackles a highly relevant topic, it is recommended that the authors address the concerns raised to improve the robustness and clarity of their research. By doing so, the potential impact of the manuscript can be further augmented, providing a more substantial contribution to the field.", "questions": "Perform analysis to condition on dot distance to demonstrate that other more intricate relationships predict human reaction time. ", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A relevant and timely paper extending the comparison between human behavior and ANNs. ", "weaknesses": "Not clear how well they demonstrate that their metric predicts human reaction time between the more straightforward relationship of dots distance, task difficulty and reaction time. In other words do they predict reaction time and object occlusion, and more complex topology of objects (narrowness of segmentation boundaries etc). \n\nDoes their metric extend to other architectures beyond the specific one they use here. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and very helpful comments. We have provided clarifications and additional numerical analyses below to address concerns raised in this review.\n\n> **Moving beyond Euclidean distance between cue and fixation dots to other factors such as object topology**\n\nThanks for this comment. We agree with the reviewer that human reaction times are affected by factors beyond the Euclidean distance between cue and fixation dots in our incremental grouping task. We perform several analyses in the manuscript (and further ones in this rebuttal) to back this claim. We will take this opportunity to highlight these here and better position them in the main text. \n\n#### Narrowness and topological factors\nWe explore stimulus manipulations presented in Ref [41] in detail in this manuscript (subsection starting at L198 and titled “$\\xi_{cRNN}$ recapitulates the spatial anisotropy in human RT data”). We consider four distinct manipulations (A-D) in the incremental grouping task. The cue and fixation dot are nearest in Condition A. Condition B and Condition C are *matched* in Euclidean distance (larger than that in Condition A) but Condition C has the cue dot on *narrow* parts of the image. While Conditions A, B, and C correspond to the cue and fixation dot on the same object, Condition D has counterbalanced cue locations for the \"different\" case.   \n\n1. We find that $\\xi_{cRNN}$ is significantly higher in Condition C than Condition B even though they are matched in Euclidean distance, consistent with prior work on human RTs. This suggests that our metric clearly captures non-Euclidean factors (**Fig. 2c; L212** in the manuscript). \n\n1. The variance in $\\xi_{cRNN}$ is significantly better explained by a non-Euclidean distance metric (introduced in [46], explanation provided in our response to **4B3C**). We report this on **L208**.\n\n1. To further test effects beyond Euclidean distance we generated a novel stimulus set that explores two additional manipulations. In the first test, we keep *both* Euclidean distance and topological distance constant, but manipulated whether the path between the dots passed through narrow regions. $\\xi_{cRNN}$ was significantly higher in the Narrow condition than in the Wide condition. In the second test, we kept the Euclidean distance constant but manipulated whether there was a straight path between the dots or not (making the latter condition higher in topological distance). $\\xi_{cRNN}$ was significantly higher in the latter condition. **These results are summarized starting from L216 in the main text and discussed in more detail in the supplementary information. Fig. S5 and Fig. S6 of the SI also show all stimuli used.**\n\n#### Fixed Euclidean distance. Varying distances to boundaries.\n\nIn a follow-up experiment, we varied the distance between the fixation dot and an object boundary while keeping the Euclidean distance between the cue and fixation dot constant (**Fig. R4a**). We found that our model exhibits higher $\\xi_{cRNN}$ values near the boundaries (**Fig. R4b,c**). This is strikingly similar to human behavior presented in prior literature (Ref [41], Fig. 3d).\n\n#### Occlusions\n\nAs we discuss in our response to reviewer **yVHQ**, we also test our model on \"occluded\" stimuli (**Fig. R1**). When compared to control conditions, we find that our model exhibits extended periods of higher uncertainty in the occluded stimuli (**Fig. R1b**).\n\n> **Specificity of the proposed metric to the architecture considered in the manuscript**\n\nThis question motivated us to clarify an aspect of our framework and we have now taken steps to perform additional model comparisons. We believe this makes our manuscript stronger and we thank the reviewer for that.\n\nFirst, we take this opportunity to clarify that, mathematically, the choice of cRNN architecture and our evidential metric can be disentangled. Suppose $f_\\theta$ is our cRNN with parameters $\\theta$ specifying the model architecture. For the evidential loss term, we rely on interpreting the outputs of a readout function $g(.)$ operating on the cRNN states in this following manner: $\\hat{\\boldsymbol{\\alpha}}(t) = g(f_{\\theta}^t(x))$. Here $x$ denotes the stimulus and $f^t$ denotes the recursive application of function $f$, $t$ times. The subsequent construction of the evidential loss as well as our metric rely only on $\\hat{\\boldsymbol{\\alpha}}(t)$. Within this framework, we are able to swap out various choices of $\\theta$ and $f$(model architecture) without affecting the downstream formulation. This makes our metric an ideal choice to compare and contrast models in terms of their temporal alignment.\n\nHaving said that, our choice of architecture was motivated by the demands that a task like incremental grouping imposes. The visual cognitive psychology literature posits that incremental grouping requires feedback instantiated by lateral connectivity. Empirically, we find this as well. A recurrent CNN without horizontal connectivity, such as a convolutional LSTM (convLSTM), fails to learn generalizable solutions on this task (**Fig. S1**) and hence was not an appropriate model choice to test our metric.\n\nHowever, the rapid scene categorization task presents an opportunity to demonstrate the generality of our metric. This task is more akin to purely perceptual processing and relies less heavily on lateral feedback. We implement and train a convLSTM equipped with our metric on this task and indeed find that our metric captures variations in scene discriminability (**Fig. R3**), $b = -0.48,SE = 0.03,t = -16.88,p < .001. $ The correlation between $\\xi_{convLSTM}$ and human RT is $.17, p<.001$.\n"}, {"review_id": "FQEfTg5bRt", "reviewer": "Reviewer_fhSL", "summary": "The authors in this work propose a combination of model output uncertainty predictions along with stable recurrent vision architectures in order to derive a proxy for models' reaction time to process input static images. The authors use the previously published horizontal GRU architecture combined with stable RNN training methods (contractive RBP) in their experiments. The proposed work is technically sound with extensive evaluation on 4 diverse datasets (some inspired by prior work in visual psychology) where the computed proxy model reaction time ($\\xi$) trends correlate positively with that of humans performing the same task. Overall, this paper contributes a new way to estimate output uncertainty of recurrent convolutional networks and to evaluate similarity between model & human perception by taking into account temporal dynamics of processing static inputs.", "questions": "Please refer to my review above for questions.", "limitations": "The authors have adequately addressed limitations of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "+ The authors make several key contributions: (1) combining Evidential Deep Learning-based output uncertainty prediction with convolutional RNNs and using the AUC of the output uncertainties over time as a measure of the model's reaction time; (2) training stable recurrent vision models using the above EDL-based readout + objective function in order to obtain models whose temporal dynamics of processing visual inputs matches that of humans performing the same task.\n+ The experiments performed are thorough on all datasets, it is clear that the positive correlation between model and human reaction times is present on the tasks evaluated.\n+ If the authors were to release the code for some of these datasets (especially the incremental grouping task in Fig 2), this would make the contribution even more valuable as it would encourage further exploration of these understudied grouping problems\n+ Clear presentation; The authors have exactly stated their contributions and have written the paper with good clarity and detail. The figures are intuitive and appreciate the video presentations showing model activations and uncertainty through time in the Supplementary.", "weaknesses": "- Choice of tasks: While the presented 4 visual cognition tasks are interesting and relevant to cognitive scientists, only the scene recognition task concerns real-world stimuli and natural vision. It would be great if the authors found any interesting patterns of reaction time on natural images or more naturalistic tasks than the ones shown here.\n- Advantages of similarity in reaction time: It is very interesting that the authors are presenting recurrent networks that are correlated with humans in terms of reaction time (and the benefits as a model of biological visual processing are clear), but does this similarity translate to any significant advantages for machine learning? Are models with better RT correlation with humans also well calibrated, or more robust compared to others? In summary, inter-model comparison based on how well they rank on RT similarity with humans would be an interesting direction that the authors don't comment about here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their extremely positive feedback and very helpful comments. We have provided clarifications and additional numerical analyses below to address concerns raised in this review.\n\n> **The figures are intuitive and appreciate the video presentations**\n\nWe really appreciate the reviewer for going through the supplementary information in detail and for watching the videos.\n\n> **Code and data sharing**\n\nWe plan to make all code (model training/analyses) and datasets (with the accompanying data generation code) publicly available for the community to build on.\n\n> **Extensions to more naturalistic tasks**\n\nWe agree! This is a great direction and one that we envision for this work in the future. In this manuscript, we curtail our focus to detailing a framework without over-engineering very large-scale models that typically are needed to be performant on naturalistic benchmarks. However, our framework does not preclude scalability. We look forward to accomplishing this in future work.\n\n> **Are there benefits for ML conferred by temporal alignment?**\n\nThis is an excellent point! In the past, aligning deep neural network models to neuroanatomy, neural, and behavioral data has shown benefits in terms of performance (R1), interpretability (R2), and adversarial robustness (R3). One of our motivations for developing this framework was to enable investigations into the benefits of temporal alignment. \n\nWhile this was not a primary focus at the time of writing this manuscript, intrigued by the reviewer's comments we piloted experiments to test what we think is an important computational benefit conferred by temporal alignment: *generalization ability*. \n\nWe consider two model classes (A and B) in the context of our incremental grouping task. cRNNs in Model Class A were *aligned* models that learned to solve the task. cRNNs in Model Class B (15x15 kernels, BPTT, 6 timesteps) also learned to solve the task (val. accuracy $\\geq 0.99$) but were *not aligned*. They did not capture narrow vs. wide or curved vs. straight effects. They also did not correlate significantly with human RT: $r =  -.08,  p  = .51$. Interestingly, when we tested both these model classes for strong generalization (experimental details in **SI 2**), the aligned model class showed superior average performance (**0.70** vs **0.61**).\n\nThough this result is anecdotal and needs a more thorough investigation, we believe that there are benefits for ML that accompany model temporal alignment.\n\n[R1] Fel et al. (2022) Harmonizing the object recognition strategies of deep neural networks with humans.\n\n[R2] Kubilius, Schrimpf et al. (2019) Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs.\n\n[R3] Dapello et al. (2020) Simulating a primary visual cortex at the front of CNNs improves robustness to image perturbation.\n"}, {"review_id": "5bUGQxRnce", "reviewer": "Reviewer_4B3C", "summary": "This paper introduces a new measure of visual artificial network computation 'time', $\\xi_{cRNN}$, as the time-averaged uncertainty of a convolutional RNN trained with an evidential deep learning loss (Sensoy et al.). The authors proceed to analyse the dynamics of this network as it solves a range of classification tasks. Importantly, beyond the network learning to solve the tasks, $\\xi_{cRNN}$ is generally well correlated with human reaction times, with which it also shares several qualitative features across tasks.", "questions": "L106: what does 'without loss of generality' mean here?\n\nL123 \"our control models lack inference-time flexibility.\" What does this mean?\n\nL144: It might be worth comparing to or discussing recent work by Bansal et al. (NeurIPS 2022), which develops a way of stably training convolutional RNNs with long computation times that generalize from small to large problem settings.\n\nL156 \"for a complete mathematical specification of this loss, refer to SI 1\". $\\epsilon$ does not seem to be defined in either the main text or SI beyond a qualitative description as 'uncertainty'. It would be good to provide a precise mathematical definition somewhere.\n\nL204: it would be good to briefly explain how this distance measure by Jeurissen et al. is defined (or at least refer to SI where it can be described in more detail).\n\nFigure 3b and 5c: it's not entirely clear to me how these 'activity maps' are computed?\n\nFigure 7b: it might be worth talking a bit more about the differences between the human and model data here instead of doing linear regression and calling it a day. The cRNN data is decidedly not linear and exhibits something resembling a plateau followed by a sharp drop at high discriminability. This seems rather different from the monotonic decrease in human RT, and the authors could perhaps speculate on or discuss this difference.\n\nSupplementary material: it would be worth having a separate folder with just a few example videos of each condition in addition to the larger set of videos currently provided.", "limitations": "The authors have adequately discussed the limitations of the work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Deep network models of the visual system have often focused purely on classification accuracy and neural representations when comparing to biological data. However, this neglects an important aspect of human behaviour, namely the time it takes to arrive at a decision. The present paper tackles this important question using recent ideas from the machine learning literature and represents an interesting step in the direction of capturing the temporal variability in human behaviour.\nThe authors analyse an impressive breadth of tasks and behavioural data and include many interesting analyses, both qualitative and quantitative.", "weaknesses": "The major weakness of the submission is that while there are strong _correlations_ between $\\xi_{cRNN}$ and human reaction times, there is less evidence that $\\xi_{cRNN}$ is mechanistically similar to human reaction times, since it is fundamentally a measure of uncertainty rather than computation time. In particular, human reaction times generally involve a tradeoff between computation/evidence accumulation and decision making (as in common drift diffusion models). On the contrary, the cRNN has a fixed computational budget and has no need or even capacity for evaluating this tradeoff. This is in contrast to a few previous deep learning models in the literature that are capable of explicitly trading off computation and actions (e.g. Graves et al., Pascanu et al., and refs by the authors in the ML literature, and Jensen et al. in the Neuro/Cogsci literature). These considerations are important because e.g. task difficulty is likely to correlate with both uncertainty and reaction time, and this raises the question of whether a model of one is automatically a model of the other.\n\nIt might the interesting to compare the current model to alternative models that more explicitly have adaptive/variable computation time, such as a cRNN that computes until $\\epsilon$ reaches a certain threshold (akin to classical drift diffusion models).\n\nAs the authors mention in L326, another potential weakness is that the present approach does not easily generalize beyond classification tasks, which form a small subset of the types of problems humans and animals are faced with during natural behaviour.\n\n_References:_ \\\nGraves et al.: \"Adaptive computation time for recurrent neural networks\", arXiv 2016.\\\nPascanu et al.: \"Learning model-based planning from scratch\", arXiv 2017.\\\nJensen et al.: \"A recurrent network model of planning explains hippocampal replay and human behavior\", bioRxiv 2023.\\\nBansal et al.: \"End-to-end Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking\", NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and useful comments. We offer clarifications, additional numerical analyses, and visualizations to address concerns raised in this review. We hope the reviewer sees value and effort in our responses and is willing to adjust their score.\n\n> **Fixed computational budget of the cRNN**\n\nThanks for allowing us to clarify this point. We draw an important distinction between a model having a fixed \"total\" budget versus a budget that can be expended *dynamically* in a stimulus-dependent manner. Consider the results presented in **Fig. R5**. Inspecting the step sizes of the recurrent dynamics (in a low-dimensional latent space) reveals that the cRNN utilizes its computational budget differently for  easy (**Fig. R5b**) versus hard (**Fig. R5c**) stimuli. The network practically *stops* (i.e., step size becomes small) after $t=t^*$, where $t^*$ is stimulus-dependent. This is unlike standard BPTT-trained cRNNs and shows how our cRNN computes dynamically and has the potential to evaluate the tradeoff between computation/evidence accumulation.\n\n> **Alternative models with explicit adaptive computational time**\n\nWe appreciate this suggestion. We now implement and include comparisons to cRNNs trained with ACT on the incremental grouping task. Here, we report quantitative results for one out of three well-performing (val. acc. $\\geq 0.95$)  runs. For the other two, we note that the results were qualitatively similar. None of these models showed a significant *positive* correlation between the ACT step count ($\\rho$) and human RT, $r= -.26, p=.04$. While for some (but not all), regressing $\\rho$ onto Euclidean distance yielded a significant positive linear slope, $b=0.07$, $SE=0.02$, $z=4.20$, $p<.001$, none of the models seemed to capture the effects of other factors known to affect human RT: narrow vs. wide, $d=0.38$, $t(21)=-1.82$, $p=.08$; curved vs. straight, $d=0.03$, $t(21)=-0.13$, $p=.90$; and A vs. B vs. C vs. D in **Fig. 2c**, $F(3,63)=1.84$, $p=.15$. \n\nWe hypothesize that despite ACT being a clever scheme for small networks, the fundamental bottleneck arises from the memory demands imposed by BPTT that limit step counts for cRNNs condemning any derived time metric to be coarse. We also note that these are only preliminary results and warrant further exploration.\n\n> **Beyond classification tasks**\n\nWe agree that this is a very promising direction for future research. Alternative forced choice tests comprise a dominant part of visual cognitive psychology paradigms for which researchers have amassed a wealth of human RT data. And thus, our motivation to start here. To the best of our knowledge, we present one of the early attempts to leverage RT metrics for alignment, and we are confident in expanding the scope of this work in the future.\n\n> **L106: what does 'without loss of generality' mean here?**\n\nWe have clarified this as follows:\n\nThe model of interest chosen in this work is the horizontal gated recurrent unit (hGRU; [19]), a convolutional recurrent neural network model that we will canonically refer to as cRNN throughout the rest of the paper. This choice does not imply any loss of generality, as our framework applies to any cRNN architecture.\n\n> **L123 \"our control models lack inference-time flexibility.\" What does this mean?**\n\nBy the *lack of inference-time flexibility*, we refer to the inability of our control models trained on easier task settings to generalize to arbitrarily harder task settings.\n\n> **Comparison to Bansal et al. (2022)**\n\nThanks for this reference. We will factor this into our revised discussion.\n\n> **Mathematical definition of $\\epsilon$**\n\nAgreed! We have added this in the description.\n\nStarting at **L41** in **S1.2**: “$S = \\sum_{j=1}^{K}\\alpha_j$ represents the \"total evidence\" accumulated by the model (or the Dirichlet strength). $D_{KL}$ is the Kullback–Leibler divergence measure. $\\boldsymbol{\\hat{\\alpha}}$ are the Dirichlet parameters of just the \"misleading\" evidence given by $\\boldsymbol{\\hat{\\alpha}} = \\mathbf{y} + (1-\\mathbf{y})\\boldsymbol{\\alpha}$. We define the instantaneous uncertainty as $\\epsilon = \\frac{K}{S}$.\n\n> **Distance measure from Jeurissen et al.**\n\nWe will add this description to the SI. \"Jeurissen et al. [41] describe a growth cone distance metric that is sensitive to the topology of the object, proximity of boundaries, and narrowness. The authors construct a series of non-overlapping, varying-sized discs whose centers trace a curve (originating at the fixation location and ending at the cue location). The radii are such that a disc is the maximum inscribed disc at its location. Disks never cross object boundaries. The final distance metric is defined as the total number of discs that constitute this curve.\"\n\n> **Computing the activity maps**\n\nThanks for the comment. We mention this in the main text (**L174**). We will repeat it in the figure caption to make it easier for the reader.\n\n\"First, we track cRNN dynamics by looking at its latent activity $\\mathbf{h_t}$ at every time step $t$, averaged across the channel dimension and smoothed by averaging $\\mathbf{h_t}$ and $\\mathbf{h_{t+1}}$.\"\n\n> **Clarification on Figure 7b**\n\nPoint well taken! We apply our cRNN on this scene categorization task to showcase the generality of our framework in extending to naturalistic stimuli. While the ability of our metric+cRNN to capture aspects of variations in scene discriminability is a promising sign, we acknowledge that there could be other cRNN architectures that are better suited for this particular task and are subsequently better aligned. We intended to propose our metric as a tool to precisely enable these sorts of comparisons.\n\n> **Re-organizing SI videos**\n\nThanks for this suggestion. We will keep your comment in mind to highlight representative examples (per-condition) in our project webpage (URL currently protected for double-blindness).\n"}]

Author rebuttal:
We thank the reviewers for their time in reading our manuscript and for their extensive feedback. In this general response, we address some common themes across the reviews. We provide detailed answers to specific reviewers' comments in subsequent responses. To go with this rebuttal, we also provide a PDF with additional figures labeled **Fig. R1-R5**, as well as an interactive version of our SU maps (URL supplied to the AC).

First, we sincerely thank the reviewers for their overwhelmingly positive feedback. All reviewers noted the importance and novelty of our work, as well as the thoroughness of our experiments.

One common critique across reviews pointed towards the need for a comparison across model architectures and alternative training frameworks.

We do believe that our framework is broadly applicable. And towards addressing questions raised by the reviewers, we perform and include several new analyses in this rebuttal. Specifically, 

1. To demonstrate the generality of our metric, we trained a convolutional LSTM (convLSTM) model on the scene categorization task and show that our metric can work with alternative model architectures as well (**Fig. R3**) (More details in our response to **JuhY**)

1. We *implemented* and made a direct comparison to a model that explicitly has adaptive computational time (ACT) trained on our incremental grouping task (as suggested by **4B3C**). More details in our response to **4B3C**.

1. We probe the generalization benefits of models exhibiting temporal alignment with humans (RTs). Specifically, we demonstrate that poorly-aligned models have a propensity to fail at out-of-distribution generalization while well-aligned models are consistently good in their generalization performance. We refer to our response to **fhSL** for more details in this regard.

We hope you agree that our manuscript has improved through your feedback and that our findings will have a significant impact on computational cognitive neuroscience and machine learning.


Meta-review: The paper proposes a recurrent vision model that accounts for human reaction times in visual decision making. All reviewers agree that it is a strong paper that makes a valuable contribution. One issue raised in the reviews and discussion was that the metric the paper proposes is more akin to uncertainty than reaction time, but this issue does not appear to undermine the overall value of the paper but rather provides some food for thought and future work.
Decision: Accept (spotlight)