SlowGuess/NeurIPS_2023_OCR huggingface dataset:
First, Jeurissen et al. [47] quantified their stimuli on an alternative dot distance measure that went beyond Euclidean distance to reflect the other aforementioned factors and was more predictive of human RT (see SI §F). We found this measure to be a better predictor of  $\xi_{cRNN}$  ( $r = .80$ ) as well (versus  $r = .45$  for Euclidean distance),  $t(63) = 4.68, p < .001$ . Second, the stimulus set featured distinct experimental conditions, varying in expected difficulty based on the position of the dots (Fig. 2c left). As shown in Fig. 2c (middle), the effects of this manipulation on  $\xi_{cRNN}$  were qualitatively aligned with those on human RT. A one-way repeated measures ANOVA across 22 outline stimuli revealed a significant effect of "Condition" on  $\xi_{cRNN}$ ,  $F(3,63) = 67.65, p < .001$ , as well as on the average RTs from [47],  $F(3,63) = 16.95, p < .001$ . Furthermore, Fig. 2c (middle) presents the results of planned contrasts (paired t-tests).  $\xi_{cRNN}$  was significantly higher for B compared to A, Cohen's  $d = 2.00$ ,  $t(21) = 9.39$ ,  $p < .001$ , but also for the more difficult Condition C compared to B,  $d = 0.73$ ,  $t(21) = 3.42$ ,  $p = .003$ . Third, we observed a significant overall correlation between human RT and  $\xi_{cRNN}$  on Yes-stimuli,  $r = .31$ ,  $p = .01$ . Fourth, on a novel test set of our own, paired t-tests showed that  $\xi_{cRNN}$  was significantly higher when two dots on the same object were placed in narrow versus wide object regions,  $d = 0.70$ ,  $t(22) = 3.34$ ,  $p = .003$ , and when the path was curved (i.e., longer topological distance) versus when it was not,  $d = 0.98$ ,  $t(22) = 4.71$ ,  $p < .001$  (see SI §G, Fig. S5). Finally, SI §G also describes an experiment in which we varied the distance between one dot and the object boundary while keeping the distance between the dots themselves constant. We found that our model exhibits higher values near the boundaries, which is strikingly similar to human behavioral results presented in [47].
Ambiguous stimuli. As a final way of gaining insight into the cRNN's task behavior, we looked at those stimuli in the validation set that elicited the highest  $\xi_{cRNN}$  values. We observed that some were characterized by uncertainty curves that remain high, even at  $t = T$ , suggesting the model deems them ambiguous. In a test set we designed to be ambiguous (it features occlusions), we frequently observed such non-decreasing uncertainty curves too (see SI §H). We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.
Taken together, these results suggest that our metric is able to capture the dynamical signatures of object-based attention in our cRNNs and provides a means to quantify alignment with human visual strategies.
5 Visual Simulation - Planko
5.1 Task and Stimuli
The Planko task (and corresponding dataset) was introduced in [53] and features stimuli  $(64\times 64\mathrm{px};$ $80K$  training,  $5K$  validation) displaying a ball about to fall down and eventually land in one of two baskets ("left"/"right"). The outcome depends on the position and orientation of a set of planks placed in between the ball and the baskets (Fig. 4a). Humans are thought to solve this task through mental
b
Figure 5: cRNN learns filling-in strategy to solve mazes. a. Task description. b. The cRNN model takes more time to solve mazes with longer path lengths, an effect previously found in human participants too. c. Latent activity visualizations for two yes-mazes. The cRNN gradually fills the segment containing the cues. The strategy can be appreciated even better in the videos supplied in the SI. d. Uncertainty curves for the two inputs shown in Panel c. The cRNN remains uncertain for much longer for the maze featuring the longest path. The uncertainty evolution is visualized dynamically in conjunction with the change in latent activity in the supplementary videos.
visual simulation [53]. Notably, feedforward CNNs struggle to solve the task [54] and when they do, they appear to adopt non-aligned strategies to humans [53]. In contrast, cRNNs show evidence of simulation-like behavior on a variant of the task [54]. With our metric, we are now able to directly compare human (collected by [53]) and cRNN RTs. To this end, we trained a cRNN from scratch on Planko (see Section 3.2 and SI §A for details on training).
The stimuli displayed in Fig. 4 illustrate the challenge of the Planko task. Our cRNN achieves a performance of  $84\%$  on the validation set, well within the confidence interval of human accuracy [53]. Stimuli presented to human participants in the original study  $(N = 200)$  were used here as a held-out set. We observed a significant correlation between model  $\xi_{cRNN}$  values and human RTs,  $r = .31, p < .001$  (Fig. 4b). This result indicates that stimuli that were harder for humans tended to be computationally challenging for the cRNN (see examples in Fig. 4c).
What makes a stimulus challenging? Variance in human RTs is known to be partially explained by the statistics of the planks. For instance, stimuli for which the outcome is more likely to change as a consequence of perturbing the plank configurations by a small amount were deemed "uncertain". Indeed, participants responded slower to these stimuli [53]. We were able to replicate this result in our cRNN,  $r = .31$ ,  $p < .001$ . These results showcase how our methodology can be used to make stimulus-dependent RT predictions. This could help shed light on, for example, the remarkable yet under-examined ability of humans to flexibly adopt different strategies in a Planko task depending on the stimulus demand.
6 Mazes
6.1 Task and Stimuli
We devised a maze task where the goal is to tell whether two cued locations are connected by a path (Fig. 5a). To build our training  $(N = 34.5K)$  and validation  $(N = 14K)$  set, we started from mazes provided by [40] but modified them to make them suitable for classification (the originals did not feature any disconnected paths). The cues were placed such that the two classes ("yes"/"no") did not differ in Euclidean cue distance. Each maze was  $144 \times 144$  px.
Here too, we observed that the uncertainty curves and derived  $\xi_{cRNN}$  values differed considerably depending on the input, even if nearly all mazes were classified correctly at  $t = T$  (99% correct
b
Figure 6:  $\xi_{cRNN}$  is predictive of the discriminability of a visual scene. a. Task description. b. Human RTs (collected in [57]) decrease as a function of discriminability, as do  $\xi_{cRNN}$ . Highly discriminable scenes are easier for both human participants and the cRNN model. Error bars represent the standard error of the mean.
on validation mazes). For the Yes-mazes contained in a random subset of the validation set, we found that the model spent more time being uncertain (higher  $\xi_{cRNN}$ ) when the length of the path between the two cued locations was longer,  $r = .84$ ,  $p < .001$  (Fig. 5b,d). Moreover, path length was a significantly better predictor than simple Euclidean distance ( $r = .39$ ),  $t(234) = 5.81$ ,  $p < .001$ . Similarly, human RTs on maze tasks have been shown to increase with path length [55]. When the cued locations were on two distinct maze segments (No-mazes),  $\xi_{cRNN}$  was significantly predicted by the mean of both segment lengths,  $r = .56$ ,  $p < .001$ . The kind of visual strategy that produced such distance effects can be understood by inspecting the visualizations of the latent activities (Fig. 5c). The model gradually spread activity along the maze segment(s) containing the cues. We strongly encourage the reader to watch our videos (see SI §I) to better appreciate this spread in conjunction with the evolution of uncertainty over time. Finally, we did not find evidence for an effect on  $\xi_{cRNN}$  of the number of T-junctions on the path, which are known to predict human RT [56]. We speculate that this potential misalignment, worth investigating in future work, could be due to the model not being restricted in how many possible paths it can explore at once.
7 Scene Categorization
7.1 Task and Stimuli
To showcase how our general approach can also be applied to study higher-level visual tasks, we consider a natural versus manmade scene categorization task as our final challenge (Fig. 6). Human RT data from a rapid-scene categorization version of this task were already available from [57]. We selected images from the SUN database [58] querying the same classes as in [57], but excluding the images shown to human participants, and performed a stratified train ( $N = 8.8K$ ) and validation ( $N = 979$ ) split. The image preprocessing pipeline entailed cropping the image to be square, resizing to  $150 \times 150$  px, converting to grayscale, and random horizontal flipping. In addition, the images were matched for overall brightness.
The model was able to correctly classify  $83\%$  of the validation images and  $73\%$  of the held-out test images from [57]. The errors can be understood, at least partly, in light of the less-than-clear-cut labels (e.g., an islet with a house on top, a building behind many trees). Each of the test images had previously been assigned a discriminability score [57], based on the average error rate of logistic classifiers trained on "gist" features [59]. Human participants were reported to be slower with decreasing discriminability [57]. Here, we show a similar trend in the model (Fig. 6b), with a linear regression analysis revealing a significant, negative discriminability slope,  $b = -0.47$ ,  $SE = 0.03$ ,  $t = -16.45$ ,  $p < .001$ . Directly comparing  $\xi_{cRNN}$  to human RT, we found them to be significantly correlated,  $r = .19$ ,  $p < .001$ . While our cRNN model of choice here does not perfectly account for empirical patterns in human reaction times, specifically in their non-monotonic reaction time readouts for the low-discriminability stimuli, they provide a promising
start. Our framework is model-agnostic (e.g., see SI §J for results on a convLSTM) and thus can be used to test hypotheses on architectural improvements.
We are in the midst of a paradigm shift in visual neuroscience. The continued development of computational models that rival human behavioral performance has called for tools to quantify how well the internal mechanisms of such models compare to biological truths. These efforts have demonstrably been beneficial for both the AI and neuroscience ecosystems.
In this work, we push this frontier by introducing a framework to characterize the evidence accumulation strategies of stable recurrent vision models. Our experiments thus far show the potential to generate novel hypotheses of interest to visual neuroscientists. The most direct application of our framework for experimental design is in taxonomizing stimuli in terms of the putative computational demand based on our metric toward understanding neural processes at varying timescales. Furthermore, the value proposition of this framework includes the ability to adjudicate between models based on the consistency of their latent dynamics with human reaction time data. For example, our metric from a cRNN trained on the maze task is uncorrelated with the number of T-junctions in the stimulus, indicating a potential divergence between serial and parallel exploration strategies (SI §I). It is worth noting that distinguishing between serial and parallel attention models has been of long-standing interest [60]. Similarly, testing a zoo of hierarchical models on the scene categorization task while keeping an eye on the degree of alignment with our metric can potentially lend novel mechanistic insights and help delineate bottom-up, lateral, and top-down processes [61]. We are also enthusiastic about drawing theoretical parallels between our metric summarizing the cRNN's evidence accumulation strategy and the complexity of natural data statistics toward understanding the sample efficiency of learning in models and primates.
Limitations and an outlook for the future. Within the scope of this work, we have considered four disparate visual cognition tasks to showcase how our approach offers the means to study the temporal alignment between recurrent vision models and the human visual system. Of course, in the face of the myriad of tasks that cognitive scientists have used over centuries, our selection still needs to be improved. Our results do not directly speak to the diversity in response types, stimulus types, or naturalism, to name a few. However, our approach is general-purpose enough to adapt to newer tasks as long as they are cast as a classification problem. We believe this should cover a significant share of cognitive science and AI tasks. Extending the approach beyond choice tasks would require a considerable revision of how to make the model's uncertainty explicit. Another limitation is that there could easily be other cRNNs that align with human RTs better. While we motivate the choice for the cRNN in Section 3.1, our main contribution here is to introduce the overall framework integrating stable recurrent vision models with evidential learning theory to support the derivation of a novel RT metric. A comprehensive comparison of different models, apart from those in SI §B, was hence beyond this work's scope. However, we are confident our framework paves the way to build a taxonomy of models in the future as well as generate testable hypotheses for neuroscience.
Broader impacts. With AI vision models becoming ubiquitous in our daily lives, the call to make them fairer, transparent, and robust is growing louder. Our framework facilitates this by allowing researchers to peek into the inner working of these AI models for transparency and thinking about their match to biological systems, which are arguably more robust. From a neuroscience perspective, computational models of RT are gaining prominence in computational psychiatry and hold the key to understanding many neurobiological disorders [62, 63]. We generally do not anticipate any negative impact on society in either scenario.
The authors would like to explicitly thank Jeurissen et al. [47] and Sofer et al. [57] for generously sharing their stimuli and human reaction time data, and Bansal et al. [40] for their maze stimuli. We are also grateful to Drew Linsley for his helpful comments and feedback. This work was supported by ONR (N00014-19-1-2029), and NSF (IIS-1912280). Additional support provided by the Carney
Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the computing hardware resources supported by NIH Office of the Director grant S10OD025181.
A.1 Model Implementation
We instantiate our cRNN as a canonical recurrent convolutional neural network layer inspired by the primate visual cortex's anatomical and neurophysiological properties. This model, termed hGRU and first introduced in Linsley et al. [19], implements an hypercolumnar organization with distinct excitatory and inhibitory subpopulations and local- and long-range feedback mechanisms. For the sake of completeness, we outline the governing equations of the model below.
First compute suppressive interactions:
$$
\mathbf {G} ^ {S} = \operatorname {s i g m o i d} \left(U ^ {S} * \mathbf {H} [ t - 1 ]\right) \quad \# \text {C o m p u t e s u p p r e s s i o n g a t e a s f u n c t i o n o f c u r r e n t s t a t e}
$$
$$
\begin{array}{l} \mathbf {C} ^ {S} = B N \left(W ^ {S} * \left(\mathbf {H} [ t - 1 ] \odot \mathbf {G} ^ {S}\right)\right) \quad \# \text {A p p l y i n g t h e g a t e} \\ \mathbf {S} = \left[ \mathbf {Z} - \left[ (\alpha \mathbf {H} [ t - 1 ] + \mu) \mathbf {C} ^ {S} \right] _ {+} \right] _ {+}, \quad \# \text {A d d i t i v e a n d m u l t i p l i c a t i v e s u p p r e s s i o n o f} \mathbf {Z} \\ \end{array}
$$
Then compute the facilitation terms followed by the recurrent update:
$$
\begin{array}{l} \mathbf {G} ^ {F} = \operatorname {s i g m o i d} \left(U ^ {F} * \mathbf {S}\right) \quad \# \text {F a c i l i t a t i o n g a t e} \\ \mathbf {C} ^ {F} = B N \left(W ^ {F} * \mathbf {S}\right) \quad \# \text {F a c i l i t a t i o n} \\ \tilde {\mathbf {H}} = \left[ \nu (\mathbf {C} ^ {F} + \mathbf {S}) + \omega (\mathbf {C} ^ {F} * \mathbf {S}) \right] _ {+} \quad \# \text {M o d u l a t i o n o f} \mathbf {S} \text {t h r o u g h a d d i t i v e a n d m u l t i p l i c a t i v e f a c i l i t a t i o n} \\ \mathbf {H} [ t ] = \left(1 - \mathbf {G} ^ {F}\right) \odot \mathbf {H} [ t - 1 ] + \mathbf {G} ^ {F} \odot \tilde {\mathbf {H}} \quad \# S t a t e u p d a t e \\ \end{array}
$$
In these equations,  $\mathbf{H},\mathbf{Z}\in \mathcal{R}^{X\times Y\times C}$  represent the cRNN's latent state and bottom-up feedforward drive, respectively, with height/width/channels  $X,Y,C$ .  $W^{S},W^{F}\in \mathbb{R}^{E\times E\times C\times C}$  are learnable kernels controlling the suppressive and facilitatory interactions and  $E$  is the spatial width of the lateral interactions. For all our experiments, we set  $E = 5$ , except for Planko where  $E = 3$ . Similarly,  $U^S,U^F\in \mathbb{R}^{1\times 1\times C\times C}$  are learnable kernels controlling the gates. Model dynamics are discretized, with  $t\in [1\dots T]$  representing the timesteps of processing. We use softplus as our rectifying nonlinearity (denoted by  $[\cdot ]_{+})$ . We also use Batch Normalization (BN) in the model to control exploding/vanishing gradients [64]. BN kernels are shared across timesteps of processing.
A.2 Training Implementation Details
Leveraging equilibrium dynamics for training. Training cRNNs has traditionally posed a problem due to the severe memory bottlenecks imposed by the backpropagation through time (BPTT) algorithm. To remedy this, Linsley et al. [8] introduced a variant of the classic recurrent backpropagation algorithm (RBP) that leveraged equilibrium dynamics to compute model gradients with  $O(1)$  memory complexity. We outline the high-level idea here and refer the reader to [8] for more details.
Our discretized cRNN dynamics can be written as follows,
$$
\mathbf {H} [ t ] = f (\mathbf {H} [ t - 1 ], \mathbf {X}; \mathbf {W}),
$$
where  $f(.)$  is the model architecture described,  $\mathbf{H}[t]$  is the cRNN's state at timestep  $t$ , and  $\mathbf{X}$  is the bottom-up sensory input. For convenience, we use  $\mathbf{W}$  to denote all the trainable parameters of  $f(.)$ . If  $f(.)$  is a contraction mapping, then  $\mathbf{H}[t] \to \mathbf{H}^*$  as  $t \to \infty$ . One can define an implicit function as follows to aid in the gradient computation:  $\psi(\mathbf{H}; \mathbf{X}, \mathbf{W}) = \mathbf{H} - f(\mathbf{H}, \mathbf{X}; \mathbf{W})$ . By construction, we can observe that  $\psi(\mathbf{H}^*; \mathbf{X}, \mathbf{W}) = 0$  and by extension,
$$
\frac {\partial \psi (\mathbf {H} ^ {*} ; \mathbf {X} , \mathbf {W})}{\partial \mathbf {W}} = 0
$$
Following a sequence of algebraic manipulations we can compute the gradient of the equilibrium state  $\mathbf{H}^*$  w.r.t. parameters  $\mathbf{W}$  as follows.
$$
\frac {\partial \mathbf {H} ^ {*}}{\partial \mathbf {W}} = (I - J _ {f, \mathbf {H} ^ {*}}) ^ {- 1} \frac {\partial f (\mathbf {H} ^ {*} , \mathbf {X} ; \mathbf {W})}{\partial \mathbf {W}},
$$
where  $J_{f,\mathbf{H}^*}$  is the Jacobian of the cRNN  $f(.)$  evaluated at  $\mathbf{H}^*$ . The ability to compute this gradient, however, relies on  $(I - J_{f,\mathbf{H}^*})$  being invertible. To guarantee this, Linsley et al. [8] introduced the Lipschitz Coefficient Penalty (LCP) as  $\mathcal{L}_{LCP}(\mathbf{W}) = \| (\mathbf{1}\cdot J_{f,\mathbf{H}^*} - \lambda)^+ \|_2$  which controls the degree of contraction in  $f(.)$  and by extension the invertibility of the term discussed above. Moreover, this regularization can be added to any task loss which we exploit.
Evidential learning formulation. To support our goal of characterizing evidence accumulation strategies in our cRNN, we parameterize our model readout to explicitly express uncertainty. Specifically, we follow the evidential learning strategy introduced in Sensoy et al. [30] to interpret cRNN readouts as the parameters of the Dirichlet density on the class predictors,  $\alpha = < \alpha_{1},\dots,\alpha_{K}>$ , where  $K$  is the total number of classes in the classification problem of interest. As in [30], our evidential loss comprises of two components, one that supports belief accumulation for the "correct" class  $(\mathcal{L}_{risk})$  and a second that ensures that the beliefs for all the other "wrong" classes have uniform mass  $(\mathcal{L}_{balance})$ .
$$
\mathcal {L} _ {E D L} (\mathbf {W}) = \sum_ {i = 1} ^ {i = N} \mathcal {L} _ {r i s k} ^ {(i)} + \rho \mathcal {L} _ {b a l a n c e} ^ {(i)} (\mathbf {W})
$$
$$
\mathcal {L} _ {r i s k} (\mathbf {W}) = \sum_ {j = 1} ^ {j = K} (y _ {j} - \hat {p} _ {j}) ^ {2} + \frac {\hat {p} _ {j} (1 - \hat {p} _ {j})}{S + 1}
$$
$$
\mathcal {L} _ {\text {b a l a n c e}} (\mathbf {W}) = D _ {K L} \left(D (\mathbf {p} | \hat {\boldsymbol {\alpha}}) \mid \mid D (\mathbf {p} | <   1, \dots , 1 >)\right)
$$
In the above,  $\mathbf{y}$  denotes a one-hot ground truth label for a given sample with  $y_{j} = 1$  if the sample belongs to class  $j$ , and 0 otherwise.  $\hat{p}_j$  is from the model that denotes the expected probability of the  $j$ -th singleton under the corresponding Dirichlet such that  $\hat{p}_j = \frac{\alpha_j}{S}$ .  $S = \sum_{j=1}^{K} \alpha_j$  resembles the "total evidence" accumulated by the model (or the Dirichlet strength).  $D_{KL}$  is the Kullback-Leibler divergence measure.  $\hat{\alpha}$  are the Dirichlet parameters of just the "misleading" evidence given by  $\hat{\alpha} = \mathbf{y} + (1 - \mathbf{y})\alpha$ . We define the instantaneous uncertainty as  $\epsilon = \frac{K}{S}$ .
Finally,  $\rho$  is an annealing term that is used to balance the relative importance of  $\mathcal{L}_{\text{risk}}$  and  $\mathcal{L}_{\text{balance}}$ . In practice, we use the following increment schedule:  $\rho_{\text{epoch}} = \min \left[1, \frac{\text{epoch}}{\tau}\right]$ , where  $\tau$  is a hyperparameter. For a complete derivation of the loss formula, we refer the reader to [30]. We choose  $\tau = 16$  for incremental grouping,  $\tau = 10$  for Planko,  $\tau = 20$  for the maze task, and  $\tau = 16$  for scene categorization.
Training objective. Our final training objective is thus  $\mathcal{L}(\mathbf{W}) = \mathcal{L}_{EDL}(\mathbf{W}) + \gamma \mathcal{L}_{LCP}(\mathbf{W})$ .
B Generalization Experiments
B.1 Task and Stimuli
In the main text, we identified the flexibility to generalize to more complex task settings at inference as a critical desideratum for obtaining a human-like RT metric from any model. We tested generalizability in the cRNN as well as in a set of control models (see further) using the incremental grouping task described in the main text (dots on the same object “yes”/“no”?). We trained on an easy version of the task by only selecting training stimuli for which the Euclidean dot distance was at most 44 px ( $50^{\text{th}}$  percentile). We refer to this reduced training set as “short training stimuli” ( $N = 170K$ ). For evaluation, we created two subsets of the validation set, one easy and one hard. The easy set (“short validation stimuli”,  $N = 7.7K$ ) consisted of those validation stimuli for which the dots were at most 44 px apart. The hard set (“long validation stimuli”,  $N = 1.5K$ ) consisted of those stimuli for which the dots were more than 82 px apart ( $90^{\text{th}}$  percentile). All thresholds were computed across the training stimuli in the original set.
B.2 Models and Training
All models listed below were trained from scratch with a batch size of 128 and a standard cross entropy loss. Training was done in a data-parallel manner on four Nvidia RTX GPUs (Titan/3090/A5000) with 24GB of memory each.