SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lore Goetschalckx\*1,2, Lakshmi N. Govindarajan\*3, Alekh K. Ashok\*1,2, Aarit Ahuja4, David L. Sheinberg4, and Thomas Serre1,2
$^{1}$ Department of Cognitive, Linguistic, & Psychological Sciences, Brown University, Providence, RI 02912
$^{2}$ Carney Institute for Brain Science, Brown University, Providence, RI 02912
{lore_goetschalckx,alekh_karkada_ashok,thomas_serre}@brown.edu
$^{3}$ Integrative Computational Neuroscience (ICoN) Center, MIT, Cambridge, MA 02319
lakshmin@mit.edu
<sup>4</sup>Neuroscience Department, Brown University, Providence, RI 02912
{aarit_ahuja,david_sheinberg}@brown.edu
The meteoric rise in the adoption of deep neural networks as computational models of vision has inspired efforts to "align" these models with humans. One dimension of interest for alignment includes behavioral choices, but moving beyond characterizing choice patterns to capturing temporal aspects of visual decision-making has been challenging. Here, we sketch a general-purpose methodology to construct computational accounts of reaction times from a stimulus-computable, task-optimized model. Specifically, we introduce a novel metric leveraging insights from subjective logic theory summarizing evidence accumulation in recurrent vision models. We demonstrate that our metric aligns with patterns of human reaction times for stimulus manipulations across four disparate visual decision-making tasks spanning perceptual grouping, mental simulation, and scene categorization. This work paves the way for exploring the temporal alignment of model and human visual strategies in the context of various other cognitive tasks toward generating testable hypotheses for neuroscience. Links to the code and data can be found on the project page: https://serre-lab.github.io/rnn_rts_site/.
The symbiosis between visual neuroscience and machine learning has created a distinctive and unprecedented phase in the field of vision. On the one hand, formidable advances in machine learning have supported the development of computational models of visual cognition that rival, and in some cases surpass, human-level performance on naturalistic tasks [1-5]. On the other hand, incorporating computational and neurophysiological constraints from biological visual systems has yielded significant benefits for AI models in terms of their performance [6, 7], efficiency [8], and robustness [9-11]. Additionally, performant computational vision models that operate directly on high-dimensional sensory inputs have shown great promise for scientific discovery through the generation of testable hypotheses for neuroscience [12, 13]. At the core of this synergistic loop lies the need for reliably quantifying the degree of "alignment" between models and the primate visual system, which can subsequently serve as a control knob for closed-loop hypothesis testing.
The notion of "alignment" is subjective and can be operationalized at varying levels of analysis [14]. One of the most prominent alignment techniques is to quantify the degree to which model feature
representations predict time-averaged firing rates of single neurons across various stimuli [11, 15-17]. In contrast, computational-level alignment analyses typically use axiomatic attribution methods [18] to extract minimal sub-circuits from larger models and probe them to gain mechanistic insights [12]. A third popular approach to alignment is at a higher behavioral level, where techniques focus on understanding the extent to which models and primates utilize similar "visual strategies". Top-down attention masks, diagnostic of human observers' visual strategies, are a commonly employed axis of comparison in this type of model alignment [7, 19]. At the same time, other behavioral measures such as object confusion patterns [20, 21] or human-derived pairwise image similarity ratings [22] are also widely considered effective quantities that can form the basis of alignment.
Each of these techniques has a common tenet: their purview is limited to measures of alignment that are global, time-independent summaries of neural or behavioral states. While this may be sufficient for task paradigms primarily involving feedforward processing, a large part of visual cognition is highly dynamic [23-25]. Understanding visual computations, thus, necessarily involves characterizing and modeling the time-varying properties of neural responses to complex and varied inputs. Why, then, is there a dearth of alignment methods focusing directly on the level of visual processing dynamics? We posit a two-fold bottleneck. First, there is a need for more performant models that exhibit and mimic visual cortical dynamics at scale. Second, we lack the mathematical tools to extract meaningful dynamical signatures from such models and compare them with biological systems. While recent progress in biologically-inspired machine learning has started to open up solutions for the former [8, 19, 26-28], the latter remains a wide-open research area.
Contributions. In this study, we consider this temporal alignment perspective. Recent advances in training biologically motivated large-scale convolutional recurrent neural network models (cRNNs henceforth) on visual cognition tasks [8] have provided candidate models to study the temporal alignment problem. A cRNN's inherent design affords access to its time-varying dynamical properties that can be explicitly analyzed. Throughout this paper, we specifically focus on extracting a metric that captures a cRNN's dynamical signature and on quantifying the ability of this metric to explain primate decision time courses in the form of reaction times (RTs) in concomitant visual psychophysics tasks.
- We introduce a novel computational framework to train, analyze, and interpret the behavior of cRNNs on visual cognitive tasks of choice. Our framework triangulates insights from an attractor dynamics-based training routine [8] and evidential learning theory [29] to support stable and expressive evidence accumulation strategies.
- We derive a stimulus-computable, task- and model-agnostic metric to characterize evidence accumulation in cRNNs. Our metric does not require extra supervision and purely leverages internal cRNN activity dynamics.
- We comprehensively demonstrate the efficacy of our metric in capturing stimulus-dependent primate decision time courses in the form of reaction times (RTs) in four disparate visual cognitive challenges that include incremental grouping, mental simulation, and scene categorization. To the best of our knowledge, this is the first demonstration of qualitative temporal alignments between models and primates across task paradigms.
Cognitive process models. The speed-accuracy tradeoff in human behavior is ubiquitous in cognitive science [31] and has garnered much attention from the modeling perspective. The most prominent of them is the Drift-Diffusion Model (DDM) family, which specifies RTs by determining when the amount of accumulated evidence for a behavioral choice from a noisy process reaches a threshold value [32, 33]. Though such models have had phenomenal successes in aligning neural and behavioral data [34], they are typically not stimulus-computable or rely on handcrafted heuristics [35]. Moreover, such models are usually directly fitted to RT data [35, 36], unlike our approach, where RT is a derived quantity without any additional supervision.
Computing reaction-time proxies from neural network models. Several notable efforts have attempted to derive RT measures from neural network models to serve as a proxy for "computational cost" [26, 37-42]. Spoerer et al. [26] introduced the notion of energy demand in their objective function that motivated parsimony regarding the number of computations the cRNN executes. Bansal
Figure 1: Computing a reaction time metric from a recurrent vision model. a. A schematic representation of training a cRNN with evidential deep learning (EDL; [30]). Model outputs are interpreted as parameters  $(\alpha)$  of a Dirichlet distribution over class probability estimates, with higher values (symbolized here by a darker gray) reflecting more generated evidence in favor of the corresponding class. In this framework, the width of the distribution signals the model's uncertainty  $(\epsilon)$  about its predictions. b. Visualization of our metric,  $\xi_{cRNN}$ , computed for an example stimulus (see Panel a) used in the task studied in Section 4. The metric, denoted  $\xi_{cRNN}$ , is defined as the area under the uncertainty curve, i.e., the evolution of uncertainty  $(\epsilon)$  over time.
et al. [40] discussed "algorithmic extrapolation" where their networks solved problems of greater difficulty than those in their training set by performing additional computations. Graves [37] proposed the use of a "halting unit" whose activation level determined the probability of continuing model computations. Finally, Figurnov et al. [43], motivated by [37], adaptively determined the spatial extent and number of processing layers in a deep convolutional network. In a broader sense, "conditional computing" approaches aim to learn dropout-like policies [44, 45] to train neural networks that can balance parsimony and performance. While the approaches presented in [37, 43-45] are impressive advances, they neither operate explicitly with cRNNs nor yield RT values for downstream comparisons on visual cognitive tasks. The elegant approach of [26] suffers from the primary drawback that it relies on the backpropagation through time (BPTT) algorithm. BPTT imposes high memory demands, limiting the number of timesteps a cRNN can perform and therefore condemning any derived measure to be coarse. Furthermore, vanilla BPTT forces the dynamics to be constant (i.e., the cRNN arrives at a solution at  $t = T$  for any input) making it non-stimulus-adaptive [8].
Probabilistic interpretation of model uncertainty. Decision certainty and computational parsimony are duals of each other. In other words, easier tasks require fewer computational resources, with models typically being most certain about their predictions, while the opposite holds for harder tasks. Modeling decision uncertainties provide another effective angle to capture the "computational demand" of a stimulus condition. While deterministic neural networks suffer from ill-calibrated prediction confidence in Softmax outputs, Bayesian neural networks derive prediction confidence from their weight uncertainties but are generally not performant. Inspired by the Dempster-Shafer theory of evidence [29], Sensoy et al. [30] propose a formulation that treats model outputs as parameters of a Dirichlet distribution that signifies the total amount of evidence accumulated for each class. Although influential, this body of work, called Evidential Deep Learning (EDL), has never been linked to RT measures or applied to visual cognitive tasks. We refer the interested reader to Ulmer et al. [46] for a comprehensive survey of this literature. In this paper, we specifically leverage EDL in conjunction with attractor dynamics to support the derivation of an RT proxy.
3 General Methods
3.1 Models
cRNN. The model of interest chosen in this work is the horizontal gated recurrent unit (hGRU; [19]), a convolutional recurrent neural network model that we will canonically refer to as cRNN throughout the rest of the paper. In principle, our framework applies to other cRNN architectures as well. The hGRU is inspired by the anatomical and neurophysiological properties of the primate
a
TASK
No
Are the dots on the same object?
Yes
Figure 2: Human versus cRNN temporal alignment on an incremental grouping task. a. Description of the task (inspired by cognitive neuroscience studies [47]). b. Visualization of the cRNN dynamics. The two lines represent the average latent trajectories across  $1K$  validation stimuli labeled "yes" and "no" respectively. Marker size indicates average uncertainty  $\epsilon$  across the same stimuli. Evident from the graph is that the two trajectories start to diverge after some initial time passes, clearly separating the two classes. Owing to the C-RBP training algorithm and attesting to the dynamics approaching an equilibrium, the step sizes become increasingly small over time. We also include snapshots of the latent activity in the cRNN for two example stimuli (one "yes", one "no"; see Panel a) along the respective trajectory. Each snapshot represents  $\mathbf{h}_{\mathbf{t}}$  at time step  $t$ , averaged across the channel dimension and smoothed by averaging  $\mathbf{h}_{\mathbf{t}}$  and  $\mathbf{h}_{\mathbf{t} + 1}$ . Notice the spread of activity over time. The cRNN gradually fills the segment containing the dots. The strategy can be appreciated even better in video format (see SI §D). c. Comparison against data from the experiment with human participants in [47]. The position of one dot (white, labeled "Fix") was kept constant, while the position of the other was manipulated to create experimental conditions of varying difficulty. These manipulations have qualitatively similar effects on the cRNN as they do on human behavior. Error bars represent the standard error of the mean across stimuli. The significance levels for shown contrasts were adjusted downward from .1 (*), .05*, .01**, and .001*** using Bonferroni. The spatial uncertainty map shown on the right visualizes the spatial anisotropy observed in the model for the same example stimulus shown on the left.
visual cortex. Architectural facets of the hGRU include an hypercolumnar organization, distinct excitatory and inhibitory subpopulations, and local- and long-range feedback mechanisms, rendering it overcomplete and an ideal candidate cRNN for our purposes. The hGRU and its extensions have previously demonstrated proficiency in learning long-range spatial dependency challenges, including naturalistic ones [8, 19, 48]. For implementational details, refer to SI §A.
Control models. To further motivate our cRNN choice, we refer to its ability to systematically generalize to more challenging task settings after being trained on less challenging ones [8]. Inference-time flexibility and performance are critical desiderata, given that our goal is to probe the computational demands of a task via stimulus manipulations. We first demonstrate that the cRNN's performance remained well above chance on exceedingly challenging versions of the first task we consider (see Section 4.1) after training on a simplified version (see SI §B for full experimental details). We trained pure feedforward convolutional neural networks (CNNs) as control models under identical training
data distributions. The CNNs struggled to generalize effectively, as did a naive control RNN model (Fig. S1). Because CNNs lack explicit temporal dynamics and our recurrent control models failed our generalization test, we limit the experimental results presented in this paper to our cRNN. For completeness, we remark that a Vision Transformer [49] passed the generalization test. However, we exclude these models from our analyses for two reasons. First, as such, Vision Transformers lack biological analogs. Second, our focus in this work is to derive RT measures from model dynamics, which transformers do not exhibit.
3.2 Training Framework
General remarks. All model runs reported herein were done from scratch using only the ground truth class labels for supervision. Any alignment between the cRNN model and humans would have had to emerge as a byproduct of task constraints. We emphasize that no human RT data were involved in our training routines. For the cRNNs, we computed a task loss only at the network's end state to preserve the ability to remain flexible during inference. The cRNNs were trained in a data-parallel manner on 4 Nvidia RTX GPUs (Titan/3090/A5000) with 24GB of memory each. Each training run took approximately 48 hours, except those in Section 7 (2 hours), which were faster due to the smaller training dataset size. They were furthermore trained with Adam [50], a learning rate of  $1e - 3$ , and  $\gamma = 100$  for the C-RBP penalty ([8]; also see SI §A). We use  $T = 40$  recurrent timesteps during the training phase, with an exception for the task in Section 6, where  $T = 80$  and  $\gamma = 1000$ .
Stabilizing cRNNs. We train our cRNN model with the contractor recurrent back-propagation algorithm (C-RBP; [8]). C-RBP imposes an attractor-dynamic constraint on the cRNN's computational trajectory, imparting several benefits over the standard BPTT algorithm. C-RBP is significantly more memory efficient and affords arbitrary precision in the cRNN temporal dynamics (i.e.,  $T$  can be high). C-RBP also allows the cRNN to learn stable and expressive solutions. We can observe this from the cRNN's latent trajectories, as visualized in Fig. 2b, where extrapolation in time (beyond  $T$ ) is non-catastrophic. We argue that model instability is why BPTT-trained controls failed the generalization test (Fig. S1). Furthermore, stable solutions imparted by C-RBP allow our cRNNs to dynamically adapt their number of processing steps in a stimulus-dependent manner. That is, we pose that a C-RBP-trained cRNN can utilize its computational budget differently depending on the input (see Fig. S2 for an illustration). Finally, we note that our strategy to achieve stable recurrent solutions is in contrast to that employed by Bansal et al. [40], wherein cRNNs were stabilized through architectural choices along with an incremental training process.
Evidential loss. We augment our cRNN training objective with an evidential deep learning loss (EDL; [30]). In contrast to the classic cross-entropy loss, EDL considers cRNN outputs to be the parameters of a belief distribution rather than point estimates of class probabilities (Fig. 1a). We formalize cRNN beliefs as a Dirichlet distribution. Strong evidence in favor of one class causes the Dirichlet distribution to be peaked, reflecting low uncertainty  $(\epsilon)$ . In contrast, when there is a lack of overall evidence, the Dirichlet distribution becomes more uniform, thereby increasing  $\epsilon$ . For a complete mathematical specification of this loss formulation, refer to SI §A.
3.3 RT Metric  $(\xi_{cRNN})$
We derive a novel metric to quantify the differential computational demands imposed on the cRNN by stimulus manipulations and capture the essence of model decision time courses for alignment. Modeling uncertainty  $(\epsilon)$  explicitly allows us to track its evolution over time from the cRNN (Fig. 1b). We formulate our RT metric  $(\xi_{cRNN})$  as the area under this uncertainty curve. Precisely,  $\xi_{cRNN} = \int_0^T \epsilon(t) dt$  (Fig. 1b). Intuitively, high (low)  $\xi_{cRNN}$  can be understood as the cRNN spending a large (small) amount of time in an uncertain regime. Our metric affords direct comparisons to human RTs. In the following sections, we apply this metric to cRNNs trained on a host of visual cognitive tasks and study their alignment to human reaction time data.
Figure 3:  $\xi_{cRNN}$  is spatially anisotropic. We present spatial uncertainty (SU) maps to visualize the impact a given fixed dot position will have on model RTs when the position of the second dot is varied. Higher (lower) values of  $\xi_{cRNN}$  represent slower (faster) model responses.
4 Incremental Grouping
4.1 Task, Stimuli, and Visualizations
Inspired by an incremental perceptual grouping task in humans [47], we propose an equivalent challenge for the cRNN where the model is required to determine whether two dots are on the same object (Fig. 2a). The stimuli consist of white outlines of natural shapes on a black background  $(150 \times 150 \mathrm{px})$ . We created  $340K$ $(15K)$  training (validation) outline stimuli from MS COCO images [51]. The dots were placed semi-randomly such that there was no difference in the distribution of dot distances between the two stimulus classes.
We additionally introduce two novel types of visualizations to probe cRNN visual strategies. First, we track cRNN dynamics by looking at its latent activity  $\mathbf{h}_{\mathbf{t}}$  at every time step  $t$ , averaged across the channel dimension and smoothed by averaging  $\mathbf{h}_{\mathbf{t}}$  and  $\mathbf{h}_{\mathbf{t} + 1}$ . To facilitate interpretation, we provide these videos for the reader (see SI §D). Second, we compute a "spatial uncertainty" (SU) map for each outline stimulus by keeping one dot fixed and varying the position of the other. Each spatial coordinate has a value equal to  $\xi_{cRNN}$  for that corresponding dot configuration. Examples are shown in Fig. 2c (right) and Fig. 3.
Despite its lightweight architecture, our cRNN solved the incremental grouping task, achieving  $98\%$  validation accuracy. Visualizing the cRNN latent activities revealed that the model learned a cognitively-viable filling-in strategy. Network activity originated at each of the dot locations and gradually spread, eventually filling up the whole object(s) (Fig. 2b). We note that this behavior is an emergent property from purely the classification task constraint instead of direct supervision for any form of segmentation. We then sought to ask if  $\xi_{cRNN}$  effectively captures these visual decision-making dynamics. In the following sections, we report results detailing the relationship between  $\xi_{cRNN}$  and a host of complex, topological stimulus properties.
$\xi_{cRNN}$  captures the serial spread of attention in our models. Human visual strategies in solving the incremental grouping task are thought to involve a serial spread of attentional resources, resulting in longer RTs for higher distances [47]. To test this effect in our model, we took a subset of our validation outlines and repositioned the dots to be in the center of an object, 14 px apart. We then systematically kept moving the dots 14 px more apart until reaching the object boundary. A linear mixed-effects regression analysis revealed a significant positive effect of this manipulation on  $\xi_{cRNN}$ ,  $b = 1.46$ ,  $SE = 0.11$ ,  $z = 13.58$ ,  $p < .001$  (Fig. S3, Fig. S4). The SU maps (Fig. 2c right, Fig. 3) exhibit this distance effect too. Placing the second dot further away from the fixed dot (indicated in white) yielded higher  $\xi_{cRNN}$  values.
$\xi_{cRNN}$  recapitulates the spatial anisotropy in human RT data. Euclidean distance only paints a partial picture. Human RTs are affected by several other factors, such as topological distance, the proximity of the dots to boundaries, and the need to spread through narrow object regions, all amounting to a well-documented spatial anisotropy in the spread of visual attention [47, 52]. While anisotropy is evident in the SU maps from our cRNN (Fig. 2c right, Fig. 3), we also formally assess the alignment between humans and the cRNN in this regard, using human RT data from [47].
Figure 4:  $\xi_{cRNN}$  from our model recapitulates human RTs on the Planko task. a. The stimuli consist of a ball on top, followed by ten randomly placed planks and finally two baskets in the bottom. Participants are shown these static stimuli and are asked to guess if the ball will end up in the left or the right basket when released and allowed to bounce around on the planks. b. Human RTs (collected in [53]) are predicted by  $\xi_{cRNN}$ . The five colored symbols reflect the data points obtained from the stimuli shown in the next panel. c. Five example stimuli of increasing  $\xi_{cRNN}$ .