SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- AlexNet-BN. This model was included as a pure feedforward control. We used the PyTorch implementation for AlexNet [65] with  $BN$  provided in [66]. The optimizer was stochastic gradient descent (SGD) [67] and the learning rate was  $1e - 2$ . We report the results for the best epoch out of 59 (based on performance on the short validation stimuli) for three individual runs.
- VGG-19-BN. This model was included as an additional pure feedforward control. We trained a variant of the original VGG architecture [68] referred to as "vgg19_bn" in the torchvision model zoo [69]. We used an SGD [67] optimizer and a learning rate of  $1e - 2$ . We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs. Interestingly, in two additional runs with the same hyperparameters, VGG-19-BN was unsuccessful in solving even the easy version of the task and performed at chance level.
- hGRU-BPTT. This is a recurrent control model with the same architecture (see SI §A) as the model studied in the main paper (referred to there as cRNN), but trained with the classic BPTT. Furthermore, we found that in order to learn the easy version of the task with BPTT, we had to replace  $BN$  with group normalization. The optimizer was Adam [50],  $lr = 1e - 3$ , and  $T = 15$ . Our hypothesis was that this control model would fail the generalization test and perform near chance level on the harder long validation stimuli. We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.
- hGRU-CRBP. This model shares the same recurrent architecture (see SI §A) and training algorithm algorithm (C-RBP) as the cRNN used in the main paper. We trained the hGRU-CRBP models with an Adam [50] optimizer,  $lr = 1e - 3$ ,  $T = 40$ , and  $\gamma = 100$ . Our hypothesis was that this model would still perform well above chance level on the harder long validation stimuli, in contrast to both the feedforward controls and the RNN control without C-RBP. This hypothesis was motivated by findings in [8]. We report the results for the best epoch out of 32 (based on performance on the short validation stimuli) for three individual runs.
- ConvLSTM. This is a recurrent control model with the LSTM architecture [70] that shares the C-RBP training algorithm used to train the cRNNs in the main text, but lacks the anatomical and neurophysiological constraints. We trained the ConvLSTM models [71] with Adam [50],  $lr = 1e - 3$ ,  $T = 40$ , and a  $\gamma$  of either 100 or 1000. We analyzed the generalization performance of the runs that reached at least  $90\%$  accuracy on the short validation stimuli.
- ViT-B-16. ViT-B-16 was included as a control model from the Vision Transformer family [49]. We used the implementation included in the torchvision zoo [69] as "vit_b_16". To
train the model, we used an Adam [50] optimizer, a patch size of 10, a learning rate of  $1e - 2$ , a dropout rate of .10, and an attention dropout rate of .10. We report the results for the best epoch out of 299 (based on performance on the short validation stimuli).
B.3 Results
Fig. S1 summarizes the results of the generalization experiments. The models that share the same architecture and training algorithm (hGRU-CRBP) as the cRNNs studied in the main text performed well above chance on the long validation stimuli. Despite having high accuracy on the short validation stimuli (i.e., those with dot distances within the range they were trained for), the control models performed near or at chance on the harder long validation stimuli. The only exception to this finding was ViT-B-16.
Train: Short training stimuli
Figure S1: Control models fail to generalize. The graph summarizes the results from generalization experiments involving the incremental grouping task. To study generalization, all models were first trained on an easy version of the task with training stimuli for which the distance between the dots was short ( $\leq 44$  px). They were then evaluated on validation stimuli with the same, short dot distances (transparent bars), as well as on validation stimuli with longer dot distances ( $>82$  px; cross hatched bars). The longer distances constituted a harder version of the task. hGRU-CRBP models, sharing the same architecture and contractor recurrent back-propagation (C-RBP) algorithm as the cRNN models studied in the main text, still performed well above chance level (0.50) on the hard version, attesting to their generalizability. The control models, in contrast, struggled with the hard version of the task, with ViT-B-16 being a noteworthy exception.
C Dynamically Expendable Computational Budget
Figure S2: Dynamic computational time. a. Low-dimensional projections of our model dynamics for two input conditions in the incremental grouping task (Section 4 of the main text). For the easy input (Panel b), the model effectively "stops" computing at an earlier time point  $(t^{*})$ , as evidenced by dramatically reduced step sizes in the latent space. For the harder input (Panel c), step sizes remain large (above the dotted line) for a longer period. Overall this demonstrates that our model dynamically utilizes its compute budget in a stimulus-dependent manner. b,c. Alternative visualization of the step size as a function of time for the easy and hard stimuli, respectively. The dotted line here is arbitrarily chosen for visual comparison.
D Videos
For the incremental grouping task (Section 4 in the main text) and the maze task (Section 6 in the main text), the cRNN learned a filling-in strategy that was evident from inspecting the latent activity  $\mathbf{h}_{\mathbf{t}}$  over time steps  $(t)$ . The gradual spread of activity can be appreciated best in video-format. Examples of such videos can be found on our project page: https://serre-lab.github.io/rnn_rts_site.
Each video consists of three panels. The left panel shows the input to the model. The middle panel shows the uncertainty curve, with a marker added at every time step to indicate the progression of time. The right panel shows a visualization of the latent activity over time. Each frame shows  $\mathbf{h}_{\mathbf{t}}$  averaged over the channel dimension and smoothed by additionally averaging between  $\mathbf{h}_{\mathbf{t}}$  and  $\mathbf{h}_{\mathbf{t} + 1}$ . With each frame in the video, one can observe the spread of activity and the corresponding change in uncertainty.
E Distance Experiment
When humans perform the incremental grouping task (Section 4 in the main text), their RT tends to increase with increasing distances between the dots. This effect has been attributed to a serial spread of attentional resources [47, 52]. To test whether the effect was also present in the cRNN we trained, we created a dedicated test set, derived from a random subset of the validation outlines ( $N = 150$ ). As explained in the main text, we repositioned the dots to be in the center of an object and  $14\mathrm{px}$  apart in Euclidean distance. We then systemically moved them  $14\mathrm{px}$  further apart until they hit an object boundary, thus creating different distance conditions for a given outline (see Fig. S3 for an example). This procedure resulted in a total of 1852 stimuli. Fig. S3 demonstrates how the distance manipulation affected the uncertainty curves for one example outline. The cRNN spent more time being uncertain if the dots were placed further apart, which is captured in our  $\xi_{cRNN}$  metric. Analyzing the relation between distance and  $\xi_{cRNN}$  across the whole test set using a linear mixed-effects model, we found a significant positive effect,  $b = 1.46$ ,  $SE = 0.11$ ,  $z = 13.58$ ,  $p < .001$  (Fig. S4).
Figure S3: Manipulating dot distance in the incremental grouping task. The top row demonstrates the manipulation of the dot distance for an example outline. The dots are colored orange here for visualization purposes. The bottom row represents the corresponding uncertainty curves, with  $\xi_{cRNN}$  annotated on top. With the dots further apart, the cRNN spends more time being uncertain, resulting in higher  $\xi_{cRNN}$ .
Figure S4: Overall distance effect in the incremental grouping task. The graph shows the average  $\xi_{cRNN}$  across stimuli as we systematically moved the dots 14 px more apart. Error bars represent the standard error of the mean. The cRNN exhibits a clear distance effect, as did human RTs in [47].
F Non-Euclidean Distance Metric
Jeurissen et al. [47] describe a growth cone distance metric that is sensitive to the topology of the object, proximity of boundaries, and narrowness. The authors construct a series of non-overlapping, varying-sized discs whose centers trace a curve (originating at the fixation location and ending at the cue location). The radii are such that a disc is the maximum inscribed disc at its location. Disks never cross object boundaries. The final distance metric is defined as the total number of discs that constitute this curve
G Spatial Anisotropy in Incremental Grouping - Additional Experiments
As a further test of how factors beyond Euclidean distance might affect  $\xi_{cRNN}$  in the incremental grouping task, we created two sets of stimuli (Fig. S7, Fig. S8), each derived from the same 23 novel outlines by placing dots in different configurations. Following the example of [47], each stimulus featured two adjacent objects of the same object class. In one set (Fig. S7), we created two conditions by placing one of the dots (or both) in a narrow region of an object in one condition, and moving the pair of dots to a wider region in the other condition. The Euclidean dot distance, as well as the topological distance were matched between both conditions. In the other stimulus set (Fig. S8), we used the same outlines but with different dot conditions. Here, one condition had the dots placed in a such a way that they could be connected by a straight path. In the other, the path was curved making that the higher topological distance condition. The Euclidean dot distance, on the other hand, was still matched. Fig. S5 visualizes the effect of both these manipulations on  $\xi_{cRNN}$ . As reported in the main text, paired t-tests showed that  $\xi_{cRNN}$  was significantly higher in the narrow versus wide condition,  $d = 0.70$ ,  $t(22) = 3.34$ ,  $p = .003$ , and in the curved versus straight condition,  $d = 0.98$ ,  $t(22) = 4.71$ ,  $p < .001$ .
path path
Figure S5: Effects beyond those of Euclidean distance in the incremental grouping task. We created two novel test sets (Fig. S7, Fig. S8). In one we manipulated whether the dots were placed in a wide versus narrow region of an object. In the other the path between the dots was either straight or curved, thereby manipulating topological distance. In both manipulations, Euclidean distance was kept constant. The boxplots show the effect of these manipulations on  $\xi_{cRNN}$ . The star markers inside the boxes represent the condition mean. The asterisks on top represent the  $p$ -values resulting from a paired t-test,  $** < .01, *** < .001$ .
Finally, in a last follow-up experiment, we varied the distance between one dot and an object boundary while keeping the Euclidean distance between the dots themselves constant (Fig. S6). We found that our model exhibits higher values near the boundaries (Fig. S6b,c). This is strikingly similar to human behavior presented in prior literature ([47], Fig. 3d).
Figure S6: Fixed Euclidean distance; varying distance to boundaries. a. Wedge stimuli presented to our model. b. We observe higher  $\xi_{cRNN}$  closer to boundaries, as was observed for human RT in [47]. Data from [47] are shown on the left here. c. Visualizing our model RT predictions for a fixed chosen radius d. The complete SU-map reveals a further disparity between the "yes" and "no" conditions.
Figure S7: Novel outline stimuli for the incremental grouping task featuring a narrow versus wide manipulation. Dots are colored orange for visualization purposes.
Curved
Figure S8: Novel outline stimuli for the incremental grouping task featuring a topological distance manipulation. The dots were placed such that it was either a straight or a curved path (higher topological distance) connecting them, whereas the Euclidean distance was the same. Dots are colored orange for visualization purposes.
H Ambiguous Grouping Stimuli and Non-zero Uncertainties
Here, we take a closer look at which stimuli elicit very high  $\xi_{cRNN}$  values. Looking at our validation set first, we observed that some high- $\xi_{cRNN}$  stimuli are characterized by uncertainty curves that remain high, even at  $t = T$ , suggesting the model deems them ambiguous (see Fig. S9 for an example).
Figure S9: High uncertainty example. Some challenging samples in our validation set display non-zero uncertainty.
Furthermore, in a test set we designed to be ambiguous (it features occlusions), we frequently observed such non-decreasing uncertainty curves too (see Fig. S10). We note that while our training procedure promotes stable attractor states in the model's latent dynamics, this is disparate from readout dynamics allowing the framework to truly explore the spectrum of computation times and uncertainties.
b
Figure S10: Testing model uncertainty on occluded stimuli. a. We created a dataset  $(N = 76)$  of occluded images, half of which have both dots on the same side of the occluding object (control). The Euclidean dot distance is matched between the control and occluded condition. We performed zero-shot transfer with our trained cRNN on this set and observed that for the occluded samples, model uncertainty often remained at high (non-zero) values in the time limit. b. A paired t-test reveals that our metric is significantly higher in the occluded condition compared to the control condition: Cohen's  $d = 2.85$ ,  $t(37) = 17.54$ ,  $p < .001$ .
I Additional Maze Analyses
Human participants have been found to take more time to solve a maze when the solution path features more junctions [56]. To study whether the cRNN we trained exhibited a similar slow-down, we computed two measures for the Yes-mazes (Fig. S11). One was the number of T-junctions encountered on the shortest path between the two cues. The other was the total number of T-junctions on the maze segment containing the cues. Linear regression analyses that statistically controlled for path length by including it as a predictor, did not yield evidence for an effect of the first measure,  $b = -0.23$ ,  $SE = 0.29$ ,  $t(234) = -0.79$ ,  $p = .43$  or the latter,  $b = 0.28$ ,  $SE = 0.18$ ,  $t(234) = 1.56$ ,  $p = .12$ . We treat this finding as an important example for how our metric can help identify potential mismatches between human- and model-behavior and ultimately guide the development of better aligned models.
Figure S11: Counting T-junctions in Yes-mazes. a. An example maze. b. Visualization of the T-junctions. The solution path connecting the cued locations for the same maze shown in the previous panel is indicated in blue here. We found no evidence for an effect of the number of T-junctions included in the solution path (orange squares), nor of the number of T-junctions included in entire maze segment containing the cues (orange + green squares) on  $\xi_{cRNN}$ .
J ConvLSTM Results on Scene Categorization
b
Figure S12: A convLSTM equipped with our metric captures human RT patterns in a scene categorization task. Within our framework, we are able to swap out various model (or architecture) choices without affecting the downstream formulation. Whereas our choice of architecture in the main text was motivated by the demands that a task like incremental grouping imposes, we show here that  $\xi$  can also be computed for other architectures, such as a convLSTM. The fact that our metric is model-agnostic makes it ideally suited for comparing and contrasting models in terms of their temporal alignment.