RTify: Aligning Deep Neural Networks with Human Behavioral Decisions
Authors: Yu-Ang Cheng
Ivan F Rodriguez Rodriguez
Sixuan Chen
Kohitij Kar
Takeo Watanabe
Thomas Serre
Venue: NeurIPS 2024 poster (2024)
Keywords: Alignment; Recurrent neural networks; Reaction times; Visual decision making; Speed-accuracy tradeoff
Primary area: neuroscience_and_cognitive_science
TL;DR: We present a novel differentiable framework to effectively align current vision models with human reaction times and behavioral choices

Abstract: Current neural network models of primate vision focus on replicating overall levels of behavioral accuracy, often neglecting perceptual decisions' rich, dynamic nature. Here, we introduce a novel computational framework to model the dynamics of human behavioral choices by learning to align the temporal dynamics of a recurrent neural network (RNN) to human reaction times (RTs). We describe an approximation that allows us to constrain the number of time steps an RNN takes to solve a task with human RTs. The approach is extensively evaluated against various psychophysics experiments. We also show that the approximation can be used to optimize an ``ideal-observer'' RNN model to achieve an optimal tradeoff between speed and accuracy without human data. The resulting model is found to account well for human RT data. Finally, we use the approximation to train a deep learning implementation of the popular Wong-Wang decision-making model. The model is integrated with a convolutional neural network (CNN) model of visual processing and evaluated using both artificial and natural image stimuli. Overall, we present a novel framework that helps align current vision models with human behavior, bringing us closer to an integrated model of human vision.

RTify: Aligning Deep Neural Networks with Human Behavioral Decisions
Yu-Ang Cheng") Ivan Felipe Rodriguez", Sixuan Chen', Thomass Kohitij Kar Takeo Watanabe' Thomas Serre!
Brown University 2 York University
mruang_cheng iiva__ffflll wellpe_rodrieeeeeeeeeeeeeeeeeeee sixuan_chent. takeeaakewwwaaae watanabe, chomas_serre))brooo k0htt1j@yorku.ca
Abstract
Current neural network models primate vision focus replicating overall levels of behavioral accuracy, often neglecting perceptual decisions' rich, dynamic nature. Here, we introduce a novel computational framework to model the dynamics of human behavioral choices by learning to align the temporal dynamics of a recurrent neural network (RNN) human reaction times (RTs). We describe an approximation that allows us to constrain the number of time steps an RNN takes to solve a task with human RTs. The approach extensively evaluated against various psychophysics experiments. We also show that the approximation can be used to optimize an "ideal-observer" RNN model to achieve an optimal tradeoff between speed and accuracy without human data. The resulting model is found account well for human RT data. Finally, we use the approximation to train a deep learning implementation of the popular Wong-Wang model is integrated with convolutional neural network (CNN) model of visual processing and evaluated using both artificial and natural image stimuli. Overall, we present a novel framework that helps align current vision models with human behavior, bringing us closer to integrated model of human vision.
model. The
1 Introduction
Categorizing visual stimuli is crucial for survival, and it requires an organism to make informed decisions in dynamic and noisy environments. This critical aspect of visual perception has driven the development of computational models to understand and replicate these processes. Traditionally, the field has followed two distinct paths.
On the one hand, image-computable vision models are used to predict behavioral decisions during (rapid) visual categorization tasks ranging from models of early- [133. mid- 4 and high-level vision (7-9 (see [10] for a review). More recently, these earlier models were superseded by deep convolutional neural networks (CNNs), which have become the de-facto choice for modeling behav- ioral decision [1) Models are typically evaluated by estimating confidence scores computed for individual images, which are then correlated with similar scores derived for human observers(such as the proportion of correct human responses for each image). Such metrics ignore human reaction times (RTs); hence, current vision models only partially account for human decisions.
On the other hand, pireision-making models have been used to explain how visual information gets integrated over time predicting behavioral choices and RTs jointly. Notably, mathematical models, exemplified by evidence accumulation models such as the drift-diffusion [5 and linear ballistic
"These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stimuli
Neural Network
Stop
h
t
No
Continue
t
0
D
ei
Steps
i=0
et = (
Compute Evidence
Te
To
RNN activity h;
0.8
Threshold
Trainable
accumulated
Evidence
0.6
Frozen
Right Direction
0.2
Left Direction
0.0
60
40
80
Steps
Steps
Figure I: Illustration of our RTify method. The input is a visual stimulus represented by random moving dots, but the model can also accommodate color images and video sequences. We take pretrained task-optimized RNN and use trainable function fw to transform the activity of the network into real-valued evidence measure, Ct, that will be integrated over time by an evidence accumulator, Φ, When the evidence accumulator reaches the threshold 6, processing stops, and decision is taken. The time step at which the accumulated evidence passes this threshold Tg taken as the model RT for this stimulus.
accumulators 18 19), have been quite successful in modeling array of behavioral data (see 20 for a review). addition, mechanistic models, including the Wong-Wang (WW) model, have provided insights into the underlying neural mechanisms (21)2). However, these efforts have primarily relied on traditional psychophysics tasks using simple, artificial stimuli, such as Gabor patterns [23] and random moving dots (24). Beyond these easily parameterizable stimuli, these models have not been extended to deal with more complex, natural stimuli.
While both vision and decision-male making models have contributed distinctively to understanding of visual processes, a complete understanding of human vision will require their integration to explain the whole dynamics of visual decision- decision-making Recent neuroscience studies have leveraged recurrent
neural network (RNN) models as the starting point for this integration [25] (see (26) for a review).
More generally, recurrent processing has been shown to be necessary to account for both behavioral
and neural recordings in object recognition tasks [27 (27
Two promising approaches have been described that leverage RNNs to bridge the gap between decision-making and vision models thanppagation-thrugg-----t
[29). an RNN was trained to solve visual classi- (BPTT) learning algorithm. The entropy of
fication task using the the RNN outputs is computed each timestep and used as a proxy for the network confidence. A decision is taken when the entropy reaches the threshold, halting further recurrent computation. In this approach, the recurrence steps are not differentiable, which prevents the use of gradient methods and inherently limits the complexity of the corresponding decision function. The resulting model may have difficulty predicting the entire distribution of RT's. Besides, the method is only applicable when human RTs are available because it requires extensive search for the correct threshold value to fit human RT data.
convolutional RNN was trained on visual classification task using contractor recurrent propapronnnn (C-RBP). Besides, instead of searching for optimal threshold to fit human RT data, surrogate time-evolving 'uncertainty'
An alternative approach was described in 30 In 30

---

metric was estimated with evidential deep learning (31). In this framework, model outputs are treated as parameters of a Dirichlet distribution, with the width of the distribution reflecting uncertainty. Remarkably, the resulting approach fits a range of experimental data well, without any supervision from human data. It is true that uncertainty and RT's are tightly coupled and are both affected by task difficulty. However, uncertainty and RT's are conceptually different. Experimental results show that uncertainty and RTs can be positively or negatively correlated [32), and even double dissociated [33) under different experimental conditions. Therefore, good model of uncertainty is not guaranteed to be a good model of RTs.
Here, based on extensive cognitive neuroscience research 34 37 we introduce a novel trainable module called RTify to allow an RNN to dynamically and nonlmearly accumulate evidence. First, this module can be trained to learn to make human-like decisions using direct human RT supervision. Our results suggest that incorporating a dynamic evidence accumulation process, compared to the entropy heuristics used in (29), can help better capture human RT's. Second, we show how the same general approach can also be used to train an RNN learn to solve task with an optimal number of time steps via self-penally Our results show that human-like RTs naturally emerge from such ideal-observer models without explicit supervision from human RT data. Hence, our framework is general enough to allow the fitting of human RT data as done in 29 and/or the training of neural architecture that can optimally trade speed and accuracy via self-penalty as done in 30
Overall, our work makes the following contributions: (i) We present RTify, a novel computational approach to optimize the recurrence steps of RNNs to account for human RTs. This enables dynamic nonlinear evidence accumulation learned through back socck-ropagation (a) to fit human data (b) optimally balance speed and accuracy. (ii) We comprehensively demonstrate the effectiveness of our framework for modeling human RTs across a diverse range of psychophysics tasks and stimuli. Our method consistently outperforms alternatives with and without explicit training on human data. (iii) As an illustrative example of the framework's ' potential, we extend the WW decision-make making model (22) create biologically plausible, multi-class compatible, and fully differentiable RNN module. We show that the enhanced neural circuit can be used as a drop-in module for CNNs fit human RTs.
Contributions:
2 RTify: Overview of the method
First, we explain how our RTify module is applied to a pre-trained RNN. Then, we will explain how to tune deep-learning RNN-based implementation of the WW model to RTify feedforward networks.
We start with task-optimized RNN with hidden state h, which remains frozen. We then train learnable mapping function fw : R + R that summarizes the state of the neural population at each time step t by mapping the RNN hidden state h, some "evidence": e = (h+). At every time step, the evidence is integrated via an "evidence accumulator" Φ, Ei= Cj and when the accumulated evidence passes a learnable threshold 6, the model is read out, and decision is made. The time step which the accumulated evidence first passes this threshold given by To = minft: 4, 0}. and is treated as model RTs. In summary, Te () is directly influenced by the threshold 0 and by w through d.,.
To align the model RTs with human RTs, or to penalize the model for excessive time steps, we need to optimize loss function over (T). In the most general case, we first consider F(Ta( ()) as our loss function to illustrate how we approximate its gradient. Since our goal is to minimize F, we will and @F(r=(+)) Following the chain rule, we get dw
need to calculate the gradient aF(r.(0))
06
OF (1)) = (F((T(()) @Te( (F(r)()) (F(re(1)) @Ta(2) OTo(T)
(1)
dw
θrg ())
dw d0
00
This means crucial step involves estimating the gradient of (o( over the trainable parameters and 0 of the RTify.
The primary challenge that arises when extracting the gradient of To ( over the trainable parameters w and 0 is the non-differentiabiliy of (T)), which prevents the direct use of the backpropagation algorithm. This is because Te($) lies in the integer space and requires non-differentiable operations
3

---

such as the minimum function and the inequality. We will relax the original formulation to circumvent this issue, enabling us to approximate the gradient.
Assume that (()) is a continuous function on the closed interval [1, MI. representing the accumulation of evidence time. Define To ( min(t E [J (()) > 0), which is the earliest time when (t)) exceeds the threshold f. We can use a first-order Taylor expansion to find the following approximation:
TT aw
aTg aw
0 Dw
TT
TTg 00
;-1
00
4
4 P1-1
Fig.
provides a visual intuition for the approximation, and the full derivation can found the This leads to the following gradients for our trainable parameters:
SI A.1
OF (Te()))
(F(T)(
04:
(T+))
(F(T((
1
Ow
(T)
aw
00
rTo
0; 0,-1
Since and & are both differentiable by nature, the gradients in Eqs. and 3 are all computable after
this approximation.
Under this framework, we consider two different scenarios: Training with human data directly ("supervised"; see Section training with "self-penalty" which involves no explicit human data but uses a penalty term that spontaneously leads decision times similar to human RTs (e.g.. achieving an optimal speed-accuracy trade-off;; see Section 2.2
2.1 Predicting human decisions with direct "supervision"
Human behavioral decisions in the random dot motion (RDM) tasks used in decision- stud- ies [16 24 38 are typically summarized histograms similar to those shown in Fig. 2 Here, histograms are computed for RTs for correct and incorrect trials corresponding to individual experi- mental conditions (such as coherence levels shown here; see section 3 for details). Moreover, RTs for incorrect trials are turned into negative RTs. Combined with correct RTs (which stay positive), one single histogram is used for capturing both accuracy (the proportion of positive values) and RTs. To measure the goodness of fit between human RTs and model RTs, we use a mean squared error loss (MSE) between histograms of model RTs and human RTs.
In the object recognition task [39], only RTs averaged across all participants were available. We can match human data on a stimulus-by by-stiuulus basis using the negative correlation loss between model and human RTs.
2.2 Predicting human decisions with "self-penalty"
Our framework allows us to develop an "ideal-observe" RNN model explicitly trained to balance the computational time required for solving particular classification task and its own accuracy for the task, i.e., speed-accurace trade-off.
To achieve this, add a regularizer the cross-entropy loss encourage the RNN jointly maximize task accuracy while minimizing the computational time needed to solve the task. With as the output logits of the network, ÿ as the output probabilities of the network, and y as the ground truth, we can write our penalty term for single sample as:
peeatizenalized = LCCE 9) + + (ly TT)
where À hyperparameter for controlling the strength of the penalty, ly refers the logit value of the correct label, Tę is the model decision time. This penalty means that the model will be penalized for using too much time, especially for higher confidence (higher ly).
2.3 RTifying feedforward networks
To integrate temporal dynamics into feedforward neural networks (e.g., CNNs), we describe an RNN module that approximates the WW neural circuit model (22]. The original WW model biophysically-realisti neural circuit model of two-altemative forced choices via the temporal

---

A
Extracting RTs with supervision (fitting human RTs)
Human data (Green et al., 2010)
RTified RNN
Entropy-threshooldng
1e-2
coh: 51.2
1e-2
coh: 25.6
10-2
coh: 12.8
1e-2
coh: 6.4
Probability
4-
1.5
1.0
1.0
2-
0.5
55
0.0
0.0
2.5
1.25
0
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
Comrect
Incomect
Correct
Incomeet
Corvect
Incomeet
Oonect
Reaction times
(s)
Reaction times (s)
Reaction times (s)
Reaction times (s)
Extracting RTs via self penalty (leveraging RNN dynamics)
Human data (Green et al., 2010)
Self-penalized RTified RNN
Uncertainty proxy
1e-2
coh: 51.2
3 16-2
coh: 25.6
2.0
1e-2
coh: 12.8
1.5
1e-2
coh: 6.4
Probabilty
2
1.0
1.0
0.5
0.5
2.5
1.25
1.25
2.5
2.5
1.25
1,25
2.5
2.5
1.25
1.25
2.5
25
2.5
Incomeet
Donnet
Incomert
Cormet
Incomet
Pormec
Incomrett
Correct
Reaction times
Reaction times (s)
Reaction times (s)
Reaction times
Figure 2: RTified model evaluation on RDM task (24). Human data are shown gray shaded area, and model fits are shown for (A) the "supervised" setting where human behavioral responses are used train the models and (B) the "self-penalized" setting where no human data is used. Our approach (green) outperforms the two alternative approaches (brown), i.e., mntropy-thresholdee for the "supervised" and uncertainty proxy (30) for the "self-penalized" settings (see Fig. for MSE comparisons and Fig. SS for all coherences).
(29)
accumulation of sensory evidence in two distinct neural populations. takes constant scalar input (representing stimulus parameter such as the degree of coherence for randomly moving dot stimuli). outputs an RT when the activity of either population reaches decision threshold.
However, the original WW model has limitations: its parameters must be manually tuned to fit observed data, and is restricted binary classification tasks with simple artificial parametric stimuli (e.g.. Gabor patterns). To overcome these limitations, we extend the WW model. First, we replace the scalar input with a feedforward neural network (e.g., CNN), enabling the model to process complex stimuli such as natural images. Second, we generalize the model from two populations to M populations. effectively increasing the number of neural populations handle multi-class classification problems. Third, we RTify the model to make all parameters trainable via backpropagation. This allows the model automatically learn the optimal parameters to human RTs (see Fig. 3 for an illustration of the multi-population case and SI Fig. A.2 for illustration of the two-population case).
3 Experiments
In this section, we validate our RTify framework on two psychophysics datasets: the RDM dataset [24]
38) and natural image categorization dataset (39). a side note, all models were trained on single Nvidia RTX GPUs (Titan33990/60000 with 24/24/48GB of memory each. All training can be
completed approximately 48 hours. Code and data are available at
com/
Yu- AngCheng/gTeeae /RTify
5

---

A
Excitatory Connection
Inhibitory Connection
'Chair population activity
50
population activity
Zebra
0.8-
Elsphant
40
Face
30
0.4
Car
0.5
Dog
20
0.2-
Apple
10
Plane
0.0
Bird
0.0
threshold
0.0
0.5
1.0
20
40
'Bear' population activity
Steps
Figure 3: Illustration of RTifying feedforward neural networks. We develop a multi-class compatible and fully differentiable RNN module based the WW model 22]. This module is implemented as an attractor- based RNN, and is stacked on top of feedforward neural network. The feedforward neural network first takes an image the input. Outputs from classification units the network are then sent to RTified WW (A). Information is accumulated by multiple populations of neurons RTified WW while they compete with each other (B). decision made and the process stops when one of the populations reaches threshold. The number of time steps needed for the RTified WW to reach the threshold is used to predict human RT (C).
3.1 Random dot motion task
The RDM task is a classic experimental paradigm used to test temporal integration that has been extensively used in psychophysics [40], human imaging (4T), and electrophysiology stud- ies [42]. The stimuli in this task consist of dots moving
screen toward a predefined di- specific probability (coherence)
rection vs. randomly. For each time step, each dot only (0.8%, 1.6%, 3.2%, 6.4%. 12.8%, 25.6%, 51.2%) to move towards the pre-defined direction, mak- ing the task non-trivial. The participants must integrate motion information across time and report it when they sufficiently confident. The original experimental data are from 24 38
where 21
young adult participants performed around 40,000 trials total over consecutive days.
First, we trained an RNN consisting of convolutional blocks (Convolution, BatchNorm, ReLU, Max pooling) and 4096-unit LSTM with BPTT. In the original experiment, the stimuli were shown on 75 Hz CRT monitor for up to 2 seconds, and therefore, we also trained our RNN for the RDM stimuli for 150 frames. The RNN was trained for 100 epochs using the Adam optimizer with a learning rate of le-4 at full coherence (c = 99.9%) for the first 10 epochs as warm-up and le-5 at all coherence levels for the remaining 90 epochs. Next, we trained our two different RTify modules. For fitting human RTs, it trained for 10,000 epochs, and for self-penalty, it was trained for 20,000 epochs. In both cases, the Adam optimizer were used, and the weights of the task-optimized RNN were frozen while training the RTify modules.
We trained the first RTify module to predict human RTs by fitting human RT distributions. Here, positive RTs refer to RT with correct choices, and negative RTs refer to RTs with incorrect choices. Therefore, one distribution incorporates both speed and accuracy information from behavioral choices. Results are shown in Fig. 2 Our RTify model can predict the full RT distribution across all coherence levels. In comparison, the
approach by (29 fails to capture the full distribution (see Fig. 2 for coherence = 51.2% to 6.4% and Fig. S3 for all coherences). Importantly, our method

---

Entroyy-hhresholding
Human
Uncertainty proxy
RTified C3D WW
RTified RNN
pretrained RNN
self-penalized RTifed RNN
pretrained C3D
A 10"
B
1.0
10%
Accuracy
0.9
MSE
0.8
10 -6
0.7
0.6
10 -7
0.5
12.8
25.6
51.2
12.8
25.6
51.2
Coherence (%)
Coherence (%)
Figure 4: (A) MSE comparisons for the RDM task (24) for all coherence levels. The RTified model trained in the "supervised" setting (i.e., with human behavioral responses; green solid line) performs better (lower MSE) than hatropy-threesolding (29] (brown solid line) under all coherence levels. Similarly, the RTified model trained in the self-penallied" setting (i.e., without human data; green dash line) performs better than uncertainty proxy [30] (brown dash line). With the help of RTified WW module (orange solid line), a convolution neural network (C3D) also fit the data better than entropy-threshee threolliirrs [29]. (B) Classification accuracy comparisons between pretrained and RTified models for the RDM task [24). The RTified model trained with human RTs data in the "supervised" setting (green solid line) and in the "self-penalized" setting (green dash line) achieve human-like classification accuracy under all coherence levels compared with the pretrained model without RTify (green dotted line). With the help of our RTified WW module (orange solid line), CNN (C3D) matches human accuracy better than the pretrained model without RTify (orange dotted line).
surpasses seropp--trresooddng approach MSE comparisons, see Fig. 4
29 (two-sided Wilcoxon signed-rank test, p < .05; for
well known that there trade-off between RT' and accuracy in cognitive tasks. To investigate this relationship further, we extended our analysis to examine whether the model's accuracy would approximate human accuracy when it was fitted solely on human RTs without using human accuracy data. Given that conventional RT distributions encompass both RTs and accuracy information, we restricted our fitting procedure to the positive part of the distribution. That is, RTs corresponding to correct responses to prevent inadvertently incorporating human accuracy into the model. Remarkably, by fitting the model on human RTs only, the model naturally reached a classification accuracy comparable human performance (see Fig. 4).
We also trained a self- eelf-penalized RTify module without any human data. This ideal-observer RNN model was trained to minimize the time steps needed for solving the RDM task (see section 2.2 for details). Human-like RT's emerge naturally in this neural network (See Fig. D Our model predicts RT data much better than previous approaches, which use a measure of uncertainty computed over the RNN as surrogate metric. As can be seen, the resulting model tends to overfit the modes of the distribution. (see Fig. 2 for coherence = 51.2% to 6.4% and Fig. S3 for all coherences). Our method surpasses uncertainty proxy approach [30] (two-sided Wilcoxon signed-rank test, p < .05, for MSE comparison, see Fig. D.. We also checked the model classification accuracy before and after self-penalized RTify. Interestingly, we also found the self penaliznd RTified RNN demonstrated a human-likk like classification accuracy (see Fig,
3.2 Object recognition task
Unlike previous visual decision eocision-maaing models, we want to show that our method can also be applied to natural images and multi-class datasets. Specifically, we consider an object recognition task, a classic paradigm used extensively in computer vision [43 44 and cognitive neuroscience studies (45] The original data is from [39], where participants perform the task through Mturk. The stimuli in this task belong to 10 categories, and for each category, there are 20 natural images
7

---

A
Extracting RTs with supervision (fitting human RTs)
Linear regression line
Filtered linear regression line
Pearson's r= 551, .011
Pearson's r= .39, p .001
Pearson's r= 52, p < .001
Pearson's r= .33, p < .001
1,4
1.4
Human RT (5)
1.2
Human RT (s)
1.2
1.0
1.0
0.8
0.8
0.6
0.6
6
10
14
18
10
12
14
16
18 20
RTified RNN (a.u.)
thrrooigt (a.u.)
B
Extracting RTs via self penalty (leveraging RNN dynamics)
Linear regression line
Filtered linear regression line
Pearson's F= _40, p < .001
Pearson's r= -336 p _001
Pearson's _41, p < .001
Pearson's r= =05,, p 0.32
1.4
1.2
1.2
Human RT
Human RT
1.0
1.0
0.8
0.8
0.6
0.6
8
12
16
20
0.2
0.3
0.4
0.5 0.6
liffppenalzze
Uncertainty proxy (a.u.)
RTified RNN RT (a.u.)
Figure 5: RTified model evaluation on an object categorization task (39]. Model vs. human RT predictions for our RTified model (green) vs. alternative approaches (brown) (A) the "supervised" setting where human behavioral responses are used to train the model and (B) the "self- helf-penalized" setting where no human data is used. Solid lines are linear regression fits between model and human RT'. Crossed-shaded areas and the dashed lines are controls to show the fits after removing the highest model RTs. Our approach outperforms the two alternative approaches, i.e., suroopythreesolding (29 for the "supervised" setting and uncertainty proxy 30 for the "self-penalized" setting.
taken from the COCO dataset 47 and 112 synthetically generated images with different backgrounds and object positions.
We first train our RNN with BPTT to perform 10-way classification task. the original study, participants performed binary classification task. However, since the individual binary pairs were not saved, we trained the model a 10-class classification task. We used Cornet-S (48) pretrained on the Imagenet dataset [43 because it was used in the original study and it achieves a relatively high brainscore terms of explaining neural activities [49]. We trained the network for 100 epochs, using the Adam optimizer with learning rate of le-4 and learning scheduler (StepLR) with step size of 2,000. We used 20 timesteps (instead of 2 in the original model) for the IT layer in the Cornet to achieve high temporal resolution. Results show that our RNN achieves 75.2% set. Similarly, we trained our two different RTify modules. For fitting human RTs, was trained for 100,000 epochs, while for self- self-penalty, it trained for 10,000 epochs with learning scheduler (StepLR) with a step size of 2,000 and a gamma of 0.3. both cases, Adam optimizers were used, and the weights of the task-optimized RNN were frozen while training the RTify modules.
held-out test
We then extracted RT's from our RNN to fit human RTs. Notably, model RTs significantly correlate with the human RT observed in the psychophysics experiment (r .51, p .001). Besides, our

---

A
Human data (Green et al., 2010)
B
Linear regression line
RTified C3D WW
Filtered linear regression line
1o-2
coh: 51.2
1e-2
coh: 25.6
Probability
1.4
2-
1-
1.2
Human RT
2.5
1.25
0
1.25
2.5
2.5
1.25
1.25
2.5
1.0
Incomrett
Commet
Incormect
Comroct
Reaction times (s)
Reaction times (s)
2.0
1e-2
coh: 12.8
1.5
1e-2
coh: 6.4
0.8
1.5
1.0
0.6
1.0
0.5
0.5
0.0
0.2
0.4
2
1.25
1,25
2.5
2.5
1.25
1.25
2.5
RTified
Reaction times
Reaction times
VGG_WW RT (a.u.)
Figure 6: RTified WW model evaluation. We combine our RTified WW module with (A) 3D CNN to fit human RTs collected an RDM task (24) (see Fig. f for MSE comparisons with other methods) and (B) a VGG to fit human RTs in rapid object categorization task (39) (Crossed-shaded areas and the dashed lines are controls to show the fits after removing the highest model RTs).
method also surpasses entropy-thresholdin theirs with probability 99.9%) We also extract RTs from an ideal-observer RNN model trained with time self-penaly (see section 2.2 for details). We show here that model RTs are significantly .40. < .001). This failed to be captured by uncertainty proxy (30). We argue this in two ways. First, bootstrapping shows that method is superior to theirs with probability of 87.1%. Second, and most importantly, although the uncertainty seems to correlate with human RTs in the dataset (r .36, .001), it only applies to high-uncertainty cases. When trials with high model uncertainty are excluded, uncertainty shows no significant correlation with human RTs (r .05, p 0.32). However, using method, the correlation remains strong and significant (r = 41, < .001).
(9 (bootstrapping shows that method is superior to
correlated with human RT's (see Fig.
3.3 RTifying feedforward neural networks
Given the prevalence of feedforward networks (e.g. CNNs) and their incredible performance in visual tasks, a natural question how to align such networks in the temporal domain of decision-making We thus developed biologically plausible, multi-class, differentiable RNN module based on the WW recurrent circuit model [1 22). This module can be stacked on top of any neural network, even if not recurrent, and can be used to align model RTs with human RTs.
For the RDM task, we take 3D CNN with convolutional blocks (convolution, BatchNorm, ReLU, Max pooling) and an MLP. We train the network for 100 epochs using the Adam optimizer with learning rate le-4 at full coherence (c 99.9%) for the first 10 epochs as a warm-up and le-5 at all coherence levels for the remaining 90 epochs. Since this model is not an RNN, it has no temporal dynamics. Therefore, we drop in our WW module and further train the WW module for 5,000 epochs using the Adam optimizer with a learning rate of Ic-4, a StepLR scheduler with step size of 1,000 and gamma of 0.3, and a grad clip at Ic-5. Interestingly, when we RTify the C3D model using the WW module, it is able to capture the distribution of human RTs across all coherence levels, see Fig. 6 for coherence 51.2% to 6.4% and Fig. S4 for all coherences, for MSE comparison see Fig. Furthermore, also observed human-lik like classification accuracy for the model when it is solely trained to human RTs but not human accuracy (see Fig. 4
Similarly, we take a VGG-19 pre-trained on Imagenet for the object recognition task and fine-tune it on the dataset provided by Kar et al. (39 in a 10-class classification way for the abovementioned reasons. We train the model for 100 epochs using batch size of 32. The optimizer was AdamW,

---

with learning rate of le-5. We use a OneCycleLR scheduler adjusted after 10 epochs of warm-up up. Results show that our RNN achieves 81.6% on a held-out test set. We further train the WW module for 100,000 epochs using the Adam optimizer with learning rate of le-4 and grad clip at 0.0001. By using the multi-class version of the WW model, we show that the RTify VGG also exhibited a significant correlation with human data (r .49, p < .001, see Fig. 6
4 Conclusion
We have described computational framework to train RNNs to learn to dynamically accumulate evidence nonlinearly that decisions can be made based on variable number of time steps to approximate human behavioral choices, including both decisions and RTs. We showed that such optimization can be used to fit an RNN directly human behavioral responses. We also showed that such a framework can be extended to an ideal -oserveeerreer model whereby the RNN is trained without human data but with selff-penalty that encourages the network to make a decision as quickly as possible. Under this setting, human-like behavioral responses naturally emerge from the RNN consistent with the hypothesis that humans achieve speed-accurane trade-off. Finally, we provided an RNN implementation of popular neural circuit decision-make making model, the WW model, trainable deep learning module that can be combined with any vision architecture to human behavioral responses. Our computational framework provides way forward to integrating image-computable models with decision- decision-making models, advancing toward a more comprehensive understanding of the brain mechanisms underlying dynamic vision.
Limitations Certain limitations will need to be addressed in future work. Most of the human data used in study remains relatively small-scale and is limited primarily to synthetic images because more naturalistic benchmarks only include behavioral choices [44 (0) and lack RT data. To properly evaluate our approach and that of others, larger-scale psychophysics datasets using more realistic visual stimuli will be needed. There is already evidence that large-scale psychophysics data can be used to effectively align Al models with humans [5T 52]. We hope this work will encourage researchers to collect novel intereetscale benchmarks that include both behavioral choices and RTs.
Broader Impacts As AI vision models become more prevalent in our daily lives, ensuring their trustworthy behavior is increasingly important [53 54). Our framework contributes to this effort by exploring how to align certain aspects of models' behavior with human responses in specific contexts. While our approach is limited to predicting RT distributions, it constitutes first step toward more human-aligned AI models.
Acknowledgments and Disclosure of Funding
This work was supported by NSF (IIS (IS-2402275). ONR (N00014-24-1-2020 and the ANR- -IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PP) 1AN--19-P13A-0004 to T.S and National Institutes of Health (NIH R01EY019466 and RO1EY027841) to T.W. Computing hardware was supported by NIH Office of the Director grant (S100D02518)) via Brown's Center for Computation and Visualization (CCV).
References
[1] Oliva, A., Torralba, A.:
Modeling the shape of the scene: holistic representation of the spatial envelope. International Journal of Computer Vision 42(3) (2001) 145-175
[2] Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat's
visual cortex. The Journal of physiology 160(1) (1962) 106
[3] Itti, L., Koch. C., Niebur, E.: A model of aaliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence 20(11) (1998) 1254-1259
[4] Jagadeesh, A.V., Gardner, J.L.: Texture-like representation of objects in human visual cortex. Proceedings of the National Academy of Sciences 119(17) (2022) e21 15302119
[5] Doshi, F.R., Konkle, T., Alvarez, G.A.: A feedforward mechanism for human-like contour integration. bioRxiv (2024) 2024-06
10

---

[6] Parthasarathy, N., Simoncelli, E.P.: Learning texture model for representing cortical area v2. In:
Computational and Systems Neuroscience (CoSyNe). Denver, CO (Feb 2020)
[7] Riesenhuber, M., Poggio, T.: Hierarchical models of object recognition in cortex. Nat Neurosci 2(11) (Nov
1999) 1019-1025
[8] Serre, T., Wolf, I.,, Bileschi, S., Riesenhuber, M., Poggio, T.: Robust object recognition with cortex-like mechanisms. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(3) (2007) 411-426
Biederman, L: review 94(2) (1987) 115
theory of human image understanding. Psychological
[10] Crouzet, S.M., Serre, T.: What are the visual features underlying rapid object recognition? Frontiers in Psychology 2 (2011)
[11] Kheradpisheh, S S.R., Ghodrati, M., Ganjtabesh, M., Masquelier, T.: Deep networks can resemble human feed-forward vision in invariant object recognition. Scientific Reports 6(1) (September 2016)
[12] Eberhardt, S., Cader, J.G., Serre, T.: How deep is the feature analysis underlying rapid visual categorization? In Lec, D., Sugiyama, M., Luxburg, U., Guyon, I., Garnett, R., eds.: Advances in Neural Information Processing Systems. Volume 29., Curran Associates, Inc. (2016)
[13] Rajalingham, R., Issa, E.B., Bashivan, P., Kar, K., Schmidt, K., DiCarlo, JJ.: Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and otate-of ofrthe-rr deep artificial neural networks. J Neurosci 38(33) (Aug 2018) 2265-7269
[14] Wichmann, F.A., Geirhos, R.: Are deep neural networks adequate behavioural models of human visual perception? (2023)
[15] Rateliff, R.: A theory of memory retrieval, Psychological review 85(2) (1978)
[16] Ratcliff, R., Smith, P.I.,, McKoon, G.: Modeling regularities in response time and accuracy data with the diffusion model. Curr Dir Psychol Sci 24(6) (Dee 2015) 458-470
[17] Wiecki, T.V., Sofer, I., Frank, MJ..: Hddm: Hierarchical bayesian estimation of the drift-diffusion model in python. Frontiers in neuroinformatics 7 (2013) 55610
[18] Brown, S.D., Hcathcote, A.: The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive psychology 57(3) (2008) 153-178
[19] Trueblood. J.S., Brown, S.D., Heathcote, A.: The multiattribute linear ballistic accumulator model of context effects multialternative choice. Psychological review 121(2) (2014) 179
[20] De Boeck, P., Jeon, M.: An overview of models for response times and processes in cognitive tests. Front Psychol (2019) 102
[21] Wang, X.J.: Probabilistic decision making by slow reverberation in cortical circuits. Neuron 36(5) (Dec 2002) 955 968
[22] Wong, K.F., Wang, X.J.: A recurrent network mechanism of time integration in perceptual decisions. J. Neurosci. 26(4) (January 2006) 1314-1322 1324-1228
[23] Petrov, A.A., Van Horn, N.M., Ratcliff, R.: Dissociable perceptual- mechanisms revealed by
diffusion-model analysis. Psychonomic bulletin & review 18 (2011) 490-497
[24] Green, C.S., Pouget, A., Bavelier, D.: Improved probabilistic inference as a general learning mechanism with action video games. Curr Biol 20(17) (Sep 2010) 1573-1579
[25] Spoerer, C.J., McClure, P., Kriegeskorte, N.: Recurrent convolutional neural networks: A hetter model of
biological object recognition. Front. Psychol. 8 (September 2017) 1551
[26] Kreiman, G., Serre, T.: Beyond the feedforward sweep: feedback computations in the visual cortex. Ann.
N. Y. Acad. Sci. 1464(1) (March 2020) 222-241
[27] Kar, K., Kubilius, J., Schmidt, K., Issa, E.B., DiCarlo, JJJ.: Evidence that recurrent circuits are critical to the ventral stream's 's execution of core object recognition behavior. Nat. Neurosci. 22(6) (June 2019)
974-983
[28] Van Kerkoerle, T., Self, M.W., Dagnino, B., Gariel-Mathis, M.A., Poort, J., Van Der Togt, C., Roelfsema, P.R.: Alpha and gamma oscillations characterize feedback and feedforward processing in monkey visual cortex. Proceedings of the National Academy of Sciences 111(40) (2014) 14332-14341
[29] Spoerer, CJ., Kietzmann, T.C., Mehrer, J., Charest, I,, Kriegeskorte, N.: Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLOS Computational Biology 16(10) (10 2020) 1-27
[30] Goetschalekx, L., Govindarajan, L.N., Karkada Ashok, A., Ahuja, A., Sheinberg, D., Serre, T.: Computing human- like reaction time metric from stable recurrent vision models. Advances in Neural Information Processing Systems 36 (2024)

---

[31] Sensoy. M., Kaplan, L.. Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In Bengio, S., Wallach, H., Larochelle, H.. Grauman, K., Cesa--iaachhi N., Garnett, R.,. eds.: Advances in Neural Information Processing Systems. Volume 31., Curran Associates, Inc. (2018)
[32] Vickers, D., Packer, J.: Effects of alternating set for speed or accuracy response time, accuracy and confidence in unidimensional discrimination task. Acta psychologica 50(2) (1982) 179-197
[33]
Rahnev, D.: A robust confidence- accuraac dissociation via criterion attraction. Neuroscience of Con- sciousness 2021(1) (2021) niab039
[34]
Winkel, J., Keuken, M.C., van Maanen, L.,, Wagenmakers. E.J., Forstmann, B.U.; Early evidence affects later decisions: Why evidence accumulation is required to explain response time data. Psychonomic bulletin & review 21 (2014) 777-784
[35] Bogacz, R., Brown, E.,. Moehlis. J., Holmes, P., Cohen, J.D.; The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks. Psychological review 113(4) (2006) 700
[36] Drugowitsch, J., Moreno-Bote, R., Churchland. A.K., Shadlen, M.N., Pouget, A.: The cost of accumulating evidence in pereeptual decision making. Journal of Neuroscience 32(11) (2012) 3612-3628
[37] Cisek, P., Puskas, G.A., EI-Mum. S.: Decisions in changing conditions: the urgency-gating model. Journal of Neuroscience (97) (2009) 11560- 11571
[38] Cochrane, A., Sims, C.R., Bejjanki, V.R., Green, C.S., Bavelier, D.:
Multiple timescales of learning indicated by changes in evidence-accumulation processes during perceptual secision-making. npj Science of Learning 8(1) (2023)
[39] Kar, K., Kubilius, J. Schmidt, K., Issa, E.B., DiCarlo, JJJ:: Evidence that recurrent circuits are critical to the ventral stream's 's exccution of core object recognition behavior. Nature neuroscience 22(6) (2019) 974-983
[40] Palmer. J., Huk. A.C., Shadlen, M.N.: The effect of stimulus strength on the speed and accuracy of a perceptual decision. Vis 5(5) (May 2005) 376 404
[41] Shibata, K.. Chang, L.H., Kim, D., Náñez. Sr, J.E., Kamitani, Y., Watanabe. T., Sasaki, Y.: Decoding reveals plasticity in v3a as a result of motion perceptual learning. PLOS ONE 7(8) (08 2012) 1-7
[42] Law, C.T., Gold, J.I.: Neural correlates of perceptual learning in a sensory-motor, but not a sensory, cortical
area. Nature neuroscience 11(4) (2008) 505 513
[43] Krizhevsky, A., Sutskever, 1., Hinton, G.E.: Imagenet classification with deep convolutional neural
networks. Advances neural information processing systems 25 (2012)
[44] Mehrer, J.. Spoerer, C.J., Jones. E.C., Kriegeskorte, N., Kietzmann, T.C.: An ecologically motivated image dataset for deep learning yields better models of human vision. Proceedings of the National Academy of Sciences 118(8) (2021) c201 1417118
[45] DiCarlo, J.J., Zoccolan, D., Rust, N.C.: How does the brain solve visual object recognition? Neuron 73(3)
(2012) 415 434
[46] Serre, T., Oliva, A., Poggio. T.: A feedforward architecture accounts for rapid categorization. Proceedings of the national academy of sciences 104(15) (2007) 1424-6429
[47] Lin. T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D.,, Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision ECCV 2014: 13th Europcan Conference, Zurich, Switzerland, September 6- 62, 2014, Proceedings, Part V 13, Springer (2014) 740-755
[48] Kubilius, J.. Schrimpf. M., Kar, K., Rajalingham. R., Hong. H., Majaj. N., Issa, E., Bashivan, P., Prescott- Roy. J., Schmidt, K., et al.: Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems 32 (2019)
[49] Schrimpf. M., Kubilius, Lee, M.J., Ratan Murty, N.A., Ajemian, R., DiCarlo, JJ:: Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron 108(3) (November 2020) 413-423
[50] Hebart, M.N., Contier, O., Teichmann, L., Rockter, A.H., Zheng, C.Y., Kidder, A., Corriveau, A., Vaziri- Pashkam, M., Baker, C.I.: Things a multimodal collection of large-scale datasets for investigating object representations in human hrain and behavior. el 12 (feh 2023) e82580
[51] Fel, T., Rodriguez, I.F., Linsley, D., Serre, T.: Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS) (2022)
[52] Linsley, D., Rodriguez, I.F., Fel, T., Arcaro, M., Sharma, S., Livingstone, M., Serre, T.: Performance- optimized deep neural networks are evolving into worse models of inferotemporal visual cortex (2023)
[53] Liang, W., Tadesse, G.A., Ho, D., Fei-Fei, L., Zaharia, M.. Zhang, C., Zou, J.: Advances, challenges and opportunities in creating data for trustworthy ai. Nature Machine Intelligence 4(8) (2022) 669-677
12

---

[54] B.. Q., Liu, B., S.,
S., Liu, J. Pei, J., J., Zhou, B.: Trustworthy ai: From principles to practices. ACM Computing Surveys 55(9) (2023) 1-46
13

---

A Appendix Supplemental material
A.1 Complete derivation of the differentiable framework
Proposition 1. Let us define Tg (2:) = minft R1.,1 : 2t > 8 as the time in which D, reaches the threshold of activity 0. Provided that D, is continuously differentiable:
θτă dw D 2
OΦ,
(5)
dw
Proof. By definition,
(#,) min E [p : Dz 03. Let us consider small change 64. By the Taylor expansion, can write:
0rg 84. 0Φ.
T (1 + +8) TØ (2:)
On the other hand, é is considered continuous value. By our definition, we can induce small change in the value of & by introducing a small change time o④, which can write in the following way:
0 Öt
P1t5t 2 B. +
8t
Without loss of generality, let take 81
00:
i and then
0Φ.-
P1+5e R Tb +
5T = +; 5
Öt
Öt
Now,
T (#, + 00b)
min(1 E [L 4. + 64 > 0}
min [L ④.- > 0}
min (t Bi.M] (- 00, 64 0}
Note that, by definition, if the evidence is evolving by 2 It just means the whole function is moving along the t-axis, and therefore the minimum time to pass the threshold would be given by T A, therefore
T..
min(t E RI,M) : 1 (+((2)) (22)- = min(t E RI.NJ : 2.)
5
Ot
OΦ
-1
(Ø)
8$
01
Finally, we can join the two sides and obtain:
iΦ
T 41 + 84) = (2) +
84 = (2)
8
OΦ
Øt
From this, deduce:
ØTg
1Φ
-1
(6)
OΦ
Ot
ØTă
4 1-
(7)
06,
(1
ØTg
(8)
Fw
1.-1 Ow
So the first part of Eq in the main text is proved.
Similarly, we can derive
ØTH
14

---

8T
4
1--1
L To=
~
dw
dw
ow
8 t-
accumulated
Φ1-1
Ow
Evidence
1-
8T0
S 1-2
-
8TØ
slope
Φt- $ --1
To
Steps
Figure S1: Illustration of Mathematics Proof The discrete RNN step To is not differentiable. Therefore, introduce piecewise linear approximation of the accumulated evidence time t;- Consider the effect of changes in t on Ta (+). A small perturbation in & will produce a proportional change the time it takes for the accumulated evidence cross the threshold thereby inducing shift in time. In simple terms, fine-tuning w to decrease 4 will delay the time at which the network crosses the threshold 0 thus increasing To (o()), while fine-tuning increase 4 will cause the threshold to be crossed earlier thus decreasing TD(
15

---

A.2
Illustration of the RTified WW circuit
A
J12
LoUneise
12
$2
J12
Ju
Steps
S
11
Stimuli
Excitatory Connection
Jn
Inhibitory Connection
Population 2 Activity
Population Activity
80
0.6
0.6
60
Steps
0.4
0.4
40
Winning
0.2
0.2
Losing Populatton
0.0
0.4
Population Activity
Steps
Forward Function of WW circuit Algorithm for Random Dot Motion
Input: Image, threshold
Output: decision times
Initialize 82, $2, dlecisionnttmms
while $1 threshold and 82 thrreshold do
J, 12 t [=((((Imaee)
Combining excitatory and inhibitory currents
H1 f(Jn 81 J12 82 + To + lou noisel)
H2 f(222 82 J21 81 + lo I2 + lou noise2)
Calculate rate of change
10:
31 H1 (1
11:
deg
52 H2 (1 $2) To
Update 81, $2 decisioo_ttmee
end while
14: return decision,tingg
Figure S2: Illustration of the RTified WW circuit. RTify feedforward neural networks, we used an RNN based on the WW circuit. Here, consider binary classification on random moving dots for illustration (A). When receiving visual input, the two populations sensitive to left/right directions compete with each other (B) and accumulate evidence until one of them reaches threshold (C). The number time steps needed for the RNN to reach the threshold is defined as the "model RT" and is used to predict human RT. We provide pscudo-code for more detailed description of the circuit (D). Here, f(r) = max (az b, 0)/(1 - exp(- -d(ax - b))) is a fixed nonlinear function. And the model and equations marked blue how extend WW model.
16

---

A.3 Extended results for RDM task
A
Extracting RTs with supervision (fitting human RTs)
Human data (Green et al., 2010)
RTified RNN
Entropy-thresholding
51.:51.2
coh: 51.2
1e-2
coh: 25.6
1e-2
coh: 12.8
coh: 6.4
Probability
1.5
2-
1.0
2-
1-
0.5
0
0
0.0
0.0
2.5
1.25
1.25
2.5
2.5
1.25
0
1.25
2.5
2.5
25
0
1.25
2.5
2.5
1
26
1.25
2.5
incorrect
Correct
incomrect
Correct
incomeet
Comrect
Reaction times (s)
Reaction times (s)
Reaction times (s)
Reaction times (s)
1e-2
coh: 3.2
1e-2
coh: 1.6
1e-2
coh: 0.8
0.5
0.0
0.0
0.0
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
incompct
Correct
Incorrect
Correct
Correct
Reaction times
(s)
Reaction times (s)
Reaction times (s)
B
Extracting RTs via self penalty (leveraging RNN dynamics)
Human data (Green et al., 2010)
Self-penalized RTified RNN
Uncertainty proxy
16-2
coh: 51.2
3 16-2
coh: 25,6
2.0
16-2
coh: 12.8
1.5
16-2
coh: 6.4
Probability
4-
1.5
1.0
2-
1
0.5
0.0
0.0
2.5
25
1.25
2.5
2.5
1.25
0
1.25
2.5
2.5
1.25
0
1.25
2.5
2.5
1.25
0
1.25
2
Incorrect
Conrect
Incomeet
Conrect
Reaction times (s)
Reaction times (s)
Reaction times (s)
Reaction times (s)
1e-2
coh: 3.2
1a-2
coh:
1e-2
coh: 0.8
1.0
1.0
1.0
0.5
0.5
0.5
2.5
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
1.25
25
Incomest Reaction
Conrect
Incerrect
Comeect
Incomect
Correct
times (s)
Reaction times (s)
Reaction times (s)
Figure S3: RTified model evaluation on a RDM task (24) across all coherences. Human data are shown as a gray shaded area, and model fits are shown for (A) the "supervised" setting where human behavioral responses are used train the models and (B) the "self- seelf-penalized" setting where no human data is used. Our approach (green) outperforms the two alternative approaches (brown). i.e., entropy- threpo-grressoldinn 29 for the "supervised" and uncertainty proxy (30 for the "self self-penaliized settings.
17

---

Extracting RTs with supervision (fitting human RTs)
Human data (Green et al., 2010)
RTified C3D WW
1e-2
coh: 51.2
3 1e-2
coh: 25.6
2.0
1e:2
coh: 12.8
1.5
10-2
coh: 6.4
Probabiiity
1.5
1.0-
2-
0.5
0.5
0.0
0.0
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
1.25
2.5
2.5
1.25
0
1.25
2.5
Correct
Reaction times (s)
Reaction times (s)
Reaction times (s)
Reaction times (s)
coh: 3.2
coh: 1.6
coh: 0.8
0 5-
1.25
1.25 2.5
1.25
1.25
2.5
2.5
1,25
1.25
Irscrnedt
Coneet
Incomect
Conrect
Reaction times (s)
Reaction times (s)
Reaction times (s)
Figure S4: RTified WW model on RDM task [24] across coherences. We combined our RTified WW module with 3D CNN to fit human RTs collected in an RDM task 24). Human data are shown as gray shaded area and model results are shown as orange solid lines.
18

---

A.4 Additional Experiment for RDM task
Recurrent neural networks generate a sequence of outputs across time steps, raising the question of how to convert these multiple outputs into a single final prediction. In the main text, we combine all the network's outputs up until the decision time point. Here, as an additional experiment, we only use the network's output at the decision time point. In both "supervised" and "self-penaliza" settings, our approach surpasses two alternative approaches (29 (see Fig. (S5).
Extracting RTs with supervision (fitting human RTs) Human data (Green al., 2010)
RTifeed RNN
intropy-thressoldiig
1e2
1o2
coh:
2.0
1o>
coh: 12.8
1.5
1o2
Probabillty
2-
1.0
1.0
0.5
2.5
1.25
2.5
2.5
1.25
2.5
2.5
1.25
2.5
2.5
1.25
2.5
Incomed
Commc
Canrst Roaction times (s)
hounee
Inomest
Cumve:
homest
Carruci
Roaction times (5)
Roaction times (s)
Reaction times (s)
1ex2
coh:
coh:
1o>
coh:
1.0
0.5
2.5
1.25
1.25
2.5
2.5
1.25
2.5
2.5
2.5
Incenedt
Commet
rncened
Corrctt
Incorret
Comest
Reaction times (s)
Reaction times (s)
Reaction times (s)
Extracting RTs via self penalty (leveraging RNN dynamics)
Human data (Green al., 2010)
Self-penalized RTified RNN
Uncertainty proxy
1e-2
coh: 51.2
3 fe-22
coh: 25.6
20-
1e-2
coh: 12.8
1.5m
1e-2
coh: 6.4
Probability
4-
2-
1.5-
1.0-
2
0.5-
0.5
1.25
2.5
1.25
1.25
2.5
0.0
2.5
1.25
1.25
0.0
1.25
Reaction times (s)
Reaction times (s)
Reaction times (s)
Reaction times (5)
1e-2
coh: 3.2
1e-2
coh: 1.6
1ex2
coh:
1.0
1.0-
0.5
125
2.5
2.5
1.25
1.25
2.5
1.25
1.25
Conwca
Reaction (8)
Reaction times (s)
Reaction times
C
10
Entropy-thresholdee
RTified RNN
10
MSF
Uncertainty proxy
10
penalized RTified RNN
12..66651..
Coherence (%)
RDM task (24) across all Coherences.Human data are shown as a gray shaded area, and model fits are shown for (A) the "supervised" setting where human behavioral responses are used train the models and (B) the "self-penalized" setting where no human data is used. Our approach (green) outperforms the two alternative approaches (brown), i.e., entropy-three thrrop-thresholdigg [29] for the "supervised" and uncertainty proxy (30] for the "self pelff-penalized settings. MSE comparison is shown in (C).
Figure $5: RTified model evaluation
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the main contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The limitations of the proposed method are discussed, the scope and assump- tions are reflected.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification: All theoretical results, along with their assumptions and proofs, are provided in main text and supplementary files.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Detailed descriptions of the experimental setup, including datasets, hyperpa- rameters, and evaluation metrics, are provided. Links to the code and data can be found in our GitHub.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer:[Yes]
The code and datasets used in the experiments publicly available in GitHub, with detailed instructions provided.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
aublic//uuddes/Codeuubbsss catpssiooooolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
ssioiil for more details.
The authors should provide instructions on data access and preparation, including how to access raw data, preprocessed data, intermediate data, generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Comprehensive details on the training and test settings, including data splits, hyperparameters, and optimizer types, are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experimental results are presented with appropriate information about the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables Fig.s symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding Fig.s tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper specifies the types of compute resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://////eeaae....//////aaaaaa
Answer: [Yes]
Justification:
The research conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed the broader impacts of the proposed method, including potential applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
23

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve the release of high-risk data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper only involves published human data where all relevant experimental information can be found in the original papers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper only involves published human data where all IRB information can be found in the original papers.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
25

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Vn31nJW35H", "reviewer": "Reviewer_6b9L", "summary": "The authors present RTify, a novel approach that leverages Recurrent Neural Networks (RNNs) to model decision response times. RTify offers a dual benefit: it can align human and RNN response times, and it can self-supervise RNNs to optimize the speed-accuracy tradeoff. Through evaluations on both synthetic and natural image recognition tasks, the authors demonstrate that RTify outperforms existing methods (Goetschalckx et al., 2024) in fitting human response times, achieving superior results in both supervised and unsupervised settings.", "questions": "NA. Please refer to my review above.", "limitations": "Yes, the authors have sufficiently discussed limitations of their proposed work.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- The authors provide a compelling motivation for modeling human response time behavior using RNNs, addressing a significant gap in prior large scale vision models (used in machine learning) that often neglect temporal dynamics of decision-making. This work effectively bridges that gap, offering a more comprehensive approach to modeling human response behavior using RTified deep neural networks.\n- A significant advancement over prior work is the proposed method's ability to learn the stopping criterion end-to-end, either with human RT supervision or by optimizing the speed-accuracy tradeoff. This departure from relying on human-defined thresholds for stopping the RNN's processing marks a substantial improvement, enabling a more adaptive and autonomous modeling of reaction times.\n- The paper is well-written, with articulate language and high-quality figures that clearly elucidate the proposed method. \n- The experimental findings are intriguing, and the comparative analysis with Goetschalckx et al., 2024 is particularly informative, highlighting the notable improvements introduced by the current technique.", "weaknesses": "Weaknesses:\n- Although the proposed RTify method is a significant extension of prior art in modeling human response times, it bears notable similarities with existing approaches, such as Adaptive Computation Time [1] and its adaptation to ConvRNNs, AdRNNs in [2]. A more explicit discussion of these similarities and differences would strengthen the paper. Specifically, the analogy between [1, 2]'s halting probability ($p_t$, $P_t$) and RTify's evidence accumulation ($e(t)$, $\\phi(t)$), as well as the resemblance between the self-penalized loss function in Eq. 4 and the objective functions in [1] and [2], warrants clarification. The key distinction between RTify and ACT appears to be the learnability of $\\tau$ in RTify, whereas ACT treats this threshold as a hyperparameter (to clarify, I find closing this gap to be a strong contribution); a clear discussion of this difference (and any others that I have missed) would be beneficial for the readers.\n- In lines 67-69, the authors argue that prior work fails to provide a mechanistic account of the underlying decision processes governing temporal dynamics. However, the current work I believe also suffers from this limitation. This paper may further benefit from a more detailed mechanistic explanation of the temporal dynamics of decision-making in RTify. While the paper demonstrates impressive empirical results, a deeper understanding of the underlying mechanisms would further enhance its impact.\n- Minor: please check for spelling errors (e.g. add subscript t to $\\phi$ in Figure 1, \"Neural Netwok\" -> \"Neural Network\" in same figure etc.). Changing the variable name for halting step $\\Tau_\\tau$ would improve readability as the threshold for evidence accumulation ($\\tau$) also uses the same alphabet. \n\nReferences:\n1. Graves, A. (2016). Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.08983.\n2. Veerabadran, V., Ravishankar, S., Tang, Y., Raina, R., & de Sa, V. (2024). Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levels. Advances in Neural Information Processing Systems, 36.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Comparison between ACT**\n\nWe will add a comparison in the main text (see general rebuttal section).\n\n**Mechanistic account of RT**\n\nWe will revise our wording to provide a more accurate claim about the scope of our work. Here, we emphasize how RTify helps our understanding of RTs. First, our results provide computational evidence that reaction times (RTs) are generated through an evidence accumulation process. Second, our findings provide further evidence that the computational principle underlying RTs is to achieve an optimal speed-accuracy trade-off. Third, RTify offers a tool to model other factors, such as learning and adaptation, that are known to affect RTs. Researchers can apply RTify to various experimental conditions and compare the resulting parameters. \n\n**Spelling errors**\n\nWe will fix all these spelling mistakes in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kYMuxPr8vP", "reviewer": "Reviewer_imBE", "summary": "The paper introduces a new framework for training vision systems using human reaction times. The approach allows for dynamic integration of visual reasoning with decision making by incorporating human behavior over time. The approach is shown to be beneficial over a range of psychophysics tasks.", "questions": "* I was wondering how RT impacted performance for a while. Was there a good sense of this? Maybe the presentation could be cleaned up to present this more systematically. \n* I wasn’t able to get a sense of runtime for the WW circuit. This may also need discussion.", "limitations": "* I feel like some limitations were missing. For example, even though you don’t theoretically need human data in the self-supervised set-up, from my perspective, there is some reliance on human data for validation of this approach. Does this bottleneck the amount of data you can use?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A novel incorporation of new human behavior data, reaction time. The authors correctly point out a gap in incorporating this sort of behavior into models.\n* A novel and intuitive framework for incorporating reaction time in a non-linear method. The evidence accumulator was intuitive and well-explained. \n* Interesting results that demonstrate the RTify can train models that match human reaction times. \n* Application to both simulated (random dot) and real (object detection) visual tasks.", "weaknesses": "* The paper doesn’t explain why human RTs are useful. I read the introduction a few times and couldn’t explain the intuition about what incorporating RTs is solving. At some level, I believe there is an intuitive explanation: reaction times give a distribution of responses over time and therefore, this measure directly incorporates decision-making (a process of time) and visual reasoning at each time step. \n* A more robust explanation of human RT would be extremely helpful to explain more of the deep intuition. This might be entirely obvious but the paper makes RT seem like a novel contribution without explaining why it might be important. \n* It was a bit hard to connect your results to the reported performance of the models. Could results like ones reported in line 277 be put in a table? \n* (Nit) I found figure 7a a bit hard to understand. What do the correct and incorrect arrows mean?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Why is modeling RT important?**\n\nModeling RT is crucial for two main reasons. First, because of the so-called speed-accuracy trade-off (fast decisions come with high error rates, and vice versa), a model that explicitly accounts for behavioral decisions and RTs is expected to reflect human brain processing better.\n\nSecond, RT itself is an important measure reflecting the processing speed and efficiency of cognitive functions (Heitz, 2014) . It captures the dynamic nature of the visual and decision-making systems (Kar et al., 2019; Wyatte et al., 2014; Ratcliff 2002; 2008). Numerous studies have used RT as a behavioral measure to understand how stimulus complexity, number of potential responses, or stimulus-response compatibility affect human decision making (Kaswan and Young, 1965; Fitts and Seeger, 1953; Fischman, 1984; Reddi and Carpenter, 2000). Integrating mechanisms responsible for behavioral choices and RTs will thus lead to a more complete model of human visual processing.\n\n**Organizing the model performance results**\n\nThank you for the suggestion. We have organized the model results into a table (see rebuttal pdf).\n\n**Figure 7a**\n\nThe distribution shown in Figure 7a is a distribution of signed RT, where the RT of a correct trial is positive, while the RT of an incorrect trial is negative. The distribution of signed RT is routinely used in decision-making studies to summarize experimental data into a single plot: (1) it shows the distribution for RT in all trials (2) the relative proportion of positive RTs over negative RTs reflects the accuracy. In this case, when our model shows a close match with such a distribution, we are successful in modeling both the accuracy and RT at the same time.\n\n**How does RT impact performance?**\n\nWe are not entirely sure whether the reviewer is referring to the performance of human participants or the RTify model.\n\nIn terms of human performance:  RT and performance are interdependent, as both behavioral choices and RTs are functions of the stimuli. Recent studies have found that if one is prone to error, one may deliberately take more time to decide (Adkins et al., 2024). This example shows how two factors may dynamically regulate each other. We believe this feature makes our RTify an important method since it now helps explain both the choices and the RTs.\n\nIn terms of the performance of how RTify predicts human decisions: this is a great question and we would love to look into this more closely. We expect a model trained on combined choice and RT data to better predict choice data. This could in theory be easily tested. Unfortunately, the datasets we use either provide choice data without the stimuli (Green et al., 2010) or they do provide the stimuli but do not provide choice data (Kar et al., 2019). We hope our paper can encourage researchers to collect more comprehensive datasets in order to better validate our method.\n\n**Runtime of the WW circuit**\n\nThank you for raising this. We do see that more discussion over the WW circuit will help readers better understand the application of RTify over a specific canonical circuit. To this end, we put a detailed discussion of WW circuits in the general rebuttal section. \n\n**Limitations missing**\n\nWe apologize for not being explicit about the limitations. We carefully took your suggestion into consideration and have written a separate limitation section (see general rebuttal section.)\n\n**Reference:**\n\nKar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.\n\nGreen, C. S., Pouget, A., & Bavelier, D. (2010). Improved probabilistic inference as a general learning mechanism with action video games. Current biology, 20(17), 1573-1579.\n\nHeitz, R. P. (2014). The speed-accuracy tradeoff: history, physiology, methodology, and behavior. Frontiers in neuroscience, 8, 150.\n\nKar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.\n\nWyatte, D., Jilk, D. J., & O'Reilly, R. C. (2014). Early recurrent feedback facilitates visual object recognition under challenging conditions. Frontiers in psychology, 5, 674.\n\nRatcliff, R., & Tuerlinckx, F. (2002). Estimating parameters of the diffusion model: Approaches to dealing with contaminant reaction times and parameter variability. Psychonomic bulletin & review, 9(3), 438-481.\n\nRatcliff, R., & McKoon, G. (2008). The diffusion decision model: theory and data for two-choice decision tasks. Neural computation, 20(4), 873-922.\n\nKaswan, J., & Young, S. (1965). Effect of stimulus variables on choice reaction times and thresholds. Journal of Experimental Psychology, 69(5), 511.\n\nFischman, M. G. (1984). Programming time as a function of number of movement parts and changes in movement direction. Journal of Motor Behavior, 16(4), 405-423.\n\nReddi, B. A. J., & Carpenter, R. H. (2000). The influence of urgency on decision time. Nature neuroscience, 3(8), 827-830.\n\nAdkins, T. J., Zhang, H., & Lee, T. G. (2024). People are more error-prone after committing an error. Nature Communications, 15(1), 6422.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGPPhIAXKs", "reviewer": "Reviewer_UxtZ", "summary": "Here the authors present a method for fitting neural network outputs to reaction times. The main technical step is setting up the calculation of a reaction time as an accumulation of a decision signal over time steps and creating a differentiable computation of the time when this accumulation crosses a threshold. Here the authors use a simple linear interpolation between the time points to allow reaction times between the time points and a differentiable dependency on the input signal for accumulation. The authors use this possibility to optimise reaction times to match average reaction times of human observers and to minimise reaction time of networks optimised for performance. By using an existing network architecture (“Wong-Wang”) the authors are able to create realistic reaction time distributions as well.", "questions": "Can this approach be extended towards creating predictions for other manipulations of reaction time?", "limitations": "While I don’t think the authors are hiding any negative social impacts or severe limitations, I do think they are vastly overestimating the broader impact of a method for predicting RT distributions based on neural networks. I like this work, but it does not allow alignment with the full spectrum of human behavioural data, it will not make DNNs more trustworthy for now and there is exactly one supervised and one less supervised situation, not many.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is successful in fitting Reaction times and is very generally applicable as the only requirement is a scalar input for the preferred decision.\nAlso the method is well grounded in the literature on how reaction time distributions are generated.\nAlso the fit is better than two existing competitor methods.", "weaknesses": "While the method presented here is fairly general it is still fairly specific in terms of the data it applies to: Timed categorisation tasks without cues, planning, rewards etc. This is a step forward from just accuracy, but clearly far from all decision making behaviour of humans.\n\nWhile the deep neural network based methods are sensible comparisons, the adapted Wong-Wang architecture received little test in this manuscript. For insights into the modelling, I believe a more thorough discussion of the concrete model used and which aspects of it are relevant for getting the RT distributions right could improve my trust into this method.\n\nThe data used here for the natural image categorisation task is not a great choice. While it has a substantial amount of images it is an MTurk dataset with corresponding limitations in accuracy of stimulus presentations and measurements. There are plenty of better controlled psychophysics datasets that could be used for evaluations of RT models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Psychophysics tasks**\n\nFirst, for the natural image dataset, we are using the dataset already peer-reviewed and published in Nature Neuroscience (Kar et al., 2019). As the reviewer pointed out, this is done via the Amazon Mechanical Turk (Mturk) platform. The reviewer is correct that these online experiments are not as easy to control compared to in-lab experiments but they come with many benefits (most prominently data collection is orders of magnitude faster to collect compared to in-lab experiments) and with they are now widely used by a large number of psychology labs (see Buhrmester et al., 2018 for a review). \n\nWe acknowledge that behavioral choices and RTs provide only a partial picture of human decision-making. However, we believe that these RTs and accuracy are by far the two most crucial metrics in the field of visual decision-making (Ratcliff, 2002; 2008; Chuan-Peng et al., 2002; Fengler et al., 2002), as they often reflect fundamental mechanisms of information processing. We hope our work will inspire experimentalists to conduct both well-controlled in-lab experiments and online experiments to further explore RTs and decision-making behavior.\n\n**WW architecture**\n\nFirst, we will add a detailed discussion of the details of the WW model in supplementary information (see general rebuttal) and we hope it can clarify the confusion. \n\nSecond, we also followed this reviewer’s suggestion and conducted an ablation study of the circuit for the random dot motion tasks. Our findings revealed an intriguing result that may prompt further research: ablating self-excitation causes the model to fail, resulting in flat RT distribution (MSE increases by 3369%), while ablating cross-inhibition deteriorates model performance (MSE increases by 38.5%). We thank the reviewer for bringing this up.\n\n**Predictions for other manipulations of reaction time**\n\nBecause RTify tackles the task on a frame-by-frame basis, a practical future experiment could investigate how individual frames of the random moving dots stimulus affect RTs. Typically, in random dot motion tasks, researchers use a consistent coherence level and direction throughout each trial. However, our method enables frame-based interventions. For example, researchers could enhance the coherence of certain frames to determine the importance of early, mid, or late frames. Alternatively, specific frames could be reversed to an opposite direction or masked entirely. RTify opens up these possibilities, offering researchers new avenues for experimentation.\n\n**Lack of limitations and overstating broader impacts**\n\nWe will soften our claims regarding broader impacts and include a limitation section so that readers can correctly understand the pros and cons of our current methods. \n\n**Reference:**\n\nKar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019). Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nature neuroscience, 22(6), 974-983.\n\nFengler, A., Bera, K., Pedersen, M. L., & Frank, M. J. (2022). Beyond drift diffusion models: Fitting a broad class of decision and reinforcement learning models with HDDM. Journal of cognitive neuroscience, 34(10), 1780-1805.\n\nBuhrmester, M. D., Talaifar, S., & Gosling, S. D. (2018). An evaluation of Amazon’s Mechanical Turk, its rapid rise, and its effective use. Perspectives on psychological science, 13(2), 149-154.\n\nChuan-Peng, H., Geng, H., Zhang, L., Fengler, A., Frank, M., & ZHANG, R. Y. (2022). A Hitchhiker’s Guide to Bayesian Hierarchical Drift-diffusion Modeling with Dockerhddm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H6j0wTRjYk", "reviewer": "Reviewer_QgJJ", "summary": "The  paper proposes a novel framework to align the temporal dynamics of RNNs and human reaction times for visual recognition tasks. The framework starts from a task-optimized RNNs and trains a function $f_w$ to transform the activity (hidden state) into a real-valued evidence measure $e(t)$ (learnable) that will be integrated over time by an evidence accumulator (seems to be a summation). To train the stopping threshold and the function $f_w$, authors propose 2 approaches: supervised and self-supervised.", "questions": "1. The formulation of $\\Phi$ function is not clear, and how the loss is actually propagated in (3) and (4). \n2. Figure 3, the Wong-Wang model is mentioned, the structure should be illustrated for the reader to understand it without going to the reference. This is because RTifing is one of the main contributions of the paper. \n3. If understand correctly, the dynamic computation time comes only from WW model, and the CNN stays exactly the same, and runs only once. Probably CNN run time dominates the full model, please comment on this.", "limitations": "Not listed in the paper as a separate section.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem sounds interesting - adaptive computation time and its relationship to human reaction times. \n2. The solution is novel, using WW models and particular training loss is are not common in NN field.\n3. The method, RTify, is a nice way to train with non differentiable loss.", "weaknesses": "1. It sounds like the paper writing is a bit overcomplicated. The actual approach is applying RNN (in the WW form that is not clear) on top of the model like CNN, and then learning to reduce the number of RNN iterations via the supervision or via regularization. \n2. Benchmarks selected for visual experiments are not strong. Why not to consider CIFAR10/100, ImageNet, COCO detection etc? \n3. Details of WW models are not mentioned in the main paper. Probably, listing PyTorch pseudo code for forward pass will be helpful here. \n4. The problem of adaptive compute is not novel for RNNs. Please consider comparing with other methods in the filed like ACT (\"Adaptive Computation Time for Recurrent Neural Networks\" by Alex Graves).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Methods of the current paper and the comparison between ACT**\n\nWe will definitely strive to make the paper clearer and simplify our writing. We think our writing may have caused some confusion and here are some points we would like to clarify. \n\nFirst, our main goal is to align the computational steps of an RNN with human RTs. Our main approach is not simply to place an RNN on top of a convolutional neural network (CNN) but to develop a differentiable framework that allows us to train an integrated model end-to-end on experimental data. Results from Figures 4-6 clearly show that we are able to derive such an RT measure better than previous models. After establishing that RNNs can be aligned with human decisions, we extended our work to align CNNs with human decisions. As this reviewer correctly pointed out, we, therefore, integrated an RNN with a CNN. Combined with our RTify method, we obtained an RT measure directly from the RNN steps required to complete the task. Figure 7 shows results similar to those in Figures 4-6.\n\nSecond, we used RTify for a dual purpose: (1) to fit human RTs, and (2) to flexibly minimize the number of computational steps needed to complete the task. This requires a computational method to directly optimize RTs. This cannot be done with ACT because it does not incorporate a way to directly train a loss function with respect to RTs. Instead, ACT can only indirectly be used for (2) by maximizing cumulative activities. Our gradient approach in Section 2.0 and Appendix D allows us to effectively achieve both goals (1) and (2) within the same framework.\n\nWe have provided a detailed comparison and clarification in the general rebuttal section to aid understanding. In brief, ACT is only compatible with image tasks using a self-supervision goal. Our results show that the model RT derived from the ACT method does not significantly correlate with human RT (r = 0.05, p = 0.24), see Fig R2.\n\n**Dataset issue**\n\nWe understand that a large dataset will help strengthen our work. However, to our knowledge, none of the large datasets mentioned here (CIFAR10/100, ImageNet, COCO detection) come with human behavioral data and, hence, cannot be used to evaluate our model.\n\n**WW architecture**\n\nWe added a detailed explanation (see general rebuttal.) \n\n**Formula of Phi**\n\nFor the $\\Phi$ function, it is a summation of $e(t)$, and $e(t)$ is $f_w (h_t)$. Where $f_w$ is achieved with an MLP.\n\nThere are two two different loss formulations for two purposes  in the paper (i.e. fitting human RTs and minimizing time steps.)\n\nFor fitting RT, please directly refer to formula (3). Here $F$ is the mean square error(MSE) between the human RT distribution and model RT distribution. \n\nFor minimizing time steps, please refer to formula (4). And here we elaborate more on how to get the gradient. There are two terms in formula(4). One is the regular cross-entropy loss, which is totally differentiable and not worth discussion. The other is the penalty of RT, i.e. ($l_y \\cdot \\Tau_{\\tau}$). When we are trying to get the gradient of the penalty term over $w$, since it is bivariate function, we first apply the chain rule $\\partial(\n  l_y \\cdot \\Tau_{\\tau})/\\partial w = \\Tau_{\\tau} \\cdot \\partial l_y / \\partial  w + l_y \\cdot \\partial \\Tau_{\\tau}/\\partial w $.\n(formula *) For the first term of (formula *), since $l_y$ are just the logits of the network, that part is differentiable in the regular way. For the second term of (formula *), we need to apply formula (3) with a special case that $F$ is the identity function.\n\n**Runtime analysis of WW model**\n\nWe are not entirely sure we understand the question because runtime is not really relevant here. The RT in the WW model is the number of time steps it takes for the circuit to make a decision. And we use that measure to model human RT. \n\nIn case we misunderstood, here is the actual runtime complexity of the WW model:  \n\nInput computation (2 matrix operations: one for each neuron).\n\nSynaptic activity update (2 differential equations solved).\n\nNoise update (2 noise generation and update operations).\n\nTrajectory update (storing the values for later use).\n\nTherefore, the WW circuit runtime is primarily governed by the number of time steps and the size of the batch to be processed, which results in a complexity of O(time_steps * batch_size). The reviewer is right that the total runtime of the RTified CNN is dominated by the forward step of the CNN since it contains most of the parameters, but we want to emphasize again that it has nothing to do with the RT.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for providing valuable feedback. We appreciate the time and expertise shared with us, and we are confident that we have addressed all raised concerns. Our paper is now stronger than before. The converging concerns are listed and answered in our general rebuttal, while each unique comment is answered individually. 

**What is the difference between our method (RTify) and the existing adaptive computing method in RNNs (e.g. adaptive computing time, ACT)?**

There are three main points we want to mention: 

First, the time steps in ACT are not differentiable and thus ACT cannot be used to fit human RTs where the loss is a function of time steps. Therefore, ACT cannot be used in our supervision case.

Second, ACT cannot be applied to image sequences (videos). There are two time components in ACT: the number of recursive steps for processing each frame (N or ponder time), and the number of frames an RNN ‘watches’ (t). ACT optimizes the former but not the latter, which is suitable for minimizing computational resources but not for modeling RT data. 

Third, although ACT can technically be run static images (i.e., a sequence of length 1 frame; the ponder time would then be treated as a reaction time) in the self-supervision case, our experiments show that RTs derived from the ACT method do not significantly correlate with human RTs (r = 0.05, p = 0.24) in the image classification task, see Fig. R2.

**What is the specific architecture of the Wong Wang (WW) model?**

Unfortunately, we had to omit details in the original submission due to space constraints. However, we agree with the reviewer and we will provide the missing details. In particular, we have created Figure R1 (please find it in the attached pdf), which will be included in the revised manuscript, along with pseudocode illustrating the model's forward pass.
In our extension of the WW model, we replaced the linear transformation of the coherence levels with a feedforward network. Additionally, we extend the model from 2 neural populations to M neural populations (We put the figure for M neural populations in Figure 3 and also a figure for 2 neural populations in Figure R1). Following these modifications, the parameter count in the WW circuit increases to O(M^2). As mentioned in the main text, the original WW model's manual tuning is practical for small M but infeasible as M grows. With RTify, the entire architecture is trained with backpropagation, as the ordinary differential equations (ODEs) in the WW model can be transformed into a discrete-time recurrent neural network (RNN), making it compatible with RTify.
The results using the WW model are summarized in Figure 7 in the main paper. It is important to note that we only used the WW model together with convolutional neural networks (CNNs) as opposed to RNNs. We selected the Wong-Wang model because it is the leading neural circuit model of decision making.

**What are the limitations of the current paper?** 

Reviewers imBE, UxTz, and Qgjj pointed out that we did not clearly state the limitations of the current paper. In order to make readers understand our method better, we provide a separate limitation section here. And surely we will include this in our camera-ready version.
In our paper, we have described a computational framework (RTify) to align RNNs with human behavioral decisions, including both choices and RTs. RTify offers two benefits: it can be used to align RNN steps with human RTs, and it can be used as a self-supervised approach to train RNNs to learn to find an optimal speed-accuracy tradeoff. However, there are certain limitations that need to be addressed in future work.
First, most of the human datasets used in our study remain relatively small-scale and largely limited to synthetic images because most current benchmarks focus exclusively on behavioral choices and lack RT data. To properly evaluate our approach and that of others, larger behavioral datasets using more realistic visual stimuli will be needed. We hope that this work will encourage researchers to collect novel internet-scale benchmarks that include not just behavioral choices but also RTs.
Second, although behavioral choices and RTs are widely studied in the field of visual decision-making, they only partially reflect the nature of human decision-making. For a more complete model of decision making, non-visual factors including confidence and specific aspects of cognitive strategies (e.g. search) will need to be integrated into the model. We recognize that incorporating all known factors into deep neural network models will be crucial for building a trustworthy AI. Nevertheless, we feel that RTify as a method for modeling both behavioral choices and RTs is a positive first step in this direction.

Meta-review: This submission addresses the challenge of modeling human reaction times (RTs) using recurrent evidence accumulation. The main contribution of this work is a differentiable approximation of simulated RTs. This approximation enables the incorporation of RTs, operationalized as the number of recurrent computation steps until an adaptive stopping criterion is met, into the training loss. This approach is employed to train networks either to fit human RTs or to reach decisions as rapidly as possible. In the latter application—training networks to reach decisions quickly—the authors demonstrate improvements over state-of-the-art methods. To the best of my knowledge, the former application—directly fitting human RTs—is novel.

Most of the reviewers believe that the contribution of this work warrants publication at NeurIPS, and I share their assessment. However, the reviewers commonly pointed out several weaknesses: overreaching conclusions, insufficient discussion of limitations, and a lack of clarity in some technical sections, particularly regarding the Wong-Wang RNN models. The authors have committed to addressing these issues in the revised manuscript. It is crucial that the authors provide well-documented and functional public code. Another point to consider in the revision is the omission of a relevant reference: Rafiei et al. (2024), published online shortly before the submission deadline:

Rafiei F, Shekhar M, Rahnev D. The neural network RTNet exhibits the signatures of human perceptual decision-making. Nature Human Behaviour. 2024 Jul 12:1-9.
Decision: Accept (poster)