CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Authors: Songlin Xu
Xinyu Zhang
Venue: ICML 2025 poster (2025)
Keywords: Cognitive Response
Deep Reinforcement Learning
Dynamic Stimuli
Primary area: applications->neuroscience_cognitive_science
TL;DR: This paper presents a computational framework integrating cognitive models and data-driven deep reinforcement learning to model human cognitive response modulated by dynamic environmental stimuli.

Abstract: Using deep neural networks as computational models to simulate cognitive processes can provide key insights into human behavioral dynamics. Challenges arise when environments are highly dynamic, obscuring stimulus-behavior relationships. However, the majority of current research focuses on simulating human cognitive behaviors under ideal conditions, neglecting the influence of environmental disturbances. We propose CogReact, which integrates drift-diffusion with deep reinforcement learning to simulate granular effects of dynamic environmental stimuli on the human cognitive process. Quantitatively, it improves cognition modeling by considering the temporal effect of environmental stimuli on the cognitive process and captures both subject-specific and stimuli-specific behavioral differences. Qualitatively, it captures general trends in the human cognitive process under stimuli. We examine our approach under diverse environmental influences across various cognitive tasks. Overall, it demonstrates a powerful, data-driven methodology to simulate, align with, and understand the vagaries of human cognitive response in dynamic contexts.

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Songlin Xu'
Xinyu Zhang'
Abstract
Battleday et al., 2021; Peterson et al., 2021) predominantly concentrates on modeling human cognition under standard and ideal conditions, often neglecting the nuanced impact of environmental stimuli (Do et al., 2021). Alternatively, some studies treat environmental stimuli as a constant factor throughout the cognitive process (Bourgin et al., 2019).
Using deep neural networks as computational models to simulate cognitive processes can pro- vide key insights into human behavioral dynamics. Challenges arise when environments are highly dynamic, obseuring stimulus-behavior relation- ships. However, the majority of current research focuses on simulating human cognitive behaviors under ideal conditions, neglecting the influence of environmental disturbances. We propose CogRe- act, which integrates drift-diffusion with deep reinforcement learning to simulate granular ef- fects of dynamic environmental stimuli on the hu- man cognitive process. Quantitatively, it improves cognition modeling by considering the temporal effect of environmental stimuli on the cognitive process and captures both subject-specific and stimuli-specific behavioral differences. Qualita- tively, it captures general trends in the human cognitive process under stimuli. We examine our approach under diverse environmental influences across various cognitive tasks. Overall, it demon- strates a powerful, data-driven methodology to simulate, align with, and understand the vagaries of human cognitive response in dynamic contexts.
We propose that more nuanced modeling approach is im- perative, particularly when dealing with dynamic stimuli that can fluctuate over time, contingent upon user perfor- mance. This nuanced approach involves stimuli variation at fine-grained timescales, exerting a continuous influence on human cognitive behaviors. To illustrate, consider an ani- mated visual stimulus conveying time pressure (Slobounov et al., 2000). Such stimuli inform users of the passage of time, evoking sensations of pressure. Representing these stimuli as a binary existence indicator would oversimplify their nuanced effects. Therefore, this paper raises a funda- mental question: How to simulate the impact of dynamic environmental stimuli on the regulation of human cogni- tive behaviors with precision at a fine-grained level?
We address this question starting by examining how dy- namic time pressure stimuli (Zur & Breznitz, 1981) influ- ence cognitive performance, particularly within the context of a math arithmetic task, a widely utilized benchmark for evaluating human cognition and logical reasoning (Lin et al., 2011; Judd & Klingherg, 2021; Daitch et al., 2016). The dynamism inherent in time pressure feedback encompasses two primary facets. Firstly, the presentation of time pres- sure can be dynamic, involving the delivery of progressively changing visual frames over time (Fig. 5(a)), thereby in- stilling a sense of urgency. Secondly, the presence of time pressure may vary dynamically across different trials. Since time pressure stimuli represent a well-established feedback modality to modulate human cognitive performance (Cheng, 2017; Slobounov et al., 2000; Moore & Tenney, 2012; Ed- land & Svenson, 1993; Whittaker et al., 2016), modeling such modulation effect holds the promise to offer valu- able insights in understanding human cognition (Jaffe et al., 2023) and facilitating adaptive intervention design for regu- lating user performance (Costa et al., 2019).
1. Introduction
Modeling human cognition is a fundamental challenge in understanding human behaviors (Jaffe et al., 2023). In partic- ular, modeling the effects of environmental dynamics (e.g.. stress (Cheng, 2017) and feedback (Costa et al., 2019)) on cognitive performance could elucidate behavioral responses to tasks (Cheng, 2017) and inform the design of feedback mechanisms to augment cognition (Costa et al., 2019). How- ever, prior research (Jaffe et al., 2023; Peterson et al., 2018;
Department of Electrical and Computer Engineering, Univer- sity of California San Diego, La Jolla, CA 92093, USA. Corre- spondence to: Songlin Xu <soxu uuc...edu>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we introduce a systematic hybrid framework (CogReact) depicted in Fig. 1. This framework integrates

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
b
d
Drift-Diffusion Model
stimui
Observatio Space Rt
perturbation
M
Start Point
Disturbed
RL
Agent Logical Reasoning
Transfer Agent
Decoding Logical Reasoning Process from Resporse
Simulating Perturbation Logical Reasoning Process
Figure 1. Illustration of the overall framework, First, we train a logical reasoning agent to solve cognitive tasks without considering user response. Sccond, transfer features extracted from the logical reasoning agent without time pressure to real user choice and response time (initial estimation). Third, the initial estimated response time and predicted choice probability generate an evidence accumulation trajectory the drift-diffusion model. Lastly, the DRL agent simulates influence stimuli perturbation on cognitive process by taking dynamic environmental stimuli input and takes specific action to modulate evidence accumulation process. When evidence accumulator achieves the boundary threshold, the final prediction response time is generated and DRL agent achieves terminate state.
a classical closed-form cognitive model into a data-driven deep reinforcement learning (DRL) approach, allowing for a comprehensive and explainable simulation of the impacts of dynamic, fine-grained time pressure stimuli.
datasets more diverse task (decision making, learning) and feedback modalities (numeric, textual). summary, our contribution three-folded:
While neural networks (NNs) are recognized for their pro- ficiency in function approximation and have been applied to model cognitive behaviors (Bourgin et al., 2019), their inherent black- box nature poses challenges in representing the internal mechanisms of the cognitive process.
We propose CogReact, a hybrid framework incorpo- rate classical cognition models (drift-diffusion model) with deep reinforcement learning to simulate perturba- tion of environmental stimuli on evidence accumula- tion process during human cognitive response.
To address this limitation, our framework integrates DRL with the drift-diffusion model (DDM), sequential sampling method widely employed in cognitive modeling (Ratcliff & McKoon, 2008; Steyvers et al., 2019). DDM posits that humans make decisions by accumulating evidence until reaching a boundary threshold (Fudenberg et al., 2020). The simulated choice and response time are then determined based on the corresponding boundary and accumulation time. While DDM excels in representing the cognition pro- cess in explicable and fine-grained manner, it primarily focuses on posterior estimation of user decisions rather than predicting user future performance under stimuli.
We comprehensively validate the effectiveness of our framework by comparing with range of baselines and extensive ablation studies. Additionally, we are open- sourcing our code and a newly collected large dataset', comprising 21, logical reasoning responses, as a contribution to the research community.
We demonstrate how framework is adapted to three representative cognitive tasks: mathematical reason- ing, decision making, and learning. This adaptation for both discrete user inputs and continuous behaviors establishes a foundation for extending the framework accommodate diverse user cognitive responses.
On the other hand, DRL, with NNs at its core, offers a step- by-step interaction environment. This environment enables the incorporation of the fine-grained cognitive process in- herent in DDM while retaining the function approximation capabilities of NNs. This hybrid approach bridges the gap between the transparency of classical cognitive models and the flexibility of data-driven methods, presenting a promis- ing avenue for modeling the intricate dynamics of cognition under dynamic time pressure stimuli.
2. Related Work
Cognitive Process Models. The existing literature has amassed empirical evidence supporting feasibility of model- ing human cognition (De Boeck & Jeon, 2019). Traditional cognitive models, exemplified by BEAST (Erev et al., 2017) and the trift-diffusio model (DDM) (Rateliff & McKoon, 2008; Steyvers et al., 2019), are characterized by closed- form structures. For example, DDM (Ratcliff & McKoon,
In addition, further show the generalization ability of our framework by extending it into additional public
http://ghhhbhoommogrra//CCggge
2

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
2008) treats the cognitive process as an evidence accumula- tion process for humans to make decisions, thus simulating the speed-accuracy tradeoff (Heitz, 2014).
enhance dataset diversity and evaluate our model under dy- namic environmental stress, participants were randomly and uniformly distributed across four distinct groups: None Group: Participants experienced no time pressure for any trial. Static Group Time pressure was consistently applied for each trial. Random Group: There was a 50% probability of time pressure being applied for each trial. Rule Group: Time pressure was adaptively applied based on user past per- formance using rule-based strategy (more details of such strategy are provided in Appendix A.3.4). This collection has been approved by the Institutional Review Board (IRB) in our local institution. More details are in Appendix A.2.
Cognitive Simulation with Machine Learning. More re- cently, there has been a notable shift toward the integration of machine learning techniques (Cichy & Kaiser, 2019) for simulating human behaviors (Peysakhovich & Naecker, 2017; Lake al., 2017; Ma & Peters, 2020) across an ar- ray of tasks, including visual cognition (Cho et al., 2023; Wenliang& & Seitz, 201 8), categorization (Battleday et al., 2017), decision making (Binz& & Schulz, 2024; Peterson et al., ; Bourgin et al., 2019), game strategy (Hartford et al., 2016), human exploration (Binz& & Schulz, 2022), word learning (Ritter et al., 2017), etc.
Our dataset analysis in Appendix A.4 revealed that human accuracy remained unaffected by external stimuli, as partic- ipants were instructed to prioritize accuracy over speed to control the speed-accuracy tradeoff (Heitz, 2014). Conse- quently, to model cognitive response due to external stimuli, we focus on simulating response time rather than choice accuracy, aligning with (Goetschalckx et al., 2024).
Response Time Simulation. Of particular note, recurrent neural networks (RNNs) (Jaffe et al., 2023; Song et al., 2017; 2016) have been adapted to execute various cogni- tive tasks (Yang et al., 2019) emulating human performance and the balance between accuracy and response time ob- served in biological vision (Spoerer et al., 2020). Recently, (Goetschalckx et al., 2024) computed human-likk reaction time from convolutional RNN using evidential deep learning (Sensoy et al., 2018). Furthermore, task- -yk VA (Jaffe et al., 2023) modeled cognitive response time with RNN-based latent dynamical systems.
3.2. CogReact Framework
Inspired by exploratory analysis (Appendix A.4) and exist- ing cognitive theories (Roseboom et al., 2019; Yang et al., 2019; Mickey & McClelland, 2014), our framework com- prises four key steps, as illustrated in Fig. 1. In the initial step, we train a long short-term memory (LSTM)-based logi- cal reasoning agent (termed math agent) to proficiently solve the designated cognitive task. The second step involves the knowledge transfer from these trained agents to establish mappings from the LSTM agent to human performance met- rics. This yields predictions for human response time and accuracy for each trial. Moving to the third step, we em- play a fine ene-grained Drift-Diffusion Model (DDM) to decode human performance, extracting detailed information about response time and accuracy. This step is pivotal in gener- ating the evidence accumulation process (EA) reflective of the underlying cognitive mechanisms. In the final step, we introduce a deep reinforcement learning (DRL.) agent to the framework. This agent plays a crucial role in simulating the impact of stimuli perturbation on the evidence accumulation process. By leveraging DRI,, we can capture the nuanced dynamics of how external stimuli, such as time pressure, influence the intricate logical reasoning processes modeled by the DDM. We describe details of the first two steps in Section 3.3 and the last two steps in Section 3.4.
These existing models predominantly simulate response time under ideal conditions. In contrast, limited work has focused on modeling the impact of external stimuli pertur- bations, such environmental stress, on task performance. We argue that more nuanced modeling approach is essen- tial, especially when addressing dynamic external stimuli that fluctuate over time based on user performance. This refined approach requires capturing stimuli variations at fine- grained timescales, which exert a continuous and evolving influence on human cognitive behaviors.
3. Model and Methodology
3.1. Math Reasoning Task and Dataset
We used a math arithmetic task with time pressure visual stimuli as our initial model exploration context. The illus- tration of the task and stimuli is depicted in Fig. 5 and in Appendix A.2. In each math trial, participants were presented with two two-digit numbers and tasked with de- termining whether their subtraction result was divisible by given one-d number. Participants made binary decision for each trial, with varying settings of time pressure stimuli described below.
3.3. Math Agent and Transfer to Humans
To simulate the impact of time pressure, it is imperative to first predict user baseline performance in ideal condi- tions without time pressure. Drawing inspiration from prior research that models human subjects' time perception by capturing internal activities in perceptual classification net-
We collected an extensive dataset encompassing 21,157 valid responses (choice accuracy and response time) from 44 participants engaged in the task (see Fig. 6(a)). To
3

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
varying levels of difficulty, leading to user choice biases and variations in response time (Hanich et al., 2001). The LSTM-based agent has the capacity to capture these poten- tial differences in difficulty levels (Mickey & McClelland, 2014; Zaremba & Sutskever, 2014), and the SVM is em- ployed to map these to user choice (via SVC, a classification model in SVM) and response time (via SVR, a regression model in SVM). More details on the rationale of the math answer agent and SVM models are provided in Appendix
Table 1. Evaluution of selected baselines in math reasoning task. For MAPE, we show its mean value (Mean), standard deviation (STD). More complete results with more baselines are in Table 4.
MАРE
Model Input Type
Model Type Name
Mcan
STD
hGRU
0.3335 0.2486
I.
LSTM + AlexNet
0.3344 0.2602
Task: Video, Feedback: Video
LSTM + VGG-16
0.3355 0.2708
LSTM VIT-B-16
0.3339 0.2573
A.5, A.6, Fig. 7.
MLP+ +D ResNet
0.3330 0.2507
LSTM-VI + 33 ResNet
0.3334 0.261
II.
3.4. Hybrid DRL to Simulate Stimuli Perturbation
L.STM-V2 + 3D ResNet
0.3376 0.2169
Task: Encoded String,
MI. + 3D ResNet
0.3331 0.2550
Feedback: Video
To simulate how dynamic time pressure perturbs human logical reasoning process, we conceptualize this process
Transformer + 3D ResNet
0.3306 0.2496
CogReact
0.2999 0.2318
as an evidence accumulation (EA) process in line with the Drif--Diffusiin Model (DDM) (Ratcliff & McKoon, 2008) (Fig. 1(c)). The EA process segments user cognition into sequential steps, facilitating the fine-grained modeling of dynamic time pressure. The boundary threshold and accu- mulation time parameters in the DDM are derived from the predicted responses obtained from the previous SVM model. In order to simulate the dynamic impact of time pressure visual stimuli, we introduce a DRL agent. The visual stimuli are segmented into frames, aligning with the steps in the EA process. For each frame, the specific visual stimuli are applied to the DRI, agent (Fig. (d)), which, akin to how participants' logical reasoning processes may be influenced by each frame of stimuli, modulates the EA process. In particular, for each frame of time pressure stimuli, the DRL. agent adjusts the EA process by introducing positive, neu- tral, or negative bias (action space of the DRL agent). This modulation may result in the evidence accumulator reaching the boundary threshold either earlier or later. The output from this DRL-modulated EA process serves as the final prediction of user response time (Details in Appendix A.7).
LSTM-V1 + +3D ResNet
0.3341 0.2617
III.
LSTM-V2 + 3D ResNet
0.3286 0.2538
Task: Numeric, Feedback: Vidco
MLP 3D ResNet
0.3333 0.2579
Transformer + 3D ResNet
0.3315 0.2526
Decision Tree
0.3617 0.3640
IV.
Linear Regression
0.3595 0.3608
Task: Numeric,
LSTM
0.3059 0.2434
Feedback: Numeric
MLP Random Forest
0.3293 0.2441
0.3650 0.3684
SVM
0.3299 0.3108
Transformer
0.3052 0.2446
CogReact
0.2703 0.2224
Decision Tree
0.3639 0,3639
V.
Lincar Regression
0,3512 0.3469
Task: Encoded String,
LSTM
0.3278 0.2478
Feedback: Numeric
MLP
0.3333 0.2577
Random Forest
0.3600 0.3630
SVM
0.3245 0.3101
Transformer
0.3299 0.2481
works (Roseboom et al., 2019), we have devised a baseline prediction model. Specifically, (Roseboom et al., 2019) con- structed a neural network functionally akin to human visual processing for image classification. The network was then exposed to input videos of natural scenes, causing changes in network activations. The accumulation of salient changes in activation was subsequently used to estimate duration, effectively gauging the perceived passage of time in the video through a Support Vector Machine (SVM).
4. Evaluation
4.1. Human Response Time Simulation Performance
We first demonstrate the effectiveness of our CogReact framework in human response time simulation by compar- ing with baselines using different stimuli encoding schemes.
The model input is composed of three parts: math task stimuli, environmental feedback stimuli (time pressure), and task question ID. The question ID is a numeric value indicating the trial number for participants in the math task. Our exploratory analysis in Appendix A.4 has depicted the relevance of question ID in human response time.
Similarly, our baseline prediction model employs an LSTM neural network to address cognitive tasks (Yang et al., 2019). In particular, we train an LSTM-based math answer agent (Fig. I(a)) to learn and respond to math questions, thereby achieving functional similarity with human cognition in math tasks (Yang et al., 2019). The intermediate output of the LSTM layer serves as input features for the SVM, estab- lishing mappings between agents and humans to estimate user choice and response time (Fig. 1(b)). The rationale of this approach is that distinct math questions may pose
In CogReact, the math task stimuli are represented by one- hot encoded textual strings and the feedback stimuli are represented by videos. However, there are also different ways to extract features from the model input. For example,
4

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Ground Truth
Hybrid DRL
Pure DRL
6-
Rasponse Time
4-
2
5
15 none
20
25
30 Index
10
20
25
30
C
random
8
6
4-
0
5
10
15
20
25
30
5
10
15
20
25
30 Index
rule
static
e
f
go
Overall MAPE
0.6
25-
Episode Reward
HD:1OPO
0.2905
0.4
PD:LOPO
0.3037
20
HDllnd
0.2667
15-
0.2
Hrouroup
PD1idd
-0.3045
0.2713
10
0.0
PD:Group
00.3127
38.30 min
-0.2
4.44 min
HD
0.2999
PD:Al
00.3034
0
2
4
6
10
2
3
4
5
-0.1
0.1
0.4
Step (x100000)
Step (x10000)
Correlation between model and human response time
[Pearson's f]
Pure DRL
Hybrid DRL
Figure 2. Experimental results in the logical reasoning task. a.b,c,d: Examples of user response time in chronological order from participant in each group predicted by the Hybrid/Pure DRL, agent in LOPO-level training, compared with ground truth. e: Pearson correlation between predictions from Hyhrid/ Pure DRI. agent (HD: Hyhrid DRI., PD: Pure DRI.) and human real response time (ground truth) in four training strategies (All: General-level, Group: Group-level, Ind: Individual- LOPO: LOPO-level). Small gray dots, medium dots, and large gray dots represent Pearson correlation prediction results from each participant's 's testing set, each group's testing set (red:none, yellow.static, black:random, bluc:rule) and whole testing set, respectively. The right axis depicts overall average MAPE two agents four training strategies. fg: Training curve Pure DRL (f) and Hybrid DRL (g) models.
we can treat both task stimuli and feedback stimuli as nu- meric values directly or we can put both math task stimuli and feedback stimuli into the whole video as model input, just as humans watch them in the task. Therefore, we eval- uate five types of model input to represent the features of task stimuli and feedback stimuli and use corresponding baseline models, as depicted in Table and Table 4. More details of each model input type and the training/testing hyperparametter//poooss are depicted in Appendix A.9.
(Faust et al., 1999). Therefore, for deep learning models, we use MAPE loss function instead of MSE loss function. Training details are in Appendix A.9.
Results depicted in Table 1 and Table 4, showing that CogReact in both Type II and Type IV has the best response time prediction performance (lowest MAPE) by comparing with other models in both the same and different model input types. Specifically, CogReact in Type IV has a more efficient representation (numeric encoding) during the model infer- ence process and achieves the best performance. We also perform statistical analysis in both Kolmogorov test and Permutation test because they do not necessarily assume the data to be normally distributed. We applied them in all baselines. Results in Table 5 show that CogReact in Type achieves significantly lower MAPE (p < 0.001) than most baselines except for LSTM/Transformer in Type IV (numeric for task/feedback). This due to different task/feedback modality encoding since CogReact uses Type II (Task: String, Feedback: Video). When we apply CogRe- act to numeric encoding of Type IV as well, results in Table 6 show that ours can also achieve significantly lower MAPE (p < 0.001) than
When encoding both task stimuli and feedback stimuli into whole video, we hGRU (Goetschalekx et al., 2024), LSTM with pre-trained vision models (Jaffe et al., 2023), and MLP with pre-trained 3D ResNet (Bourgin et al., 2019) as the bascline. These models are adapted into our problem corresponding to the recent state-of-the-ard (SOTA) mod- els in human decision making (Bourgin et al., 2019) and response time prediction (Goetschalekx al., 2024; Jaffe et al., 2023). Similarly, for other types of model input, we also use related SOTA models in the specific input type domain. More details of baseline models and adaptation into our problems are depicted in Appendix A.9.
in Type IV.
We use Mean Average Percentage Error (MAPE) instead of Mean Squared Error (MSE) to evaluate the response time difference between real humans and simulations because human response times exhibit high individual differences
MAPE Variance. Despite the superiority, we observed high variance MAPE, largely driven by individual differences across users and variability in the math trials they completed.
5

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
These factors interact, amplifying prediction error variance. To isolate their effects, held one factor constant while averaging over the other: (a) Fixing Users: Averaging pre- dicted and actual response times across all trials per user, then computing MAPE, yielded a mean of 0. 1388 (STD: 0.0641, 95% CI: [0.0316, 0.2378J). (b) Fixing Trials: Aver- aging across users per trial and computing MAPE resulted in mean of 0.1403 (STD: 0.0865, 95% CI: [0.0292, 0.2855)). Both approaches significantly reduced variance compared to the overall results (Mean: 0.2703, STD: 0.2224, 95% CI: [0.0093, 0.7631)), suggesting that the high variability stems from the interaction of user and trial differences.
combinations of three numbers in math questions and got a dataset containing 20,414 samples, which is split into train- ing set (80%) and testing set (20%). Given that the first two numbers of math questions are both two-digit, the arithmetic reasoning result is chosen from 0 to 8. Consequently, the ground truth encompasses 9 classes.
We experimented with different numbers of output neu- rons (32. 64, 128, 256) from the LSTM layer. After 100 epochs of training, the logical reasoning agent with 256 neu- rons achieved remarkable results, attaining a training loss of 0.0001 and 100% accuracy (Fig.9(b)) The confusion matrix (Fig. (()) for the testing set also demonstrates that this neu- ron configuration yields over 99% accuracy for all classes, resulting in an overall test accuracy of 99.93%. Moreover, even for other neuron numbers, the test accuracy is also high enough (more than 95%). These outcomes affirm that the LSTM- logical reasoning agent adeptly solves math arithmetic problems in the majority of cases. This aligns with existing work (Mickey & McClelland, 2014; Zaremba & Sutskever, 2014), which demonstrated the capacity of neural networks to learn mathematical equivalence. The success of the logical reasoning agent in solving arithmetic problems lays a foundation and explains its capability for extracting representative features from math questions to construct cognition models.
The performance improvement stems from the entire frame- work, including useful features from the math logical rea- soning agent and the integration of the drift-diffusion model in the DRL agent to simulate feedback stimuli in fine- grained manner. In what follows, we run ablation studies to show the unique importance of each component.
4.2. Importance of Task Encoding with Math Agent
To demonstrate that the math answer agent indeed captures representative features from math questions, in the first ablation study, we compare SVM models (second step in our framework) with two additional settings where the SVM models do not take features captured from the math answer agent as input. Instead, they take raw three-digit numbers from the math questions or one-hot encoded vectors (same as the input of the math logical reasoning agent in Appendix A.5) of raw numbers as input, along with the question ID. The SVM performance in the three settings is depicted in Table 3. Notably, SVM models with features from the math answer agent exhibit significantly higher accuracy (0.9613) and Fi-score (0.8996) for user choice prediction and lower MAPE (0.3652) for response time estimation than other settings, underscoring the effectiveness of the math answer agent in capturing representative math question features and the feasibility of predicting user baseline performance in ideal conditions without environmental stimuli using SVM
4.4. Importance of Integrating DDM into DRL Agents
The third ablation examines the importance of DDM in DRL agents, to simulate the perturbation of external stimuli on hu- man response time in a fine-grained manner. We introduce a baseline DRL model called the pure DRL agent, which does not incorporate the DDM. Specifically, this pure DRL agent does not segment time pressure visual stimuli into frames. Instead, for each trial from the dataset, it directly takes the entire time pressure visual stimuli as input and outputs one action representing the overall change in response time due to time pressure. The final estimation of regulated response time is the sum of this action and the basic response time estimated by the SVR models (details in Appendix A.8, Fig. 8). Moreover, we also directly remove the whole hybrid DRL agent and only use SVR models to predict response time as another ablation baseline.
4.3. Why Does the Logical Reasoning Agent Work?
The second ablation study explores why the math logical rea- soning agent can extract useful features from math questions (first step of our framework). We answer this question by exploring its math task solving performance under different numbers of output neurons from LSTM layer.
We employ both MAPE and Pearson correlation to compare the performance of the hybrid DRL and pure DRL agents. Four model training strategies are used below: (a). General- level involves splitting the entire dataset into training (80%) and testing (20%) sets for overall model evaluation. (b). Group-level trains and tests specific model using data from each group, revealing performance across different time pressure stimuli. (c). Individual-level trains and tests model using data from a specific participant, assessing per- sonalized model feasibility incorporating subject-specific
Note that the math answer agent aims to solve math tasks correctly instead of predicting human choice. In short, given one math question as input, it could directly output the arith- metic reasoning answer. Therefore, its training and testing have no correlation with real-user responses. Hence, we prepare a separate dataset that is independent of the human dataset to train the agent. Finally, we traverse all possible
6

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
a
b
C
0.3
0.3
0.2-
0.2
0.2
0.2
0.1
0.1
0.1
0.1
0.0
0.0-
-0.1
-0.1
0.0
10
20
30
0
10
20
30
10
20
30
20
40
e
none
f
random
g
rule
static
0.4
0.15-
0.8-
0.020-
0.2
0.10
0.015
0.0
0.05
0.6-
0.010
-0.2
0.00
0.4-
0.005
-0.05
-0.4
0.000
0.2
none static random rule Response Time
none static randommulee Trajectory Mean
nonessation randommulle Action Variance
none static random rule Trajectory Slope
Figure 3. a.b,c,d: Time pressure effect trajectories of four groups, respectively. e: Box plot of relative response time change across four groups in the whole dataset. fgg.:: Box plot of mean value of time pressure effect trajectories (f), standard deviation of action trajectories (g). slope of time pressure effect trajectories (h) of four groups predicted testing dataset Hybrid DRL agent. The slope of one trajectory is calculated from the start point to the end point of the trajectory.
behavioral differences. Shuffling is applied to training and testing sets to prevent overfitting artifacts. (d). As shuf- fled testing disrupts the temporal trend user response time across different math trials, we incorporate Leave- One-Participant--Ou (LOPO) training as an additional training strategy. This strategy selects all data from participant as the testing set and uses data from other partic- ipants in the same group as the training set. By traversing every participant's data as the testing set, we ensure com- prehensive assessment of model performance in capturing temporal trends of response time.
DRL agent in modeling the effect of time pressure stimuli.
To compare which agent design better captures the trend of response time change in user overall tasks, we visualize the prediction results and real user response time for the testing set from one participant of each group LOPO-level in chronological order. Fig. 2(a.b.c,d) clearly demonstrate that the hybrid DRL agent more accurately captures the trend of user response time compared to the pure DRL agent.
4.5. Training Efficiency
The training curves for both hybrid and pure agents presented in Fig. 2(f.g). The pure DRL and hybrid DRL agents converge at approximately 800,000 steps and 20,000 steps, respectively. is important to note that the meaning of one step differs between the two agents. For the hybrid DRL agent, one step represents one frame of time pressure stimuli during trial, whereas represents the entire trial. Consequently. direct comparison of steps is not meaningful. Instead, we compare the training time required for both agents to achieve convergence on the same hardware (GeForce RTX 2080 Ti) and the same dataset. The results in Fig. 2(f.g) indicate that the hybrid DRL agent converges in approximately one-tenth of the time compared to the pure DRL agent (4.42 minutes vs. 38.30 minutes). This outcome underscores the advantage of incorporating an explicit cognitive model (i.e., the DDM) in
Fig. 10 illustrates the average MAPE of the testing set for each individual user (a,b,c,d) and each group (e.f.g.h). Both the hybrid DRL and pure DRL agents show an improve- ment in response time estimation compared to SVM results. However, the hybrid DRL agent consistently achieves lower MAPE compared to the pure DRL agent in most cases, indi- cating the superiority of the hybrid DRL agent in response time estimation. The overall average MAPE for the entire testing set by both agents is depicted the right y-axis of Fig. 2(e), further supporting this conclusion. Fig. 2(e) also reveals that the hybrid DRL agent exhibits larger Pear- son correlation in individual testing sets (small dots), group testing sets (medium dots), and the whole testing set (large dots) compared to the pure DRL agent in most cases, across all four training strategies. Both MAPE and Pearson corre- lation demonstrate the superior performance of the hybrid
step for the pure DRL agent

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
the hybrid DRL agent to improve training efficiency.
Hybrid DRL
0.357
Hybrid DRL
Pure CRL
1.616
Pure DRL
2,682
4.6. Interpretability
Unencaded Hybrid (R)
Unencoded Hybrid DRI
0.360
0.956
An essential advantage of the regnition-inspieee hybrid DRL agent is its interpretability, compared to deep learning and the pure DRL agent, which directly output estimated re- sponse time changes for each trial, obscuring the internal mechanism regarding how time pressure stimuli modulate the logical reasoning process. In contrast, the hybrid DRL agent can generate a trajectory of the time pressure effect on response time corresponding to user logical reasoning process. Therefore, visualizing the trajectories of the hybrid DRL agent enables extracting new insights of how time pressure stimuli affect the human logical reasoning process.
Encoded SVM
0,558
Encoded SVM
174
SVM
0.385
LSTM
1.274
LSTM
0 461
Transformer
1.207
Transformer
0.535
MLP
1.204
MIPP
0.476
SVM
1.167
Random Forest
Random Forest
0.512
1.224
Dedsioo Tree-
0.536
Decision Tree-
1.222
Lincar Regression
0 632
Lincar Regression
1.218
2
3
Decision Making Task
Learning Task
We explore this benefit in Fig. 3(a.b.....e.fgghh.. Here, the action trajectory represents the trajectory of actions taken by the hybrid DRL agent during one episode, with each episode corresponding to one math trial of users. The time pressure effect trajectory is the accumulated actions multiplied by 6p. 6p represents one unit of evidence per step, transforming the normalized action value into the evidence accumulation process. We visualize the time pressure effect trajectories across the four groups in Fig. 3(,,b. Each curve represents one trajectory predicted by the hybrid DRL agent during one trial.
Figure 4. Results in decision making(left) and learning(right) task.
agent designs across three key aspects (response time esti- mation performance, training efficiency, and interpretability) highlights the advantages of the hybrid DRL approach in capturing the nuanced dynamics of time pressure stimuli on user response time in the logical reasoning process.
We observe that the time pressure effect trajectories are more concentrated in the random and rule groups but divergent in the none and static groups (Fig. 3(a,b,c,d)) This suggests that participants in the random and rule groups, especially the random group, are better regulated by the corresponding type of time pressure stimuli, resulting in similar trends in all time pressure effect trajectories in this group. Quantitatively, the random group has the lowest standard deviation (STD) of action trajectories (Fig. 3g)) and the highest average value and slope for the time pressure effect trajectories (Fig. 3(f.h)). These findings in the simulation results indicate that the random group experiences the most effective regulation of user cogmition performance.
5. Generalization
We further evaluate the generalization ability of our model in two additional public datasets: CPC182 for decision making and PeerEdu for learning (Xu et al., 2025). Table 2 depicts the datasets' properties and summarizes diverse tasks and feedback modalities for modeling. Dataset details and our framework adaptation are depicted in Appendix A.11.
Baselines: As both datasets lack video input, we adopt the same baselines as Type IV/V in Table 1, aligning with base- lines commonly used in prior work (Bourgin et al., 2019) that benchmarks the CPC18 dataset. For fair comparison, in the PeerEdu dataset, we use the same embeddings from OpenAl's
e This observation aligns with the expectation that users may quickly adapt to none static time pressure, ceasing to be regulated by them after a few trials. However, users may not anticipate the time pressure in the random group, leading to a more prolonged regulation effect. This result in the hybrid DRI. simulation is also consistent with real human results our initial exploratory findings (Appendix A.4, Fig. 6(e)), where participants the random group demonstrated significantly larger reduction in response time, compared with other groups. These experiments affirm the hybrid DRL agent's 's capability to explain and support observations in the real humans' response time performance.
-mmeeddngg3--mme model to encode textual data for model input of baselines.
Consistent with prior experiments, we use MAPE to evaluate modeling error. As shown in Fig. 4, our hybrid DRL model achieves the lowest MAPE across both datasets compared to all baselines, highlighting its effec- tiveness in generalizing across diverse tasks and feedback modalities. Moreover, the statistical analyses (same with previous evaluations) also show significant improvement of our model over all baseline models in Table 7 and Table 8.
Performance:
Ablation Study: We further explore the role of each com- ponent using variants of our framework, as shown in the
The comparative analysis between the hybrid and pure DRL
Phups://cpc-18 18.11.....m com/data/
8

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table 2. Task/feedback information and dataset properties.
Task Information
Simulation Modality
Dataset Information
Task Type
Response Type
User Action
Cognitive Response
Task
Feedback Stage 1
Source
Size
User
Math Reasoning
Active
Hinary
Response Time
String
Visual Math Agent
Ours
21,157
50
Decision Making
Active
Binary Response Time
Numeric Numerie
Risk Agent
Public
30,489
240
Learning
Passive
Contiauous
Curiosity
Textual
Textual
LLM Agent
Public
12,804
300
textured bar results in Fig. 4. The pure DRL variant re- moves the DDM component from the DRL loop, where the action space directly represents response time/curiosity changes without segmenting the evidence accumulation pro- cess (similar to Section 4.4). The unencoded hybrid DRL variant removes the first step of the framework (risk/I.LM agent for embedding extraction), while the encoded SVM variant only includes the first two steps (risk/LLM agent+ + SVM) without the DDM or the DRL loop.
then derived from these predicted parameters. Additionally, we introduced a variant of MLP (MLPv2), which shares the same neural network architecture as our Hybrid DRL model, to assess whether the observed performance gains could be attributed solely to the combination of DL and DDM, This experiment was conducted across three datasets, with the same statistical analyses performed in previous evalua- tions. The results (Fig. 11) indicate that while DL+DDM integration sometimes outperforms standalone DL models, its performance remains significantly inferior to that of the Hybrid DRL model (p < 0.001). These findings highlight that the improvement observed in the hybrid DRL model cannot be attributed solely to DL+DDM integration; rather, the RL component plays a critical and complementary role in achieving superior performance.
Fig. 4 shows that all variants perform worse than the full framework (hybrid DRL), underscoring the unique and crit- ical roles of each component. Moreover, the statistical anal- yses (same with previous evaluations) also show significant improvement of our model over most ablation models in Table 7 and Table 8. Removing the first step (unencoded hybrid DRL) causes a slight performance drop, whereas removing the DDM (pure DRL) leads a significant error increase, performing worse than all baselines. This high- lights the dominant role of incorporating DDM into the DRL loop in modeling feedback modulation effects on cognitive responses. Additionally, removing both DDM and DRL (encoded SVM) also degrades performance, emphasizing the importance of the hybrid DRL loop.
6. Discussion, Limitations, and Conclusion
We propose computational framework for simulating en- vironmental stimuli perturbations on human cognitive pro- cesses, including logical reasoning, decision-making and learning, across diverse task and feedback modalities. By integrating the drift-diffusion model from cognitive science with deep reinforcement learning, our framework achieves higher simulation accuracy, improved training efficiency, and enhanced interpretability, capturing the granular effects of dynamic stimuli on cognitive processes. The successful adaptation of our framework for continuous behaviors (e.g.. curiosity in learning tasks) lays a foundation for extending it to handle continuous user inputs beyond binary responses. This advancement has the potential to offer new insights into machine learning and neuroscience by fostering com- putational models that better understand human cognition.
Interestingly, the encoded SVM variant (risk/LLM agent + SVM) performs worse than the straightforward SVM base- line (without risk/LLM agent), suggesting potential draw- backs of using risk/LLM embeddings for SVM. However, removing the risk/I.L.M agent (unencoded hybrid DRL) re- sults in worse performance compared to the full hybrid DRL model. This discrepancy suggests that, while the extracted features from the risk/LLM agent may not directly bene- fit the SVM, they still improve the hybrid DRL. model by enhancing and expanding the observation space during the evidence accumulation process.
One limitation is our focus on response time simulation. Future extensions could incorporate additional cognitive measures. A potential path involves training task-solving agents, like the logical reasoning agent in math tasks, to em- ulate human task performance. Building on prior research that highlights the effectiveness of machine learning models in more than 20 cognitive tasks (Yang et al., 2019), our framework could be extended to other domains by linking extracted features from task-solving agents to real user re- sponses using models like SVM. Finally, by dynamically adapting the action and observation spaces to task- speciffc feedback, the DRL agent could simulate the nuanced effects of stimuli across diverse cognitive scenarios.
DDM in Deep Learning Models. Our previous findings demonstrated the advantage of integrating DDM into DRL. To investigate whether this performance gain arises primar- ily from the deep learning (DL) architecture within DRL or from the reinforcement learning (RL) component itself, we conducted an additional study that directly incorporated DDM into DL models without RL. Specifically, we adapted LSTM, MLP, and Transformer architectures--deniie those used in Baseline Model Type IV-with unchanged hy- perparameters, modifying them to predict DDM parameters instead of response times. The final response times were
to

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Acknowledgments
feature representations. arXiv preprint arXiv:1711.04855 2017.
This work is supported by National Science Foundation CNS-2403124 CNS-2312715, CNS-2128588 and the Uni- versity of California San Diego Center for Wireless Com-
Battleday, R. M., Peterson, C., and Griffiths, L. Captur- ing human categorization of natural images by combining deep networks and cognitive models. Nature communica- tions, 1 11(1):1-14, 2020.
munications.
Impact Statement
Battleday, R. M., Peterson, J. C.,, and Griffiths, T. L. From convolutional neural networks to models of higher-level cognition (and back again). Annals of the New York Academy of Sciences, 1505(1):55-78, 2021.
Our framework bridges cogmitive science and reinforcement learning by modeling human-like becision-making in dy- namic environments. It extends prior work by capturing how environmental stimuli shape cognitive responses, a key challenge in human in the-loop AI. This approach enables neuroscientists to simulate cognitive adaptation through reinforcement-ban models and supports biologically in- spired AI designs and intervention strategies. It also pro- vides cogmitive scientists with tools to design behavioral experiments, while helping AI researchers align algorithms with human reasoning. By improving transparency, trust, and usability in adaptive systems, the framework advances human-centered Al and serves practical foundation for interdisciplinary research across AI, cognitive science, and neuroscience. Future research inspired by this framework could refine its scope to encompass additional cognitive measures, expand its application to diverse task-solving agents, and deepen its alignment with real-world user data. The framework's is adaptability to continuous behaviors and dynamic feedback underscores its potential to drive innova- tion across cognitive modeling, neuroscience, and machine learning. We anticipate that this research will encourage interdisciplinary exploration into cogmitive and behavioral modeling, with minimal societal risks. We adhered to ap- propriate licenses in using public datasets. Our experiment for human data collection in logical reasoning tasks approved by the Institutional Review Board (IRB) at our local institution. The data we model is strictly limited to anonymous behavioral responses under experimental tasks, with no links to real-world identities, nor demographics. As such, the model itself does not encode or have access to demographic-sensinne
Binz, M. and Schulz, E. Modeling human exploration through resource-rational reinforcement learning, Ad- vances in neural information processing systems, 35: 31755-31768, 2022.
Binz, M. and Schulz, E. Turning large language models into cognitive models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openreviens www. net
C4BKKpp11..
Bourgin, D. D., Peterson, J. C., Reichman, D., Russell, S. J., and Griffiths, T. L. Cognitive model priors for predicting human decisions. In International conference on machine learning, pp. 5133-5141. PMLR, 2019.
Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang. J., and Zaremba, W. Openai gym. arXiv preprint arXiv: 010.6005440,. 2016.
Cheng, S.-Y. Evaluation of effect on cognition response to time pressure by using eeg. In International conference on applied human factors and ergonomics, pp. 45-52, Springer, 2017.
Cho, J., Yoon, J., and Ahn, S. Spatially-aware transform- ers for embodied agents. In The Twelfth International Conference on Learning Representations, 2023.
Chollet, F. al. Keras. https://kerag ic, 2015.
Cichy, R. M. and Kaiser, D. Deep neural networks as scien- tific models. Trends in cognitive sciences, 2344)3305-317, 2019.
features, reducing the risk of biased outputs, and also reducing the risk of potential usage for justifying employment, acceptance to college, and so on for targeted people.
Costa, J., Guimbretière, F., Jung, M. F., and Choudhury, T. Boostmeup: Improving cognitive performance in the mo- ment by unobtrusively regulating emotions with smart- watch. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(2):1-23, 2019,
References
Alexander, C., Paul, M., Michael, M., et al. The effects of practice on the cognitive test performance of neuro- logically normal individuals assessed at brief test-retest intervals. Journal of the International Neuropsychologi- cal Society, 93))419 428, 2003.
Daitch, A. L., Foster, B. L.,, Schrouff, J., Rangarajan, V., Kaşikçi, I., Gattas, S., and Parvizi, J. Mapping human temporal and parictal neuronal population activity and functional coupling during mathematical cognition. Pro- ceedings of the National Academy of Sciences, 113(46): E7277-E7286, 2016.
Battleday, R. M., Peterson, J. C., and Griffiths, T. L. Mod- eling human categorization of natural images using deep
10

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
De Boeck, P. and Jeon, M. An overview of models for response times and processes in cognitive tests. Frontiers in psychology, 10: 10,1 2019.
Hartford, S., Wright, J. R., and Leyton-Brown, K. Deep learning for predicting human strategic behavior. Ad- vances in neural information processing systems, 29, 2016.
Do, S., Chang, M., and Lee, B. A simulation model of intermittently controlled point-and- behaviour. In Proceedings of the 2021 CHI Conference on Human Fac-
Heitz, R. P. The speed-accuracy tradeoff: history, physi- ology, methodology, and behavior. Frontiers in neuro- science, 8:86875, 2014.
tors in Computing Systems, pp. 1-17, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv 2020.
Huys, Q J., Maia, T. V., and Frank, M. J. Computational psychiatry as a bridge from neuroscience to clinical ap- plications. Nature neuroscience, 19(3):404-413, 2016.
preprint arXiv:2010.1
Jaffe, P. I., Poldrack, R. A., Schafer, R. J., and Bissett, P. G. Modelling human behaviour in cognitive tasks with latent dynamical systems. Nature Human Behaviour, pp. 1-15, 2023.
Edland, A. and Svenson, O. Judgment and decision making under time pressure. In Time pressure and stress in hu- man judgment and decision making, pp. 27-40. Springer, 1993.
Judd, N. and Klingberg, T. Training spatial cognition en- hances mathematical learning in a randomized study of 17, 17,000 children. Nature Human Behaviour, 5(11):1548- 1554, 2021.
Erev, I., Ert, E., Plonsky, O., Cohen, D., and Cohen, O. From anomalies forecasts: Toward descriptive model of de- cisions under risk, under ambiguity, and from experience. Psychological review, 124(4):369, 2017.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6940 2014.
Faust, M. E., Balota, D. A., Spieler, D. H., and Ferraro, F. R. Individual differences in information processiooeeegeeeee rate and amount: implications for group differences in response latency. Psychological bulletin, 125(6):777, 1999.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances neural information processing systems, 25, 2012.
Fudenberg, D., Newey, W., Strack, P., and Strzalecki, T. Testing the drift-diffusion model. Proceedings of the National Academy of Sciences, 117/52)33141--3314, 2020.
Kumbhar, O., Sizikova, E., Majaj, N., and Pelli, D. G. Any- time prediction a model of human reaction time. arXiv preprint arXiv:2011.12855 2020.
Goetschalckx, L., Govindarajan, L. N., Karkada Ashok, A., Ahuja, A., Sheinberg, D., and Serre, T. Computing human-like reaction time metric from stable recurrent vi- sion models. Advances in Neural Information Processing Systems, 36, 2024.
Lake, M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Legg, A. M. and Locker Jr, L. Math performance and its relationship to math anxiety and metacognition. North American Journal of Psychology. 11(3), 2009.
Golan, T., Raju, P. C., and Kriegeskorte, Controversial stimuli: Pitting neural networks against each other as models of human cognition. Proceedings of the National 2020.
Academy Sciences, 117(47):2330-2293)
Lin, C.-T., Chen, S.-A., Chiu, T.-T., Lin, H.-Z., and Ko, L.-W. Spatial and temporal eeg dynamics of dual-task driving performance. Journal of neuroengineering and rehabilitation, 8(1):1-13, 2011. 2011.
Han, J. and Moraga, C. The influence of the sigmoid func- tion parameters on the speed of backpropagation learning. In Proceedings of the International Workshop on Artifi- cial Neural Networks: From Natural to Artificial Neural Computation, IWANN '96, pp. 195-201, Berlin, Heidel- berg, 1995. Springer- Verlag. ISBN 3540594973.
Linsley, D., Kim, J., Veerabadran, V., Windolf, C., and Serre, T. Learning long-range spatial dependencies with horizon- tal gated recurrent units. Advances in neural information processing systems, 31, 2018.
Hanich, L. B., Jordan, N. C., Kaplan, D., and Dick, J. Perfor- mance across different areas of mathematical cognition in children with learning difficulties. Journal of educational psychology, 93());615, 2001.
Ma, W. J. and Peters, B. A neural network walks into a lab: towards using deep nets as models for human behavior. arXiv preprint arXiv:2005.02 2020.

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Mehrer, J., Spoerer, C. J., Kriegeskorte, N., and Kietzmann, T. C. Individual differences among deep neural network models. Nature communications, 11(1):1-12, 2020.
Plonsky, O., Erev, I., Hazan, T., and Tennenholtz, M. Psy- chological forest: Predicting human behavior. In Thirty- first AAAI conference on artificial intelligence, 2017.
Mickey, K. W. and McClelland, J. L. A neural network model of learning mathematical equivalence. In Proceed- ings of the annual meeting of the cognitive science society, volume 36, 2014.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines33 Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http:
/ //jmlr.org wttpoorppppppe//////0/0/ -064 html.
Moore, D. A. and Tenney, E. R. Time pressure, performance, and productivity. In Looking back, moving forward: A review of group and team-based research, volume 15, pp- 305-326. Emerald Group Publishing Limited, 2012.
Ratcliff, R. and McKoon, G. The diffusion decision model: theory and data for two-choice decision tasks. Neural computation, 20(4):873-922, 2008.
Noti, G., Levi, E., Kolumbus, Y., and Daniely, A. Behavior- based machine-learning A hybrid approach for predicting human decision making. arXiv preprint arXiv: 1.1.v: 2016.
Ritter, S., Barrett, D. G., Santoro, A., and Botvinick, M. M. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, pp. 1940-2949 PMLR, 2017.
Pajares, and Miller, M. D. Role self-efficacy and self- concept beliefs in mathematical problem solving: A path analysis. Journal of educational psychology, 86(2):193, 1994.
Rodríguez, P., Bautista, M. A., Gonzalez, J., and Escalera, S. Beyond one-hot encoding: Lower dimensional target embedding. Image and Vision Computing, 75:21-31, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z. Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang. E., Vito,, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: an imperative style, piih-pperorrance deep learning librury. Curran Associates Inc., Red Hook, NY, USA, 2019.
Roseboom, W., Fountas, Z., Nikiforou, K., Bhowmik, D., Shanahan, M., and Seth, A. K. Activity in perceptual clas- sification networks as a basis for human subjective time perception. Nature communications, 10(1):1-9, 2019.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv, abs/1707 bos/1707.06347, .06347, 2017.
Pedersen, M. L., Frank, M. J., and Biele, G. The drift diffu- sion model as the choice rule in reinforcement learning. Psychonomic bulletin review,
Sensoy, M., Kaplan, L., and Kandemir, M. Evidential deep learning to quantify classification uncertainty. Advances neural information processing systems, 31, 2018.
2017.
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cour- napeau, D., Brucher, M., Perrot, M., and Duchesnay, E. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556,, 2014.
Singh, P., Peterson, C., Battleday, R. M., and Grif- fiths, T. L. End-to-end deep prototype and exemplar models for predicting human behavior. arXiv preprint arXiv:2007.08723 2020.
Peterson, J. C., Abbott, T., and Griffiths, T. L. Evaluating (and improving) the correspondence between deep neural networks and human representations. Cognitive science, 42(8))264,,2669, 2018.
Slobounov, S., Fukada, K., Simon, R., Rearick, M., and Ray, W. Neurophysiological and behavioral indices of time pressure effects visuomotor task performance. Cognitive Brain Research, 93):2877-988, 2000.
Peterson, J. C., Bourgin, D. D., Agrawal, M., Reichman, D., and Griffiths, L. Using large-scale experiments and machine learning discover theories of human decision- making. Science, 372(6547); 221--222-11 2021.
Smith, Diffusion theory of decision making in contin- uous report. Psychological Review, 123(4):425, 2016.
Peysakhovich, A. and Naecker, J. Using methods from machine learning to evaluate behavioral models of choice under risk and ambiguity. Journal of Economic Behavior & Organization, 133:373-38.. 2017.
Song, H. F., Yang, G. R., and Wang, X.J. Training excitatory-iniibitry recurrent neural networks for cog- nitive tasks: a simple and flexible framework. PLoS computational biology, 12(2):e1004792, 2016.
12

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Song, H. F., Yang, G. R., and Wang. X.-J Reward-based training of recurrent neural networks for cognitive and value-based tasks. Elife, 6:e21492, 2017.
trained perform many cognitive tasks. Nature neuro- science, 22(2):297-306, 2019.
Zaremba, W. and Sutskever, I. Learning to exccute. arXiv preprint arXiv: 140.4664 2014.
Spoerer, C. J., Kietzmann, T. C., Mehrer, J., Charest, I., and Kriegeskorte, N. Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS computational biology, 1008211,, 2020.
Zur, H. B. and Breznitz, S. The effect of time pressure on risky choice behavior. Acta Psychologica, 4702):99-104, 1981.
Steyvers, M., Hawkins, G. F.,, Karayanidis, F., and Brown, S. D. A large-scale analysis of task switching practice effects across the lifespan. Proceedings of the National Academy of Sciences, L16366)17733--177400 2019.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion, 2015. URL htw.s: /ttp///axvvvv//aa///a//e.eeaaava wrw. haagg//bs/// /abb/111... 00567.
Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., and Paluri, M. A closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEEE con- ference Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Viejo, G., Khamassi, M., Brovelli, A., and Girard, B. Model- ing choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Frontiers in behav- ioral neuroxcience, 9:225, 2015.
Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, Improving text embeddings with large language models. arXiv preprint (tt::2401.000666 2023.
Wenliang, L. K. and Seitz, A. R. Deep neural networks for modeling visual perceptual learning. Journal of Neuro- science, 38(27);6028 6044, 2018.
Whittaker, S., Kalnikaite, V., Hollis, V., and Guydish, A. 'don't waste my time' use of time information improves focus. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 1729-1738, 2016.
Xu, S., Hu, D., Wang, R., and Zhang, X. Peeredu: Bootstrap- ping online learning behaviors via asynchronous area of interest sharing from peer gaze. In Proceedings of the 2025 CHIl Conference on Human Factors Computing Systems, pp. 1-14, 2025.
Yang, G. R., Joglekar, M. R., Song, H. F., Newsome, W. T., and Wang, X.-J. Task representations neural networks
13

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
A. Appendix
from all participants beforehand. More dataset details are in Appendix A.3.
A.1. Ethics Statement
Our experiment for human data collection in logical reason- ing tasks was approved by the Institutional Review Board (IRB) our local institution. We do not anticipate any risk during data collection and we have obtained informed consent from all participants beforehand. Our work may provide insights to integrate classical cognitive theories into machine learning models. In neuroscience, effective com- putational models for response time could pave the way for understanding many key cognitive behaviors and neurobi- ological disorders disorders(Goottscallte Goetschalckk et al., 2024; Huys et al., 2016). We do not anticipate the negative impact on society in this context.
A.3. Dataset Collection
1.3.1. PARTICIPANTS
We recruited 50 participants in total (age 21.44 + 3.22 y (mean + SD); 27 female) from our local institution to finish the math modular task (details in Fig. 5(a)). Participants were recruited by email groups at our local institution and came from a variety of majors including engineering, com- puter science, biology, and so on. Six participants took part in the preliminary study to explore potential configura- tions of study design, whose results were removed. Other 44 participants were randomly and uniformly divided into 4 groups in order to fully capture the potential effects of time pressure in cognition performance, as described before. Two participants withdrew from the study and three did not finish the study completely. We also removed another three participants' results whose study duration was longer than 3 hours. This was much longer than normal study duration of other participants (within hour) and suggested that partici- pants neither focused on the task nor took this experiment seriously. Finally, we had 36 participants: None Group (10), Static Group (9), Random Group (7), Rule Group (10). This study has been approved by the Institutional Review Board (IRB) at our local institution. We have obtained informed consent from all participants before study.
A.2. Task and Dataset
As depicted in Section. 1, we used math arithmetic task with time pressure visual stimuli as our model explo- ration context. The illustration of the task and stimuli is depicted in Appendix Fig. 5. In short, each math trial was composed of two two-digit numbers Num Numa and one one-digit number N Numn, formatted as: Num, = N J'mm(( mod N umg). To solve this question, participants first subtracted Nu Numz from Num, and judged whether the subtraction result could be divisible by Num,, If it was divisible, they selected "True" button. Otherwise, they selected "False" button. When the time pressure stimuli hap- pened, a progress bar was shown on top of the math question, which added one unit for each second and reset and added again when it accumulated five units. The human response time was then calculated from the time when the math task appeared per trial, to the time when the participants clicked one button to answer it.
A.3.2. PROCEDURE
All participants took part in a two-day study. For each day, they were asked to first finish an exercise session containing 20 math trials and then finish a formal session containing 300 math trials. The exercise session aimed to familiar- ize the users with tasks and measure users' bascline per- formance (without time pressure). In the formal session, different time pressure mechanisms were provided for differ- ent groups as mentioned above. Additionally, participants were requested to rate their current attention/anxiety status on 7-point Likert scale every 30 trials. There was also a 5-min rest between exercise session and formal session. It took each participant an average of one hour for the study per day. In the study, participants were told to always take accuracy as the priority and then try their best to answer questions soon as possible. The compensation rule for each participant (ranging from $10 to $100) also prioritized average accuracy over response time in order to encourage participants to follow our instructions. We finally obtained a large data set of 21,157 logical responses after removing invalid user response.
We collected an extensive dataset encompassing 21,157 valid responses from 44 participants engaged in the math task (see Fig. 6(a)). To enhance datasct diversity and eval- uate our model under dynamic environmental stress, par- ticipants were randomly and uniformly distributed across four distinct groups: None Group: Participants experienced no time pressure for any trial. Static Group: Time pressure was consistently applied for each trial. Random Group: There was a 50% probability of time pressure being ap- plied for each trial. Rule Group: Time pressure was adap- tively applied based on users' past performance using a rule-based strategy (more details of such strategy are in Ap- pendix A.3.4). Each participant engaged in two-day study, featuring one exercise session (20 trials) and one formal session (300 trials) per day, when we collected participants' choices and response time per trial. This collection has been approved by the Institutional Review Board (IRB) at our local institution. We do not anticipate any risk dur- ing data collection and we have obtained informed consent
14

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
A.3.3. MATH QUESTION GENERATION AND DISTRIBUTION
threshold, it delivers time pressure feedback. In addition, if the strategy delivers time pressure for too many times (exceeding PC threshold), the time pressure feedback is still not delivered to users. Therefore, rule-based strategy is a relatively conservative strategy which cares more about avoiding introducing additional distraction to users.
All math questions are composed of two two-digit numbers (Num1, Numm)) and one one-digit number (N (uum)). We denote the three numbers as Nu Num1 = ab, Num, = cd, Num3= = e, respectively. So each math question could be denoted as ab= = od ( mood e). where a E [1, 10). b E [2, 10). C [1, 10). d E [1, e E [3, 10). All math questions are randomly generated for each trial. We have traversed all possible combinations of math digits in the math question format, which are distributed uniformly in the whole math space for the four groups. Participants' accuracy and the provided time pressure feedback are also distributed uni- formly.
A.4. Dataset Exploration
To investigate the impact of different time pressure stim- uli on cognition performance, we conducted an initial ex- ploratory analysis on the dataset. To mitigate the influence of chance factors, we divided the 300 trials of the formal session into five blocks of equal size and calculated the block-wise averages for accuracy, response time, attention, and anxiety scores. Recognizing the inherent variability in users' baseline performance, we aimed to elucidate the im- pact of time pressure across different groups by comparing the relative change in user performance and status across the four groups. Specifically. let R, denote the average re- sult of Block,, where R₁ (Block)) represents the baseline performance. The final relative result for Block; (i > 1) is R4 R1)/ R4 for accuracy and response time change and R. R4 for attention and anxiety change. This adjustment accounts for the fact that attention/anxiety scores linearly reflect user status, while response time/accuracy changes need to be normalized against participants' individual base- line performances. The obtained results were then analyzed using repeatedd-mezsurss ferences, Bonferroni-correcte paired post hoc t-tests were employed for pairwise comparisons between the groups, enabling a thorough exploration of the impact of different time pressure stimuli on cognition performance and user status.
A.3.4. GROUPS
Here we describe details of four groups in dataset collection. None Group:
Participants experienced no time pressure for any trial. Static Group: Time pressure was consistently applied for each trial. Random Group: There was 50% probability of time pressure being applied for each trial. Rule Group: Time pressure was adaptively applied based on users' past performance using a rule-based strategy. More details about such strategy are depicted below.
Rule-based strategy is designed to provide adaptive time pressure feedback for each trial according to participants' past performance in the Rule group. There is a response buffer to update and save user response of most recent 20 tri- als. For each new user response, is updated in the response buffer. Then we calculate five metrics (mean response time, delta response time, mean accuracy, push counter, and toler- ant counter) in the buffer to decide whether the time pressure feedback is delivered to participants in the next trial. The time pressure feedback only happens if: (a). Mean response time exceeds its threshold RT. Here we use the average re- sponse time in exercise session of each specific participant to be RT. (b). Delta response time exceeds its threshold deltaRT second. (c). Mean accuracy is lower than its threshold accuracy TA. Here we use the average accuracy in exercise session of each specific participant to be TA. (d). Push counter is lower than its threshold PC= 3. (e). Tolerant counter achieves its threshold = When the time pressure feedback is decided to be delivered to the participant in the next trial, push counter adds unit and tolerant counter is reset to O.
measures ANOVA. To discern specific dif-
1.4.1. RESPONSE TIME
In the analysis of between-subjects effects, the ANOVA re- vealed a significant effect of Group (F3,32 = 3.015, P 0.044 < 0.05) (Fig. 6(c)). Specifically, a significant differ- ence was identified between the none group (mean + SD: 0..022 + 0.021) and the random group (-0.105 + 0.025) with p = 0.039 < 0.05. The rule group showed a larger reduction in response time (-0.034 + 0.021) compared to the none group but smaller reduction compared to the static group (-0.054 + 0.022). Notably, the random group exhibited the most substantial reduction in response time. These results suggest that different types of time pressure stimuli may exert varying effects on response time.
These five metrics aim to ensure that time pressure feedback does not increase user response time but could increase user accuracy. Push counter and tolerant counter are designed to avoid introducing too much distraction to users. The strategy tolerates for a few trials and does not deliver time pressure feedback even if the first three metrics achieve the threshold. After the tolerant counter achieves the TC
Regarding within-subjects tests, a significant effect was observed across blocks (Fa,96 = 7.121, P < 0.001) (Fig. 6(c)), specifically between the following blocks: Block2 (-0.031 + 0.011) vs. Block, (-0.070 + 0.014): p = 0.023 < 0.05, Block2 vs. Blocks (-0.072 + 0.014):
15

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
b
a
Modular Math Task
0.5
0.00
attention anxiety
Progress Bar
88 73 mod
Math Trial
-0.02
0..0
-0.04
True Artine
Selection Button
-0.5
-0.06
110
-0.08
response time
accuracy
3
4
5
Block ID
Figure 5. a: Math arithmetic task and time pressure feedback. Each math trial is composed of two two-digit numbers Numi, Num and one-digit numer Nums, formatted as: Num, Numa (mod Nums). To solve this question, participants first subtract Numn from N Num and judge whether the subtraction result could be divisible by Numn. If it divisible, they select "True" button. Otherwise, they select "False" button. When adds one unit for each sccond and reset and add again when it accumulates five units. b: Overall trend of relative change of response time/accuracy (left y axis), and atlention/anxiety (right y axis), respectively, across blocks.
pressure feedback happens, progress bar will shown on top the math question, which
p = 0.026 < 0.05, Blocks (-0.033 + 0.013) vs. Block4: p 0.008 < 0.01, Block3 vs. Blocks: p 0.025 0.05.
et al., 2003), providing a foundation to inform the design of our cognition model. The observed effects underscore the relevance of considering both math task and question ID in modeling and understanding the dynamics of user response time under varying conditions.
No interaction was found between Block and Group (F9.96 0.958, P = 0.48). Furthermore, there was significant effeet of Date (Fi.32 0.003, P = 0.959) (Fig. 6(a)), and no other significant interaction effects were iden- tified (all > 0.05). These findings provide valuable insights into the differential impact of time pressure stimuli on response time and underscore the significance of within- subject variations across different blocks.
A.S. Math Logical Reasoning Agent
Existing work revealed humans' varied performance on dif- ferent cognitive tasks of diverse difficulty levels (Hanich et al., 2001). Therefore, it is essential to first encode features such as difficulty levels of cognitive tasks so that we could model participants' varied responses to different math ques- tions stem from features inherent in the questions. These features may influence user choice and response time even in ideal conditions (i.e., without external stimuli). To cap- ture such features, train logical reasoning agent capable of solving math questions in a manner similar to humans. Subsequently, feature representations are extracted from the intermediate output of this logical reasoning agent.
A.4.2. ACCURACY
No significant effect was observed in Group (F3.32
0.081, P 0.97 > 0.05), Block (F3,30 = 0.313, P 0.816 > 0.05) (Fig. 6(f)), or Date (Fi,32 = 0.861, P
0.36 > 0.05) (Fig. 6(b)). Additionally, no other signif- icant interaction effects were identified (all P > 0.05). This outcome aligns with expectations, as participants were instructed prioritize accuracy over response time consis- tently. Consequently, the accuracy of users' choices should generally be high, while response time may vary depending on the stimuli. The lack of significant effects in these factors supports the study design and participants' adherence to the specified priority in their decision- makion process.
Illustrated Fig. and Fig. 7, we employ an LSTM- based logical reasoning agent that takes math question as input and outputs the corresponding answer. For example, given the sequence "61 = 26(mod 4)" input, the agent outputs "3" (the remainder of the subtraction result, "35," of "61" and "26," divided by "4"). It is essential to note the distinction from the data collection process, where users are required to choose whether the subtraction result ("35") of "61" and "26" is divisible by "4" binary selection task.
The above results suggest that both time pressure stimuli and block number (not experiment date) may impact sponse time. This evidence contributes valuable insights and aligns with prior theory (Slobounov et al., 2000; Alexander
16

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
b
d
a 2.0
6
C
5
0.20
0.20
Density
1.5
Aositty
pensity
4
Density
0.15
Density
0.15
0.5
1.0
day
d1
2
3
day
d1
d2
0.10
0.05
day
d2
d1
0.10
0.05
day
d2
d1
0.0
-0.50 -0.250 0.25 0.50
-011
0
0.1
0.2
0.00
-5.0 -..5
2.5
0.00
-5.0
5.0
Response Time
Accuracy
Attention
Anxiety
Response Time (D
2
3
4
5
2
3
4
5
0.4
0.2
0.1
Accuracy
0.0
0.0
0.2
0.1
-0.4
none
static
random
rule
none
static
random
rule
o
2
3
4
5
2
3
2
4
0
Attention
0
Anxiety
2
0
-2
-2
-4
-4
1 random
none
static
rule
none
static
random
rule
Figure 6. a,b,c,d: overall distribution of relative change of response time (a), accuracy (b), attention (c), and anxiety (d), respectively, across 2 days. e.f.g.h: box plot of relative change response time (e), accuracy (0, attention (g), and anxiety (h), respectively, across groups and 4 blocks.
17

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
In other words, the logical reasoning agent is trained an- swer math arithmetic tasks correctly, rather than to predict user responses. This design choice ensures that the agent learns the potential arithmetic reasoning process and gener- ates representative features of math questions, rather than performing a binary classification task.
corresponding trial number in the dataset, resulting in total of 257 features for predicting user response for each sam- ple/trial. Users' response encompasses both user choice and response time. Consequently, the SVM models consist of a binary SVM classifier (SVC) to predict user choice (True or False selection) and an SVM regressor (SVR) to estimate user response time.
The logical reasoning agent is a eqquence-to-sqquene model based on an LSTM model. Before inputting the math question into the LSTM, the math question is encoded into sequence vectors from original string format. Each math question is denoted as ab = cd ( (mode), comprising 11 characters. We use one-hot encod- ing to deal with the characters. Specifically, each character is mapped into x 17 vector, where the location of this character in a pre-built character dictionary (L'O', "',,',,"""" ""","""""""" ",",",""" ""',",'"''""""" '6', '8', '9','",, ==", '',,'mi '''m," 'o', ''',''' 'D) is denoted as 1, and other locations are denoted as O. So we finally obtain the 11 x 17 vector for each math question.
The SVM model is implemented with
et al., 201 1). We use default regularization parameter, kernel, and other parameters for both SVM classifier (SVC) and regressor (SVR). The SVR takes 256 features from LSTM layer of math logical reasoning agent as well as question id for input and predicts user response time. The SVC not only predicts user response (choice) but also the probability Rp for each possible response, which serves as the boundary threshold in the drift-diffusion model.
A.7. Hybrid DRL Agent with Drift-Diffusion Model
For each math question string (1 x 11), we use sequence encoding mentioned above to encode it into a sequence vector (11 x 17), which is then fed into the LSTM model. The hidden unit is 256 neurons, which is then connected with 17 neurons with softmax activation function. Finally, the neuron with the highest probability is the final output answer. We use Keras(Chollet et al., 2015) to implement the model (loss function: categorical cross entropy, optimizer: Adam, learning rate: 0.001).
A.7.1. DRIFT-DIFFUSION MODEL (DDM)
The DDM assumes that users make decision by accumulat- ing evidence for each choice and make the final selection when the evidence accumulator achieves the threshold. Our framework incorporates the SVM model's predicting results into the DDM. Specifically, we use the output probability of SVC as the accumulated evidence, whose start point is 0.5. The boundary threshold is Rp. which is the probability when SVC makes the predictions. Different from traditional DDM that uses Bayesian modelling draw distribution of user response time, need to have fine-graines trajectory from start point to end point for each math trial to support our reinforcement learning process. Here we use Sigmoid function(Han & Moraga, 1995) to represent the trajectory from the start point to the end point. When users are solving math questions, they are usually more confident given more time to answer (Legg & Locker Jr, 2009; Pajares & Miller, 1994). Therefore, we could use monotonic function to represent the trajectory T, i.e. the Sigmoid function. More- over, we use Brownian motion (Smith, 2016) to add noise into the Sigmoid curve in order to introduce the randomness in decision making trajectory (Smith, 2016). Note that the final simulated trajectory is not always monotonic because such trajectory is modulated and modified by the DRL agent adaptively according to the environmental stimuli.
The logical reasoning agent aims to solve math tasks cor- rectly. In short, given one math question input, it directly outputs the arithmetic reasoning answer. Therefore, the training and testing of logical reasoning agent have no cor- relation or connection with real users' response. Hence, we prepare a separate dataset that is independent with users' dataset to train the logical reasoning agent. Finally, we have traversed all possible combinations of three numbers in math questions and gotten a dataset including 20414 sam- ples, which is split into training set (80%) and testing set (20%).
A.6. SVM Model Configuration
As previously mentioned, the second step in our simulation framework (Fig. 1) involves transferring features captured by the logical agent to real responses of humans by utilizing SVM models to predict users' baseline performance without time pressure. The features comprise the intermediate out- put of the LSTM layer, with the output neuron number set to 256, resulting in 256 features captured by the math answer agent. During cognition performance analysis, we observed that users' performance is influenced by the block number. Therefore, for each trial, we introduce the question id as an additional input feature, concatenated with the previous 256 features for SVM models. The question id denotes the
7.7. DRL TRAINING LOOP
The DRL training loop is composed of observation space, action space, reward, terminal state, and learning policy. The observation space serves as the model entrance to accept math question information and external stimuli as input. The action space contains set of potential actions that the DRL agent could take to perform simulation. The reward is used to guide the DRL agent to update its strategy powered
18

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table Performance of user choice classification of SVC models and response time estimation SVR models across three math question representations: Feature label: SVM (both SVC and SVR) takes features extracted from logical reasoning agent input, String label: SVM (both SVC and SVR) takes encoded vectors of raw math numbers as input, Digits label: SVM (both SVC and SVR) takes raw numeric math numbers as input.
Choice Classification
Response Time Regression (MAPE)
Input
Accuracy
F1-Score
Precision
Recall
Mean
STD
Lower
Upper
Digits 0.8107
0.0000
0,0000
0.0000
0.3740
0.3772
0.0121
1.4185
String
0.8174
0.0724
0.9333
0.0377
0.3813
0.3847
0.0135
1.4891
Feature
0.9613
0.8996
0.8833
0.9166
0.3652
0.3648
0.0108
1.3612
Observation Space
Math Answer Agent
b
Decoding
Answer
11x17
Math
Encoding
intermediate output
SVR
Estimated Response Time: Rg
Question
DynamicThm Time Presoure ntiuul Frame
Initial User Response Estimation
SVC
Estimated Choice: R., Probabiliyy: Rg
DRL Policy
Drift-Diffusion Model
action [-1,11]
CNN feature extractor
Start Point
Terminats
Figure The detailed architecture our CogReact framework. First, use math questions train math answer agent to solve them without considering users' response. Second, for each math question, we transfer features extracted from LSTM layer in math answer agent without time pressure make predictions user choice response time using SVM (initial estimation). The initial estimated response time and predicted choice probability will generate evidence accumulation trajectory in the drift-diffssion model. Third, the DRL agent will take math question and frame dynamic time pressure stimuli as input and specific action modulate evidence accumulation process. When evidence accumulator achieves boundary threshold, the final prediction of response time is generated and DRL agent achieves terminate state.
19

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
by the learning policy to take the optimal action so as to achieve highest possible reward. Terminal state represents the end of one training episode.
the reward function is:
Ert 0,
Esum Fs + P Ert < Egum
(1)
Fitt 2 Eeem
A.7.3. OBSERVATION SPACE
The observation space consists of two parts: math question information and dynamic time pressure visual stimuli. For each math trial, the math question encoded as a sequence vector (11 17) just like the logical reasoning agent. The dynamic time pressure visual stimuli is segmented into vi- sual frames just like what users perceive in the study. Given frame rate f. for each frame i, we can obtain the specific image S; of the visual stimuli for input in the observation space (we set f = 5). In order to encode the frame for input, we use default CNN feature extractor in Stable Baselines3 (Raffin al., 2021) to extract features automatically from the time pressure image.
where Er! and Eson are the estimated error rate of the hy- brid DRL's predicting response time (R,1) and the SVM's predicting response time (Regmm = RR) compared with real response time (R.,)) respectively, i.e. E,, = | R,,l/ R... Evem = |Raem Rul/Ru. P the penalty caused by terminal state if the hybrid DRL agent's step number ex- ceeds the maximum step threshold (P* = -1). Otherwise, P = 0.
A.7.7. LEARNING ALGORITHIM AND POLICY
We use Proximal Policy Optimization (PPO) (Schulman et al., 2017) as the learning algorithm and multilayer per- ceptron (MLP) be the policy for agent training. All hyperparameters and network architectures follow the de- fault settings in Stable (Rselin et al., 2021). The hybrid DRL model is implemented with PyTorch(Paszil (Paszee et al., 2019), Stable Baselines3(Rafiin et al., 2021), and Gym(Brockman et al., 2016).
A.7.4. ACTION SPACE
The action space contains one action with continuous nu- meric value from -1 to 1. The hybrid DRL agent takes one step for each frame i. When the output action a is O, it means that the current time pressure frame has no effect on evidence accumulator in drift-diffusion model. When the output action is from -1 to 0 or 0 to +1, then it means current time pressure frame leads to negative positive change 8 on evidence accumulator. The change 5 is ob- tained from the trajectory of drift diffusion model. Given boundary threshold Rp. start point Sp. response time RG and frame rate f. the change of evidence accumulation in each frame is 5 À x 6p. 6p Rp Spl/(f Re), where is the discounting factor to avoid the DRL agent introducing too aggressive bias.
A.8. Pure Deep Reinforcement Learning (DRL) Agent
The pure DRL model is implemented with PaTorchPaszkke et al., 2019), Stable (Rstlin et al., 2021), and Gym(Brockman et al., 2016).
Most parts of the pure DRL agent is the same as the hybrid DRL agent. The main difference lies in the way to represent effect of time pressure in human cognition performance. The hybrid DRL agent segments cognition process of each trial into frames and each action represents specific effect on each frame/step. However, for the pure DRL agent, directly takes the whole visual stimuli as input and output one action which represents the whole response time change due to time pressure. The final estimation of regulated response time is the sum of this action and basic response time estimated by SVR models.
A.7.5. TERMINAL STATE
Terminal state happens when the evidence accumulator achieves boundary threshold (Rt) or the hybrid DRL agent achieves maximum steps in one episode. Here, one episode represents one math trial in the dataset. Here we set the maximum response time to 10 seconds, consistent with the largest response time in our dataset. So the maximum step number N = RT X f = 10 x 5 = 50 steps. If the DRL agent takes Sn steps when the evidence accumula- tor achieves RG, then the new predicted response time is R,, = S./f.
1. DRL TRAINING LOOP
The DRL training loop is similar with the hybrid DRL agent, which still composed of observation space, action space, reward, terminal state, and learning policy. More details are depicted below.
A.7.6. REWARD
A.8. A.8.2. OBSERVATION SPACE
For each step during per episode, the hybrid DRL agent only gets reward in the terminal state. For other situations, the reward is O. The reward mainly aims to encourage the hybrid DRL agent to behave similarly with real users. Therefore,
The observation space still consists of two parts: math ques- tion information and dynamic time pressure visual stimuli. For each math trial, the math question encoding is the same as the hybrid DRL agent, For time pressure, different from
20

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Observation Space
a
Math Answer Agent
b
9S2K1
LSTM
Decoding
1ixi
11x17
Math Info
Encoding
intermediate output
SVR
Estimated Response Time: Rg
Time Pressure Stimal Video (S teconds)
Initial User Response Estimation C
SVC
Estimated Choice: Rg, Probabiiity:
Pre-trained model: Inception V3
DRL Policy
d
23x2048
action: Rg [11++1]
Final Response Time Estimation: R,, R. Rg XTT
max
Terminate
Roward
Figure 8. The detailed architecture of the pure DRL agent without drift-diffusion model.
the hybrid DRL agent, the pure DRL agent does not seg- ment visual stimuli into frames. Instead, it takes whole time pressure stimuli video (lasting 5 seconds) as input. We first use a pre-trained Inception-V V3 model(Szegedy et al., 2015) in Keras(Cholll) et al., 2015) to extract features from this video. The dimension of output features from each frame of the video is x 2048. For the whole video, use the same frame rate the hybrid DRL agent (f 5). So finally we have 5 seconds
A.8.5. REWARD
Different from the hybrid DRL agent that could only obtain reward in terminate state, for the pure DRL agent, it gets reward during each step (each trial in user dataset). The reward mainly aims to encourage the pure DRL agent to simulate effect of time pressure visual stimuli that is similar with real users' response. Therefore, the reward function is:
= 25 frames. The final feature dimension of this time pressure visual stimuli in observation space of the pure DRL agent is 25 x 2048.
Ert F/uml/Esum P 0,
E,t E Exgm Ev1 Egom
r
(2)
A.8.3. ACTION SPACE
where Ept and Esom are the estimated error rate of the pure DRL's predicting response time (R,)) and the SVM's predicting response time (Ramm R₂) compared with real response time (R.,) respectively, i.c. E, RRtt R.,l Ru,. Esom R Ru,l/ R., Pa is the penalty caused by terminal state if the pure DRL agent's estimated response time exceeds the normal range (0 to 10 seconds) (P* -1). Otherwise, P = 0.
The action space contains one action (Rş) with continuous numeric value which is normalized into the range from -1 to 1. Different from the hybrid DRL where each step is frame of user cognition process, here each step of the pure DRL agent is just one trial of users' response. For each trial, user baseline performance is obtained from SVM models. The action of the pure DRL agent represents perturbation for baseline response time (R₂) because of time pressure stimuli. Therefore, the final estimation of user response time is R2 R₂ Rs x RTmaz, where RT 10 is the maximum of user response time in the dataset.
A.8.6. LEARNING ALGORITHM AND POLICY
We Proximal Policy Optimization (PPO)(Schulman et al., 2017) the learning algorithm and multilayer percep- tron (MLP) to be the policy for agent training. All hyperpa- rameters and network architectures follow default settings in Stable Baselines3 et al., 2021).
A.8.4. TERMINAL STATE
The terminal state happens when final estimated response time R, exceeds normal range (smaller than 0 or larger than RT maz 10) or the pure DRL agent achieves maximum steps in one episode. Here, one step represents one math trial in the dataset. Here set the maximum step number to be 60 steps, which is the same as the trial number of each block in our user study result analysis.
A.9. Baseline Models
Our baseline models are adapted into our problem corre- sponding to the recent State-of-the- (SOTA) computa- tional models in human decision making (Bourgin et al., 2019) and response time prediction (Goetschalckx al.,
21

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
2024; Jaffe et al., 2023).
port the results for the best epoch out of 30 (based performance the validation set).
The whole dataset is first split into raw training (80%) and test set (20%). The raw training set is then split into model training set (80%) and validation set (20%). The validation set is used to select the best epoch.
LSTM ViT-B-16: This model is similar with LSTM+ + AlexNet but we replace the AlexNet with pre-trained ViT-B-16 (Dosovitskiy et al., 2020) in TorchVision (Paszke et al., 2019) to extract visual features from video frames. Each epoch takes about 60 minutes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
All neural network-based models use MAPE loss function, Adam optimizer (Kingma Ba. 2014) with learning rate of 0.001 and batch size of 16. All models trained on 2 Nvidia RTX A6000 GPUs (48GB GPU memory). All neural network models are implemented by PyTorch (Paszke et al., 2019) and other machine learning models are implemented by scikit-learn (Pedregosa et al., 2011).
MLP + 3D ResNet: This model is based on (Bour- gin et al., 2019) that uses MLP to predict human decision making. We follow the same ML.P ar- chitecture as (Bourgin al., 2019). To adapt to accept video input, we first pre-trained 3D ResNet (Tran al., 2018) in Torch Vision (Paszke et al., 2019) to extract features from the video directly (instead of each video frame). The ex- tracted features are then concatenated with ques- tion ID for input into the MI.P model. Each epoch takes about 25 minutes for training. We report the results for the best epoch of 30 (based performance the validation set).
Baseline Type I: Model Input Format: Task: Video, Feedback: Video, Question ID: Numeric Value.
hGRU (Linsley et al., 2018): This model comes from (Goetschalckx et al., 2024) to simulate human response time in visual tasks. We use (Goetschalckx al., 2024; Linsley et al.. 2018) to implement this model. The original hGRU model accepts image input. We adjust the dimensions accept video (including both task and time pres- sure visual feedback) as input. This model is trained from beginning without pre- preitraened mod- els. The output hGRU model is then concate- nated with question ID for input into linear layer (64 neurons) to predict response time. Each epoch takes about 40 minutes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
Baseline Type 2: Model Input Format: Task: Encoded String, Feedback: Video, Question ID: Numeric Value
LLTM-VI + 3D ResNet: This model is based on (Jaffe al., 2023) that uses LSTM to simulate human response time in cognitive tasks. Here we use the same LSTM configurations as (Jaffe et al., 2023). To adapt it to accept video input, we first use pre-trained 3D ResNet (Tran et al., 2018) in Torch Vision (Paszke al., 2019) to extract features from the video directly (instead of each video frame). The extracted feedback video fea- tures are then concatenated with both math task string with one-hot encoding and question ID for input into the LSTM model. The output of the LSTM layer is then passed into a linear layer (64 neurons) to predict response time. Each epoch takes about 12 minutes for training. We report the results for the best epoch out of 30 (based on performance the validation set).
LSTM + AlexNet: This model is based on (Jaffe al., 2023) that uses LSTM to simulate human response time in cognitive tasks. Here use the same LSTM configurations 2023). To adapt it to accept video as input, we first use pre-trained AlexNet (Krizhevsky et al., 2012) from Torch Vision (Paszke et al., 2019) to extract features from each frame of the video. The sequence of features from all frames are then in- put into LSTM layer. The output of the LSTM layer is then concatenated with question ID for input into a linear layer (64 neurons) to predict response time. Each epoch takes about 40 min- utes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
(Jaffe et al.,
LSTM-V2 3D ResNet: This model is similar with LSTM-VI V1 + 3D ResNet. The difference is that the extracted feedback video features are first fed into the LSTM layer and then the output is concatenated with both math task string with one- hot encoding and question ID to predict response time. Each epoch takes about 12 minutes for train- ing. We report the results for the best epoch out of 30 (based on performance on the validation set). MLP + 3D ResNet: This model is based on (Bour- gin et al., 2019) that uses ML.P to predict human
LLSTM VGG-16: This model is similar with LSTTM + AlexNet but we replace the AlexNet with pre-trained VGG-16 (Simonyan & Zisser- man, 2014) in TorchVision (Paszke et al., 2019) to extract visual features from video frames. Each epoch takes about 40 minutes for training. We re-
22

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
decision making. We follow the same MLP chitecture (Bourgin et al., 2019). To adapt it accept video imput, we first use pre-trained 3D ResNet (Tran et al., 2018) in Torch Vision (Paszke et al., 2019) to extract features from the video directly (instead of each video frame). The ex- tracted features are then concatenated with both math task string with one-hot encoding and ques- tion ID for imput into the MLP model. Each epoch takes about 15 minutes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
ResNet (Tran al., 2018) in Torch Vision (Paszke et al., 2019) extract features from the video directly (instead of each video frame). The ex- tracted features are then concatenated with both math task digits and question ID for input into the MLP model. Each epoch takes about 15 minutes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
Transformer + 3D ResNet: This model is simi- lar with MLP + 3D ResNet. The difference that we replace the MLP model with the trans- former model. We follow the default architecture of transformer in (Vaswani et al., 2017). Each epoch takes about 12 minutes for training. We re- port the results for the best epoch out of 30 (based on performance on the validation set).
Transformer 3D ResNet: This model is simi- lar with MI.P + 3D ResNet. The difference is that we replace the MLP model with the trans- former model. We follow the default architecture of transformer in (Vaswani et al., 2017). Each epoch takes about 12 minutes for training. We re- port the results for the best epoch out of 30 (based on performance on the validation set).
Baseline Type 4: Model Input Format: Task: Numeric Value, Feedback: Numeric Value, Question ID: Nu- meric Value. For this baseline type, all input features (lask, feedback, question ID) are directly concatenated into 1D array for input into models. The baseline models in this type are mainly based (Bourgin et al., 2019), which presents several machine learning models to predict human decision making with similar model input.
Baseline Type 3: Model Input Format: Task: Numeric Value, Feedback: Video. Question ID: Numeric Value.
LS + 3D ResNet: This model is based on (Jaffe et al., 2023) that uses LSTM to simulate human response time in cognitive tasks. Here the same LSTM configurations (Jaffe al., 2023). To adapt it to accept video input, we first use pre-trained 3D ResNet (Tran et al., 2018) in TorchVision (Paszke et al., 2019) to extract features from the video directly (instead of each video frame). The extracted feedback video fea- tures are then concatenated with both math task digits and question ID for input into the LSTM model. The output of the LSTM layer is then passed into linear layer (64 neurons) to predict response time. Each epoch takes about 12 min- utes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
Decision Tree: We use scikit-learn (Pedregosa et al., 2011) to implement this model and follow all default settings scikit-leam. The training process takes within 10 minutes.
Linear Regression: We use scikit-learn (Pe- dregosa et al., 2011) to implement this model and follow all default settings in scikit-learn. The training process takes within 10 minutes.
LSTM: This model is based (Jaffe et al., 2023) that uses LSTM to simulate human response time in cognitive tasks. Here use the same LSTM configurations as (Jaffe et al., 2023). Each epoch takes about 2 minutes for training. We report the results for the best epoch out of 100 (based on performance on the validation set).
LSTM-V2 + 3D ResNet: This model is similar with LSTM-V1 + 3D ResNet. The difference is that the extracted feedback video features are first fed into the LSTM layer and then the output concatenated with both math task digits and question ID to predict response time. Each epoch takes about 12 minutes for training. We report the results for the best epoch out of 30 (based on performance on the validation set).
MLP: This model is based on (Bourgin et al., 2019) that uses MLP to predict human decision making, We follow the same MLP architecture as (Bourgin et al., 2019). Each epoch takes about minutes for training. We report the results for the best epoch of 100 (based on performance on the validation set).
MLP 3D ResNet: This model is based on (Bour- gin et al., 2019) that uses MLP to predict human decision making. We follow the same MLP ar- chitecture as (Bourgin et al., 2019). To adapt it to accept video input, we first use pre-trained 3D
Random Forest: We use scikit-learn (Pedregosa et al., 201 to implement this model and follow all default settings seikit-leam. The training process takes within 10 minutes.
23

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
-VVM:
We use scikit-learn (Pedregosa al., 2011) implement this model and follow all default settings in scikit-learn. The training process takes within 10 minutes.
which predominantly concentrated on the coarse-grained posterior estimation of decision- through reinforce- ment learning (Viejo et al., 2015; Pedersen et al., 2017). The elucidative nature of our framework significantly aug- ments our capacity to comprehend and interpret the intricate interplay between environmental stimuli and cognitive be- haviors.
Transformer: We follow the default architecture of transformer in (Vaswani et al., 2017). Each epoch takes about 2 minutes for training. We report the results for the best epoch out of 100 (based on performance on the validation set).
The principles underlying CogReact may be extended to the analysis of neural and physiological responses to external stimuli. Although such Rata-whether derived from neural activity or wearable sensors-poos significant challenges for direct modeling due to their high-frequency, noisy time- series nature, they hold considrable potential. Specifically, these recordings can serve as proxies for the trajectory of evidence accumulation in human decision-making and other cognitive processes. Mapping these novel forms of evi- dence accumulation within specific cognitive tasks offers a promising avenue for capturing human cognition at a highly fine-grained temporal and representational level.
Baseline Type 5: Model Input Format: Task: Encoded String, Feedback: Numeric Value, Question ID: Nu- meric Value. For this baseline type, the math task ques- tions come with textual string format and get encoded with one-hot encoding (Rodríguez et al., 2018), which are then concatenated with feedback and question ID for input into models. The baseline models in this type are mainly based on (Bourgin et al., 2019), which presents several machine learning models to predict human decision making with similar model input. The settings of these baseline models (Decision Tree, Lin- ear Regression, LSTM, MLP, Random Forest, SVM, Transformer) are the same as Baseline Type 4.
A.11. Generalization in New Tasks and Datasets
A. 11.1. DATASETS
We further evaluate the generalization ability of our model in two additional public datasets. The first is CPC18, a widely used benchmark for modeling human cognition in a decision making task (Bourgin et al., 2019). In this dataset, participants engaged in a gambling game where they made binary decisions in each trial. Each choice offers different reward/loss with certain probabilities, and feedback indi- cates the alternative reward or loss if the participant had selected the other option. Consistent with previous experi- ments, the model predicts user response times per trial based on the model input including task information (reward/loss probability configurations of two choices) and feedback in- formation (alternative reward/loss), represented as numeric arrays. We obtained total of 30,489 trials from 240 partici- pants with valid response time from the raw dataset.
A. Further Discussion
Modelling dynamics in human cognitive responses to ex- ternal stimuli is fundamental understand how the brain dynamically reacts to the environment. However, the pre- vailing trend in contemporary research (Jaffe et al., 2023; Peysakhovich& & Naecker, 2017; Lake et al., 2017; Ma & Peters, 2020; Mchrer et al., 2020; Golan et al., 2020; Kumb- har et al., 2020; Battleday et al., 2017: 2020; Singh et al., 2020; Peterson al., 2018; Battleday al., 2021; Peterson et al., 2021; Noti et al., 2016; Bourgin et al., 2019; Plonsky et al., 2017) predominantly centers on the modeling of hu- man cognition within standardized and idealized contexts, thereby often neglecting the nuanced influence exerted by external stimuli. Conversely, certain investigations adopt an oversimplified perspective by treating external stimuli as persistent and unchanging factor throughout the cognitive processes (Bourgin al., 2019). A more sophisticated mod- eling methodology is deemed essential, particularly when addressing dynamic environmental stimuli that exhibit tem- poral fluctuations contingent upon user performance. This refined approach advocates for a nuanced consideration of stimuli variation at fine temporal scales, thereby perpetuat- ing+ a continuous impact on human cognitive behaviors.
The second public dataset, PeerEdu (Xu et al., 2025), cap- tures students' cognitive states during learning with exter- nal peer feedback. Students watched online video lectures while their cognitive states, were passively recorded using sensors. Specifically, we use one cognitive state named cu- riosity (continuous value from 0 to 1) for evaluation, which is the most significantly impacted by peer feedback from (Xu et al., 2025). Unlike previous tasks requiring active but binary human choice input, PeerEdu focuses on passive yet continuous cognitive states without students' active in- put. Peer feedback was delivered by highlighting specific regions on video lecture slides that peer students focused on, updated continuously based on lecture progress.
Our hybrid modeling approach, characterized by the incor- poration of Deep Reinforcement Learning (DRL) to emulate external stimuli within the explainable drift-diffusion model at a granular level, takes into account subject-specifee and stimuli- couull-speciee specific behavioral distinctions. This distinctive feature sets our framework apart from antecedent studies,
To simulate curiosity in this learning task, each lecture was
24

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
divided into transcripts, with each transcript representing sentence spoken by the lecturer. Task information includes transcript content, while feedback information comprises the highlighted text in peer feedback regions on the slides. The model predicts curiosity for each transcript by taking both task and feedback information (textual format) as input. We obtained a total of 12,804 samples from 300 students in PeerEdu, where each sample corresponds to the curiosity of one student during transcript with specific feedback.
A.1..2. FRAMEWORK ADAPTATION
To adapt our framework for the decision making task, we replace the math agent with an LSTM-based risk agent in the first step in Fig. This risk agent, using the same architecture as before, predicts potential reward/loss (feed- back information) from task inputs in each gambling trial, extracting risk features for the SVM model in the second step. The second step keeps the same as Fig. I to predict user choice and response time without feedback, which are used to generate the evidence accumulation process in the DDM step. The DRL loop incorporates an adjusted obser- vation space to handle the task and feedback information in decision making, while maintaining the same action space and reward functions the previous logical reasoning task for simulating response time changes.
For the learning task adaptation, we replace the math agent in our framework with large language model (LLM),, re- ferred to as the LLM agent, to extract features from tex- tual task and feedback information, leveraging the strong textual data mining capabilities of LLMs (Wang et al., OpenAl's text-embedding-3 small model to generate embeddings from both task and feedback information, which are then input into SVM mod- els in the second step of Fig. 1. To handle the absence of binary user input, we categorize curiosity values into high or low levels based on their position relative the median, enabling SVM models to predict curiosity level (SVC: high low) and actual value (SVR), similar predicting binary choice and response time in previous tasks. This approach enables adaptation to continuous response modeling without significant changes to the framework and can be extended to other continuous behaviors in the future. The DRL loop incorporates an updated observation space to process embed- dings from task and feedback information, with action space and reward functions adjusted to align with the curiosity scale.
2023). Specifically,
25

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table Results baseline model performance on response time simulation Math Task. MAPE, we show mean value
(Mean), standard deviation (STD), 2.5th (Lower) and 97.5th (Upper) percentiles of the MAPE distribution (95% confidence interval).
MAPE
Model Input Type
Model Type Name
Mean
STD
Lower
Upper
hGRU
0.3335
0,2486
0.0153
0.9406
Task: Video
LSTM + AlexNet
0.3344
0.2602
0.0132
0,9954
Feedback: Video
LSTM VGG-16
0.3355
0.2708
0.0128
1.0393
LSTM ViT-B-16
0.3339
0.2573
0.0145
0.9852
MLP + 3D ResNet
0.3330
0,2507
0.0121
0,9390
LSTM-V1 + 3D ResNet
0.3334
0,261
0.0151
0,9866
Task: Encoded String
LSTM-V2 + 3D ResNet
0.3376
0,2169
0.0185
0.7618
Feedback: Video
MLP + ResNet
0.3331
0,2550
0.0125
0.9601
Transformer + 3D ResNet
0.3306
0.2496
0.0145
0.9462
CogReact
0.2999
0.2318
0.0131
0.8029
LSTM-V1 + 3D ResNet
0.3341
0,2617
0.0152
0,9923
Task: Numeric Value
LSTM-V2 3D ResNet
0.3286
0.2538
0.0147
0.9707
Feedback: Video
MLP 3D ResNet
0.3333
0.2579
0.0147
0.9731
Transformer + 3D ResNet
0.3315
0,2526
0.0152
0.9615
Decision Tree
0.3617
0.364
0.015
1,3729
Task: Numeric Value
Linear Regression
0.3595
0.3608
0.0113
1.3399
Feedback: Numeric Value
LSTM
0.3059
0.2434
0.0141
0.9253
ML.P Random Forest
0.3293
0.2441
0.0151
0.9257
0.3650
0.3684
0.0117
1.3448
SVM
0.3299
0.3108
0,0113
1.1827
Transformer
0.3052
0,2446
0,0112
0,9309
CogReact
0,2703
0.2224
0.0093
0.7631
Decision Tree
0.3639
0.3639
0.0112
1.3917
Task: Encoded String
Linear Regression
0.3512
0.3469
0.0105
1.3176
Feedback: Numeric Value
LSTM
0.3278
0,2478
0,0142
0,9397
MLP
0.3333
0,2577
0.0145
0,9724
Random Forest
0.3600
0,3630
0.0130
1.3620
SVM
0.3245
0.3101
0.0123
1.1952
Transformer
0.3299
0.2481
0.0142
0.9350
26

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table Statistical results by comparing our CogReact model in Type (Task: Encoded String, Foodback: Vidco) with each of baseline model respectively in the Math Task.
Statistical Tests for CogReact in Type II
Model Input Type
Model Type Name
Kolmogorov- Smimrov
Permutation
hGRU
< 0.001
p < 0.001
Task: Video
LSTM AlexNet
< 0.001
P < 0.001
Feedback: Video
LSTM VGG-16
< 0.001
p 0.001
LSTM VIT-B-16
< 0.001
p < 0.001
MLP + 3D ResNet
< 0.001
< 0.001
LSTM-VI 3D ResNet
p < 0.001
P < 0.001
Task: Encoded String
LSTM-V2 3D ResNet
p < 0.001
p 0.001
Feedback: Video
MLP 3D ResNet
< 0.001
p 0.001
Transformer 3D ResNet
< 0.001
< 0.001
LSTM-V1 3D ResNet
< 0.001
< 0.001
Task: Numeric Value
LSTM-V2 3D ResNet
p < 0.001
P < 0.001
Feedback: Video
MLP 3D ResNet
< 0.001
P 0.001
Transformer 3D ResNet
< 0.001
P 0.001
Decision Tree
< 0.001
< 0.001
Task: Numeric Value
Linear Regression
< 0.001
P 0.001
Feedback: Numeric Value
LSTM
0.819
P 0.263
MLP
< 0.001
P 0.001
Random Forest
< 0.001
< 0.001
SVM
< 0.001
< 0.001
Transformer
= 0.920
= 0.332
Decision Tree
p < 0.001
P 0.001
Task: Encoded String
Linear Regression
< 0.001
P 0.001
Feedback: Numeric Value
LSTM
< 0.001
< 0.001
MLP
< 0.001
< 0.001
Random Forest
< 0.001
< 0.001
SVM
< 0.001
< 0.001
Transformer
P < 0.001
P 0.001
27

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table Statistical results by comparing our CogReact model in Type (Task: Numeric Value, Feedback: Numeric Value) with each
baseline model respectively in the Math Task,
Statistical Tests for CogReact in Type IV
Model Input Type
Model Type Name
Sratnogoro--mmiioo
Permutation
hGRU
p < 0.001
p < 0.001
Task: Video
LSTM + AlexNet
p < 0.001
p < 0.001
Feedback: Video
LSTM + VGG-16
p 0.001
p 0.001
LSTM + ViT-B-16
p 0.001
p 0.001
MLP + 3D ResNet
P 0.001
p 0.001
LSTM-V1 + +3D ResNet
p < 0.001
P < 0.001
Task: Encoded String
LSTM-V2 + 3D ResNet
p 0.001
p < 0.001
Feedback: Video
MLP + 3D ResNet
p 0.001
p 0.001
Transformer 3D ResNet
P 0.001
P 0.001
LSTM-VI + +3D ResNet
p < 0.001
P < 0.001
Task: Numeric Value
LSTM-V2 + 3D ResNet
p 0.001
p 0.001
Feedback: Video
MLP + 3D ResNet
p 0.001
p 0.001
Transformer 3D ResNet
p 0.001
p 0.001
Decision Tree
p < 0.001
P < 0.001
Task: Numeric Value Feedback: Numeric Value
Linear Regression
P < 0.001
P < 0.001
LSTM
p 0.001
p 0.001
MLP
P 0.001
P 0.001
Random Forest
P 0.001
P 0.001
SVM
P < 0.001
P 0.001
Transformer
P < 0.001
P < 0.001
Decision Tree
P 0.001
P 0.001
Task: Encoded String
Linear Regression
p 0.001
p 0.001
Feedback: Numeric Value
LSTM
p 0.001
p 0.001
MLP
P < 0.001
P < 0.001
Random Forest
< 0.001
P < 0.001
SVM
P < 0.001
P < 0.001
Transformer
P 0.001
P 0.001
Table Statistical results by comparing CogReact with each baseline model and ablation model (our model variants in ablation studies) respectively in PeerEdu dataset.
Statistical Tests for CogReact in PeerEdu
Model Input Type
Model Type Name
Kolmogorov-
Permutation
Pure DRL
p < 0.001
p < 0.001
Task: Numeric Value
Unencoded Hybrid DRL
p 0.037
p 0.001
Feedback: Numeric Value
Encoded SVM
P 0.001
p 0.001
LSTM
p 0.001
p 0.001
Transformer
p < 0.001
p 0.001
MLP
p < 0.001
p 0.001
SVM
< 0.001
p < 0.001
Random Forest
< 0.001
P < 0.001
Decision Tree
P < 0.001
P 0.001
Linear Regression
P 0.001
P 0.001
28

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Table 8. Statistical results by comparing CogReact with cach of baseline model and ablation model (our model variants in ablation studics)
respectively in CPC dataset.
Statistical Tests for CogReact in CPC
Model Input Type
Model Type Name
Kolmogorov-Smirnor
Permutation
Pure DRL.
p < 0.001
< 0.001
Task: Numeric Value
Unencoded Hybrid DRL
= 0.058
= 0.677
Feedback: Numeric Value
Encoded SVM
p < 0.001
< 0.001
LSTM
p < 0.001
< 0.001
Transformer
P < 0.001
p < 0.001
MLP
p 0.001
p 0.001
SVM
P 0.001
0.001
Random Forest
P 0.001
0.001
Decision Tree
< 0.001
< 0.001
Linear Regression
< 0.001
< 0.001
b
a
795
700
25
50
75
100
Loss
600
2.00
2
762
1.75
3
567
500
1.50
% of Prediction Accuracy
1.25
4
461
0
400
1.00
5
320
-30
0.75
Neuron
0.50
6
203
-00
0.25
32
64
0.00
Mekoz
7
0
0
0
145
0
-100
128
256
8
49
0
0
20
40
60
80
100
2
3
4
5
6
7
8
Epoch
Figure 9. a. Confusion matrix (x axis: ground truth, axis: prediction) for testing set prediction of the logical reasoning agent (LSTM
neuron 256). Training loss accuracy with training epochs across four kinds LSTM neurons of logical reasoning agent.
29

---

CogReact: A Reinforced Framework to Model Human Cognitive Reaction Modulated by Dynamic Intervention
Hybrid DRL Group-level
Pure DRL
SVM
a
Individuul-level
C
General-level
d
LOPO-Ievel
41
41
2
41
38
36
35
6
34
34
32
32
6
32
8
32
10
31
10
31
10
10
13
0.1
29
13
0.2
29
0.2
14
0.2
26
14
28
14
Y04
26
14
V0.4
28
15
0,3
0.8
0.4
0.8
16
0.8
26
16
0.8
19
20
21
0.5
22
24
18
20
22
23
24
18
20
24
18
20
21
22
24
f
h
e
g
B
C B
C B
C B
C
0.26
0.26
0.30
0.30
0.30
0.30
0.34
0.34
0.34
0:34
0:38
0.38
0,38
D
A: none group, B: random group, C: static group, D: rule group
Figure 10. Evaluation results in the logical reasoning task for MAPE in different levels. a.b.c.d..ff.g.hh Average MAPE for each participant (a.b.c.d\/roup (e.f.g.h) in predictions of testing set from Hybrid DRL agent, Pure DRL agent, and SVM model in four training strategies (a.e. Individual-Level b.f. Group-Level, c.g. General-Level, d.h. LOPO-Level), respectively. (The number around the circle represents participant id in a.b.c.d).
Tybrid DRL
0.270
Hybrid DRL
0.357
Hybrid CRL
0.950
LSTM
LSTM
0.461
LSTM DDM
0.480
LSTM-DDM
Transformer
1.207
Donsfomme++DOM
0.331
Transtrmrr-
Trensformer+ DDM
1.198
MLP
0.329
0.476
MLP-
1.204
DONV
0,312
MLVI-DDM
0.463
MUNV-DDDM
1.236
MURUZ+DOM
MLPV2-DDM
0.468
MLPV++DDM
0.4
0.5
1.5
2.0
Math Task
Decision Making Task
(c). Learning Task
Figure II. Comparison results between CogReact Model (Hybrid DRL) and baseline deep learning models with without DDM three datasets. For statistical analysis with both Dolmogorov-Smimne Smirnov test and Permutation test, indicates p < 0.05, as indicates p 0.01, indicates < 0.001. The results show that the MAPE of our model is significantly (p 0.001) smaller than all deep learning models with / without DDM in all three datasets.
30


Reviews:
[{"review_id": "RXtqXsVL6l", "reviewer": "Reviewer_kUQA", "summary": "This work introduces a problem in accounting for the effect of dynamic environmental stimuli on human behavior, such as a continuous time limit indicator that induces a time pressure on participants. The CogReact framework has many moving pieces but is quite elegant. An LSTM is trained to solve the task humans are tasked with, and its embeddings are used to estimate user responses (answers and reaction times). These predictions are then inputted into a DDM model and a Deep RL agent is trained to modulate the evidence accumulation process through its actions (where its actions is a continuous value that modulates the DDM). They collected a mathematical reasoning dataset to test their model on accounting for human behavior (mainly RTs), comparing it a large number of baselines, showing many ablation studies to show how the individual components contribute, and show its ability to reveal different time pressure effect trajectories for different experimental groups in which time pressures varied. They also applied their framework to a decision making and learning task, highlighting the ability for this method to generalize to other kinds of tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful and encouraged by the reviewers’ recognition of our work’s contribution, including sound methodology and novel model contribution in ICML community (reviewer kUQA, uxDz, xPom), valuable dataset contribution (reviewer uxDz), extensive literature review (reviewer kUQA), convincing and well-supported claim (reviewer kUQA) with rigorous evaluation and comprehensive experiments/analysis for generalization across different task domains (reviewer kUQA, xPom). Moreover, reviewer uxDz, 3FF7, and kUQA also acknowledge the suitable benchmark datasets and evaluation metrics. Both reviewer kUQA and reviewer uxDz acknowledge that our work explores an understudied problem to study computational neuroscience with neural networks in ICML community, which can be a particularly powerful tool in analyzing human behavior. Reviewer xPom also suggests our valuable insights to fill a gap in literature by focusing on dynamic environmental stimuli and aligns with integrating machine learning in cognitive modeling. Finally, reviewer xPom suggests that our generalization across tasks and use of real-world datasets contribute to a more practical understanding of human cognition. \n\n---\n\n**Individual Response to Reviewer kUQA**\n\nWe sincerely appreciate your recognition of our work and constructive suggestions. Your encouragement and insightful feedback further motivate us to continue exploring this important research area, to deepen our understanding of human cognitive processes and their interaction with environmental stimuli. Below, we provide a point-by-point response to each comment.\n\n**Experimental Designs Or Analyses**: We appreciate your recognition of our experimental rigor. To strengthen our evaluation, we have added statistical tests (please check our response to point [Claim] for reviewer uxDz) and additional experiments to explore DL+DDM (please check our response to point [Experiment] for reviewer 3FF7). \n\n**Supplementary Material**: We are glad that the interpretability analyses were engaging. Given this feedback, we will consider moving some of the most critical insights from the supplement into the main paper, ensuring that these contributions are more visible to readers.\n\n**Other Strengths And Weaknesses**: We completely agree that expanding CogReact beyond reaction time is an exciting avenue for future work. Please see our detailed answers to your question 2 below.\n\nQuestions for Authors\n\n**Question 1**: Thank you for the insightful question. The difference in representation strategies between artificial agents and humans can indeed impact response time prediction due to the different embeddings, which may potentially impact the final simulation. However, in our exploration, we find that the constraints of problem space are more related to the different modalities that can be handled by different agents, rather than the different strategies/representations of these agents. This means that the different strategies/representations may mainly impact the simulation accuracy and interpretation, rather than the constraints of problem space. By contrast, the modalities that can be supported by different agents are more related to constraining the problem space. For example, some agents like CNN are good at processing visual stimuli to model visual cognition [1] while other agents like LLMs are good at processing textual stimuli. Therefore, we can insert different agents in our framework to adapt to different problem spaces for generalization, just like our paper that uses different agents to process different tasks and feedback modalities across different datasets (table 2 in our paper).\n\n**Question 2**: We appreciate this thought-provoking suggestion. The principles behind CogReact may be adapted to analyze neural and physiological responses to external stimuli. \nHowever, these neural or wearable recordings are more challenging for direct modelling due to the time series format with high frequency and much noise, different from response time. However, such neural/wearable recordings can potentially serve as the representations of evidence accumulation trajectory in human decision making or other cognitive processes. Mapping such novel formats of evidence accumulation in specific cognitive tasks is promising to represent human cognitive processes in a very fine-grained manner. In the final version, we will expand our discussion on future applications, including potential extensions to wearable sensor data and neural activity modeling.\n\nWe once again appreciate your thoughtful and constructive feedback. Your insights have helped us refine our presentation and clarify our contributions. Given the opportunity, we are confident that minor revisions incorporating these valuable suggestions will significantly enhance the clarity and impact of our paper.\n\nReference:\n\n[1]. Grace W. Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future, 2021", "claims_and_evidence": "They claim its the SOTA method for accounting for human behavior when modulated by external dynamic stimuli and I think this is very well supported due to very rigorous comparing with other methods across different kinds of tasks. They also claim superior training efficiency and interpretability, which is also well supported from Figures 9-10. The paper was quite convincing to me.", "methods_and_evaluation_criteria": "The datasets tested were very appropriate for the problem at hand and their metrics (e.g. MAPE) make sense to me.", "theoretical_claims": "This is not so much a theory paper. Did not see any proofs.", "experimental_designs_or_analyses": "The experimental design and analyses are extremely rigorous. Lots of baselines, ablation studies, multiple human datasets, creative interpretability analyses, etc.", "relation_to_broader_scientific_literature": "I think this is an exciting paper that explores a very understudied problem, the effect of external dynamic stimuli on internal cognitive processing of humans. I think this will be a particularly powerful tool in analyzing human behavior.", "essential_references_not_discussed": "The related work and literature review are quite extensive.", "other_strengths_and_weaknesses": "I highlighted a lot of the strengths previously in this review. One potential (small) weakness is the focus on human reaction time, which is very rich but it would be exciting if CogReact could be applied to other kinds of measures.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Many artificial agents (e.g. LSTMs, LLMs) can solve these problems with a different strategy or representation than that of humans. Does this affect the ability to make the initial response time prediction? If so, could it possibly constrain the space of problems you can use CogReact on?\n\n2. Can the ideas from CogReact be applied to other kinds of measures?  A version of CogReact that could be applied to wearable recordings or non-invasive neural measurements would be very exciting.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eWgKHViB9O", "reviewer": "Reviewer_uxDz", "summary": "This paper tackles a fundamental question: How can we accurately simulate the influence of dynamic environmental stimuli on human cognitive behavior at a fine-grained level? To address this, CogReact is introduced, a hybrid framework that integrates drift-diffusion models (DDMs) with deep reinforcement learning (DRL) to model the perturbation effects of environmental stimuli. Additionally, the 21,157 logical reasoning responses dataset represents a valuable contribution to the field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and your recognition of our contribution in the understudied problem in ICML with suitable benchmarks. We made significant efforts with extra experiments to address your concerns. Below, we provide a point-by-point response.\n\n**Claim**: \n\nUnlike MSE, MAPE’s range is not from 0 to 1. For example, MAPE can be as high as 2 if the predicted response time is 0.3 seconds but the label is 0.1 seconds. \n\nThe high variance of MAPE is because our prediction problem itself is very challenging due to the diverse nature of human behaviors and user variance, i.e. different people have different behavioral patterns. This challenge is also evidenced by Table 4 where most baselines also have much higher variance than our model. \n\nWe also added statistical tests following your suggestion. We use both Kolmogorov-Smirnov test and Permutation test because they do not necessarily assume the data to be normally distributed. We applied them in all baselines in Table 4 and detailed statistical results are in Table 6 in our repo: https://github.com/cogreact/CogReact/. We find that CogReact achieves significantly lower MAPE (p value<0.001) than most baselines except for LSTM/Transformer in Type IV (numeric for task/feedback). This is due to different task/feedback modality encoding since CogReact uses Type II (Task: String, Feedback: Video). When we apply CogReact to numeric encoding of Type IV as well, results in Table 5/Table7 show that ours can also achieve significantly lower MAPE (p<0.001) than LSTM/Transformer in Type IV. Moreover, statistical tests in other two datasets also show significant improvement of our model in Table 8/Table 9. All tables are in our github repo.\n\nPlease also check our response to point [Claim] for reviewer 3FF7.\n\n**Experiment**:\n\nWe denote time pressure effect trajectory as Tp trace (generated from our model), which represents how feedback modulates human evidence accumulation in cognitive process. In Fig. 10, each curve represents the Tp trace of each specific user, and (a,b,c,d) represent users’ Tp traces in different groups applying different time pressure feedback. \n\nThus, higher concentration (i.e. Tp traces are more similar across users) in random group shows that users are more consistently impacted by random feedback, compared to other groups. This suggests that random groups’ feedback has a higher impact on human cognition compared to other groups. Such simulations of Tp traces also echo the real user data, which also shows a higher impact of random group’s feedback compared to other groups. In this way, our simulated Tp traces are consistent with real user behavioral patterns modulated by feedback. With this consistency, Tp traces serve as convincing visualization to represent and explain (i.e. **the interpretability**) how human cognitive process is modulated by feedback, because x axis in Fig 10 represents the time during human cognitive process and y axis represents positive or negative impact of feedback on human cognitive process for each specific time point.\n\nNote: we also use quantitative measurements beyond visualization, depicted in Section 4.6.\n\nTo strengthen our study, we also had extra experiments to explore DL+DDM (please check our response to point [Experiment] for reviewer 3FF7).\n\n**Relation To Broader Literature**: please see point [Claim] above for first concern, and Question 2 for second concern.\n\n**Other Comments**:\nWe appreciate the suggestion. We will restructure the figures and text accordingly to make it more concise while retaining essential details.\n\n**Question 1**: Please see point [Experiment].\n\n**Question 2**: \n\nOur model serves as a potential abstraction of human cognition by capturing how dynamic stimuli modulate cognitive responses and generalizing across different cognitive tasks. It integrates RL and DDM, and both are well-established mechanisms grounded in cognitive theories [1,2] to model and explain human cognition.\n\nWhile our model exhibits predictability, we do not claim to be a complete abstraction of human cognition. Instead, we aim to develop a predictive and interpretable tool that approximates certain cognitive behaviors under dynamic stimuli.\n\nAI and neuroscientists can benefit in several ways. For AI, we offer a novel hybrid model to enhance human-in-the-loop AI system design where understanding human responses are crucial. It also provides a benchmark to integrate cognitive constraints into ML to improve human-like adaptability. Neuroscientists can use it to investigate how reinforcement learning mechanisms interact with cognitive models like DDM, leading to new hypotheses of human cognition under uncertainty. It can also be generalized to other cognitive processes like attention / learning, offering broader applications in cognitive science.\n\nReference\n\n[1]. A. Ross Otto et al. Cognitive Control Predicts Use of Model-based Reinforcement Learning\n\n[2]. Patricio Farrell et al. Drift-diffusion models", "claims_and_evidence": "I feel several claims are not well supported.\n\n> \"Results are depicted in Table. 1 and Table. 4, showing that CogReact has the best response time prediction performance (lowest MAPE) by comparing with other models in both the same and different model input types.\"\n\nThe high variance observed across all baselines raises concerns about the reliability of the results, making it difficult to draw meaningful conclusions. Notably, the performance of the proposed system fluctuates significantly, ranging from 0.0131 to 0.8029, which suggests instability in the reported outcomes. The absence of a paired t-test or any statistical significance analysis weakens the validity of the comparisons, leaving open the question of whether the observed improvements are significant. Then it makes very little sense to claim further on:\n\n> \"Why does the logical reasoning agent work? The second ablation study explores why the math logical reasoning agent could extract useful features from math questions (in the first step of our framework).\"", "methods_and_evaluation_criteria": "* Method: Using DDM for human-like decision-making has been a standard thing. The chosen variants of the neural models are naive, but the authors demonstrate a list of different choices. Given that the goal is to predict human cognitive processes rather than leaderboard chasing, I am fine with the method.\n\n* Benchmark: The benchmark is standard math reasoning, which makes sense in this context.", "theoretical_claims": "There is no theoretic claim in this work.", "experimental_designs_or_analyses": "I find the analysis on interpretability confusing:\n> \"We observe that the time pressure effect trajectories are more concentrated in the random and rule groups but divergent in the none and static groups\"\n\nFigure 10 is not very well presented, and its relevance to interpretability is not established. I tried my best to understand what the authors tried to convey in this section but I am not able to fully understand.", "relation_to_broader_scientific_literature": "I am concerned about the goal and position of this work in the landscape of AI and neuroscience.\n\n> \"This advancement has the potential to offer new insights into machine learning and neuroscience by fostering computational models that better understand human cognition.\"\n\nFirst, the proposed computational model does not convincingly simulate human cognition at a satisfactory level. While it may capture certain patterns, the degree to which it truly reflects human cognitive processes remains questionable due to the huge variance. Second, even if the model demonstrates some level of predictability, this does not logically guarantee that it serves as a true abstraction of human cognition. Correlation does not imply a faithful mechanistic representation, and without stronger theoretical grounding or empirical validation, its claim to cognitive plausibility remains uncertain.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I have discussed the weaknesses in previous sections.\n\nI think one of the strengths is that contributions of this kind are rare in the field of ML, especially in the ICML community. Although I am not convinced that this work meets the acceptance bar of ICML, I think it is meaningful to study computational neuroscience with neuro network models.", "other_comments_or_suggestions": "Please consider moving the figures to the main paper and condense the main text. There are many verbose and unnecessary detail in the main paper.", "questions_for_authors": "1. Could you elaborate on what you mean by \"the time pressure effect trajectories are more concentrated\"?\n2. Even if the model demonstrates some level of predictability, why could it logically serve as a true abstraction of human cognition? If not, what is the goal of this study? How can AI / neurosci researchers benefit from the results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E6Jb0I1PYa", "reviewer": "Reviewer_xPom", "summary": "This paper presents a hybrid framework that integrates the Drift-Diffusion Model (DDM) from previous cognitive methods with deep reinforcement learning to simulate the impact of dynamic environmental stimuli on human cognitive processes. Unlike prior work, this paper considers human cognitive processes under time pressure. Experimental results across multiple tasks demonstrate that the proposed method outperforms baseline approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our comprehensive experiments and our contribution to more practical understanding of human cognition across different tasks and real-world datasets to fill the literature gap. We have made significant efforts including additional experiments and new analysis following your suggestions. Below, we provide a point-by-point response to each comment.\n\n**Claim**\n\nWe appreciate this insightful comment and acknowledge that our current work focuses primarily on three specific kinds of stimuli types in our three tasks. While they represent important aspects of cognitive processes, we agree that additional factors like emotional stress and cognitive load also contribute to real-world cognition. We will revise our claims to better reflect the scope of our study and explicitly state that while our framework effectively captures time pressure effects, extending it to broader cognitive influences is an important direction for future work. We have also discussed how our approach could be adapted to incorporate other dynamic stimuli in follow-up studies in section 6.\n\n**Method**\n\nThank you for this valuable suggestion of more efficient visual stimuli encoding. Using CNN to study human visual cognition and biological vision is commonly adopted in existing research [1]. This is why we use CNN to process visual stimuli. However, we agree that using a more efficient representation may further improve our model. Thus, we have adapted our model from encoding Type II (Task: String, Feedback: Video) to Type IV (numeric for task/feedback), i.e. using numeric values to represent time pressure stimuli. We find that this indeed improves our performance to be much better than existing performance (results hosted in https://github.com/cogreact/CogReact/). Please see further details in our response to point [Claim] for reviewer 3FF7 and point [Claim] for reviewer uxDz.\n\n**Theoretical Claims about Biases**\n\nWe appreciate this observation and recognize that dataset biases can impact model performance. We will explicitly discuss potential biases in our data and their possible effects on our findings. Additionally, we will include a section outlining steps for mitigating such biases in future iterations of our framework. It’s also worth noting that, during our user study data collection, we also have explicit instructions for participants to reduce biases. Please see details in point [Ethical Review Concerns] below.\n\n**Experiment**\n\nThank you for your recognition of our comprehensive experiments. To strengthen our study, we also had extra experiments to explore DL+DDM (please check our response to point [Experiment] for reviewer 3FF7).\n\n**Other Comments**\n\nThank you for pointing this out. We will correct this typo and carefully proofread the manuscript to address any additional language issues.\n\n**Questions for Authors:**\n\nThank you for the valuable suggestion. We have obtained experimental results using the more efficient visual stimuli encoding following your suggestions. Please see point [Method] above.\n\n**Ethical Review Concerns**: It is uncertain whether the participants are influenced by other factors when making decisions: \n\nTo reduce biases introduced by other factors, during the data collection of our user study, we have several explicit instructions to participants. First, we asked participants to find a whole experiment duration when they were not interrupted by other things during the study. Second, participants were asked to have a rest after each study session to avoid fatigue. Third, to deal with the speed-accuracy trade-off, we asked participants to first prioritize their accuracy and then respond as soon as possible. Our study is also approved by the IRB office in our local institution and we obtained written consents from participants before the study.\n\nOnce again, we extend our heartfelt gratitude for your thoughtful feedback and constructive suggestions. Your insights further help us refine our work, and we believe these improvements will significantly enhance the clarity and impact of our paper.\n\nReference:\n\n[1]. Grace W. Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future, 2021", "claims_and_evidence": "The paper claims to simulate the impact of dynamic environmental stimuli on human cognitive processes. While the use of time pressure stimuli in the math task provides valuable insights, real-world cognitive processes are influenced by a multitude of factors beyond time pressure. The current evidence mainly focuses on time pressure stimuli, and it's not clear if the framework can accurately capture the effects of other important factors like emotional stress, cognitive load from multiple concurrent tasks, or social influence. So, the claim may be over generalizing without sufficient evidence for a comprehensive representation of real-world cognitive processes.", "methods_and_evaluation_criteria": "This method uses CNN to directly process images as visual stimuli, which may be unreasonable because CNNs may struggle to understand the key information. The authors could consider whether a more efficient representation can be used.", "theoretical_claims": "The experimental methodology is technically sound, but biases in the data may still affect the results.", "experimental_designs_or_analyses": "Experiments are sufficient", "relation_to_broader_scientific_literature": "The paper's CogReact framework builds on traditional cognitive models like DDM and fills a gap in literature by focusing on dynamic environmental stimuli. It aligns with the trend of integrating machine learning in cognitive modeling, improving response time simulation. Its generalization across tasks and use of real-world datasets contribute to a more practical understanding of human cognition.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "Line 115: We focuses -> we focus", "questions_for_authors": "The authors could consider employing a more efficient representation during the model inference process, such as substituting the images that represent time pressure with a vector that encodes time information.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Discrimination / Bias / Fairness Concerns"], "ethical_review_concerns": "It is uncertain whether the participants are influenced by other factors when making decisions."}}, {"review_id": "lS2YawXbdf", "reviewer": "Reviewer_3FF7", "summary": "The paper proposed a novel method to model humans' response time under varying time pressure. The method combines the traditional model called Drift-Diffusion Model with Reinforcement learning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and your recognition of our suitable benchmark datasets and evaluation metrics. We made significant efforts with additional experiments/ analysis to address your concerns following your suggestions. Below, we provide a point-by-point response.\n\n**Claim**\n\nFor the reason of high standard deviation of MAPE, please see our response to point [Claim] for reviewer uxDz.\n\nTo address concerns of misleading significance by over-precision, we added statistical tests: Kolmogorov-Smirnov and Permutation tests, as they do not necessarily assume data normality. They were applied to all baselines in Table 4, with detailed results in Table 6 in our repo: https://github.com/cogreact/CogReact/. We found CogReact achieved significantly lower MAPE (p < 0.001) than most baselines, except for LSTM and Transformer in Type IV (numeric for task/feedback). \n\nThe close performance (only 0.005 difference) of vanilla LSTM/Transformer in Type IV is due to differences in task/feedback encoding, as CogReact originally used Type II (Task: String, Feedback: Video). To ensure a fair comparison, we adapted a new version of CogReact with numeric encoding (Type IV) as well, whose new MAPE (0.2703) is significantly lower (p < 0.001) than LSTM (0.3059) and Transformer (0.3052) (Table 5/Table 7). Additional statistical tests on two other datasets (Table 8/Table 9) further confirm CogReact’s significant improvements. All tables are in our GitHub repo.\n\n**Experiment**\n\nWe added experiments following your insightful suggestion to integrate DL models with DDM. Specifically, we used LSTM, MLP, and Transformer (Type IV) with the same hyperparameters as before but modified them to predict DDM parameters instead of response time. The final response time was then derived from these predicted parameters. Additionally, we introduced another MLP (MLPv2), which shares the same neural network architecture inside our Hybrid DRL model, to assess whether performance gains stem solely from DL+DDM integration. Other ML models like SVM were not used as they are not applicable to this experimental goal.  \n\nThis experiment was conducted on three datasets, with statistical tests applied as before. Results (Figure 11, in our github repo) show that while DL+DDM occasionally improves over standalone DL, performance remains significantly worse (p<0.001) than our Hybrid DRL model, underscoring the necessity of our full hybrid approach.\n\n**Other Weaknesses:**\n\n*Figure 1 description*: Please see the improved Figure 1 with more captions in github repo.\n\n*More backgrounds in methods*: We appreciate the valuable feedback. Some methods like DDM were described in introduction section / Appendix. We will organize our paper to move key details to improve the clarity.\n\n*Table 1 annotation*: Thank you for the insights. The numbers in Table 1 refer to mean and standard deviation of MAPE. We intend to use more decimals to deliver more precise results. Please see our improved Table 1 with more captions in github repo.\n\n**Question 1**:\n\nWe acknowledge the need for a clearer explanation. \n\n(Roseboom et al., 2019) aims to model human time perception when watching visual stimuli. To do so, it first uses a CNN model to solve visual stimuli recognition tasks and then uses SVM to connect CNN’s embeddings to humans’ time perception. \n\nBy contrast, our initial step in Section 3.3 aims to model human initial response time when solving math problems. To do so, we similarly first use the LSTM model to solve math problem tasks and then use SVM to connect the LSTM’s embeddings to humans’ response time.\n\nThe similarity lies in the high-level idea to first use a problem-solving model to solve cognitive tasks, and then use SVM to connect this model’s embeddings to humans’ cognitive behaviors.\n\nBut we do not intend to claim the similarity between CNN and LSTM. The selection of either CNN or LSTM depends on the cognitive task type. In (Roseboom et al., 2019), CNN is widely adopted for their visual cognition task in neuroscience [1]. However, for our math task, existing literature (Yang et al., 2019) [reference in line 162, page 3] proves that LSTM is proficient in such math problems. Therefore, we use LSTM instead of CNN.\n\n**Question 2**:\n\nThe risk agent (defined in line 374 right side, paper page 7) adapts the LSTM-based math agent to encode risk information for decision-making in the CPC18 dataset. Similarly, the LLM agent (line 389 left side, page 8) replaces the LSTM-based agent in past math tasks to better encode learning task information in the PeerEdu dataset.\n\nWe once again appreciate your valuable suggestions, which further help us improve the presentation, rigor, and overall significance of our paper. We are grateful for the opportunity to incorporate these improvements to contribute to the ICML community.\n\nReference:\n\n[1]. Grace W. Lindsay. Convolutional neural networks as a model of the visual system: Past, present, and future, 2021", "claims_and_evidence": "I have a serious concern that the claim is not correct.\n\nMentioned in the paper \"Results are depicted in Table. 1 and Table. 4, showing that\nCogReact has the best response time prediction performance\n(lowest MAPE) by comparing with other models in both the\nsame and different model input types.\"\n\nIf we look at Table 1, a vanilla transformer can reach the same performance (0.3052 with std of 0.2446) as CogReact (0.2999 with std 0.2318). The difference is only 0.005 with standard deviation of 0.2. It is misleading significance by over precision.", "methods_and_evaluation_criteria": "I have no concern regarding benchmark datasets and evaluation metrics.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The authors suggest applying Deep Reinforcement Learning (DRL) to predict Drift-Diffusion Model (DDM) parameters. However, I find the comparison in Figure 2 to be unfair. While pure DRL performs the worst, Hybrid DRL (which combines DDM) shows the best prediction. This leads me to believe the improvement is due to the Deep Learning (DL) and DDM integration, not necessarily the Reinforcement Learning (RL). To properly test the impact of RL, the authors should also try integrating other types of DL with DDM.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "There are some aspects of writing that can be improved\n\n\n1. Figure 1 does not have any description and it is difficult to understand purely based on the graphic.\n\n\n2. As the paper is submitted to ML venue, it should contain some methods that are not well-known among ML researchers, such as Drift-Diffusion Model (DDM). This will make the paper much easier to read by those who are not familiar with DDM.\n\n\n3. Table 1. Evaluation in the math reasoning task. What are these numbers? They are not included in the table caption. Moreover, is it necessary to report several decimals? Most of the values are ranged from 0.30-0.33 with standard deviation of 0.23-0.26, why do you not use only 2-3 decimals?", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. I cannot see the connection between your work and  (Roseboom et al., 2019). In Section 3.3, Similarly, our baseline prediction model employs an LSTM neural network to address cognitive tasks. Why is LSTM similar to CNN in (Roseboom et al., 2019)?\n\n2. What is risk/LLM? I have not seen it is defined somewhere in the manuscript.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a hybrid framework integrating drift-diffusion models (DDM) and deep reinforcement learning (DRL) to simulate human cognitive responses under dynamic stimuli, validated on a novel dataset of 21,157 logical reasoning trials. The introduced method is novel at the intersection of cognitive science and machine learning. While initial results showed high variance, post-rebuttal statistical analyses strengthened claims of superiority over baselines. Ethical concerns were adequately addressed through anonymized data practices, though broader societal impacts warrant continued caution. The manuscript received three positive reviews (including a strong acceptance) from four reviewers. The only concern of the weak reject reviewer is that the writing quality needs to be improved. Given the technical novelty, statistically validated improvements, and responsive revisions during rebuttal, I recommend ​​Acceptance​​. Besides, I also suggest the authors further refine the writing and figures to improve the clarity in the final version.
Decision: Accept (poster)