Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Authors: Chao Yang
Shuting Cui
Yang Yang
Shuang Li
Venue: ICML 2025 poster (2025)
Keywords: Logic-Informed Temporal Point Process
Amortized Variational EM
Human-AI Collaboration
Primary area: general_machine_learning

Abstract: Understanding human mental states—such as intentions and desires—is crucial for natural AI-human collaboration. However, this is challenging because human actions occur irregularly over time, and the underlying mental states that drive these actions are unobserved. To tackle this, we propose a novel framework that combines a logic-informed temporal point process (TPP) with amortized variational Expectation-Maximization (EM). Our key innovation is integrating logic rules as priors to guide the TPP’s intensity function, allowing the model to capture the interplay between actions and mental events while reducing dependence on large datasets. To handle the intractability of mental state inference, we introduce a discrete-time renewal process to approximate the posterior. By jointly optimizing model parameters, logic rules, and inference networks, our approach infers entire mental event sequences and adaptively predicts future actions. Experiments on both synthetic and real-world datasets show that our method outperforms existing approaches in accurately inferring mental states and predicting actions, demonstrating its effectiveness in modeling human cognitive processes.

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Chao Yang' Shuting Cui
Yang Yang'
Shuang Li
Abstract
Human behavior, though complex, often follows simple, generalizable logic. In Fig.l, preparing oatmeal involves sequential steps (e.g., taking cup, adding oats, and pour- ing milk) governed by underlying intentions like wanting oatmeal soaked in milk (Damen et al., 2018). Logic rules that define the mutual relationship between intentions and actions provide compact, interpretable representations of this knowledge. Using such rules, Al systems can better predict behavior and infer mental states.
Understanding human mental states-such in- tentions and desires-is crucial for natural AI- human collaboration. However, this is challeng- ing because human actions occur irregularly over time, and the underlying mental states that drive these actions are unobserved. To tackle this, we propose novel framework that combines logic- informed temporal point process (TPP) with amor- tized variational
Anpectaaion-Maxiiizate Our key innovation is integrating logic rules as pri- ors to guide the TPP's intensity function, allowing the model to capture the interplay between actions and mental events while reducing dependence on large datasets. To handle the intractability of men- tal state inference, we introduce a discrete-time renewal process to approximate the posterior. By jointly optimizing model parameters, logic rules, and inference networks, our approach infers entire mental event sequences and adaptively predicts future actions. Experiments on both synthetic and real-world datasets show that our method outper- forms existing approaches in accurately inferring mental states and predicting actions, demonstrat- ing its effectiveness in modeling human cognitive processes.
(HM).
Prior work has focused on forecasting actions from observed sequences (Abu Farha et al., 2018; Cramer et al., 2021; Darvish et al., 2020), but critical challenges remain. First, actions and mental states occur irregularly in time: inten- tions shift unexpectedly (e.g., abandoning task due to distraction), and the time intervals between actions encode contextual cues about these shifts. Ignoring temporal irreg- ularity risks misinterpreting intentions, delays or abrupt transitions often signal evolving goals (Hu & Clune, 2023; Zolotas & Demiris, 2022). Second, mental states are dy- namic and interdependent: past actions influence intention formation, while intentions guide future behavior. Inferring the entire sequence of mental events-when and what inten- tions occur occur-requires requires modeling these mutual dependencies with precise temporal granularity. Traditional methods strug- gle to disentangle irregular, intertwined processes, leading to oversimplified or stochastic predictions.
To address these challenges, we propose novel framework that unifies logic-informed temporal point processes (TPPs) and amortized variational inference model the bidirec- tional dynamics between irregular human actions and latent mental states. Our approach jointly learns to infer inten- tions, predict future actions, and discover interpretable logic rules-all within a single, cohesive learning paradigm.
1. Introduction
Rapid advancement of Al has spurred interest in developing autonomous agents that collaborate with humans in tasks such as healthcare, education, and robotics (Carroll et al., 2019; Puig et al., 2020; Strouse et al., 2021). Effective col- laboration requires AI agents to understand human actions and infer latent intentions mental states that drive behav- ior but remain unobserved. This capability is essential for enabling agents provide timely, context-aware assistance in dynamic environments.
Central to our framework a logic-informed TPP, where do- main knowledge is encoded probabilistic logic rules that shape the intensity functions governing actions and mental events. These rules (Li al., 2020; Mei et al., 2020), act as priors to constrain the learning space, reducing reliance on vast datasets while preserving interpretability. Crucially, our model captures two-way interactions: past actions prob- abilistically trigger or reset intentions (e.g., prolonged inac- tivity may signal a shift in goals), while inferred intentions guide the likelihood and timing of future actions.
'School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, China. Correspondence to: Shuang Li <lishuang @ www.eeccc..c.c.eeeeeeeeeeeeeeeee edu.en>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

H((T))dT, Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Time: 0m2s action event
Time: 0m33s
Time: 0m35s action event
Time: 1m27s
Time: 1m32s Time: 1m58s Time: 2m10s action event action event action event
Time: 2m21s mental event
Time: 2m46s action event
mental event
action event
Timeline
Walk into the Kitchen
patmeal soaked
cup
Take the oatmeal Pour some oats
Take the milk
Pour the milk
Want keep the table clean
Put the table mat
Figure 1. An illustrative example of complex interaction between intentions and action events: at 0m33s, the individual forms the intention have oatmeal with milk, and 2m21s, another intention arises, focused keeping the table clean. These intentions are driven by past actions and consequently result in a sequence of future actions.
To address the challenge of latent mental states, we develop an amortized variational EM algorithm that alternates be- tween the following steps:
cesses of action and mental events, while DT-RPs will be introduced to approximate latent mental state posteriors.
2.1. Temporal Point Processes (TPPs)
E-step: Refining an amortized inference network to ap- proximate the posterior distribution of mental states. The discrete-time renewal
Intensity Function TPPs model the occurrence of events in continuous time through intensity functions that encode dependencies on historical context. Let X' denote event types (c.g., actions like adding oats or mental states like cooking intent). Each event 6 := (t,, (z)) occurs time t; with type æi X, and the history up time & is H(I) {ey t2 < t). The conditional intensity A₂ (t H(t)) quan- tifies the instantaneous rate of event at time t
inference network is modeled process (DT-RP) with neural hazard functions, effectively capturing intention resetting and long-term dependencies.
Optimizing continuous TPP parameters and dis- crete logic rules. The logic rule set evolves dynamically dur- ing training: predefined rules bootstrap learning, while an automated column generation strategy (Barnhart et al., 1998; Li et al., 2021) expands the rule space with data-driven pat- terns (e.g., discovering new intention-action correlations).
M-step:
IP(x occurs in f + At) H(t))
A H(t)) = lim Ar-50
At
The integration of symbolic priors with neural inference allows our model to adapt to individual behavioral nuances while maintaining explainability. Once trained, the model predicts future actions by inferring mental events and sam- pling action timing/type. A backtracking mechanism is used to ensure temporal consistency by checking for mental state shifts between observed and predicted actions, aligning predictions with inferred intentions.
A classic example is the Hawkes process (Hawkes, 1971; Zuo et al., 2020), which models self- and mutual-excitation between event types. Its intensity for type is:
Ae H((t) = 1zz +
Σ az.zc Pa,y
by (H(((()
where Pa is the base rate, Orr,z defines how strongly a past event ej (of type æry) excites future events of type x, and Br,,, controls the decay of this influence.
Our contributions can be summarized as follows: (i) We propose a logic-informed TPP framework that captures irreg- ularly spaced actions and latent mental states. By integrating logic rules as symbolic priors, our approach enhances data efficiency and ensures model interpretability. (ii) We intro- duce a variational EM algorithm with a DT-RP for efficient and scalable inference of latent mental event sequences. The M-step jointly optimizes model parameters and the logic rule set, leveraging automated rule discovery to reveal novel and overlooked intention-aee action patterns. (iti) Extensive experiments demonstrate the effectiveness of our method in inferring complex mental event sequences and accurately predicting future actions, advancing human-centric AI.
Learning and Sampling For observed events e = {cI...., ..N } over [0, T), the log-likelihood is:
log logpo(ee = log da, (t. 1 - 1.' {
xEX
(1)
where 0 includes TPP parameters (e.g., Hax Ola,ary 32,2, for Hawkes process). By maximizing this likelihood, we learn the generative dynamics of the events encoded in Sam- pling events in TPPs often requires iterative methods like thinning algorithm (Ogata, 1981; Rasmussen, 2018), 8), where candidate event times are proposed and accepted/rejected based on the intensity function.
2. Preliminaries
We present key foundations for modeling event sequences. TPPs will be used to model the intertwined generative pro-
Modeling Actions and Mental States While TPPs like neural Hawkes processes capture complex triggering pat- terns, their dense parameterization poses challenges. The
2

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
pairwise parameters opaque, making it hard explain how actions mental states influence each other. Learn- ing all pairwise interactions requires large datasets, which are scarce in behavioral domains. Moreover, these models cannot easily represent conditional triggering logic (e.g-, "if A and B occur, trigger C"), which is essential for cap- turing complex mental state-behavior relationships. To ad- dress these limitations, we integrate logic rules as inductive biases, grounding the model in domain knowledge. For example, rule like "cook meal + hunger A available ingredients" explicitly links desire to actions, pruning irrele- vant connections (c.g., setting Qs.æ, = O for irrelevant pairs) and reducing need for dense parameter learning. Crucially, these rules can be predefined or learned in our framework, adapting to new scenarios and improving generalization. For more details, see Sec. 3.3.
where Og (A: 1 indicates that an event æ occurs in the k-thh interval and oz (k) 0 otherwise. This grid-wise factoriza- tion enables efficient training by decomposing likelihoods across time and event types.
Efficient Computation and Sampling We use DT-RPs to approximate mental state posteriors because they allow for efficient sampling and likelihood evaluation. Unlike TPPs, which rely on costly iterative thinning and numerical inte- gration, DT DT-RPs divide time into intervals. This approach enables fast sampling through cumulative hazards and cal- culates event probabilities separately for each interval. By prioritizing simplicity and speed over continuous-time pre- cision, DT-RPs align well with the amortized variational EM framework's 's need for rapid latent event sampling and repeated likelihood evaluations.
2.2. Discrete-Tine Time Renewal Processes (DT-RPs)
3. Amortized Variational EM
In our method, we propose using DT-RPs to approximate the intractable posterior distribution of latent mental processes, for ease of implementation.
Human decision making is inherently shaped by complex cognitive processes that remain largely unobservable. We propose an amortized variational EM framework built upon event data models. This framework, visualized in Fig. 2, can learn coupled generative processes of actions and mental events, infer latent mental events and predict future actions. Consider the following action and mental events:
Discretization and Hazard Rates Time horizon T is di- vided into K" = [T/At] grids V2- = ((k - 1)At, k For each event type x (e.g., mental states), the hazard function h. (k) E [o. 1] indicates the probability of x occurring in Vk given no x-events since last occurrence at grid ko. i.e.,
Action: Observed sequence a = x E A (e.g., "boil water").
No where
h.p (k) P (ar in V& No x in Vi... VE-1).
Nm where Mental: Latent sequence {(tT """""""")) )
Recall that in TPPs, the hazard rate corresponds to the instan- tancous intensity finction, which can exceed I (Rasmussen, 2018). In contrast, in DT-RPs, the hazard represents the con- ditional probability of an event occurring within discrete time interval and thus restricted to values between 0 and 1. Correspondingly, the survival probability S,(k),, indicating likelihood of no xeevents from ku + to k, is computed as
x! E M (e.g., "cooking intenf")
We use the logic-informed TPPs (TLPP) to model the in- terleaved generative processes between action and mental events, parameterized by po(a, m)), where contains both continuous logic model parameters and the discrete rule set, Details of the logic-informed model is provided in Sec. 3.1. To approximate the intractable posterior for latent mental states po (m a), we use DT-RP as the variational distri- bution (m a). The parameters & are amortized across sequences via a neural encoder:
k Sk) = II (1-hx(T)) +1
(3)
From the definition, each event x "restarts its own clock" upon occurrence, resetting the survival function to 1 by updating ko.
Ls (m (m | a) = NN₂(a).
(5)
to extract beddings from a. dynamically shaping DT-RP's hazard. Details of this neural hazard model are provided in Sec. 3.2.
where NN, & employs
Learning and Sampling Given the hazard function hz),), we sample the next x-event according to Alg. 2 in Appendix. A. using inverse transform sampling method. With a hazard model parametrized by o, the likelihood for binary event indicators o(k) E {0, 1)1x1 over K grids is
Our goal is to optimize both (m a) that enables latent mental event inference and ps (a, m)) that enables rule min- ing and events dynamic learning, through Evidence Lower Bound (ELBO), which serves as the objective for training:
(0) = II (K)) (1 h1 (k)) (R)
(4)
C(O, oi a) = Egg [log pe (p((e (, m)] - Eg [log (m | a)]
A= 1 X
Eago [log Po (a, m)) + HI [gs (m a)). (6)
3

---

Evolving Minds: Lngic-Informed Inference from Temporal Action Patterns
E-Step (Training Encoder)
Hazard Function Modeling
History
Future
Predicted Trajectory
f
Encodler
Cross-Attemtion
Haxard
Sampling
Accept during
backtracking
Montal
Probiom
Sul-
Prior Knowledge
Masier- Problem
Decoder
Query
Value
Key
Time Gnd
Sampled Latent
Learned Rulos
Time Embed
Type Embedd
Embed
Time
Extract
Rule f1
Rule f2
W1
W2
[4
Precection with Bockkracckng
Rule fs
W's
Observed Event
Biscrete Time Horizon
Observed Action Trijectory
M-Step (Training Decoder)
Figure 2. Model architecture. E-Step: freezing 0 to optimize ø for training encoder. M-Step: freezing e to optimize for training decoder. Prediction: the prediction with backtracking mechanism is presented in dashed box.
3.1. Decoder (Logic-Informed TPP pa(a, m))
where Tlatt is the most recent event time the combination, and 8 >> O (can either be pre-defined or learned) controls how quickly the rule's influence diminishes. The intensity for event type combines base rates and rule contributions:
The decoder models joint dynamics of action and mental events, which integrates domain-specific temporal logic rules capture the structured dependencies between events.
Ax H((t) = Me + Σ wy
Event Representation and Temporal Logic action and mental event types into a single set X' = AUM, where x E X' is represented as a Boolean variable:
We unify
JEF.
where Me is the base rate, Wy the rule weight, and Fz are rules targeting The model parameters 0 = [Moleex [Wy] include continuous base rates [1z [ and rule weights [ as well as discrete rule set Uzex (will also be refined during training).
{
1
if an event of type occurs at time t
((1)
otherwise
For observed actions a and inferred mental states m, these variables are grounded as x= (4?) and x (LTT).
For observed actions a and inferred mental states m, the joint likelihood (according to Eq. (1)) is:
Domain knowledge is encoded temporal logic rules F, which define structured dependencies between events. Each horn rule (i.e., "if-then" rule) f E F follows:
log po (a, m)
Σ Xay (t H((t)) + Σ Auy H(t""))
St Σ de(T H(T)dd
f Thead
A
A R
Rí (x" se")
Z"EX,
REALM
Body conditions
3.2. Encoder (Amortized Posterior qo (m(m) a))
where the head of the rule is the target event Phead E X influenced by the rule, the body conditions are the log- ical conjunction of event occurrences (xx) and tempo- ral relations (Rz). The temporal relations are constraints like Before (x", After (x" xx", or Equal (x",x"), grounded in event timestamps.
The encoder leverages DT-RPs to approximate posterior of latent mental states, combining sampling efficiency and tractable likelihoods for scalable variational inference. Its novelty lies in its neural hazard architecture, dynamically synthesizing action context, mental history, and temporal dynamics to parameterize hazard functions. Our encoder builds the Neural Hazurd Architecture by integrating the following critical components:
Dynamic Rule Influence with Decay The influence of past events current intensities decays time. For each rule f. the feature (I))) counts valid event combina- tions in history H(t) that satisfy the rule's body, weighted by an exponential decay:
Action Context via Cross-Attention Observed actions x/)} are encoded into timeline-aligned embed-
a = {(t dings using attention mechanism. Each action embedded as x, = u, + zabs (tf)). where u, is learnable type embed- ding and Zahs encodes absolute timestamps. Discrete time
Φy ()((((t))
Σ e (By(t-
valid comb

---

Evolving Minds:
Logic-Informed Inference from Temporal Action Patterns
grids Vi= ((k- kA]] are similarly embedded as 4 Zuhs (k A cross-attention layer aligns grids with historical actions:
strategies for different scenarios, including autonomous rule learning and the use of predefined rules. We use col- umn generation algorithm which maximizes the expected complete-date log-likelihood over sampled mental states m ~ 9o (m a), computed during the M-step. The algo- rithm alternates between optimizing continuous parameters
S = Softmax (QKT/V) V.
where Q L K,V XW* XW
i 0= [eeleex [WI] (Master Problem) and refines the temporal logic rule set F (Sub-Problem).
where L = 11...... 4 and X .,,,..,,;,. 2..... represent stacked timeline queries and stacked action keys/values, respectively. The output S& E TR. captures entire action context relevant to grid Vi, ensuring temporal alignment critical for inferring mental states.
Master Problem (Continuous Parameter Optimization) Given samples {m )== q0(++1) a) and the cur- rent rule set F3 optimize the continuous parameters 0 as
Hazard Function Design The hazard rate ht for men- tal event type at grid & is constructed as:
8 log po (a, (a.ml))) - 2t ((W)))EE))
0j arg max
S
sml
(k:) a( Se (k)
+
Us (k typo-spocific timing
(9)
Expected log likelihood
typo-specific emission
s.t. wy 2 (OVVEE))
(11)
where fo R x R + R is tppe-specific final layer for each 9x N - R is type-specific MLP modeling elapsed time k - ki since last z-event, mh E I is mental history state (shared across types), updated autoregressively, and (.) is sigmoid function, constraining (k) E (0, 1).
where the log-likelihood for a sample (a, ml()) is computed as in Eq. (7), (()) is a differentiable rule complexity regu- larization term, such as &EYEF, w3. This regularization encourages simpler rules while preventing overfitting.
Sub-Problem (Rule Proposal via Sampled Gradients) New rules are proposed by identifying those with the most negative expected reduced cost, computed over samples {m(*1) s=1
Autoregressive Updates After sampling o(k) {0, MMI via Alg. 2, the mental history updates as
E
7++1 = RNN (Mk o(k))
AF= US
Reduced Cost(f) < 0}
resetting Kn to k if (k) 1. This enables dynamic, context-sensitive hazard modeling.
where the reduced cost for rule f is defined as:
1
8 log pa
8Ω
This design of hazard function balances parameter efficiency (shared initial layers across event types) with expressivity (type-specific final layers for emissions), which is critical for modeling heterogeneous mental processes. The hazard parameter @ is learned via E-step, where the entropy term is explicitly defined using hazard-based likelihood, i.e.
ReducedCost (f)
+
s Owy
Bws 68
mx)
is derived from the sam- pled mental events m ( Among all candidate rules, the one with the most negative reduced cost (i.e., arg miny ReducedCost(f)) is prioritized for addition to Fj and obtain Fz+1- This ensures that the proposed rules maximally improve the expected log-likelihood while ac- counting for regularization. The algorithm for the column generation is presented in Alg. 3, Appendix. A.2.
The gradient Blgg chey
K
H (m a)] Jam ZE.M
Σ
[he (k) log (k))
+ (1 - 12 (1-h (k)) log (1 h (k))].
(10)
where hr (k) is parameterized as in Eq. (9). This follows because each term in 4a (0) is a Bernoulli distribution with parameter 1e (k) as in Eq. (4). Sampling m ~ (m a) can be efficiently implemented via Alg. 2, which leverages inverse transform sampling over discrete grids.
3.4. Learning Procedure
Now building blocks are prepared, we employ amortized variational EM algorithm that alternates between E-step and M-step. with algorithm shown in Alg, 4. Appendix. A.3.
3.3. Adaptive Rule Learning via Column Generation
E-Step: Amortized Variational Inference Update the inference network & to infer mental event trajectories from actions:
To accommodate various applications, especially those lack- ing prior knowledge datasets, we introduce an adaptive rule learning module. It allows us to implement tailored
d (e+1) arg max Eng Plog Poco (a, m)) + H (gs (m|a)]
5

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
The gradient of the expectation term is estimated via the REINFORCE estimator with baseline (e.g., empiri- cal average) to reduce variance, Vollng [log pa m)] x Bgg (V & log (m a)- (log (o(a, m) c)]- The en- tropy term H [mm) a has a closed-form expression for DT-RPs as in Eq. (10), ensuring efficient optimization via SGD.
Algorithm 1 Predicting Actions with Backtracking
1: Step 1: Infer Mental States
Given historical actions a, sample mental events m
qo(m a) over discrete time grids.
3: Define the augmented history: H((t) = a m.
4: Step 2: Propose Candidate Action
5: Generate a candidate action time and associated value:
"next - ~ po H(t)).
Logic Model Parameter Update Maximize the expected complete-data log-likelihood with respect to 0:
M-Step:
6: Step 3: Check for Mental Events
7: For the interval [Must: mext); forward-sample mental
g((+1) = arg max EE 4 (++1) (m/a) [log Po (a, m)]
events in discrete time:
8: if mental event occurs at tthh [Musts "ne
9:
Update the history by integrating the mental event at ttm.
The expectation is approximated by sampling mental event trajectories m (0 a).
10: - Sett flast = tt
11:
- Reegenerate thea: T ~ po(- H((tT)).
Comparison with VAES Unlike standard VAEs, which leverage the reparameterizatioo trick for joint, differentiable updates of 0 (decoder) and Φ (encoder) (Kingma, 2013; Mehrasa et al., 2019), our framework employs staged E-M steps, alternating between freezing 0 to optimize & (E-step) and freezing ø to optimize 9 (M-step). This modular ap- proach mimics classical EM, reducing instability from com- peting parameter updates while accommodating rule-based refinement via column generation non- con-differentiable pro- cess incompatible with VAE-style gradients. To ensure sta- bility, we run the E-step to near-convergence (ELBO change < e) before each M-step, anchoring updates in reliable pos- terior estimates.
12: Step Iterate with Maximum Rounds
13: Repeat Steps 3 until no new mental events occur in [Mast *mex))
14: Step 5: Accept MexT next as Final Predicted Action
actions average. For real-world datasets, select four interesting datasets that capture human behaviors, which are highly likely to be driven by human mental states: i) Hand- Me-That (Wan et al., 2022): We focus on the change-stare type episodes and extract 503 sequences with average action trajectory length 30.5. ii) Car-following (Li et al., 2023): We extract 2000 car-following behavior sequences with 3.6 average action events and average time horizon 19.44s. tii) MuliTHUMOS (Yeung et al., 2018): We focus only on the basketball dataset with 2000 sequences. The time horizon of each sequence is 208.32s with 38.41 actions on average. iv) MIIC-Kitchen--A 100 (Damen et al., 2018): We focus on two goals: cut onion and pour water, and extract 131 se- quences contains related key actions. The time horizon of each sequence is 500s with 5.41 actions on average. For all the datasets, we split each dataset into 80%, 10%, 10% train/dev/test by the total sequences. Detailed introduction about datasets can be found in Appendix. B
4. Future Action Prediction
To predict future actions, we condition on both observed actions and inferred mental states sampled from the DT-RP posterior. The backtracking mechanism (as shown in Alg. 1) ensures temporal coherence by dynamically check- ing for mental state shifts between the last observed action and the predicted next action. If mental event ge safety- check intent) occurs in this interval, the sampler regenerates predictions using the updated mental state, aligning actions with the most recent intent.
During the training process, only the action trajectories are given. For synthetic datasets, the ground truth mental events are known, allowing us compare the inferred mental events with the ground truth. In real-world datasets, the ground truth mental events are hidden, and thus we cannot directly compare the accuracy of sampled mental events, but resort to compare the accuracy of action predictions.
5. Experiments
5.1. Experimental Setup
Datasets We conduct our experiments on synthetic and real-world datasets. For synthetic datasets, we simulate two datasets with same sample size (2000 sequences) and same time horizon (15s), but with different number of predicates and ground truth logic rules: i) Syn Data-1: 3 ground truth rules, I mental predicates and 2 action predicates. Each se- quence has 18.60 actions on average, ii) Syn Data-2: a more complicated scenario with 4 ground truth rules, 2 mental predicates and 2 action predicates. Each sequence has 13.25
Baselines We choose state-of-the-art baselines consider- ing three different fields: i) Neural Temporal Point Process Model: RMTPP (Du et al., 2016), THP (Zuo et al., 2020), PromptTPP (Xue et al., 2023), and HYPRO (Xue et al., 2022), ii) Logic-Bases Based Model: TELLER (Li et al., 2021)
6

---

Evolving Minds: Ingic-Informed Inference from Temporal Action Patterns
Fitted hazard TruwSammpeed mental events
parned Rule Content
Ruie Parems
Rule-1:
m1 < a1 before m
0.8
0.4
Sampled
Rule-2: m2 a1 A ag
0.7
d1 before d2
0.6
02 before m2
0.4
3u a1 M2 A a2
U2 before m2,
0.-
M2 before a1
0.2
01
Rule-4: a2
m,, M2 before a,
0.0
0.0
113
Figure 3. Results Syn Data-2. Left: fitted hazards of mental events for one sequence, corresponding sampled mental events, and ground truth mental events. Middle: results learned rules. Green stars indicate that the rules are correctly learne.. Right: ground truth rule parameters and learned parameters.
Syn Data-1
Syn Data-2
Hand-Me Me-That
Category
Model
ER% +
MAE $
ER% |
MAE 4
ER% 4
MAE 4
RMTPP
48.37 + 1.47
3.11 + 0.09
52.14 + 1.55
3.20 = 0.07
79.86 = 2.12
2.13 + 0.10
Neural
THP
45.46 + 1.32
2.83 0.08
48.93 + 1.29
2.99 = 0.07
78.85 + 1.85
2.08 + 0.12
TPP
PromptTPP
$3.47
0.06
47.51 = 1.75
=.17
76.35 2.06
1.68 + 0.08
HYPRO
44.53 1.57
2.46
47.85 + 1.62
2.60 +000
75.88 + 2.22
1.70 0.08
TELLER
N6.72:18886
0.15
49.15 2.23
3.04
78.28 2.90
1.86
Rule-based Model
CLNN
46.25 0.12
48.32 = 1.74
2.13
77.74 1.95
1.82 + 0.07
STLR
45.05
2.12
48.23 + 1.83
2.72
77.25
1.81
1.80 0.09
AVAE
0.93
2.08
47.53 = 1.62
2.12
80.12 +
2.10 0.12
Gen.
GNTPP
47.22 1.84
2.97 0.17
51.86 + 2.08
+.19
85.38 3.02
2.69 + 0.12
Model
VEPP
47.58 + 1.58
0.04
52.02 + 1.52
3.22 = 0.13
83.32
2.51 + 0.11
STVAE
46.81 + 1.63
2.76
0.07
49.27
+ 1.76
3.02 1 0.10
79.12
2.56
2.17 + 0.09
Ours
41.72 +: 1.45
2.32 + 0.09
46.85
+ 1.56
2.52
0.15
75.28 + 2.12
1.26 + 0.07
Table Comparison between model and baselines all synthetic datasets and Hand-Me-Thal datasets for prediction tasks. Bold text represents the best result and underline denotes the second-best result. 'he performance is averaged over three different seeds and the standard deviation stored after "4". Results from our model shaded in red.
CLNN (Yan et al., 2023), STLR (Cao et al., 2024), tii) Gen- erative Model: AVAE (Mehrasa al., 2019), GNTPP (Lin et al., 2022), VEPP (Pan et al., 2020), and STVAE (Wang et al., 2023). For PromptTPP and HYPRO, we choose At- INHP (Yang et al., 2021) as their base model. For GNTPP, we choose the revised attentive history encoder and VAE probabilistic decoder. Details are in Appendix. C.
3.8
3.6
3.4
3.0
2.6
Figure 4. Performance of all the methods on predicting future 3 actions for Syn 2. Left: Comparison of event type average emorr rate ER%. Right: Comparison of event time average MAE.
Comparison Metric Following common next-event pre- diction task in TPPs (Du al., 2016; Zuo et al., 2020; Yang al., 2023), our model as well other baselines attempt to predict the next event from history. Moreover, autoregres- sively predicting multiple future events is also considered in our experiments. We evaluate the event type prediction with the Error Rate (ER%) and evaluate the event time prediction with the Mean Absolute Error (MAE).
5.2. Experiments on Synthetic Dataset
Infer Latent Mental Events and Learn Rule Parameters Results in Fig. 3 demonstrate a general alignment between the locations of fitted high-hazard and the actual occur- rences of mental events. The mental events sampled based on these hazards also correspond reasonably well to the
7

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Car-Follow
MuliTHUMOS
Pitcheitchen
Category
Model
ER% +
MAE 4
ER% +
MAE 4
ER% 4
MAE 4
RMTPP
35.71 + 1.33
2.64 + 0.08
67.01 = 2.64
8.72 = 0.36
49.02 = 2.38
41,17 + 2.62
Neural
THP
33.43 + 1.62
2.31 = 0.08
62.32 + 2.34
7.12 = 0.28
42.19 = 2.36
37.13 - 2.30
TPP
PromprTPP
33.29 + 0.87
2.11+0.00 0.05
60.35 + 1.92
7.00 + 0.24
40.82 + 1.80
33.21 + 1.78
HYPRO
32.86 + 1.70
2.03
58.25 2.42
6.98 = 0.37
42.28 2.02
35.98 = 1.97
TELLER
37.83 2.96
3.41
64.77 2.53
7.52 =
43.49 = 2,35
38.05 = 3.02
Rule-based
CLNN
37.09 2.20
3.25 0.18
63.10 + 2.28
7.33 + 0.34
42.86 2.06
37.13 + 1.89
Model
STLR
32.75 + 1.90
2.12
63.38 : 2.58
7.69 - 0.32
43.37 2.12
36.85 = 1.79
AVAE
35.08 + 1.52
2.95 0.13
61.17 + 2.09
8.32 0.42
43.56 + 2.00
39.24 + 2.18
Gen.
GNTPP
39.22 + 1.77
3.89 + 0.06
63.75 + 2.30
8.37 + 0.43
46.25 + 2.17
38.11 + 2.25
Model
VEPP
40.25 2.39
3.78 - 0.16
64.23 = 2.63
8.42 = 0.50
47.56 + 2.42
38.93 - 2.03
STVAE
37.23 + 1.78
3.18 - 0.17
64.28 = 2.64
8.24 = 0.35
45.83 = 2.09
37.48 + 2.15
Ours
32.72 + 2.90
1.80 + 0.18
57.20 = 2.32
6.76 = 0.45
40.26 = 2.20
32.19 + 2.24
Table 2. Comparison between our model and baselines on Car-Follow, MuldTTHMMS,, and EPPIC-Kicheer datasets for prediction tasks.
actual time points of occurrence. As we infer latent mental events through probability- based sampling, there is a certain degree of error involved. However, this error remains within an acceptable range. Our rule learning module correctly un- covers most ground truth rules and accurately learns the rule parameters. In particular, our model effectively addresses the challenge of limited data by incorporating logic rules as guidings. Even with a dataset size of only 2000 samples, it achieves promising results.
MOS dataset, we aim to predict next 3 actions. The experi- mental results, presented in Tab. 1 and Tab. 2, demonstrate that our model performs exceptionally well in predicting both future event types and timings, outperforming all other methods. In practice, thanks to the inherent capabilities of our rule learning module, we can employ two strategies tailored to distinct scenarios: autonomous rule learning for learning for scenar-
data-rich domains and template-guics ios with limited data but ample prior knowledge. In our ex- periments, we provide prior rule templates to enhance model performance as in Appendix. B. The rule learning module can refine these templates and learn rules from scratch. In- deed, our model uncovers overlooked rules with real-world significance, as in Appendix. D.4. In the Appendix. D.2, we also attempt remove these prior knowledge. Despite this, our model still achieved satisfactory results in pre- diction tasks, showcasing its adaptability when applied to real-world datasets lacking prior knowledge.
Next Single Event Prediction The experiments on two synthetic datasets to predict the next single future action events are presented in Tab. I, from which one can observe that our model outperforms all the baselines.
Next Multiple Events Prediction Auto-regressive long- horizon prediction may cause cascuding error in TPP (Xue et al., 2022). The HYPRO method considered in the base- lines is purely data-driven, which is flexible neural-based model combined with expressive energy- -based models. In contrast, our method employs a neural black-box encoder, but with a rule-based white-box decoder. The prediction results rely on the learned logic rules. Our design choice in- herently trades off interpretability for model expressiveness. Encouragingly, with enhanced interpretability, our model also mitigates cascading error. Shown in Fig. 4, in task of predicting next 3 actions, our model achieves comparable and even lower ER% and MAE than HYPRO.
Prediction Examples As exemplified in Fig. 9 for Hand- Me-Thal dataset and Fig. 10 for Car-Follow with analysis in Appendix. D.5, our model demonstrates intriguing ap- plicability in real-life scenarios. Given our model's ' precise inference of historical human intentions and accurate fore- casting of future human actions with corresponding time indexes, Al agents can promptly assist humans, significantly enhancing convenience.
5.4. Scalability, Ablation Study, and Hyper-Parameter
5.3. Experiments on Real-World Dataset
The scalability experiments on synthetic datasets shown in Fig. 5 validate that our model has potential to apply effec- tively to large-scale datasets. On real-world datasets, as dis- cussed in Appendix. D.I, our model also demonstrates good scalability. The results of the ablation study on synthetic datasets are shown in Tab. 3, with detailed analysis can be found in Appendix. D.2, confirming that appropriate prior
Experiment Results On real-world datasets, we have de- signed prediction tasks that are specifically tailored to the characteristics of each dataset, taking into account variations in the number of future events to be predicted. For Hand-Me- 100 datasets, we
That, Car- far-Following and focus on predicting the next action, whereas for MuliTHU-
8

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Time Cost v.s. Samples
ER% v.5. Samples
MAE v.5 Semples
57.5
Ruies
Rules
2 Rules Rules Rules Rulcs
Ruies
Rules
3,0
525
2.8
47.
24-
42.
40.
37.5
1000
1.500
2000
2500
30000
1500
6000
6500
5000
1500
1500
2000
2500
3000
3500
4200
5000
1000 1500
2000
3000
3500
4200
4500
5000
Samples
Samp
Samples
Figure 5. Scalability and the computation time cost of our model on synthetic datasets with varying sample size, ground truth temporal
logic rules, and latent state domains. All the experiments are conducted over three random runs and the standard error is reflected in the shaded areas.
knowledge enhances the accuracy of model predictions, and the inclusion of backtracking indeed improves prediction accuracy, even when there exists noise in prior knowledge. Based on empirical results, our hyper-yapanee parameter selection, including time grid resolution and maximum number of backtracking rounds, strikes a balance between model per- formance and training efficiency, with experimental results reported in the Appendix. E.I.
ies through intensity of TPP. The follow-up work (Li et al., 2021) employed column generation for data-driven rule min- ing but faces scalability issues. Kuang et al. (2024) utilized EM algorithm to optimize the rule set in a differential man- ner. And the works of (Yan et al., 2023) and (Yang et al., 2024) developed a differentiable neuro- syurolyymmolic framework for modeling TPPs. These works improve efficiency but overlook the importance of prior knowledge. Zhang et al. (2021) incorporated neural TPP with temporal logic prior knowledge, achieving good performance. In our paper, we customized iigicinnfrmme TPPs to model actions and men- tal events in continuous time. Using logic rules as priors, we enhance the inference of mental states and improve future action prediction, even with limited data.
Ablation Settings
Syn Data-1
Syn Data-2
Prior
Back
ER(%)
MAE
ER(%)
MAE
X
x
48.28
3.13
52.50
3.24
/
X
45.64
2.64
48.37
2.86
X
/
41.72
2.32
46.85
2.52
/
/
42.08
2.19
46.43
2.47
Table Ablation study on synthetic datasets. Our current selec- tion of modules are highlighted in blue. "Prior" stands for prior knowledge and "Back" stands for backtracking mechanism
Latent Mental Inference Considering human intentions as latent variables, Wei al. (2017) developed an EM-based approach for their inference but only model intentions as discrete latent variables. Mehrasa et al. (2019) and Zolotas & Demiris (2022) leveraged the VAE architecture to synthe- size human trajectories, albeit without addressing the inter- pretability the latent variables. Hidden Markov Models (HMMs) have also been employed for modeling sequential data and inferring hidden states. Jeong et al. (2021) inte- grated HMMs with VAEs to infer human activity sequences. The study by Cao et al. (2024) used learned spatial-tempone rules and human intentions to guide actions, most similar to our work. However, it overlooks the time interval informa- tion between actions and mental events, where the elapsed time also carries valuable contextual insights.
6. Related Work
Neural Temporal Point Process (TPP) Recent advance- ments in TPP field primarily concentrate on enhancing the flexibility of intensity functions. Some approaches utilized RNN and LSTM networks, such as (Du et al., 2016; Mei & Eisner, 2017; Xiao al., 2017; Mei et al., 2020). Others leveraged Transformer architectures, including (Zuo et al., 2020; Zhang et al.. 2020; Zhu et al., 2021; Yang et al., 2021). Despite these advancements, the reliance on black- box models raises interpretability issues, particularly in contexts requiring explanations for events. Recently, Xue al. (2022) combined the base TPP with an energy function to overcome the cascading errors of predictions, which is also what our model aims to avoid.
7. Conclusion
We propose novel framework combining logic-informed TPPs with amortized variational EM to jointly infer latent mental states, learn the model parameters, and predict future actions. Guided by temporal logic rules, our model performs well even in small data scenarios and enhance interpretabil- ity. The introduction of the backtracking mechanism further improves the stability of our model prediction tasks.
Logic-Informed Temporal Point Process Logic rules effectively represent domain knowledge and hypotheses, offering explanations for real-world event data. A pioneer- ing work (Li et al., 2020) introduced a unified framework integrating first-order logic rules to model event dynam-

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Impact Statement
epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp- 720-736, 2018.
Our research proposes a novel framework combining logic- informed temporal point processes with amortized varia- to jointly infer unobserved mental states, learn the model parameters such as discrete rule content and continuous rule parameters, and predict future actions. Our method also holds practical significance. By inferring latent mental events and incorporating rule learning, our method enhances Al's ability to understand human thoughts and predict their actions, enabling timely assistance and promoting human-AI collaboration. The flexibility, adaptability, and interpretability showcase its potential to inspire future research endeavors.
tional Dnnectation-Maximiaae
Darvish, K. Simetti, E., Mastrogiovanni, F,, and Casalino, G. hierarchical architecture for human- roobtt cooper- ation processes. IEEE Transactions Robotics, (7): 567-586, 2020.
Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez- Rodriguez, M., and Song, L. Recurrent marked temporal point processes: Embedding event history vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1555-1564, 2016.
Acknowledgements
Hawkes, A. G. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58( (8(1):83-90 1971.
Shuang Li's research was in part supported by the Key Program of the NSFC under grant No. 72495131, NSFC under grant No. 62206236, Shenzhen Stability Science Pro- gram 2023, Shenzhen Science and Technology Program ST2E00233066091002000000 Longgang District Key Labo- ratory of Intelligent Digital Economy Security and SRIBD Innovation Fund S1F202400100.
Houston, J., Zuidhof, G., Bergamini, L., Ye, Y., Chen, L., Jain, A., Omari, S., Iglovikov, V., and Ondruska, P. One thousand and one hours: Self-driving motion prediction dataset. In Conference Robot Learming, pp. 409-418. PMLR, 2021.
Hu, S. and Clune, J. Thought cloning: Learning to think while acting by imitating human thinking. arXiv preprint SSKY::330600322. 2023.
References
Abu Farha, Y., Richard, A., and Gall, J. When will you do anticipniii temporal occurrences of activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5343-5352) 2018.
Jeong, S., Kang. Y., Lee, J., and Sohn, K. Variational embedding of hidden markov model to generate hu- man activity sequences. Transportation research part C: emerging technologies, 131:103347, 2021.
Barnhart, Johnson, E. L.., Nemhauser, G. L.., Savelsbergh, M. W., and Vance, P. H. Branch-and-price: Column generation for solving buge integer programs. Operations research, 46(3):316-33) 1998.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114. 2013.
Krajzewicz, D., Hertkorn, G., Rössel, C., and Wagner, P. Sumo (simulation of urban mobility)-an open-source traf- fic simulation. In Proceedings of the 4th middle East Symposium on Simulation and Modelling (MESM20002), pp. 183 183-187, 2002.
Cao, C., Yang, C., Zhang, R., and Li, S. Discovering intrin- sic spatial-temporal logic rules to explain human actions. Advances in Neural Information Processing Systems, 36, 2024.
Kuang, Y., Yang, C., Yang, Y., and Li, S. Unveiling latent causal rules: A temporal point process approach for ab- normal event explanation. In International Conference on Artificial Intelligecce and Statistics, pp. 2935-2943. PMLR, 2024.
Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Cramer, M., Kellens, K., and Demeester, E. Probabilistic decision model for adaptive task planning in human-robot collaborative assembly based on designer and operator intents. IEEE Robotics and Automation Letters, 6(4): -7332, 2021.
Li, G., Jiao, Y., Knoop, V. L., Calvert, S. C., and van Lint, J. Large car-following data based on lyft level-5 open dataset: driven vehicles. arXiv preprint arXiv:2305. 18900...1.9..... 2023.
Following autonomous vehicles vs. human-
Li, S., Wang, L., Zhang, R., Chang, X., Liu, X., Xie, Y., Qi, Y., and Song, L. Temporal logic point processes. International Conference on Machine Learning, pp. 5990-6000. PMLR, 2020.
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D.,, Munro, J., Per- rett, T., Price, W., et al. Scaling egocentric vision: The
10

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Li, S., Feng, M., Wang, L.,, Essofi, A., Cao, Y., Yan, J., and Song. L.. Explaining point processes by learning inter- pretable temporal logic rules. In International Conference on Learning Representations, 2021.
Wang, H., Zhang, Q. Wu, Jin, D., Wang. X., Zhu, L.,, and Yu, L.. Synthesizing human trajectories based on varia- tional point processes. IEEE Transactions on Knowledge and Data Engineering, 2023.
Lin, H., Wu, L.., Zhao, G., Liu, P., and Li, S. Z. Exploring generative neural temporal point process, arXiv preprint arXiv:2208.0/ 0774,, 2022.
Wei, P., Xie, D., Zheng, N., Zhu, S.-C., et al. Inferring human attention by learning latent intentions. In IJCAI, pp. 1297-1303, 2017.
Mehrasa, N., Jyothi, A. A., Durand, T., He, J., Sigal, L.,, and Mori, G. A variational auto- uncoder model for stochastic point processes. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3165-317 3174, 2019.
Xiao, S., Yan, J.. Yang, X., Zha, H., and Chu, S. Modeling the intensity function of point process via recurrent neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Xue, S., Shi, X., Zhang, J., and Mei, H. Hypro: A hybridly normalized probabilistic model for long-horize predic- tion of event sequences. Advances in Neural Information Processing Systems, 35:34641-34650, 2022.
Mei, H. and Eisner, J. M. The neural hawkes process: A neurally self-modulad uofulaoullaing multivariate point process. Advances in neural information processing systems, 30, 2017.
Xue, S., Wang, Y., Chu, Z., Shi, X., Jiang, C., Hao, H., Jiang, G., Feng, X., Zhang. J. Y., and Zhou, J. Prompt- augmented temporal point process for streaming event sequence. arXiv preprint arXiv:2310.04993, 2023.
Mei, H., Qin, G., Xu, M., and Eisner, J. Neural datalog through time: Informed temporal modeling via logical specification. In International Conference on Machine Learning, pp. 6808-6819. PMLR, 2020.
Yan, R., Wen, Y., Bhattacharjya, D., Luss, R., Ma, T., Fok- oue, A., and Julius, A. A. Weighted clock logic point process. In International Conference on Learning Re- search (ICLR) 2023, 2023.
Ogata, Y. On lewis' simulation method for point processes. IEEE transactions information theory, 27(1):23-31, 1981.
Yang, C., Mei, H., and Eisner, Transformer embeddings of irregularly spaced events and their participants. arXiv preprint wwxiv:2201.00044 2021.
Pan, Z., Huang, Z., Lian, D., and Chen, E. A variational point process model for social event sequences. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 173-180, 2020.
Yang, C., Wang. L., Gao, K., and Li, Reinforcement logic rule learning for temporal point processes. arXiv preprint arXiv: DrXXiv2308.006 060944 2023.
Puig, X., Shu, T., S., Wang, Z., Liao, Y.-H., Tenenbaum, J. B., Fidler, S., and Torralba, A. Watch-and-help: A chal- lenge for social perception and human-ai collaboration. arXiv preprint wrXiv:2010.098000 2020.
Yang, Y. Yang, C., Li, B., Fu, Y., and Li, S. Neuro- symbolic temporal point processes. arXiv preprint arXiv:2406.03914, 2024.
Rasmussen, G. Lecture notes: Temporal point processes and the conditional intensity function. arXiv preprint arXiv: 1806.8000000 00221, 2018
Yeung, S., Russakovsky, O., Jin, N., Andriluka, M., Mori, G., and Fei-Fei, L. Every moment counts: Dense detailed labeling of actions in complex videos. International Jour- nal of Computer Vision, 126:375-389, 2018.
Song. J. Wu, Y., Xu, Z., and Lin, X. Research car-following model based on sumo. In The 7th IEEE//nternational Conference on Advanced Infocomm Technology, pp. 47-55. IEEE, 2014.
Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, E. Self- attentive hawkes process. In International conference on machine learning, pp. 11183-1 193. PMLR, 2020.
Strouse, D., McKee, K., Botvinick, M., Hughes, E., and Ev- erett, R. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
Zhang, Y., Sharma, K., and Liu, Y. Vigdet: Knowledge informed neural temporal point process for coordination detection on social media. Advances in Neural Informa- tion Processing Systems, 34:3218 1231118--11111 2021.
Wan, Y., Mao, J., and Tenenbaum, J. Handmethat: Human- robot communication in physical and social environments. Advances in Neural Information Processing Systems, 35: 1201 12022,, 2022.
Zhu, S., Zhang, M., Ding, R., and Xie, Y. Deep fourier kernel for self-attentive point processes. In International Conference on Artificial Intelligence and Statistics, pp. 856-864 PMLR, 2021.

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Zolotas, M. and Demiris, Y. Disentangled sequence clus- tering for human intention inference. In 2022 IEEF/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pp.
IEEE, 2022.
Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In International conference on machine learning, pp. 11692-1120011 117022 PMLR, 2020.
12

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Appendix Overview
Section A presents key additional technical details, including pseudocodes for each building block and the overall proposed amortized variational EM framework.
Section provides comprehensive explanation of the generating process of synthetic datasets, along with an overview of the real-world datasets and the corresponding pre-processing procedure. also delineates the temporal logic rule templates as prior knowledge for real-world datasets.
Section C elaborates the bascline methods discussed in paper.
Section D shows more experimental details, including scalability experiments, ablation study, prediction examples on real-world datasets, et al.
Section provides reproducibility analysis, such as Syper-parameter selection and computing infrastructure.
Section F states the limitation and broader impacts of our proposed method.
A. Technique Details
A.1. DT-RP Sampling
In Alg. 2, we provide the pseudocodes for sampling from discrete-time renewal processes.
Algorithm 2 DT-RP Sampling for Event Type æ
1: Input: Last event grid kn, hazard function hrr()
2: Sample
Uniform (0,
3: for k == ho + 1,2.... ... do
4: Compute (k) = -o== (1 1e (T))
5: if Fk) > u then
6: 1 1= k&t, update kn -- D. call "restart"
end if
8: end for
A.2. Adaptive Rule Learning via Column Generation
In Alg. provide the pseudocodes for adaptive rule learning module via column generation.
Algorithm 3 Column Generation
1: Initialize: Start with Jo or a set predefined rules.
2: while not converged do
3: Step 1 (Master Problem): Solve for 0; given Fj-
4: Step 2 (Sub Problem): Generate candidates AF using a heuristic optimization strategy. arg miny ReducedCost(f).
Select f*
5: Step 3: Update F,+1 = F, UF.
6: end while
7: Terminate: Stop when F,+1 = Fj (reach the maximum iteration).
A.3. Amortized Variational EM
In Alg. provide the overall training procedures of proposed amortized variational EM framework.
A.4. Backtracking Mechanism
In Alg. we provide the pseudocodes for the backtracking sampling mechanism (we have reported in the main text).
13

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Algorithm 4 Integration with Variational EM
1: Within the M-step the EM algorithm:
2: E-Step: Freeze f 0 FF optimize & infer (m a).
3: M-Step: Alternates the following steps to obtain 6:
Continuous Update: Optimize 0 via master problem.
Discrete Update: Expand F, via subproblem.
Algorithm 5 Predicting Actions with Backtracking
Step 1: Infer Mental States
2: Given historical actions a, sample mental events m qm(m)) a) over discrete time grids.
3: Define the augmented history: H(tt)
m.
4: Step 2: Propose Candidate Action
5: Generate a candidate action time and associated value: mext: F ~ po(- H(()).
6: Step 3: Check for Mental Events
7: For the interval [ast ). soapled-sample mental events in discrete time:
8: if mental event occurs t E [Mast: next):
9: Update the history by integrating the mental event tm
10: SStt last 1.
11: Regenerate t next p-(- H((t))).
12: Step 4: Iterate with Maximum Rounds
13: Repeat Steps 3 until no new mental events occur in [tss f
14: Step 5: Accept 1ixt: Text as Final Predicted Action
B. Datasets and Prior Knowledge
B.1. Datasets
Synthetic Dataset
Syn Data-1: ground truth rules, mental predicates and action predicates. Each sequence has 18.60 actions on average.
Syn Data-2: more complicated scenario with 4 ground truth rules, 2 mental predicates and 2 action predicates. Each sequence has 13.25 actions on average.
World Dataset
Hand-Me That (Wan et al., 2022): contains 10,000 episodes of human-robot interactions in houschold tasks with a textual interface. In each episode, the robot first observes a trajectory of human actions towards her internal goal. Next, the robot receives a human instruction and takes actions to accomplish the subgoal behind the instruction. Here we consider robot's actions as the expert human trajectory. We combine human's history trajectory and robot's subsequent actions a whole sequence from single agent (human) and transfer the intermediate human instruction into human't 's mental state (e.g., human instruction: Please soak the piece of cloth the toilet be regard as human's mental: want soak the cloth). The 10,000 episodes can be classified based on 3 human's 's instruction types: bring-me, move-to and change-state. Considering the diversity and practicability of defined logic rule templates, we focus on involving more action and mental predicates instead of complex objects' names, we mainly use change-state episodes. Abandoning episodes without human history trajectories, we finally get 503 sequences with average length 30.5.
Car- Fol-Followng (Li et al., 2023): is processed from Lyft level-5 open dataset. The Lyft level-5 dataset (Houston et al., 2021) is a large-scale dataset of high-resolution sensor data collected by fleet of 20 self-driving cars. The dataset includes 1000+ hours of perception and motion data collected over a 4-month period from urban and suburban environments along a fixed route in Palo Alto, California. The dataset covers diverse Car-Following (CF) regimes and the enhanced dataset provides smooth, ready-to-use motion information for Car- hollFooloiing behaviors
14

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
investigation. A regime refers driving situation experienced by the following vehicle (usually restricted by the leading vehicle). 29k+ Human Vehicle HlV]-following-Autooooo Vehicle (HV) pairs were selected and enhanced in similar environments from the Lyft level-5 dataset, with the total duration spanning over 460+ hours, covering a total distance of 15,000+ km. We mainly focus on HV-following-HH CF pairs because the essential information of HV-following-AV CF pairs (AV's speed and acceleration which are used to segment vehicle's regimes) were estimated by Kalman filtering, while all the information in HIV-following-HV CF pairs are truly recorded with slight imputation of missing data. We extracted 2000 sequences from HV-following-HV CF pairs and defined reasonable logic rule templates to explain the change of regimes in those sequences.
Vehicle (AV) pairs and 42k+ Human
Vehicle (HV))following-Hill
MullTTUMMO (Yeung et al., 2018): 8): challenging dataset for action recognition, containing 400 videos of 65 different human actions. In this paper, we focus only on the basketball dataset with 2000 sequences. The time horizon each sequence is 208.32s with 38.41 actions on average.
SPPIC-Kitchen-00 (Damen et al., 2018): large-scale dataset in first-person (egocentric) vision, which are multi-faceted audio-visual, non-scripted recordings in native environments i.e. the wearers' homes, capturing all daily activities in the kitchen over multiple days. In this paper, we focus on two goals in the kitchen: cut onion and pour water, and extract 131 sequences contains related key actions. The time horizon of each sequence is 500s with 5.41 actions on average.
B.2. Prior Knowledge
For synthetic datasets, we know the ground truth temporal logic rules so that we can compare the rule learning accuracy. For real-world datasets, we have defined set of temporal logic rule templates as prior knowledge that align with intuition and experiential knowledge, capturing the time-based patterns associated with human mental intentions. The prior knowledge temporal logic rule templates truth rules. Our model aims to base on these kinds of prior knowledge to refine and generate more accurate logic rules on real-world dataset. It is noteworthy that these menta---mnaal-rlateed predicates are latent and do not actually exist within these real-world datasets.
partial and may not entirely correct but potentially capture
patterns of the ground
Predicates
Explanation
m mental event-1
a1
action event- 1
a2
action event-2
Table 4. Defined predicates and corresponding explanation for Syn data-1.
Rule Num
Rule Content
Rule Weight
Rule-1
my f- a1 (a) before m,)
0.6
Rule-2
01 02 (02 before (1)
0.6
Rule-3
02 <- mi, (m1 before a2)
0.8
Table 5. Ground truth temporal logic rules and corresponding weights for Syn Data-1.
Predicates
Explanation
m
mental event-1
1/12
mental event-2
01
action event-1
02
action event-2
Table 6. Defined predicates and corresponding explanation for Syn data-2.
Synthetic Dataset
Syn Data- 1: Defined predicates and ground truth temporal logic rules are shown in Tab. 4 and Tab. 5 respectively.
We can compare the rule learning accuracy based on synthetic dataset, since we know the ground truth.
Syn Data-22 2: Defined predicates and ground truth temporal logic rules are shown in Tab, 6 and Tab. respectively.
15

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Rule Num
Rule Content
Rule Weight
Rule-1
m1 f 01, (01 before m1)
0.6
Rule-2
m2 f- 01 A d2, (a, before a2). (a2 before m²)
0.6
Rule-3
01 <- m2 A d2, (m2 before a2), (#2 before a1)
0.8
Rule-4
0g <- mi. (m² before (z)
0.8
Table Ground truth temporal logic rules and corresponding weights for Syn Data-2.
Real- World Dataset
Hond-Me- That: Extracted predicates and prior knowledge temporal logic rule templates are shown in Tab, 8 and Tab. 9 respectively.
Extracted predicates and prior knowledge temporal logic rule templates are shown in Tab. 10 and Tab. 11 respectively.
For-Following:
-MulaTHUMOS: Extracted predicates and prior knowledge temporal logic rule templates in Tab. 12 and Tab. 13 respectively
SPPCC-Ktccen--000 Extracted predicates and prior knowledge temporal logic rule templates are defined in Tab. 14 and Tab. 15 respectively.
Predicates
Explanation
MoveTo
Move to a location or an object
PickUp
Pick up an object from a location or a receptacle
Put
Put an object on a location or into a receptacle
ToggleOn
Toggle on toggleable- thing, like electric device Soak an object
Soak
Open
Open openable thing, like cabinet
Clean
Clean an object or a location
Cool
Freeze food
Slice
Slice food
Heat
Heat food
Close
Close openable thing
WantToPickUp
Want to get an object
WonaToSoak
Want to soak an object
Want VontToOOeenToGr
Want to open an openable thing to get an object
Wo ToTggleOn
Want to toggle on an electric device
Want'ToPut
Want to put an object on a location or into receptacle
WantToClean
Want to clean a location or an object
WaatT
Want to heat food
WantToCool
Want to freeze food
ToltCo
Want to slice an object
Table 8. Defined predicates and corresponding explanation for Hand-Me- That dataset.
C. Baselines
In this paper, we primarily focus on baselines from three different fields: neural temporal point process model, logic-based model, and generative model. Below, we provide a detailed introduction to these baselines.
Neural Temporal Point Process Model
RMTPP (Du et al., 2016): The approach considers the intensity function of a temporal point process as a nonlinear function that depends on the history. It utilizes a recurrent neural network to automatically learn a representation of the influences from the event history, which includes past events and time intervals, thereby fitting the intensity function of the temporal point process.
16

---

Evolving Minds:
Logic-Informed Inference from Temporal Action Patterns
Rule Num
Rule Content
Rule-1
PickUp + WantToPickUpp A Move'To, (WantToPickUPp before MoveTTo, (Move'To before PickUp)
Soak f- WantToSoak A Move'To A Put A ToggleOn, (WantToSoak equal MoveTTo),
Rule-2
(MoveTo before Put), (Put before ToggleOn), (ToggleOn before Soak)
PickUp f WanTToOpenToCiet A MoveTo A Open (Wan before MoveTo), (MoveTo before Open), (Open before MoveTTo)
Rule-3
Rule-4
ToggleOn f-
WantToToggleOn A Move'To, (WantTologgleon equal MoveTTo, (MoveTo before ToggleOn)
Rule-5
Put + WantToPut A MoveTTo (WantToPut equal MoveTo), (MoveTo before Put)
Clean +- WantToClean A Soak A PickUp MoveTTo (WantToClean before Soak),
Rule-6
(Soak before PickUp), (PickUp before Mo),To), (MoveTo before CLean)
Heat <- WantToHHet PickUp A Move'To Put ToggleOn, (WantToHeat before PickUp),
Rule-7
(PickUp before MoveTo), Toove before Put), (Put before ToggleOn),
(ToggleOn before Heat)
Cool <- WantToCool A PickUp A Move'To A Open A Put A Close, (WantToCool before PickUp),
Rulc-8
(PickUp before MoveTTo), (MoveTTo before Open), (Open before Put),
(Put before Close), (Close before Cool)
Slice <- Want'ToSlice A Put A PickUp, (WantToSlice before Put),
Rule-9
(Put before PickUp), (PickUp before Slice)
Table 9. Temporal logic rules prior knowledge for Hand-Me- That dataset.
Predicates
Explanation
Fa
Free acceleration
C
Cruising at a desired speed
A
Acceleration following a leading vehicle
D
Deceleration following a leading vehicle
F
Constant speed following
Conservativelntentionn Aggressivelntention
The driver has a conservative intention, maintaining their speed
The driver has an aggressive intention, tending to accelerate Table 10. Defined following car predicates and corresponding explanation in Car-Following dataset.
THP (Zuo et al., 2020): The model employs a concurrent self self-ateention module to embed historical events and generate hidden representations for discrete time stamps. These hidden representations are then used to model the interpolated continuous time intensity function. THP can also incorporate additional structural knowledge. Importantly, THP surpasses RNN-based approaches in terms of computational efficiency and the ability to capture long-term dependencies.
PromptTPP (Xue et al., 2023): The model incorporates a continuous-tine retrieval prompt pool into the base TPP, enabling sequential learning of event streams without the need for buffering past examples or task-specific attributes. Specifically, this approach consists of a base TPP model, a pool of continuous time retrieval prompts, and a prompt-event interaction layer. By addressing the challenges associated with modeling streaming event sequences, this mode enhances the model's performance.
HYPRO (Xue al., 2022): The hybridly normalized probabilistic (HYPRO) model is capable of making long- horizon predictions for event sequences. This model consists of two modules: the first module is an auto-regun regressgeee
Rule Nun
Rule Content
A f C A AggressiveIntention, (C before Aggressivelntention),
Rule-1
(AggressiveIntention equal A)
C - Fa A Conservativelntention, (Fa before Conservativelntenioon)
Rule-2
(Conservaatvelntention equal C)
P+AAD Conservativelntentionn, (A before D),
Rule-3
(D before Conservativelntentionn) (Conservativelntention before F)
Table 11. Temporal logic rules as prior knowledge for Car- Car-Following datasct.
17

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Predicates
Explanation
Dribble
Dribbling the basketball
Pass
Passing the basketball from one person to another
Shot
An attempt to put the basketball in the baskethall hoop
PoorShootingOppotuue
Mental event, a player think that this is not a good shooting opportunity Mental event, a player think that this is a good shooting opportunity Table 12. Defined predicates and corresponding explanation for MuliTHUMOS baskethall dataset.
GoodShootingggpoottuity
Rule num
Rule Content
Rule-1
Dribble t- PoonShoctingOpporuuiiy. (PoorShociin@Oppotuniy before Dribble)
Rule-2
Pass <- PporShootingOppottuityy (PoorShooiin@Oppotunnyy before Pass)
Rule-3
Shot <- CoooSShociig@pporuunityy (GoodShooting@Oppruuniyy before Shot)
Table 13. Temporal logic rules prior knowledge for MuliTHUMOS basketball dataset.
base TPP model that generates prediction proposals, while the second module is an energy function that assigns weights to the proposals, prioritizing more realistic predictions with higher probabilities. This design effectively mitigates the cascading errors commonly experienced by auto-regressive TPP models in prediction tasks, thereby improving the model's accuracy in long-term forecasting.
Logie-Based Model
TELLER (Li ct al., 2021): It is a non-differemtiable algorithm that can be described as a temporal logic rule learning algorithm based on column generation principles. This method formulates the process of discovering rules from noisy event data as maximum likelihood problem. It also designs a tractable branch-and-pree -price algorithm to systematically search for new rules and expand existing ones. The algorithm alternates between a rule generation stage and a rule evaluation stage, gradually uncovering the most significant set of logic rules within a predefined time limit.
CLNN (Yan et al., 2023): The model learns weighted clock logic (wCL) formulas, which serve as interpretable temporal logic rules indicating how certain events can promote or inhibit others. Specifically, the CLNN model captures temporal relations between events through conditional intensity rates guided by set of wCL formulas that offer greater expressiveness. In contrast to conventional approaches that rely on computationally expensive combinatorial optimization to search for generative rules, CLNN employs smooth activation functions for the components of wCL formulas. This enables a continuous relaxation of the discrete search space and facilitates efficient learning of wCL formulas using gradient- based methods.
-STRR (Cao et al., 2024): A model specifically designed for learning -patial-tempora logic rules in order to explain human actions. It consists of two main modules: the rule generator, employing the transformer to infer logic rules by treating them as latent variables, and the reasoning evaluator, which predicts future entity trajectories based on the generated rules. While STLR demonstrates flexibility in generating logic rules without relying on prior knowledge, it lacks the ability to infer fine- ane-grrined latent mental events in a real-time manner, a capability inherent in our method.
Generative Model
Predicates
Explanation
Take
Retrieve the plate for future use
TakeEggs
Retrieve the eggs for further use
TakeOnion
Retrieve the onion for further use
TakeGlass
Retrieve the glass for further use
CutOnion
Retrieve the onion for further use
PourWater
Pour water to glass
NeedOnion Toook
Mental event, one has the intention
onion to cook
NeedWater
Mental event, one needs water in kitchen
Table 14, Defined predicates and corresponding explanation for EPIC-Kitchen- 100 datasct.
18

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Rule num
Rule Content
CutOnion <
NeedOnion ToCook A TakeOnion,
Rule-1
(NeedOnionToCook before TakeOnion), (TakeOnion before CutOnion)
PourWater < Need A TakeGlass,
Rule-2
(NeedWater before TakeGlass), (TakeGlass before PourWater) Table 15. Temporal logic rules as prior knowledge for GPIC-Kitchen- 100 datasct.
AVAE (Mehrasa et al., 2019): The model is a recurrent variational auto-encoder designed for modeling asyn- chronous action sequences. At each time step, the model utilizes the history of actions and inter-arrival times to generate distribution over latent variables. A sample from this distribution is then decoded into probability distributions for the inter-arrival time and action label of the next action. To address the limitations of using fixed prior in the traditional VAE framework, this model incorporates a prior net that enhances the learning process.
GNTPP (Lin al., 2022): The model is comprehensive generative framework for neural temporal point process modeling. It utilizes deep generative models probabilistic decoders to approximate the target distribution of occurrence time. For the encoder, the model considers both RNN-based methods and self-attention- based mechanisms. As for the decoder, the model incorporates multiple generative models, such as the temporal conditional diffusion denoising model, temporal conditional VAE model, temporal conditional GAN model, temporal conditional continuous normalizing flow model, and temporal conditional noise score network model. The various combinations of encoders and decoders make the GNTPP highly flexible.
VEPP (Pan et al., 2020): This model is a probabilistic generative model for event sequences. It introduces VAE to event sequence modeling that can better use the latent information and capture the distribution over inter-arrival time and types of event sequences. Specifically, it employs LSTM to embed the event sequence and utilizes VAE for modeling the event sequence.
-TVAE (Wang et al., 2023): This model combines the classical temporal point process with the neural variational inference framework, leading to its good ability to model human trajectories with continuous temporal distribution, variable length, and multi-dimensional context information.
D. Experimental Details
D.1. Scalability
To test the scalability of our proposed model, we have varied the training sample size within {1000, 2000, 3000, 4000, 5000} as well the ground truth rules within {2, 4, 5, 6) for synthetic datasets. Synthetic datasets with complex ground truth rules have larger corresponding domains for latent mental states. For example, synthetic datasets with 5 ground truth rules typically have latent mental state domains encompassing 4 mental states, ensuring that each of these 4 mental states is associated with at least one ground truth rule. Synthetic datasets with 6 ground truth rules are the most challenging with latent mental state domains encompassing 6 mental states. We have also extracted more data sequences for Hand-Me- Me-Thtt and Car-Following datasets to investigate the scalability of our model to handle large-scale real-world datasets.
As depicted in Fig. 6 and Fig. 7, our model swiftly converges with acceptable training time even with large-scale datasets for synthetic dataset and real-world dataset. The model's prediction performance improves with larger sample sizes, demonstrating its good scalability.
For more intricate rules and larger mental space domains like dataset with 6 ground truth rules (colored in orange in Fig. 6). the prediction ER% decreased to 52.08% and the MAE reduced to 2.52 when provided with 5000 samples, highlighting its ability to handle complex domains.
For these two real-world datasets with different sample sizes (results are in Fig. 7), the prediction performance improved with more samples and also resulted in more computational cost. For small-scale real-world dataset, our model well-handles challenges posed by small datasets. Even with dataset size of only few samples, our model delivers satisfactory results. For the Hand-Me- dataset with 100 samples, the ER% and MAE are 78.33% and 1.32. For the Car-Following dataset with 1000 samples, the ER% and MAE are 35. 38% and 2. 2.12, which are comparable with the majority basclines trained with larger sample size. For large-scale real-world dataset, the ER% and MAE decrease to 72.09% and 1.24 for Hand- hand-Me-Tie Me-Thar dataset with 1000 samples. And these two metrics decrease to 18% and 1.69 for the Following dataset with 5000 samples. Even on large-scale datasets, our algorithm converges relatively quickly with current computational infrastructure,
19

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
indicating the ability of our proposed model handle large-scale real-world datasets. This is attributed the constraints of rule length reduce search space, thereby mitigating computational complexity to some extent.
Time Cost Samples
ERS v.s. Samples
MAE v.s Samples
575
Rules
0
3.2
52.9
10.
47.
2.8
45.0
24-
42.-
40.0
2.0
375
1000
1500
2000
2500
2000
2500
4000
4500
5000
1000
1500
2000
2500
3000
3500
4000
4500
5000
1000
1500
2000
2500
3000
3500
4000
4500
5000
Samples
Sampes
Samples
Figure 6. Scalability and the computation time cost of our model on synthetic dataset with varying sample size, ground truth temporal
logic rules, and latent state domains. All the experiments are conducted three random runs and the standard error reflected in the shaded areas.
Time Cost v.s. Samples
Samples
Samples iie
Hand Ne That
Hand
Hand
1.40
Hand- Band-Me-TBe That
7-
78-
Time Cost (1)
1.33
6-
1:30
74-
1:29
1:15
200
600
1000
200
1000
230
1002
Samples
Time Cost Samples
ERS Samples
MAE v.s. Samples
2.2
36
Car-Follow Tore Cast 08
10
2.0
32-
1.0
7-
1.6
1500
2000
2500
30000
3500
4500 5000
Samples
Samples
Samples
Figure 7. Scalability and the computation time cost of model real-world datasets (Hand-Me-Thatd and Car-Following datasets) with varying sample size. All the experiments are conducted over three random runs and the standard error is reflected in the shaded areas.
D.2. Ablation Study
We conducted an ablation study to assess the importance of different components, using the following ablation settings: (i) removing the prior knowledge and removing the backtracking module, (ii) solely removing the backtracking module, (iii) solely removing the prior knowledge module, (iv) evaluating the full model. The results on synthetic datasets and real-world datasets are shown in Tab. 16 and Tab. 17.
Note that if we exclude the prior knowledge module, the rule learning module will commence with an empty rule set. Consequently, the training process will require additional time to converge due to increased iterations between solving the master problem and solving the sub problems within the rule learning module. The experiments were conducted with consistent settings and hyper-parameters as described in our paper.
For synthetic dataset, the results from different ablation settings confirm that appropriate prior knowledge enhances the accuracy of model predictions. Even though, our model without prior knowledge still perform well on prediction tasks on two synthetic datasets, demonstrating the capability of our model being adapt to situations lacking prior knowledge. Additionally, the inclusion of a backtracking mechanism plays more vital role which significantly improves model
20

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
Ablation Settings
Syn Data-1
Syn Data-2
Prior Knowledge
Backtracking
ER(%)
MAE
ER(%)
MAE
X
X
48.28
3.13
52.50
3.24
X
45.64
2.64
48.37
2.86
X
41.72
2.32
46.85
2.52
42.08
2.19
46,43
2.47
Table 16. Ablation study on synthetic datasets. Our current selection of modules are highlighted in blue.
Ablation Settings
Hand-Me- That
Car-Follow
MultiTHUMOS
EPIC-
Prior Knowledge
Backtracking
ER(%)
MAE
ER(%)
MAE
ER(%)
MAE
ER(%)
MAE
X
82.20
2.48
36.23
2.52
65.41
7.29
47.52
42.68
X
79.45
1.72
35.88
2.49
67.26
7.47
46.82
39.51
77.14
1.38
34.60
1.92
58.10
6.52
40.15
33.23
75.28
1.26
32.72
1.80
57.20
6.76
40.26
32.19
Table 17. Ablation study real-world datascts, Our current selection of modules are highlighted in blue.
performance, even when there is some level of noise in the prior knowledge. Overall, both modules contribute to enhancing the model's efficiency.
We want to emphasize that the practical significance of the ablation study experiments on real-world datasets, as obtaining prior knowledge in real data settings is sometimes challenging. The ability our model to perform well in scenarios lacking prior knowledge is a crucial aspect assessing its performance. Encouragingly, even after removing prior rules and starting with an empty rule set, our model still achieved good predictive performance by learning rules from scratch using the proposed rule learning module.
Furthermore, when providing the model with prior rules or rule templates, there was an improvement in predictive accuracy on real-world datasets. It is noteworthy that these prior logic rules are simple, readily available, and align with human cognitive logic. This indicates that our model effectively leverages prior knowledge to enhance performance while demonstrating the capability to handle datasets lacking such prior knowledge.
Overall, in practice, we can employ two strategies tailored distinct scenarios: autonomous rule learning for data-rich domains and template-guided learning for scenarios with limited data but ample prior knowledge. These approaches enhance the adaptability of our model when applied to real-world datasets.
D.3. Illustration of Attention Weight for Observed Actions
In our proposed model, the temporal point processes involve triggering between latent mental and action events, where historical actions can influence latent events and vice versa. Therefore, resort the attention mechanism to map the information of the entire action sequence on the spanning time horizon on each discrete time grid. Note that the attention in our model cannot capture how the mental state influences actions. The influence of mental process on action process is reflected on the intensity function after the inference of latent mental process.
In Fig. 8, we provide an example of attention weights for action sequence of Syn Data-1 I on different discrete time grids, which visualizes attention patterns of different attention heads. Pixel (i, k) in each figure signifies the attention weight of the event (4g, =rf attending the discrete time grid k. We can see that each attention head employs different pattern to capture dependencies. For each attention head, the impact of one event is different each discrete time grid, reflected by various attention weights. The impact of the entire action sequence on a discrete time grid can be conceptualized as a weighted combination of the entire action sequence, with weights derived through attention mechanisms. The attention mechanism effectively captures the potential influence of historical events at specific time points on the intensity of future events.
D.4. Learned Rules with Real-World Significance on Real-World Dataset
In Tab. 18, we report several temporal logic rules that identified with real-world significance on real-world datasets, which are learned by our rule learning modules. These rules are not provided as prior knowledge and are easily overlooked. For Hand-Me- Me-That our rule learning model mines that after and intention WanrToPickUp, human may sequentially take actions MoveTo and Open, thus increase the likelihood of taking action of PickUp. This rule overlooked in our prior knowledge,
21

---

Evolving Minds: Inforeed Inference from Temporal Action Patterns
Event Index
Event Index
Event Index
Event Index
320
Grid
Grld
Grid
Grid
Figure 8. Visualization of attention patterns of different attention heads for action sequences in Syn Data-1 I on different discrete time grids.
but aligns with human behavioral logie. The learned rule on MuliTHUMO dataset representative, as a basketball player dribbles to seek offensive opportunities, once they identify favorable moment, they take the shot. Our learned logic rules effectively capture this behavior the court. Although did not provide such prior knowledge during model training, model still identified this logic rule adaptively through column generation in data-driven manner. Other interesting rules found for other two real-world datasets also hold practical meaning, showcasing the good adaptability and the effectiveness of our rule learning module.
Real-World Dataset
Rule Content
PickUp +- WantToPickUp A MoveTo A Open, (WantToPickUpp before MoveTo), (MoveTTo before Open), (Open before PickUp)
Hand-Me-That
A - C A Aggressivelntention A Fa, (C before Aggressivelntention)
Car-Follow
(Aggressivelntention equal Fa). (Fa before A)
Shot < Dribble A wwwaahoootinngporuuuen (Dribble before GoodShootiing@ppootuning
MultTTHUMS
GGoodShhotinng@pporuuti before shot)
Need Water t- NeedOnionToCook A TakeOnion,
EPIC-Kitchen
(NeedOnionTooCee before TakeOnion), (TakeOnion before Need Water)
Table 18. Learned temporal logic rules (we only report some of them) with real-world significance real-world datasets.
D.5. Prediction Example on Real-World Dataset
Our model demonstrates intriguing applicability in real-life scenarios due its ability of accurately predicting events and speculating human thoughts. As exemplified by the Hand-Me- dataset, in Fig. 9, our proposed model effectively infers human historical intentions like want clean the grill, and want to soak. It also forecasts future human actions with correct time indexes. In this instance, if the Al-Agent infers person's 's intention to clean the grill at time index 20 and predicts that the person will clean the grill at time index 22, can promptly retrieves the grill for him, which will significantly enhance convenience.
Mental Intention
Time index: 3 Want to clean the grill Want clean the grill (mental event)
Time index; 8
Time index: 15 Want to soak
Time index: Want to clean the grill (mental event)
(mental event)
mental event)
Prediction
555
History
Inferred Human Intention
Timeline
Prediction
Histony
Human Action
Time index: 1
Time index: Open the door (action event)
Time index: 12 Pick up the cloth (action event)
Time index: 19
index: Clean the grill (action event)
Move grill
Move to the sink (action event)
Soak the cloth (action event)
(action event)
Figure 9. Inferred history mental predicted human mental events. Bottom: observed and predicted human actions. Notice that the original dataset only includes the order of action occurrences. During data pre-processing, we assume actions to have equal time intervals of and then discretize the timeline. Then, we can use time indices to represent specific time point.
and predicted future events for human trajectory aiming clean the grill. Top: inferred and
22

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
We also provide another prediction example for Car-Following dataset. In Fig. 10, our model infers a driver's historical intentions and predicts their future car-following actions. In the field of autonomous driving, self-driving vehicle can adjust its lane and drive reasonably by considering the inferred intentions of human drivers in neighboring lanes and their predicted behaviors, all while adhering to traffic regulations.
Conservative Intention
Cruising desired speed
Aggressive intention
Acceleration
Deceleration follow car
Constant speed following
acceleration
follow
Time Line
History
Future
Satellite map of Palo Alto, California, extracted from Google Earth, Right: Car following process (pink car) for one car trajectory. The historical mental event inferred by our method is indicated within the pink boxes. The next action in the future of the pink car predicted by our method is represented in the blue box to the right of the dashed line. The visualization enhanced via SUMO simulator (Krajzewicz et al., 2002; Song et al., 2014).
Figure 10. Left:
E. Reproducibility Analysis
E.1. Hyper-Paramen Parameter Selection
We present the selected hyper- parameters on both synthetic datasets and real-world datasets in Tab. 19. The hyper-parameter selection metrics reflect the trade-off between training converged log-likelihhn oogllikelhhood, prediction performance, and time efficiency.
Hyper- parameters
Value Used
Syn Data-1
Syn Data-2
Hand-Met That
Car-Following
MultTHHUMSS
EPIC-Kitchen
Batch Size
32
32
32
64
64
16
Time Grid Resolution
0.50
0.50
1.00
0.25
1,50
2.50
Max. Backtracking Rounds
2
2
5
3
Embedding Dimension
16
16
Dropout Learning Rate
0.15
0.15
0.10
0.15
0.15
0.10
le-3
le-3
5e-4
le-3
le-3
5e-4
Table Descriptions and values of
used for models trained on both synthetic dataset and real-world datasets.
such as time grid resolution and maximum number of backtracking rounds is based on empirical results, which strikes a balance between model performance and training efficiency. We use metrics including training ELBO, training time, and prediction accuracy to select appropriate resolution of a discretized time grid and the number of backtracking rounds, with experiment results shown in Tab. 20 and Tab. 21.
Specifically, the selection of some key
For the resolution of time grid, as shown in Tab. 20, from the results can that under current selection of 0.50, the proposed model achieves the second highest ELBO on both two synthetic datasets. Reducing the resolution to 0.25 only marginally increases the ELBO to 18.45 on Syn Data-1 and 22.08 on Syn Data-2, but extends the model convergence time by 1.78 hours on Syn Data-1 and 1.84 hours on Syn Data-2 respectively. Conversely, while increasing the resolution can only slightly shorten training time. Moreover, it also decreases the model's converged ELBO and results in a decline in prediction accuracy when using the well-trained model for prediction tasks. This could be due to missing some crucial latent mental events. Hence, balaneing between accuracy and computational efficiency, we opt for the current resolution of 0.5.
For the maximum number of backtracking rounds, as shown in Tab. 21, under current selection of up 3 rounds of backtracking, satisfactory results have been achieved in prediction tasks. Increasing the maximum backtracking rounds to 5 shows minimal improvement in prediction accuracy while adding to computational complexity. Conversely, reducing the backtracking rounds to a maximum of 2 or fewer significantly decreases prediction accuracy. This pattern is consistent across both synthetic datasets.
For real-world datasets, the selection of key hyper-parameter follow similar selection strategy, namely balancing the model
23

---

Evolving Minds: Logic-Informed Inference from Temporal Action Patterns
performance and training efficiency. Our current selection for real-world datasets can be found in Tab. 19.
Syn Data-1
Syn Data-2
Resolution
0.25
0,50 *
0.75
1.00
1.25
0.25
0.50
0.75
1.00
1.25
ELBO 1 ER% MAE +
18.45
18.33
16.62
15.39
15.22
22.08
21.42
19.46
16.80
16.25
40.67
41.72
42.74
45.60
45.93
46.52
46.85
48.78
49.60
51.25
2.28
2.32
2.36
2.53
2.62
2.45
2.52
2.67
2.88
2.85
Time Cost (h) 4
6.57
4.79
4.41
3,85
3.69
6.82
4.98
4.36
4.22
3.94
Table 20. Compare various time grid resolutions and their impact on the model performance for synthetic dataset. "x" indicates our current hyper-parameter selection for specific dataset. Other hyper-parameters and training sample size remain consistent.
Max. Back-
Syn Data-1
Syn Data-2
tracking Rounds
0
0
1
2
3
ER% +
45.64
45.33
43.82
41.72
42.07
41.28
48.37
47.49
46.92
46.85
46.70
45.86
MAE $
3.12
3.06
2.75
2.32
2.19
2.11
2.86
2.77
2.74
2.52
2.46
2.42
Table 21. Compare various maximum number backtracking rounds and their impact on model performance for synthetic dataset. "x" indicates our current hyper- parameter selection for specific dataset. Other hyper- parameters and training sample size remain consistent.
E.2. Computing Infrastructure
All synthetic data experiments and real-world data experiments, including the comparison experiments with baselines, are performed on Ubuntu 20.04.3 LTS system with Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz, 227 Gigabyte memory.
F. Limitation and Broader Impacts
Our work has vast potential applications in the field of human-Al collaboration. The proposed approach enables timely and accurate inference of human mental events, well precise prediction of future human behavior. This will assist Al in providing timely, accurate, and useful assistance. For instance, it can aid elderly individuals with limited mobility in managing daily activities or help self-driving vehicles navigate roads more safely and smoothly.
One limitation of our current approach the reliance on hand-crafted logic rule templates in the decoder. While these rules provide interpretability, they may introduce additional biases. mitigate this limitation, we have already proposed a rule learning module via column generation, which can learn rule from an empty rule set. This approach enhances the model's flexibility by adapting to the nuances the data and reduces the risk associated with manually introduced biases. We can explore more rule mining algorithms to improve the efficiency in the future work.
Additionally, the discretization of the timeline might introduce some noise when sampling the latent mental events. In real-world scenarios, establishing well-defined and fine-grained discrete time grid can necessitate conducting numerous experiments. It's worth noting that choosing the interval for discretization is tunable hyperparameter. We can explore methods to automate hyperparameter tuning streamline this process and ensure optimal performance without the need for extensive manual experimentation.
24


Reviews:
[{"review_id": "x4vKjV3gWr", "reviewer": "Reviewer_JJ34", "summary": "The paper presents an amortized variational EM framework for understanding human mental states by modeling the relationship between actions and hidden mental events over time. Some innovations include using logic rules as priors to improve interpretability and approximating the posterior distribution of latent mental states by discrete-time renewal process.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback!\n\nRegarding your question:\n\n_\"One question regarding autonomous rule learning for data-rich domains: in Table 17, the elements of temporal logic rules, such as PickUp and WantToPickUp, are still predefined, right?\"_\n\nYes, in our current framework, predicates (i.e., _\"elements of temporal logic rules\"_ as shown in Appendix D.4, Table 17) are drawn from a predefined candidate pool. This reflects a **closed-world assumption**, where all possible actions and mental states are specified in advance.\n\nHowever, our approach can be extended to an **open-world setting** by incorporating **predicate invention mechanisms**. Inspired by prior work [1] on using vision-language models for predicate generation, we can leverage LLMs to handle novel situations—first prompting them to generate diverse candidate actions and mental states, and then integrating these into the rule-learning and inference process.\n\n[1] Liang, Y., Kumar, N., Tang, H., Weller, A., Tenenbaum, J. B., Silver, T., ... & Ellis, K. Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. ICLR 2025.", "claims_and_evidence": "The paper makes several claims about the effectiveness of their proposed method, and overall, these are supported by experimental evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria generally align well with the problem of inferring human mental states and predicting future actions from temporal data. The use of two strategies tailored to different scenarios is useful. One question regarding autonomous rule learning for data-rich domains: in Table 17, the elements of temporal logic rules, such as PickUp and WantToPickUp, are still predefined, right?", "theoretical_claims": "No formal theoretical claims are provided in this work.", "experimental_designs_or_analyses": "The experiments on multiple datasets demonstrate improved event prediction performance, and the baselines from three categories—Neural TPP, Rule-Based, and Generative models—provide fair comparisons. Additionally, the ablation study on backtracking supports its effectiveness.", "relation_to_broader_scientific_literature": "This work builds on existing work by combining logic reasoning with neural networks, dynamically discovering rules while balancing model flexibility and explainability.", "essential_references_not_discussed": "NAN", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AnuQSNJRyM", "reviewer": "Reviewer_w7nu", "summary": "This paper proposes combined logic-informed temporal point processes with amortized variational EM, allowing their method to infer underlying mental states reliably, even in low-data regimes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer w7nu for the detailed analysis and insightful comments, which benefit us to further improve our paper! To address your concerns, we have prepared a detailed point-by-point response below.\n\n**$\\star$ Scalability**: Your comments are very valuable! Yes. We acknowledge that our original statement was imprecise. A more accurate statement would be: \"Our model demonstrates potential for scaling to large datasets.\"\n\nCompared with related methods, regarding our rule learning module, existing related approaches like TELLER [1] and CLNN [2] require only 600-2,400 training synthetic sequences. While event prediction models with latent states like AVAE [3] use up to 2,000 sequences. In comparison, our reported experiments utilize relatively large sample sizes. \n\nMoreover, our fine-grained model's computational complexity stems from three key factors: \n\n(1) The inherently combinatorial nature of rule learning.\n\n(2) Temporal discretization requiring inference at each tiny time grid for accurate continuous-time mental event inference.\n\n(3) Event density per sequence. In our scalability experiments, the most complex synthetic dataset comprises 5,000 sequences, with an average of 47.60 action events per sequence—totaling 238,000 events. This represents a relatively large-scale dataset. \n\nTherefore, scalability assessment must consider **not just sample size**, **but also rule learning complexity, temporal resolution, and event density** -- making our 5,000-sample experiments relatively large-scale under these demanding conditions. \n\nConsidering the above key factors influencing scalability, we have added new experiments using Syn Data-2 and larger sample size to further assess the potential of our model. Please find **Table 1** in \n\nhttps://anonymous.4open.science/r/paper9060-F13F\n\nThese new experimental results show that the model performance exhibits asymptotic stabilization with increasing dataset size, and our model still has potential to perform well in 10K+ sequences scale dataset within satisfactory training time cost.\n\n[1] Li, S., Feng, M., Wang, L., Essofi, A., Cao, Y., Yan, J., & Song, L. Explaining point processes by learning interpretable temporal logic rules. ICLR 2021.\n\n[2] Yan, R., Wen, Y., Bhattacharjya, D., Luss, R., Ma, T., Fokoue, A., and Julius, A. A. Weighted clock logic point process. ICLR 2023.\n\n[3] Mehrasa, N., Jyothi, A. A., Durand, T., He, J., Sigal, L., and Mori, G. A variational auto-encoder model for stochastic point processes. CVPR 2019.\n\n**$\\star$ Illustrative Examples**: Thanks for your advice! We will incorporate additional illustrative examples in subsequent revisions to enhance comprehension. Following your suggestion **regarding the distinction between action and mental events, we added example** as:\n\n“Consider a person who intends to start exercising and later maintains regular strength training at the gym. This behavioral sequence involves: (1) a mental event (the unobservable intention to exercise) and (2) action events (the observable gym workouts). Crucially, mental events are internal and unobservable, while actions are external and directly observable—constituting what we actually perceive.”\n\n**$\\star$ Unfamiliar Situation**:  Our rule-generation algorithm is inherently capable of learning logic rules from scratch, even in scenarios with limited prior knowledge, making it highly adaptable to unfamiliar situations. In practice, when encountering such scenarios, our system operates in two modes: If pre-trained on similar datasets, it directly transfers the well-trained model and takes the learned rules as priors to the new data; Otherwise, it autonomously learns new rules without requiring predefined logic rules, with the built-in backtracking mechanism further improving accuracy. \n\nWe have assessed our method under two distinct unfamiliar conditions: **transfer adaptation (for domain-related cases)** and **fully unfamiliar application (for entirely unseen datasets)**, with comparative results presented in **Table 2** in \n\nhttps://anonymous.4open.science/r/paper9060-F13F\n\nThese new experimental results confirm our model **maintains robust performance in fully unseen scenarios**. When **pre-trained** on similar datasets, the learned rules **boost both training efficiency and final model performance**.\n\n**$\\star$ Correction for Typos**: Thanks for your careful review! All typos have been corrected and will be updated into the final version.", "claims_and_evidence": "Their experimental results show the effectiveness of this framework on some synthetic as well as real-world datasets.\n\nHowever, I do have some concerns regarding scalability. First of all, I think it is an impressive result that the method works well in low-data regimes and is efficient through the use of EM and injection of priors as logic rules. However, I am less convinced by the current dataset sizes that there is evidence of scalability, despite the authors' pointing this out.", "methods_and_evaluation_criteria": "Yes, the baselines, datasets and the metrics used for evaluation all seem reasonable to me.", "theoretical_claims": "There aren't theoretical claims. The derivations used when presenting the method looked good to me.", "experimental_designs_or_analyses": "Overall the authors conduct experiments both on synthetic and real-world datasets to showcase their method.", "relation_to_broader_scientific_literature": "Learning to infer human intent, and predicting next actions from existing datasets is a relevant problem for human-centric AI and the authors propose a method that achieves good results in low-data regimes, with the logic rules offering additonal interpretability as well.", "essential_references_not_discussed": "I am not aware of missing literature at this point.", "other_strengths_and_weaknesses": "I think overall this is a nice work, with the logic-informed priors offering interpretability in human-centric AI and the method showcasing good results in low-data regimes. \n\nI do have some worries though regarding scalability claims -- I do think it is impressive that the framework works well in low-data regimes, I am bringing up scalability mainly because the authors mention it throughout the paper (that the method can scale), but the largest synthetic dataset used has 5K trajectories, and a small number of underlying rules (up-to 6 ground truth rules App D.1). I do not think this is enough evidence to support the scalability claims, I would be curious to hear the authors' thoughts on this.\n\nAnother weakness I find is in the presentation of the method: I think the overall setup of the method can be better motivated with some examples in the beginning of the Preliminaries section. For example, the explicit actions and mental events differentiation first comes up in Section 3.", "other_comments_or_suggestions": "- Line 270 column 1: is computd --> is computed\n- Line 272 column 2: nearconvergence --> near convergence or near-convergence\n- Lines 292-292 colum1: model is easily adopt to large-scale --> model can easily be adopted ...?\n- Line 282 column 2: other baseline attempt to predict **the** next event from history\n- Line 303 column 2: In paticular --> in particular\n- Line 403 column 2: faces scalability issue**s**", "questions_for_authors": "- As mentioned before, I would be curious to hear the authors' thoughts on how well their method can scale or at which point they realistically would expect it to not scale anymore. \n\n- How does the method perform in unfamiliar/novel situations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92AELBgQ8z", "reviewer": "Reviewer_BKJo", "summary": "The paper introduces a single framework to infer human intentions, predict future actions, and interpretable logical rules. The motivation is that human actions occur irregularly and are driven by unobserved mental states/intentions. To address this, the paper proposes a framework combining the temporal point process (TPP) and amortized variation Expectation-Maximization (EM) to model the bidirectional dynamics between irregular human actions and latent mental states. Logical rules are used as priors to guide TPP to build the relationship between intention and actions, reduce dependency on large datasets, and, ensure interpretability. The framework jointly optimizes the model parameters, logical rules, and inference networks. Experiments are conducted on synthetic and real-world datasets on action prediction task (next event prediction from history) and evaluated using error rate % and mean absolute error metrics and the framework shows a good improvement on both metrics for all datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer BKJo for the insightful analysis and recognition of our work! We hope our responses listed below can address your concerns.\n\n**$\\star$ Examples Revealing Newly Uncovered Rules**: We have reported a subset of temporal logic rules that have been identified as having real-world significance based on real-world datasets, and due to the page limitation, we put the results in the **Appendix D.4, Table 17**. These rules are learned by our model which are not provided as prior knowledge and easily overlooked. For the reviewer’s convenience, below we analyze one of the learned rules as a concrete example, explaining both its formal interpretation and real-world significance:\n\n$$PickUp \\leftarrow WantToPickUp\\ \\wedge\\ MoveTo\\ \\wedge\\  Open, (WantToPickU\\ before\\ MoveTo), (MoveTo\\ before\\ Open), (Open\\ before\\ PickUp)$$\n\nThis rule is learned from the Hand-Me-That dataset, capturing a sequential intention-action pattern: the user first develops an intention to pick up an object ($WantToPickUp$), then moves toward a destination ($MoveTo$), and opens the container ($Open$), ultimately leading to the target action ($PickUp$).\n\nThe practical value lies in using this learned rule as guidance for mental state inference. When our model detects the user's WantToPickUp intention, the AI agent can proactively assist -- for instance, by opening the container in anticipation of the user's need. This inference and prediction capability can enhance human-agent interaction efficiency.\n\n**$\\star$ Discussion on Limited/Few Mental States**: In real-world settings, mental states are typically **high-level and inherently sparse within specific contexts**. Our model operates under a **closed-world assumption**, focusing on constrained datasets tailored for specific tasks. As a result, the mental states in our experiments are confined to a **predefined domain-relevant set**—justifying our consideration of only a limited number of mental states for real-world applications. Encouragingly, our model **has the potential to adapt to scenarios with unknown or numerous mental states**. Inspired by [1], which uses vision-language models for predicate invention, we can also leverage LLMs to generate diverse mental states, circumventing the limitations of pre-defined states while producing arbitrarily numerous variations, before proceeding to rule learning and inference.\n\n[1] Liang, Y., Kumar, N., Tang, H., Weller, A., Tenenbaum, J. B., Silver, T., ... & Ellis, K. Visualpredicator: Learning abstract world models with neuro-symbolic predicates for robot planning. ICLR 2025.\n\n**$\\star$ Training Time & Computing Infrastructure**: We have reported the training time for both synthetic datasets and real-world datasets as well as the computing infrastructure in our previous submission, and due to the page limitation, we put the results in the Appendix. Below we provide the section indexes for reviewer's convenience. The training time for varying sample sizes and the number of ground truth rules have been shown in **Appendix D.1**, for both synthetic datasets (**Figure 5**) and real-world datasets (**Figure 6**). Additionally, we have compared the impact of different hyper-parameter choices on training time and model performance in **Appendix E.1**. Details of the computational resources and computing infrastructure used in our experiments were provided in **Appendix E.2**.\n\n**$\\star$ Open-Source**: Yes, we will release the codes for the final version.\n\n**$\\star$ Correction for Typos**: Thanks for pointing it out! We have updated the legend in Figure 3 and made corresponding revisions to the final manuscript.", "claims_and_evidence": "1. The claim that mental states drive human behavior and building a relationship between then can be helpful in predicting next actions has strong evidence as the framework performs well across all synthetic and real-world datasets.\n1. The method optimizes the logical rules (initialized from scratch) and shows results that the rule learning module is able to learn the 4 ground truth rules in Figure 3.\n2. Mental events are sampled based on the hazards and probabilistic sampling is able to sample reasonably within an error range as shown in Figure 3.\n4. The paper also mentions the scalability of the approach and shows supported results in the supplementary.", "methods_and_evaluation_criteria": "The objective of the paper is to improve next action prediction task by strengthening the relation between mental state and human actions. \nThe proposed method makes sense to address this objective. The evaluation criteria using error rate % and mean absolute error as metrics is valid and is being by the prior TPP method [1] as well. The paper also shows the accuracy on learning logical rules and the sampling efficiency of mental states which evaluates the method well.\n\n[1]. Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In International conference on machine learning, pp. 11692–11702. PMLR, 2020", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "The evaluation criteria using error rate % and mean absolute error as metrics is valid and is being by the prior TPP method [1] as well. The paper also shows the accuracy of learning logical rules and the sampling efficiency of mental states which evaluates the method well. The comparison metric shows significant improvement when compared with baselines and further analysis on logical rules and sampling efficiency shows accuracy as well. In supplementary, the scalability of the approach is measured which supports the claim of the paper. \n\n[1]. Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H. Transformer hawkes process. In International conference on machine learning, pp. 11692–11702. PMLR, 2020", "relation_to_broader_scientific_literature": "The key contribution of the paper of jointly inferring the human mental state and the human actions is helpful in the scientific literature of video understanding where it is essential to understand the context of the video and human intentions to anticipate future actions. The key contribution would especially be helpful for tasks such as action anticipation.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The motivation and contribution of the paper are strong and establish a relation between understanding latent human mental state and human actions.\n2. The paper is written well and has a good explanation for the needed areas such as the subsection on comparison with VAEs.\n3. The idea of logical rules is also helpful in reducing reliance on datasets and improving interpretability.\n\nWeakness:\n1. In Line 95, the authors claim that the rule discovery reveals novel and overlooked intention-action patterns. Is there any result/evidence in the paper that can support this?\n2. As the idea of inferring human mental states seems to be a key contribution, the datasets used for experiments have very limited number of mental states such as EK-100 considers 2 states, hand-me-that considers 4, and car-following has a single mental state of human vehicle following human vehicle. Can the authors discuss why limited/few mental states in datasets are considered for experiments?", "other_comments_or_suggestions": "In Figure 3, in the fitted hazard graph, there is a typo in the label : Ture -> True.", "questions_for_authors": "1. Will the work be open-sourced?\n2. What is the training time and how much resources does the framework require?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers voted accept based on (1) the paper proposes a reasonable idea, (2) the experiments are solid, (3) the work helps reduce reliance on datasets and improve interpretability, etc., which I mostly agree with. Some concerns were raised but have been properly addressed.
Decision: Accept (poster)