Discovering Intrinsic Spatial-Temporal Logic Rules to Explain Human Actions
Authors: Chengzhi Cao
Chao Yang
Ruimao Zhang
Shuang Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Logic rule
human actions
sports analyze

Abstract: We propose an interpretable model to uncover the behavioral patterns of human movements by analyzing their trajectories. Our approach is based on the belief that human actions are driven by intentions and are influenced by environmental factors such as spatial relationships with surrounding objects. To model this, we use a set of spatial-temporal logic rules that include intention variables as principles. These rules are automatically discovered and used to capture the dynamics of human actions. To learn the model parameters and rule content, we design an EM learning algorithm that treats the unknown rule content as a latent variable. In the E-step, we evaluate the posterior over the latent rule content, and in the M-step, we optimize the rule generator and model parameters by maximizing the expected log-likelihood. Our model has wide-ranging applications in areas such as sports analytics, robotics, and autonomous cars. We demonstrate the model's superior interpretability and prediction performance on both pedestrian and NBA basketball player datasets, achieving promising results.

Discovering Intrinsic Spatial-Temporry Logic Rules to Explain Human Actions
Chengzhi Cao 1,2 Chao Yang', Ruimao Zhang', Shuang Li
The Chinese University of Hong Kong (Shenzhen) University of Science and Technology of China chengzhi caodmail. ustc. edu. en, 2220430110lind cuhk edu. cn, zhangruimao@cuhi: edu. cn, lishuang@cuhk: edu.
Abstract
We propose a logic-informed knowledge-driven modeling framework for human movements by analyzing their trajectories. Our approach inspired by the fact that human actions are usually driven by their intentions or desires, and are influenced by environmental factors such as the spatial relationships with surrounding objects. In this paper, we introduce a set of sparial-temporet logic rules as knowledge explain human actions. These rules will be automatically discovered from observational data. To learn the model parameters and the rule content, we design an expectation-maxiiizeee latent variables. The EM algorithm alternates between the E-step and M-step: the E-step, the posterior distribution over the latent rule content is evaluated; in the M-step, the rule generator and model parameters jointly optimized by maximizing the current expected log-likelihood. Our model may have a wide range of applications in areas such as sports analyties, robotics, and autonomous cars, where understanding human movements are essential. We demonstrate the model's superior interpretability and prediction performance on pedestrian and NBA basketball player datasets, both achieving promising results.
(EM) algorithm, which treats the rule content as
Introduction
For human, although the exhibited movements be complex, the logic behind the actions is usually simple, clear, and can be generalized [17.. The logic rules present a compact and high-level knowledge representation, defining what actions tend to be executed under what conditions. It emphasizes reasoning, and encourages structuring the world in terms of objects, properties, and relations (22). There has been great interest and business value in unveiling the human logic from the observational movements and actions [88. We provide two motivating examples below.
In sports analytics, understanding each player's behavior preferences tendencies under various scenarios will provide coaches with valuable information [. Usually, coaches need to watch the game or training videos for hundreds of hours before they can summarize the discoveries into compact principles. Could we design an algorithm to synthesize these principles from the raw action data automatically? One can imagine, such a tool will significantly reduce the workload of coaches, providing more granular insight into each player's 's capabilities and strategies, and aiding in personalized training and match strategy design [5].
In self- drividn car, it's essential to enable the self-driving cars to "read human's 's mind" like humans. This requires the self-driving cars to automatically understand human intentions and reasoning when
"Wor done graduate research assistant at The Chinese University of Hong Kong (Shenzhen)
"Corresponding author.
37th Conference on Ncural Information Processing Systems (NcurIPS 2023).

---

Dynamics for Event Data. Since the human actions are irregular spatial- temporal event data, we also briefly discuss probabilistic models for such event sequences. Modeling the teatial- dynamics of discrete events foundational in many scientific fields and ap- events they are running on the same roads with human drivers [2] If self-driving cars can automatically distill logic rules from their observed low-level noisy human actions and movement trajectories, will increase the technical reliability and accelerate the widespread use of self-driving cars.
For human actions, lots of the governing rules would be regarding the spatial- relation with the surrounding environments and their intentions [2] For example, when a basketball player with a ball within the scoring range, his/her action, such as shoot, pass, or triple threat, is influenced by historical and current surrounding factors, such as the current locations of the player and the defenders, the time clapses of the game, the success shooting rate of the player in today's 's game, and so on. The quick decision made by the player is actually reflecting composition of all these factors, which can be described as a collection of spatial-temport temporal logic rules in our model.
Formally, in our introduced include spatial-temporal relation predicates, in addition to the commonly defined object property and relation predicates. The rule content will capture the spatial relation of the object with surrounding objects, as well as the temporal ordering constraints of the events.
-terporll logic rules, the logic variable (i.e., predicate) set will
Our methods have the following distinct features:
From the modeling perspective: (i) Our human action model is a rule-based probabilistic model, which treats each hidden rule "soft" constraint. We assume each rule will be executed by humans with probabilities, and this tolerates the uncertainties in data. (ii) Our model directly uses low-level, fine-grained, and (may) irregulatly-spaeed action times and locations (i.c., original 3d coordinates) as inputs, as opposed to other rule-based models, where one needs to first extract relational data as inputs. (iii) Our spatial and temporal predicates are also probabilistic. For predicates such as "left of" or "before", we model them as kernel functions with learnable variables. In this way, our introduced spatial-temport predicates are smooth functions of the input locations and times, which increases model flexibility.
From the learning perspective: We propose a tractable and differentiable algorithm that can jointly learn the rule content and model parameters from observational data. The learning framework is designed maximize the likelihood of the human action trajectories. Specifically, we propose to use a neural rule generator to generate the spatial-temporal logic rule set. Our continuous rule generator parameters will be optimized in a differentiable way. The overall learning procedure is an expectation- maximization (EM) algorithm, where we treat the rule set as latent variables. In the E-step, the posterior distribution over the latent rule set is evaluated. In the M-step, both the rule generator parameters and the model parameters are optimized by maximizing the expected log-likelihood with respect to the current posterior. We demonstrated the promising performance of model in terms of human action prediction and explanation on two interesting real datasets.
2 Related Work
Logic Rule Learning. Learning logic rules from raw data has been widely studied for various downstream tasks, such as motion inference [4] and healthcare analysis [[]. Learning rules via an exact search requires enumerating all combinations of the logic predicates and is intractable in most problems. One has to design heuristic searching algorithms by leveraging some structural properties of the problems. For example, Dash et al. [5] formulated convex rule learning problem and proposed a column generation algorithm to expand the rule set gradually. Wang et al. [25] designed a Bayesian framework for learning rule classifiers and derived bounds on the support of rules in MAP solution.
Recently, Yang et al. (28) proposed an interesting end-to-end differentiable approach (Neural LP) to learn the parameters and structure of logical rules. Qu al. [19) and Sadeghian et al. [22 proposed an efficient logic rule mining algorithm based on the knowledge graph data. However, none of these advanced rule mining methods can directly work on spatial-temporee human action data when the imputs are raw event 3d coordinates and types.
plications (20). Shen et al. 23 proposed a novel deep learning model for
such as taxi data and achieved promising prediction accuracy. Zhou et al. (3 integrated deep learning methods with spatiotemporal point processes and modeled the intensity function as latent stochastic process. Chen et al. K
deployed two novel architectures, including jump and attentive continuous time normalizing flows, to learn the dynamics of the spatiotemporal event data. Repe et al.

---

21 learned canonical spatiotemporal point cloud representation using latent ODE and continuous normalizing flows to generate shapes continuously in spacetime. However, these event models are governed by hard-to- hord--t-interpret dynamic functions and cannot be generalized to model human action events. Could we propose model with logic-informed dynamic functions to explain the spatial-temports human action events?
3
Our Model
3.1 Data: Human Actions Recorded as Spatial-Temporal Event Sequences
Consider a set of objects, denoted as C. For the object E C, its trajectories and the key actions observed up to & can be summarized as a sequence of temporally ordered events:
HG {ei (fi, st, .i)...,, ... (f Su, (R)) th <th,
(I)
where E R the time, 8 E R² is the location, and is E K. is the event (i.e., action) type.
3.2 Definition Spatial-
Predicates
Static Predicate Given the object set C, the predicate is defined as the property or relation of objects, which is logic function as follows:
x() :C x C.... xC ++ (0.1).
(2)
For example, Smokes(c) is property predicate and Friend(c, c') is a relation predicate.
Temporal relation predicates
They can be used to define the temporal relations of two action events. We consider three types of temporal relation predicates (before, equal, after) as:
Raetue (4,, .2)) = (4 12 03.
Raser (11, L2) = (11
12 > 0):
(3)
Regual (f1, f2) = 1 t2).
We will treat the temporal relation predicate either boolean variable real-valued function. If the time information is imprecisely recorded, we can parameterize the temporal relation predicates as temporal kernel funetions that map [0, 1), which is a function of t, t2 with learnabee parameters.
Spatial-Tempon Predicate In our paper, we extend the above static predicates to spatial-temporal predicates, which include spatial-temporal property predicates and spatial-temporal relation predi-
cates. Specifically, the spatial-temporal property predicates are defined as: X(-) : ... C 7 x + {0,1)
(4)
For example, PickupKey (c,t, f, 8) is a spatial-temporal property predicate. Suppose an entity ei picked up the key at time i in location 81, then the predicate will be grounded as True (1) at = 1; otherwise it is false.
(c1,&1, 81), ie., PickupKey(cr)
Given the observational human action data, the grounded predicate {PickupKey (c,t, s) can be modeled as a sequence of discrete events when the predicate becomes True, an event happens. In general, the grounded spatial-temporal property predicate is discrete event sequence, where the event occurrence times and locations are irregular.
The -ertiall relation predicates are introduced to define the spatial and temporal relations of two entities. Specifically, they are defined as:
(5) relations of two objects, where we further divide them into temporal relation predicates, static spatial relation predicates, and dynamic spatial relation predicates. More details can be found in Appendix.
R(:,)) (C x T x S) x 5) (0,
Spatial-temport relation predicates are logic variables indicating the
It is noteworthy that all these boolean predicates can be converted to probabilistic ones. We soften these logic functions by kernel functions with learnable parameters to tolerate uncertainties in data.
3.3 Definition of
Logic Rules
We will consider ppatial-tempora logic rules where the body part contain spatial-temporal predicates relation constraints. For example, sensible rule will look like:
:YTunArmaa (c, f, 8) XPasupkey (c, t, A
(6)
Hinroaa ((d t, (c, 8)) / Huehina t,s"),) (c,t,
3

---

where E Cperson, d' E Colock. and c" E Ckey. "person", "block" and "key" are the specific instances in the object set C, and this rule represents that person wants pick up key, while one block is in front of him and the key is behind him, so he turns around. We utilize this example to manifest the meaning of logic rules. In general, the spatial-tempon logic rule in our paper is defined as a logical connectives of predicates, including property predicates and spatial-temporal relation predicates:
{:Y(o)+
Xpropent (v)
Repucial temporal (u', ,v);
Nprppeyy E.Y.y
terepood ER,y
where (v) is the head predicate evaluated at the aatity-time-locattee triplet v, X2 is the set of property predicates defined in rule f. and R denotes the set of spatial-temporal relation predicates defined in rule f.
3.4 Logic-Informed Action Event Models
We consider setting where we can fully observe the trajectories of all the moving objects, including their real-time locations and key actions (i.e., events), denoted as H4. We aim to propose logic- informed
model to predict and explain the action type given the entity-time antiy-time-location triplet v = (c,t.s) (i.e., query) and H4. The main idea is to construct the model features using spatial-temporal logic rules, as defined in Eq. ( Intuitively, given the entire trajectories H. and the query v = (c, s), the body part of the rule defines the evidence to be selectively gathered from history to deduce the event type for query entity for each possible event type
1
Figure I: Illustration of feature construction using a simple logic formula with temporal relation predicate (f1 < f22). f Y <- A B A C (A Before B). The rule defines the template to gather combinations of the body predi- cate history events. Here pred- icate A has 2 events and predi- cate B has I event, the tempo- ral relation constraint would lead to valid combinations (also called "paths"). This type of feature construction can be extended to spatial-temporal cases, where we count the valid paths as the fea- ture.
(c, t, Assume that K., there exist multiple rules such as Eq. (7) to explain its occurrence, with being the head predicate.
Given an individual rule as Eq. (7), we propose to build the feature that is conditional on history and query as
6 lu, Hi) = sign(s E f)-
Σ
91 (path),
(8)
where we introduce function g-(-) check the body conditions of ff given a "path". We use a simple example to explain how to compute features, as shown in Figure D As illustrated, the feature computes the valid total number of "paths" given the data and query.
Suppose there is rule set Fx. where the event is the head predicate. All the rules will play together to reason about the occurrence of π.. For each f E Fxs one can compute the features as above. Given the rule set Fr,, we model the probability of the event K as a log-linear function of the features, i.e.,
p(K|v, H.) x exp EEEF wy by (cllu,H.).
(9)
where w = [wy]ser 2 0 are the learnable weight parameters associated with each rule. All the model parameters can be learned by maximizing the likelihood, which can be computed using the above Eq. (9). We intend to train a rule generator pe and an evaluator Pw to maximize the likelihood of training data as:
max O00, w) (1, [log Ep, (x/o,, H.,]].
(10)
More details can be found as follows.
4 Our Learning Algorithm
Our goal is to jointly learn the set of spatial-temporal logic rules {Fiche hrek and their weights by the maximum likelihood method, where each rule has general form as Eq. 7
To discover each rule, the algorithm needs to navigate through the combinatorial space considering all the combinations of the property predicates and their spatial and temporal relations. To address this computational challenge, we propose a tractable (functional) EM algorithm that treats the rule set as latent variable z. The rules will be generated by a hidden neural rule generator. The overall learning framework alternates between an E-step, where the posterior distribution of the latent rule space evaluated (rule generation), and the M-step, where the model parameters and rule generator parameters are optimized. Please refer to Figure. 2 for illustration.

---

Rule Generating
Event-type
B
Sporp
O
Head predicate
Trijectoy Input
Rule Generator
Logie Co
Pa
Symbol
Spatio-temporal Relation
Deeader
B
Entities
Intention
Reasening Evaluatio (.(x
High-quality & Pe..(E, (V,))
Logic Reasoning
Figure 2: The overview of our proposed framework. It contains two important processes: rule generating and logic reasoning. Given the past motion of cach entity generates logic rules for the reasoning predictor. The reasoning predictor takes the generated rules as input, and predict the intention of each entity. It is optimized EM algorithm. In the E-step, set of top K rules selected from all generated rules via posterior inference. Finally in the M-step, the rule generator is updated to be consistent with the high-quality rules identified in E-step.
over the last few seconds, the rule generator
Our goal is to maximize the likelihood of the observed human action events (x(()) (e(())==i...... Ji=l...,. Using the chain rule, have
21
log Poo ((x((( (:=I,.))))
log (x ,,0 H4, 17).
(11)
To simplify the notation, we will use Pw (n:(0) to stand for Pa (n:(0) u((), H4(--1) in the following. Given a latent rule set 2, have to marginalize the posterior of to get the above log- le-likelihood. However, the exact inference ()(2)1e(9) approximate the true posterior. We have
is intractable. We will introduce an amortized recognition network
lo (x. Dx (plp= + C(,, w; x)),
(12)
where the first term is the KL divergence of the approximate from the true posterior, and the second term C(O, wi (+) the variational lower bound (ELBO). can be represented as:
C(O, wi (0) - (p) (p..())) + E (p(()]=()) [log [logge. P.. (x. (xx))
(13)
And log P (x) > C(O, w; (0). The bound becomes tight when the approximate posterior matches the true one. Our goal is to optimize the variational parameters 0 and model parameters w from the ELBO lower bound.
4.1 Rule Generator
We deploy Transformer- based framework to model the rule generator Pa. We define the distribution of set of rules as follows:
(=(2 H.) V (2|N, Transs (v, H.)).
(14)
where (-) is multinomial distributions, N is the number of the top rules, and Trans (v, H.) defines distribution over compositional rules with spatial-temporal states. The generative process of the rule set is quite intuitive, where we simply generate N rules to form z. In fact, this pe v. H4) is flexible posterior approximation function, which will be optimized by the EM type algorithm.
We choose transformer over graph neural network (GNN) our baseline because transformer architectures based on self-attention mechanism that is able to capture long-range relationships, as opposed to recurrent neural networks that process sequence elements recursively and can only take into account short-term context. Note that the graph operations GNN are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a changeable graph that consists of various types of nodes and edges.
5

---

4.2 Rule Evaluator
Eq. (9) is our rule evaluator (suppose know the rule content). Here assume the rule content latent, and the rule evaluator is given as
exp (Eren wy ws (xkl,, He))
Pu 2, = En exp (Erer wy w (,(1"),, Hi)
(15)
4.3 Optimization
We optimize the rule generator po and reasoning evaluator Pw maximize the objective in Eq. (00) At each training iteration, we first update the reasoning predictor pw: according to some rules generated by the generator, and then update the rule generator pa-
In our network, the latent rule set will be automatically discovered. The best set of logic rules is approximately obtained by sampling and preserving the top-K rules according to their posterior probabilities. Specifically, as shown in Eq. (14), the posterior probabilities of the latent rule is obtained by Transformer type of encoder, which maps the input observed action trajectories to latent explanatory rule space. Each candidate rule is generated in the latent rule space token-by-token (token means logic variable/predicate in our setting) in a sequential manner meanwhile the posterior probability of each rule sequence can be evaluated. When optimizing the evaluator, we draw several rules Σ for each query and let the evaluator uses try to identify top-K rules 2, from all generated rules å. It is accomplished by taking into account the posterior probabilities of each subset of logie rules 2 with prior from the rule generator pe and likelihood from the reasoning predictor Pa- Specifically, when a series of rules produced from the
predict K. For each query,
rule generator pa. we calculate the weights of each rule 21) as follows:
J(z(9) (k)2)))
[ } log Transo He),
(16)
where A is the set of all candidate event type inferred by logic rules. (2a I, H.) is the probability of rule computed by the generator. For subset of rules 21 2, the log- re-probability can be approximated as: log pa. (27 He. = Exten, ((x + log v (zr [N, Transs (v, Hc)) + const. This equation inspired us to use the distribution ((2r) x exp(Ex()es) (() + log V (2r NN, Transe ())))))) as approximation of the posterior. Each rule 200 sampled from (221) independently can be formed with N logic rules.
Clearly, (1) can be regarded as the quality of candidate rules, with consideration of the evaluator Pw- It calculated as the contribution of rule the correct event type minus the average contribution of this rule to the other candidate responses. A rule is more significant if it obtains a higher score to the correct event type and a lower score other potential predictions.
After getting several high- iigh-tuaitty rules from training data, we further utilize these rules to update the parameters of rule generator pa. Concretely, we regard the generated nigh-quality rules as part of training data, and update the rule generator by maximizing the log- oogliikel as follows:
( Po Po H.) = Σ
log Transø (v, Hc) + const.
(17)
By learning to generate high-qualin rules, the rule generator will reduce the search area and produce better empirical results for the reasoning predictor.
Experiments
In this section, we provide some implementation details and show ablation studies as well as visualization to evaluate the performance of our framework. We compare our model with several
state-of-the-art approaches, including PECNet [14), NMMP (7). STGAT B SOPHIE (22]. STAR
29.. Y-Net 13
MID 6 Social-SSL. 24 Social-VAE 26 Social-Implicit (16), and NSP-SFM
30
5.1
Implementation
We follow the same data prepossessing strategy as PECNet [4] for our method. All models were trained and tested on the same split of the dataset, as suggested by the benchmark. We train the network using Adam optimizer with learning rate of 0.001 and batch size 16 for 500 epochs.

---

MAE=0.10
MAE=0.11
MAE=0.06
MAE=0.17
Truth Learned
Truth Learned
Truth Learned
Learned
Value
2-
Value
2-
Value
Value
0
w2
wD
w1
wD
Rule Weight
Rule Weight
Rule Weight
Rule Weight
Figure 3: Rule discovery and weight learning results 4 synthetic datasets (2.4K seqs).
Table 1: Quantitative results (AD F22// F.20, and accuracy) of trajectory prediction in NBA dataset.
The bold/underlined font represent the best/second best result.
Social Implicit
Social VAE
Times
Y-Net
MID
NSP-SFM
Social-SSL
Ours
1.0s
0.38/0.48
0.45/0.59
0.41/0.52
0.48/0.61
0.45/0.53
0.49/0.66
0.30/0.40
2.0s
0.63/0.93
0.76/1.06
0.67/0.94
0,76/1.08
0.72/0.96
0.77/1.11
0.58/0.88
3.0s
0.94/1.34
1.06/1.40
0.98/1.35
1.06/1
1.00/1.39
1.11/1.46
0.87/1.31
4.0s
1.17/1.61
1.32/1.74
1.18/1.63
1.35/1.78
1.19/1.66
1.37/1.79
1.13/1.60
Acc.
0.69
0.65
0.70
0.68
0.69
0.64
0.73
5.1.1
Datasets
Stanford Drone Dataset. This dataset consists of more than 11,000 persons in 20 scenes captured from the campus of Stanford University in bird's eye view. We follow the [27] standard train-test split, and predict the future 4.8s (12 frames) using past 3.2s (8 frames). Note that SDD dataset does not provide explicit pedestrian's action. Instead, record them abstract encoding of the pedestrian's ' speed and location. The action contains [left, right, straight, turn around].
NBA Vp Dataset. It is collected by NBA using the S tracking system, which reports the trajectories of the ten players and the ball in real basketball games. Each trajectory contains the 2D positions and velocities of the offensive team, consisting of 5 players. We predict the future 10 timestamps (4.0s) based on the historical 5 timestamps (2.0s). Each player's action contains [left, right, straight, turn around, pass, shoot].
5.1.2 Metrics
Here, we adopt two metrics for evaluation: Average Displacement Error (ADF) and Final Displace- ment Error (FDF). Specifically, Ex is deñnedd as the minimum average distance between N predicted trajectories and the ground truth over all the involved entities within the prediction horizon. FDEN defined as the minimum deviated distance of predicted trajectories the predicted time step. Moreover, we also calculate the accuracy and Fl score of event-types predicted from each network.
5.2 Synthetic Experiment
We follow [10] verify our model's rule discovery ability on synthetic datasets with known set of ground-truth rules and weights. The synthetic events are generated from TLPPs [2] with a known of rules and weights. We prepared synthetic datasets, and each setting corresponds to different rule weights, rule length and number, type of temporal relation, and intensity of free predicates. Note that it was originally utilized for the temporal point process, so we modify by adding spatial variables (such "left, right, front, and behind")) to fit in our settings. The weight learning results on 4 synthetic datasets are shown the Figure 3 where 2400 sequences are used for evaluation. Each plot at the bottom compares the genuine rule weights to the learned rule weights and reports the Mean
7

---

(a) GT
(b) NMMP
(c) Ours
Figure 4: Visual results on the NBA dataset. We plot the predicted trajectories from the state-of-the-art method NMMP (b). Ours (c) and ground truth (a). The red/blue color represents players two teams and the green color represents the basketball. Light color represents the past trajectory.
Table 2:
Quantitative results E2D F20 and F1 score) of trajectory prediction in SDD dataset.
The bold/underlined font represent the best/second best result.
Social SSL
Metrics
PECNet
Y-Net
MID
NMMP
STGAT
SOPHIE
NSP-SFM
STAR
Ours
A E2o
20.03
7.85
7.61
14.12
14.43
15.56
6.63
6.52
10.76
6.41
F FDE20
33.86
11.85
14.30
20.68
22.59
24.32
12.23
10.61
17.03
10.23
Fl score
0.37
0,54
0.49
0.41
0.33
0.39
0.53
0.58
0,56
0.59
Absolute Error (MAE). As we can see, almost all truth rule weights are learned correctly. It shows an accurate performance of our algorithm in terms of both the rule discovery and parameter learning.
5.3
Analysis
We compare our method with several state-of-the-ard approaches, and Table presents the qualitative results on the SDD dataset. The proposed model achieved the best performance in ADE, FDE and accuracy. We observe that our method significantly outperforms all baselines measured by ADE and FDE. It achieves an ADE of 6.41 and FDE of 10.23 at K = 20 in SDD datset, which exceeds the previous state-of-te satte-of-the-ee the-art performance of Y-Net [13 by 18.3% on ADE and 13.6% FDE. In Table I our method also achieve higher performance than Y-Net in NBA dataset. This is because Y-Net firstly assume that the waypoint lies straight line segment connecting the sampled goal and the past trajectory, then use a multivariate Gaussian prior centered at the assumed location. This assumption not well suit in other complex conditions, such as the trajectory of players in the NBA dataset.
Compared with MID 6 we also obtain 15.7% ADE and 28.4 28.4% FDE improvement. Note that they carefully design Transformer-basee architecture model the temporal dependencies in trajectories, but ignore the spatial correlation of agents. Our transformer-
network aims at generating high- quality logic rules based on spatial-temporal relation under the principle to maximize the likelihood of the observational human actions. For NSP-SFM (30], it obtains high performance in SDD dataset but can achieve the same level in NBA dataset. It combines physies with deep learning for trajectory prediction and accommodates arbitrary physics models. The limitation of lies in specific physies models, such as pedestrians, and deterministic. So can deal with some strategy- based conditions, including basketball and football games. But our logic-learming method tries to use set of spatial-temporal logic rules with intention variables involved as principles model the dynamics of human actions, not restrained into specific conditions.
Further, The proposed model achieved the best scores in Fl score, which is an balanced metric considering both recall and precision. This is because the rule generator and evaluator can collaborate with each other to reduce search space and learn better rules. More experiments and ablation studies can be found in Supplementary Material.

---

E
Player A Rulc @:
Explanation:
Player A is carrying the ball
D
Player A is defended by player C and player D
Player B is on the left of player A
Player passes the ball to player
G.
Player B Rule @: LoLffff..B FronttBrr))
FR
Explanation:
Player E. and player F is moving the player B
Player B is defended by player E on left side right side
D
Player B is defended by player
G
Player B goes front and carries the ball
B Rule @: BLLLCEE.B)AARRhhGBBAA)OAoooo
Explanation:
Player B is defended by player E on left side
Player B is defended by player G on right side
Player B is defended by player F on front side
Player B shoots at the basket
basketball player
basketbull
Figure 5: Visualization and explanation logic rules in NBA dataset.
To evaluate the robustness our method, We added some noise and randomly removed several tracks in the NBA dataset, then evaluated all methods in Table ("Ours** is the original results without noises or missing tracks). As we can see, our method still achieves the best performance. Moreover, by comparing "Ours" and "Ours*", we can see that the quality of tracks has less influence on our method, which demonstrates the robustness of our method.
Our method not only outperforms all the above comparative methods in quantitative evaluations, but also produces visually pleasing results among them. Aegure the predicted trajectories of NMMP, ground-truth (GT) and our methods NBA dataset. It is obvious that our method produces more precise predictions than state-of-the art method NMMP. This is because our proposed spatial-temporal logic rules can actually captures the spatial relation of the players with surrounding players, as well as the temporal ordering constraints the events.
5.4 Generated Logic Rules
We add visualization and explanation about the logic rule and corresponding actions from NBA dataset Figure 5 Note that the static spatial relation [Left (B, } represents that the player is on the left of player A. And the dynamic spatial relation (Away (A, E) means that player A is getting away from player We can see that these logic rules are meaningful and diverse. In this picture, player A is defended by two players and then passes the ball to player B. Player B goes front and crossover to bypass three defenders and shoot at the basket. Our rule can actually represent their offensive strategy. In fact, our framework can actually adapt to some complex motions, such as cutting toward the ball, because these spatial-temporal predicates are fed into the rule generator and evaluator to obtain high-quality rules to explain the intention of players.

---

Table 3: Robustness of our method in NBA dataset some noise.
2.0
3.0
4.0
Methods
ADE
FDE
ADE
FDE
ADE
FDE
ADE
FDE
Y-Net
0.51
0.62
0.81
1.11
1.08
1.49
1.37
1.86
NSP-SFM
0.48
0.63
0.82
1.13
1.06
1.46
1.36
1.82
Ours
0.31
0.48
0.67
0.91
0.93
1.39
1.18
1.66
Ours
0.30
0.40
0.58
0.88
0.87
1.31
1.13
1.60
6
Limitation
It's challenging to define some complex predicates with richer meanings for different datasets. Given a more informative dataset, our method can discover more principles-like complex rules. Our motivation is consider the spatial-temporaa relation between pedestrians and generate some high-quality logic rules to explain their behaviors. Although we only choose some simple actions in experiments, they bring some benefits for understanding the principles of biological agents' behaviors. Our framework is suitable for more complex conditions, supposing more sophisticated action predicates can be obtained from the data.
7
Conclusion
We proposed a framework for learning intrinsic spatial-temporal logic rules for explaining human actions. We regard logic rules as latent variables, and the rule generator as well as the rule evaluator are jointly learned with EM-based algorithm. In the experiments, our method can analyze the biological movement sequence pedestrians and players, and obtained novel insights from generated logic rules. In the future, plan to incorporate other physical laws into the models, such conservation of energy and momentum to enhance robustness of our model.
Acknowledgement
Shuang Li's research was in part supported by the NSFC under grant No. 62206236, Shenzhen Science and Technology Program JCY12021032412001100 National Key R&D Program of China under grant No. 20227D01 16004, and Guangdong Key Lab Mathematical Foundations for Artificial Intelligence.
References
[1] David Adam. Science and the world cup: how big data is transforming football. Nature, 611(7936):4444466 2022.
[2] Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert Systems with Applications, 165:1 133:1, 2021.
[3] Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference Learning Representations, 2021.
[4] Chang Choi, Junho Choi, Eunji Lee, Ilsun You, and Pankoo Kim. Probabilistic spatio-temporal inference for motion event understanding. Neurocomputing, 122:24-32, 2013.
[5] Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances neural information processing systems, 31, 2018
[6] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 17113-17122,
2022.
10

---

[7] Yue Hu, Siheng Chen, Ya Zhang, and Xiao Gu. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 6319 12--6228, 2020.
[8] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6272-6281, 2019.
[9] Rainer Kartmann, You Zhou, Danqing Liu. Fabian Paus, and Tamim Asfour. Representing spatial object relations parametric polar distribution for scene manipulation based on verbal commands. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8373 8380. IEEE, 2020.
[10] Shuang Li, Mingquan Feng, Lu Wang, Abdelmajid Essofi, Yufeng Cao, Junchi Yan, and Le Song. Explaining point processes by learning interpretable temporal logic rules. In International Conference on Learning Representations, 2021.
[11] Shuang Li, Mingquan Feng, Lu Wang, Abdelmajid Essofi, Yufeng Cao, Junchi Yan, and Le Song. Explaining point processes by learning interpretable temporal logic rules. In International Conference on Learning Reprexentations, 2022.
[12] Shuang Li. Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song. Temporal logic point processes. pages 5990-6000, 2020.
[13] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals. waypoints paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15233-15242. 2021.
[14] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 759-776. Springer, 2020.
[15] Andrew Miller, Luke Bornn, Ryan Adams, and Kirk Goldsberry. Factorized point process intensities: A spatial analysis of professional basketball. In International conference on machine learning, pages 235-243. PMLR, 2014.
[16] Abduallah Mohamed, Deyao Zhu, Warren Vu, Mohamed Elhoseiny, and Christian Claudel. Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In European Conference Computer Vision, pages 463-479. Springer, 2022.
[17] Filmer Stuart Cuckow Northrop. The logic of the sciences and the humanities. 1947.
[18] Clayton Peterson. A logic for human actions. Applications of Formal Philosophy: The Road Less Travelled, pages 73-112, 2017.
[19] Meng Qu, Junkun Chen, Mouis-Pascaa Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In International Conference on Learning Representations, 2021.
[20] Alex Reinhart. A review of self-exciting spatio-temporal point processes and their applications. Statistical Science, 33(3):299-318, 2018
[21] Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojeic, Srinath Sridhar, and Leonidas J Guibas, Caspr: Learning canonical spatiotemporal point cloud representations. Advances in neural information processing systems, 33: 13688-13811,,, 2020.
[22] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems, 32, 2019.

---

[23] Bilong Shen, Xiaodan Liang, Yufeng Ouyang, Miaofeng Liu, Weimin Zheng, and Kathleen M Carley. Stepdeep: A novel spatial-temporal mobility event prediction framework based on deep neural network. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery data mining, pages 724-733, 2018.
[24] Li-Wu Tsao, Yan-Kai Wang. Hao-Siang Lin, Hong-Han Han Shuai, Lai-Kuan Wong, and Wen- Huang Cheng. Social-ssl: Self-supervised cross-sequence representation learning based transformers for agent trajectory prediction. Computer Vision-ECCV 2022: 17th European Conference. Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 250. Springer, 2022.
[25] Tong Wang, Cynthia Rudin, Finale Doshi-Velez. Yimin Liu, Erica Klampfl, and Perry MacNeille. bayesian framework for learning rule for interpretable classification. The Journal of Machine Learning Research, 18(1):2357-2393 2017.
[26] Baowen Xu, Xuelei Wang. Shuo Li, Jingwei Li, and Chengbao Liu. Social-evae: Pedestrian trajectory prediction using conditional variational auto-encoder. International Conference on Neural Information Processing, pages 489. Springer, 2023.
[27] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2022.
[28] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems, 30, 2017.
[29] Cunjun Yu, Xiao Ma. Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Osion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 507-523 Springer, 2020.
[30] Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Eision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 376-394. Springer, 2022.
[31] Zihao Zhou, Xingyi Yang. Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference, pages 777-789. PMLR, 2022.
12


Reviews:
[{"review_id": "wj9YYK2oun", "reviewer": "Reviewer_gEM9", "summary": "This study presents a logic-informed, knowledge-driven modeling framework designed to predict and understand human movements, based on the analysis of their trajectories. It takes into account that human behaviors are commonly guided by intentions, desires, and spatial relationships with surrounding objects. The research integrates a set of spatial-temporal logic rules, derived from observational data, into the model, utilizing an expectation-maximization (EM) algorithm to infer model parameters and rule content. The performance of the model is evaluated using datasets of NBA basketball player movements and pedestrian trajectories, demonstrating high levels of interpretability, predictive accuracy, and efficacy in its results.", "questions": "- There is a typo in equation (3); the correct symbol should be $\\varphi_f$ instead of $\\phi_f$ .\n- The references for the datasets utilized in the experiments are currently absent and need to be included.\n- Where did you get the heatmap depicted in Fig3 (b) (d)?", "limitations": "If my understanding is correct, the spatial-temporal logic rule space, encompassing both static spatial and dynamic spatial relations, needs to be explicitly defined. This implies that prior knowledge or assumptions about the environment and the interactions within it are required to structure and define these rules. Thus, it's not a process automatically derived from the data but requires human intervention and understanding to set up appropriately.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The objectives and motivation are well-articulated, offering a clear understanding of the reasons behind the research and the problem it aims to address.\n- The performance of the proposed model is outstanding when compared to other state-of-the-art methods signifying an advantage of the work.\n- The proposed method is based on theoretical principles (EM method). It's not a product of arbitrary choices, but rather a thoughtful construction based on the EM algorithm, which adds credibility to the model's outcomes.", "weaknesses": "- If additional specifics regarding the architecture could be provided, such as the input and output formats utilized by the transformer, it would be incredibly beneficial for a deeper understanding of the method.\n- The evaluation of this work is currently based on only two datasets. If the authors could expand this to include results from more universally employed pedestrian trajectory prediction datasets (ETH [i] or UCY [ii] ), it would significantly strengthen the claim of the model's generalizability.\n  \n  [i] Improving data association by joint modeling of pedestrian trajectories and groupings ECCV10.  \n  [ii] Crowds by example. In Computer Graphics Forum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your recognition of our work and your insightful reviews! We hope our responses can address your questions. Our responses are listed below.\n\n**Q1: If additional specifics regarding the architecture could be provided, such as the input and output formats utilized by the transformer, it would be incredibly beneficial for a deeper understanding of the method.**\n\nA1: Thanks for your suggestion. The input of our framework is the historical trajectory coordinates of entities, and the output is the predicted future trajectory. Coordinates as input would be first encoded into a vector by three-layer MLPs before being fed into the Transformer, and a ReLU nonlinearity following each of the first two layers. The dimensions of keys, values, and queries are all set to 256, and the hidden dimension of feed-forward layers is 512. The number of heads for multi-head attention is 8. We will add these details in a revision.\n\n**Q2: The evaluation of this work is currently based on only two datasets. If the authors could expand this to include results from more universally employed pedestrian trajectory prediction datasets (ETH [i] or UCY [ii] ), it would significantly strengthen the claim of the model's generalizability.**\n\nA2: We have evaluated our method and other state-of-the-art methods in the ETH/UCY dataset, and **due to the page limitation** in the main paper, the results are shown in **the supplementary material Table 6**. Below we show the results for the reviewer’s convenience. These new experimental results show that we still achieve superior results in most metrics.\n\n| Methods  | ETH | HOTEL | UNIV | ZARA1 | ZARA2 |AVG|\n|--|--|--|--|--|--|--|\n|Y-Net          |0.28/0.33|0.10/0.14|0.24/0.41|0.17/0.27|0.13/0.22|0.18/0.27|\n|MID            |0.39/0.66|0.13/0.22|0.22/0.45|0.17/0.30|0.13/0.27|0.21/0.38|\n|NSP-SFM        |0.25/0.44|0.09/0.13|0.21/0.38|0.16/0.27|0.12/0.20|0.17/0.24|\n|Social SSL     |0.69/1.37|0.24/0.44|0.51/0.93|0.42/0.84|0.34/0.67|0.44/0.85|\n|Social Implicit|0.66/1.44|0.20/0.36|0.32/0.60|0.25/0.50|0.22/0.43|0.33/0.37|\n|Social-VAE     |0.41/0.58|0.13/0.19|0.21/0.36|0.17/0.29|0.13/0.22|0.21/0.33|\n|ABC+           |0.31/0.44|0.16/0.21|0.25/0.47|0.21/0.28|0.20/0.26|0.23/0.32|\n|Ours           |**0.22/0.30**|**0.07/0.13**|**0.16/0.34**|**0.14/0.25**|**0.07/0.16**|**0.13/0.24**|\n\nTable: Quantitative results (ADE20/FDE20) of trajectory prediction in ETH/UCY dataset. \n\n**Q3: There is a typo in equation (3); The references for the datasets utilized in the experiments are currently absent and need to be included.**\n\nA3: These typos and references (SDD dataset [1] and ETH [2]/UCY [3] dataset) have been modified in a revision. \n\n\n[1] Robicquet A, Sadeghian A, Alahi A, et al. Learning social etiquette: Human trajectory understanding in crowded scenes[C]//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer International Publishing, 2016: 549-565.\n\n[2] Pellegrini S, Ess A, Schindler K, et al. You'll never walk alone: Modeling social behavior for multi-target tracking[C]//2009 IEEE 12th international conference on computer vision. IEEE, 2009: 261-268.\n\n[3] Lerner A, Chrysanthou Y, Lischinski D. Crowds by example[C]//Computer graphics forum. Oxford, UK: Blackwell Publishing Ltd, 2007, 26(3): 655-664.\n\n**Q4: Where did you get the heatmap depicted in Fig3 (b) (d)?**\n\nA4: Following [4][5], we utilize Fig. 3 to show the heatmaps of goals and waypoints for t=30 seconds in the future from the estimated distribution. These heat maps are achieved through an estimated distribution map obtained by the decoder in our framework.\n\n[4] Mangalam K, An Y, Girase H, et al. From goals, waypoints & paths to long term human trajectory forecasting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 15233-15242.\n\n[5] Jacobs H O, Hughes O K, Johnson-Roberson M, et al. Real-time certified probabilistic pedestrian forecasting[J]. IEEE Robotics and Automation Letters, 2017, 2(4): 2064-2071."}, {"review_id": "6HdNkyFF80", "reviewer": "Reviewer_zUPP", "summary": "The paper proses a method for learning spatio-temporal logic rules to explain human actions, utilizing the EM framework. The method results are more easy interpretable by humans, thanks to the logic rules. The method beats some state-of-the-art methods on two real-world motion prediction datasets.", "questions": "It seems that using the full 3D human pose would be a much better indicator for human actions, interactions with objects, and interactions with other persons. Why did the authors use only trajectory data? For example, the Haggling dataset [a] contains triadic interactions under a well-defined social protocol which could have been utilized.\n\nL94: “Consider a set of objects denoted as C”: does object mean humans? This should be explicitly stated if true.\n\nL96: could the authors elaborate how k is encoded?\n\n—\n\n[a] Liu, Zhenguang, et al. \"Towards natural and accurate future motion prediction of humans and animals.” CVPR 2019.", "limitations": "The methods challenges are discussed to some degree.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The method is well-motived, i.e. that human actions are driven by intention as well as social and environmental factors. The explainability of the model is impressive and well-visualized.", "weaknesses": "My major concern with this work is the need for dataset-depended actions. Biasing the model with additional labels, which are not provided to other methods, allows it to spend more capacity on forecasting, effectively producing an unfair advantage. It would have been better if the authors provide an approach that can automatically extract the actions, without human pre-selection.\n\nWhile the general intention is well-motivated, the method overview and method motivation in the introduction section is not well described, making it difficult to get a first good understanding of what the method tries to achieve and how this is done. For example, the authors explain that “[] logic rules present a compact knowledge representation” (L19) but they do not further explain what “logic” they are using and how it is defined. \n\nIn the method section a lot of important details are missing, making it difficult to follow:\n* In the Equations in L104.5 and L113.5 the authors utilize variable v and v’ without introducing what this variable means. The authors should explain the variable appropriately. \n* In L122 the authors introduce various new object types “person”, “block” and “key” which are not explained. \n* In L117 the paper says that the binary predicated values can be softened into “probabilistic” ones using kernel functions without details or reference to other parts of the paper. This makes it particularly difficult to understand as the paper has not yet revealed how the logic rules are applied to obtain the predictions.\n\nSpatio-temporal predicates (L104) contain the time and space dimension. This necessitates that both time and area must be discretized. The discretization rate would be an important hyper-parameter and should be ablated accordingly. \n\nWhy do the authors not follow the experimental setup in [23] and also evaluate on the ETH-UCY dataset? Can the dataset not be labeled with actions? If so: why? What makes the dataset difficult for this method setup?\n\nThe organization of the experiments section is not convincing: while the authors conducted a lot of experiments they should have added them into the main paper.\n\n**Minor issues**:\n\nThe authors claim their models “superior interpretability” (L15) but do not properly explained in text (only attempted in Figure 5). This however should be explained briefly.\n\nEquations should be numbered to make referencing them easier", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful reviews! We hope our response below addresses your concerns. \n\n**Q1: My major concern with this work is the need for dataset-depended actions**\n\nA1: Thanks for your suggestion. To learn specific actions would require learning recursion and predicate invention [a]. Invented predicates can be interpreted as a set of phrases to express the meaning of actions. Following [a], we embed meta-interpretive learning (MIL), which supports efficient predicate invention and learning of recursive logic programs built as a set of metalogical substitutions by a modified Prolog meta-interpreter, into our framework and evaluate its performance in the NBA dataset. Obviously, this operation actually brings slight improvements.\n\n\n| Times  | Ours | Ours+ |\n|--|--|--|\n| 1.0s | 0.30/0.40 | 0.28/0.37 |\n| 2.0s | 0.58/0.88 | 0.58/0.87 |\n| 3.0s | 0.87/1.31 | 0.85/1.28 |\n| 4.0s | 1.13/1.60 | 1.11/1.54 |\n\nTable: Quantitative results (ADE/FDE) of trajectory prediction in NBA dataset. “Ours+” means that we automatically extract the actions.\n\n[a] Muggleton S H. Meta-interpretive learning of higher-order dyadic datalog: Predicate invention revisited[J]. Machine Learning, 2015.\n\n\n**Q2: They do not further explain what “logic” they are using and how it is defined.**\n\nA2: Indeed, “logic” emphasizes high-level reasoning, and encourages structuring the world in terms of objects, properties, and relations [b]. “logic” can provide the formal machinery to reason about some concepts, such as time, space, abstraction, and causality, in a rigorous way.\n\n[b] Belle V. Symbolic logic meets machine learning: A brief survey in infinite domains[C]//International conference on scalable uncertainty management. 2020.\n\n**Q3: In the method section a lot of details are missing ...**\n\nA3: (1) v and v’ represent the entity-time-location triplet (described in Line 125), so X(v) and R(v,v’) are logic random variables that define the property or relation of entities. (2) “person”, “block” and “key” are the specific instances in the object set C, so that in Line 121, this rule represents that a person wants to pick up a key, while one block is in front of him and the key is behind him, so he turns around. We utilize this example to manifest the meaning of logic rules. (3) Following [c], we use the softened representation and a weighted combination of logic rules to deal with uncertainty in events. The soft constraint has been utilized in other papers (Ref [10] [24] in the main paper).\n\n[c] Li S, et al. Temporal logic point processes[C]// ICML, 2020.\n\n**Q4: The discretization rate would be an important hyper-parameter and should be ablated accordingly.**\n\nA4: The discretization rate can directly influence the computation cost and the loss of information. The larger discretization rate means more loss of information but less computation cost. In fact, our method can tackle different discretization rates properly. In the main paper, we set the smallest discretization rate in our experiment. Following the reviewer’s advice, we also set different discretization rates (presented as d, and \"d=0.1\" means that we remove 10% trajectory data regularly) in the SDD dataset and obtain some results in the following table.\n\n| Metric | d=0.1 | d=0.2 | d=0.3 | d=0.4 |\n|--|--|--|--|--|\n|ADE  | 8.24 | 12.21 | 15.03 | 19.49 |\n|FDE  | 16.74|19.58  | 24.98 | 30.87  |\n\nTable: Ablation study of discretization rate in SDD dataset.\n\n**Q5: Why do the authors not follow the experimental setup in [23] and also evaluate the ETH-UCY dataset?**\n\nA5: In Line 235, we have demonstrated that “We follow the [23] standard train-test split, and predict the future 4.8s”. Moreover, we have evaluated our method in the ETH-UCY dataset and the results are shown **in the supplementary material Table 6**. Clearly, this dataset can be labeled with actions and our method can obtain great performance in this dataset. \n\n**Q6: The organization of the experiments section is not convincing.**\n\nA6: Due to the page limitation, most of the experimental results are put into supplementary material, including ablation studies of backbones, additional results in the ETH/UCY dataset, etc. We will add more results in the main paper.\n\n**Q7: The authors claim their models have “superior interpretability” (L15) ...**\n\nA7: In Figure 5, we have added some explanation about the logic rule from the NBA dataset to show the superior interpretability of our method. Each rule can vividly manifest the corresponding actions of each player. Additionally, we also show more generated rules **in the attached PDF Table 1**. These rules also conform to the common sense. All equations will be numbered in the final version.\n\n**Q8: The Haggling dataset [a] contains triadic interactions which could have been utilized.**\n\nA8: We choose trajectory data because it’s easy to define spatial-temporal relation predicates and object property to explore some intrinsic logic rules. Moreover, following the reviewer’s advice, we also evaluate our method in the Haggling dataset, and the results are shown in the following table. Our method achieves promising results for both short-term and long-term predictions of complex activities. **More results are shown in the attached PDF Table 3 and Figure 1.**\n\n|Time|80ms|160ms|320ms|400ms|560ms|640ms|720ms|1000ms|\n|--|--|--|--|--|--|--|--|--|\n|Walking|0.38|0.58|0.80|0.89|0.98|1.03|1.11|1.22|\n|Eating|0.25|0.39|0.60|0.76|0.94|1.01|1.03|1.29|\n\nTable: Performance evaluation (in MAE) of comparison methods over on the Haggling (H3.6) dataset.\n\n**Q9: Does object mean humans?**\n\nA9: Generally, the object set C represents all objects in the specific environment. In our settings, all entities are human, so the “object” means human.\n\n**Q10: Could the authors elaborate on how k is encoded?**\n\nA10: Note that we have defined several event types k in Line 238 and Line 243. These actions can be encoded from the history of each entity as a booling logic variable. "}, {"review_id": "eUSyIFAOIi", "reviewer": "Reviewer_7WeJ", "summary": "This paper presents a novel approach for human trajectory prediction, introducing a learnable rule-based framework that combines rule generation/reasoning and EM optimization. Unlike previous works in the field, this framework utilizes a neural rule generator to generate rules and treats them as latent variables. Additionally, it selects the top k rules from the entire rule set to predict future trajectories. The experimental results on two datasets demonstrate the effectiveness of the proposed framework.", "questions": "- Regarding the latent rule space, what is the dimension of the latent space z?\n\n- I have reservations about fully accepting the argument presented in L207 that \"our method can discover more principles-like complex rules.\" The generation of more rules for complex conditions necessitates a larger latent space (higher dimension), which can pose challenges in generating high-quality rules.\n\n- In L207-L208, it is mentioned that \"For each query, we aim to identify top K rules $z_I$ from all generated rules $\\hat{z}$,\" with K set to 5 in the experiments. Not all movements require up to K rules. Some movements may be simple and only require 1-2 rules (as seen in the second example in Figure 2). Is it possible to automatically determine the number of rules based on their weights?", "limitations": "The authors have provided a discussion of the limitations in Section 6.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-structured and written in a clear manner, making it easy to follow. The authors have used appropriate notation and included equations where necessary, enhancing the readability of the paper.\n\n- The detailed analysis and visualizations presented in sections 5.4-5.6 and the ones in the supplementary materials contribute to a better understanding of the proposed method. I found them enjoyable and informative to read.\n\n- The paper introduces a novel framework that incorporates separate steps for rule generation and logic reasoning. This approach is original and reasonable for the given task.\n\n- The proposed framework demonstrates improved results on the Stanford Drone Dataset and NBA SportVU dataset.\n\n- The authors have put effort into providing numerous illustrative examples that showcase the behavior of the proposed framework. The explanation of logic rules in the NBA dataset (Figure 5) is particularly interesting.", "weaknesses": "- The predicates are required to be manually defined and these predicates may need to be redefined when applied to new scenarios, such as transitioning from 2D to 3D scenarios.\n\n- ETH/UCY benchmarks have been widely used for benchmarking short-term trajectory prediction but missing from this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your recognition of our work and your insightful reviews! We hope our responses can address your questions. Our responses are listed below.\n\n**Q1: The predicates are required to be manually defined and these predicates may need to be redefined when applied to new scenarios, such as transitioning from 2D to 3D scenarios.**\n\nA1: Thanks for your advice. To demonstrate the benefit of our method in 3D scenarios, we also evaluate it in the 3D motion prediction dataset, called Haggling dataset [1], over 7 different action types, and the results are shown in Table 1. Our method achieves promising results for both short-term and long-term predictions of complex activities.\n\n|Time|80ms|160ms|320ms|400ms|560ms|640ms|720ms|1000ms|\n|--|--|--|--|--|--|--|--|--|\n|Walking|0.38|0.58|0.80|0.89|0.98|1.03|1.11|1.22|\n|Eating|0.25|0.39|0.60|0.76|0.94|1.01|1.03|1.29|\n|Discussion|0.31|0.57|0.88|0.99|1.48|1.65|1.81|1.96|\n|Phoning|0.55|0.83|1.22|1.35|1.58|1.65|1.72|1.92|\n|Posing|0.27|0.56|1.19|1.48|1.93|2.14|2.29|2.58|\n|Sitting|0.40|0.63|1.02|1.18|1.28|1.34|1.40|2.02|\n|Waiting|0.36|0.69|1.25|1.46|1.80|1.95|2.12|2.57|\n\nTable: Performance evaluation (in MAE) of comparison methods over on the Haggling (H3.6) dataset.\n\n[1] Liu Z, Wu S, Jin S, et al. Towards natural and accurate future motion prediction of humans and animals[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 10004-10012.\n\n**Q2: ETH/UCY benchmarks have been widely used for benchmarking short-term trajectory prediction but missing from this paper.**\n\nA2: We have evaluated our method in the ETH/UCY dataset, and **due to the page limitation, we put the result in the supplementary material Table 6**. Below we show the results for the reviewer’s convenience. These new experimental results show that we still achieve superior results in most metrics.\n\n| Methods  | ETH | HOTEL | UNIV | ZARA1 | ZARA2 |AVG|\n|--|--|--|--|--|--|--|\n|Y-Net          |0.28/0.33|0.10/0.14|0.24/0.41|0.17/0.27|0.13/0.22|0.18/0.27|\n|MID            |0.39/0.66|0.13/0.22|0.22/0.45|0.17/0.30|0.13/0.27|0.21/0.38|\n|NSP-SFM        |0.25/0.44|0.09/0.13|0.21/0.38|0.16/0.27|0.12/0.20|0.17/0.24|\n|Social SSL     |0.69/1.37|0.24/0.44|0.51/0.93|0.42/0.84|0.34/0.67|0.44/0.85|\n|Social Implicit|0.66/1.44|0.20/0.36|0.32/0.60|0.25/0.50|0.22/0.43|0.33/0.37|\n|Social-VAE     |0.41/0.58|0.13/0.19|0.21/0.36|0.17/0.29|0.13/0.22|0.21/0.33|\n|ABC+           |0.31/0.44|0.16/0.21|0.25/0.47|0.21/0.28|0.20/0.26|0.23/0.32|\n|Ours           |**0.22/0.30**|**0.07/0.13**|**0.16/0.34**|**0.14/0.25**|**0.07/0.16**|**0.13/0.24**|\n\nTable: Quantitative results (ADE20/F DE20) of trajectory prediction in ETH/UCY dataset. The\nbold font represents the best result.\n\n**Q3: Regarding the latent rule space, what is the dimension of the latent space z?**\n\nA3: The dimension of latent embedding in our method is set to 32 on all datasets. We will add it in a revision.\n\n**Q4:I have reservations about fully accepting the argument presented in L207.**\n\nA4: Thanks for your advice. Complex rule generation actually requires higher-dimensional latent space, which is also our future work. We will modify this sentence as “our method has the potential to discover more principles-like complex rules”.\n\n**Q5: Is it possible to automatically determine the number of rules based on their weights?**\n\nA5: Yes, the number of rules can also be automatically learned by neural networks or other deep learning methods. In our settings, we manually set K as the upper limit to reduce the computation cost. In E-step, some movements are so simple that they only require less rules because most of the candidate rules’ weights are minimal. Moreover, following the reviewer’s advice, we set a weight threshold to automatically determine the number of rules, and the results are shown in the following table. Obviously, this operation can bring slight improvements.\n\n\n| Times  | Ours | Ours++ |\n|--|--|--|\n| 1.0s | 0.30/0.40 | 0.29/0.38 |\n| 2.0s | 0.58/0.88 | 0.57/0.86 |\n| 3.0s | 0.87/1.31 | 0.85/1.29 |\n| 4.0s | 1.13/1.60 | 1.11/1.55 |\n\nTable: Quantitative results of trajectory prediction in NBA dataset. “Ours++” means that we automatically determine the number of rules."}, {"review_id": "APjoWcnGjB", "reviewer": "Reviewer_AEbP", "summary": "This paper proposes a model for inferring rules based on observation trajectories of (entity, time, location), with the aim of maximizing the probabilities of certain \"events\". The aim is to predict the next events based on past trajectories, conditioned on the latent \"rules\" which need to be marginalized over. But this is intractable, so an \"encoder\" or \"recognition\" network is used, similar to the logic of variational autoencoder. In the encoder, a transformer is used to define the probability distribution over possible rules based on input trajectories, and a decoder can sample these rules to generate the next event. The model is used for trajectory prediction tasks on two datasets with better results than existing trajectory prediction model. Examples of rules learnt are also given in case of one dataset.", "questions": "1) Can we have an experiment on synthetic setting with ground truth data generated according to specific rules (eg. like a cellular automata), and then show if the proposed model can recover the rules?\n2) Please add an algorithm to explain how the trajectory prediction is carried out, step-by-step.\n3) What can be other applications of this framework beyond trajectory prediction? Also, can the \"rules\" here be related to \"policies\" to choose actions as in Reinforcement Learning?\n4) Can the rule probability distribution be modelled by a cheaper model than Transformer?\n\n\n", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) A nice framework is proposed for representation of spatio-temporal action rules based on logic\n2) The \"rules\" are expressed as a latent variable model, where inference can be done using E-M algorithm applied to an encoder-decoder model\n3) Strong results are shown on trajectory prediction tasks, sample rules are also shown\n\nIn my opinion, the rule definition and rule inference framework is quite general and can find wider applications than the experiments described in the paper.", "weaknesses": "It is not very clear how exactly the goal states are estimated (eg. in Fig 3), or how the future trajectories are generated (Fig 3 and experiments), I get the broad idea, but precision is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your recognition of our work and your insightful reviews! We hope our responses can address your questions. Our responses are listed below.\n\n**Q1: It is not very clear how exactly the goal states are estimated (eg. in Fig 3), or how the future trajectories are generated.**\n\nA1: Following [1][2], the goal and waypoint heatmap are extracted from the decoder in our framework. We estimate distributions of future waypoint positions which along with the goal points are used to obtain explicit maps over all the remaining intermediate trajectory positions.\n\n[1] Mangalam K, An Y, Girase H, et al. From goals, waypoints \\& paths to long-term human trajectory forecasting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 15233-15242.\n\n[2] Jacobs H O, Hughes O K, Johnson-Roberson M, et al. Real-time certified probabilistic pedestrian forecasting[J]. IEEE Robotics and Automation Letters, 2017, 2(4): 2064-2071.\n\n**Q2: Can we have an experiment on the synthetic setting with ground truth data generated according to specific rules (eg. cellular automata), and then show if the proposed model can recover the rules?**\n\nA2: Yes. Thanks for your advice. We follow [3] to verify our model’s rule discovery ability on synthetic datasets with a known set of ground-truth rules and weights. Note that it was originally utilized for the temporal point process, so we modify it by adding spatial variables (such as “left, right, front, and behind”) to fit in our settings. The weight learning results on 4 synthetic datasets are shown in the following table. More results are  shown **in the attached PDF Figure 2.**\n\n\n| Weight  | Dataset-1 | Dataset-2 | Dataset-3 |Dataset-4|\n|--|--|--|--|--|\n| w0 | 1.0/0.98 | 0.5/0.45 |1.5/1.47|2.0/1.82|\n| w1 | 1.0/0.91 | 0.5/0.40 |1.5/1.44|1.0/0.97|\n| w2 | 1.0/0.81 | 0.5/0.34 |1.5/1.39|1.0/0.92|\n\nTable: Rule discovery and weight learning results (GT weights/learned weights) on 4 synthetic datasets.\n\n[3] Li S, Feng M, Wang L, et al. Explaining point processes by learning interpretable temporal logic rules[C]//International Conference on Learning Representations. 2021. \n\n**Q3: Please add an algorithm to explain how the trajectory prediction is carried out, step-by-step.**\n\nA3: Following the reviewer's advice, we explain the trajectory prediction step by step.\n\n---\n\n**Input**: observed history trajectory $H$ of entity $v$, time horizon $T$, the number of rules $N$\n\n**while** not converge **do**\n\n$\\qquad$ Use the rule generator $p_\\theta$ to generate a set of rules $z$ by $p_\\theta(z | v, H_t) = Ψ(z|N, Trans_\\theta(v, H_t))$;\n\n$\\qquad$    Update the predictor $p_w$ based on generated rules $z$;\n\n$\\qquad$    Identify K high-quality rules from $z$ according to Eq.10;\n\n$\\qquad$    Update the rule generator $p_\\theta$ according to the identified rules;\n\n**end**\n\n Use $p_\\theta$ to generate rules and feed them into $p_w$ for prediction;\n\n**for** $i = 1,\\dots,T$ **do**\n\n$\\qquad$    Calculate posterior probability $P(z_i|CurrentPosition)$;\n\n**end**\n\nUpdate parameters based on computed posterior probabilities;\n\n**for $i = 1,\\dots,T$ do**\n\n$\\qquad$    Compute the weighted average of predicted positions from each component;\n\n**end**\n\n---\n\n**Q4: What can be other applications of this framework beyond trajectory prediction? Also, can the \"rules\" here be related to \"policies\" to choose actions as in Reinforcement Learning?**\n\nA4: (1) This framework can also be applied to other spatial-temporal events, such as social analysis, mobile robots and epidemic forecasting. (2) Yes, the generated rule can be utilized to choose actions in Reinforcement Learning. It's our future work to represent the policies in reinforcement learning by first-order logic based on policy gradient methods and differentiable inductive logic programming, which has significant advantages in terms of interpretability and generalisability.\n\n\n**Q5: Can the rule probability distribution be modeled by a cheaper model than Transformer?**\n\nA5: Yes, the rule probability distribution be modeled by other backbones, such as CNN. In the supplementary material Section 7, we have compared our method (transformer-based) with three widely used backbones, including CNN, RNN, and GNN (graph neural network), and evaluated them in the NBA dataset. The results have been shown **in supplementary material Table 3**. Obviously, our architecture can actually achieve superior results in all metrics. Below we show the results for the reviewer’s convenience.\n\n| Method  | 1.0s | 2.0s | 3.0s | 4.0s|\n|--|--|--|--|--|\n| CNN | 0.41/0.60 | 0.78/1.07 |0.98/1.53|1.24/1.76|\n| GNN | 0.38/0.59 | 0.76/1.03 |0.96/1.49|1.19/1.69|\n| RNN | 0.36/0.49 | 0.69/1.00 |0.95/1.39|1.17/1.67|\n| Ours |0.30/0.40 | 0.58/0.88 |0.87/1.31|1.13/1.60|\n\nTable: Comparison of different backbones in the rule generator in the SDD dataset. "}]

Author rebuttal:
Dear Reviewers, Area Chairs, and Program Chairs,

We are greatly thankful for the insightful comments and suggestions, which are very helpful for us to further improve this work. We are very excited that the reviewers hold positive feedback and find our work "well-articulated, offering a clear understanding....the performance of the proposed model is outstanding" (Reviewer gEM9), "the explainability of the model is impressive and well-visualized" (Reviewer zUPP), "introduces a novel framework that incorporates separate steps for rule generation and logic reasoning" (Reviewer 7WeJ), "the rule definition and rule inference framework is quite general and can find wider applications than the experiments" (Reviewer AEbP). In our response, we provide additional figures and tables in the attached PDF for visualization. Recognizing the importance of additional explanations and experiments, we are committed to making these enhancements. To ensure transparency and clear communication, we've summarized our main responses as follows:

- **Experiment on ETH/UCY dataset.** We would like to address the fact that some reviewers may not have had the opportunity to review our supplementary material, leading to concerns regarding the ETH/UCY dataset. It's worth noting that we have indeed evaluated our method alongside other state-of-the-art techniques using the ETH/UCY dataset. However, owing to page constraints in the main paper, we've presented the results in *Table 6 of the supplementary material*.

- **Experiment on 3D scenarios.** In response to the reviewer's suggestion, we have extended our evaluation to the 3D Haggling dataset, encompassing diverse action types. The outcomes are detailed in *Table 3* and *Figure 1* in the attached PDF. Encouragingly, our method demonstrates noteworthy performance for both short-term and long-term predictions, particularly in intricate activity scenarios.

- **Experiment on synthetic datasets.** We use the approach outlined in [1] to test our model's ability to uncover rules on synthetic datasets with known ground-truth rules and weights. We compare the learned rule weights with the actual ones and report the Mean Absolute Error (MAE) as a measure of accuracy.

- **Ablation study.** Based on the feedback from reviewers, we have included more ablation studies in our work. These studies cover aspects such as rule selection, discretization rates, and variations in different backbones.

- **Clarification about some important concepts.** We have revised our explanation of "logic" and "interpretability" based on the comments provided by the reviewers.

Beyond the academic contributions presented in the paper, our approach also offers practical significance. We introduce a feasible and differentiable algorithm capable of simultaneous learning of rule content and model parameters from observational data. Our model directly utilizes precise, detailed, and irregularly-spaced action times and original 3D coordinates as inputs. Through extensive experimentation on real datasets, we have showcased our model's strong performance in human action prediction and explanation. We hold the view that this distinct approach has the potential to inspire a host of future research endeavors.

[1] Li S, Feng M, Wang L, et al. Explaining point processes by learning interpretable temporal logic rules[C]//International Conference on Learning Representations. 2021. 

Meta-review: This paper presents an interesting framework for discovering rules for action recognition in a spatial-temporal logic setup. The methodological contribution includes a well-formulated EM approach for learning logic rules to explain a dataset of actions. The empirical results are also convincing, demonstrating the efficacy of the approach.

In the rebuttal/discussion, the authors clarified details, terminology, and pointed to synthetic/real dataset results that support the claims in the paper. The reviewers concur on an accept rating for the paper.
Decision: Accept (poster)