Title: Discovering Intrinsic Spatial-Temporal Logic Rules to Explain Human Actions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Logic rule

human actions

sports analyze
Abstract: We propose an interpretable model to uncover the behavioral patterns of human movements by analyzing their trajectories. Our approach is based on the belief that human actions are driven by intentions and are influenced by environmental factors such as spatial relationships with surrounding objects. To model this, we use a set of spatial-temporal logic rules that include intention variables as principles. These rules are automatically discovered and used to capture the dynamics of human actions. To learn the model parameters and rule content, we design an EM learning algorithm that treats the unknown rule content as a latent variable. In the E-step, we evaluate the posterior over the latent rule content, and in the M-step, we optimize the rule generator and model parameters by maximizing the expected log-likelihood. Our model has wide-ranging applications in areas such as sports analytics, robotics, and autonomous cars. We demonstrate the model's superior interpretability and prediction performance on both pedestrian and NBA basketball player datasets, achieving promising results.

Discovering Intrinsic Spatial-Temporal Logic Rules to Explain Human Actions

Abstract

We propose a logic-informed knowledge-driven modeling framework for human movements by analyzing their trajectories. Our approach is inspired by the fact that human actions are usually driven by their intentions or desires, and are influenced by environmental factors such as the spatial relationships with surrounding objects. In this paper, we introduce a set of spatial-temporal logic rules as knowledge to explain human actions. These rules will be automatically discovered from observational data. To learn the model parameters and the rule content, we design an expectation-maximization (EM) algorithm, which treats the rule content as latent variables. The EM algorithm alternates between the E-step and M-step: in the E-step, the posterior distribution over the latent rule content is evaluated; in the M-step, the rule generator and model parameters are jointly optimized by maximizing the current expected log-likelihood. Our model may have a wide range of applications in areas such as sports analytics, robotics, and autonomous cars, where understanding human movements are essential. We demonstrate the model's superior interpretability and prediction performance on pedestrian and NBA basketball player datasets, both achieving promising results.

1 Introduction

For a human, although the exhibited movements can be complex, the logic behind the actions is usually simple, clear, and can be generalized [17]. The logic rules present a compact and high-level knowledge representation , defining what actions tend to be executed under what conditions. It emphasizes reasoning, and encourages structuring the world in terms of objects, properties, and relations [22]. There has been a great interest and business value in unveiling the human logic from the observational movements and actions [18]. We provide two motivating examples below.

In sports analytics , understanding each player's behavior preferences or tendencies under various scenarios will provide coaches with valuable information [1]. Usually, coaches need to watch the game or training videos for hundreds of hours before they can summarize the discoveries into compact principles. Could we design an algorithm to synthesize these principles from the raw action data automatically ? One can imagine, such a tool will significantly reduce the workload of coaches, providing more granular insight into each player's capabilities and strategies, and aiding in personalized training and match strategy design [15].

In self-driving car , it's essential to enable the self-driving cars to 'read human's mind" like humans. This requires the self-driving cars to automatically understand human intentions and reasoning when

they are running on the same roads with human drivers [2]. If self-driving cars can automatically distill logic rules from their observed low-level noisy human actions and movement trajectories, it will increase the technical reliability and accelerate the widespread use of self-driving cars.

For human actions, lots of the governing rules would be regarding the spatial-temporal relation with the surrounding environments and their intentions [9]. For example, when a basketball player with a ball is within the scoring range, his/her action, such as shoot, pass, or triple threat, is influenced by historical and current surrounding factors, such as the current locations of the player and the defenders, the time elapses of the game, the success shooting rate of the player in today's game, and so on. The quick decision made by the player is actually reflecting a composition of all these factors, which can be described as a collection of spatial-temporal logic rules in our model.

Formally, in our introduced spatial-temporal logic rules, the logic variable (i.e., predicate) set will include spatial-temporal relation predicates, in addition to the commonly defined object property and relation predicates. The rule content will capture the spatial relation of the object with surrounding objects, as well as the temporal ordering constraints of the events .

Our methods have the following distinct features:

From the modeling perspective: ( i ) Our human action model is a rule-based probabilistic model, which treats each hidden rule as a 'soft" constraint. We assume each rule will be executed by humans with probabilities, and this tolerates the uncertainties in data. ( ii ) Our model directly uses low-level, fine-grained, and (may) irregularly-spaced action times and locations (i.e., original 3d coordinates ) as inputs, as opposed to other rule-based models, where one needs to first extract relational data as inputs. ( iii ) Our spatial and temporal predicates are also probabilistic. For predicates such as 'left of" or 'before", we model them as kernel functions with learnable variables. In this way, our introduced spatial-temporal predicates are smooth functions of the input locations and times, which increases model flexibility.

From the learning perspective: We propose a tractable and differentiable algorithm that can jointly learn the rule content and model parameters from observational data. The learning framework is designed to maximize the likelihood of the human action trajectories. Specifically, we propose to use a neural rule generator to generate the spatial-temporal logic rule set. Our continuous rule generator parameters will be optimized in a differentiable way. The overall learning procedure is an expectation-maximization (EM) algorithm, where we treat the rule set as latent variables. In the E-step, the posterior distribution over the latent rule set is evaluated. In the M-step, both the rule generator parameters and the model parameters are optimized by maximizing the expected log-likelihood with respect to the current posterior. We demonstrated the promising performance of our model in terms of human action prediction and explanation on two interesting real datasets.

2 Related Work

Logic Rule Learning. Learning logic rules from raw data has been widely studied for various downstream tasks, such as motion inference [4] and healthcare analysis [11]. Learning rules via an exact search requires enumerating all combinations of the logic predicates and is intractable in most problems. One has to design heuristic searching algorithms by leveraging some structural properties of the problems. For example, Dash et al. [5] formulated a convex rule learning problem and proposed a column generation algorithm to expand the rule set gradually. Wang et al. [25] designed a Bayesian framework for learning rule classifiers and derived bounds on the support of rules in a MAP solution. Recently, Yang et al. [28] proposed an interesting end-to-end differentiable approach (Neural LP) to learn the parameters and structure of logical rules. Qu et al. [19], and Sadeghian et al. [22] proposed an efficient logic rule mining algorithm based on the knowledge graph data. However, none of these advanced rule mining methods can directly work on spatial-temporal human action data when the inputs are raw event 3d coordinates and types.

Spatio-Temporal Dynamics for Event Data. Since the human actions are irregular spatialtemporal event data, we also briefly discuss probabilistic models for such event sequences . Modeling the spatial-temporal dynamics of discrete events is foundational in many scientific fields and applications [20]. Shen et al. [23] proposed a novel deep learning model for spatial-temporal events such as taxi data and achieved promising prediction accuracy. Zhou et al. [31] integrated deep learning methods with spatiotemporal point processes and modeled the intensity function as a latent stochastic process. Chen et al. [3] deployed two novel architectures, including jump and attentive continuous-time normalizing flows, to learn the dynamics of the spatiotemporal event data. Repe et al.

[21] learned canonical spatiotemporal point cloud representation using a latent ODE and continuous normalizing flows to generate shapes continuously in spacetime. However, these spatial-temporal event models are governed by hard-to-interpret dynamic functions and cannot be generalized to model human action events. Could we propose a model with logic-informed dynamic functions to explain the spatial-temporal human action events?

3 Our Model

3.1 Data: Human Actions Recorded as Spatial-Temporal Event Sequences

Consider a set of objects, denoted as C . For the object c ∈ C , its trajectories and the key actions observed up to t can be summarized as a sequence of temporally ordered events:

where t ∈ R + is the time, s ∈ R 2 is the location, and κ ∈ K is the event (i.e., action) type.

3.2 Definition of Spatial-Temporal Predicates

Static Predicate Given the object set C , the predicate is defined as the property or relation of objects, which is a logic function as follows:

For example, Smokes ( c ) is a property predicate and Friend ( c, c ′ ) is a relation predicate.

Temporal relation predicates They can be used to define the temporal relations of two action events. We consider three types of temporal relation predicates { before, equal, after } as:

✶ We will treat the temporal relation predicate as either a boolean variable or a real-valued function. If the time information is imprecisely recorded, we can parameterize the temporal relation predicates as temporal kernel functions that map to [0 , 1] , which is a function of t 1 , t 2 with learnable parameters.

Spatial-Temporal Predicate In our paper, we extend the above static predicates to spatial-temporal predicates, which include spatial-temporal property predicates and spatial-temporal relation predicates.

Specifically, the spatial-temporal property predicates are defined as:

For example, PickupKey ( c, t, s ) is a spatial-temporal property predicate. Suppose an entity c 1 picked up the key at time t 1 in location s 1 , then the predicate will be grounded as True (1) at ( c 1 , t 1 , s 1 ) , i.e., PickupKey ( c 1 , t 1 , s 1 ) = 1 ; otherwise it is false.

Given the observational human action data, the grounded predicate { PickupKey ( c, t, s ) } t =1 , 2 ,... can be modeled as a sequence of discrete events - when the predicate becomes True, an event happens. In general, the grounded spatial-temporal property predicate is a discrete event sequence, where the event occurrence times and locations are irregular.

The spatial-temporal relation predicates are introduced to define the spatial and temporal relations of two entities. Specifically, they are defined as:

Spatial-temporal relation predicates are logic variables indicating the spatial-temporal relations of two objects, where we further divide them into temporal relation predicates, static spatial relation predicates, and dynamic spatial relation predicates. More details can be found in Appendix.

It is noteworthy that all these boolean predicates can be converted to probabilistic ones. We can soften these logic functions by kernel functions with learnable parameters to tolerate uncertainties in data.

3.3 Definition of Spatial-Temporal Logic Rules

We will consider spatial-temporal logic rules where the body part contain spatial-temporal predicates as relation constraints. For example, a sensible rule will look like:

where c ∈ C person , c ′ ∈ C block, and c ′′ ∈ C key. 'person', 'block' and 'key' are the specific instances in the object set C , and this rule represents that a person wants to pick up a key, while one block is in front of him and the key is behind him, so he turns around. We utilize this example to manifest the meaning of logic rules. In general, the spatial-temporal logic rule in our paper is defined as a logical connectives of predicates, including property predicates and spatial-temporal relation predicates:

where Y ( v ) is the head predicate evaluated at the entity-time-location triplet v , X f is the set of property predicates defined in rule f , and R f denotes the set of spatial-temporal relation predicates defined in rule f .

3.4 Logic-Informed Action Event Models

We consider a setting where we can fully observe the trajectories of all the moving objects, including their real-time locations and key actions (i.e., events), denoted as H t . We aim to propose a logicinformed spatial-temporal model to predict and explain the action type given the entity-time-location triplet v = ( c, t, s ) (i.e., query) and H t . The main idea is to construct the model features using spatial-temporal logic rules, as defined in Eq. (7). Intuitively, given the entire trajectories H t and the query v = ( c, t, s ) , the body part of the rule defines the evidence to be selectively gathered from history to deduce the event type for query entity v = ( c, t, s ) . Assume that for each possible event type κ ∈ K , there exist multiple rules such as Eq. (7) to explain its occurrence, with κ being the head predicate . Given an individual rule as Eq. (7), we propose to build the feature that is conditional on history and query as

where we introduce a function g f ( · ) to check the body conditions of f given a 'path". We use a simple example to explain how to compute features, as shown in Figure 1. As illustrated, the feature computes the valid total number of 'paths" given the data and query. Suppose there is a rule set F κ , where the event κ is the head predicate. All the rules will play together to reason about the occurrence of κ . For each f ∈ F κ , one can compute the features as above. Given the rule set F , we model the probability of the event κ

B

A

C

x

y

GLYPH(cmap:d835)2

GLYPH(cmap:d835)1

GLYPH(cmap:d835)3

GLYPH(cmap:d835)4

GLYPH(cmap:d835)5

κ as a log-linear function of the features, i.e.,

where w = [ w f ] f ∈F ≥ 0 are the learnable weight parameters associated with each rule. All the model parameters can be learned by maximizing the likelihood, which can be computed using the above Eq. (9). We intend to train a rule generator p θ and an evaluator p w to maximize the likelihood of training data as:

More details can be found as follows.

4 Our Learning Algorithm

Our goal is to jointly learn the set of spatial-temporal logic rules {F κ } κ ∈K and their weights by the maximum likelihood method, where each rule has a general form as Eq. (7).

To discover each rule, the algorithm needs to navigate through the combinatorial space considering all the combinations of the property predicates and their spatial and temporal relations. To address this computational challenge, we propose a tractable (functional) EM algorithm that treats the rule set as latent variable z . The rules will be generated by a hidden neural rule generator. The overall learning framework alternates between an E-step, where the posterior distribution of the latent rule space is evaluated (rule generation), and the M-step, where the model parameters and rule generator parameters are optimized. Please refer to Figure. 2 for an illustration.

Rule Generator

(

|

)

p

z

v



A

B

C

...

1



N



...

Logic Rules

ˆ

z

Score

Trajectory

Input

Raw Data

Rule Generating

Reasoning Evaluator

(

|

,

)

w

p

v z



Intention

A

B

C

Prediction

Decoder

...

1

ˆ



ˆ

N



...

Logic Reasoning

1

2

(

,

,...)





,

ˆ

~

(

|

,

)

k

w

k

z

p

z

v





High-quality Rule

A

B

C

D

D

D

∧

∨

Entities

Spatio-temporal

Relation

Symbol

Event-type

Head predicate

A

B

C

D

A

B

C

D

Our goal is to maximize the likelihood of the observed human action events { κ ( i ) } i =1 ,...,n . Using the chain rule, we have

To simplify the notation, we will use p w ( κ ( i ) ) to stand for p w ( κ ( i ) | v ( i ) , H t ( i -1) ) in the following. Given a latent rule set z , we have to marginalize the posterior of z to get the above log-likelihood. However, the exact inference of z is intractable. We will introduce an amortized recognition network p θ ( z | κ ( i ) ) to approximate the true posterior. We have

where the first term is the KL divergence of the approximate from the true posterior, and the second term L ( θ, w ; κ ( i ) ) is the variational lower bound (ELBO). It can be represented as:

And log p w ( κ ( i ) ) ≥ L ( θ, w ; κ ( i ) ) . The bound becomes tight when the approximate posterior matches the true one. Our goal is to optimize the variational parameters θ and model parameters w from the ELBO lower bound.

4.1 Rule Generator

We deploy Transformer-based framework to model the rule generator p θ . We define the distribution of a set of rules as follows:

where Ψ( · ) is multinomial distributions, N is the number of the top rules, and Trans θ ( v, H t ) defines a distribution over compositional rules with spatial-temporal states. The generative process of the rule set is quite intuitive, where we simply generate N rules to form z . In fact, this p θ ( z | v, H t ) is a flexible posterior approximation function, which will be optimized by the EM type algorithm.

We choose transformer over graph neural network (GNN) as our baseline because transformer architectures are based on a self-attention mechanism that is able to capture long-range relationships, as opposed to recurrent neural networks that process sequence elements recursively and can only take into account short-term context. Note that the graph operations in GNN are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a changeable graph that consists of various types of nodes and edges.

4.2 Rule Evaluator

Eq. (9) is our rule evaluator (suppose we know the rule content). Here we assume the rule content is latent, and the rule evaluator is given as

4.3 Optimization

We optimize the rule generator p θ and reasoning evaluator p w to maximize the objective in Eq. (10). At each training iteration, we first update the reasoning predictor p w according to some rules generated by the generator, and then update the rule generator p θ .

In our network, the latent rule set will be automatically discovered. The best set of logic rules is approximately obtained by sampling and preserving the top-K rules according to their posterior probabilities. Specifically, as shown in Eq. (14), the posterior probabilities of the latent rule z is obtained by a Transformer type of encoder, which maps the input observed action trajectories to a latent explanatory rule space. Each candidate rule is generated in the latent rule space token-by-token (token means logic variable/predicate in our setting) in a sequential manner and meanwhile the posterior probability of each rule sequence can be evaluated. When optimizing the evaluator, we draw several rules ˆ z for each query and let the evaluator uses ˆ z to predict κ . For each query, we try to identify top-K rules z I from all generated rules ˆ z . It is accomplished by taking into account the posterior probabilities of each subset of logic rules z I with prior from the rule generator p θ and likelihood from the reasoning predictor p w . Specifically, when a series of rules produced from the rule generator p θ , we calculate the weights of each rule z ( i ) as follows:

where A is the set of all candidate event type inferred by logic rules. Trans θ ( z ( i ) | v, H t ) is the probability of rule computed by the generator. For a subset of rules z I ⊂ ˆ z , the log-probability can be approximated as: log p θ,w ( z I | v, H t ) ≈ ∑ z ( i ) ∈ z I J ( z ( i ) ) + log Ψ( z I | N, Trans θ ( v, H t )) + const . This equation inspired us to use the distribution q ( z I ) ∝ exp( ∑ z ( i ) ∈ z I J ( z ( i ) ) + log Ψ( z I | N, Trans θ ( v, H t ))) as approximation of the posterior. Each rule z ( i ) sampled from q ( z I ) independently can be formed with N logic rules.

Clearly, J ( z ( i ) ) can be regarded as the quality of candidate rules, with consideration of the evaluator p w . It is calculated as the contribution of a rule to the correct event type minus the average contribution of this rule to the other candidate responses. A rule is more significant if it obtains a higher score to the correct event type and a lower score to other potential predictions.

After getting several high-quality rules from training data, we further utilize these rules to update the parameters of rule generator p θ . Concretely, we regard the generated high-quality rules as part of training data, and update the rule generator by maximizing the log-likelihood as follows:

By learning to generate high-quality rules, the rule generator will reduce the search area and produce better empirical results for the reasoning predictor.

5 Experiments

In this section, we provide some implementation details and show ablation studies as well as visualization to evaluate the performance of our framework. We compare our model with several state-of-the-art approaches, including PECNet [14], NMMP [7], STGAT [8], SOPHIE [22], STAR [29], Y-Net [13], MID [6], Social-SSL [24], Social-VAE [26], Social-Implicit [16], and NSP-SFM [30].

5.1 Implementation

We follow the same data prepossessing strategy as PECNet [14] for our method. All models were trained and tested on the same split of the dataset, as suggested by the benchmark. We train the network using Adam optimizer with a learning rate of 0.001 and batch size 16 for 500 epochs.

MAE=0.10

MAE=0.11

MAE=0.06

MAE=0.17

Truth

Truth

Truth

Truth

Learned

Learned

Learned

Learned

Value

e

2

三

e

2

Value

2

Value

2

1

1

1

1

0

0

0

0

wo

w1

w2

wo

w1

w2

wo

w1

w2

wo

w1

w2

Rule Weight

Rule Weight

Rule Weight

Rule Weight

5.1.1 Datasets

Stanford Drone Dataset. This dataset consists of more than 11,000 persons in 20 scenes captured from the campus of Stanford University in bird's eye view. We follow the [27] standard train-test split, and predict the future 4.8s (12 frames) using past 3.2s (8 frames). Note that SDD dataset does not provide explicit pedestrian's action. Instead, we record them as an abstract encoding of the pedestrian's speed and location. The action contains [left, right, straight, turn around].

NBA SportVU Dataset. It is collected by NBA using the SportVU tracking system, which reports the trajectories of the ten players and the ball in real basketball games. Each trajectory contains the 2D positions and velocities of the offensive team, consisting of 5 players. We predict the future 10 timestamps (4.0s) based on the historical 5 timestamps (2.0s). Each player's action contains [left, right, straight, turn around, pass, shoot].

5.1.2 Metrics

Here, we adopt two metrics for evaluation: Average Displacement Error ( ADE ) and Final Displacement Error ( FDE ). Specifically, ADE N is defined as the minimum average distance between N predicted trajectories and the ground truth over all the involved entities within the prediction horizon. FDE N is defined as the minimum deviated distance of N predicted trajectories at the last predicted time step. Moreover, we also calculate the accuracy and F1 score of event-types predicted from each network.

5.2 Synthetic Experiment

We follow [10] to verify our model's rule discovery ability on synthetic datasets with a known set of ground-truth rules and weights. The synthetic events are generated from TLPPs [12] with a known set of rules and weights. We prepared 4 synthetic datasets, and each setting corresponds to different rule weights, rule length and number, type of temporal relation, and intensity of free predicates. Note that it was originally utilized for the temporal point process, so we modify it by adding spatial variables (such as 'left, right, front, and behind') to fit in our settings. The weight learning results on 4 synthetic datasets are shown in the Figure 3, where 2400 sequences are used for evaluation. Each plot at the bottom compares the genuine rule weights to the learned rule weights and reports the Mean

(a) GT

(b) NMMP

(c) Ours

Absolute Error (MAE). As we can see, almost all truth rule weights are learned correctly. It shows an accurate performance of our algorithm in terms of both the rule discovery and parameter learning.

5.3 Analysis

We compare our method with several state-of-the-art approaches, and Table 2 presents the qualitative results on the SDD dataset. The proposed model achieved the best performance in ADE, FDE and accuracy. We observe that our method significantly outperforms all baselines measured by ADE and FDE. It achieves an ADE of 6.41 and FDE of 10.23 at K = 20 in SDD datset, which exceeds the previous state-of-the-art performance of Y-Net [13] by 18.3% on ADE and 13.6% on FDE. In Table 1, our method also achieve higher performance than Y-Net in NBA dataset. This is because Y-Net firstly assume that the waypoint lies on a straight line segment connecting the sampled goal and the past trajectory, then use a multivariate Gaussian prior centered at the assumed location. This assumption can not well suit in other complex conditions, such as the trajectory of players in the NBA dataset.

Compared with MID [6], we also obtain 15.7% ADE and 28.4% FDE improvement. Note that they carefully design a Transformer-based architecture to model the temporal dependencies in trajectories, but ignore the spatial correlation of agents. Our transformer-based network aims at generating highquality logic rules based on spatial-temporal relation under the principle to maximize the likelihood of the observational human actions. For NSP-SFM [30], it obtains high performance in SDD dataset but can not achieve the same level in NBA dataset. It combines physics with deep learning for trajectory prediction and accommodates arbitrary physics models. The limitation of it lies in specific physics models, such as pedestrians, and is deterministic. So it can not deal with some strategy-based conditions, including basketball and football games. But our logic-learning method tries to use a set of spatial-temporal logic rules with intention variables involved as principles to model the dynamics of human actions, not restrained into specific conditions.

Further, The proposed model achieved the best scores in F1 score, which is an balanced metric considering both recall and precision. This is because the rule generator and evaluator can collaborate with each other to reduce search space and learn better rules. More experiments and ablation studies can be found in Supplementary Material.

basketball player

basketball

Player A Rule ① : Left(B,A) ∧ Right(C,A) ∧ Front(D,A)→Pass(A,B) Explanation:

Player A is carrying the ball

Player A  is defended by player C and player D

Player B is on the left of player A

Player A passes the ball to player B

Player B Rule ② : Left(E,B) ∧ Right(D,B)→Go_Front(B) Explanation:

Player E and player F is moving to the player B

Player B is defended by player E on left side

Player B is defended by player D on right side

Player B goes front and carries the ball

B

A

②

E

F

D

C

G

Player B Rule ③ : Left(E,B) ∧ Right(G,B) ∧ Front(F,B)→Shoot(B) Explanation:

Player B is defended by player E on left side

Player B is defended by player G on right side

Player B is defended by player F on front side

Player B shoots at the basket

B

A

③

E

F

D

C

G

B

A

①

E

F

D

C

G

To evaluate the robustness of our method, We added some noise and randomly removed several tracks in the NBA dataset, then evaluated all methods in Table 3 ('Ours*' is the original results without noises or missing tracks). As we can see, our method still achieves the best performance. Moreover, by comparing 'Ours' and 'Ours*', we can see that the quality of tracks has less influence on our method, which demonstrates the robustness of our method.

Our method not only outperforms all the above comparative methods in quantitative evaluations, but also produces visually pleasing results among them. Figure 4 compares the predicted trajectories of NMMP, ground-truth (GT) and our methods on NBA dataset. It is obvious that our method produces more precise predictions than state-of-the art method NMMP. This is because our proposed spatial-temporal logic rules can actually captures the spatial relation of the players with surrounding players, as well as the temporal ordering constraints of the events.

5.4 Generated Logic Rules

We add visualization and explanation about the logic rule and corresponding actions from NBA dataset in Figure 5. Note that the static spatial relation { Left(B,A) } represents that the player B is on the left of player A . And the dynamic spatial relation { Away(A,E) } means that player A is getting away from player E . We can see that these logic rules are meaningful and diverse. In this picture, player A is defended by two players and then passes the ball to player B. Player B goes front and crossover to bypass three defenders and shoot at the basket. Our rule can actually represent their offensive strategy. In fact, our framework can actually adapt to some complex motions, such as cutting toward the ball, because these spatial-temporal predicates are fed into the rule generator and evaluator to obtain high-quality rules to explain the intention of players.

6 Limitation

It's challenging to define some complex predicates with richer meanings for different datasets. Given a more informative dataset, our method can discover more principles-like complex rules. Our motivation is to consider the spatial-temporal relation between pedestrians and generate some high-quality logic rules to explain their behaviors. Although we only choose some simple actions in our experiments, they can bring some benefits for understanding the principles of biological agents' behaviors. Our framework is suitable for more complex conditions, supposing more sophisticated action predicates can be obtained from the data.

7 Conclusion

We proposed a framework for learning intrinsic spatial-temporal logic rules for explaining human actions. We regard logic rules as latent variables, and the rule generator as well as the rule evaluator are jointly learned with EM-based algorithm. In the experiments, our method can analyze the biological movement sequence of pedestrians and players, and obtained novel insights from generated logic rules. In the future, we plan to incorporate other physical laws into the models, such as conservation of energy and momentum to enhance robustness of our model.

References

David Adam. Science and the world cup: how big data is transforming football. Nature , 611(7936):444-446, 2022.

Claudine Badue, Rânik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert Systems with Applications , 165:113816, 2021.

Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representations , 2021.

Chang Choi, Junho Choi, Eunji Lee, Ilsun You, and Pankoo Kim. Probabilistic spatio-temporal inference for motion event understanding. Neurocomputing , 122:24-32, 2013.

Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances in neural information processing systems , 31, 2018.

Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17113-17122, 2022.

Yue Hu, Siheng Chen, Ya Zhang, and Xiao Gu. Collaborative motion prediction via neural motion message passing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6319-6328, 2020.

Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6272-6281, 2019.

Rainer Kartmann, You Zhou, Danqing Liu, Fabian Paus, and Tamim Asfour. Representing spatial object relations as parametric polar distribution for scene manipulation based on verbal commands. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 8373-8380. IEEE, 2020.

Shuang Li, Mingquan Feng, Lu Wang, Abdelmajid Essofi, Yufeng Cao, Junchi Yan, and Le Song. Explaining point processes by learning interpretable temporal logic rules. In International Conference on Learning Representations , 2021.

Shuang Li, Mingquan Feng, Lu Wang, Abdelmajid Essofi, Yufeng Cao, Junchi Yan, and Le Song. Explaining point processes by learning interpretable temporal logic rules. In International Conference on Learning Representations , 2022.

Shuang Li, Lu Wang, Ruizhi Zhang, Xiaofu Chang, Xuqin Liu, Yao Xie, Yuan Qi, and Le Song. Temporal logic point processes. pages 5990-6000, 2020.

Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15233-15242, 2021.

Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 759-776. Springer, 2020.

Andrew Miller, Luke Bornn, Ryan Adams, and Kirk Goldsberry. Factorized point process intensities: A spatial analysis of professional basketball. In International conference on machine learning , pages 235-243. PMLR, 2014.

Abduallah Mohamed, Deyao Zhu, Warren Vu, Mohamed Elhoseiny, and Christian Claudel. Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In European Conference on Computer Vision , pages 463-479. Springer, 2022.

Filmer Stuart Cuckow Northrop. The logic of the sciences and the humanities. 1947.

Clayton Peterson. A logic for human actions. Applications of Formal Philosophy: The Road Less Travelled , pages 73-112, 2017.

Meng Qu, Junkun Chen, Louis-Pascal Xhonneux, Yoshua Bengio, and Jian Tang. Rnnlogic: Learning logic rules for reasoning on knowledge graphs. In International Conference on Learning Representations , 2021.

Alex Reinhart. A review of self-exciting spatio-temporal point processes and their applications. Statistical Science , 33(3):299-318, 2018.

Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojcic, Srinath Sridhar, and Leonidas J Guibas. Caspr: Learning canonical spatiotemporal point cloud representations. Advances in neural information processing systems , 33:13688-13701, 2020.

Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems , 32, 2019.

Bilong Shen, Xiaodan Liang, Yufeng Ouyang, Miaofeng Liu, Weimin Zheng, and Kathleen M Carley. Stepdeep: A novel spatial-temporal mobility event prediction framework based on deep neural network. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 724-733, 2018.

Li-Wu Tsao, Yan-Kai Wang, Hao-Siang Lin, Hong-Han Shuai, Lai-Kuan Wong, and WenHuang Cheng. Social-ssl: Self-supervised cross-sequence representation learning based on transformers for multi-agent trajectory prediction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII , pages 234-250. Springer, 2022.

Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeille. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research , 18(1):2357-2393, 2017.

Baowen Xu, Xuelei Wang, Shuo Li, Jingwei Li, and Chengbao Liu. Social-cvae: Pedestrian trajectory prediction using conditional variational auto-encoder. In International Conference on Neural Information Processing , pages 476-489. Springer, 2023.

Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6507, 2022.

Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems , 30, 2017.

Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 , pages 507-523. Springer, 2020.

Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV , pages 376-394. Springer, 2022.

Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference , pages 777-789. PMLR, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a model for inferring rules based on observation trajectories of (entity, time, location), with the aim of maximizing the probabilities of certain \"events\". The aim is to predict the next events based on past trajectories, conditioned on the latent \"rules\" which need to be marginalized over. But this is intractable, so an \"encoder\" or \"recognition\" network is used, similar to the logic of variational autoencoder. In the encoder, a transformer is used to define the probability distribution over possible rules based on input trajectories, and a decoder can sample these rules to generate the next event. The model is used for trajectory prediction tasks on two datasets with better results than existing trajectory prediction model. Examples of rules learnt are also given in case of one dataset.\n\nSTRENGTHS:\n1) A nice framework is proposed for representation of spatio-temporal action rules based on logic\n2) The \"rules\" are expressed as a latent variable model, where inference can be done using E-M algorithm applied to an encoder-decoder model\n3) Strong results are shown on trajectory prediction tasks, sample rules are also shown\n\nIn my opinion, the rule definition and rule inference framework is quite general and can find wider applications than the experiments described in the paper.\n\nWEAKNESSES:\nIt is not very clear how exactly the goal states are estimated (eg. in Fig 3), or how the future trajectories are generated (Fig 3 and experiments), I get the broad idea, but precision is missing.\n\nQUESTIONS:\n1) Can we have an experiment on synthetic setting with ground truth data generated according to specific rules (eg. like a cellular automata), and then show if the proposed model can recover the rules?\n2) Please add an algorithm to explain how the trajectory prediction is carried out, step-by-step.\n3) What can be other applications of this framework beyond trajectory prediction? Also, can the \"rules\" here be related to \"policies\" to choose actions as in Reinforcement Learning?\n4) Can the rule probability distribution be modelled by a cheaper model than Transformer?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach for human trajectory prediction, introducing a learnable rule-based framework that combines rule generation/reasoning and EM optimization. Unlike previous works in the field, this framework utilizes a neural rule generator to generate rules and treats them as latent variables. Additionally, it selects the top k rules from the entire rule set to predict future trajectories. The experimental results on two datasets demonstrate the effectiveness of the proposed framework.\n\nSTRENGTHS:\n- The paper is well-structured and written in a clear manner, making it easy to follow. The authors have used appropriate notation and included equations where necessary, enhancing the readability of the paper.\n\n- The detailed analysis and visualizations presented in sections 5.4-5.6 and the ones in the supplementary materials contribute to a better understanding of the proposed method. I found them enjoyable and informative to read.\n\n- The paper introduces a novel framework that incorporates separate steps for rule generation and logic reasoning. This approach is original and reasonable for the given task.\n\n- The proposed framework demonstrates improved results on the Stanford Drone Dataset and NBA SportVU dataset.\n\n- The authors have put effort into providing numerous illustrative examples that showcase the behavior of the proposed framework. The explanation of logic rules in the NBA dataset (Figure 5) is particularly interesting.\n\nWEAKNESSES:\n- The predicates are required to be manually defined and these predicates may need to be redefined when applied to new scenarios, such as transitioning from 2D to 3D scenarios.\n\n- ETH/UCY benchmarks have been widely used for benchmarking short-term trajectory prediction but missing from this paper.\n\nQUESTIONS:\n- Regarding the latent rule space, what is the dimension of the latent space z?\n\n- I have reservations about fully accepting the argument presented in L207 that \"our method can discover more principles-like complex rules.\" The generation of more rules for complex conditions necessitates a larger latent space (higher dimension), which can pose challenges in generating high-quality rules.\n\n- In L207-L208, it is mentioned that \"For each query, we aim to identify top K rules $z_I$ from all generated rules $\\hat{z}$,\" with K set to 5 in the experiments. Not all movements require up to K rules. Some movements may be simple and only require 1-2 rules (as seen in the second example in Figure 2). Is it possible to automatically determine the number of rules based on their weights?\n\nLIMITATIONS:\nThe authors have provided a discussion of the limitations in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proses a method for learning spatio-temporal logic rules to explain human actions, utilizing the EM framework. The method results are more easy interpretable by humans, thanks to the logic rules. The method beats some state-of-the-art methods on two real-world motion prediction datasets.\n\nSTRENGTHS:\nThe method is well-motived, i.e. that human actions are driven by intention as well as social and environmental factors. The explainability of the model is impressive and well-visualized.\n\nWEAKNESSES:\nMy major concern with this work is the need for dataset-depended actions. Biasing the model with additional labels, which are not provided to other methods, allows it to spend more capacity on forecasting, effectively producing an unfair advantage. It would have been better if the authors provide an approach that can automatically extract the actions, without human pre-selection.\n\nWhile the general intention is well-motivated, the method overview and method motivation in the introduction section is not well described, making it difficult to get a first good understanding of what the method tries to achieve and how this is done. For example, the authors explain that “[] logic rules present a compact knowledge representation” (L19) but they do not further explain what “logic” they are using and how it is defined. \n\nIn the method section a lot of important details are missing, making it difficult to follow:\n* In the Equations in L104.5 and L113.5 the authors utilize variable v and v’ without introducing what this variable means. The authors should explain the variable appropriately. \n* In L122 the authors introduce various new object types “person”, “block” and “key” which are not explained. \n* In L117 the paper says that the binary predicated values can be softened into “probabilistic” ones using kernel functions without details or reference to other parts of the paper. This makes it particularly difficult to understand as the paper has not yet revealed how the logic rules are applied to obtain the predictions.\n\nSpatio-temporal predicates (L104) contain the time and space dimension. This necessitates that both time and area must be discretized. The discretization rate would be an important hyper-parameter and should be ablated accordingly. \n\nWhy do the authors not follow the experimental setup in [23] and also evaluate on the ETH-UCY dataset? Can the dataset not be labeled with actions? If so: why? What makes the dataset difficult for this method setup?\n\nThe organization of the experiments section is not convincing: while the authors conducted a lot of experiments they should have added them into the main paper.\n\n**Minor issues**:\n\nThe authors claim their models “superior interpretability” (L15) but do not properly explained in text (only attempted in Figure 5). This however should be explained briefly.\n\nEquations should be numbered to make referencing them easier\n\nQUESTIONS:\nIt seems that using the full 3D human pose would be a much better indicator for human actions, interactions with objects, and interactions with other persons. Why did the authors use only trajectory data? For example, the Haggling dataset [a] contains triadic interactions under a well-defined social protocol which could have been utilized.\n\nL94: “Consider a set of objects denoted as C”: does object mean humans? This should be explicitly stated if true.\n\nL96: could the authors elaborate how k is encoded?\n\n—\n\n[a] Liu, Zhenguang, et al. \"Towards natural and accurate future motion prediction of humans and animals.” CVPR 2019.\n\nLIMITATIONS:\nThe methods challenges are discussed to some degree.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study presents a logic-informed, knowledge-driven modeling framework designed to predict and understand human movements, based on the analysis of their trajectories. It takes into account that human behaviors are commonly guided by intentions, desires, and spatial relationships with surrounding objects. The research integrates a set of spatial-temporal logic rules, derived from observational data, into the model, utilizing an expectation-maximization (EM) algorithm to infer model parameters and rule content. The performance of the model is evaluated using datasets of NBA basketball player movements and pedestrian trajectories, demonstrating high levels of interpretability, predictive accuracy, and efficacy in its results.\n\nSTRENGTHS:\n- The objectives and motivation are well-articulated, offering a clear understanding of the reasons behind the research and the problem it aims to address.\n- The performance of the proposed model is outstanding when compared to other state-of-the-art methods signifying an advantage of the work.\n- The proposed method is based on theoretical principles (EM method). It's not a product of arbitrary choices, but rather a thoughtful construction based on the EM algorithm, which adds credibility to the model's outcomes.\n\nWEAKNESSES:\n- If additional specifics regarding the architecture could be provided, such as the input and output formats utilized by the transformer, it would be incredibly beneficial for a deeper understanding of the method.\n- The evaluation of this work is currently based on only two datasets. If the authors could expand this to include results from more universally employed pedestrian trajectory prediction datasets (ETH [i] or UCY [ii] ), it would significantly strengthen the claim of the model's generalizability.\n  \n  [i] Improving data association by joint modeling of pedestrian trajectories and groupings ECCV10.  \n  [ii] Crowds by example. In Computer Graphics Forum.\n\nQUESTIONS:\n- There is a typo in equation (3); the correct symbol should be $\\varphi_f$ instead of $\\phi_f$ .\n- The references for the datasets utilized in the experiments are currently absent and need to be included.\n- Where did you get the heatmap depicted in Fig3 (b) (d)?\n\nLIMITATIONS:\nIf my understanding is correct, the spatial-temporal logic rule space, encompassing both static spatial and dynamic spatial relations, needs to be explicitly defined. This implies that prior knowledge or assumptions about the environment and the interactions within it are required to structure and define these rules. Thus, it's not a process automatically derived from the data but requires human intervention and understanding to set up appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}