SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\mathcal{A}$  is the set of all candidate event type inferred by logic rules.  $\mathrm{Trans}_{\theta}(z^{(i)}|v,\mathcal{H}_t)$  is the probability of rule computed by the generator. For a subset of rules  $z_{I}\subset \hat{z}$ , the log-probability can be approximated as:  $\log p_{\theta ,w}(z_I|v,\mathcal{H}_t)\approx \sum_{z^{(i)}\in z_I}J(z^{(i)}) + \log \Psi (z_I|N,\mathrm{Trans}_{\theta}(v,\mathcal{H}_t)) + \mathrm{const}$ . This equation inspired us to use the distribution  $q(z_I)\propto \exp (\sum_{z^{(i)}\in z_I}J(z^{(i)}) + \log \Psi (z_I|N,\mathrm{Trans}_{\theta}(v,\mathcal{H}_t)))$  as approximation of the posterior. Each rule  $z^{(i)}$  sampled from  $q(z_I)$  independently can be formed with  $N$  logic rules.
Clearly,  $J(z^{(i)})$  can be regarded as the quality of candidate rules, with consideration of the evaluator  $p_w$ . It is calculated as the contribution of a rule to the correct event type minus the average contribution of this rule to the other candidate responses. A rule is more significant if it obtains a higher score to the correct event type and a lower score to other potential predictions.
After getting several high-quality rules from training data, we further utilize these rules to update the parameters of rule generator  $p_{\theta}$ . Concretely, we regard the generated high-quality rules as part of training data, and update the rule generator by maximizing the log-likelihood as follows:
$$
\mathcal {O} (\theta) = \log p _ {\theta} \left(z _ {I} \mid v, \mathcal {H} _ {t}\right) = \sum_ {z ^ {(i)} \in z _ {I}} \log \operatorname {T r a n s} _ {\theta} (v, \mathcal {H} _ {t}) + \text {c o n s t .} \tag {17}
$$
By learning to generate high-quality rules, the rule generator will reduce the search area and produce better empirical results for the reasoning predictor.
In this section, we provide some implementation details and show ablation studies as well as visualization to evaluate the performance of our framework. We compare our model with several state-of-the-art approaches, including PECNet [14], NMMP [7], STGAT [8], SOPHIE [22], STAR [29], Y-Net [13], MID [6], Social-SSL [24], Social-VAE [26], Social-Implicit [16], and NSP-SFM [30].
5.1 Implementation
We follow the same data prepossessing strategy as PECNet [14] for our method. All models were trained and tested on the same split of the dataset, as suggested by the benchmark. We train the network using Adam optimizer with a learning rate of 0.001 and batch size 16 for 500 epochs.
Figure 3: Rule discovery and weight learning results on 4 synthetic datasets (2.4K seqs).
Table 1: Quantitative results  $(ADE_{20} / FDE_{20}$ , and accuracy) of trajectory prediction in NBA dataset. The bold/underlined font represent the best/second best result.
Times	Y-Net	MID	NSP-SFM	Social-SSL	Social Implicit	Social VAE	Ours
1.0s	0.38/0.48	0.45/0.59	0.41/0.52	0.48/0.61	0.45/0.53	0.49/0.66	0.30/0.40
2.0s	0.63/0.93	0.76/1.06	0.67/0.94	0.76/1.08	0.72/0.96	0.77/1.11	0.58/0.88
3.0s	0.94/1.34	1.06/1.40	0.98/1.35	1.06/1.43	1.00/1.39	1.11/1.46	0.87/1.31
4.0s	1.17/1.61	1.32/1.74	1.18/1.63	1.35/1.78	1.19/1.66	1.37/1.79	1.13/1.60
Acc.	0.69	0.65	0.70	0.68	0.69	0.64	0.73
5.1.1 Datasets
Stanford Drone Dataset. This dataset consists of more than 11,000 persons in 20 scenes captured from the campus of Stanford University in bird's eye view. We follow the [27] standard train-test split, and predict the future 4.8s (12 frames) using past 3.2s (8 frames). Note that SDD dataset does not provide explicit pedestrian's action. Instead, we record them as an abstract encoding of the pedestrian's speed and location. The action contains [left, right, straight, turn around].
NBA SportVU Dataset. It is collected by NBA using the SportVU tracking system, which reports the trajectories of the ten players and the ball in real basketball games. Each trajectory contains the 2D positions and velocities of the offensive team, consisting of 5 players. We predict the future 10 timestamps (4.0s) based on the historical 5 timestamps (2.0s). Each player's action contains [left, right, straight, turn around, pass, shoot].
5.1.2 Metrics
Here, we adopt two metrics for evaluation: Average Displacement Error (ADE) and Final Displacement Error (FDE). Specifically,  $ADE_N$  is defined as the minimum average distance between N predicted trajectories and the ground truth over all the involved entities within the prediction horizon.  $FDE_N$  is defined as the minimum deviated distance of N predicted trajectories at the last predicted time step. Moreover, we also calculate the accuracy and F1 score of event-types predicted from each network.
5.2 Synthetic Experiment
We follow [10] to verify our model's rule discovery ability on synthetic datasets with a known set of ground-truth rules and weights. The synthetic events are generated from TLPPs [12] with a known set of rules and weights. We prepared 4 synthetic datasets, and each setting corresponds to different rule weights, rule length and number, type of temporal relation, and intensity of free predicates. Note that it was originally utilized for the temporal point process, so we modify it by adding spatial variables (such as "left, right, front, and behind") to fit in our settings. The weight learning results on 4 synthetic datasets are shown in the Figure 3, where 2400 sequences are used for evaluation. Each plot at the bottom compares the genuine rule weights to the learned rule weights and reports the Mean
(a) GT
(b) NMMP
(c) Ours
Figure 4: Visual results on the NBA dataset. We plot the predicted trajectories from the state-of-the-art method NMMP (b), Ours (c) and ground truth (a). The red/blue color represents players of two teams and the green color represents the basketball. Light color represents the past trajectory.
Table 2: Quantitative results ( $ADE_{20}$ ,  $FDE_{20}$  and F1 score) of trajectory prediction in SDD dataset. The bold/underlined font represent the best/second best result.
Metrics	PECNet	Y-Net	MID	NMMP	STGAT	SOPHIE	Social SSL	NSP-SFM	STAR	Ours
ADE20	20.03	7.85	7.61	14.12	14.43	15.56	6.63	6.52	10.76	6.41
FDE20	33.86	11.85	14.30	20.68	22.59	24.32	12.23	10.61	17.03	10.23
F1 score	0.37	0.54	0.49	0.41	0.33	0.39	0.53	0.58	0.56	0.59
Absolute Error (MAE). As we can see, almost all truth rule weights are learned correctly. It shows an accurate performance of our algorithm in terms of both the rule discovery and parameter learning.
We compare our method with several state-of-the-art approaches, and Table 2 presents the qualitative results on the SDD dataset. The proposed model achieved the best performance in ADE, FDE and accuracy. We observe that our method significantly outperforms all baselines measured by ADE and FDE. It achieves an ADE of 6.41 and FDE of 10.23 at  $K = 20$  in SDD dataset, which exceeds the previous state-of-the-art performance of Y-Net [13] by  $18.3\%$  on ADE and  $13.6\%$  on FDE. In Table 1, our method also achieve higher performance than Y-Net in NBA dataset. This is because Y-Net firstly assume that the waypoint lies on a straight line segment connecting the sampled goal and the past trajectory, then use a multivariate Gaussian prior centered at the assumed location. This assumption can not well suit in other complex conditions, such as the trajectory of players in the NBA dataset.
Compared with MID [6], we also obtain  $15.7\%$  ADE and  $28.4\%$  FDE improvement. Note that they carefully design a Transformer-based architecture to model the temporal dependencies in trajectories, but ignore the spatial correlation of agents. Our transformer-based network aims at generating high-quality logic rules based on spatial-temporal relation under the principle to maximize the likelihood of the observational human actions. For NSP-SFM [30], it obtains high performance in SDD dataset but can not achieve the same level in NBA dataset. It combines physics with deep learning for trajectory prediction and accommodates arbitrary physics models. The limitation of it lies in specific physics models, such as pedestrians, and is deterministic. So it can not deal with some strategy-based conditions, including basketball and football games. But our logic-learning method tries to use a set of spatial-temporal logic rules with intention variables involved as principles to model the dynamics of human actions, not restrained into specific conditions.
Further, The proposed model achieved the best scores in F1 score, which is an balanced metric considering both recall and precision. This is because the rule generator and evaluator can collaborate with each other to reduce search space and learn better rules. More experiments and ablation studies can be found in Supplementary Material.
Player A Rule ①: Left(B,A) ∧ Right(C,A) ∧ Front(D,A) → Pass(A,B) Explanation:
- Player A is carrying the ball
- Player A is defended by player C and player D
- Player B is on the left of player A
- Player A passes the ball to player B
Player B Rule ②: Left(E,B) ∧ Right(D,B)  $\rightarrow$  Go_Front(B) Explanation:
- Player E and player F is moving to the player B
- Player B is defended by player E on left side
- Player B is defended by player D on right side
- Player B goes front and carries the ball
Figure 5: Visualization and explanation of logic rules in NBA dataset.
Player B Rule ③: Left(E,B) ∧ Right(G,B) ∧ Front(F,B) → Shoot(B) Explanation:
- Player B is defended by player E on left side
- Player B is defended by player G on right side
- Player B is defended by player F on front side
- Player B shoots at the basket
$\bigcirc / \bigcirc$  basketball player
To evaluate the robustness of our method, We added some noise and randomly removed several tracks in the NBA dataset, then evaluated all methods in Table 3 ("Ours*") is the original results without noises or missing tracks). As we can see, our method still achieves the best performance. Moreover, by comparing "Ours" and "Ours*", we can see that the quality of tracks has less influence on our method, which demonstrates the robustness of our method.
Our method not only outperforms all the above comparative methods in quantitative evaluations, but also produces visually pleasing results among them. Figure 4 compares the predicted trajectories of NMMP, ground-truth (GT) and our methods on NBA dataset. It is obvious that our method produces more precise predictions than state-of-the-art method NMMP. This is because our proposed spatial-temporal logic rules can actually captures the spatial relation of the players with surrounding players, as well as the temporal ordering constraints of the events.
5.4 Generated Logic Rules
We add visualization and explanation about the logic rule and corresponding actions from NBA dataset in Figure 5. Note that the static spatial relation  $\{\text{Left}(\mathsf{B},\mathsf{A})\}$  represents that the player B is on the left of player A. And the dynamic spatial relation  $\{\text{Away}(\mathsf{A},\mathsf{E})\}$  means that player A is getting away from player E. We can see that these logic rules are meaningful and diverse. In this picture, player A is defended by two players and then passes the ball to player B. Player B goes front and crossover to bypass three defenders and shoot at the basket. Our rule can actually represent their offensive strategy. In fact, our framework can actually adapt to some complex motions, such as cutting toward the ball, because these spatial-temporal predicates are fed into the rule generator and evaluator to obtain high-quality rules to explain the intention of players.
Table 3: Robustness of our method in NBA dataset in some noise.
Methods	1.0	2.0	3.0	4.0
ADE	FDE	ADE	FDE	ADE	FDE	ADE	FDE
Y-Net	0.51	0.62	0.81	1.11	1.08	1.49	1.37	1.86
NSP-SFM	0.48	0.63	0.82	1.13	1.06	1.46	1.36	1.82
Ours	0.31	0.48	0.67	0.91	0.93	1.39	1.18	1.66
Ours*	0.30	0.40	0.58	0.88	0.87	1.31	1.13	1.60
It's challenging to define some complex predicates with richer meanings for different datasets. Given a more informative dataset, our method can discover more principles-like complex rules. Our motivation is to consider the spatial-temporal relation between pedestrians and generate some high-quality logic rules to explain their behaviors. Although we only choose some simple actions in our experiments, they can bring some benefits for understanding the principles of biological agents' behaviors. Our framework is suitable for more complex conditions, supposing more sophisticated action predicates can be obtained from the data.
We proposed a framework for learning intrinsic spatial-temporal logic rules for explaining human actions. We regard logic rules as latent variables, and the rule generator as well as the rule evaluator are jointly learned with EM-based algorithm. In the experiments, our method can analyze the biological movement sequence of pedestrians and players, and obtained novel insights from generated logic rules. In the future, we plan to incorporate other physical laws into the models, such as conservation of energy and momentum to enhance robustness of our model.
Shuang Li's research was in part supported by the NSFC under grant No. 62206236, Shenzhen Science and Technology Program JCYJ20210324120011032, National Key R&D Program of China under grant No. 2022ZD0116004, and Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence.
[1] David Adam. Science and the world cup: how big data is transforming football. Nature, 611(7936):444-446, 2022.
[2] Claudine Badue, Rănik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M Paixao, Filipe Mutz, et al. Self-driving cars: A survey. Expert Systems with Applications, 165:113816, 2021.
[3] Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. In International Conference on Learning Representations, 2021.
[4] Chang Choi, Junho Choi, Eunjil Lee, Ihsun You, and Pankoo Kim. Probabilistic spatio-temporal inference for motion event understanding. Neurocomputing, 122:24-32, 2013.
[5] Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. Boolean decision rules via column generation. Advances in neural information processing systems, 31, 2018.
[6] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122, 2022.
[23] Bilong Shen, Xiaodan Liang, Yufeng Ouyang, Miaofeng Liu, Weimin Zheng, and Kathleen M Carley. Stepdeep: A novel spatial-temporal mobility event prediction framework based on deep neural network. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 724-733, 2018.
[24] Li-Wu Tsao, Yan-Kai Wang, Hao-Siang Lin, Hong-Han Shuai, Lai-Kuan Wong, and Wen-Huang Cheng. Social-ssl: Self-supervised cross-sequence representation learning based on transformers for multi-agent trajectory prediction. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXII, pages 234–250. Springer, 2022.
[25] Tong Wang, Cynthia Rudin, Finale Doshi-Velez, Yimin Liu, Erica Klampfl, and Perry MacNeill. A bayesian framework for learning rule sets for interpretable classification. The Journal of Machine Learning Research, 18(1):2357-2393, 2017.
[26] Baowen Xu, Xuelei Wang, Shuo Li, Jingwei Li, and Chengbao Liu. Social-cvae: Pedestrian trajectory prediction using conditional variational auto-encoder. In International Conference on Neural Information Processing, pages 476–489. Springer, 2023.
[27] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6507, 2022.
[28] Fan Yang, Zhilin Yang, and William W Cohen. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems, 30, 2017.
[29] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 507-523. Springer, 2020.
[30] Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 376-394. Springer, 2022.
[31] Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference, pages 777-789. PMLR, 2022.