SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chengzhi Cao $^{1,2}$ ; Chao Yang $^{1}$ , Ruimao Zhang $^{1}$ , Shuang Li $^{1\dagger}$
<sup>1</sup>The Chinese University of Hong Kong (Shenzhen)
$^{2}$ University of Science and Technology of China
chengzhicao@mail.ustc.edu.cn, 222043011@link.cuhk.edu.cn,
zhangruimao@cuhk.edu.cn, lishuang@cuhk.edu.cn
We propose a logic-informed knowledge-driven modeling framework for human movements by analyzing their trajectories. Our approach is inspired by the fact that human actions are usually driven by their intentions or desires, and are influenced by environmental factors such as the spatial relationships with surrounding objects. In this paper, we introduce a set of spatial-temporal logic rules as knowledge to explain human actions. These rules will be automatically discovered from observational data. To learn the model parameters and the rule content, we design an expectation-maximization (EM) algorithm, which treats the rule content as latent variables. The EM algorithm alternates between the E-step and M-step: in the E-step, the posterior distribution over the latent rule content is evaluated; in the M-step, the rule generator and model parameters are jointly optimized by maximizing the current expected log-likelihood. Our model may have a wide range of applications in areas such as sports analytics, robotics, and autonomous cars, where understanding human movements are essential. We demonstrate the model's superior interpretability and prediction performance on pedestrian and NBA basketball player datasets, both achieving promising results.
For a human, although the exhibited movements can be complex, the logic behind the actions is usually simple, clear, and can be generalized [17]. The logic rules present a compact and high-level knowledge representation, defining what actions tend to be executed under what conditions. It emphasizes reasoning, and encourages structuring the world in terms of objects, properties, and relations [22]. There has been a great interest and business value in unveiling the human logic from the observational movements and actions [18]. We provide two motivating examples below.
In sports analytics, understanding each player's behavior preferences or tendencies under various scenarios will provide coaches with valuable information [1]. Usually, coaches need to watch the game or training videos for hundreds of hours before they can summarize the discoveries into compact principles. Could we design an algorithm to synthesize these principles from the raw action data automatically? One can imagine, such a tool will significantly reduce the workload of coaches, providing more granular insight into each player's capabilities and strategies, and aiding in personalized training and match strategy design [15].
In self-driving car, it's essential to enable the self-driving cars to "read human's mind" like humans. This requires the self-driving cars to automatically understand human intentions and reasoning when
*Work done as a graduate research assistant at The Chinese University of Hong Kong (Shenzhen)
†Corresponding author.
they are running on the same roads with human drivers [2]. If self-driving cars can automatically distill logic rules from their observed low-level noisy human actions and movement trajectories, it will increase the technical reliability and accelerate the widespread use of self-driving cars.
For human actions, lots of the governing rules would be regarding the spatial-temporal relation with the surrounding environments and their intentions [9]. For example, when a basketball player with a ball is within the scoring range, his/her action, such as shoot, pass, or triple threat, is influenced by historical and current surrounding factors, such as the current locations of the player and the defenders, the time elapses of the game, the success shooting rate of the player in today's game, and so on. The quick decision made by the player is actually reflecting a composition of all these factors, which can be described as a collection of spatial-temporal logic rules in our model.
Formally, in our introduced spatial-temporal logic rules, the logic variable (i.e., predicate) set will include spatial-temporal relation predicates, in addition to the commonly defined object property and relation predicates. The rule content will capture the spatial relation of the object with surrounding objects, as well as the temporal ordering constraints of the events.
Our methods have the following distinct features:
From the modeling perspective: (i) Our human action model is a rule-based probabilistic model, which treats each hidden rule as a "soft" constraint. We assume each rule will be executed by humans with probabilities, and this tolerates the uncertainties in data. (ii) Our model directly uses low-level, fine-grained, and (may) irregularly-spaced action times and locations (i.e., original 3d coordinates) as inputs, as opposed to other rule-based models, where one needs to first extract relational data as inputs. (iii) Our spatial and temporal predicates are also probabilistic. For predicates such as "left of" or "before", we model them as kernel functions with learnable variables. In this way, our introduced spatial-temporal predicates are smooth functions of the input locations and times, which increases model flexibility.
From the learning perspective: We propose a tractable and differentiable algorithm that can jointly learn the rule content and model parameters from observational data. The learning framework is designed to maximize the likelihood of the human action trajectories. Specifically, we propose to use a neural rule generator to generate the spatial-temporal logic rule set. Our continuous rule generator parameters will be optimized in a differentiable way. The overall learning procedure is an expectation-maximization (EM) algorithm, where we treat the rule set as latent variables. In the E-step, the posterior distribution over the latent rule set is evaluated. In the M-step, both the rule generator parameters and the model parameters are optimized by maximizing the expected log-likelihood with respect to the current posterior. We demonstrated the promising performance of our model in terms of human action prediction and explanation on two interesting real datasets.
Logic Rule Learning. Learning logic rules from raw data has been widely studied for various downstream tasks, such as motion inference [4] and healthcare analysis [11]. Learning rules via an exact search requires enumerating all combinations of the logic predicates and is intractable in most problems. One has to design heuristic searching algorithms by leveraging some structural properties of the problems. For example, Dash et al. [5] formulated a convex rule learning problem and proposed a column generation algorithm to expand the rule set gradually. Wang et al. [25] designed a Bayesian framework for learning rule classifiers and derived bounds on the support of rules in a MAP solution. Recently, Yang et al. [28] proposed an interesting end-to-end differentiable approach (Neural LP) to learn the parameters and structure of logical rules. Qu et al. [19], and Sadeghian et al. [22] proposed an efficient logic rule mining algorithm based on the knowledge graph data. However, none of these advanced rule mining methods can directly work on spatial-temporal human action data when the inputs are raw event 3d coordinates and types.
Spatio-Temporal Dynamics for Event Data. Since the human actions are irregular spatial-temporal event data, we also briefly discuss probabilistic models for such event sequences. Modeling the spatial-temporal dynamics of discrete events is foundational in many scientific fields and applications [20]. Shen et al. [23] proposed a novel deep learning model for spatial-temporal events such as taxi data and achieved promising prediction accuracy. Zhou et al. [31] integrated deep learning methods with spatiotemporal point processes and modeled the intensity function as a latent stochastic process. Chen et al. [3] deployed two novel architectures, including jump and attentive continuous-time normalizing flows, to learn the dynamics of the spatiotemporal event data. Repe et al.
[21] learned canonical spatiotemporal point cloud representation using a latent ODE and continuous normalizing flows to generate shapes continuously in spacetime. However, these spatial-temporal event models are governed by hard-to-interpret dynamic functions and cannot be generalized to model human action events. Could we propose a model with logic-informed dynamic functions to explain the spatial-temporal human action events?
3 Our Model
3.1 Data: Human Actions Recorded as Spatial-Temporal Event Sequences
Consider a set of objects, denoted as  $\mathcal{C}$ . For the object  $c \in \mathcal{C}$ , its trajectories and the key actions observed up to  $t$  can be summarized as a sequence of temporally ordered events:
$$
\mathcal {H} _ {t -} ^ {c} = \left\{e _ {1} ^ {c} = \left(t _ {1} ^ {c}, s _ {1} ^ {c}, \kappa_ {1} ^ {c}\right), \dots , e _ {n} ^ {c} = \left(t _ {n} ^ {c}, s _ {n} ^ {c}, \kappa_ {n} ^ {c}\right) \mid t _ {n} ^ {c} <   t \right\}, \tag {1}
$$
where  $t \in R^{+}$  is the time,  $s \in R^{2}$  is the location, and  $\kappa \in \mathcal{K}$  is the event (i.e., action) type.
3.2 Definition of Spatial-Temporal Predicates
Static Predicate Given the object set  $\mathcal{C}$ , the predicate is defined as the property or relation of objects, which is a logic function as follows:
$$
X (\cdot): \mathcal {C} \times \mathcal {C} \dots \times \mathcal {C} \mapsto \{0, 1 \}. \tag {2}
$$
For example,  $Smokes(c)$  is a property predicate and  $Friend(c, c')$  is a relation predicate.
Temporal relation predicates They can be used to define the temporal relations of two action events. We consider three types of temporal relation predicates {before, equal, after} as:
$$
R _ {\text {B e f o r e}} \left(t _ {1}, t _ {2}\right) = \mathbb {1} \left\{t _ {1} - t _ {2} <   0 \right\},
$$
$$
R _ {\text {A f t e r}} \left(t _ {1}, t _ {2}\right) = \mathbb {1} \left\{t _ {1} - t _ {2} > 0 \right\}, \tag {3}
$$
$$
R _ {\text {E q u a l}} \left(t _ {1}, t _ {2}\right) = \mathbb {1} \left\{t _ {1} = t _ {2} \right\}.
$$
We will treat the temporal relation predicate as either a boolean variable or a real-valued function. If the time information is imprecisely recorded, we can parameterize the temporal relation predicates as temporal kernel functions that map to  $[0,1]$ , which is a function of  $t_1, t_2$  with learnable parameters.
Spatial-Temporal Predicate In our paper, we extend the above static predicates to spatial-temporal predicates, which include spatial-temporal property predicates and spatial-temporal relation predicates.
Specifically, the spatial-temporal property predicates are defined as:
$$
X (\cdot): \mathcal {C} \times \dots \times \mathcal {C} \times \mathcal {T} \times \mathcal {S} \mapsto \{0, 1 \}. \tag {4}
$$
For example, PickupKey  $(c, t, s)$  is a spatial-temporal property predicate. Suppose an entity  $c_1$  picked up the key at time  $t_1$  in location  $s_1$ , then the predicate will be grounded as True (1) at  $(c_1, t_1, s_1)$ , i.e., PickupKey  $(c_1, t_1, s_1) = 1$ ; otherwise it is false.
Given the observational human action data, the grounded predicate  $\{PickupKey(c,t,s)\}_{t = 1,2,\ldots}$  can be modeled as a sequence of discrete events - when the predicate becomes True, an event happens. In general, the grounded spatial-temporal property predicate is a discrete event sequence, where the event occurrence times and locations are irregular.
The spatial-temporal relation predicates are introduced to define the spatial and temporal relations of two entities. Specifically, they are defined as:
$$
R (\cdot , \cdot): (\mathcal {C} \times \mathcal {T} \times \mathcal {S}) \times (\mathcal {C} \times \mathcal {T} \times \mathcal {S}) \mapsto \{0, 1 \}. \tag {5}
$$
Spatial-temporal relation predicates are logic variables indicating the spatial-temporal relations of two objects, where we further divide them into temporal relation predicates, static spatial relation predicates, and dynamic spatial relation predicates. More details can be found in Appendix.
It is noteworthy that all these boolean predicates can be converted to probabilistic ones. We can soften these logic functions by kernel functions with learnable parameters to tolerate uncertainties in data.
3.3 Definition of Spatial-Temporal Logic Rules
We will consider spatial-temporal logic rules where the body part contains spatial-temporal predicates as relation constraints. For example, a sensible rule will look like:
$$
f: Y _ {\text {T u r n A r o u n d}} (c, t, s) \leftarrow X _ {\text {P i c k U p K e y}} (c, t, s) \bigwedge \tag {6}
$$
$$
R _ {\text {I n F r o n t}} \left(\left(c ^ {\prime}, t, s ^ {\prime}\right), (c, t, s)\right) \bigwedge R _ {\text {B e h i n d}} \left(\left(c ^ {\prime \prime}, t, s ^ {\prime \prime}\right), (c, t, s)\right),
$$
where  $c \in \mathcal{C}_{\mathrm{person}}$ ,  $c' \in \mathcal{C}_{\mathrm{block}}$ , and  $c'' \in \mathcal{C}_{\mathrm{key}}$ . "person", "block" and "key" are the specific instances in the object set  $\mathcal{C}$ , and this rule represents that a person wants to pick up a key, while one block is in front of him and the key is behind him, so he turns around. We utilize this example to manifest the meaning of logic rules. In general, the spatial-temporal logic rule in our paper is defined as a logical connectives of predicates, including property predicates and spatial-temporal relation predicates:
$$
f: Y (v) \leftarrow \bigwedge_ {X _ {\text {p r o p e r t y}} \in \mathcal {X} _ {f}} X _ {\text {p r o p e r t y}} (v) \bigwedge_ {R _ {\text {s p a t i a l - t e m p o r a l}} \in \mathcal {R} _ {f}} R _ {\text {s p a t i a l - t e m p o r a l}} \left(v ^ {\prime}, v\right), \tag {7}
$$
where  $Y(v)$  is the head predicate evaluated at the entity-time-location triplet  $v$ ,  $\mathcal{X}_f$  is the set of property predicates defined in rule  $f$ , and  $\mathcal{R}_f$  denotes the set of spatial-temporal relation predicates defined in rule  $f$ .
3.4 Logic-Informed Action Event Models
We consider a setting where we can fully observe the trajectories of all the moving objects, including their real-time locations and key actions (i.e., events), denoted as  $\mathcal{H}_t$ . We aim to propose a logic-informed spatial-temporal model to predict and explain the action type given the entity-time-location triplet  $v = (c,t,s)$  (i.e., query) and  $\mathcal{H}_t$ . The main idea is to construct the model features using spatial-temporal logic rules, as defined in Eq. (7). Intuitively, given the entire trajectories  $\mathcal{H}_t$  and the query  $v = (c,t,s)$ , the body part of the rule defines the evidence to be selectively gathered from history to deduce the event type for query entity  $v = (c,t,s)$ . Assume that for each possible event type  $\kappa \in \mathcal{K}$ , there exist multiple rules such as Eq. (7) to explain its occurrence, with  $\kappa$  being the head predicate. Given an individual rule as Eq. (7), we propose to build the feature that is conditional on history and query as
$$
\phi_ {f} (\kappa \mid v, \mathcal {H} _ {t}) = \operatorname {s i g n} (\kappa \in f) \cdot \sum_ {\text {p a t h} \in \left\{\mathcal {H} _ {t}, v \right\}} g _ {f} (\text {p a t h}), \tag {8}
$$
where we introduce a function  $g_{f}(\cdot)$  to check the body conditions of  $f$  given a "path". We use a simple example to explain how to compute features, as shown in Figure 1. As illustrated, the feature computes the valid total number of "paths" given the data and query. Suppose there is a rule set  $\mathcal{F}_{\kappa}$ , where the event  $\kappa$  is the head predicate. All the rules will play together to reason about the occurrence of  $\kappa$ . For each  $f \in \mathcal{F}_{\kappa}$ , one can compute the features as above. Given the rule set  $\mathcal{F}_{\kappa}$ , we model the probability of the event  $\kappa$  as a log-linear
Figure 1: Illustration of feature construction using a simple logic formula with temporal relation predicate  $(t_1 < t_2)$ ,  $f: Y \leftarrow A \land B \land C \land (A \text{ Before } B)$ . The rule defines the template to gather combinations of the body predicate history events. Here predicate A has 2 events and predicate B has 1 event, the temporal relation constraint would lead to valid combinations (also called "paths"). This type of feature construction can be extended to spatial-temporal cases, where we count the valid paths as the feature.
$$
p (\kappa | v, \mathcal {H} _ {t}) \propto \exp \left(\sum_ {f \in \mathcal {F} _ {\kappa}} w _ {f} \cdot \phi_ {f} (\kappa | v, \mathcal {H} _ {t})\right), \tag {9}
$$
where  $w = [w_{f}]_{f\in \mathcal{F}}\geq 0$  are the learnable weight parameters associated with each rule. All the model parameters can be learned by maximizing the likelihood, which can be computed using the above Eq. (9). We intend to train a rule generator  $p_{\theta}$  and an evaluator  $p_w$  to maximize the likelihood of training data as:
$$
\max  _ {\theta , w} \mathcal {O} (\theta , w) = \mathbb {E} _ {(\kappa , v, \mathcal {H} _ {t})} [ \log \mathbb {E} _ {p _ {\theta}} [ p _ {w} (\kappa | v, \mathcal {H} _ {t}) ] ]. \tag {10}
$$
More details can be found as follows.
4 Our Learning Algorithm
Our goal is to jointly learn the set of spatial-temporal logic rules  $\{\mathcal{F}_{\kappa}\}_{\kappa \in \mathcal{K}}$  and their weights by the maximum likelihood method, where each rule has a general form as Eq. (7).
To discover each rule, the algorithm needs to navigate through the combinatorial space considering all the combinations of the property predicates and their spatial and temporal relations. To address this computational challenge, we propose a tractable (functional) EM algorithm that treats the rule set as latent variable  $z$ . The rules will be generated by a hidden neural rule generator. The overall learning framework alternates between an E-step, where the posterior distribution of the latent rule space is evaluated (rule generation), and the M-step, where the model parameters and rule generator parameters are optimized. Please refer to Figure. 2 for an illustration.
Figure 2: The overview of our proposed framework. It contains two important processes: rule generating and logic reasoning. Given the past motion of each entity on a scene over the last few seconds, the rule generator generates logic rules for the reasoning predictor. The reasoning predictor takes the generated rules as input, and predict the intention of each entity. It is optimized by EM algorithm. In the E-step, a set of top K rules are selected from all generated rules via posterior inference. Finally in the M-step, the rule generator is updated to be consistent with the high-quality rules identified in E-step.
Our goal is to maximize the likelihood of the observed human action events  $\{\kappa^{(i)}\}_{i = 1,\dots ,n}$ . Using the chain rule, we have
$$
\log p _ {w} \left(\left\{\kappa^ {(i)} \right\} _ {i = 1, \dots , n}\right) = \sum_ {i = 1} ^ {n} \log p _ {w} \left(\kappa^ {(i)} \mid v ^ {(i)}, \mathcal {H} _ {t (i - 1)}\right). \tag {11}
$$
To simplify the notation, we will use  $p_w(\kappa^{(i)})$  to stand for  $p_w(\kappa^{(i)} \mid v^{(i)}, \mathcal{H}_{t^{(i-1)}})$  in the following. Given a latent rule set  $z$ , we have to marginalize the posterior of  $z$  to get the above log-likelihood. However, the exact inference of  $z$  is intractable. We will introduce an amortized recognition network  $p_\theta(z|\kappa^{(i)})$  to approximate the true posterior. We have
$$
\log p _ {w} \left(\kappa^ {(i)}\right) = D _ {K L} \left(p _ {\theta} \left(z \mid \kappa^ {(i)}\right) \mid \mid p _ {w} \left(z \mid \kappa^ {(i)}\right)\right) + \mathcal {L} (\theta , w; \kappa^ {(i)}), \tag {12}
$$
where the first term is the KL divergence of the approximate from the true posterior, and the second term  $\mathcal{L}(\theta ,w;\kappa^{(i)})$  is the variational lower bound (ELBO). It can be represented as:
$$
\mathcal {L} (\theta , w; \kappa^ {(i)}) = - D _ {K L} \left(p _ {\theta} \left(z \mid \kappa^ {(i)}\right) \| p _ {w} (z)\right) + \mathbb {E} _ {p _ {\theta} \left(z \mid \kappa^ {(i)}\right)} \left[ \log p _ {w} \left(\kappa^ {(i)} \mid z\right) \right]. \tag {13}
$$
And  $\log p_w(\kappa^{(i)})\geq \mathcal{L}(\theta ,w;\kappa^{(i)})$ . The bound becomes tight when the approximate posterior matches the true one. Our goal is to optimize the variational parameters  $\theta$  and model parameters  $w$  from the ELBO lower bound.
4.1 Rule Generator
We deploy Transformer-based framework to model the rule generator  $p_{\theta}$ . We define the distribution of a set of rules as follows:
$$
p _ {\theta} (z \mid v, \mathcal {H} _ {t}) = \Psi (z | N, \operatorname {T r a n s} _ {\theta} (v, \mathcal {H} _ {t})), \tag {14}
$$
where  $\Psi(\cdot)$  is multinomial distributions,  $N$  is the number of the top rules, and  $\mathrm{Trans}_{\theta}(v, \mathcal{H}_t)$  defines a distribution over compositional rules with spatial-temporal states. The generative process of the rule set is quite intuitive, where we simply generate  $N$  rules to form  $z$ . In fact, this  $p_{\theta}(z \mid v, \mathcal{H}_t)$  is a flexible posterior approximation function, which will be optimized by the EM type algorithm.
We choose transformer over graph neural network (GNN) as our baseline because transformer architectures are based on a self-attention mechanism that is able to capture long-range relationships, as opposed to recurrent neural networks that process sequence elements recursively and can only take into account short-term context. Note that the graph operations in GNN are designed to learn node representations on the fixed and homogeneous graphs. The limitations especially become problematic when learning representations on a changeable graph that consists of various types of nodes and edges.
4.2 Rule Evaluator
Eq. (9) is our rule evaluator (suppose we know the rule content). Here we assume the rule content is latent, and the rule evaluator is given as
$$
p _ {w} (\kappa | v, z, \mathcal {H} _ {t}) = \frac {\exp \left(\sum_ {f \in z _ {\kappa}} w _ {f} \cdot \phi_ {f} (\kappa | v , \mathcal {H} _ {t})\right)}{\sum_ {\kappa^ {\prime}} \exp \left(\sum_ {f \in z _ {\kappa^ {\prime}}} w _ {f} \cdot \phi_ {f} \left(\kappa^ {\prime} | v , \mathcal {H} _ {t}\right)\right)}. \tag {15}
$$
We optimize the rule generator  $p_{\theta}$  and reasoning evaluator  $p_w$  to maximize the objective in Eq. (10). At each training iteration, we first update the reasoning predictor  $p_w$  according to some rules generated by the generator, and then update the rule generator  $p_{\theta}$ .
In our network, the latent rule set will be automatically discovered. The best set of logic rules is approximately obtained by sampling and preserving the top-K rules according to their posterior probabilities. Specifically, as shown in Eq. (14), the posterior probabilities of the latent rule  $z$  is obtained by a Transformer type of encoder, which maps the input observed action trajectories to a latent explanatory rule space. Each candidate rule is generated in the latent rule space token-by-token (token means logic variable/predicate in our setting) in a sequential manner and meanwhile the posterior probability of each rule sequence can be evaluated. When optimizing the evaluator, we draw several rules  $\hat{z}$  for each query and let the evaluator uses  $\hat{z}$  to predict  $\kappa$ . For each query, we try to identify top-K rules  $z_{I}$  from all generated rules  $\hat{z}$ . It is accomplished by taking into account the posterior probabilities of each subset of logic rules  $z_{I}$  with prior from the rule generator  $p_{\theta}$  and likelihood from the reasoning predictor  $p_w$ . Specifically, when a series of rules produced from the rule generator  $p_{\theta}$ , we calculate the weights of each rule  $z^{(i)}$  as follows:
$$
J \left(z ^ {(i)}\right) = \left\{p _ {w} \left(\kappa \mid z ^ {(i)}\right) - \frac {1}{| \mathcal {A} |} \right\} + \log \operatorname {T r a n s} _ {\theta} \left(z ^ {(i)} \mid v, \mathcal {H} _ {t}\right), \tag {16}
$$