Temporal and Object Quantification Nets

1 INTRODUCTION . When watching a soccer match ( Fig . 1 ) , we see more than just players and a ball moving around . Rather , we see events and actions in terms of high-level concepts , including relations between agents and objects . For example , people can easily recognize when one player has control of the ball , or when a player passed the ball to another player . This cognitive act is effortless , intuitive , and fast . Machines can recognize actions , too , but generally based on limited windows of space and time , and with weak generalization . Consider the variety of complicated spatio-temporal trajectories that passing can refer to ( Fig . 1 ) , and how an intelligent agent could learn this . The act of passing does not seem to be about the pixel-level specifics of the spatio-temporal trajectory . Rather , a pass is a high-level action-concept that is composed of other concepts , such as possession ( A pass begins with one player in possession of the ball and ends with another player in possession ) or kick . The concept of passing can itself be re-used in a compositional way , for example to distinguish between a short pass and a long pass . We propose Temporal and Object Quantification Nets ( TOQ-Nets ) , structured neural networks that learn to describe complex activities by quantifying over both entities and time . TOQ-Nets are motivated by the way in which humans perceive actions in terms of the properties of and relations between agents and objects , and the sequential structure of events ( Zacks et al. , 2007 ; Stränger & Hommel , 1996 ) . A TOQ-Net is a multi-layer neural network whose inputs are the properties of agents and objects and their relationships in a scene , which may change over time . In a soccer game , these inputs might be the 3D position of each player and the ball . Each layer in the TOQ-Net performs either object or temporal quantification , which can emulate and realize disjunctive and conjunctive quantification over the properties and relationships between entities , as in “ the kicker is behind all the other players ” , as well as quantification over time , as in “ the player controls the ball until it moves fast towards the goal. ” The key idea of TOQ-Nets is to use tensors to represent the relational features between agents and objects ( e.g. , the player controls the ball , and the ball is moving fast ) , and to use tensor pooling operations over different dimensions to realize temporal and object quantifiers ( all and until ) . Thus , by stacking these object and temporal quantification operations , TOQ-Nets can learn to construct higher-level concepts of actions based on the relations between entities over time , starting from low-level position and velocity input and supervised with only high-level class labels . We evaluate TOQ-Nets on two perceptually and conceptually different benchmarks for action recognition : trajectory-based soccer event detection and 6D pose-based manipulation concept learning . We show that the TOQ-Net makes several important contributions . First , TOQ-Nets outperform both convolutional and recurrent baselines for modeling relational-temporal concepts across both benchmarks . Second , by exploiting the temporal-relational features learned through supervised learning , TOQ-Nets achieve strong few-shot generalization to novel actions . Finally , TOQ-Nets exhibit strong generalization to scenarios where there are more agents and objects than were present during training . They are also robust w.r.t . time warped input trajectories . Meanwhile , the learned concepts can also be easily transferred to other prediction tasks in a similar domain . 2 RELATED WORK . Action concept representations and learning . TOQ-Nets impose a structural inductive bias that describes actions with quantification over objects ( every entity ... , there exists an entity ... ) and quantification over times ( an event happens until some time and then some other event begins ) , which is motivated by first-order and linear temporal logics ( Pnueli , 1977 ) . Such representations have been studied for analyzing sporting events ( Intille & Bobick , 1999 ; 2001 ) and daily actions ( Tran & Davis , 2008 ; Brendel et al. , 2011 ) using logic-based reasoning frameworks . However , these frameworks require extra knowledge to annotate relationships between low-level , primitive actions and complex ones . By contrast , TOQ-Nets enable end-to-end learning of complex action descriptions with only high-level action-class labels . Cognitive science has long recognized the importance of structural representations of actions and events in human reasoning , including in language , memory , perception , and development ( see e.g . Stränger & Hommel , 1996 ; Zacks et al. , 2001 ; 2007 ; Pinker , 2007 ; Baldwin et al. , 2001 ) . In this paper , we focus on the role of object and temporal quantification in learning action representations . Temporal and relational reasoning . This paper is also related to work on using neural networks and other data-driven models for modeling temporal structure . Early work includes ADL description languages ( Intille & Bobick , 1999 ; Zhuo et al. , 2019 ) , hidden Markov models ( Tang et al. , 2012 ) , and and-or graphs ( Gupta et al. , 2009 ; Tang et al. , 2013 ) . These models need human-annotated action descriptions ( e.g. , pick up x means a state transition from not holding x to holding x ) and specialpurpose inference algorithms such as graph structure learning algorithms . In contrast , TOQ-Nets have an end-to-end design and can be integrated with arbitrary differentiable modules such as convolutional neural networks . People have also used structural representations to model object-centric temporal concepts with temporal graph convolution networks ( Yan et al. , 2018 ; Materzynska et al. , 2020 ; Wang & Gupta , 2018 ; Ji et al. , 2020 ) . Meanwhile , Deng et al . ( 2016 ) proposed to integrate graph neural networks with RNNs by replacing the message aggregation step in GNN propagation with RNNs to capture temporal information . Bialkowski et al . ( 2014 ) ; Ibrahim et al . ( 2016 ) ) have proposed to build a personal-level and a group-level feature extractor for group activity recognition . The high-level idea is to use RNNs to encode per-person features across the video and use another RNN to combine features of individual persons into a group feature for every frame . StagNet Qi et al . ( 2018 ) and Spatial-Temporal Interaction Networks ( Materzynska et al. , 2020 ) combines graph neural networks with RNNs and spatial-temporal attention models to model temporal information . TOQ-Nets use a similar object-centric relational representation , but different models for temporal structures . There are also alternative models for temporal structures from pixels , such as convolutions ( Carreira & Zisserman , 2017 ; Ji et al. , 2020 ) , recurrent networks ( Dong et al. , 2019 ) , attention ( Wang et al. , 2018 ) , or segmentation consensus and its variants ( Wang et al. , 2016 ; Zhou et al. , 2018 ) . In contrast to these methods , TOQ-Nets works on an object-centric temporal-relational feature space , and uses the proposed temporal quantification operations to model the sequential structures of events . 3 TEMPORAL AND OBJECT QUANTIFICATION NETS . TOQ-Nets , shown in Fig . 2 , are multi-layer neural networks with a structural inductive bias for learning composable action concepts using object and temporal quantification . The input to a TOQNet is a tensor representation of the properties of all entities at each moment in time . For example , in a soccer game , the input encodes the position of each player and the ball , as well as the team membership of each player . The output is a label for the action performed by a target entity . The first layer of a TOQ-Net ( Fig . 2 ( i ) ) extracts temporal features for each entity with an input feature extractor that focuses on entity features within a fixed and local time window . These features may be computed by a convolutional neural network , such as STGCN ( Yan et al. , 2018 ) or a bank of parametric feature templates , such as thresholding the velocity of an entity : whether the velocity of an entity x is greater than a learnable parameter θ at time t. The output of this step is thus a collection of nullary , unary , and binary relational features over time for all entities . Throughout the paper we will assume that all output tensors of this layer are binary-valued , although the one can easily extend this feature extractor to any real-valued function . This input feature extractor is usually task- and dataset-specific and is not the main focus of this paper . Second , the temporal-relational features go through several relational reasoning layers , which we present in detail in Section 3.2 . In short , each layer at this step performs linear transformations , sigmoid activation , and object quantification operations . The linear and sigmoid functions allow the network to realize Boolean logical forms , and object quantification operators can realize quantifiers . With more relational reasoning layers , the network can realize logical forms with more quantifiers , as illustrated in Fig . 2 . All operators in these layers are performed for all time steps in parallel . Next , the relational reasoning layers summarize each event at a particular time step as a nullary feature that is passed to the to the temporal reasoning layer , as detailed in Section 3.3 . Each layer at this step performs linear transformations , sigmoid activation , and temporal quantification . They allow the model to realize logical forms in a simple linear temporal logic ( Pnueli , 1977 ) , such as always ∃x . q1 ( x ) ∧ q2 ( x ) . This is semantically equivalent to the temporally quantified expression ∀t . ∃x . q1 ( x , t ) ∧q2 ( x , t ) . Similar to the relational reasoning layers , adding more temporal reasoning layers enables the network to realize logical forms with more deeply nested temporal quantifiers . In the last layer , all object and time information is projected into a representation in the initial time step , which summarizes what takes place in the entire trajectory ( e.g. , the event “ the kicker eventually scores ” may be recognized at the first time step ) . We feed this representation into a final softmax unit , and get classification probabilities for the event that the network is trained to detect . Remark . It is important to understand the representational power of this model . The input transformation layer can be seen as defining basic predicates and relations that will be useful for defining more complex concepts , but no specific predicates or relations are built into it . The learned outputs might be properties that humans can interpret , such as “ moving fast ” or “ far from. ” The relational reasoning layers can build quantified expressions over these basic properties and relations , and might construct expressions that could be interpretable as “ the player is close to the ball. ” Finally , the temporal reasoning layer can represent temporal operations applied to these complex expressions , such as “ the player is close to the ball until the ball moves with high speed. ” Critically , none of the symbolic properties or predicates are hand defined—they are all constructed by the initial layer in order to enable the network to express the concept it is being trained on . Compared with the expressiveness of first-order linear temporal logic , TOQ-Nets only work in finite universes : all quantifiers only have finite interpretations . Meanwhile , the depth of the logical forms that it can learn is fixed . However , our goal is not to fully replicate temporal logic , but to bring ideas of object and temporal quantification into neural networks , and to use them as structural inductive biases to build models that generalize better from small amounts of data to situations with varying numbers of agents , objects , and time courses .

Summary: This paper proposes TOQ-Nets which is a structured neural network that learns to describe complex activities over entities and time. The model leverages relational reasoning layers which are the Neural Logic Machines (NLM) to capture the spatial information. To further capture the temporal information, this paper proposes temporal reasoning layers.The results show that their method outperforms conventional graph neural networks with high accuracy and generalization with a large margin.