Temporal and Object Quantification Nets
Authors: Jiayuan Mao
Zhezheng Luo
Chuang Gan
Joshua B. Tenenbaum
Jiajun Wu
Leslie Pack Kaelbling
Tomer Ullman
Venue:  (2021)
Keywords: Temporal Modeling
Object-Centric Representations

Abstract: We aim to learn generalizable representations for complex activities by quantifying over both entities and time, as in “the kicker is behind all the other players,” or “the player controls the ball until it moves toward the goal.”  Such a structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses. In this paper, we present Temporal and Object Quantification Nets (TOQ-Nets), which provide such structural inductive bias for learning composable action concepts from time sequences that describe the properties and relations of multiple entities. We evaluate TOQ-Nets on two benchmarks: trajectory-based soccer event detection, and 6D pose-based manipulation concept learning. We demonstrate that TOQ-Nets can generalize from small amounts of data to scenarios where there are more agents and objects than were present during training. The learned concepts are also robust with respect to temporally warped sequences and easily transfer to other prediction tasks in a similar domain.

Under review as a conference paper at ICLR 2021
TEMPORAL AND OBJECT QUANTIFICATION NETS
Anonymous authors Paper under double-bling blind review
ABSTRACT
We aim to learn generalizable representations for complex activities by quantifying over both entities and time, in "the kicker is behind all the other players," "the player controls the ball until it moves toward the goal." Such structural inductive bias of object relations, object quantification, and temporal orders will enable the learned representation to generalize to situations with varying numbers of agents, objects, and time courses. In this paper, we present Temporal and Object Quantification Nets (etQ,Nets), which provide such structural inductive bias for learning composable action concepts from time sequences that describe the proper- ties and relations of multiple entities. We evaluate TOQ-Nets on two benchmarks: trajectory-basee based soccer event detection, and 6D pose- based manipulation concept learning. We demonstrate that TOQ-Nets data to scenarios where there are more agents and objects than were present during training. The learned concepts also robust with respect to temporally warped sequences, and easily transfer to other prediction tasks in a similar domain.
generalize from small amounts of
INTRODUCTION
When watching a soccer match (Fig. 1), we see more than just players and ball moving around. Rather, we see events and actions in terms of high-level concepts, including relations between agents and objects. For example, people can easily recognize when one player has control of the ball, or when a player passed the ball to another player. This cognitive act is effortless, intuitive, and fast.
Machines can recognize actions, too, but generally based on limited windows of space and time, and with weak generalization. Consider the variety of complicated spatio-temporal trajectories that passing can refer to (Fig. 1), and how an intelligent agent could learn this. The act of passing does not seem to be about the pixel-level specifics of the spatio-temporal trajectory. Rather, pass is high-level action-concept that is composed of other concepts, such as possession (A pass begins with one player in possession of the ball and ends with another player in possession) or kick. The concept of passing can itself be re-used in compositional way, for example to distinguish between a short pass and a long pass.
We propose Temporal and Object Quantification Nets (TOQ-Nets), structured neural networks that learn describe complex activities by quantifying over both entities and time. TOQ-Nets motivated by the way in which humans perceive actions in terms of the properties of and relations between agents and objects, and the sequential structure of events (Zacks et al., 2007; Stränger Hommel, 1996).
A TOQ-Net is a multi-layer neural network whose inputs are the properties of agents and objects and their relationships in scene, which may change over time. In a soccer game, these inputs might be the 3D position of each player and the ball. Each layer the TOQ-Net performs either object or temporal quantification, which can emulate and realize disjunctive and conjunctive quantification over the properties and relationships between entities, as in "the kicker behind all the other players", as well as quantification over time, in "the player controls the ball wntil it moves fast towards the goal." The key idea of TOQ-Nets is to use tensors to represent the relational features between agents and objects (e.g., the player controls the ball, and the ball is moving fast), and to use tensor pooling operations over different dimensions to realize temporal and object quantifiers (all and until). Thus, by stacking these object and temporal quantification operations, TOQ-Nets can lear to construct higher-level concepts of actions based on the relations between entities over time, starting from low-level position and velocity input and supervised with only high-level class labels.

---

Under review as a conference paper at ICLR 2021
Frame I-1
Framel
Frame ++1
Player
Ball
Plaver
Player
Ball
Plavert
Plaver
States
Ball 62, -0. -0.19, 0.113
10:58, -0.20, 0.111
10.65, -0.13, 0.401
3x3 (N=1)
Phoer 10.65, -0.17, 0.00] Player 2: 79, -0.06, 0,001
[9.60, -0.20, 0.001
10,56, -0.23 0.00]
10.76, -0.64,
10.69, -0.05 0.001
Action
Short
Possos(lhyerl, Ball)
Kick(PlayerI,
Posses<Player2, Ball)
Figure I: Illustrative example of action recognition in a soccer simulator. The action concept short
pass is composed of other concepts, such possession and kick.
We evaluate TOQ-Nets two perceptually and conceptually different benchmarks for action recog- nition: trajectory-baat -based soccer event detection and 6D pose-based manipulation concept learning. We show that the TOQ-Net makes several important contributions. First, TOQ-Nets outperform both convolutional and recurrent baselines for modeling relational-temporal concepts across both benchmarks. Second, by exploiting the temporal-relations features learned through supervised learning, TOQ-Nets achieve strong few-shot generalization to novel actions. Finally, TOQ-Nets exhibit strong generalization to scenarios where there more agents and objects than were present during training. They are also robust w.r.t. time warped input trajectories. Meanwhile, the learned concepts can also be easily transferred to other prediction tasks in similar domain.
RELATED WORK
Action concept representations and learning. TOQ-Nets impose structural inductive bias that describes actions with quantification over objects (every entity there exists an entity ...) and quantification over times event happens until some time and then some other event begins), which is motivated by first-order and linear temporal logics (Pnueli, 1977). Such representations have been studied for analyzing sporting events (Intille Bobick, 1999; 2001) and daily actions (Tran & Davis, 2008; Brendel et al., 2011) l) using logic-based reasoning frameworks. However, these frameworks require extra knowledge annotate relationships between low-level, primitive actions and complex ones. By contrast, TOQ-Nets enable end-to-end learning of complex action descriptions with only high-level action-class labels.
Cognitive science has long recognized the importance of structural representations of actions and events in human reasoning, including language, memory, perception, and development (see e.g. Stränger & Hommel, 1996; Zacks et al., 2001; 2007; Pinker, 2007; Baldwin et al., 2001). In this paper, we focus on the role of object and temporal quantification learning action representations.
Temporal and relational reasoning. This paper is also related to work on using neural networks and other data-driven models for modeling temporal structure. Early work includes ADL description languages (Intille & Bobick, 1999; Zhuo et al., 2019), hidden Markov models (Tang al., 2012), and and-or graphs (Gupta et al., 2009; Tang et al., 2013). These models need human-annotated action descriptions (e.g.. pick up x means state transition from not holding purpose inference algorithms such as graph structure learning algorithms. In contrast, TOQ-Nets have an end-to-end design and can be integrated with arbitrary differentiable modules such as convolutional neural networks. People have also used structural representations to model object-centric temporal concepts with temporal graph convolution networks (Yan et al., 2018: Materzynska al., 2020; Wang & Gupta, 2018; Ji et al., 2020). Meanwhile, Deng et al. (2016) proposed to integrate graph neural networks with RNNs by replacing the message aggregation step GNN propagation with RNNs to capture temporal information. Bialkowski et al. (2014); Ibrahim et al. (2016) have proposed to build a personal-leve and group-level feature extractor for group activity recognition. The high-level idea is to use RNNs to encode per-person features across the video and use another RNN to combine features of individual persons into group feature for every frame. StagNet Qi et al. (2018) and Spatial-Tempprrl Interaction Networks (Materzynska et al., 2020) combines graph neural networks with RNNs and ppatial-temport attention models to model temporal information. TOQ-Nets use a similar object-centric relational representation, but different models for temporal structures. There
holding 2) and special-
2

---

Under review as a conference paper at ICLR 2021
Input Trajectories
Action Label
Predicted Action
ypestx.0 O bal00u
Short Poss
Supervision
xpos0r.1)
(1 Input Feature Extractor
(i) Relational Reasoning Layers (K byyer, Section S.2]
Temporal Reasining Layers ( bean,, Saction 3.3)
represenatiionraaadll
(E)
Nullary
U2(..0
Unary:
Ox
PS)
y 200))
Binary: R
Increased Complexity
Increased Complexity in Temporal Quanti fication
Objectt Quantification
(1(x.)
Vx3y./100...0) (2.x..
always (16r.) ((y.,( A2(0. until until abways Vo. (+)
( (2(0.)
Figure 2: A TOQ-Net contains three modules: (i) an input feature extractor, (ii) relational reasoning layers, and (iii) temporal reasoning layers. To illustrate the model's 's representational power, we show logical forms of increasing complexity that can realized by stacking multiple layers.
also alternative models for temporal structures from pixels, such as convolutions (Carreira & Zisserman, 2017; Ji et al., 2020), recurrent networks (Dong et al., 2019), attention (Wang et al., 2018), segmentation consensus and its variants (Wang et al., 2016; Zhou al., 2018). In contrast to these methods, TOQ-Nets works on an object-centric temporal-relational feature space, and uses the proposed temporal quantification operations model the sequential structures of events.
TEMPORAL AND OBJECT QUANTIFICATION NETS
TOQ-Nets, shown in Fig. 2, are multi-layer neural networks with a structural inductive bias for learning composable action concepts using object and temporal quantification. The input to TOQ- Net is tensor representation of the properties of all entities each moment in time. For example, in soccer game, the input encodes the position of each player and the ball, as well as the team membership of each player. The output is label for the action performed by a target entity.
The first layer of TOQ-Net (Fig. 2 (i)) extracts temporal features for each entity with an input feature extractor that focuses on entity features within a fixed and local time window. These features may be computed by convolutional neural network, such as STGCN (Yan et al., 2018) parametric feature templates, such as thresholding the velocity of an entity: whether the velocity of an entity x is greater than a learnable parameter 0 at time t. The output of this step is thus collection of nullary, unary, and binary relational features over time for all entities. Throughout the paper we will assume that all output tensors of this layer binary-valued although the this feature extractor to any real-vat real-valued function. This input feature extractor is usually task- and dataset-specific and not the main focus of this paper.
bank of
easily extend
Second, the remporal-relational features go through several relational reasoning layers, which we present in detail in Section 3.2. In short, each layer at this step performs linear transformations, sigmoid activation, and object quantification operations. The linear and sigmoid functions allow the network to realize Boolean logical forms, and object quantification operators can realize quantifiers. With more relational reasoning layers, the network can realize logical forms with more quantifiers, as illustrated in Fig. All operators these layers are performed for all time steps in parallel.
Next, the relational reasoning layers summarize each event at particular time step as nullary feature that is passed to the to the temporal reasoning layer, as detailed in Section 3.3, Each layer at this step performs linear transformations, sigmoid activation, and temporal quantification. They allow the model to realize logical forms in a simple linear temporal logic (Pnueli, 1977). such as always 3x. 91 (x:) A (z))). This is semantically equivalent to the temporally quantified expression Vt. 3x. q, (xx, t) A qe (x. t). Similar to the relational reasoning layers, adding more temporal reasoning layers enables the network to realize logical forms with more deeply nested temporal quantifiers.
3

---

Under review as a conference paper at ICLR 2021
Relational Reasoning Layer 3.2)
6 (t. A (2(0))
3x. 3x,0,0000A
Nullary
Expand
NN
Nullary P.
Expand
Nullary
(1.xtt) 92(x, ()
Reduce
Reduce
Unary: Qu
Expand
NN
Unary: Q:
Expand
NN
Unary: Oz
y.,)
Rinary: R
y.t)
Reduce
NN
Binary: R₁
Reduce
NN
Binury: R2
Relational Reasoning Layer #1
Relational Reasoning Layer #2
(II) Temporal Reasoning Layer (See 3.3)
Dim
Time
Reduce NN
(222)
PP3 X++
X+2
Reduce-
Temporal Reasoning
0-13 R+1
12:0 K+1
Temporal Reasoning Layer #2
g
Layer #1
X42
X=2
always
until always By step I)
Figure 3: Illustration of (i) relational reasoning layers and (ii) temporal reasoning layers. We provide two illustrative running traces. (i) The first relational reasoning layer takes unary predicates ai and 92 input and its output Q1 is able to represent 9ı A 42- The max(Q1, dim = 0) in layer 2 can represent 3r. qi (x, t) A q2 (x, t). (ii) Assume PKK encodes the occurance events e1 and e2 at each time step. The first temporal reasoning layer realize always e2 with temporal pooling from time step 3 to time step T. In the second temporal reasoning layer, the temporal pooling summarizes that el holds true from time step to 2. Thus, the NN should be able realize et until (always e2).
In the last layer, all object and time information is projected into representation in the initial time step, which summarizes what takes place in the entire trajectory (e.g.. the event "the kicker eventually scores" may be recognized at the first time step). We feed this representation into a final softmax unit, and get classification probabilities for the event that the network is trained to detect.
Remark. It is important understand the representational power of this model. The input transfor- mation layer can be seen as defining basic predicates and relations that will be useful for defining more complex concepts, but specific predicates relations are built into it. The learned outputs might be properties that humans can interpret, such as "moving fast" or "far from." The relational reasoning layers can build quantified expressions over these basic properties and relations, and might construct expressions that could be interpretable as "the player is close to the ball." Finally, the temporal reasoning layer can represent temporal operations applied to these complex expressions. such as "the player is close to the ball until the ball moves with high speed." Critically, none of the symbolic properties or predicates are hand defined they are all constructed by the initial layer in order to enable the network to express the concept it is being trained on.
Compared with the expressiveness of first-order linear temporal logic, TOQ-Nets only work in finite universes: all quantifiers only have finite interpretations. Meanwhile, the depth of the logical forms that it can learn is fixed. However, our goal not to fully replicate temporal logic, but to bring ideas of object and temporal quantification into neural networks, and to use them as structural inductive biases to build models that generalize better from small amounts of data to situations with varying numbers of agents, objects, and time courses.
3.1
FEATURE REPRESENTATION
TOQ-Nets use tensors to represent object-centre and relational features over time. The input to the network as well as the feature output of intermediate layers is represented as tuple of three tensors.
4

---

Under review as a conference paper at ICLR 2021
vector of dimension Do represent aspects of the state that apply to depend on all entities at a specific time step t. We use a matrix of shape N x D1 to represent the state of each entity time t. where N is the number of entities and D1 is the hidden dimension size. Similarly, use a tensor of shape N x N x D2 to represent a pairwise relation between each pair of entities at time step f. As concrete example, illustrated in Fig. 2, the number of entities N is the total number of players plus one (the ball). For each entity and each time step, the inputs are their 3D position, type (ball or player) and team membership. The TOQ-Net outputs the action performed by the target player. Since there are only entity features, the input trajectory is encoded with "unary" tensor of shape T X N x D1, where T is the length of the trajectory. That is, there are no nullary or binary inputs in this case.
Specifically,
3.2 RELATIONAL REASONING LAYERS
Our relational reasoning layers follow prior work on Neural Logic Machines (Dong et al., 2019), illustrated in Fig. (i). Consider specific time step At each layer I, the input to a neural logic layer is a 3-tuple (P-1, Qi-1, R(-1), which corresponds to nullary, unary, and binary features respectively. Their shapes are Do. x D1, and N N x D2. The output is another 3-tuple (Pr, Q1, Rt), given by
P = NNp (Concat max() dim = =]]),
Q = NNo (Concat
= 0): max dim 1); expand(PI-uppp dim 1)]), R. NNR (Concal [x dim exx dim = 1)]),
where NN, are single fully-connected layers with sigmoid activations. For unary and binary features, NNę and NNR & are applied along the feature dimension. That is, we apply the same linear transfor- mation the unary features of all entities. A different linear transformation is applied the binary features of all entity pairs. Concat -] is the concatenation operation, applied to the last dimension of the tensors (the feature dimension). max, also called the "reduced" max operation, takes the maximum value along the given axis of a tensor. The expand operation, also called "broadcast," will duplicate the input tensor N times and stack them together along the given axis. The relational reasoning layer is applied identically to the input features at every time step t. That is, we use the same neural network weights in a relational reasoning layer for all time steps parallel.
Remark. The design of relational reasoning layers motivated by relational logic rules a finite and fully grounded universe (i.e., under the closed-world assumption; see Dong et al. (2019) for a detailed discussion.) The max reduction operations implement differentiable version of an existential quantifier over the finite universe of individuals, given that the truth values of the propositions are represented as values in (0.0, 1.0). Because preceding and subsequent relational reasoning layers can negate propositions needed, we omit explicit implementation of finite-doma universal quantification, although it could be added simply by including analogous min reduction operations. Thus, illustrated in Fig. 3 (i), given input features qr (3r, () and ga (1, l), we can realize the logical formula 3r. Qr (Ir, L) A q2 (I,, I) by stacking two object reasoning layers.
Throughout the paper have been using only nullary, unary, and binary features, but the pro- posed framework itself can be easily extended to higher-order relational features. From a graph network (Bruna et al., 2014; Kipf & Welling, 2017; Battaglia al., 2018) point of view, one can treat these features as the node and edge embeddings of fully-connected graph and the relational reasoning layers are instances of graph neural network layers.
3.3 TEMPORAL REASONING LAYERS
Temporal reasoning layers perform similar quantification operations as relational reasoning layers, but along the temporal dimension. They take the summarized event representation Px at each time step as input, produced by the K-th relational reasoning layer, as matrix of shape T D. The output Px+ of a temporal reasoning layer 7 is
K+ = max NN;
Concat
((C) Expand (ppand(Pktt--in K+-- dim = 1); max p(l") "see ct' PR---))
est
where we use the superscript p(() K+; to denote the feature vector at time step t; NN1 is single-layer fully connected neural network with sigmoidal output; maxege"s" p(t") KKt-1 is three-dimensional
tensor indexed by t, t', and d, where d is the channel index.
5

---

Under review as a conference paper at ICLR 2021
Model
Feat.
Reg.
Few-Shot
Full
STGCN
Raw 43.1
28.
63.
68
STGCN-MAX
Raw: 72.9+1.4
25.1 62.320.7
SIGAN
SIGCN femacaic
STGCN-LSTM Raw 74.1
25.726.4 63.320.8
Macro Accuracy.
SIGUN-STM
Space-Time Raw. 69.221.4
26.926.5
59.8.0..8
SIGUNGSIN bonieat
STGCNMAN
Non-Local Raw. 69.5+1.4
44.1
63.9
Span-Tine
STGCN Phy. 73.2
26.0.5.7
62.8.0.6
Spare Time bonsarch
STGCN-MAX
Phy. 73.611.5 28.625.0
63.640.7
Nentoal Fan wnd
Nin-tall
STGCN-LSTM Phy. 72.7+1.4
23.8
61.940.6
Space- Time Phy. 74.8+1.5
31.726.1
65.240.6
Non-Local Phy. 76.5 45.026.3
69.5
TOQ-Net (ours) Phy. 87.7 6.. 79.8
18
20 #Finctune examples
30
Table 1: Results the soccer event dataset. Dif- ferent columns correspond to different action sets (the regular, few-shot, and full action sets). The performance is measured by per-action (macro)
Figure 4: Generalization to soccer environments with a different court size and agent speeds. The accuracy, averaged over nine few-shot splits. The standard errors are computed based on three ran- + values indicate standard errors. TOQ-Net sig- nificantly outperforms all baseline methods on the few-shot action set.
dom seeds.
Remark. The max operations act existential quantifiers over time points. A max pooling of event feature from time to time /' evaluates to iff event i eventually happens within the time period. Since the preceding and subsequent fully connected networks can negate the proposition as needed, this operation can also represent universal quantification: event is always true from time to Note that since all temporal quantification is applied on the summarized state Pk, TOQ-Nets can represent always 3z. 91 (z) A (2(2), but not 3z. always qı (x) A 92(z). This can be solved with interleaved relational and temporal quantification, but we find such interleaved models difficult to train and as yet out of scope of this paper.
EXPERIMENTS
We compare our model with other approaches to object-centric action recognition on two datasets The setups and metries focus on data efficiency and generalization.
4.1 BASELINE
We compare TOQ-Nets against five baselines. The first two are spatial-temporal graph convolutional neural networks (STGAN; Yan et al., 2018) and its variant STGCN-MAN which models entity relationships with graph neural networks, and temporal structure with temporal-domain convolutions. The third is STGCN-LSTM, which uses STGCN layers for entity relations but LSTM (Hochreiter & Schmidhuber, 1997) for temporal structures. The last two baselines are based on space-time graphs: Space- Time Graph (Wang & Gupta, 2018) and Non-Local networks (Wang et al., 2018). We provide details about our implementation and how we choose the model configurations in the appendix.
4.2 TRAJECTORY- --ASED SOCCER EVENT DETECTION
We start our evaluation with an event detection task in soccer games. The task is to recognize the action performed by a specific player at specific time step soccer game, given a short game clip.
Dataset and setup. We collect training and evaluation datasets based on the gfootball simulator', which provides a physics-based based 3D football simulation. also provides AI agents that can be used to generate random plays. The AI outputs abstract action commands: different types of passes (short pass, long pass, high pass, trap), ball shooting (shoot with the head foot), and defense (deflect, trip, slide, interfere. and catch). We run the simulator with Al-controlled players to generate plays, and formulate the task as classifying the action (9-way classification) of a specific player specific time step given temporal context (25 frames). For all input training examples, we perform temporal alignment for the action so that the action happens in the middle the clip. We randomly split all
"https://researce" carteerrrh--o-obbeaa footbal devv

---

Under review as a conference paper at ICLR 2021
Model
Feat.
3v3
4v4
6v6
6v6 (Time Warp)
8v8
11v11
4.0
63.2.+4.9
68.2.+2..8
42.8
55.4
44.4
STGCN
Phy.
(-40.4%)
(-7.4%)
(0.05%
(-22.6%)
(-18.8%)
(-34,9%)
47.4
68.822.0
71.5
56.5
29.1
45.612.5
STGCN-MAX
Phy.
(-33.7%)
(3388%
(0.0%)
(-21,0%)
(-17.3%)
(-36.2%)
1.1
60.42...2
69.
30.6110.6
55.8
44. 10.7
STGCN-LSTM
Phy.
(43..1%)
(-13.5%)
(0.0%)
(-56.1%%
(-20.0%)
(336.8%)
1..0=1.6
53.5
73.3
70.7
33.942.8
15.2
Space-Time
Phy.
(-60,4%)
(-27.0%)
(0.0%)
(-3,5%)
(-53.7%)
(-79.3%)
25.
70.7
78.1
77.7
41.8 58.5
Non-Loccl
Phy. (-41.25%
(-9.5%)
(0.0%)
(-0.5%)
(-25.156)
(-46.5%)
23.5
88.3
88.4
10.4
81.3
77.1 #1.7
TOQ-Net
Phy.
(-12.4%)
(0.0%)
(0.0%)
(-1.7%)
(-8.0%)
(-12.8%)
Table 2: Results on generalization to scenarios with more agents and temporally warped trajectories on the soccer event dataset. The standard error of all values are smaller than 2.5%, computed based on three random seeds.
actions into two categories: seven "regular" actions, for which 3,000 video clips are available, and two "few-shot" actions, for which only 50 clips are available.
Input features. We implemented two types of input features. trajectories of all agents. The second, Physics-basee extracts the velocity and acceleration for each entity, as well as the distance between all pairs of agents. This pre-processing necessary for TOQ-Nets, because they only model sequential structures of events, but not temporally local features such as velocity and acceleration. See the appendix for details.
The first, Raw, uses the raw coordinate
Results. Table 1 shows the result. Our model significantly outperforms all the baselines in all three action settings. This suggests that our model able to discover a set useful features at both input and intermediate levels, which can be further used to compose classifiers for actions with only few examples. In general, the models with physics-basee based features outperform those with raw coordinate inputs. Thus, for the rest of the experiments, will use physics-based input feature for all models.
Generalization to different input scales. Intuitively, after learning how to detect events in a soccer game, the learner should also generalize concepts analogs with the same basic structure but enacted on proportionately larger or smaller spatial or temporal scales-as as in the soccer-inspired game futsal. To evaluate generalization to different input scales, we first train all models on the original dataset with 9-way classification supervision. After that, we finetune each model on new dataset where all input values are doubled (i.c., the court now double the size. and the players now move at four times the speed), but with only a small number of examples. Results are summarized in Fig. 4. While all baselines can benefit from the pre-training on the original dataset, our TOQ-Net outperforms all baselines by a significant margin when the finetuning dataset is small.
Generalization to more players. Due to their object-centric design, TOQ-Nets can generalize to soccer games with varying number of agents. After training on 6v6 soccer games (i.e., 6 players on each team), we evaluate the performance of different models on games with different numbers of players: 3v3, 4v4, 8v8, and 11v11. Table 2 summarizes the result and the full results are provided in the appendix. Comparing the columns highlighted in yellow, we notice a significant performance drop for all baselines while TOQ-Net performs the best. By visualizing data and predictions, found that misclassification of instances of shot as short pass contribute most to the performance degradation of our model when we have more players. Specifically, the recall of shot drops from 97% to 60%. In soccer plays with many agents, a shot is usually unsuccessful and a player from another team steals the ball in the end. In such scenarios, TOQ-Net tends to misclassify such trajectories as a short pass. Ideally, this issue should be addressed by understanding actions based on agents' goals instead of the actual outcome (Intille Bobick, 2001). We leave this extension as a future direction.
Generalization to temporally warped trajectories. Another crucial property of TOQ-Nets is to recognize actions based on their sequential order in the input trajectory, instead of binding features to specific time steps in the trajectory. To show this, we test the performance of different models time warped trajectories. The results are shown in Table 2. Specifically, our test set consists of 25- 25-frame trajectories, and the action may occur at anytime between the 6th and the 19th frame. By comparing rows with and without time warping. we notice a 60% performance drop for STGCN, STGCN-MAX, and STGCN-LSTT STM.. In contrast, TOQ-Nets still have reasonable performance. Note that Space-
7

---

Under review as a conference paper at ICLR 2021
Model
Reg.
1-Shot
Full
--Shot Set
I-Shot Set #2
Ice
100
STGCN 99. 99.89=0.00 =0.05
11.02
20.29
45-
STGGN-LSTI =9.92.0.03
95.48
18.86
TOQ-Net
99.96-0002 28.04
29.48
85
Table Few-shot learning on the RLBench dataset, measured by per-action (macro) accuracy and averaged of four I-shot splits and four random seeds per split. The + values indicate standard errors. On the right we shows the sampled per- formance of different models each individual 1-shot split.
STOCE STOCN-LSSM Vet
STOCN STGGN-LSIM ToQ::Ne
--Shot Set #3
1-Shot Set 64
193
50
STOCN STOCN-LSIM NeQ
STGCN STGCN-LSIM TeQ
and Non-Local model experienced almost performance drop against time warping because they are completely agnostic to temporal ordering.
4.3 MANIPULATION CONCEPT LEARNING FROM 6D POSES
Structural action representations can also be usefully applied to other domains. Here we show the result robotic environment, where the goal classify the action performed by robotic arm.
Dataset and setup. We generated dataset based the RLBench simulator (James al., 2020), which contains a set of robotic ihject-manipulation actions in a tabletop environment. We chose 24 actions from the RLBench, including opening box, closing fridge, push block, etc. Details are provided in the appendix. For each object, at each time step. the input contains its bounding box (as an approximation of the shape of the object) and the 6D pose. Deformable objects are split into several pieces so that the learner can infer their state (open closed). For example, the lid of the box a separate object. We also input the 6D pose and the state the robot gripper. During training, we used on average 65 videos for regular actions.
Input features. All methods take the raw trajectories as input. In TOQ-Nets, we use extra single linear layer with sigmoid activation as the input feature extractor. evaluate temporal modeling with recurrent neural networks, STGCN-LSTM uses kernel size in its (STGCN) temporal convolution layers. Ablation studies can found the appendix.
Results. We start with evaluating different mod- els on the standard 24-way classification task. We summarize results in Fig. 5. also illustrates each model's performance under different time stretching factors of the input trajectory, from l/8x to 8x. Sim- ilar the soccer dataset, the TOQ-Net outperforms both convolutional and recurrent models across all time stretching factors.
Macro Accuracy
90
Generalization to new action concepts. We also evaluate how well different models can generalize the learned knowledge about opening and closing other objects to novel objects. Specifically, we hold out 50% of all Open X actions and 50% of the Close actions to form the 1-shot action learning set. For example, the learner may only see single instance of the opening of box during training. Actions with more examples are "regular." All models are tested on 3-way classification task: Open-X, Close-X, and Other. Results are summarized in Table 3. There are several held-out sets that are noticeably more "difficult" than others, such as the Set #1 in Table 3. Our visualization shows that actions close jar and close drawer are harder generalization targets, compared to other actions. This is possibly because the motion close jar involves some rotation, and the motion of close drawer looks more like pushing than other close-X actions.
TOQ-Net STGCN
STGCN-LSTM
Time Stretching Factor
Figure 5: Comparing different models with different time stretching factors the RL- Bench dataset.
4.4 EXTENSION TO WORLD DATASETS
The proposed TOQ-Net can also extended to other real-world datasets human activities. We further conduct experiments two datasets: Toyota Smarthome and volleyball activity dataset.
8

---

Under review as a conference paper at ICLR 2021
Toyota Smarthome (Das et al. (2019)). Toyota Smarthome is dataset that contains videos of daily living activities such as "walk", "take pills" and "use laptop". It also comes with 3D-skeleton detections. There are around 16.1k videos in the dataset, including 19 activity classes. The videos' length varies between a few seconds to minutes and we subsample 30 frames for each video. We split them into training (9.9k), validation (2.5k), and testing (3.6k). Similar the soccer event dataset, we treat the joints of the human as entities and use the physical feature extracted from the 3D-skeletons as the input. The input consists of the position of joints, the velocity of joints, the distance between pairs of joints, and joint angles. We evaluated our model and STGCN on the 19-way classification task. We also test model performance on uarew sequences (2x quick motion).
19-way classification
Time warp (QuickMo 2x)
TOQ-Net STGCN
42.0
41.2
43.0
32.3
Our model achieves comparable accuracy STGCN on the standard classification task. Importantly, on the generalization test to time-warped sequences, our model encounters only a 0.8 performance drop where STGCN drops 10.7, which indicates that the learned temporal structures by our model improve model generalization to varying time courses.
Volleyball Activity (Ibrahim et al. (2016). The volleyball dataset contains 4830 video clips collected from 55 youtube volleyball videos. They are labeled with 8 group activities such as "left spike" and "right pass". Each video contains 20 frames with the labeled group activity performed at the 10-th frame. The dataset also includes manual annotations for players at each frame, which consists of the bounding box, the indicator of whether the player is involved in the group activity, and the individual action such as "setting". "digging" and "spiking". We use the manual annotations, processed by an MLP as the input feature for each player. We train models to classify the video into one of the eight group activities. In our experiment, adapt the train/val/test split in [2), i.e., 24, 15, and 16 of 55 videos are used for training, validation, and testing, respectively.
8-way classification
Time warp (Shift)
Time warp (QuickMo 2x)
TOQ-Net STGCN
73.3
70.3
70.7
73.6
39.5
48.6
On the standard classification task, TOQ-Net achieves a comparable performance with STGCN. When we perform time warping on the input sequences, STGCN's performance drops by >25.00 25.0% while our model drops only 3%. This further suggests the generalization ability of TOQ-Net w.r.t. varying time courses.
CONCLUSION AND DISCUSSION
We have presented TOQ-Nets, neural module for describing complex activities by quantifying over both entities and time. TOQ-Nets use tensors to represent the time-var varying properties and relations of different entities, and tensor pooling operations over different dimensions to realize temporal and object quantifiers. TOQ-Nets can learn to describe complex actions, and generalize well to scenarios with a varying number of entities and time courses.
The design of TOQ-Nets suggests multiple research directions. For example, the generalization of the acquired action concepts to novel objects, such as from opening fridges to opening bottls, needs further exploration. Mcanwhile, TOQ-Nets are currently based on physical properties. e.g. 6D poses. Incorporating representations mental variables such as goals, intentions, and beliefs can aid in action and event recognition (Baker et al., 2017; Zacks et al., 2001; Vallacher & Wegner, 1987). Finally, future research can examine connecting TOQ-Nets to perceptual processing, in order individuate objects and compute their properties.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chris L Baker, Julian Jara- Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):1-10, 2017.
Dare A Baldwin, Jodie A Baird, Megan M Saylor, and M Angela Clark. Infants parse dynamic action. Child development, 72(3):708-717, 2001.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. / 2018.
Vinicius Zambaldi, Mateusz
A. Bialkowski, Lucey, Carr, Y. Yue, S. Sridharan, Matthews. Large-scale analysis of soccer matches using spatiotemporal tracking data. In 2014 IEEE International Conference on Data Mining, pp. 725-730, 2014. doi: 10.1 10.100CCC..20044...
William Brendel, Alan Fern, and Sinisa Todorovic. Probabilistic event logic for interval-based event recognition. In CVPR, 2011.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. In ICLR, 2014. 5
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? model and the kinetics dataset. In CVPR, 2017.
Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities daily living. In The IEEE International Conference on Computer Vision (ICCV), October 2019. 9
Zhiwei Deng, Arash Vahdat, Hexiang Hu, and Greg Mori. Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Honghua Dong, Jiayuan Mao, Tian Lin, Chong Wang, Lihong Li, and Denny Zhou. Neural logic machines. In ICLR, 2019. 3,5
Abhinav Gupta, Praveen Srinivasan, Jianbo Shi, and Larry S Davis. Understanding videos, constructing plots learning visually grounded storyline model from annotated videos. In CVPR, 2009. 2
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 93):1 1997.
Mostafa S. Ibrahim, Srikanth Muralidharan, Zhiwei Deng, Arash Vahdat, and Greg Mori. hierarchical deep temporal model group activity recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 9
Stephen S Intille and Aaron Bobick. A framework for recognizing multi-agent action from visual evidence. In AAAI, 1999.
Stephen S Intille and Aaron F Bobick. Recognizing planned, multiperson action. CVIU, 81()):144444 2001. 2,7
Stephen James, Zicong Ma. David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark & learning environment. RA-I, 2020. 8, 12
Jingwei Ji, Ranjay Krishma, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions composition of spatio-temporal scene graphs. CVPR, 2020. 2,3
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 16
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017. 5
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-elss Compositional action recognition with apatial-temport interaction networks. CVPR, 2020.
Steven Pinker. The stuff of thought: Language as a window into human nature, Penguin, 2007.
Amir Pnueli. The temporal logic of programs. In FOCS, 1977. 2,3
10

---

Under review as a conference paper at ICLR 2021
Mengshi Qi. Jie Qin, Annan Li, Yunhong Wang, Jiebo Luo, and Luc Van Gool. stagnet: An attentive semantic for group activity recognition. In Proceedings of the European Conference Computer Vision (ECCV), September 2018. 2
Jürgen Stränger and Bernhard Hommel. perception of action and movement. In Handbook of perception and action, volume 1, pp. 397-451. Elsevier, 1996. 1, 2
Kevin Tang, Fei-Fei, and Daphne Koller. Learning latent temporal structure for complex event detection, In CVPR, 2012.
Kevin Tang, Bangpeng Yao, Fei-Fei, and Daphne Koller. Combining the right features for complex event recognition. In ICCV, 2013. 2
Tran and Larry Davis, Event modeling and recognition using markov logic networks. In ECCV, 2008.
Robin R Vallacher and Daniel M Wegner. What do people think they're doing? action identification and human behavior. Psychol. Rev, 94(1):3, 1987.
Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. ECCV, 2016.
Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs. In ECCV, 2018. 2, 6, 15
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018. 3, 6, 15
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In AAAI, 2018. 2,3, 13
Jeffrey M Zacks, Barbara Tversky, and Gowri Tyer. Perceiving, remembering, and communicating structure in events. Exp. Psychol. Gen., 130(1):29 2001. 2.9
Jeffrey M 7acks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: mind-brain perspective. Psychol. Bull., 133(2):273, 2007. 1,2
Bolci Zhou, Alex Andonian, Aude Oliva, and Antonio Torrala. Temporal relational reasoning in vidcos. In ECCV, 2018. 3
Tao Zhuo, Zhiyong Cheng, Peng Zhang, Yongkang Wong, and Mohan Kankanhalli, Explainable video action rcasoning prior knowledge and state transitions. In MM, 2019. 2

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 SOCCER EVENT DATASET
We used the gfootball simulator' to generate simulated soccer match replays by letting the built-in Al play against each other. The simulator provides the 3D coordinates of the ball and the players as well as the action each player is performing at each time step. There are total 13 actions deñned in the simulator, including movement, call_control, trap, short_ deffect, catch, interfere, trip and sliding. We exclude header and catch actions, as they never appear in AI games. We also exclude mall_control and movement, since they just mean the agent is moving (with or without the ball). Thus, in total, we have nine action categories.
long_pase
tiihhppsss header, shot,
For each action, we have generated 5,000 videos, expect for sliding, for which we generated 4,000 videos because is rare in the Al games. Among the generated examples, 60% (2,400 or 3,000) are used for training, 15% are used for validation, and 25% are used for testing.
Our data is collected from 6v6 soccer games. Each trajectory is an 8-fps replay clip that contains 17 frames (about two seconds). There is a single "target" player in each trajectory. The action label of the trajectory is the action performed by this target player at frame #9.
Input. Each trajectory is represented with 7 time-varying unary predicates, including the 3D coordinate of each player and the ball and four extra predicates defining the type of each entity GameTTeam('"
x: [sBall(:). Is IsTargetPPayeer Opponent'Teamu indicates whether x is of the same team as the target player. We also add a temporal indicator function which is a Gaussian function centered at frame of interest with variance 0² = 25.
(e)mm))) where Same (z) and
Generalization to different input scales. In this task, we multiply the coordinates of the ball and the agents by 2, and subsample the trajectories by a factor of 2. Thus, time flows twice fast, and the player moves four times the speed compared with the original dataset,
Generalization to more players and temporally warped trajectories. In these tasks, we gener- ate, on average, 1,500 examples for each action in 3v3, 4v4, 8v8, and 11v1 I games. We use the same input and label format for non-time-warper trajectories trajectories, the trajectory length is 25, and each trajectory is labeled by the action performed by the target player at any time step between the 6-th and 19-th frame. We ensure that the target player performs only one action during the entire input trajectory. Thus, the label is unambiguous.
the original dataset. For wareed
A.2 RLBENCH DATSAET
We generated the RLBench dataset with the RLBench simulator (James al., 2020). The simulator uses motion- planning algorithms that can generate robot arm trajectories that manipulate various objects in tabletop environment. We use 24 actions from the dataset, including CloseBox, Close- Drawer, CloseFridge, CloseGrill, CloseJar, CloseLaptopLid, CloseMicrowave,, GeilceFromFridge OpenBox, OpenFridge, OpenMicrowave, OpenWineBottle, PickUpCup, PressSwitch, PushButtons, PutGroceriesinnuppoore PutltemlnDraverr PurRubbishlnBin. PutTrayInOven, Scoop WithSpat- ula, SeiTheTable,
and TakeToi- letRollOf[Siand. We randomly initialize the position and orientation of different objects in the scene and use the built-in motion planner to generate robot arm trajectories. Depending on the task and the initial poses of different objects, trajectories may have different lengths, varying from 30 to 1,000. Most of the trajectories have ~0 frames.
For each action category, we have generated 100 trajectories. Among the generated examples, 60% of the examples are used for training, 15% are used for validation, and the other 25% are used for testing.
Input. Each trajectory contains the poses of different objects, at each time step. Each object except for the robot arm is represented 13 unary predicates, including the 3D position, the 3D orientation
hhttps://researdh footbal dev/
12

---

Under review as a conference paper at ICLR 2021
represented as quaternions (4D vector), and the 3D bounding box (axis-aligned, 6D vector). The robot arm is represented as 8 unary predicates, including the 3D position, the 3D orientation, and a binary predicate that indicates whether the gripper is open closed.
A.3 IMPLEMENTATION DETAILS
In this section, we present the implementation detail of our model, the TOQ-Nets, and five base- lines (STGCN, STGCN-LSTM, STGCN-MAX, Space-Time and Non-local), including the model architecture, input features, and the training methods.
A.3.1 TOQ-NETS
In the soccer event recognition task, we use three object quantification layers and three temporal quantification layers. Each object quantification layer has hidden dimension of 16 (i.e.. all nullary, unary, and binary tensors have the same hidden dimension 16). Each temporal quantification layer has hidden dimension of 48.
In the manipulation concept learning task, we use three object quantification layers and three temporal quantification layers. Each object quantification layer has hidden dimension of 48. Each temporal quantification layer has a hidden dimension of 144.
Input feature extractor. We use the following physics-based input feature extractor for soccer event recognition. Given the trajectories of all entities, including all players and the ball, we first compute the following physical properties: ground speed (i.e., the velocity the ryy-llne),, vertical speed (i.e., the velocity along the z direction), and acceleration. We also compute the distance between each pair of entities.
After extracting the physical properties, we use the following feature templates to generate the input features. For each physical property, X, we first normalize X across all training trajectories, and then we create = 5 features:
X ---3 0g B
where is the sigmoid function, 3 is a scalar hyperparameter, 8, 3,4, are trainable parameters. These feature templates can be interpreted as differentiable implementations of 1[X > 0:), where [--] is the indicator function. During training, we make 3 decay exponentially from 1 to 0.001.
In the manipulation concept learning task, we use single fully-connected layer with sigmoid activation as the feature extractor. The hidden dimension of the layer is 64.
A.3.2 STGCN
We use the same architecture Yan al. (Yan et al., 2018). Table 4 summarizes the hidden dimensions and the kernel sizes.
The STGCN model's output is tensor of size (T/4) N x 256, where T is the length of the input trajectory, and N is the number of entities. Following Yan et al. (Yan et al., 2018), we perform an average pooling over the temporal and the entity dimension and get feature vector of size 256. We apply a softmax layer on top of the latent feature to classify the action.
Input feature extractor. In the soccer event recognition task, we have tested two different input features, namely the Raw feature and the Physics- feature. For the Raw feature, the input to the STGCN model contains seven oone-vrrying unary predicates (i.e., the 3D coordinates, and four predicates for entity types). For the Physics-based feature, in addition to the seven raw predicates, also input the ground speed, the vertical speed, and the acceleration for each entity, Note that we do not input the distance between each pair of entities does not support binary predicate inputs.
do for the TOQ-Nets, the STGCN model
In the manipulation concept learning task, STGCN uses the same input format as the TOQ-Nets. Specifically, we represent the state of each entity with 13 unary predicates, including 3D position, 3D orientation, and bounding boxes.
13

---

Under review as a conference paper at ICLR 2021
Model
Input Dim.
Output Dim.
Kernel Size
Stride
Dropout
Residual
64
1
0
False
64
64
0.5
True
64
64
I
0.5
True
64
64
1
0.5
True
64
128
7
2
0.5
True
STGCN
128
128
0.5
True
128
128
7
1
0.5
True
128
256
2
0.5
True
256
256
7
0.5
True
256
256
0
True
16
1
0
False
16
16
7
I
0.5
True
16
32
7
2
0.5
True
STGCNs s
32
32
0.5
True
32
64
7
2
0.5
True
64
128
1
0
True
0
False
STGCNr
8
12
7
2
0.5
True
12
64
1
0
True
Table 4: The STGCN architecture used in the paper.
Model
Input Dim.
Output Dim.
Stride
Residual
64
False
64
64
I
True
True
64
64
1
True
64
128
2
True
Space-Time
128
128
Truc
128
128
True
128
256
2
True
256
256
True
256
256
True
False
16
16
1
True
Space-Times
True
32
64
2
True
64
128
True
32
False
Space-Timey
32
64
2
True
64
64
2
True
Table 5: The Space-Time Region Graph architecture used in the paper.
A.3.3 STGCN-MAX
The STGCN-MAX model is variation of STGCN. During graph propogation, for each node, instead of taking average of all propagated information from all neighbors, we take their maximum value over all feature dimensions, and update the hidden state of each node with this. This baseline replicates a very similar propagation rule as our relational reasoning layer. However, it still uses temporal convolutions to model temporal structures.
14

---

Under review as a conference paper at ICLR 2021
Model
Input Dim.
Output Dim.
Stride
Residual
False
64
64
True
64
64
True
64
64
True
64
128
2
True
Non-Local
128
128
True
128
128
True
128
256
True
256
256
True
256
256
True
32
False
32
32
True
Non-Locals
32
64
2
True
128
True
32
False
Non-Localf
32
32
2
True
True
Table 6: The Non-local Neural Networks architecture used in the paper.
A.3.4 STGCN-LSTM
The STGCN-LSTM model first encodes the input trajectory with an STGCN module. The output of the STGCN module is tensor of shape T x N 256, where T is the length of the input trajectory, and N the number of entities. We perform an average pooling over the entity dimension and get tensor of shape I 256. Next, we encode this feature with 2-layer bidirectional LSTM module. The hidden dimension for the LSTM module is 256. We use a single softmax layer on top of the LSTM module classify the action.
STGCN-LSTM uses the same architecture as STGCN for both tasks, except for the kernel size and the stride, because goal is to evaluate the performance of recurrent neural networks (the LSTM model) on modeling temporal structures. In the soccer event recognition task, we use kernel size 3 and stride that the STGCN module will have the representational power encode physical properties, such as velocity and acceleration. In the manipulation concept learning task, we use a temporal kernel size of and stride 1. Meanwhile, Table 7 shows the ablation study of different kernel sizes, evaluated by the few-shot learning performance on the RLBench dataset. Different kernel sizes and have similar performance.
A.3.5 SPACE-TIME REGION GRAPH AND NON-LOCAL NEURAL NETWORKS
We use the same architecture for each Space- Time layer Wang al. proposed in Space-Time Graph (Wang & Gupta, 2018). Table summarises the detailed parameters of all layers.
A.3.6 NON-LOCAL NEURAL NETWORKS
use same architecture for each Non-local layer as Wang et al. proposed Non-Local networks (Wang et al., 2018). The original non-local network works pixels. Here instead, we apply it to the space-time graph constructed the same way as Wang Gupta (2018). The major difference between the Space-Tima Graph and the Non-local Network is that Space- Time Graph uses graph convolutions while Non-local uses transfomer like attentions over all nodes in the graph. Table 6 summarises the detailed parameters of all layers.
15

---

Under review as a conference paper at ICLR 2021
Model
Reg.
1-Shot
Full
STGCN
99.89 10.05
94.9211.03
98.79 10.23
STGCN-LSTM (kernel size
99.89 10.05
96.16.11
99.04 10.31
STGCN-LSTM (kernel size 1)
99.9240.03
95.4841.67
98.8640.43
TOQ-Net
99.96
98.04
99.48 .0.24
Table 7: Few-shot learning the RLBench dataset, measured by per-action (macro) accuracy and averaged of four 1-shot splits and four random seeds per split. The + values indicate standard errors.
A.3.7 TRAINING
Soccer event recognition. We use the identical training strategy for our model and both baselines. Each training batch contains 128 examples, which are sampled as following: we first uniformly sample action category, and then uniformly sample trajectory labeled this action category.
In all the soccer event tasks, we optimized our model using Adam (Kingma & Ba, 2015), with the learning rate initialized to 7) 0.002. The learning rate is fixed for the first 50 epochs. After that, decrease the learning rate 17 by a factor of 0.9 the best validation loss has not been updated during the last epochs.
All models are trained for 200 epochs. In the task of generalization to different input scales, we first train different models the original dataset for 200 epochs (~40,,000 iterations) and finetune them on the new dataset with few samples for 10,000 iterations.
Manipulation concept learning. We optimized different models using Adam with the initial learning rate = 0.003. We applied the same learning rate schedule as in the soccer event recognition task. The batch size for the manipulation concept learning task 32 for models with STGCN backbone and is 4 for TOQ-Nets. All models are trained for 150 epochs.
A.4 STGCN-2D
The original STGCN does not support input features of higher dimensions, so extend STGCN to STGCN-2D to add binary inputs such as the distance. This extension slightly improves the model performance on the standard 9-way classification test. It also helps in the few-shot setting. Specifically, on the few-shot learning setting:
Reg.
New
All
TOQ-Net
87.7
52.2
79.8
STGCN-2D
84.1
46.9
75.8
Mcanwhile, adding extra binary inputs does not improve the generalization of the network to games with different number of agents time- wanped sequences. This following table extends the Table 2 in the main text. Specifically, we train STGCN-2D on 6v6 soccer games and test on scenarios with a different number of agents (3v3 and 1Iv1l) and also temporally warped trajectories. Our model shows a significant advantage.
9-way
3v3
1Iv11
6v6(Time warp)
TOQ-Net
88.4
77.4
77.1
86.9
STGCN-2D
84.5
17.5
16.6
39.7
A.5 NETWORK ARCHITECTURE
In general, TOQ-Net has smaller number of weights than baselines. We add additional comparisons to models of different #params. Specifically, models is a smaller version of the model and modely is the tiny version of the model whose #params about equal To obtain smaller models we typically reduce the number of layers and the hidden state dimensions. On the fho- task of the soccer event dataset, we test all baselines with all combination of features
smaller than #params of TOQ-Net.
16

---

Under review as a conference paper at ICLR 2021
Model
Feat.
#params
Reg.
Few-Shot
Full
STGCN
Raw.
3.43M
71.3
26.8_4.6
61.410.4
STGCNs
Raw.
262K
13.1
28.5-4.7
63.2+0..
STGCN,
Raw.
34K
68.7
27.3-6.7
59.5
STGCN
Phy.
3.42M
72.0
5..
61.0
STGCN;
Phy.
263K
73.0
22.5-4.9
61.7.0.5
STGCNr
Phy.
34K
73.2+1.6
26.0_5.7
62.8
STGCN-LSTM
Raw.
2.08M
74.1
25.7
63.3
STGCN-LSTM;
Raw.
233K
73.4
24.2_6.3
62.5
STGCN-LSTMr
Raw.
33K
71.9112
27.4-6.6
62.040.8
STGCN-LSTM
Phy.
2.08M
72.5
20.2=5.0
60.9
STGCN-LSTM;
Phy.
233K
72.1
20.5_5.0
60.610.7
STGCN-LSTMr
Phy.
33K
72.7
23.8
61.9
STGCN-MAX
Raw.
3.42M
68.2
17.9_3.7
57.013.3
STGCN-MAX,
Raw.
262K
72.9
=5.
62.3
STGCN-MAXy
Raw.
34K
69.9
29.1 29.1
60.8
STGCN-MAX STGCN-MAX5
Phy.
3.42M
73.5
20.2
61.7
Phy.
263K
73.611.5
28.6
63.6
STGCN-MAXT
Phy.
34K
74.5+1.2
25.4
63.640.7
Space-Time Time
Raw.
263K
67.9+1.9
28.3
59.1
Space-Times
Raw.
24K
69.2+1.6
26.9
59.8
Space-Time
Raw.
13K
68.0+1.6
30.1
59.6+0.3
Space-Time
Phy.
263K
71.3+1.4
26.9
61.441.2
Space-Times
Phy.
24K
74.7+1.4
30.7 30.7-6.8
64,9
Space-Timer
Phy.
14K
74.8
31.7-6.1
65.210.6
Non-Local
Raw.
1.23M
53.2+4.9
44.3_7.6
51.343.8
Non-Locals
Raw.
107K
69.5
44.1
63.941.2
Non-Localr
Raw.
32K
63.811.8
44.0=6.2
59.4
Non-Local
Phy.
1.23M
74.5
44.3_7.0
67.843.7
Non-Locals
Phy.
108K
76.5+2.4
45.0=m.
69.5
Non-Localy
Phy.
32K
75.641.2
44.8-6.4
68,8
TOQ-Net
Phy. 35K
87.7
52.2_6.3
79.8
Table 8: Results on the soccer event dataset for baselines with different capacities, where model and modely denote the small and the tiny variant for each model. The performance is measured by per-action (macro) accuracy, averaged over nine few-shot splits. The + values indicate standard errors. For each baseline we showed the best performance over three levels of capacities. In fact, performances of most of the models are not affected much by the capacity. We highlight the best best-performing variants of all baselines, and use them in all comparisons in the main text.
17

---

Under review as a conference paper at ICLR 2021
Model
Feat.
Time Warp
3v3
4v4
6v6
8v8
11v11
40.7+1.0
63.2
68.2
45.4
44.4
STGCN
Phy.
N
(-40.4%)
(-7.4%)
(0.0%)
(-18.8%)
(-34.%)
32.6+2.8
50.2+4.4
52.8=7.0
43.21449
34.043.7
STGCN
Phy.
(.52.3%)
(-26.5%)
(-22.6%)
(336.7%)
(.50.2%)
47.4
68.812.0
71.521.9
59.1
45.612.5
STGCN-MAX
Phy.
N
(.33.7%)
(3.8%)
(0.0%)
(-17.3%)
(-36.2%)
37,5
52.3
4..5
46.643.7
36,941.9
STGCN-MAX
Phy.
(-47.5%)
(-26.9%)
(-21,0%)
(-34.8%)
(-48.4%)
39.7
60.410.2
69.8
55.812.0
44.1
STGCN-LSTM
Phy.
19)
(-13.5%)
(0.0%)
(-20.0%)
(.36.8%)
21.8
27.8
3..6
25.8
22.610..8
STGCN-LSTM
Phy.
(-68.88 896
(-60.25)
(-56.1%)
(66..1%)
(-67.6%)
29.0+1.6
53.5
73.3
33.942.8
15.2
Space-Time Time
Phy.
(-60.4%))
(-27.0%)
(0.0%)
(-53.7%%
(-79.3%)
29.7
$1.6+2.5
70.7
33.8
14,9+1.6
Space-Time
Phy.
Y
(-59.5%)
(-29.6%)
(-3.5%)
(-53.9%)
(-79.7%%
45.
70.7
78. #o.s
18.5
41.8448.6
Non-Local
Phy.
(-41.2%)
(-9.5%)
(0.0%)
(-25.1%)
(-46.5%%
46.7144.1
69.9
77.7=6.0
58.7
41.3
Non-Local
Phy.
Y
(-40.2%)
(-10.5%)
(40.5%)
(.24.9%)
(-47.1%)
77.4
88.3
88.4
81.3
77.1
TOQ-Net
Phy.
(-12.4%)
(40.0%)
(0.0%)
(-8.0%)
(-12.8%)
76.0+2.6
87.7+1.4
86.9
80.3
74.9
TOQ-Net
Phy.
Y
(-14.0%)
(-0.8%)
(-1.7%)
(-9.1%)
(-15.2%)
Table 9: Full results generalization scenarios with all combinations of #agents and temporally warping on the soccer event dataset. The standard error of all values are smaller than 2.5%, computed based on three random seeds.
and capacities. Table 8 summarizes the results. In general, the "small" variations are the best for all models (except for STGCN-LSTM), Across all experiments presented in the main paper, we use the best architecture Table 8.
A.6 FULL RESULTS FOR THE TIME- WARPING GENERALIZATION
In the paper we have evaluated the generalization performance of different models to time- warped sequences but with the same number of players. Table shows the results on time-warped sequences with different number of agents.
A.7 #LAYERS OF TOQ-NET
We study the TOQ-Net performance over varying number of relation reasoning layers and temporal reasoning layers the 9-way classification task of the soccer event dataset (See figure Fig. 6), and decide to use the combination relational layers and temporal layers. Both relational layers and temporal layers play important roles in the performance.
A.8 INTERPRETABILITY
Though interpretability is not the main focus of our paper, in Fig. 7, we show how TOQ-Net composes low-level concepts into high-level concepts: from inputs, intermediate features at different temporal layers, and the final output labels. See the figure caption for detailed analysis.
18

---

Under review as a conference paper at ICLR 2021
90
85
80
75
70
65
Temporal Depth
Relational Depth
60
Figure Comparing # of reasoning layers. When # relational reasoning layers vary (blue), temporal reasoning layers are fixed at 4. When of temporal reasoning layers vary (orange), relation reasoning layers are fixed at 3. Accuracy tested on 6v6 9-way classification. Adding reasoning layers improves results, and 3 relational layers + 4 temporal layers is good balance between computation and performance.
siding
Channel xeporal layer
deflect
Used in trig__pow.. pao..
shot
Dependent input feauures: opponent(S) Speed_xy(s) =2.88m/s player imteres(s))
nteeeeee
Channel leporall layer
high_pass pass
Used in short poox.
Dependent input fentures: player intersst) dist(x.3) 68m
back
shart
trap
siding
dellect
Channal #19 at umpural layer #2,
Uked in Aigh joor. Dependent input spead xxx) 0.75mh
Channel #2 temporal layer Used in vñort, past. Dependent inpur aecis)> 7.8mm 7770mm22
tri
interfere
2ix>> 1.00m's
speed x0x)> 0,51m
pos, pos__tx) 213 0.45m
knn pats
speed xy(x)> 2.68ms
short pass
Figure 7: Relevant features in temporal layers. Feature dependencies are computed by gradient. These dependencies and thresholds are learned end-to-end from data. Insets detail features for events in the first and the second stage of the high pass and short pass.
19


Reviews:
[{"review_id": "r92j6btvNAN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The authors address a well formulated and an important problem for lots of practical scenarios.\n- The paper is well written.\n- The experiments demonstrate that the proposed method outperforms several baselines.\n\nWeaknesses:\n- I personally found the proposed approach to be very cumbersome and complicated. It consists of many different components that are not conceptually intuitive. Contrary to some of the recent approaches for modeling visual relations (i.e. Non-Local Networks or Space-Time Video Graphs), the proposed model is much more difficult to understand. In my opinion, this is a big disadvantage because the models that are most useful to the community are usually conceptually (and technically) simple, yet effective.\n- The authors assume that the input features to the TOQ net are hand-engineered, and thus, are not learnable. This is very different to most modern approaches, which typically try to learn all the features from raw pixels end-to-end. Very few recent methods (to the best of my knowledge) rely on hand-engineered features for video modeling/action recognition. This begs the question how applicable the proposed approach is to modern computer vision community.\n- The biggest weakness of the paper is its experimental evaluation. The authors evaluate their approach on 2 small-scale artificial video datasets. Thus, it is not clear whether the proposed approach would generalize to real datasets such as Kinetics, Something-Something, EPIC-Kitchens, etc. Most current action recognition methods evaluate their model on these large-scale real-world datasets. Thus, I think it is imperative that the authors would conduct thorough experiments not only on their small artificial datasets but also on the datasets that are most often used by the action recognition community. In particular, datasets like Something-Something, EPIC-Kitchens, or Charades require spatiotemporal relation modeling as demonstrated by prior work (i.e. Space-Time Video Graphs). \n- The comparison with the pixel-level baselines (i.e. Non-Local Networks, or Space-Time Video Graphs) might not be exactly fair. The authors adapt these baselines to the dataset/task specific scenarios. Compared to their own model, the authors don't have as many incentives to tune these baseline models for their specific tasks. Thus, I believe that the experiments on the real-world large scale datasets such as Kinetics, Something-Something and Charades are essential for validating that the proposed approach is better than these prior methods.\n- Missing relevant work: Wang et al., \"Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks.\" (CVPR 2020).\n\nRebuttal Requests:\n- The authors should include thorough experiments on the real-world large-scale datasets such as Kinetics, Something-Something, and Charades. The currently used datasets are not sufficient to verify the generalizability and usefulness of the proposed approach.\n\n==Post Rebuttal Response:\n\nI read the rebuttal, and unfortunately it didn't address most of my pressing concerns. I appreciate the authors' efforts to add new experiments on other datasets. However, in my opinion these new datasets are not very relevant for the action recognition community, i.e. they are small, and they are rarely used to compare the effectiveness of a particular model. In my initial review, I listed a few datasets that are most commonly used for action recognition comparisons. In my view, without comparisons on these more popular datasets it is very difficult to tell the real value of the proposed approach. If the authors could demonstrate close to state-of-the-art performance on those datasets I would be more convinced that the proposed approach is effective. Currently, most of the comparison are done w.r.t baselines that are implemented by the authors which is insufficient in my opinion. Therefore, I stand by my original recommendation of rejecting the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LVaJ2Ryf6CH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper overall presents a model that defines the multi-person activities using logic expressions and uses neural logic models to generate recognitions and predictions over events. Specifically, the model follows the neural logic machines (Dong et al.) to define the operations in the networks. Authors demonstrate their model under two tasks, trajectory-based soccer event detection, and robot object manipulation event understanding. Using their self-generated datasets from simulators, the authors found that their model performs better than other baselines.\n\nPaper strengths:\n\n+ The logic forms of event understanding is a quite interesting direction, this paper could be a nice contribution towards that field.\n+ The experimental results support authors' claim on the model\n\nPotential cons:\n1. There are not very unique modeling contributions in this paper. The model used is an application of neural logic machines.\n\n2. Although I like the logical way of defining the events, it seems that the current refinement modules are quite similar to graph neural networks' message passing operations. Is the reason for the proposed model outperforming baselines that there are higher-order terms considered compared to baselines? What if a graph net/relation net with high-order interaction terms is used? Is the logical form of expressing the events really necessary (v.s. a natural language/semantic expression of events + graph nets + high-order terms)?\n\n3. Two relevant and missing citations on soccer analysis and human interaction analysis [1,2].\n\n[1] Large-Scale Analysis of Soccer Matches using Spatiotemporal Tracking Data\n\n[2] Structure inference machines: Recurrent neural networks for analyzing relations in group activity recognition", "extra_scores": {}, "extra_text": {}}, {"review_id": "AjrAR3tHI0J", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "########################################################################\n        \nSummary:\n\nThis paper proposes TOQ-Nets which is a structured neural network that learns to describe complex activities over entities and time. The model leverages relational reasoning layers which are the Neural Logic Machines (NLM) to capture the spatial information. To further capture the temporal information, this paper proposes temporal reasoning layers.The results show that their method outperforms conventional graph neural networks with high accuracy and generalization with a large margin.\n\n########################################################################       \n \nPros:\n\n- This paper extends the NLM into temporal. The proposed temporal reasoning layer borrows the idea from temporal logic \n\n- programs which is interesting and promising.\n\n- The Results show that their proposed model outperforms the STGCN with a large margin.\n\n- The paper is well-organized and easy to read.\n\n######################################################################## \n\nAlthough the proposed method is promising, the experiment settings are not convincing.\n\nCons:\n\n- Missing reference : A line of research focuses on Group Activity Recognition [1, 2] which has the similar settings as Soccer Dataset. This paper collects the soccer dataset by themself and sets STGCN as their baseline which is not originally used for this task. \n\n- Both datasets used in the paper are collected by the simulators. Compared with the Volleyball dataset [1, 2] that handles videos, the task in the paper seems much more simple and less practical.\n\n- The inputs of TOQ-net and STGCN are different. The distances between players are not input to the STGCN, however the significance of this input variable is unknown. For example, the actions “Control ball” and “Movement (without ball)” can be easily distinguished by the distance of the target player and the ball. Please explain this.\n\n- Missing Ablation study of the temporal reasoning layer. There is no experiment to show the improvement of the proposed temporal reasoning layer. Please explain how much important the temporal information is in your experiment.\n\n\n##########################################################################\n\nReasons for score: \n \nOverall, I like this idea of extending NLM into temporal. However, the experiments fail to convince me of its performance. So I tend to vote to reject temporarily. If the authors can show more evidence about the improvement of TOQ-net and eliminate my concerns, I will change my mind.\n\n##########################################################################\n\nQuestions during rebuttal period: \n \nPlease address and clarify the cons above \n\n\n\n\n\n\n##########################################################################\n\nReference\n\n[1] A Hierarchical Deep Temporal Model for Group Activity Recognition. Mostafa S. Ibrahim et. al.\n\n[2] stagNet: An Attentive Semantic RNN for Group Activity Recognition. Mengshi Qi et. al.\n\n\n--------------------------------------------------------------------------------------------------------------------------------------------\nI have read the rebuttal. The experiments in the rebuttals shows the effectiveness of TOQ-Nets in other large, real-world dataset. These experiments should be added in the revision of the paper and I would like to change my score to 6.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on temporal logic representations in neural networks.  The paper builds on work on Neural Logic Machines (Dong et al.), adding temporal quantification.  The main positives to the method are this contribution of the temporal reasoning layers (e.g. iii in Fig. 2).  This layer provides an interesting extension of existing literature in the area.

The main concerns raised by the reviewers were the following:
- Contribution of the paper over previous NLM techniques
- Relation to graph neural networks and other message passing techniques
- Use of hand-crafted initial features and resultant comparisons to baselines
- Empirical validation

After reading the authors' responses the reviewers reconsidered their positions and engaged in discussion.  While the reviewers appreciate the addition of temporal reasoning layers as an interesting contribution, there is still concern over the magnitude of this contribution and its effectiveness.

Additional points raised in the discussion include
- Lack of evaluation on standard, challenging datasets and comparisons to state of the art
- Ablation study in response (Fig. 6) does not consider absolute removal of temporal layers (main contribution).

Overall, while the paper does contribute an interesting inductive bias for learning with temporal data, the current evaluation is limited in terms of its effectiveness at the classification tasks in the experiments.  Based on the concerns raised in the initial reviews and subsequent discussion, it was determined that the paper is not ready for publication in ICLR.


Decision: Reject