Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks
Authors: Iulia Duta
Andrei Liviu Nicolicioiu
Marius Leordeanu
Venue: NeurIPS 2021 Poster (2021)
Keywords: object-centric
graph neural networks
spatio-temporal

Abstract: Graph Neural Networks are perfectly suited to capture latent interactions between various entities in the spatio-temporal domain (e.g. videos). However, when an explicit structure is not available, it is not obvious what atomic elements should be represented as nodes. Current works generally use pre-trained object detectors or fixed, predefined regions to extract graph nodes. Improving upon this, our proposed model learns nodes that dynamically attach to well-delimited salient regions, which are relevant for a higher-level task, without using any object-level supervision. Constructing these localized, adaptive nodes gives our model inductive bias towards object-centric representations and we show that it discovers regions that are well correlated with objects in the video. In extensive ablation studies and experiments on two challenging datasets, we show superior performance to previous graph neural networks models for video classification.

Discovering Dynamic Salient Regions for Graph Neural Networks
Tulia Duta" Bitdefender, Romania 1d3660cam. ac uk
Andrei Nicolicioiu" Bitdefender, Romania anicol amccolicioiu@@btteeffeeo..
Marius Leordeanu
Bitdefender, Romania
Institute of Mathematics of the Romanian Academy University "Politehnica" of Bucharest
marius leordeanu@imar..r
Abstract
Graph Neural Networks perfectly suited to capture latent interactions between various entities in the spatio-temporal domain (e.g. videos). However, when an explicit structure is not available, it is not obvious what atomic elements should represented as nodes. Current works generally use pre-trained object detectors fixed, predefined regions extract graph nodes. Improving upon this, proposed model learns nodes that dynamically attach to well-delimited salient regions, which are relevant for a higher-level task, without using any object-level supervision. Constructing these localized, adaptive nodes gives our model inductive bias towards object-centric representations and show that it discovers regions that are well correlated with objects in the video. extensive ablation studies and experiments on two challenging datasets, we show superior performance previous graph neural networks models for video classification.
Introduction
Spatio-temporal data, and videos, in particular, are characterised by an abundance of events that require complex reasoning to be understood. In such data, entities classes exist at multiple scales and in different contexts in space and time, starting from lower-level physical objects, which are well localized in space and moving towards higher-level concepts which define complex interactions. We need a representation that captures such depending on the current scene and the requirements of the task. Classical convolutional nets address apatio-temporal processing in a simple and rigid manner, determined only by fixed local receptive fields (1. Alternatively, space-time graph neural nets 2 3 offer
interactions at different level of granularity,
powerful and flexible approach modeling complex short and long-range interactions between visual entities.
In this paper, we propose a novel method to enhance vision Graph Neural Networks (GNNs) by an additional capability, missing from any other previous works. That is, to have nodes that are constructed for spatial reasoning and can adapt to the current input. Prior works are limited to having either nodes attached to semantic attention maps [4] or attached fixed locations such as grids 5 3 6). Moreover, unlike works that require external object detectors 7 our method relies a learnable mechanism to adapt to the current input.
* Equul contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We propose a method that learns to discover salient regions, well-delimited in space and time, that are useful for modeling interactions between various entities. Such entities could be single objects, parts or groups of objects that perform together simple action. Each node learns to associate by itself to such salient regions, thus the message passing between nodes is able to model object interactions more effectively. For humans, representing objects is core knowledge system [8] and to emphasize them in our model, we predict salient regions
[ that give a strong inductive bias towards modeling them.
Our method, Dynamic Salient Regions Graph Neural Network (DyReg-GNN) improves the relational processing of videos by learning to discover salient regions that are relevant for the current scene and task. Note that the model learns to predict regions only from the weak supervision given by the high-level video classification loss, without supervision at the region level. Our experiments convincingly show that the regions discovered are well correlated with the objects present in the video, confirming the intuition that action recognition should be strongly related salient region discovery. The capacity to discover such regions makes DyReg-GND GNe an excellent candidate model for tackling tasks requiring spatio-temporal reasoning.
Our main contributions are summarised as follow:
1. We propose novel method to augment spatio-temporal GNNs by an additional capability: that of learning to create localized nodes suited for spatial reasoning. that adapt to the input.
2. The salient regions discovery enhance the relational processing for high-level video clas- sification tasks: creating GNN nodes from predicted regions obtains superior performance compared to both using pre-trained object detectors or fixed regions
3. Our model leads to unsupervised salient regions discovery, a novelty in the realm of GNNs: it predicts such regions in videos, with only weak supervision at the video class level. We show that regions discovered are well correlated with actual physical object instances.
2 Related work
Graph Neural Networks in Vision. GNNs have been recently used in many domains where the
data has non-uniform structure [10 [2 13 In vision tasks, it is important to model the relations between different entities appearing in the scene [14] [5] and GNNs have strong inductive biases towards relations [16 171, thus they perfectly suited for modeling interactions between visual instances. Since an explicit structure is not available in the video, it is of critical importance to establish what atomic elements should be represented as graph nodes. As main contribution revolves around the creation of nodes, we analyse other recent GNN methods regarding the type of information that each node represents, and group them into two categories, semantic and spatial.
The approaches of 18 19 global graph nodes, each one receiving information from all the points in the input, regardless of spatio-temporal position. In [4] the nodes assignments are predicted from the input, while in [18] the associations between input and nodes are made by a soft clusterization. The work of [2] discovers different representation groups by using an iterative clusterization based on self-attention similarity.
21 22 capture the purely semantic interactions by reasoning over
The downside of these semantic approaches is that individual instances, especially those belonging to the same category, are not distinguished in the graph processing. This information is essential in tasks such as capturing human-object interactions, instance segmentation or tracking.
Alternatively, multiple methods, including ours, favour modeling instance interactions by defining spatial nodes associated with certain locations. We distinguish between them by how they extract the nodes from spatial location: as fixed regions or points [23 24], or detected object boxes [25] 26 22 28 29). The method [5] creates nodes from every point in 2D convolutional features maps, while Non-Local 30) uses self-attention [3T] between all spatio-temporal positions to capture distant interactions. Further, [3] extract nodes from larger fixed regions at different scales and processes them recurrently. Recent methods based on Transformer [32 33 also model the interactions between fixed locations on a grid using self-attention. In 77]. nodes are created from object boxes extracted by an external detector and are processed using two different graph structures, one given by location and one given by nodes similarity. A related approach is used in 27 in a streaming setting while 34 learns to hop over unnecessary frames. Hybrid approaches use nodes corresponding to
points and object features [35] 36) propagate over both semantic and spatial nodes [37 38 39].

---

PEATURSS
GENERATION
DXTRACTIONS
PROCESSIIN
RC WARPING
OUTPUT
M
X;
Nr
Y
C
Figure 1: (Left) DyReg-GNN extracts localized node useful for relational processing of videos. For each node i, from the features Xt, we predict params o, denoting the location and size region. They define a kernel K,, used to extract the localized features
from the corresponding region of X,, We process the nodes with a spatio-temporal GNN and project each node v. into its initial location. (Right) B) Node Region Generation: Functions and {o: } generate the regions params o f extracts latent representation shared between nodes, while each 9i has different params for each node i. C) Node Features Extraction: Each o, creates a kernel that is used in differentiable pooling w.r.t. og, This allows us to optimize the generation of these regions' params from the final classification loss, resulting in unsupervised discovery of salient regions.
However, methods that rely on external modules trained on additional data, such as object detectors, too dependent the module's performance. They are unable adapt to the current problem, being limited to the set of pre-defined annotations designed for another task. Differently, our module optimized to discover regions useful for the current task, using only the video classification signal.
Recently, the method 40 uses multiple position-awarn aware nodes that take into account the spatial structure. This makes it more suitable for capturing instances, but the nodes have associated static learned location, where each one is biased towards specific position regardless of the input. On the other hand, dynamically assign location for each node, based on the input, making the method more flexible to adapt to new scenes.
Dynamic Networks. Several works second-order computations by dynamically predicting different parts of their model from the input, instead of directly optimising parameters. Our work is related to STN [41] that aggregates features by interpolating from area given by predicted global transformation and to the differentiable pooling used in some object detectors The method [45). replaces the parameters
42
standard convolution with weights predicted from the
47 predict, based on the input, an offset for each position in the convolutional kernel. Similar, [48] use the same idea of predicting offsets but in a graph formulation. The common topic of these methods is to predict dynamically support for all points in a convolutional operation while we dynamically generate the input for a set of nodes designed to process high-level interactions. Related ideas, involving high-level processing of a small set of powerful modules, is also highlighted in 49 and [40].
input, resulting in a dynamically generated filter. Deformable convolutions [46]
Unsupervised Object Representations. There is an entire area of work devoted to extracting
They are successful in leveraging a reconstruction task to decompose the scene into objects, for synthetic images. In [54] it is shown that representations learned from unsupervised decomposition are also helpful in relational reasoning tasks. Methods for generating unsupervised keypoints or entities
representations centered on objects 500] in a fully unsupervised setting [5]
52
53 22
[55 56 57 58 have been generally used synthetic setting. The method (55) generates keypoints
from real images of people and faces but they use an image reconstruction objective that could not be aligned with the downsteam task. Our goal is to relate spatio-temporal entities, but without enforcing clear decomposition of the scene into objects. This allows us use simpler but effective method that learns from classification supervision of real-world videos and obtain representations that are correlated to objects.
Activity Recognition. Video classification has been influenced by methods designed for 2D images 59] 60 61 62 More powerful 3D convolutional networks have been later proposed (63) while
3

---

other methods factorise the 3D convolutions 664 65 66 bringing both computational speed and
accuracy. Methods like TSM 667 and [68 showed that simple shift in the convolutional features results in improved accuracy at a low computational budget.
3 Dynamic Salient Regions GNNs
We investigate how to create node representations that useful for modeling visual interactions between various entities in space and time using GNNs. Our proposed Dynamic Salient Regions GNN model (DyReg- -NgN learns to dynamically assign each node to certain interesting region. By dynamic, we mean that we have a fixed number N of regions that change their position and size according the input each time step. The regions assigned to each of the N nodes can change from one moment of time to the next depending on their saliency.
The main architecture of our DyReg- GNN model is illustrated in Figure I Our model receives feature volume X E Tx HTXXXWW> X XC and at each time step t we predict the location and size N regions. From these regions, differentiable pooling operation creates graph nodes that are processed by GNN and then are projected to their initial position. This module can be inserted at any intermediate level in standard convolutional model.
3.1 Node Region Generation
We want attend only to few most relevant entities in the scene, thus a small number of nodes used in DyReg-GNN (in our experiments N = 9) and is crucial to assign them to the most salient regions. The number of nodes is hyperparameter that we choose such that it exceeds the expected number of relevant entities in the scene, to increase the robustness of the model. Thus, we propose a global processing (shown in Figure B) that aggregates the entire input features to produce regions defined by parameters indicating their location (Ax, Ay) and size (w: h).
To generate N salient regions, we process the input X, using position-aware functions and {g, iE.N that retain spatial information. Nodes should be consistent across time, thus we generate their regions the same way at all time steps. by sharing in time the parameters f and {g, }. The function f is convolutional network that highlights the important regions from the input.
M, f(x,) E x'C''"C"'
(1)
For each node i, we generate a latent representation of its associated region using the {a:} functions. Each gi has the same architecture, but different parameters for each node and could be instantiated as fully connected network or global pooling enriched with spatial positional information. We generate the node regions from a global view to make the decision as informed as possible.
m t = g:(M)) CC Vi E I,N
Each of the N latent representations processed independently, with GRU (69 recurrent network (shared between nodes), to take into account the past regions' representations.
Zit = GRU(Z.,--1, mi E CC ViEIN,
At each time step, the final parameters are obtained by a linear projection W, E R''4, transformed by function to control the initialisation of the position and size (e.g. regions would start at reference points either in the center of the frame or arranged on grid). For more details about how to set the transformation
refer to the Supplemental Materials.
o (Awi,e AWii wa hi.e) = a (W.2..)) R
3.2 Node Features Extraction
The following operations applied independently each time step thus, in the current subsection, we ignore the time index for clarity. We extract the features corresponding to each region using a differentiable pooling w.r.t. the predicted region parameters o,, All the input spatial locations p R are interpolated according to the kernel function k(((p)) as presented in Figure C.

---

We present the operation for single axis since the kernel is separable. acting in the same way on both axes:
1- (P.. ( (pa) (py) E R
(5)
We define the center of the estimated region Ci,z + Az, where ca, is fixed reference point for node i (located in the frame's 's center or arranged on a grid). The values of the kernel decrease with the distance to the center and is non-zero up maximal distance of W where w; and Az; the predicted parameters from Eq.
(px) max (0, w, |i A.;
Pa
(6)
For each time step f, node is created by interpolating all points in the input X, using the kernel function. By modifying (Ar; Ay:) the network controls the location of the regions, while (h,, w;) parameters indicate their size.
W
H
Σ
Σ
(p):Py)XXp..1 RC
Pa
Setting w; = 1 leads to standard bilinear interpolation, but optimising it allows the model to adapt region's size and observe that larger result in more stable optimisation (see node size ablations from Supp. Material).
The position of the region associated with each node should be taken into account. It helps the relational processing by providing an identity for the node and is also useful tasks that require positional information. We achieve this by computing positional embedding for each node é using linear projection of the kernel K, into the same space as the feature vector 2/ and summing them.
Key Properties. By construction, the nodes in our method are localized, meaning that they are clearly associated with location: they pool information from clearly delimited area in space and they maintain position information from the positional embedding. These two aspects could be helpful in tasks involving spatio-temport reasoning.
The dynamic aspect refers to the key capability of adapting the region's position and size according to the saliency of the input at each time step. This is done by predicting the regions from the input with the operations from equations
An essential aspect of this method that the final classification loss is disferentiable with respect to regions' parameters as the gradients are passing from the nodes outputs v; through the kernels k; to the parameters w!; and Ar; This allows learn regions from the final loss, without direct supervision for the region generation. Thus the method has more flexibility in learning relevant regions as appropriate for the task.
3.3 Graph Processing
For processing the nodes' features, different spatio-temporal GNNs could used. Generally, they follow framework 12 of sending messages between connected nodes, aggregating [70] [1 and
updating them.
The specific peessage-passing mechanism is not the focus of the current work, thus we follow general formulation similar to 3 for recurrent spatio-temporal graph processing. uses two different stages: one happening between all the nodes at single time step and the other one updating each node across time. For each time step f, we send messages between each pair of two nodes, computed as an MLP (with shared parameters) and aggregates them using a dot product attention a(v, U;) E R.
N
Vi,1 Σ
a(v V..,) MLL()))...V e C
(8)
We incorporate temporal information through a shared recurrent function across time, applied independently for each node.
1i = GRU(V Vs,)) e RC
(9)
The GRU output represents the updated nodes' features and the two steps are repeated K times.
5

---

Table 1: Results val. set of Annt-SmtV2 showing the importance of salient regions discovery. We compare our predicted (unsupervised) regions to fixed grid regions boxes given by an object detector using the same GNN model. The mean L2 distance between the regions and gt. objects proves that DyReG-GNN has regions correlated with
Table 2: Consistent improvements over different backbones on the vali- dation set of Smt-Smt-VI using central crop evaluation.
objects, while also having superior accuracy and efficiency.
Model
Acc (%)
TSM-R18
33.7
Model
Regions discovery
FLOPS
Dist
Acc
TSM-RI8 DyReg-GNN
35.6 (1 1.9)
+
(%:y
13D-R50
44.0
TSM-R50
65.8G
63.4
I3D-R50
DyReg DNR
45.4 1.4)
+ GNN+Fixed
Grid Obj detector
+1.4G
0.170
64.1
TSM-R50
47.2
+ GNN+Detector
+41.1G
0.125
64.0
TSM-R50 DyReg-GNN
48.8 1.6)
+ DyReg-GNN
Unsupervised
+1.6G
0.129
64.8
3.4 Graph Re-Mapping
To use our method module inside any backbone, we produce an output with the same shape as the convolutional input X, E IR Hx xc The resulting features of each node are sent to all locations in the input according the weights used in the initial pooling from Section 3.2
N Σ
(po ERC
(10)
4 Experimental Analysis
While much effort is put into the creation of different video datasets used in the literature, such as Kinetics [63] or Charades (72), it has been argued [73] that they contain biases that make them solvable without complex spatio-temporal reasoning. CATER 773 proposed alleviate this, but it is too small (5500 videos) and still has biases that make the last few frames sufficient for good performance (34 We test our model two video classification datasets that seem offer the best advantages, being large enough and requiring abilities to model complex interactions. We evaluate on wal-world datasets, Something- Sooteiinn--ottiingg--.... VI& V2 774,, while we also test on variant of the SyncMNIST 3 dataset that is challenging and requires aentio-temporal reasoning, while allowing fast experimentation. The code for our method can be found in our repository
4.1 Human-Object Interactions Experiments
Something- Soretninn-mtt-nnn--... V1&V2 74 datasets classify scenes involving human-object complex in-
teractions. They consist of 86K / 169K training videos and 11K / 25K validation videos, having 174 classes. Unless otherwise specified, all experiments on as a backbone and we add instances of our module at multiple stages.
datasets use
TSM-ResNet-50 677
Studying the Importance of Salient Regions Discovery. We test the importance of the dynamic regions for GNNs vision methods by training models where we replace the predicted regions with the same number of fixed regions on grid (GNN Fixed Regions) or boxes (GNN Detector) as given by Faster R-CNN 75 trained on MSCOCO (76
The detector based model has comparable results to the one with fixed regions, seemingly being unable to fully benefit from the correctly identified objects. The relative weaker performance of this model could be due the fact that the pre-trained detector is not well aligned the actual salient regions that are relevant for the classification problem.
On the other hand, this weakness is not applicable for DyReg GNN that learns suitable regions for the current task and it obtains the best performance as seen in Table Not only that it does not require object annotations, but is also more computationally efficient. Running the detector on video of
httpp:://github.ccom/bii

---

Table 3: Results val. set of Smt-Smt-V1. Our model achieves competitive results compared to re- cent works (best results in red), while it outperforms all other graph-based methods (best results in blue). All the methods use ResNet50 backbone.
Table 4: Results on val. set of Smt-Smt-Va in comparisons to recent works. DyReg- GNN improves the ISM-ResNet50 backbone when using either one (14) or three (r3-4-5) modules of graph processing and it obtains top results.
DyReg-GNN
Model
Regions discovery
#F
Top 1
Top 5
TSM
67
16
48.4
78.1
Model
BB
Top
Top
non-Graph
S3D 64
64
48.2
78.7
TRG
82
R50
59.8
87.4
GST 78
16
48,6
77.9
GST
R50
62.6
87.9
SmallBig
79
16
50.0
79.8
v-DP
83
D121
62.9
88.0
STM 80
16
50.7
80.4
SmallBig 79
R50
63.8
88,9
MSNct 8
16
52.1
82.3
STM 80
R50
64.2
89.8
ORN 14
Objects
8
36.0
MSNet [8
R50
64.7
89.4
NL 13D 7
Grid
32
44.4
76.0
TSM 67
R50
63.4
88.5
Graph
NL GCN 7
Objects
32
46.1
76.8
TSM--yRee-r4
R50
64.3
88.9
TRG 82
Frames
16
48.1
80.4
TSM-
R50
64.8
89.4
RSTG
Grid
32
49.2
78.8
TSM+DyReg
Dynamic
49.9
79.0
size 224 x 224 would add 39.7 GFLOPS on its own, comparing to the 1.6G of three DyReg-GNT GNN modules, from which 0.2G represents the regions prediction.
Overall, our method, with unsupervised regions obtains superior performance in terms of accuracy and computational efficiency representing a suitable choice for relational processing of a video.
Object-centric representations. The nodes represent the core processing units and their localiza- tion enforces clear decision what specific regions to focus on while completely ignoring the rest, as form of hard attention. Different from other works [77), our hard attention formulation is differen- tiable. To better understand what elements influence the model predictions, we could inspect the pre- dicted kernels, thus introducing another layer of interpretability to the model, on top of the capabilities offered by the convolutional backbone. Visualisations of our nodes' regions reveal that generally, they cover the objects in the scene. For example, in the first row of Figure 33 the nodes are placed around the phone in the first frames and then separate into two groups, one for the phone one for the hand.
The localized nodes make our model capable of discovering salient regions, leading to object-centric node representations. We quantify this capacity by measuring the mean L2 distance (normalised to the size of the input) between the predicted regions and ground-truth (gt.) objects given by (28). The metric is completely defined in the Supp. Materials. We observed that the score improves during the learning process (it reaches 0. 129 starting from 0.201), although the model is not optimized for this task. This suggests that the model actually learns object-centric representations.
In Table we also compare the final L2 distance of our best DyReg- ReReggGNN GNN model to an object detector and to fixed grid regions. Although our method is not designed and supervised to find object regions, we observe that is able to predict locations that are fairly close to gt. objects. The L2 distance sim- ilar to the one obtained by external model 10.129 vs 0.125), trained especially for detecting objects.
We observe that learning the regions' size important for the stability the optimisation and thus for the final performance (see Tab Supp. Material Regions' Size section). However, the predicted size not as well aligned with the size of the true objects. This gives us hint that for the action classification task it is important to have good region locations, but their size is less relevant. We leave a more thoroughly investigation for futures work.
These experiments prove that the high-level classification task is well inter-related with the discovery of salient regions and that, in turn, these regions improve the relational processing in the recognition task. First, we show that DyReg-GNN's region obtain superior accuracy and efficiency than other methods of extracting nodes and second, these regions are well correlated gt. object locations.
Comparison to recent methods. DyReg-GNN can be used with any convolutional model and we show that it consistently boosts the performance of multiple backbones(Table 2 We compare to recent methods from the literature in Table LableBand and Table 4 Our method improves the accuracy over the
7

---

SEM-ResNet50 backbone on both Smt-Smt- VI and Smt-Smt- V2 by 1.5% and 1.4% respectively and achieves competitive results. Compared to all the other graph based methods we obtain superior re- sults, showing that our discovery of dynamic regions is effective for space-time relational processing.
Implementation Details Unless otherwise specified, we use TSM-ResNe550 (pre-trained on Ima- geNet (84) backbone and add instances of our module in the last three stages. To benefit from ImageNet pre-training, we add our graph module as a residual connection. We noticed that models using multiple graphs have problems learning adapt the regions from certain layers. We fix this by training models containing a single graph at each single considered stage, as the optimisation process is smoother for a single module, and distill their learned offsets into the bigger model. The distillation is done for the first 10% of the training iterations to kick-start the optimization process and then continue the learning process using only the video classification signal.
In all experiments we follow the training setting of [67]. using 16 frames resized to have the shorter side of size 256, and randomly sample crop of size 224 x 224. For the evaluations, follow the setting in [67] of taking 3 spatial crops of size 256 x 256 with temporal samplings and averaging their results. For training, we use SGD optimizer with learning rate 0.001 and momentum 0.9, using a total batch-size of 10, trained on two GPUs. We decrease the learning rate by factor of 10 three times when the optimisation reaches a plateau.
4.2 Synthetic Experiments
SyncMNIST is a synthetic dataset involving digits that move on a black background, some in a random manner, while some move synchronously. The task is to identify the digits that move in the same way. We use a harder variant of the dataset (MultiSyneMNISST, where the videos could include multiple digits of the same class. The challenge consists in finding useful entities and model their relationships while being able to distinguish between instances of the same class. Each video contain 5 digits and the goal is find the smallest and the largest digit class among the subset that moves in the same way. This results in a video classification task with 56 classes. The dataset contains 600k training videos and 10k validation videos with 10 frames each.
Studying the Importance of Dynamic Nodes. We validate our assumption that the nodes should be dynamic, meaning that their regions position and size should be adapted according to the input at each time step. We investigate (Table. 5) different types of localized nodes, each adapting to the input to a varying degree, and show the benefits of our design choices. We experiment with variants of our model, all having the same backbone (2D Re (85)), the same graph processing and same pre-determined number of regions, but we constrain the node regions in different ways.
Fixed Model extracts node features from regions arranged on grid, with fix location and size.
Static Model investigates the importance of dynamic regions by optimising regions based on the whole dataset but do not take into account the current input. Effectively, the features z; from Eq. become learnable parameters.
Constant- -ime Model has regions adapted to the current video but they do not change in time.
DyReg-GNN GNN Model predicts regions defined by location and size, and we can either pre-determine a fixed size for all the regions (Position-Only Model) or directly predict it from the input as in our complete model (DyReg- (DyReg-GNN GNN Model).
These experiments (Table 5). show that the fixed region approach (Fixed Model) achieves the worst results, slightly improving when the regions are allowed to change according to the learned statistics of the dataset (Static model). Adapting to the input is shown to be beneficial, the performance improving even when the regions are invariant in time (Constant- Time Model), and further more when predicting different regions at every time steps (Position- (Position-Only). The best performance is achieved when both the location and the size of the regions are dynamically predicted from the input (DyReg-GNN),
In Figure 2 we show examples of the kernels obtained for each of these models. We observe that the Static Model's 's kernels are learned to be arranged uniformly on a grid, to cover all possible movements in the scene, while the Constant- Time Model's kernels are adapted for each video such that they cover the main area where the digits move in the current video. The full DyReg-GNN Model learns to reduce the size of its regions and we observe that they closely follow the movement of the digits.
The previous experiments show that performance increases when the model becomes more dynamic, proving that our model benefits from nodes that are adapted to higher degree to the current input.
8

---

Static
b. Ct Time
Semantic
di d.DyReG
Figure 2: Nodes' regions MultiSyneMNISTT for 3 frames. a) Static Model, ignoring the input, learns a regular grid; b) Constant-Time predicts the same regions for all time steps, covering the movement in the video; c) The attention map of single node of Semantic that can't distinguish between different instances of the same digit; d) DyReg Byyee-GGNN generally follows the digits locations each time steps while also adapting the regions' size.
Figure 3: Nodes' regions on Smt-Smt-V We show (Ist row) the center of all the N regions as predicted by DyReg-GNN (each color for node). Each node region (last rows) corresponds to zone from the latent conv features pooled by node.
Table 5: Ablation of dynamic nodes MultiSyncM- NIST. It is crucial to have regions that adapt based on the input (Dynamic), both their position (Pos.) and size at each time step.
Table 6: Semantic vs spatial nodes on MultiSyncMNIST. The localized (spa- tial) node regions of DyReg-GNN better suited than semantic nodes' maps obtained by the Semantic Model.
Model
Optimise Pos.
Time
Dynamic Pos.
Acc
Varying
Size
Model
Params (M)
Acc
Fixed
78.85
ResNet-18
2.79
52.29
Static
81.48
Fixed
2.82
78.85
Ct-Time
86.77
Semantic
2.85
82.41
Pos-Only
93.41
DyReg-GNV -reeg-GNN-Liee
2.83
91.43
DyReg-Gih GNN
95.09
DyReg-GNN
3.08
95.09
Studying the Importance of Localized Nodes. We argue that nodes should pool information from different locations according to the input, such that the extracted features correspond to meaningful entities. Depending on the goal, we could balance between semantic nodes globally extracted from all spatial positions localized (spatial) nodes that obtained from well-delimited regions.
Semantic Model creates nodes similar to locations and could represent a semantic concept. Each node is extracted by a global average pooling where the weights at every position p are directly predicted from the input features at that location. Practically, we replace the spatially delimited kernel used in our model with this global attention map.
where each node extracts features from all the spatial
A major downside of this approach is that it does not distinguish between positions with the same features, making it harder to reason about different instances. Figure C shows the attention map of single node and observe that it has equally high activations for both instances of the same digit, thus making it hard to distinguish between them.
This limitation does not exist in our DyReg- BNN model, it predicts localized nodes that favour the modeling of instances. For comparison, we use two variants with different number of parameters and show that they clearly outperform the semantic model (Table G). These experiments prove that in cases that involve spatial reasoning of entities, such as the current task, DyReg-Ghi RyReeg-GNN is a perfect choice, showing its benefits for spatio-temporal modeling.
Implementation details. All models share the ResNet-18 backbone with stages, where the graph receives the features from the second stage and sends its output to the third stage. We use N graph nodes and repeat the graph propagation for three iterations. In our main model, f from Eq. D a small convolutional network while g a fully connected layer. For the lighter model that implements g as global pooling enriched with spatial positional information, we refer to the Supp. Materials. The graph offsets are initialized such that all the nodes' regions start in the center of the frame. In all experiments, we use SGD optimizer with learning rate 0..01 and momentum 0.9, trained on single GPU.
Key Results. In the previous section, we experimentally validated that: DyReg-GGNi consistently improves multiple backbones (Table obtaining competitive results (Table 4 2. learned dynamic

---

regions are crucial for good performance (Table 5 and 3. these regions are preferable to fixed regions or external object detectors for space-tima GNNs (Table(T): predicted nodes correspond to salient regions (Fig- 23 and are well correlated with objects (Table I
5 Conclusions
We propose Dynamic Salient Regions Graph Neural Networks (DyReg-GN)) GNN), a relational model for processing spatio-temporal data (videos), that augments visual GNNs by learning to predict localized nodes, adapted for the current scene. This novel method enhances the relational processing of spatio-temporal GNNs and experimentally prove that it is superior to having nodes anchored in fixed predefined regions or linked to external pre-trained object detectors. Although we do not use region level supervision, the learning dynamies of high-level classification produces salient regions that are well correlated with object instances. We believe that our method of learning dynamic, localized nodes is a valuable direction that could lead further advances the growing number of powerful relational models in apatio-tempora domains.
We would like thank Florin Brad, Elena Burceanu and Florin Gogianu for their valuable feedback and discussions of this work. This work has been supported in part by Bitdefender and UEFISCDI, through projects EEA-RO-2018-0490 and PN--I-PAADD-PCE-220----3
Acknowledgment
References
[1] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. In D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances Neural Information Processing Systems Curran Associates, Inc., 2016.
29, pages
[2] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[3] Andrei Nicolicioiu, Julia Duta, and Marius Leordeanu. Recurrent space-time graph neural networks. In Advances in Neural Information Processing Systems 32, pages 12838-12855 Curran Associates, Inc., 2019.
[4] Y. Chen, M. Rohrbach, Z. Yan, Y. Shuicheng, J. Feng, and Y. Kalantidis. Graph-based global rea- soning networks. In 2019 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 433 043-442,240 2019.
[5] Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems 30, pages 4967-4976, 2017.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[7] Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pages 399-417, 2018.
[8] Elizabeth S Spelke. Core knowledge. American psychologist, 55(11 1:13, 2000.
[9] Bogdan Alexe, Thomas Deselaers, and Vittorio Ferrari. What is an object? In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 73-80, 2010.
[10] Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. CoRR, abs/1312.6203, 2013.
[11] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physies. In Advances in neural information processing systems, pages 4502-45100 2016.
10

---

[12] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee whye Teh, edi- tors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1263-1272 2017.
[13] Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs, 2018.
[14] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, June 2018.
[15] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. Proceedings of the European Conference Computer Vision (ECCV), pages 401-417, 2018.
[16] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1806.0080.001200 2018.
[17] Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken ichi Kawarabayashi, and Stefanie Jegelka. What can neural networks reason about? In International Conference on Learning Representations, 2020.
[18) Yin Li and Abhinav Gupta. Beyond grids: Learning graph representations for visual recognition. In Advances in Neural Information Processing Systems, pages 9225-9235, 2018.
[19] Xiaodan Liang, Zhiting Hu, Hao Zhang, Liang Lin, and Eric Xing. Symbolic graph reasoning meets convolutions. In Advances in Neural Information Processing Systems 31, pages 1853-
1863. 2018
[20] Songyang Zhang, Xuming He, and Shipeng Yan. Latentgnn: Learning efficient non-local relations for visual recognition. In International Conference on Machine Learning, pages 7374-7383. PMLR, 2019.
[21] Thomas Kipf, Elise van der Pol. and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[22] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv preprint arXiv:2006. 15055, 2020.
[23] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A' 2-nets: Double attention networks. In Advances in Neural Information Processing Systems, pages 350-359, 2018.
[24] Gao, T. Zhang, and C. Xu. Graph convolutional tracking. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4644 4654, 2019.
[25] Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and Trevor Darrell. Spatio-temporal action graph networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0-0, 2019.
[26] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. A structured model for action detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 9975-9984, 2019.
[27] Hao Huang, Luowei Zhou, Wei Zhang, Jason Corso, and Chenliang Xu. Dynamic graph modules for modeling object-object interactions in activity recognition. In Proceedings of the British Machine Vision Conference (BMVC), pages 101.1-101.12. BMVA Press, September 2019.
[28] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-ele Compositional action recognition with spatial-temporal interaction networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2020.
11

---

[29] Moshiko Raboh, Roei Herzig, Jonathan Berant, Gal Chechik, and Amir Globerson. Differen- tiable scene graphs. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
[30] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, page 4, 2018.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, pages
2017.
[32] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: (506::9900.09000 2019.
[33] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, pages 213-229, 2020.
[34] Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizit Martin Renqiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In International Conference on Learning Representations, 2021.
[35] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 318-334, 2018.
[36] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 244-253, 2019.
[37] Xinlei Chen, Li-Jia Li, Li Fei-Fei, and Abhinav Gupta. Iterative visual reasoning beyond convolutions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7239-7248, 2018.
[38] Effrosyni Mavroudi, Benjamın Bejar, and René Vidal. Representation learning on visual- symbolic graphs for video understanding. In The European Conference on Computer Vision (ECCV), 2020.
[39] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3146-3154, 2019.
[40] Nasim Rahaman, Anirudh Goyal, Muhammad Waleed Gondal, Manuel Wuthrich, Stefan Bauer, Yash Sharma, Yoshua Bengio, and Bernhard Schölkopf. S2rms: Spatially structured recurrent modules. arXiv preprint arXiv:2007.0653. 065333 2020.
[41] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial trans- former networks. Advances Neural Information Processing Systems, volume 28, pages 2017-2025, 2015.
[42] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3150-3158, 2016.
[43] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[44] Borui Jiang, Ruixuan I Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localiza- tion confidence for accurate object detection. Proceedings of the European Conference on Computer Vision (ECCV), pages 784-799, 2018.
12

---

[45] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In Advances Neural Information Processing Systems, pages 667-675, 2016.
[46] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[47] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 9308-9318 2019.
[48] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Tom: Dynamic graph message passing network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[49] Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1900 199999999999 2019.
[50] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 2020.
[51] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-object rep- resentation learning with iterative variational inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2424 2433. PMLR, 09-15 Jun 2019.
[52] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint
2019.
[53] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference on Learning Representations, 2020.
[54] Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, and Austin R. Benson. Better set representations for relational reasoning. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria-Florins Balcan, and Aisua-Tiinn Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[55] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object landmarks through conditional image generation. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[56] Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[57] Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg. Causal discovery in physical systems from videos. In Advances in Neural Information Processing Systems, volume 33, pages 9180-9192. Curran Associates, Inc., 2020.
[58] Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. In Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 1439-1456. PMLR, 30 Oct-0l Nov 2020.
[59] Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. Beyond short snippets: Deep networks for video classification. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 4694-4703 2015.
13

---

[60] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan. Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In and pattern recognition, pages CnnPrroce Proceedings of the IEEE conference on computer vision
2015.
[61] Chih- Yao Ma, Min-Hung Chen, Zsolt Kira, and Ghassan AlRegib. Ts-Istm and temporal- inception: Exploiting spatiotemporal dynamics for activity recognition. Signal Processing: Image Communication, 2018.
[62] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. Proceedings of the European Conference on Computer Vision (ECCV), pages 803-818, 2018.
[63] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 4733-4733. IEEE, 2017.
[64] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spa- tiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305-321, 2018.
[65] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1450-6459, 2018.
[66] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 5552- 1922-556,2000
[67] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understand- ing. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
[68] Linxi Fan, Shyamal Buch, Guanzhi Wang. Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[69] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
1734, 2014.
[70] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[71] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[72] Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, 2016.
[73] Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. In International Conference on Learning Representations, 2020.
[74] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. al. The" something something" video database for learning and evaluating visual common sense. In ICCV, volume 1, page 3, 2017.
[75] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances neural information processing systems, pages 91-99, 2015.
14

---

[76] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and Larry Zitnick. Microsoft coco: Common objects context. ECCV, European Conference on Computer Vision, September 2014.
[77] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho. Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pages 2015.
PMLR,
[78] Chenxu Luo and Alan Yuille. Grouped spatial-temporal aggretation for efficient action recogni- tion. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
[79] X. Li, Yali Wang, Zhipeng Zhou, and Yu Qiao. Smallbignet: Integrating core and contextual views for video classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1089- 1098, 2020.
[80] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotem- poral and motion encoding for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 2000-2009, 2019.
[81] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding. In European Conference Computer Vision, pages 345-362. Springer, 2020.
[82] J. Zhang, F. Shen, Xing Xu, and H. Shen. Temporal reasoning graph for activity recognition. IEEE Transactions on Image Processing. 29:5491-5506, 2020.
[83] Yizhou Zhou, Xiaoyan Sun, Chong Luo, Zheng-Jun Zha, and Wenjun Zeng. Spatiotemporal fusion in 3d cnns: A probabilistic view. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 9829-9838, 2020.
[84] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), pages 211 1-222. 2015.
[85] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 770-778, 2016.
15


Reviews:
[{"review_id": "riMYCusRFtH", "reviewer": "Reviewer_5S3R", "summary": "This paper introduces a graph-based model for salient region discovery in videos, called Spatio-Temporal Graph Neural Networks (DyReg-GNN). The key idea is to capture latent interactions between various entities by graph; moreover, RNN is used to capture inter-flame information. Generally, the idea is reasonable and the experiments show the effectiveness of the proposed approach.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The idea of using graph-based techniques for capturing inter-entities is reasonable; \n+ The paper describes each component clearly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- It is unclear to me why the proposed method is unsupervised. It seems that the proposed approach requires video-class level supervision for learning parameters; \n\n- The graph net/CNNs for better feature representation learning has been studied by many other research work. Even though the paper is the first to introduce it for salient region discovery, the contributions of the paper are still limited as a NeurIPS paper.\n\n- The experiments show that the proposed approach achieves competitive results; yet, it dose not outperform SOTA approaches such as MSNet.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "TSOpVj6YHl4", "reviewer": "Reviewer_y7mA", "summary": "The paper proposes a novel approach for generating dynamic visual nodes, ideally capturing salient visual regions, which can then be fed to a spatio-temporal Graph Neural Network in order to model intra-region relationships. In contrast to prior work, which uses feature map columns in a grid or object detector bounding boxes as the input visual nodes to a GNN, the proposed DyReg-GNN learns to localize a fixed number of visual nodes from a video, without using object detectors or assuming spatial annotations.\n\nTo achieve this, the authors propose using node-specific trainable neural networks, which output region locations and sizes based on the input spatio-temporal feature map. Then spatio-temporal features are differentiably-pooled from each one of these regions, are fed to a standard GNN, are refined based on interactions, and finally are mapped back to their original spatio-temporal positions to yield a refined spatial feature map. Therefore, the DyReg-GNN can be inserted at any intermediate level in a standard convolutional model.\n\nResults show that the proposed approach significantly outperforms GNNs that are applied on a fixed grid on a synthetic dataset (MultiSync MNIST), is much more efficient than methods that use object detectors, and the generated regions are qualitatively shown to correlate with objects, change size over time, and even retain object identities over time. The method is also evaluated on Sth-Sth V1, where it leads to a competitive performance, although the improvement over the grid baseline is much smaller.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n=======\n\nS1) Well-motivated approach: The paper addresses a crucial limitation of spatio-temporal GNN approaches applied to video, namely the need for object detectors in order to generate candidate salient regions. This is not only computationally expensive, but is also limited by the proposal recall of object detectors. The proposed approach aims to remove the need for this step, by automatically learning to localize a fixed number of salient regions in each video frame, without spatial annotations. \n\nS2) Original framework: The approach is very nicely positioned w.r.t the prior work and is exploring a novel direction, that of automatically generating salient regions beyond grids. It cleverly leverages networks that output location and size, differentiable pooling ideas (such as the ones in Spatial Transformer Networks[39]), and spatio-temporal GNNs[3].\n\nS3) Well-designed ablations in a synthetic dataset: Except for tackling a novel direction, another strong point of the paper are the experiments on the MultiSyncMNIST dataset. These controlled experiments clearly demonstrate: a) that using the discovered region nodes instead of a fixed grid improves video classification (Table 5), b) using the discovered, visually grounded, region nodes is better than finding latent semantic nodes and applying the GNN on them (Table 6), c) generated nodes tend to follow the digits locations, while also adapting region's size.\n\nS4) The paper is very well-written, is easy to read and equations are properly explained. The approach also seems technically sound.\n\nS5) The authors have provided rich qualitative examples, videos, implementation details, and code. A lot of limitations are also discussed.\n\nWeaknesses\n==========\nW1) Choice of benchmark dataset (Something-Something): I think that the proposed method would be useful in real-life applications if it could significantly improve upon a fixed grid approach in real datasets, without the added overhead of an object detector. I appreciate that the authors compared these 3 settings. However, I feel that the chosen real-life dataset is too limited in terms of the richness of scenes. This does not only limit the expected impact from using salient regions instead of grids, but might also be an easy dataset for the unsupervised region discovery.For example, most scenes have only a few objects and a single person (hands) interacting with them. Therefore, although action recognition in SthSth is indeed a challenging spatio-temporal reasoning task, finding salient regions is not as hard, since most of the foreground blobs are relevant. We can see that from the experimental results as well. In Table 1, we observe that using a fixed grid vs an object detector vs the proposed salient nodes yields an accuracy of 64.1 vs 64.0 vs 64.8. The fact that object detections do not improve over the grid might be because the grid already captures all the relevant objects and there are not many irrelevant grid cells.\n\nW2) Missing ablations: Was there an ablation about not using the GRU when generating nodes over time (Eq. 3)? Also, what is the impact of the attention component in Eq. 8? Could it be that grid-based GNNs without attention would be much worse than the proposed DyReg-GNN? Did the authors experiment with different number of GNN layers or different loss scalar weight \\lambda?\n\nW3) Evaluation of alignment of generated regions with ground-truth objects: Why (only) use the L2 distance for comparing regions with ground-truth object regions, and not IoU as well (Table 1)? Also, an additional reported metric could be the number of salient regions covering each object instance. The authors acknowledged this limitation, namely that because of the fixed number of nodes, a lot of the generated regions can focus on the same object. Object recall is also an important metric. It would be also interesting to have a quantitative evaluation on whether the model learns some form of tracking, although it is not the primary goal.\n\nW4) Comparison with SOTA: The approach improves marginally improved results compared to RSTG (same? GNN applied on feature grids) - 49.2 vs 49.9%. Also, the method is not state-of-the-art when compared to non-graph methods that use the same backbone and number of frames.  Question: does the RSTG[3] also use the TSM module?\n\nMinor comments\n============\nDC1) The authors could also cite: Dynamic Graph Modules for. Modeling Object-Object Interactions in. Activity Recognition, BMCV19 , which shares the same ideas of having a latent, location graph, that is visually grounded, although that method also uses object detections for the visual graph.\nDC2) In the state-of-the-art tables, it would be useful to add a column regarding the nodes used by the compared GNNs (grid vs object vs dynamic).\nDC3) Since the node position generation networks are 9 fixed networks, was there any qualitative analysis on what types of regions they generate? For example, is there any correlation between the region generated by the first node generation network for similar scenes? Is there any cluster of objects that arises?\n\nRating Justification\n===============\nOverall, the paper proposes a novel approach for spatio-temporal interaction modeling in videos, which tackles the difficult task of dynamically learning salient regions without spatial annotations. The approach is well-motivated, thoughtfully designed and experimental results on synthetic data validate its effectiveness. I am still skeptical whether the approach would be feasible in video datasets with richer scenes, such as Charades or cooking datasets, where object detection is itself more challenging than SthSth. In such datasets, not only the small fixed number of nodes could be an issue, but also training could be harder since the model is ideally trying to capture object detection and object tracking without annotations.  Still, I think the work in its current form opens up an interesting direction in the field and is showing promising results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Q6TY87ouQS_", "reviewer": "Reviewer_9uPY", "summary": "This paper proposes a graph based network for spatio-temporal understanding. The core idea of the paper is to learn to select visual features for node representations in graph dynamically using convolutional encoders. The paper shows competitive results in the Smth-Smth v1 and v2 datasets and MultiSync MNIST. The contributions of the paper are demonstrating a method about how to learn to select dynamic regions and making it work in the context of graph neural networks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses an important problem of spatio-temporal understanding. It is important to investigate approaches on understanding this modality more deeply (e.g. via reasoning) rather than superficially building deeper networks and this paper makes a step in the right direction.\n\nI have the following comments and questions about the paper:\n\n1) What are the benefit of using dynamic conv features apart from performance scores? The benefit of using static nodes (e.g. using objects) is that there is increased model interpretability and the model learns to establish between static entities in the dataset. Using convolutional features that change every time-step based on output from conv layers is not very interpretable. It may increase the performance of your model, since the model learns to attend to larger number of foreground and background features. However, this benefit is also seen with object detectors where ROIs are used [A].\n\n2) There is criticism of datasets in Section 4, some of which is valid. Kinetics does have a high spatial bias but does contain fine-grained classes (e.g. around basketball) that are far from being solved and it would be interesting to see how your model can separate actions for classes that involve similar objects (or salient regions). Similarly, the size of CATER was limited by design. Also, [68] does release an updated dataset that does require longer term reasoning and maybe worth looking into. CATER is perhaps a great fit for the central idea of the paper as the attention region should dynamically move from snitch to cones (in Task 3) as the video progresses.\n\n3) In the contributions, you mention that the paper “enhances relational processing”, it is not clear that the relational processing enhaced i.e. better higher order relations are established. My intuition is that by regressing and attending to a large number of region features the model is better regularized as well as gets additional signals that a pre-trained object detector misses.\n\n4) How did you pick 9 regions/objects? Perhaps to make the system completely dynamic, this can be a learned parameter as well.\n\nAfter reviewing rebuttal responses from authors to my review and other reviews, I decided to change my score to Accept.\n\n[A] Attend and Interact: Higher-Order Object Interactions for Video Understanding https://arxiv.org/abs/1711.06330", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately addresses the social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "NnJDS9DrxcH", "reviewer": "Reviewer_uuaS", "summary": "The paper proposes to improve spatio-temporal graph neural networks by adaptively computing the input node features from spatially localized attention regions. The attention regions are dynamically predicted from each input frame, which is in contrast to previous methods that use fixed regions or pretrained object detectors. It is shown that by only optimizing a video classification loss, the model can learn to attend to objects, thereby extracting object-centric node features and facilitating interaction modeling. On the Something-Something-V1&V2 datasets, the model achieves comparable or slightly better classification accuracy than previous work, and can consistently boost performance when combined with multiple backbones.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is an interesting idea to let the nodes of a graph net dynamically attend to relevant input regions. This has the potential for better interpretability and interaction modeling. However, similar ideas have been proposed in previous methods like OP3[1] and V-CDN[2], where they predict object masks / keypoints and use graph nets to model dynamics, all without supervision. Compared to these methods, the strength of this paper seems to be that it can work on real-world datasets like Something-Something-V1&V2, when given the video class labels. However, I had some concern about the experiments:\n\n- The main metric considered is the classification accuracy, on which using dynamically adapted regions only slightly outperforms using fixed regions (Table 1). Also, in Appendix Figure 5, it seems that each node covers a specific region. These results lead to the question: Is it necessary to use dynamic regions?\n- For object-centric representations to benefit interaction modeling, it is best if the representations are temporally consistent, i.e., same node capturing same object across time. However, I did not see much evalution on this temporal consistency aspect, and Figure 5 seems to suggest that if an object moves a long distance, it will be captured by different nodes.\n- I was a bit confused about the visualization of node regions in Figures 3 and 4. In particular, the number of regions in Figure 4 is much smaller than the number of nodes, so what nodes are visualized? Also the learned position of nodes looks similar to keypoints [2-4], which can be predicted without supervision even for realistic images. I think it would be better to have some discussion about these methods.\n\n**MINOR COMMENTS**\n\n- What is the benefit of the proposed kernel function compared to spatial transformer networks?\n\n[1] Entity Abstraction in Visual Model-Based Reinforcement Learning (CoRL 2019).\n\n[2] Causal Discovery in Physical Systems from Videos (NeurIPS 2020).\n\n[3] Unsupervised Learning of Object Landmarks through Conditional Image Generation (NeurIPS 2018).\n\n[4] Unsupervised Learning of Object Keypoints for Perception and Control (NeurIPS 2019).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In addition to the limitations mentioned in the paper, I think the proposed model may not have good temporal consistency when objects move long distances, and thus the improvement in interaction modeling can be limited in long videos.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received both positive and negative reviews. After a round of discussion between the reviewers and reading the author's rebuttal, three of the four reviewers recommended accepting the paper and I am happy to accept it. I encourage the authors to include in the final version the additional material that they mentioned in the rebuttal. 
Decision: Accept (Poster)