Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks

1 Introduction . Spatio-temporal data , and videos , in particular , are characterised by an abundance of events that require complex reasoning to be understood . In such data , entities or classes exist at multiple scales and in different contexts in space and time , starting from lower-level physical objects , which are well localized in space and moving towards higher-level concepts which define complex interactions . We need a representation that captures such spatio-temporal interactions at different level of granularity , depending on the current scene and the requirements of the task . Classical convolutional nets address spatio-temporal processing in a simple and rigid manner , determined only by fixed local receptive fields [ 1 ] . Alternatively , space-time graph neural nets [ 2 , 3 ] offer a more powerful and flexible approach modeling complex short and long-range interactions between visual entities . In this paper , we propose a novel method to enhance vision Graph Neural Networks ( GNNs ) by an additional capability , missing from any other previous works . That is , to have nodes that are constructed for spatial reasoning and can adapt to the current input . Prior works are limited to having either nodes attached to semantic attention maps [ 4 ] or attached to fixed locations such as grids [ 5 , 3 , 6 ] . Moreover , unlike works that require external object detectors [ 7 ] our method relies on a learnable mechanism to adapt to the current input . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose a method that learns to discover salient regions , well-delimited in space and time , that are useful for modeling interactions between various entities . Such entities could be single objects , parts or groups of objects that perform together a simple action . Each node learns to associate by itself to such salient regions , thus the message passing between nodes is able to model object interactions more effectively . For humans , representing objects is a core knowledge system [ 8 ] and to emphasize them in our model , we predict salient regions [ 9 ] that give a strong inductive bias towards modeling them . Our method , Dynamic Salient Regions Graph Neural Network ( DyReg-GNN ) improves the relational processing of videos by learning to discover salient regions that are relevant for the current scene and task . Note that the model learns to predict regions only from the weak supervision given by the high-level video classification loss , without supervision at the region level . Our experiments convincingly show that the regions discovered are well correlated with the objects present in the video , confirming the intuition that action recognition should be strongly related to salient region discovery . The capacity to discover such regions makes DyReg-GNN an excellent candidate model for tackling tasks requiring spatio-temporal reasoning . Our main contributions are summarised as follow : 1 . We propose a novel method to augment spatio-temporal GNNs by an additional capability : that of learning to create localized nodes suited for spatial reasoning , that adapt to the input . 2 . The salient regions discovery enhance the relational processing for high-level video classification tasks : creating GNN nodes from predicted regions obtains superior performance compared to both using pre-trained object detectors or fixed regions 3 . Our model leads to unsupervised salient regions discovery , a novelty in the realm of GNNs : it predicts such regions in videos , with only weak supervision at the video class level . We show that regions discovered are well correlated with actual physical object instances . 2 Related work . Graph Neural Networks in Vision . GNNs have been recently used in many domains where the data has a non-uniform structure [ 10 , 11 , 12 , 13 ] . In vision tasks , it is important to model the relations between different entities appearing in the scene [ 14 , 15 ] and GNNs have strong inductive biases towards relations [ 16 , 17 ] , thus they are perfectly suited for modeling interactions between visual instances . Since an explicit structure is not available in the video , it is of critical importance to establish what atomic elements should be represented as graph nodes . As our main contribution revolves around the creation of nodes , we analyse other recent GNN methods regarding the type of information that each node represents , and group them into two categories , semantic and spatial . The approaches of [ 4 , 18 , 19 , 20 , 21 , 22 ] capture the purely semantic interactions by reasoning over global graph nodes , each one receiving information from all the points in the input , regardless of spatio-temporal position . In [ 4 ] the nodes assignments are predicted from the input , while in [ 18 ] the associations between input and nodes are made by a soft clusterization . The work of [ 22 ] discovers different representation groups by using an iterative clusterization based on self-attention similarity . The downside of these semantic approaches is that individual instances , especially those belonging to the same category , are not distinguished in the graph processing . This information is essential in tasks such as capturing human-object interactions , instance segmentation or tracking . Alternatively , multiple methods , including ours , favour modeling instance interactions by defining spatial nodes associated with certain locations . We distinguish between them by how they extract the nodes from spatial location : as fixed regions or points [ 23 , 24 ] , or detected object boxes [ 25 , 26 , 27 , 28 , 29 ] . The method [ 5 ] creates nodes from every point in 2D convolutional features maps , while Non-Local [ 30 ] uses self-attention [ 31 ] between all spatio-temporal positions to capture distant interactions . Further , [ 3 ] extract nodes from larger fixed regions at different scales and processes them recurrently . Recent methods based on Transformer [ 32 , 6 , 33 ] also model the interactions between fixed locations on a grid using self-attention . In [ 7 ] , nodes are created from object boxes extracted by an external detector and are processed using two different graph structures , one given by location and one given by nodes similarity . A related approach is used in [ 27 ] in a streaming setting while [ 34 ] learns to hop over unnecessary frames . Hybrid approaches use nodes corresponding to points and object features [ 35 , 36 ] or propagate over both semantic and spatial nodes [ 37 , 38 , 39 ] . However , methods that rely on external modules trained on additional data , such as object detectors , are too dependent on the module ’ s performance . They are unable to adapt to the current problem , being limited to the set of pre-defined annotations designed for another task . Differently , our module is optimized to discover regions useful for the current task , using only the video classification signal . Recently , the method [ 40 ] uses multiple position-aware nodes that take into account the spatial structure . This makes it more suitable for capturing instances , but the nodes have associated a static learned location , where each one is biased towards a specific position regardless of the input . On the other hand , we dynamically assign a location for each node , based on the input , making the method more flexible to adapt to new scenes . Dynamic Networks . Several works use second-order computations by dynamically predicting different parts of their model from the input , instead of directly optimising parameters . Our work is related to STN [ 41 ] that aggregates features by interpolating from an area given by a predicted global transformation and to the differentiable pooling used in some object detectors [ 42 , 43 , 44 ] . The method [ 45 ] , replaces the parameters in a standard convolution with weights predicted from the input , resulting in a dynamically generated filter . Deformable convolutions [ 46 , 47 ] predict , based on the input , an offset for each position in the convolutional kernel . Similar , [ 48 ] use the same idea of predicting offsets but in a graph formulation . The common topic of these methods is to predict dynamically a support for all points in a convolutional operation while we dynamically generate the input for a set of nodes designed to process high-level interactions . Related ideas , involving high-level processing of a small set of powerful modules , is also highlighted in [ 49 ] and [ 40 ] . Unsupervised Object Representations . There is an entire area of work devoted to extracting representations centered on objects [ 50 ] in a fully unsupervised setting [ 51 , 52 , 53 , 22 ] . They are successful in leveraging a reconstruction task to decompose the scene into objects , for synthetic images . In [ 54 ] it is shown that representations learned from unsupervised decomposition are also helpful in relational reasoning tasks . Methods for generating unsupervised keypoints or entities [ 55 , 56 , 57 , 58 ] have been generally used in synthetic setting . The method [ 55 ] generates keypoints from real images of people and faces but they use an image reconstruction objective that could not be aligned with the downsteam task . Our goal is to relate spatio-temporal entities , but without enforcing a clear decomposition of the scene into objects . This allows us to use a simpler but effective method that learns from classification supervision of real-world videos and obtain representations that are correlated to objects . Activity Recognition . Video classification has been influenced by methods designed for 2D images [ 59 , 60 , 61 , 62 ] . More powerful 3D convolutional networks have been later proposed [ 63 ] , while other methods factorise the 3D convolutions [ 64 , 65 , 66 ] bringing both computational speed and accuracy . Methods like TSM [ 67 ] and [ 68 ] showed that a simple shift in the convolutional features results in improved accuracy at a low computational budget . 3 Dynamic Salient Regions GNNs . We investigate how to create node representations that are useful for modeling visual interactions between various entities in space and time using GNNs . Our proposed Dynamic Salient Regions GNN model ( DyReg-GNN ) learns to dynamically assign each node to a certain interesting region . By dynamic , we mean that we have a fixed number N of regions that change their position and size according to the input at each time step . The regions assigned to each of the N nodes can change from one moment of time to the next depending on their saliency . The main architecture of our DyReg-GNN model is illustrated in Figure 1 . Our model receives feature volume X ∈ RT×H×W×C and at each time step t we predict the location and size of N regions . From these regions , a differentiable pooling operation creates graph nodes that are processed by a GNN and then are projected to their initial position . This module can be inserted at any intermediate level in a standard convolutional model . 3.1 Node Region Generation . We want to attend only to a few most relevant entities in the scene , thus a small number of nodes are used in DyReg-GNN ( in our experiments N = 9 ) and it is crucial to assign them to the most salient regions . The number of nodes is a hyperparameter that we choose such that it exceeds the expected number of relevant entities in the scene , to increase the robustness of the model . Thus , we propose a global processing ( shown in Figure 1 B ) that aggregates the entire input features to produce regions defined by parameters indicating their location ( ∆x , ∆y ) and size ( w , h ) . To generate N salient regions , we process the input Xt using position-aware functions f and { gi } i∈1 , N that retain spatial information . Nodes should be consistent across time , thus we generate their regions in the same way at all time steps , by sharing in time the parameters of f and { gi } . The function f is a convolutional network that highlights the important regions from the input . Mt = f ( Xt ) ∈ RH ′×W ′×C′ ( 1 ) For each node i , we generate a latent representation of its associated region using the { gi } functions . Each gi has the same architecture , but different parameters for each node and could be instantiated as a fully connected network or as global pooling enriched with spatial positional information . We generate the node regions from a global view to make the decision as informed as possible . m̂i , t = gi ( Mt ) ∈ RC ′ , ∀i ∈ 1 , N ( 2 ) Each of the N latent representations is processed independently , with a GRU [ 69 ] recurrent network ( shared between nodes ) , to take into account the past regions ’ representations . zi , t = GRU ( zi , t−1 , m̂i , t ) ∈ RC ′ , ∀i ∈ 1 , N ( 3 ) At each time step , the final parameters are obtained by a linear projection Wo ∈ RC ′×4 , transformed by a function α to control the initialisation of the position and size ( e.g . regions would start at reference points either in the center of the frame or arranged on a grid ) . For more details about how to set the transformation α we refer to the Supplemental Materials . oi , t = ( ∆xi , t , ∆yi , t , wi , t , hi , t ) = α ( Wozi , t ) ∈ R4 ( 4 )

Summary: The paper proposes a novel approach for generating dynamic visual nodes, ideally capturing salient visual regions, which can then be fed to a spatio-temporal Graph Neural Network in order to model intra-region relationships. In contrast to prior work, which uses feature map columns in a grid or object detector bounding boxes as the input visual nodes to a GNN, the proposed DyReg-GNN learns to localize a fixed number of visual nodes from a video, without using object detectors or assuming spatial annotations.  To achieve this, the authors propose using node-specific trainable neural networks, which output region locations and sizes based on the input spatio-temporal feature map. Then spatio-temporal features are differentiably-pooled from each one of these regions, are fed to a standard GNN, are refined based on interactions, and finally are mapped back to their original spatio-temporal positions to yield a refined spatial feature map. Therefore, the DyReg-GNN can be inserted at any intermediate level in a standard convolutional model.  Results show that the proposed approach significantly outperforms GNNs that are applied on a fixed grid on a synthetic dataset (MultiSync MNIST), is much more efficient than methods that use object detectors, and the generated regions are qualitatively shown to correlate with objects, change size over time, and even retain object identities over time. The method is also evaluated on Sth-Sth V1, where it leads to a competitive performance, although the improvement over the grid baseline is much smaller.