Discovering Dynamic Salient Regions for Spatio-Temporal Graph Neural Networks

1 Introduction . Spatio-temporal data , and videos , in particular , are characterised by an abundance of events that require complex reasoning to be understood . In such data , entities or classes exist at multiple scales and in different contexts in space and time , starting from lower-level physical objects , which are well localized in space and moving towards higher-level concepts which define complex interactions . We need a representation that captures such spatio-temporal interactions at different level of granularity , depending on the current scene and the requirements of the task . Classical convolutional nets address spatio-temporal processing in a simple and rigid manner , determined only by fixed local receptive fields [ 1 ] . Alternatively , space-time graph neural nets [ 2 , 3 ] offer a more powerful and flexible approach modeling complex short and long-range interactions between visual entities . In this paper , we propose a novel method to enhance vision Graph Neural Networks ( GNNs ) by an additional capability , missing from any other previous works . That is , to have nodes that are constructed for spatial reasoning and can adapt to the current input . Prior works are limited to having either nodes attached to semantic attention maps [ 4 ] or attached to fixed locations such as grids [ 5 , 3 , 6 ] . Moreover , unlike works that require external object detectors [ 7 ] our method relies on a learnable mechanism to adapt to the current input . ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose a method that learns to discover salient regions , well-delimited in space and time , that are useful for modeling interactions between various entities . Such entities could be single objects , parts or groups of objects that perform together a simple action . Each node learns to associate by itself to such salient regions , thus the message passing between nodes is able to model object interactions more effectively . For humans , representing objects is a core knowledge system [ 8 ] and to emphasize them in our model , we predict salient regions [ 9 ] that give a strong inductive bias towards modeling them . Our method , Dynamic Salient Regions Graph Neural Network ( DyReg-GNN ) improves the relational processing of videos by learning to discover salient regions that are relevant for the current scene and task . Note that the model learns to predict regions only from the weak supervision given by the high-level video classification loss , without supervision at the region level . Our experiments convincingly show that the regions discovered are well correlated with the objects present in the video , confirming the intuition that action recognition should be strongly related to salient region discovery . The capacity to discover such regions makes DyReg-GNN an excellent candidate model for tackling tasks requiring spatio-temporal reasoning . Our main contributions are summarised as follow : 1 . We propose a novel method to augment spatio-temporal GNNs by an additional capability : that of learning to create localized nodes suited for spatial reasoning , that adapt to the input . 2 . The salient regions discovery enhance the relational processing for high-level video classification tasks : creating GNN nodes from predicted regions obtains superior performance compared to both using pre-trained object detectors or fixed regions 3 . Our model leads to unsupervised salient regions discovery , a novelty in the realm of GNNs : it predicts such regions in videos , with only weak supervision at the video class level . We show that regions discovered are well correlated with actual physical object instances . 2 Related work . Graph Neural Networks in Vision . GNNs have been recently used in many domains where the data has a non-uniform structure [ 10 , 11 , 12 , 13 ] . In vision tasks , it is important to model the relations between different entities appearing in the scene [ 14 , 15 ] and GNNs have strong inductive biases towards relations [ 16 , 17 ] , thus they are perfectly suited for modeling interactions between visual instances . Since an explicit structure is not available in the video , it is of critical importance to establish what atomic elements should be represented as graph nodes . As our main contribution revolves around the creation of nodes , we analyse other recent GNN methods regarding the type of information that each node represents , and group them into two categories , semantic and spatial . The approaches of [ 4 , 18 , 19 , 20 , 21 , 22 ] capture the purely semantic interactions by reasoning over global graph nodes , each one receiving information from all the points in the input , regardless of spatio-temporal position . In [ 4 ] the nodes assignments are predicted from the input , while in [ 18 ] the associations between input and nodes are made by a soft clusterization . The work of [ 22 ] discovers different representation groups by using an iterative clusterization based on self-attention similarity . The downside of these semantic approaches is that individual instances , especially those belonging to the same category , are not distinguished in the graph processing . This information is essential in tasks such as capturing human-object interactions , instance segmentation or tracking . Alternatively , multiple methods , including ours , favour modeling instance interactions by defining spatial nodes associated with certain locations . We distinguish between them by how they extract the nodes from spatial location : as fixed regions or points [ 23 , 24 ] , or detected object boxes [ 25 , 26 , 27 , 28 , 29 ] . The method [ 5 ] creates nodes from every point in 2D convolutional features maps , while Non-Local [ 30 ] uses self-attention [ 31 ] between all spatio-temporal positions to capture distant interactions . Further , [ 3 ] extract nodes from larger fixed regions at different scales and processes them recurrently . Recent methods based on Transformer [ 32 , 6 , 33 ] also model the interactions between fixed locations on a grid using self-attention . In [ 7 ] , nodes are created from object boxes extracted by an external detector and are processed using two different graph structures , one given by location and one given by nodes similarity . A related approach is used in [ 27 ] in a streaming setting while [ 34 ] learns to hop over unnecessary frames . Hybrid approaches use nodes corresponding to points and object features [ 35 , 36 ] or propagate over both semantic and spatial nodes [ 37 , 38 , 39 ] . However , methods that rely on external modules trained on additional data , such as object detectors , are too dependent on the module ’ s performance . They are unable to adapt to the current problem , being limited to the set of pre-defined annotations designed for another task . Differently , our module is optimized to discover regions useful for the current task , using only the video classification signal . Recently , the method [ 40 ] uses multiple position-aware nodes that take into account the spatial structure . This makes it more suitable for capturing instances , but the nodes have associated a static learned location , where each one is biased towards a specific position regardless of the input . On the other hand , we dynamically assign a location for each node , based on the input , making the method more flexible to adapt to new scenes . Dynamic Networks . Several works use second-order computations by dynamically predicting different parts of their model from the input , instead of directly optimising parameters . Our work is related to STN [ 41 ] that aggregates features by interpolating from an area given by a predicted global transformation and to the differentiable pooling used in some object detectors [ 42 , 43 , 44 ] . The method [ 45 ] , replaces the parameters in a standard convolution with weights predicted from the input , resulting in a dynamically generated filter . Deformable convolutions [ 46 , 47 ] predict , based on the input , an offset for each position in the convolutional kernel . Similar , [ 48 ] use the same idea of predicting offsets but in a graph formulation . The common topic of these methods is to predict dynamically a support for all points in a convolutional operation while we dynamically generate the input for a set of nodes designed to process high-level interactions . Related ideas , involving high-level processing of a small set of powerful modules , is also highlighted in [ 49 ] and [ 40 ] . Unsupervised Object Representations . There is an entire area of work devoted to extracting representations centered on objects [ 50 ] in a fully unsupervised setting [ 51 , 52 , 53 , 22 ] . They are successful in leveraging a reconstruction task to decompose the scene into objects , for synthetic images . In [ 54 ] it is shown that representations learned from unsupervised decomposition are also helpful in relational reasoning tasks . Methods for generating unsupervised keypoints or entities [ 55 , 56 , 57 , 58 ] have been generally used in synthetic setting . The method [ 55 ] generates keypoints from real images of people and faces but they use an image reconstruction objective that could not be aligned with the downsteam task . Our goal is to relate spatio-temporal entities , but without enforcing a clear decomposition of the scene into objects . This allows us to use a simpler but effective method that learns from classification supervision of real-world videos and obtain representations that are correlated to objects . Activity Recognition . Video classification has been influenced by methods designed for 2D images [ 59 , 60 , 61 , 62 ] . More powerful 3D convolutional networks have been later proposed [ 63 ] , while other methods factorise the 3D convolutions [ 64 , 65 , 66 ] bringing both computational speed and accuracy . Methods like TSM [ 67 ] and [ 68 ] showed that a simple shift in the convolutional features results in improved accuracy at a low computational budget . 3 Dynamic Salient Regions GNNs . We investigate how to create node representations that are useful for modeling visual interactions between various entities in space and time using GNNs . Our proposed Dynamic Salient Regions GNN model ( DyReg-GNN ) learns to dynamically assign each node to a certain interesting region . By dynamic , we mean that we have a fixed number N of regions that change their position and size according to the input at each time step . The regions assigned to each of the N nodes can change from one moment of time to the next depending on their saliency . The main architecture of our DyReg-GNN model is illustrated in Figure 1 . Our model receives feature volume X ∈ RT×H×W×C and at each time step t we predict the location and size of N regions . From these regions , a differentiable pooling operation creates graph nodes that are processed by a GNN and then are projected to their initial position . This module can be inserted at any intermediate level in a standard convolutional model . 3.1 Node Region Generation . We want to attend only to a few most relevant entities in the scene , thus a small number of nodes are used in DyReg-GNN ( in our experiments N = 9 ) and it is crucial to assign them to the most salient regions . The number of nodes is a hyperparameter that we choose such that it exceeds the expected number of relevant entities in the scene , to increase the robustness of the model . Thus , we propose a global processing ( shown in Figure 1 B ) that aggregates the entire input features to produce regions defined by parameters indicating their location ( ∆x , ∆y ) and size ( w , h ) . To generate N salient regions , we process the input Xt using position-aware functions f and { gi } i∈1 , N that retain spatial information . Nodes should be consistent across time , thus we generate their regions in the same way at all time steps , by sharing in time the parameters of f and { gi } . The function f is a convolutional network that highlights the important regions from the input . Mt = f ( Xt ) ∈ RH ′×W ′×C′ ( 1 ) For each node i , we generate a latent representation of its associated region using the { gi } functions . Each gi has the same architecture , but different parameters for each node and could be instantiated as a fully connected network or as global pooling enriched with spatial positional information . We generate the node regions from a global view to make the decision as informed as possible . m̂i , t = gi ( Mt ) ∈ RC ′ , ∀i ∈ 1 , N ( 2 ) Each of the N latent representations is processed independently , with a GRU [ 69 ] recurrent network ( shared between nodes ) , to take into account the past regions ’ representations . zi , t = GRU ( zi , t−1 , m̂i , t ) ∈ RC ′ , ∀i ∈ 1 , N ( 3 ) At each time step , the final parameters are obtained by a linear projection Wo ∈ RC ′×4 , transformed by a function α to control the initialisation of the position and size ( e.g . regions would start at reference points either in the center of the frame or arranged on a grid ) . For more details about how to set the transformation α we refer to the Supplemental Materials . oi , t = ( ∆xi , t , ∆yi , t , wi , t , hi , t ) = α ( Wozi , t ) ∈ R4 ( 4 )

Summary: This paper introduces a graph-based model for salient region discovery in videos, called Spatio-Temporal Graph Neural Networks (DyReg-GNN). The key idea is to capture latent interactions between various entities by graph; moreover, RNN is used to capture inter-flame information. Generally, the idea is reasonable and the experiments show the effectiveness of the proposed approach.