Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel approach for generating dynamic visual nodes, ideally capturing salient visual regions, which can then be fed to a spatio-temporal Graph Neural Network in order to model intra-region relationships. In contrast to prior work, which uses feature map columns in a grid or object detector bounding boxes as the input visual nodes to a GNN, the proposed DyReg-GNN learns to localize a fixed number of visual nodes from a video, without using object detectors or assuming spatial annotations.

To achieve this, the authors propose using node-specific trainable neural networks, which output region locations and sizes based on the input spatio-temporal feature map. Then spatio-temporal features are differentiably-pooled from each one of these regions, are fed to a standard GNN, are refined based on interactions, and finally are mapped back to their original spatio-temporal positions to yield a refined spatial feature map. Therefore, the DyReg-GNN can be inserted at any intermediate level in a standard convolutional model.

Results show that the proposed approach significantly outperforms GNNs that are applied on a fixed grid on a synthetic dataset (MultiSync MNIST), is much more efficient than methods that use object detectors, and the generated regions are qualitatively shown to correlate with objects, change size over time, and even retain object identities over time. The method is also evaluated on Sth-Sth V1, where it leads to a competitive performance, although the improvement over the grid baseline is much smaller.
main_review: Strengths
=======

S1) Well-motivated approach: The paper addresses a crucial limitation of spatio-temporal GNN approaches applied to video, namely the need for object detectors in order to generate candidate salient regions. This is not only computationally expensive, but is also limited by the proposal recall of object detectors. The proposed approach aims to remove the need for this step, by automatically learning to localize a fixed number of salient regions in each video frame, without spatial annotations. 

S2) Original framework: The approach is very nicely positioned w.r.t the prior work and is exploring a novel direction, that of automatically generating salient regions beyond grids. It cleverly leverages networks that output location and size, differentiable pooling ideas (such as the ones in Spatial Transformer Networks[39]), and spatio-temporal GNNs[3].

S3) Well-designed ablations in a synthetic dataset: Except for tackling a novel direction, another strong point of the paper are the experiments on the MultiSyncMNIST dataset. These controlled experiments clearly demonstrate: a) that using the discovered region nodes instead of a fixed grid improves video classification (Table 5), b) using the discovered, visually grounded, region nodes is better than finding latent semantic nodes and applying the GNN on them (Table 6), c) generated nodes tend to follow the digits locations, while also adapting region's size.

S4) The paper is very well-written, is easy to read and equations are properly explained. The approach also seems technically sound.

S5) The authors have provided rich qualitative examples, videos, implementation details, and code. A lot of limitations are also discussed.

Weaknesses
==========
W1) Choice of benchmark dataset (Something-Something): I think that the proposed method would be useful in real-life applications if it could significantly improve upon a fixed grid approach in real datasets, without the added overhead of an object detector. I appreciate that the authors compared these 3 settings. However, I feel that the chosen real-life dataset is too limited in terms of the richness of scenes. This does not only limit the expected impact from using salient regions instead of grids, but might also be an easy dataset for the unsupervised region discovery.For example, most scenes have only a few objects and a single person (hands) interacting with them. Therefore, although action recognition in SthSth is indeed a challenging spatio-temporal reasoning task, finding salient regions is not as hard, since most of the foreground blobs are relevant. We can see that from the experimental results as well. In Table 1, we observe that using a fixed grid vs an object detector vs the proposed salient nodes yields an accuracy of 64.1 vs 64.0 vs 64.8. The fact that object detections do not improve over the grid might be because the grid already captures all the relevant objects and there are not many irrelevant grid cells.

W2) Missing ablations: Was there an ablation about not using the GRU when generating nodes over time (Eq. 3)? Also, what is the impact of the attention component in Eq. 8? Could it be that grid-based GNNs without attention would be much worse than the proposed DyReg-GNN? Did the authors experiment with different number of GNN layers or different loss scalar weight \lambda?

W3) Evaluation of alignment of generated regions with ground-truth objects: Why (only) use the L2 distance for comparing regions with ground-truth object regions, and not IoU as well (Table 1)? Also, an additional reported metric could be the number of salient regions covering each object instance. The authors acknowledged this limitation, namely that because of the fixed number of nodes, a lot of the generated regions can focus on the same object. Object recall is also an important metric. It would be also interesting to have a quantitative evaluation on whether the model learns some form of tracking, although it is not the primary goal.

W4) Comparison with SOTA: The approach improves marginally improved results compared to RSTG (same? GNN applied on feature grids) - 49.2 vs 49.9%. Also, the method is not state-of-the-art when compared to non-graph methods that use the same backbone and number of frames.  Question: does the RSTG[3] also use the TSM module?

Minor comments
============
DC1) The authors could also cite: Dynamic Graph Modules for. Modeling Object-Object Interactions in. Activity Recognition, BMCV19 , which shares the same ideas of having a latent, location graph, that is visually grounded, although that method also uses object detections for the visual graph.
DC2) In the state-of-the-art tables, it would be useful to add a column regarding the nodes used by the compared GNNs (grid vs object vs dynamic).
DC3) Since the node position generation networks are 9 fixed networks, was there any qualitative analysis on what types of regions they generate? For example, is there any correlation between the region generated by the first node generation network for similar scenes? Is there any cluster of objects that arises?

Rating Justification
===============
Overall, the paper proposes a novel approach for spatio-temporal interaction modeling in videos, which tackles the difficult task of dynamically learning salient regions without spatial annotations. The approach is well-motivated, thoughtfully designed and experimental results on synthetic data validate its effectiveness. I am still skeptical whether the approach would be feasible in video datasets with richer scenes, such as Charades or cooking datasets, where object detection is itself more challenging than SthSth. In such datasets, not only the small fixed number of nodes could be an issue, but also training could be harder since the model is ideally trying to capture object detection and object tracking without annotations.  Still, I think the work in its current form opens up an interesting direction in the field and is showing promising results.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces a graph-based model for salient region discovery in videos, called Spatio-Temporal Graph Neural Networks (DyReg-GNN). The key idea is to capture latent interactions between various entities by graph; moreover, RNN is used to capture inter-flame information. Generally, the idea is reasonable and the experiments show the effectiveness of the proposed approach.

main_review: + The idea of using graph-based techniques for capturing inter-entities is reasonable; 
+ The paper describes each component clearly.
limitations_and_societal_impact: - It is unclear to me why the proposed method is unsupervised. It seems that the proposed approach requires video-class level supervision for learning parameters; 

- The graph net/CNNs for better feature representation learning has been studied by many other research work. Even though the paper is the first to introduce it for salient region discovery, the contributions of the paper are still limited as a NeurIPS paper.

- The experiments show that the proposed approach achieves competitive results; yet, it dose not outperform SOTA approaches such as MSNet.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes to improve spatio-temporal graph neural networks by adaptively computing the input node features from spatially localized attention regions. The attention regions are dynamically predicted from each input frame, which is in contrast to previous methods that use fixed regions or pretrained object detectors. It is shown that by only optimizing a video classification loss, the model can learn to attend to objects, thereby extracting object-centric node features and facilitating interaction modeling. On the Something-Something-V1&V2 datasets, the model achieves comparable or slightly better classification accuracy than previous work, and can consistently boost performance when combined with multiple backbones.
main_review: It is an interesting idea to let the nodes of a graph net dynamically attend to relevant input regions. This has the potential for better interpretability and interaction modeling. However, similar ideas have been proposed in previous methods like OP3[1] and V-CDN[2], where they predict object masks / keypoints and use graph nets to model dynamics, all without supervision. Compared to these methods, the strength of this paper seems to be that it can work on real-world datasets like Something-Something-V1&V2, when given the video class labels. However, I had some concern about the experiments:

- The main metric considered is the classification accuracy, on which using dynamically adapted regions only slightly outperforms using fixed regions (Table 1). Also, in Appendix Figure 5, it seems that each node covers a specific region. These results lead to the question: Is it necessary to use dynamic regions?
- For object-centric representations to benefit interaction modeling, it is best if the representations are temporally consistent, i.e., same node capturing same object across time. However, I did not see much evalution on this temporal consistency aspect, and Figure 5 seems to suggest that if an object moves a long distance, it will be captured by different nodes.
- I was a bit confused about the visualization of node regions in Figures 3 and 4. In particular, the number of regions in Figure 4 is much smaller than the number of nodes, so what nodes are visualized? Also the learned position of nodes looks similar to keypoints [2-4], which can be predicted without supervision even for realistic images. I think it would be better to have some discussion about these methods.

**MINOR COMMENTS**

- What is the benefit of the proposed kernel function compared to spatial transformer networks?

[1] Entity Abstraction in Visual Model-Based Reinforcement Learning (CoRL 2019).

[2] Causal Discovery in Physical Systems from Videos (NeurIPS 2020).

[3] Unsupervised Learning of Object Landmarks through Conditional Image Generation (NeurIPS 2018).

[4] Unsupervised Learning of Object Keypoints for Perception and Control (NeurIPS 2019).
limitations_and_societal_impact: In addition to the limitations mentioned in the paper, I think the proposed model may not have good temporal consistency when objects move long distances, and thus the improvement in interaction modeling can be limited in long videos.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a graph based network for spatio-temporal understanding. The core idea of the paper is to learn to select visual features for node representations in graph dynamically using convolutional encoders. The paper shows competitive results in the Smth-Smth v1 and v2 datasets and MultiSync MNIST. The contributions of the paper are demonstrating a method about how to learn to select dynamic regions and making it work in the context of graph neural networks.
main_review: The paper addresses an important problem of spatio-temporal understanding. It is important to investigate approaches on understanding this modality more deeply (e.g. via reasoning) rather than superficially building deeper networks and this paper makes a step in the right direction.

I have the following comments and questions about the paper:

1) What are the benefit of using dynamic conv features apart from performance scores? The benefit of using static nodes (e.g. using objects) is that there is increased model interpretability and the model learns to establish between static entities in the dataset. Using convolutional features that change every time-step based on output from conv layers is not very interpretable. It may increase the performance of your model, since the model learns to attend to larger number of foreground and background features. However, this benefit is also seen with object detectors where ROIs are used [A].

2) There is criticism of datasets in Section 4, some of which is valid. Kinetics does have a high spatial bias but does contain fine-grained classes (e.g. around basketball) that are far from being solved and it would be interesting to see how your model can separate actions for classes that involve similar objects (or salient regions). Similarly, the size of CATER was limited by design. Also, [68] does release an updated dataset that does require longer term reasoning and maybe worth looking into. CATER is perhaps a great fit for the central idea of the paper as the attention region should dynamically move from snitch to cones (in Task 3) as the video progresses.

3) In the contributions, you mention that the paper “enhances relational processing”, it is not clear that the relational processing enhaced i.e. better higher order relations are established. My intuition is that by regressing and attending to a large number of region features the model is better regularized as well as gets additional signals that a pre-trained object detector misses.

4) How did you pick 9 regions/objects? Perhaps to make the system completely dynamic, this can be a learned parameter as well.

After reviewing rebuttal responses from authors to my review and other reviews, I decided to change my score to Accept.

[A] Attend and Interact: Higher-Order Object Interactions for Video Understanding https://arxiv.org/abs/1711.06330
limitations_and_societal_impact: The paper adequately addresses the social impact.
review_title: null

Meta-review: This paper received both positive and negative reviews. After a round of discussion between the reviewers and reading the author's rebuttal, three of the four reviewers recommended accepting the paper and I am happy to accept it. I encourage the authors to include in the final version the additional material that they mentioned in the rebuttal. 
Decision: Accept (Poster)