Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a framework for video action anticipation, where the task is to observe input frames and predict the action label after an anticipation period (not observed). The proposed architecture builds on top of message passing terminology (message, update, and readout functions) to create a recurrent transformer model that uses self-attention to capture spatial dependencies between frame patches (similar to ViT w/o [cls]) and recurrent module to capture the temporal dependencies across frames. Edge affinity matrix is learned and introduced into the MHSA block (Eqn. 12) as a prior to influence the attention matrix at every layer. Three different explicit edge learning approaches are proposed, in addition to implicit edge learning. The model is evaluated on the Epic Kitchen 55 dataset.
main_review: Strengths:
		
a) The proposed approach provides a transformer-based method to encode Spatio-temporal features from a graph or RGB inputs. Adding the edge affinity matrix to the attention map allows the model to make use of edge affinity information in a self-attention encoder block.

b) The task of video action anticipation can arguably aid in learning more robust and transferable representations of inputs by forcing the model to predict further away actions.

c) Unlike other methods (Furnari et al.), this method does not require optical flow or object detections as inputs 
The paper is well written, and claims are supported by empirical results (Tables 1,2, and 3).

Weaknesses:

a) The authors present the architecture using GNN terminology (message, update, readout functions). Generally, GNNs are useful in the following scenarios where:

1) The input graphs are not fully connected, so the adjacency (or affinity) matrix introduces an inductive bias on the data forcing the nodes to pass messages in a certain manner, and inhibit message passing for disconnected nodes.

2) The input graphs are fully connected, but affinity matrix (or edge embeddings) influence the message passing function.

3) Permutation invariance is needed.

In the proposed approach, the authors use a transformer architecture to process the image patches. The vertices (image patches) of the graph are flattened into a 1D transformer sequence. These vertices are updated with every new observation using a recurrent model (similar to LSTM) where the input, update, and output functions (gates) are implemented with self-attention blocks. The input data contains no edge information between image patches. And positional encoding is added to the input, allowing the network to be aware of the positional information of patches. It becomes hard to convince the readers that GNNs terminology and edge learning is necessary unless it is well supported by empirical results. However, the results of explicit edge learning do not outperform implicit edge learning through attention, because the attention operation is capable of capturing the spatial dependencies between image patches (as implied by the authors). Introducing the edge affinity matrix into the attention block will only be useful if it is part of the input; it can be used as a prior to influence the attention maps. This framework can be more useful for different tasks with edges defined as part of the input.

b) More empirical results can be provided to discuss Tail classes (similar to table 3 in AVT paper) and more results on EK100 or a different dataset. Providing results on multiple datasets helps prove the efficacy of the approach in different settings (egocentric vs. third-person, indoor vs. outdoor, etc.). 

c) The claim that edge learning with template bank achieved the best results is not well supported by quantitative results. Ablations on the main architecture (instead of template bank size) would be more useful, i.e. different backbones, removing squeeze and excitation, etc. Ideally, this can be fixed with an additional table showing the Top-1/Top-5 performance with different backbone variations and initializations. This will allow the reader to understand which part of the main architecture causes the model to perform better than the previous approaches.

d) The authors claim to surpass current SOTA approaches but based on the reported results, improvements are not significant (and not consistent).


Minor Issues:

a) Please provide an explanation for how the output of Eqn. 18 can be inserted in Eqn. 19. The output of SABlock is an (Nxd) tensor and we need an NxN attention matrix in SABlock_v. One option is to extract the attention from SABlock_e before multiplying by the Value matrix but it is not clear in the text.

b) It would be a good idea to create a separate section for the decoder (unrolling classifier) and the loss function instead of briefly mentioning them in the implementation details. The background section can be shortened to save space.

c) Some implementation details are missing, such as:
1) Is the backbone frozen or just pretrained?
2) How many blocks of SA are used? just one for every function?
3) What is the number of patches (HW) used as input to the transformer?
4) What initialization is used for node states, template bank, etc.?
5) Any data augmentation applied to the dataset? horizontal flipping seems like a good fit for this data.
6) What is the function f_{scale} scaling the values of h_{v}^{t} to?
7) Minor typos are present (i.e., Section 4.1:L2)


Ideas that are not required for publication, but would make the paper stronger:

a) It would be an interesting idea to test the learned features on the action recognition task. Using this architecture as a pretrained encoder for an action recognition task may outperform other approaches with naive initializations. The would be a very strong section in the evaluation.

b) Other papers (Furnari et al.) use object detections and optical flow features as part of the input. It would be interesting to see the effect of including these features as input to the model.

summary_of_the_review: The authors propose an interesting architecture for encoding RGB inputs and predicting actions after the anticipation period. The method has the capacity to encode edge information of graphs in the transformer model, but the type of data/task does not make use of this advantage. Other approaches (AVT) provide very similar solutions by using a temporal transformer, instead of a recurrent model, to capture the temporal dependencies. The GNN analogy seems weak and distracts the reader from the main approach (recurrent transformer). It can be fixed by simply removing the GNN and edge related sections, this will save space for additional ablations and quantitative results on other datasets.

The claim that this approach surpasses the SOTA approaches is weakly supported by quantitative results. The slight overall increase in performance cannot be attributed to explicit edge learning (as can be seen in the results). It is unclear which component(s) of the proposed architecture is the cause; it could be the added squeeze and excitation block, or bypassing the bottleneck representation (by not using a [CLS] token). The paper is not ready for publication in its current state; however, additional ablation studies, experiments on additional datasets, and removing unnecessary sections can significantly increase the paper's chances for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Paper proposes a novel architecture for video action anticipation task. The proposed method used a graph representation via message passing. The key contribution lies in the three explicitly edge learning strategies. These strategies to escape to the trivial estimation which is  purely based on the similarity in input tensors, by bringing the flexibility of edge representations. 

1. Edge attention decouples the attention operator in message function into vertex and edge estimation separately.
2. Class token projection performs the outer-product of a trainable vector with supervision signal from class labels.
3. Template bank obtains the edge matrix by the linear combination of trainable templates, based on a selecting module conditions on inputs.

Experiment on the EK55 dataset outperforms the state-of-the-arts algorithms.
main_review: Strengths
1. Novel approach which solves an important vision task: video action anticipation.
2. Method is clearly and methodically described.
3. Experimental results are strong on the challenging EK55 dataset. The approach is compared against many state-of-the-arts algorithms.

Weaknesses
1. Paper's approach is focus on the specific task of video action anticipation task. While it is an important task in vision, it is not clear if it is of interests to the wider audience in the ICLR community.
2. Minor. Paper's main technical contributions are specific to the task and may not generalise to other problems.
summary_of_the_review: I recommend to accept this paper as the technical contributions are quite strong. However, its appeal to the wider audience in ICLR is a minor concern.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper addressed an interesting problem of action anticipation. To this end, the authors proposed a unified recurrence model that generates the graph representation of the video sequence. Extensive experimental results have shown the effectiveness of the proposed method.
main_review: Strength:

+The message passing method for learning graph representation under the video anticipation setting is novel to my knowledge

+The idea of using self-attention learning the correlation between vertices and further building the graph representation is interesting.

+Different edge learning methods are proposed and properly evaluated in the ablation study

+The experiment results seem to be promising

+The paper is overall well-written


Weakness (sorted by priority):

-The major concern is the effectiveness of the proposed three edge learning strategies. According to Table1-3, those 3 strategies have minor improvement on the overall performance.

-The method section is not well-structured. It will be nice to have an overview of how to build the graph of the video sequence with proposed operations

-It would be great to compare with previous methods on the efficiency.

-Some visualization of what has been learned by the self-attention would be interesting, and may potentially improve the paper quality/

-Only results on EPIC-Kitchens are shown. Many baseline methods also conduct experiments on the EGTEA Gaze+ dataset. 

-missing reference:

EGO-TOPO: Environment Affordances from Egocentric Video CVPR 2020

Forecasting Human Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Vision. ECCV2020


summary_of_the_review: The overall idea of using message passing for constructing graph representation for video action anticipation. But the proposed edge learning method has a minor influence on the model performance. Therefore some of the arguments made by the paper are not fully supported.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method for action anticipation based on a graph representation of the temporal structure of video. Specifically a graph is built using semantic representations of video frames extracted with a 2D CNN backbone as vertices. Temporal reasoning is hence achieved modeling spatiotemporal information flow thorugh message passing across vertices. The approach is based on a message function, an unpdate function and a readout function, all based o a multi-head self-attention block. Experiments compare the proposed approach with respect to other approaches suing RGB inputs on EPIC-KITCHENS-55.
main_review: The paper considers a topic which has attracted interest lately. Anticipation models are known to be hard to design and optimize, and learning representations good for anticipation is not trivial, so having discussion about this topic at this conference can be of interest.
As another plus, the proposed approach is compared with several other methods, which is not that common, especially for a task which has been tackled mostly recently. This gives a good basis for other works to compare with the state of the art.

Nevertheless, I think the paper has several flaws would be ideal to address in order to make it suitable for publication in this venue.

1) The main novelty of the paper seems to be the proposal of a method for action anticipation based on a graph representation and message passing. While these kind of approaches have not been much investigated for this problem, there is at least one work which the authors should compare with in order to put the scope of their work in the right context [A]. Apart from this missed comparison, the paper lacks an in-depth discussion on the main relation between the proposed work and the current art. In particular, it would be interesting to answer the following questions: In which ways the proposed approach is different from the current ones? What are the main advantages of the proposed approach? I would suggest revising the paper to make these points more explicit and put the proposed work in the right context, by explaining how it advances our current understanding of the problem at hand.

 2) Another major flaw of the paper is its lack of clarity in several points. I found the description of the method often hard to follow, partially due to the presence of unclear statements. Some of these statements are hard to grasp, while others reduce readability. I would suggest revising and proof readning. Some examples are reported in the following:
 - Page 1: "making the accumulated information gains aligned with the temporal orders" -> what is it meant by this statement? Is there a proof in the paper that this actually happens? What is it meant by aligned? How can it be measured?
 - Page 4: "The message function, update function, and redout funciton are leveraged multi-head self-attention" -> please revise (maybe "leverage multi-head self-attention"?)
 - Page 5: "Given the frame features $x^t$ at time $t$, with shape $(H,W,C)$, which is transformed by a backbone model." -> this sentence is incomplete, please revise
 - Page 5: "In this case where $N(v) \equiv v$ in equation 1." -> this sentence is also incomplete.
 - Page 5: "All edge learning strategies introduced following are all belong to explicit edge estimation category" -> please revise (maybe "introduced in the following all belong...")

3) The paper reports experiments on EPIC-KITCHENS-55, while an extension to the dataset, EPIC-KITCHENS-100 [B] has been introduced. Previous works also reported results on EGTEA-GAZE+ [C]. I think it is hard to draw general conclusions from experimenting on a single dataset, and it would have been interesting seeing experiments on these other datasets as well for completeness.
 
4) While experiments include comparisons with several other approaches. All results are restricted to the case in which only RGB inputs are processed. This is odd, as previous works have been demonstrated to be able to leverage multimodal information. For instance, RULSTM and Action Banks both use RGB, Flow and object inputs. I would have expected the paper to report comparisons with methods taking these inputs as well (I believe these would outperform the results currently reported in the paper). Additionally, it would have been interesting to study whether the proposed approach can work well also on the other representations apart from RGB. This would be natural to assess, as also Flow features can be obtained with a 2D CNN backbone and object features are proposed in RULSTM as 1D vectors, which are inputs compatible with the proposed approach.

[A] E. Dessalene, C. Devaraj, M. Maynord, C. Fermuller and Y. Aloimonos, "Forecasting Action through Contact Representations from First Person Video," in IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2021.3055233.

[B] Damen, Doughty, Farinella, Furnari, Kazakos, Moltisanti, Munro, Price, Wray (2021). Rescaling Egocentric Vision . International Journal on Computer Vision (IJCV), abs/2006.13256 .

[C] Li, Yin, Miao Liu, and James M. Rehg. "In the eye of beholder: Joint learning of gaze and actions in first person video." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
summary_of_the_review: All in all I think the paper tackles an interesting problem, but its presentation lacks clarity, which makes the paper hard to follow at times and difficult to compare to current art. Also, the experimental validation is lacking in some aspects, which should be fixed to make the paper a solid contribution.
review_title: null

Meta-review: This paper presents work on action anticipation.  The reviewers appreciated the message passing based method.  However, concerns were raised regarding novelty, effectiveness, presentation, empirical results, and magnitude of impact for ICLR.  The reviewers considered the authors' response in their subsequent discussions but felt the concerns were not adequately addressed.  Based on this feedback the paper is not yet ready for publication in ICLR.
Decision: Reject