Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an approach to use annotated images from different domain to help with the training of video-based models. The transformer can take both inputs and there are various objectives involved during training for images and videos, respectively. During inference, the only input is video, and the object prompts are used for label prediction.
strengths_and_weaknesses: The paper is generally well-written and easy to follow. It is well-motivated to use annotated images due to the relatively low-cost annotations. The experiments demonstrate consistent performance gain over existing method like MViTv2, and the ablations are providing a reasonable break-down of improvements coming from different components.
questions: 1) I think the motivation of this approach is clear, but in terms of the technical contributions, my understanding is that the joint training of video and image inputs is enabled by ViT to tokenize, and there are other works jointly training images and videos [6, 25, 73, 26, 33, 61]. The authors mentioned that this work "involves explicit interactions between the main video task and the auxiliary image task", so can you further elaborate what is the explicit interaction and how is it not applicable to other approaches like [26, 33, 61]?

2) For experimental results, it appears in Table 3 (a) and (b) that the amount of annotated images is not critical, and neither is the type of HAOG attributes. Does it indicate that the auxiliary task is in fact not functioning as providing additional information, but more as a regularization factor? In other words, it might be worth to systematically demonstrate what are the minimum requirements to make the model work, maybe not the best, but reasonably well (e.g., from ~63 to ~65 in Table 3 examples). This will provide guidance when there's no annotation available but possible annotations can be collected with minimum efforts.

3) Is it possible to use the model in a zero-shot setting and compare with CLIP on action recognition tasks like Kinetics? Also, when comparing with MViTv2 results, is it a fair comparison because additional information/computation has been used in the proposed approach?
limitations: I think they properly addressed the limitations as mentioned in "We do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend to exercise caution."
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Brief Summary: The paper tackles a few-shot / semi-supervised setup where a small number of training samples about hand-object interaction are provided to drive downstream video performance. The two key ideas are to use hand object information (bounding boxes, and contact edges), and using a frame-consistency loss so that the supervision is propagated to other frames which don't have annotations.

Experiments are carried out on Something-Something, Something-Else, Ego4D, Diving48, and show improvements over compared baselines.
strengths_and_weaknesses: Pros:

1. Interesting finding that using 2% of the in-domain data is comparable to using the entire in-domain data. 

2. Multiple datasets are considered for evaluation, 


Cons:

1. On naming conventions: I don't think it is fair to say annotated video frames count as image-annotations. They are still video annotations, just that they are sparse. 

2. On relations to previous work:

(i) The authors should distinguish their work better compared to previous work. For instance, the idea of using object tags has been previous used in [44], so improvements just based on object tags are not exactly unexpected. Similarly, [Ref1] shows using entity-prompt can improve video-text pretraining. 

(ii) Structured representations for videos using semantic roles has been previously investigated, see [Ref2] and [Ref3], but discussion on them is missing.

3. On the task setup: 

(i) L71 suggests that in training we have access to video-labels and structured scene annotations. It is not clear how scalable this is, how much preprocessing time and annotation it requires.

(ii) Use of Hand-object graphs seems to be relevant for very specific domains. What if hands are not visible, or they are not interacting with a specific object?

4. On data used: According to L196, video frames are annotated, but no details of how they are obtained are provided. Is it completely automatic / semi-automatic / purely human annotated?

5. On model and experiment:

(i) In L163, the motivation for frame-consistency loss is that image loss may not transfer to video loss. It is unclear if the proposed solution is the best way to approach this. For instance, one could simply perturb the image temporal position embedding to any random index. Alternatively, a object tracking system could interpolate / extrapolate the bounding boxes. Some experiments comparing such additional ways is needed.

(ii) The relative performance improvement in Table 1. are very small, and it is unclear if the differences are actually significant.

(iii) Improvements are not always consistent, for instance, in Table 1, MViT v2 MT outperforms SViT-SFT on Top-1 base. Similarly, and perhaps more surprisingly, SViT-2% outperforms SViT-ID on 10-shot case. 

(iv) The authors should provide SViT-2% results for Table 2a,b,c as well. 

(v) (Minor) Given that the model works for both images and videos, the authors could show results on human object interaction based datasets such as [Ref4]


[Ref1]: Li, Dongxu, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. "Align and Prompt: Video-and-Language Pre-training with Entity Prompts." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4953-4963. 2022.

[Ref2]: Sadhu, Arka, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. "Visual semantic role labeling for video understanding." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5589-5600. 2021.

[Ref3]: Chen, Brian, Xudong Lin, Christopher Thomas, Manling Li, Shoya Yoshida, Lovish Chum, Heng Ji, and Shih-Fu Chang. "Joint Multimedia Event Extraction from Video and Article." arXiv preprint arXiv:2109.12776 (2021).

[Ref4]: Chao, Yu-Wei, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. "Learning to detect human-object interactions." In 2018 ieee winter conference on applications of computer vision (wacv), pp. 381-389. IEEE, 2018.

========================
Given the extensive response by the authors, I raise my score by 1 point. In particular, the experiments on Kinetics-data, and additional ablative studies are helpful.

However, I am still confused why, according to the new expts, the proposed method shows improvements in the cases where no hands are visible? This might suggest the real improvements are due to some other hyper-parameter, and not HAOG.
questions: Q1. (Minor) In section 3.1, the dataset numbering is incorrect. (3) is skipped.

Q2. Why does diving-48 show mvit performing so much worse than slow-fast and time-sformer? 


limitations: I feel that authors should emphasize the limitation of using hand-object graph which limits the proposed methods to a subset of videos where hands are prominent and not occluded. Moreover, it is unclear how the proposed method can be extended to more general action videos such as Kinetics data.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes the StructureViT (SViT) architecture designed to incorporate structured information about hand-object interaction labels in images to aid in video-related classification tasks. The authors propose a modified video-transformer architecture that, in addition to predicting video-level labels, also predicts the hand-object graph structure for annotated images via additional learned 'object tokens'. To distill information of the hand-object graphs of images to the video domain the authors further propose a Frame-Clip Consistency loss, where the hand-object graphs predicted for a video and its individual frames are forced to be consistent. The authors assess their technique on several different datasets and video-related tasks and show improvements in accuracy over the state-of-the-art (SOTA) across-the-board.
strengths_and_weaknesses: Originality: The proposed work is novel in many aspects. It is novel in attempting to exploit the transformer architecture's seamless ability to process multiple domains (video and image in this case) using the same set of shared weights. It is also novel in proposing a clever solution to distill information from the image domain to the video domain via the frame-clip consistency loss. Lastly, it is novel in attempting to injecting information of structured hand-object interaction labels to improve several downstream video-related tasks.

Quality: All methodology and experiments in the paper are technically sound and correct.

Clarity: The material is presented in a clear and organized fashion. Some editorial errors are noted below. 
- ln 30: "perfect" should be "perfectly"
- ln 129: $r_i$ should be $r_t$
- ln 130: suggested to write this as $T \times ( H \times W + n)$ for improved clarity
- ln 199: should be  "to treat them"
- ln 305: "share" should be "shared"

Significance: This work presents a significant result towards advancing multi-modality processing and information sharing with transformer architectures. Transformers are quickly becoming the dominant architecture for visual information processing surpassing CNNs. They present the additional advantage of being able to align multi-modality information much more seamlessly versus CNNs. Hence, this work is an interesting exploration that advanced our understanding of the use of transformers for multimodal (image and video) information sharing via transformers. The proposed approach is fairly general and may be applicable with minor modifications to other video and image related tasks as well besides HAOG and the video related tasks considered by the authors.
questions: Q. What procedure/losses did the authors employ for the pre-training task?

Q. One of the main weaknesses of the proposed technique is the requirement of a large number of labeled images (~100K) to distill from. While in Table 1 and Table 3, the authors show the effect of reducing the number of annotated images for the Compositional and Few-Shot Action Recognition tasks, I am curious to know how these effects play out for the experiments presented in Table 2 as well. Does the authors' claim that the number of images can be reduced to 2% (or some similar small number) of the annotated image training images hold in general or is it only true for this task?

Q. As an additional ablation, I would be curious to see the results of an experiment where the authors first learn a vision transformer to perform the HAOG task alone and then fine-tune its backbone network for the video-related task.

------------------
Post-rebuttal:

I thank the reviewers for addressing all my concerns. However, after having considered all the other reviews, I share the concerns of the other reviewers in terms of limited novelty of the current work given the existence of prior non-transformer-based works that incorporate structured information into video-based tasks. Hence, I have lowered my original rating by 1 point.
limitations: Is the requirement for a large number of annotated images a strict one for the success of the proposed method? If it is, I would like or see the authors acknowledge this limitation more clearly.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors extend prior work on utilizing scene structure information to improve action recognition performance to transformers. In particular, their model is jointly trained to detect hands and objects in images and to classify activities in videos. The images can come from the same or from a different dataset. Differently to a naive multi-task approach they introduce dedicated tokens that are used for object detection, which improves the perfromance somewhat. Additional loss encourages consistency between object tokens (which are only supervised in images) in frames encoded separately and as a part of a clip to make sure that they are not ignored during video inference (a form of domain adaptation). This consistency loss brings further improvements. Overall, minor to moderate improvements are demonstrated on several ego-centric actions recognition datasets (and, for some reason, Diving48).
strengths_and_weaknesses: Strengths:

The paper is well written and is easy to follow.

The idea of utilizing scene structure information for improving action recognition perfromance is sound, though not novel.

The proposed video transformer architecture with object tokens is novel to the best of my knowledge.

Non-negligible improvements over baselines are reported on SomethingElse, Ego4d and Diving48.

A thorough ablation study is provided.


Weaknesses:

The authors ignore prior work which utilizes inference on object graphs to improve action recognition without requiring in-domain object detection label (there is at least A Structured Model For Action Detection by Zhang et al., CVPR'19). There is no conceptual novelty in the proposed approach, only technical novelty of adapting those ideas to the transformer architecture (which is still a valuable contribution).

The authors refer to their approach as multi-modal, but images and videos are not different modalities. These claims should be removed. 

The biggest improvement is observed on the Diving48 dataset which does not require any object reasoning. This is counterintuitive and not explained.
questions: Please improve overview of the prior work to included methods that use out-of-domain object detection labels to improve action recognition via object graph reasoning and tone down the novelty claims accordingly. 

Please provided an analysis of the method's performance on Diving48 to explain why the proposed approach demonstrated the largest improvements on this dataset despite it not containing any objects. 

Please report results on a 3rd person action recognition dataset of your choosing which does require object reasoning to demonstrate to what degree the learned object representations transfer to the 3rd person scenario. 
limitations: It's not clear whether the benefits of the proposed approach are limited to the ego-centric domain.
review_title: null

Meta-review: metareview: This paper proposes StructureViT (SViT), a network architecture to incorporate structured information from images to aid in video tasks. All four reviewers found several aspects of the paper interesting including the ability to use information from just a few images and be beneficial to video tasks. They noted the thorough experimentation on multiple datasets and also found the paper easy to follow. One of the reviewers had concerns about the positioning of the paper. The authors had multiple discussions with this reviewer and were able to comprehensively update their paper and address most concerns, which was commended by the reviewer. Another reviewer had concerns about comparisons and discussions with regards to previous work. The authors did a good job of addressing most of their concerns. One common concern that emerged from the reviews and discussions was the existence of prior work that incorporates structured information into video tasks, thus reducing the novel contributions of this paper. Having read the paper, reviews and discussions carefully, I think the paper improves upon past work and has sufficient novel contributions that are valuable to readers. I recommend acceptance.
confidence: Less certain
Decision: Accept