Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a video-based HOI detection method that formulates the spatio-temporal representation in a video clip as tubelet tokens, in which the tubelet tokens are extracted based on the window-based multi-head self-attention with learnable offsets to aggregate semantically-related patch tokens in the spatial domain, and then temporal token linking to aggregate one-to-one matched tokens from the other frames. The tubelet tokens are then refined with global contexts and used as the video embeddings for querying HOI instances by a well-known transformer decoder, similarly to QPIC. 

The tubelet tokens are better than patch tokens due to their compactness in representing spatio-temporal patterns and are more likely to align with visual dynamics from semantic instances. These two benefits are important in generating informative video embeddings for transformer-based HOI detection. Therefore, extensive experiments show that a video-based HOI detection model with tubelet tokens becomes more effective and efficient than previous patch token-based reference methods. 
strengths_and_weaknesses: --- Strengths ---

1. Video embeddings as tokens that represent spatio-temporal tubelets are beneficial for various action-related tasks, such as HOI detection. This paper proposes a nice and practical attempt toward this goal and is a good reference for readers in related research fields.
2. This paper is generally well-written with clear organization and comprehensive experiments.
3. The effectiveness and efficiency of the proposed method have been proven, with significant performance gains and runtime speedup to existing methods. 

--- Weaknesses ---

1. This paper claims that the tubelet tokens are aligned with semantic instances, but the experiments did not validate this contribution. In fact, the IWP operation is not guaranteed to extract instance-level tokens.

2. The technical details should be clarified:
- Token linking: The nms-one-hot operation seems to introduce biases since different orders of query tokens will result in the different linkage of key tokens. I am not sure whether such biases affect the prediction or not.
- Global context refining: Knowing the global contexts should be useful since contexts may help discriminate fine-grained interactions. But enhancing global contexts may not discriminate against co-occurring interactions because these interactions may be different labels describing the same spatio-temporal action patterns.

3. Experiments: 
- The analysis of CNN-based & Transformer-based methods: 
  - The reference methods ST-HOI and STGPN should be briefly introduced and compared with the proposed method. 
  - How the experiment ``spatial distance'' is conducted is not quite sure. For example, does the spatial distance means the distance between the human and the object? How to handle the case that the spatial distance is temporally varying?
  - How to generate the similarity matrix in Fig.3 (c)? Line 252-256 briefly introduces the procedure, but it is still not quite straightforward. Moreover, why 'watch' and 'feed' are likely to co-occur?

questions: Overall, this is a good paper with a sound novelty. I am almost satisfied with this manuscript, but it is encouraging if the authors can answer the questions listed in the paper's weaknesses.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a novel vision Transformer TUTOR for human-object interaction detection in videos. TUTOR structurizes a video into a few tubelet tokens by agglomerating and linking semantically-related patch tokens along spatial and temporal domains. Experiments are conducted on VidHOI and CAD-120.
strengths_and_weaknesses: Strengths:
1. Ablation studies are conducted to evaluate the effectiveness of each component.
2. The proposed method achieves a new state-of-the-art performance.

Weaknesses:
1. In the paper, token linking aims at generating complete tubelets whose length is equal to the duration of the video. However, in real scenarios, the instances don’t necessarily appear in the entire video and the tubelets are thus broken into fragments. In these cases, generating a full-length tubelet is unreliable and may introduce much noise.
2. Some closely related works, such as [17], should also be compared in Table 5 and 6.
3. The manuscript is not well written and needs to be revised carefully.

questions: See above
limitations: See above
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new method for obtaining efficient token representations in transformer-based HOI detection in videos. The key idea is to spatially aggregate feature tokens using irregular window partition in order to deal with various shapes of objects (token abstraction) and to temporally link the tokens based on the similarity (token linking). The proposed method is evaluated on benchmark datasets and achieved favorable performance.
strengths_and_weaknesses: ## Strength
1. The proposed framework is overall reasonable and sound. Especially the design of token linking module is new. I think the proposed method is a reasonable extension of transformer-based HOI detection in videos.
1. The proposed method achieved favorable results on standard benchmark datasets. 
1. The effectiveness of the proposed components are verified at least to some extent.

## Weakness
1. The proposed method is not discussed well under the context of very similar work that should have inspired the proposed method. IWP in token abstraction module is very similar to deformable DETR [52]. The appropriate credit should be given to [52] and maybe also to [A1], and the proposed method should be discussed in relation to these works. In my view, token abstraction module is not a innovation of the present paper, rather, it is adaptation of the existing concept.

1. The experiment is not very satisfactory. 
    1. There is no ablation study on global context refining.
    1. Section 4.5: The reason why different decoders (1-layer, 4-layer) are used is not explained. How is the performance differences if the normal 6-layer decoder is used?
    1. Figure 3c: I find the explanation of this figure is not convincing enough. In L256 the example of watch and feed is picked up, but watch is more strongly related to wave hand, which I think is strange. Overall, I find difficulty in drawing any solid conclusion from this similarity matrix.

1. Some of the important details are not clear and confusing. 
    1. In table 1a, what is the concrete process of “ViT-like” variant?
    1. In table 1b, does the “Gumbel-softmax” variant ensure one-to-one assignment? 
    1. In equation 1, “[(i-1)S_w, (i-1)S_w]”, does this mean the regular windows are placed only diagonally? Isn’t it something like “[(i-1)S_w, (j-1)S_w]”? 
    1. In L102, “i-th regular window” is this “i” different from “i” in L103 ([i,j], C_i)? If so, this is very confusing.
    1. L113-114, “q enumerates all integral locations”. What does “all integral locations” mean? How are they calculated?
    1. L144: Is “N” here equal to “N” in L109?
    1. L145: I recommend not using index “i” for here and there to avoid confusion.
    1. L150: Is “q” here mistake for “r”?
    1. The caption of Table 1: What is “global$^{\dagger}$”?
    1. How the accuracies of other methods in Table 4 are obtained?

1. I’m afraid the paper, not significantly but slightly, violates the formatting instruction of NeurIPS. The margin seems too small and the characters in figures are invisible without zooming in (especially Fig. 3 and Tab.1c). I’m afraid this is not fair. 

1. Typos and editorial errors.
    1. L163 “the the”. 
    1. L284 “frame. when…”. 
    1. L292 “respectively. 
    1. the mAP…”. 
    1. Table 4: wrong citation for QPIC. 
    1. [52] ICLR 2021, not 2020. 
    1. Checklist 4b: Since existing assets are used (according to the answer to 4a), the answer to this question should be either yes or no. Actually it should be “No” because I found no license statement.

[A1] Dai+, Deformable Convolutional Networks, ICCV 2017

questions: 1. I think one reasonable way to find one-to-one matching in token linking module is to use Hungarian algorithm. Is it possible to use it, and if so, how does it compare to the proposed scheme for linking tokes?
1. In the token linking module, it is implicitly assumed that a token in a frame always match to another token in another frame. However, I think this is not always the case because an object may be occluded after some frames or an object may appear at some frame. In this view, I do not think it is optimal to try to find one-to-one matching between frames. I would like to know the authors’ opinion on this point.

limitations: The limitations and potential negative impact are discussed in Section 5.
review_title: null

Meta-review: metareview: *Summary*

This paper presents a novel vision Transformer TUTOR for human-object interaction detection in videos. TUTOR structurizes a video into a few tubelet tokens by agglomerating and linking semantically-related patch tokens along spatial and temporal domains. Experiments are conducted on VidHOI and CAD-120, showing that the proposed approach is more effective and efficient than previous patch token-based reference methods.

*Reviews*

The paper received 3 reviews, with ratings: 6 (weak accept), 5 (borderline accept) and 5 (borderline accept). All reviewers voted to accept the paper, but raised some concerns:
- the claim that tubelet tokens align with semantic instances is not rigorously supported (authors added additional evidence).
- an ablation study of the global context refining mechanism is required (this has been added by the authors).
- an experimental comparison to 'Detecting Human-Object Relationships in Videos' [17/18] is required (authors note that code is not available, but instead provide a comparison with an alternative similar approach that reported even better performance).
- clarification of the relationship between this model and Deformable DETR is required
- some details of the model require clarification.

*Decision*

I am satisfied that the substantive concerns raised by reviewers have been addressed by the authors, and with all reviewers voting to accept the paper I also agree. I encourage the authors to carefully update the manuscript to address all the discussions below.

confidence: Certain
Decision: Accept