Video-based Human-Object Interaction Detection from Tubelet Tokens
Authors: Danyang Tu
Wei Sun
xiongkuo min
Guangtao Zhai
Wei Shen
Venue: NeurIPS 2022 Accept (2022)
Keywords: human-object interaction
Transformer
tubelet token
video-analysis
compution vision

Abstract: We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatial-temporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each token is learned by a selective attention mechanism to reduce redundant dependencies from others; 2) Expressiveness: each token is enabled to align with a semantic instance, i.e., an object or a human, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results show our method outperforms existing works by large margins, with a relative mAP gain of $16.14\%$ on VidHOI and a 2 points gain on CAD-120 as well as a $4 \times$ speedup.

Video-based Human-Object Interaction Detection from Tubelet Tokens
Danyang Tu', Wei Sun', Xiongkuo Min', Guangtao Zhai (7)) Wei S
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University {danyangtu, aunguwei, minziongkuo, zhaiguangtao, wei. ahenaaaejsstuu... edu.cn
Abstract
We present novel vision Transformer. named TUTOR, which is able to learn tubelct tokens, served as highly-abstracter spatiotemporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens struc- turize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each tubelet token is learned by selective attention mechanism to reduce redundant spatial dependencies from others; 2) Expressiveness: each tubelet token is enabled to align with semantic instance, i.e., object or human, across frames, thanks to agglomeration and linking. The effectiveness and efficiency TUTOR are verified by extensive experiments. Results show our method outperforms existing works by large margins, with relative mAP gain of 16.14% on VidHOI and 2 points gain CAD-120 as well as 4x speedup.
Introduction
Human-object interaction (HOI) detection is a detailed scene understanding task, which requires both localization interacted human-object pairs and recognition interaction labels. Existing methods mostly investigated detecting HOIs in static images without capturing temporal information (Figure I (a)), thus lack the ability to detect time-related interactions (e.g., shoot or pass a basketball). How- ever, interactions more of time-related in practical scenario, leading strong demand to detect HOIs from videos, a more challenging problem built on spatiotemporal semantic representations.
Transformer, originated from natural language processing (NLP), is intuitive choice for its eminent capability of reasoning long-range dependencies, in which one of the most crucial components token. A token serves However, unlike language that naturally has such a discrete signal space for building tokenized dictionaries, images lic in continuous and nigh-dimensional space. To address this issue, vision Transformer (VIT) [7] provided solution that divides each image into several local patch tokens ("visual words") structurize the entire image "visual sentence" sentence"(Figure ((b)). This solution has become a de facto tokenization standard followed by most existing Transformer based methods, which has achieved excellent performance for various vision tasks, especially image classification.
element of data representations, which is usually word in language.
Nevertheless, this patch based tokenization strategy might not be proper for video-based HOI (V-HOI) detection (as the performance degradation of the ViT-like framework shown in Table. la). We find that the reason the patch tokens generated by regular splitting are difficult to exactly capture instance-level semantics (an instance is object human, e.g., the basketball shooter in Figure I), which yet is crucial for V-HOI detection to reason the interaction labels. These patch tokens inevitably
Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

600
600
hold basketbull
bold baitetball
bold basketball
hold basketall
boskehhall
aboor baskehell
bold basketholl
play
basketblll
shoot basketball
Static interaction
Dynamic interaction
Parch
+. Proliction
Tokes abstraction
+ Token linking
(a) proposals
(b) patch tokens
(c) tubelet tokens
Figure I: Illustration of different strategies for V-HOI detection, which are huilt on different representations. (a) Image-base based HOI detection methods, which process each frame as i.i.d data, and recognize the interrelations among pre-detected proposals in each frame independently. (b) ViT-like V-HOI detection frameworks, which perform global attention mechanism patch tokens over space and time. (c) Our proposed TUTOR, which structurizes video into few tubelet tokens by token abstraction along spatial domain and token linking along temporal domain.
suffer from redundancy due information mixture from different instances well insufficiency due to only part occupancy of an instance, which limit their representation ability.
In this paper, present TUTOR, new TransformeR for V-HOI detection built on TUbelet TOkens, handling aforementioned limitations favorably. The tokenization of the tubelet tokens is not based on fixed regular splitting but is jointly performed with the learning of the Transformer encoder. This enables the tubelet tokens to progressively emerge and represent high-level visual semantics. Concretely, first, along the spatial domain, alternatively update the representation for each patch token by selective attention mechanism and agglomerate semantically-relatee patch tokens into instance tokens. The selective attention mechanism ensures that attention is performed among tokens expected to belong to the same instance, which reduces redundant spatial dependencies from others. Then, along the temporal domain, we link instance tokens across frames to form the tubelet tokens. Figure I(c) illustrates the process of tubelet token generation. Experimental results show that TUTOR outperforms existing sota methods by large margins. Specifically, we achieve relative mAP gain of 16.4% on VidHOI [5] and 2 points Fl score gain on CAD-120 [22], with a 4x inference speedup.
2 Related Work
HOI detection. Most previous works are devoted to detecting HOIs in static images [3, 10, 11, 13, 14, 16, 19, 21, 23, 24, 27, 29, 35, 39, 41, 42, 44, 46, 47, 50, 51, 52, 37, 54, 20, 38, 4, 48). Without considering temporal information, these methods fail to detect time-related interactions, restricting their value in practical applications. problem, which however is less explored [35, 33, 34, 36, 5, 43, 18]. [35, 36, 43] detected HOIs in videos by building graph neural networks to capture spatiotemporal information. In [33], HOI "hotspots" can be directly learned from videos by jointly training iideo-based action recognition network as well as an anticipation model. Inspired by image-based methods, [5] introduced a two-stage framework where the frame-wise human/object features are firstly extracted by using trajectories, and then HOIs are detected by processing the instance features as well as auxiliary features, including spatial configurations and human poses. However, these methods lack the ability to model long range contextural information, resulting in poor performance when the interacted human and object are far apart. [18] proposed spatiotemporal Transformer reason human-object relationships in videos, which detects human/object proposals firstly and then captures spatial and temporal information by using two dense-connected Transformers, respectively. However, such dense-connectee manner introduces extra computation and ambiguity in token representation.
contrast, vidco-based HOI detection a more practical
Transformer in video analysis. Transformer [40] has shown a great potential in video analysis, egg. action recognition [28, 49], video restoration [25], video question answering [12], video instance segmentation [45] and etc. However, most spatiotemporal Transformer follow the de facto scheme of VIT [7], i..,, simply dividing image into local patches and stacking global attention, which lacks sufficient exploration of the properties of visual signal, thus suffering from insufficiency token representation and explosive computation.
2

---

Backbone
Abstracion
IWP
W-MSA
Instance loken
Add
Ressie Wesdow
Nom
Offer Bied
Curv.
IWP
Patch token
Hold
Token Linking
Imggye
Global
0.
Hold-
MI.P-
Content
Refining
Temporal position
Teker
CB.
Docoder
Tubelct token
Spotial
Selective atition based
S.Hackk
Simslanties
Figure 2: The architecture of TUTOR. consists of 1) backhone generate the initial patch tokens; token abstraction module that alternatively update token representations and agglomerate patch tokens, to progressively form instance tokens; 3) token linking module that links the semantically related instance tokens across different frames to form tubelet tokens; 4) simple global attention layer to reinforce the global contextual information and 5) standard Transformer decoder to decode the HOI instances. We use different dashed squares zoom on different key modules.
3
Methodology
The main idea of TUTOR is to structurize video into a few tubelet tokens, which serve as highly- abstracted spatiotemporal representations. To this end, propose reinforced tokenization strategy. which jointly performs tokenization and optimization the Transformer encoder, as illustrated in Figure 2. The process of tubelet token generation consists of two steps: 1) Token abstraction along the spatial domain, where patch tokens are alternatively updated by selective attention mechanism and agglomerated into instance tokens; Token linking along the temporal domain, where instance tokens across frames are linked to form tubelet tokens. We describe these two steps in details below.
3.1 Backhone
Taking video clip x E RT*HxxW> that consists of T frames with size H W as the input, we use a ResNet [15] followed by feature pyramid network (FPN) [26] as the backbone on f-th frame to generate a feature map ((.b)) E f XCo where = 1,,2 and Co = 32 is the channel number of the initial feature map.
3.2 Token Abstraction
Token abstraction is organized in Stages through hierarchy of Transformer layers. Each stage performs token representation learning by selective attention mechanism and merges semantically- related patch tokens into instance tokens by an agglomeration layer. Here, we denote the feature map of t-th frame inputted into s-th stage (t.s) RR "r,,,,,,, xC, Specifically, ((t.1) Z(e.)).
Selective attention. To eliminate the redundancy caused by information mixture from different instances, we are motivated to selectively calculate attention weights among related tokens, iee. tokens belong the same instances. To this end, propose irregular window partition (TWP) mechanism (orange rectangle in the right of Figure 2), a simple yet effective strategy that samples group of related tokens into local window. IWP is inspired from regular window partition [30], where the tokens are grouped by sliding a regular rectangle R with size of S., x S., over the feature { 0), [o, 1)...., [3, 4), [4, 4]} defines regular window
map ((t.s) in s-th stage. For instance, with size of 5 x 5. Then, for the i-th regular window, we have
Zim (z) (p.. + |xt.. vinl)
E R),
(1)
where ((t.e) (Ix, v)) E RIXC. denotes the feature vector at spatial location {x, y) and [or s is the location of the lop-left point of i-th window. However, as shown in Figure 2, regular windows

---

can easily divide an instance into several parts due to the limitation of fixed shape, leads to unrelated tokens within window, i.e., belonging different instances. Inspired by deformable DETR [53] and deformalbe convolution [6]. (the detailed comparison is described in Appendix) IWP makes a simple change by augmenting regular grid with learned offsets, which allows the generated irregular windows to be aligned with humans/objects with arbitrary shapes. With offsets {Apn n 1,,,..,,N ] } and N = [3], for the tokens in i-th irregular window, we have
inw (Z(r..) (Pm + [= v + Ap.) Pa E R).
(2)
Specifically, Ap,, are learned by performing a convolutional layer with kernel size of 3 x over the input feature map ((t,a): As the offsets are typically fractional, the right part of Eq. 2 is implemented practically via bilinear interpolation as
2 (1.)
Σ B(q. p) (9),
(3)
where p - Pn. + [a' Vicl + Ap,. denotes an arbitrary location, q enumerates all neighboring integral locations, and B is the bilinear interpolation kernel. On this basis, we alternatively update the token representation by stacking several S-blocks (rectangle in white solid line in Figure 2) that are built on selective attention, i.e., performing attention mechanism within irregular windows. Specifically, each block is computed as
2 IWP(2'-1 S,,),,
(4)
e W-MSA(LLN(21. + P¹)) + Flatten z' (eshape(MLPPLN(22)) + 2²),
(5)
(6)
where z' is updated representation for all tokens, p'. is sine-based spatial position encoding at l- th S-block, and å denotes various intermediate features. Here, we factorize the conventional 3D position encoding into 2D spatial position encoding and ID temporal one since the spatial and temporal information are separately extracted. detail, "W-MSA" denotes window-based multi-head self-attention, "I.N" is layer normalization and "MIP" refers to multi-layer perceptron. Since the convolution layer in IWP is operated on 2D feature map yet attention calculated on sequential features, we use "Flatten" (2.D - 1D) operation to collapse the spatial dimension and "Reshape" (LD - 2D) operation to restore it. The computational complexity a global MSA (G-MSA) block and an irregular-window-beee -ased block (IW-MSA) for total T' frames at s-th stage are respectively:
((G-MSA) = 4H, W.,W, + ((H.W
Q(IW-MSA) = 4H,,W + 2(S + K²)H.W,TC..
(8)
where K = 3 is the kernel size of convolutional layer and Sw, is fixed as 7. In comparison, IW-MSA can effectively reduce the computational complexity. In our experiment, the number of S-block for (1-3)-th stage is set to 1, 1, 3, respectively.
Token agglomeration. We perform token agglomeration at the end of each stage to merge semanti- cally similar tokens. Specifically, we first perform IWP with a window size of 2 x 2 to dynamically sample every 4 related tokens into window. Then, concatenate the tokens within each window and apply a fully-com cally-connected (FC) layer on the concatenated 4C.-dimmnsion features. We set the output dimension 2C,. It reduces the number of tokens by multiple of 2 2 = 4 after each stage.
To sum up, token abstraction totally reduces the number of tokens by a factor of 4 64 and increases the dimension by a factor of 2 8. It structures each frame into few instance tokens on the basis of selective attention and token agglomeration, which reduces the visual redundancy and also enjoys the advantage of Transformer with an affordable computational costs.
3.3 Token Linking
Assuming that after token abstraction, each frame structured as J instance tokens. Then video clip of T frames can be denoted as Zinx = {z]] |j = 1,2, J;L = 1,,, T}, where z/ refers to the j-th token in the 1-th frame. Here, sine-based 1D temporal position encoding is additionally added to Zins- The goal of token linking link T instance tokens with the same semantic across T frames, so that a video clip can be structurized as J spatiotemporal tubelet tokens. To this end, we propose exemplar

---

based between-free fraam one-to-one matching strategy. We first chose Zq = (z," 3g the exemplar frame, where T' tokens in the exemplar frame and those in rest frames Z, = {x]" bx E Zins; + r} as query tokens and key tokens, respectively. Then, we compute a similarity matrix A between the query tokens and the key tokens via a
= 1, 2.. as [ is the index of the middle frame in the video clip. We denote the
tmax [17] operation computed over the query tokens
A (iudk) exp(Wy 2. W z + +74) Σj=1 exp(W W 7i)
s.t. t t
where W." and W are the weights of the learned linear projections for the jy-th query tokens and the jı-th key tokens, respectively, and ys are i.i.d random samples drawn from the Gumbel (0, 1) distribution that enables the Gumbel-Sof tmax distribution to be close with the real categorical distribution. Then, we introduce a modified nms - one-hot operation to determine the one-to-one correspondence between the query tokens and the key tokens of each frame. Specifically, for jk- th key token z!! in t-th frame, the norm one-hot assignment is performed by taking the value of argmax A, (Je-5h) lig = J.. J However, such operation cannot ensure a one-to-one correspondence, i.e., more than one key tokens in the same frame could be assigned the same query token. To address this issue, in nms ooe-ooneeooo scheme, for example, when -tth and my-th key token 1-th frame are assigned to jg-th query token simultaneously, we assign the one with a higher
similarity, i.e., max(A, (ax(AA)) ( A (2))))) to the ig-th query token. Then, if z!"! has been assigned to the jy-th query token, we manually set A, (26))))) as 0 and continue to conduct the assignment
operation. Since the nms-one-hot operation not differentiable, we adopt the straight through strategy in [8] to compute the assignment matrix:
Å nms cess-oneeootteee
hote-hot(A)) () + A sg(A),
(10)
where sg(-) is the stop gradient operator. À is numerically equal to nms- maes-one-hot assignments and its gradient is equal to the gradient of A, which makes the token linking module differentiable and end-to-end trainable. Finally, we link the tokens corresponding to the same query token to form the
tubelet tokens Zuue = {ziubeli 1,2,..., J}. which is computed as
ΣT 11= A (5.0()))) ((2( +(it)
Woz,
L = 2/. + W.
s.t. t t
(11)
ΣT A (106.)))
where Wo and W, are the learned weights of projectors, and ((j.)) t) is the index of token in f-th frame and being assigned to j-th query token.
3.4 Global Context Refining
After token agglomeration and linking, a spatiotemporal video representation is structurized a few tubelet tokens. On this basis, we perform an additional global attention layer to model global contextual information. Our intuition is two-fold: 1) Global context can significantly boost the performance of interaction recognition, e.g., if grassland is detected, person is more likely to be playing soccer than basketball. 2) Different interactions can be co-occurring, e.g., a person holding fork could be eating something.
3.5 Decoder & Prediction Head
Decoder. Following the standard architecture in [2], the decoder transforms Ng embeddings by stacking 6 layers consisting of self-attention and cooss-tttetionn mechanisms. These embeddings are learned position encodings which are initialized to constants and we refer them to as HOI queries. Being added to the input of each attention layer, the N, queries are transformed as output embeddings by the decoder, which performs global reasoning by using the entire video clip as context.
Prediction head. Following [37], the prediction head is composed of four feew-forward networks (FFNs): numan-bounding-box FFNsf1, object-bounding-bor FFNs for object-class FFNs fo and action-class FFNs fa. Specifically, fr. and fo are both 3-layer perceptron followed by a sigmoid function, which output normalized human- and object-bounding box b [0, b, [0, 1
5

---

respectively. f is a linear layer followed by softmax function, predicting the probability of object classes é, E [0, 1 Noat,,, where Nobj is the number of object classes and the (Noat + 1-t element in C, indicates the query has no corresponding human-object pair. Since actions could be co-occurring, fa a linear layer followed by a sigmoid function rather than the softmax function. It outputs the probability of action classes é₂ [0, 1] Neer,, which has no an additional element to indicate no-action. Here, Net is the number of action classes.
3.6 Loss Function
We follow the loss calculation scheme in [37], including bipartite matching and loss calculation. We describe the detailed calculating process in Appendix.
4
Experiments
4.1 Datasets & Meerric
We conduct experiments on VidHOI [5] and CAD-120 [22] benchmarks to evaluate the proposed methods by following the standard scheme. VidHOI is large-scale dataset for V-HOI detection, comprising 6,366 videos for training and 756 videos for validation. In VidHOI, 50 relation categories are annotated, of which half are time-related ones. Mean AP (mAP) is calculated as the evaluation metric for VidHOI, which is reported over three sets: 1) Full: all 557 categories are evaluated; 2) R5 categories with less than 25 instances and 3) Non-rare: 242 categories with more than 25 instances. CAD-120 is relatively smaller dataset that consists of 120 RGB-D videos. Here, we only use the RGB images and the 2D bounding boxes annotations humans and objects. Following standard scheme, we calculate the sub-activity F1 score as metrics.
4.2 Implementation Details
The dimension of HOI query
to 256, which is the same the tubelet tokens (32 2²)). The number of queries is set to 100 for VidHOI and 50 for CAD-120. To save computational resources, the backbone is initialized by the backbone weights of QPIC [37], and then frozen without being updated, We employed an AdamW [31] optimizer for 150 epochs. A batch size of 16 on 8 RTX-2080TI GPUs, and learning rate 4, = 2.5c for Transformer and 1e for FPN are used. The 4, decayed by half at 50-th, 90-th and 120-th epoch, respectively. We use a 1, = 10-6 to warm up the training for the first 5 epochs, and then go back to 2.5e and continue training.
4.3 Analysis of CNN-based & Transformer-baser Methods
We compare CNN-based and Transformer-base methods in terms of: 1) long-range dependency modeling, robustness to time discontinuity and 3) contextual relation reasoning.
Long-range dependency modeling. We split HOI instances into bins of size 0.1 according to the normalized spatial distances, and report the APs of each bin. As shown in Figure 3a, Transformer- based method outperforms existing CNN-based methods in all cases, which becomes increasingly evident as the spatial distance grows. It indicates that Transformer has better long-range dependency modeling capability compared to CNN-based methods that relay on limited receptive field. With this ability, Transformer can dynamically aggregate important information from global context.
Robustness to time discontinuity. we randomly sample one frame every seconds from the original video to generate new video inputs, and report the relative performance compared to the baseline (sampling frame per second). As shown in Figure 3b, the performance of CNN-based methods drop dramatically in contrast to Transformer. The main reason is that the ROI features from different frames are likely to be inconsistent due to the discontinuity of temporal domain. For Transformer, can be partly solved by learning a variable attention weights to selectively process different features of different frames.
Contextual relations reasoning. we randomly pick 5 static interaction types (represented in blue) and dynamic ones (green), each with over 10,000 images. Then, we calculate the average weights of sclf-attention in the last decoder layer on all pictures where two interactions are co-predicted. As

---

STACE
010
12
d2t
(a) Spatial distance.
(b) Temporal pooling.
(c) Similarity matrix.
Figure The performance of CNN-based and Transformer based methods under different scenarios.
shown in Figure 3c, Transformer can mine the interrelations among different HOI instances, e.g.. watch and feed, which two are likely to co-occur, get a relatively high attention weights (0.71).
4.4 Analysis of Token abstraction and Linking
Token abstraction. Table. la shows the influence of token abstraction, which is proposed to capture instance level representation. comparison, CNN-based methods process cropped proposal features (Figure (a)), which suffers from temporal inconsistencies and the lack of contextual information, leading to the worst performance. In terms of Transformer, performance significantly improved ( over 25% ), but varies under different strategies. Interestingly, adding a simple token fusion module to ViT-like framework (ViT-like') ), i.e., fusing every neighboring patch tokens after each Transformer layer, can achieve 4% relative mAP improvement. It implies that visual redundancy is obstacle for Transformer to achieve better performance. Moreover, our irregular-window based (IR-win) token abstraction mechanism achieves the optimal performance. Nevertheless, when replacing all irregular windows in TUTOR with regular windows (R-win), the performance is unexpectedly surpassed by ViT-like!. It indicates that regular windows can reduce the computational complexity, but cannot eliminate visual redundancy thoroughly.
Token linking. Table. 1b shows the influence of token linking. Here, the inputs for all methods are identical, which are the instance tokens generated by token abstraction module. Although computing global attentions along temporal domain without token linking achieves competitive performance in the detection of time-related interactions, its performance relatively poor for detecting static HOIs. We conjecture that the instance tokens in different frames are semantically similar, which introduces redundant information to static interaction detection. Moreover, the mAP decreases severely when
directly use the value of Gumbel-Sof tmax as assignment weights, i.e., replace the Å in Eq. with A in Eq.9. One possible reason is the redundancy arises within token representation due to the absence of zero value in A. contrast, one-hot assignment is sparse but cannot ensure an one-to-one assignment among frames, which can also cause ambiguity. In comparison, nms-one-hot assignment enforces every T tokens (one per frame) to be linked, which minimizes the ambiguity and redundancy in token representation, thus achieving the optimal performance. We further investigate the effect of video length on these two assignment approaches. As shown in Figure 1c. one-hot assignment surpasses nms-one-hot when video longer than 16 seconds, which caused by the simple way of choosing exemplar frame, i.e., intuitively selecting the middle frame. When a video clip is long, the middle frame is semantically inconsistent with the frames that are temporally far away. We solve this problem by splitting long video into uniform short clips and performing nms-one ans-one-hot assignment in each clip nams-one-hots respectively, which yet introduces more computation.
4.5 Analysis of Effectiveness and Efficiency
Effectiveness. We verify TUTOR's effectiveness of capturing spatial and temporal semantic by observing the performance of detecting static HOI and dynamic HOI when using quite simple decoder. For former, we use a I-layer Transformer decoder on patch tokens ViT-like method and instance tokens in TUTOR, respectively. the mAP is reported only static HOI detection. As the Table 2a shows, instance tokens generated by token abstraction can stupendously improve the performance by 70%, compared with patch tokens. It demonstrates that token abstraction mechanism
7

---

method
S
method
S
T
Proposal
22.84
16.34
global
30.07
19.58
20
ViT-like
27.64
17.30
gumbel-softmax
28.81
18.11
ViT-like
28.45
18.64
one-hot
30.64
19.27
R-win
28.17
18.24
ams-one-hot
32.21
21.28
IR-win
32.21
21.28
(a) Token abstraction.
(b) Token linking.
(c) Tokenization video length.
Table 1: Analysis on token abstraction and linking. We report mAP on detecting dynamic temporal-related (T) and static spatial-related (S) HOL, respectively. ViT-like' in (a) denotes that a agular-windows wiidow-bbsedd token fusing is performed after each Transformer layer. Nms-one-hot" in (c) means to split long video into several uniform short clips. Default settings are marked in gray
case
mAP
case
params
MAP
TFLOPs
FPS
speedup
w/ TA 16.42
global
243M
23.51
0.81
0.5
spatial
w/o TA
9.67
w/ TA
104M
25.63
0.42
1.2
2x
w/ TL
8.28
w/ TL
187M
24.28
0.76
0.8
temporal
w/o TL
2.30
w/ (TA+TL)
82M
26.84
0.25
2.0
4x
(a) Effectiveness.
(b) Efficiency.
Table Analysis on effectiveness and efficiency. TA is short for token agglomeration and TL for token linking. We use clips of size 8 x 384 x 384, with frames sampled at a rate of 1/32.
can significantly extract even the decoder simple. For dynamic HOI detection, we use 4-layer perception with RELU in between as decoder. Next, we perform global average pooling on tubelet tokens and instance tokens, which are then fed to the simple decoder to predict the dynamic interactions, respectively. Interestingly, tubelet tokens generated by token linking boost performance by 4x, showing us the importance to reduce the temporal redundancy.
instance level semantic, which can be easily captured
Efficiency. Computing attention weights accounts for most of computational overhead in Transformer. Compared to the quadratic computational costs in global attention, we achieve linear one. As shown in Table 2b, TUTOR achieves a 4x speedup, greatly improving its usability in practical applications. Here, "FPS" is reported in terms video, i.e., the number of videos being processed per second.
4.6 Ablation Study
Token agglomeration. Token agglomeration is proposed to distill the token representation by selectively merging and projecting the semantically related tokens. We first intuitively try the k- mean [32], excellent classical clustering algorithm, but obtain unexpectedly poor performance, as shown in Table 3a. The reason two-fold: it is difficult to integrate the k-mean with the main network into end-to-end pipeline: 2) it is hard determine the value of K. Then we replace irregular winodws TUTOR with regular windows to merge every neighboring tokens, an average pooling operation essentially, which reduces the feature redundancy to some extent. In comparison, irregular- window is more of an operation to selectively merge the semantically similar tokens to model highly abstracted features. It worth emphasizing that gradually increase the dimension of agglomerated token is interestingly important, which achieves a gain of more than 6% on mAP. We guess that the features are richer with increasing dimensions, as extensively adopted in CNNs. Window size. Table 3b varies the window size. The instances in an image could have variant sizes. Therefore, small-sized window is hard to overlap different instances while large-sized one could cause information mixture unrelated tokens may be included. We find 7 to be optimal.
Small tricks. We represent some small tricks for key module design in Table 3c. For mons-one-iin ussignment, another commonly used strategy for merging assigned tokens is to concatenate them and then project them with fully-connected layer. Compared with weighted-sum, it can slightly improve performance, but introduces more computation. For position encoding, we factorize the normally used 3D position encoding for spatiotemporal Transformer into a 2D spatial position encoding and ID temporal one, which two are added in token agglomeration and linking module, respectively. It

---

component
trick
method
S
T
size
S
T
32.85
21.44
nms-onc-hot
k-mean
26.71
17.56
3
24.80
17.92
w-sum
32.21
21.28
r-win
28.82
18.95
5
30.69
19.75
p-encoding
3D
31.62
20.59
(2+1)D
32.21
21.28
ir-win-C
29.34
19.43
32.21
21.28
w/o
31.41
20.63
GCR
ir-win-2C
32.21
21.28
11
30.28
19.71
w/ 32.21 21.28
(a) Token agglomeration.
(b) Window size.
(c) Small tricks.
Table Ablation study. (a), "nC" is the dimension of agglomerated token. in (c), "p- "p-encoding" is short for position encoding, "w-sum" for weighted-sum, "GCR" for global context refining.
method
backbone
P
VidHOI NoneRare
CAD-120 sub-activity 9%)
Full
Rare
T
CNNN-busee methods
PMF [41] w/ SlowFast
SlowFast
16.31
14.28
23.86
21.77
8.42
GPNN [35]
ReaNet-101
18.47
16.41
24.50
26.41
16.06
88.9
STIGPN [43]
ResNet-50
19.39
18.22
28.13
26.58
18.46
91.9
ST-HOI [5]
SlowFast
17.60
17.30
27.20
25.00
14.40
Trunsformer- bused methods
HOTR* [20]
ResNet-50
21.14
19.83
30.75
28.36
9.81
QPIC* [37]
ResNet-50
21.40
20.56
32.90
28.87
9.74
HOTR w/ SlowFast
SlowFast
22.84
21.15
32,86
27.12
13.29
90.7
QPIC w/ SlowFast
Slow/Fas
22.92
21.64
33.43
28.41
13.47
91.3
TimmSSommer [1] w/ decoder
TimeSformer
23.17
21.79
34.57
27.84
18.90
92.5
Ours
ResNet-50
26.92
23.49
37.12
32.21
21.28
94.7
Table 4: comparison with state-of-the-art. means human poses, denotes image-based method.
is an experiential operation since spatial and temporal information are separately extracted, Global
context refining, which refines global contextual information, can achieve almost point gain.
4.7 Comparison with State-of-the-art
Unlike the popularity of image-based HOI detection, relatively less works investigate iideo-based one as a more practical yet challenging problem. Interestingly, the ability of image-based methods to detect dynamic HOI can be partly improved by replacing the original 2D backbone with 3D one, but weaken the ability of detecting static HOIs. With aforementioned strategies, our methods outperforms existing sota methods by large margins. It is belief that detecting HOI from video more reasonable and practical since most interactions are time-related. Therefore, we hope our work will be useful for video-based human activity understanding research.
Discussion & Conclusion
Limitation. Our Transformer-be based method suffers from the problem of overfitting when handling with small-scale datasets. In our experiments, we have to use the pertrained weights on VidHOI to initialize the model for CAD-120 (small scale), or performance would be severely degraded.
Broader impacts. We know some applications that illegally analyze user behavior by video monitor- ing. Therefore, striet ethical review is essential to avoid our model being used for such applications.
Conclusion. In this paper, we present TUTOR, novel spatiotemporal Transformer for video-based HOI detection, which structurizes video into few tubelet tokens. To generate compact and expressive tubelet tokens, we propose a token abstraction scheme built on selective attention and token agglomeration, along with token linking strategy to link semantically-related tokens across frames. Our methods outperforms existing works by large margins. Going further, visual redundancy is one of the biggest obstacles for vision Transformer achieve the same excellent performance as language Transformer, and we will devote more exploration on this in the future works.
Acknowledgments. This work was supported by NSFC (62225112, 61831015) and National Key R Program China 2021 YFF0206700, NSFC 62176159, Natural Science Foundation of Shanghai 2ZRR 1432200 and Shanghai Municipal Science and Technology Major Project 2021SHZDZX0102.

---

References
[1] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for video understanding. ICML, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020.
[] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Deng. Learning to detect human-object interactions. In WACV, 2018.
[4] Mingfei Chen, Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In CVPR, 2021. 2
[5] Meng-Jiun Chiou, Chun- Yu Liao, Li-Wei Wang, Roger Zimmermann, and Jiashi Feng. St-hoi: spatial- temporal bascline for human-object interaction detection in vidcos. In ICDAR, pages 9 17, 2021. 2, 6,
[6] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017. 4
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2021. 1,2
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883 2021.
[9] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. recognition. In CVPR, 2019. 9
Slowfast networks for video
[10] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020. 2
[11] Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centrie attention network for human-object interaction detection. In BMVC, 2018. 2
[12] Noa Garcia and Yuta Nakashima. Knowledge-base based vidco question answering with unsupervised scene descriptions. In ECCV, pages 581-598, 2020. 2
[13] Georgia Gkioxari. Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human-objed interactions. In CVPR, 2018. 2
[14] Tammay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In ICCV, 2019. 2
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 3
[16] Zhi Hou, Xiaojiang Peng. Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. ECCV, 2020.
[17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzaiion with gumbel-softmax. In ICLR, 2017. 5
[18] Jingwei Ji, Rishi Desai, and Juan Carlos Niebles. Detecting human-object relationships videos. ICCV, pages 8106 8116, 2021. 2
[19] Bumsoo Kim, Tacho Choi, Jacwoo Kang, and Hyunwoo J. Kim. UnionDet: Union-level detector towards real-time human-object interaction detection. In ECCV, 2020.
[20] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end
human-objed interaction detection with transformers. In CVPR, 2021. 2.9
[21] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV, 2020. 2
[22] Hema Swetha Koppula, Rudhir Gupta, and Ashutosh Saxena. Learning human activities and object affordances from rgb-d videos. IJRR, pages 951-970, 2013. 6
10

---

[23] Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020. 2
[24] Yong-Lu Lu Li. Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang. Yanfeng Wang. and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In CVPR, 2019.
[25] Jingyun Liang, Jiczhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawci Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer. arXiv preprint arXiv:2201. 2022.
[26] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[27] Xue Lin, Qi Zou, and Xixia Xu. Action-guided attention mining and relation reasoning network for human-obbec interaction detection. In IJCAI, 2020. 2
[28] Xiaolong Liu, Qimeng Wang. Yao Hu, Xu Tang, Song Bai, and Xiang Bai. End-to-cns temporal action detection with transformer. arXiv preprint arXiv:2106. 102222,, 2021. 2
[29] Yang Liu, Qingchao Chen, and Andrew Zisserman. Amplifying key cucs for human-ohjjcct-nteraatiii detection. In ECCV, 2020. 2
[30] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 3
[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2017.
[32] James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, 1967. 8
[33] Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In ICCV, pages 8688-8697, 2019. 2
[34] Megha Nawhal, Mengyao Zhai, Andreas Lehrmann. Leonid Sigal, and Greg Mori. Generating videos of zero-shot compositions of actions and objects. In ECCV, pages 382-401, 2020. 2
[35] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song- Chhnn Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, 2018. 2,9
[36] Sai Praneeth Reddy Sunkesula, Rishabh Dabral, and Ganesh Ramakrishnan. Lighten: Learning interactions with graph and hierarchical temporal networks for hoi in videos. In ACMMM, pages 691-699, 2020. 2
[37] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In CVPR, 2021. 2, 5, 6,9
[38] Danyang Tu, Xiongkuo Min, Huiyu Duan, Guodong Guo, Guangtao Zhai, and Wei Shen. Iwin: Human- object interaction detection via transformer with irregular windows. arXiv preprint arX1v:2003.10537, 2022. 2
[39] Oytun Ulutan, ASM lfiekhar, and Bangalore S Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In CVPR, 2020.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 2
[41] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, 2019. 2,
[42] Hai Wang. Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020. 2
[43] Ning Wang, Guangming Zhu, Liang Zhang, Peiyi Shen, Hongsheng Li, and Cong Hua. Spatio-temporal interaction graph parsing networks for human-object interaction recognition. In ACMMM, pages 4985- 4993, 2021. 2.
[44] Tiancai Wang. Rao Muhammad Anwer, Muhammad Haris Khan, Fahad Shahbaz Khan, Yanwei Pang. Ling Shao, and Jorma Laaksonen. Deep contextual attention for human-object interaction detection. In ICCV, 2019. 2
[45] Yuqing Wang, Zhaoliang Xu. Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, pages 8741-87777 2021. 2
11

---

[46] Bingjie Xu. Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli. Interact as intend: iilention-driven human-object interaction detection. 7MM, 2019.
[47] Dongming Yang and Yuexian Zou. A graph-hased interactive reasoning for human-object interaction detection. I/CAL, 2020. 2
[48] Aixi Zhang, Yue Liao, Liu, Miao Lu, Yongliang Wang, Chen Gao, Xiaobo Li. Mining the benefits of two-stage and one-stage hoi detection, NIPs, 34, 2021.
[49] Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. arXiv preprint arXiv:2202.07991 2022. 2
[50] Xubin Zhong, Changxing Ding, Xian Qu, and Dacheng Tao. Polysemy deciphering network for robust human-object interaction detection. In ICCV, 2021. 2
[51] Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. In ICCV, 2019.
[52] Tianfei Zhou, Wenguan Wang, Siyuan Qi. Haibin Ling, and Jianbing Shen. Cascaded human-object interaction recognition. In CVPR, 2020. 2
[53] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-ene to-end object detection. In ICLR, 2021. 4
[54] Cheng Zou, Bohan Wang. Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, and Yichen Wei. End-to-end human object interaction detection with hoi transformer. In CVPR. 2021. 2
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] proprietary.
The code and the data are
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [Yes] The code
provided in supplementary material.
12

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
A Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in the appendix. This section will often be part of the supplemental material.
13


Reviews:
[{"review_id": "reFFUNfxdH1", "reviewer": "Reviewer_Jbmu", "summary": "This paper proposes a new method for obtaining efficient token representations in transformer-based HOI detection in videos. The key idea is to spatially aggregate feature tokens using irregular window partition in order to deal with various shapes of objects (token abstraction) and to temporally link the tokens based on the similarity (token linking). The proposed method is evaluated on benchmark datasets and achieved favorable performance.", "questions": "1. I think one reasonable way to find one-to-one matching in token linking module is to use Hungarian algorithm. Is it possible to use it, and if so, how does it compare to the proposed scheme for linking tokes?\n1. In the token linking module, it is implicitly assumed that a token in a frame always match to another token in another frame. However, I think this is not always the case because an object may be occluded after some frames or an object may appear at some frame. In this view, I do not think it is optimal to try to find one-to-one matching between frames. I would like to know the authors’ opinion on this point.\n", "limitations": "The limitations and potential negative impact are discussed in Section 5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. The proposed framework is overall reasonable and sound. Especially the design of token linking module is new. I think the proposed method is a reasonable extension of transformer-based HOI detection in videos.\n1. The proposed method achieved favorable results on standard benchmark datasets. \n1. The effectiveness of the proposed components are verified at least to some extent.\n\n## Weakness\n1. The proposed method is not discussed well under the context of very similar work that should have inspired the proposed method. IWP in token abstraction module is very similar to deformable DETR [52]. The appropriate credit should be given to [52] and maybe also to [A1], and the proposed method should be discussed in relation to these works. In my view, token abstraction module is not a innovation of the present paper, rather, it is adaptation of the existing concept.\n\n1. The experiment is not very satisfactory. \n    1. There is no ablation study on global context refining.\n    1. Section 4.5: The reason why different decoders (1-layer, 4-layer) are used is not explained. How is the performance differences if the normal 6-layer decoder is used?\n    1. Figure 3c: I find the explanation of this figure is not convincing enough. In L256 the example of watch and feed is picked up, but watch is more strongly related to wave hand, which I think is strange. Overall, I find difficulty in drawing any solid conclusion from this similarity matrix.\n\n1. Some of the important details are not clear and confusing. \n    1. In table 1a, what is the concrete process of “ViT-like” variant?\n    1. In table 1b, does the “Gumbel-softmax” variant ensure one-to-one assignment? \n    1. In equation 1, “[(i-1)S_w, (i-1)S_w]”, does this mean the regular windows are placed only diagonally? Isn’t it something like “[(i-1)S_w, (j-1)S_w]”? \n    1. In L102, “i-th regular window” is this “i” different from “i” in L103 ([i,j], C_i)? If so, this is very confusing.\n    1. L113-114, “q enumerates all integral locations”. What does “all integral locations” mean? How are they calculated?\n    1. L144: Is “N” here equal to “N” in L109?\n    1. L145: I recommend not using index “i” for here and there to avoid confusion.\n    1. L150: Is “q” here mistake for “r”?\n    1. The caption of Table 1: What is “global$^{\\dagger}$”?\n    1. How the accuracies of other methods in Table 4 are obtained?\n\n1. I’m afraid the paper, not significantly but slightly, violates the formatting instruction of NeurIPS. The margin seems too small and the characters in figures are invisible without zooming in (especially Fig. 3 and Tab.1c). I’m afraid this is not fair. \n\n1. Typos and editorial errors.\n    1. L163 “the the”. \n    1. L284 “frame. when…”. \n    1. L292 “respectively. \n    1. the mAP…”. \n    1. Table 4: wrong citation for QPIC. \n    1. [52] ICLR 2021, not 2020. \n    1. Checklist 4b: Since existing assets are used (according to the answer to 4a), the answer to this question should be either yes or no. Actually it should be “No” because I found no license statement.\n\n[A1] Dai+, Deformable Convolutional Networks, ICCV 2017\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dXghCT-xeZk", "reviewer": "Reviewer_JsuZ", "summary": "This paper presents a novel vision Transformer TUTOR for human-object interaction detection in videos. TUTOR structurizes a video into a few tubelet tokens by agglomerating and linking semantically-related patch tokens along spatial and temporal domains. Experiments are conducted on VidHOI and CAD-120.", "questions": "See above", "limitations": "See above", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Ablation studies are conducted to evaluate the effectiveness of each component.\n2. The proposed method achieves a new state-of-the-art performance.\n\nWeaknesses:\n1. In the paper, token linking aims at generating complete tubelets whose length is equal to the duration of the video. However, in real scenarios, the instances don’t necessarily appear in the entire video and the tubelets are thus broken into fragments. In these cases, generating a full-length tubelet is unreliable and may introduce much noise.\n2. Some closely related works, such as [17], should also be compared in Table 5 and 6.\n3. The manuscript is not well written and needs to be revised carefully.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OTDSwLaGAkU", "reviewer": "Reviewer_7mrZ", "summary": "This paper proposes a video-based HOI detection method that formulates the spatio-temporal representation in a video clip as tubelet tokens, in which the tubelet tokens are extracted based on the window-based multi-head self-attention with learnable offsets to aggregate semantically-related patch tokens in the spatial domain, and then temporal token linking to aggregate one-to-one matched tokens from the other frames. The tubelet tokens are then refined with global contexts and used as the video embeddings for querying HOI instances by a well-known transformer decoder, similarly to QPIC. \n\nThe tubelet tokens are better than patch tokens due to their compactness in representing spatio-temporal patterns and are more likely to align with visual dynamics from semantic instances. These two benefits are important in generating informative video embeddings for transformer-based HOI detection. Therefore, extensive experiments show that a video-based HOI detection model with tubelet tokens becomes more effective and efficient than previous patch token-based reference methods. ", "questions": "Overall, this is a good paper with a sound novelty. I am almost satisfied with this manuscript, but it is encouraging if the authors can answer the questions listed in the paper's weaknesses.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--- Strengths ---\n\n1. Video embeddings as tokens that represent spatio-temporal tubelets are beneficial for various action-related tasks, such as HOI detection. This paper proposes a nice and practical attempt toward this goal and is a good reference for readers in related research fields.\n2. This paper is generally well-written with clear organization and comprehensive experiments.\n3. The effectiveness and efficiency of the proposed method have been proven, with significant performance gains and runtime speedup to existing methods. \n\n--- Weaknesses ---\n\n1. This paper claims that the tubelet tokens are aligned with semantic instances, but the experiments did not validate this contribution. In fact, the IWP operation is not guaranteed to extract instance-level tokens.\n\n2. The technical details should be clarified:\n- Token linking: The nms-one-hot operation seems to introduce biases since different orders of query tokens will result in the different linkage of key tokens. I am not sure whether such biases affect the prediction or not.\n- Global context refining: Knowing the global contexts should be useful since contexts may help discriminate fine-grained interactions. But enhancing global contexts may not discriminate against co-occurring interactions because these interactions may be different labels describing the same spatio-temporal action patterns.\n\n3. Experiments: \n- The analysis of CNN-based & Transformer-based methods: \n  - The reference methods ST-HOI and STGPN should be briefly introduced and compared with the proposed method. \n  - How the experiment ``spatial distance'' is conducted is not quite sure. For example, does the spatial distance means the distance between the human and the object? How to handle the case that the spatial distance is temporally varying?\n  - How to generate the similarity matrix in Fig.3 (c)? Line 252-256 briefly introduces the procedure, but it is still not quite straightforward. Moreover, why 'watch' and 'feed' are likely to co-occur?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept