Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection
Authors: Kaifeng Gao
Long Chen
Hanwang Zhang
Jun Xiao
Qianru Sun
Venue: ICLR 2023 poster (2023)
Keywords: Prompt Tuning
Video Relation Detection

Abstract: Prompt tuning with large-scale pretrained vision-language models empowers open-vocabulary prediction trained on limited base categories, e.g., object classification and detection. In this paper, we propose compositional prompt tuning with motion cues: an extended prompt tuning paradigm for compositional predictions of video data. In particular, we present Relation Prompt (RePro) for Open-vocabulary Video Visual Relation Detection (Open-VidVRD), where conventional prompt tuning is easily biased to certain subject-object combinations and motion patterns. To this end, RePro addresses the two technical challenges of Open-VidVRD: 1) the prompt tokens should respect the two different semantic roles of subject and object, and 2) the tuning should account for the diverse spatiotemporal motion patterns of the subject-object compositions. Our RePro achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones. Extensive ablations also demonstrate the effectiveness of the proposed compositional and multi-mode design of prompt. Code is available at https://github.com/Dawn-LX/OpenVoc-VidVRD.

Published as a conference paper at ICLR 2023
COMPOSITIONAL PROMPT TUNING WITH MOTION CUES FOR OPEN- MENN-VOCBBLLARY VIDEO RELATION DETECTION
Kaifeng Gao', Long Chen"; Hanwang Zhang Jun Xiao', Qianru Sun 1 !Zhejiang University, 2The Hong Kong University of Science and Technology Naayyng Technological University, 'Singapore Management University "[kite.phone, phone, junx]0zju.ee edu.. cn, 2zjuuchnnoongggnee gmaii com á "hanwangzhangenttu ed, sg, *qianrusund smu edu. sg
ABSTRACT
Prompt tuning with large-scale pretrained vision-language models empowers open-vocabulary predictions trained on limited base categories, egg. object clas- sification and detection. In this paper, we propose compositional prompt tuning with motion cues: an extended prompt tuning paradigm for compositional pre- dictions of video data. In particular, we present Relation Prompt (RePro) for where con-
Open-vocabulary Video Visual Relation Detection POpen-VidVRIl ventional prompt tuning is easily biased to certain subject-object combinations and motion patterns. To this end, RePro addresses the two technical challenges of Open-VidVRR VidVRD: 1) the prompt tokens should respect the two different seman- tic roles of subject and object, and 2) the tuning should account for the diverse spatio-temporal motion patterns of the subject-object compositions. Without bells and whistles, our RePro achieves new state-of-the-ee performance on two Vid- VRD benchmarks of not only the base training object and predicate categories, but also the unseen ones. Extensive ablations also demonstrate the effectiveness of the proposed compositional and multi-mode design of prompts. Code is available attps: / / //gggithuu thub. com/ Dawn. LX/ http/ppeeVo----eeeeeee -ViVVDD.
INTRODUCTION
Video visual relation detection (VidVRD) aims detect the visual relationships between object tracklets in videos as (subject, predi- cate, object) triplets (Shang et al., 2017; Chen et al., 2021; 2023; Gao et al., 2021; 2022), e.g., dog-towa wiwardds rds-child ld shown in Fig- ure 1. Compared to its counterpart in still im- ages (Chen et al., 2019; Li et al., 2022b;cddze). due to the extra temporal axis, there are usu- ally multiple relationships with different tem- poral scales, and a subject-object pair can have several predicates with ambiguous boundaries. For example, shown in Figure 1, the action feed of child to dog co-occurs with sev- eral other predicates (e.g., away, towards).
14.18
22.16
23.69
27.76
20,96
dog
stool
dog
stool
sit-on
away
sit-on
away
feed
child
sit-on
floor
child
floor
breed
Open- Vocabulary
Examples of VidVRD. The relation graphs are w.r.t the whole video clip. Dashed lines denote unseen new categories in the training data.
Figure I:
This characteristic makes VidVRD have more plentiful and diverse relations between objects than its image counterpart. As result, it is impractical to collect sufficient annotations for all categories for VidVRD. Therefore, to make VidVRD practical, we should know how to generalize the model, trained on limited annotations, to new object and predicate classes unseen training data.
"Long Chen is the corresponding author. Part of the work was done when Kaifeng Gao served as visiting Ph.D. student at Singapore Management University.

---

Published as a conference paper at ICLR 2023
[a][video|oo[]]dug] [away] [chäld]
Multi-mode
[5:] [5p
[away]
Prompt Groups
Motion Cues
[ox] [0:]
[=[[video[[of]]ddgg [towards] [child]
awaay
lowaard
(a) Fixed (Handcraft) Prompt
[si
[tal
1sy
[towards]
GT: logd--wards-hhild
[0q]
[02]
legl
[wy] (wy] ... [we] [uway]
[..
class token
fixed prompt token
[Wa] [Wa] [wa] [towards]
*x
Selector
leamable prompt token
away
Lowards
away boaads
(b) Conventional Leamable Prompt
(c) Compositional Prompt with Motion
Figure 2: Comparisons of different prompt tuning methods for Open- VidVRD.
To this end, we propose a new task: Open-vocabulary VidVRD Viden-id In particular, "opee"" does not only mean unseen relationship combinations, e.g., dog-sit.on-floo. unseen objects and predicates, e.g., bread and feed, as shown in Figure 1. Recent works on such generalization only focus on the unseen combinations (Chen et al., 2021; Shang al., 2021) in Vid- VRD, or zero-shot transfer among semantically related objects in zero-shot object detection (Huang al., 2022), e.g., the seen dog class can help recognize the unseen wolf. However, they fail to generalize to the categories totally unrelated to the limited seen ones, where the transfer gap is unbridgeable, e.g., bread in testing has no visual similarity with dog and chi fild in training.
but also
Thanks to the encyclopedic knowledge acquired by large vision-language models (VLMs) pre- trained on big data (Radford et al., 2021; Li et al., 2022a), we can achieve open-vocabulary relation detection with only training data of limited base categories. To bridge the gap between the pre- trained and downstream tasks without extra fine-tuning the whole VLM model, a trending technique named prompt tuning is widely adopted (Liu et al., 2021; Jin et al., 2022; Zhou et al., 2022b). For example, we can achieve zero-shot relation classification for the tracklets pair in Figure 2. We first crop the object tracklet regions in the video, and feed them into the visual encoder of VLM to obtain corresponding visual embeddings. Then we use a simple prompt like "a video of [CLASS]*, feed it the VLM's text encoder obtain the text embedding, and classify the object based on the similarities between visual and text embeddings. Based on the tracklet classification results, for the example of the pair dog and child, we can craft a prompt like "a video of dog [CLASS] child", as shown in Figure 2(a), and similarly classify their predicates based on the predicate text embed- dings. Furthermore, we can replace the fixed prompt tokens with learnable continuous tokens, as shown in Figure 2(b), known as prompt representation learning, which has been widely applied to open-vocabulary object detection (Gu et al., 2021; Du et al., 2022; Ma et al., 2022).
Learning the prompt representation is actually introducing some priors for describing the context of target classes, and it excludes some impossible classes with the constraint of the context. However, the prompt representations (either handcrafted or learned) in above approaches are monotonous and static, and learning the prompt sometimes might break the "open" knowledge due to overfitting to the base category training data. Modeling the prompt representation for video visual relations has some specific characteristics that need to be considered:
The prompt context for predicates is highly related to the semantic roles of subject and object. A holistic prompt representation might be sub-optimal for predicates. For example, shown in Figure 1, even the same predicate (s (oon in different relation triplets dlog-sit.ont floor and
Compositional:
-illdddsstoon-ddeee stool) have totally different visual context.
Motion-related:
Predicates with different motion patterns naturally should be prompted with different context tokens. The naive prompt representation fails to consider the spatio-temporal motion cues of tracklet pairs. For example, the predicate towards shown in Figure 1 can be prompted as "a relation of [CLASS], moving closer". In contrast, eat and sit.on can be prompted as "a relation of [CLASS), relative static".
In this paper, we propose a compositional and basedd Relation Prompt learning framework: RePro, as shown in Figure 2(c). To deal with the compositional characteristic of visual relations, we set compositional prompt representations specified with subject and object respectively. With this design, we model the prompt context w.r.t. semantic roles (i.e., sub ject object). For example, a possible prompt can be "sth. doing [CLASS]" for the subject and "sth. being [CLASS]" for the object. To consider the motion-related characteristic of predicate contexts, design multi- mode prompt groups, where each group (i.e., each mode) is assigned with certain motion pattern, and has its own compositional prompts for the subject and object. During the implementation,
2

---

Published as a conference paper at ICLR 2023
select proper group according to the motion cues (patterns) in the subject-object tracklet pairs (cf. Sec. 3.3). Compared to some prompt tuning works which focus on category- based context (Zhou et al., 2022b) or instance- conditiond context (Zhou et al., 2022a; Ni et al., 2022), our motion-cue- based grouping has better cross cross-category generalization ability, and can avoid the over-fitting to base categories. We evaluate our RePro the VidVRD (Shang et al., 2017) and VidOR (Shang et al,, 2019) benchmarks. Our experiment results show that RePro trained with only the samples of base relation categories has a good generalizability to detect novel relations, and achieves the new state- of-theeart, art. For example, it outperforms the top- pop-performing method, i.e., VidVRD-II (Shang et al., 2021), by 2.54% and 3.91% % absolute mAP for SGDet and SGCIs settings, respectively.
Our contributions in this paper are thus three-fold. 1) A new open-vocabulary setting for video vi- sual relation detection task, i.e., Open-VidVRID, VidVRD. 2) compositional prompt representation learning method that models the prompt contexts for the subject and object separately. 3) A motion-cue-based multi-mode prompt groups that achieve a strong generalization ability.
2 RELATED WORK
Video Visual Relation Detection (VidVRD) was defined in Shang et al. (2017; 2019) together with the proposals of the VidVRD and VidOR benchmarks. The task aims spatio-temporally localize visual relations between object tracklets. Existing methods mainly focus on modeling better visual or spatio-temporal contexts (Qian et al., 2019; Shang et al., 2021; Cong al., 2021), and detect- ing visual relations with more granularity either by sliding windows (Liu et al., 2020) or temporal grounding (Gao al., 2022). They mainly worked on the pre-defined (closed) sets of object and predicate categories. In contrast, our work is the first one to study the open- locabulayyy VidVRD setting, where some object and predicate categories are unseen in the training set.
Zero-Shot Setting in Image and Video VRD. Existing VRD works, either in image domain (Tang et al., 2020; Kan et al., 2021) or video domain (Shang et al., 2021), only achieve zero-shot transfer on the unseen triplet combinations, where the objects and predicates are seen in the training set. They ignore the model's generalization ability to unseen object/predicate categories. There is one concurrent work (He et al., 2022) proposes the open-vocabulaty setting in image VRD. However, they put the main emphasis on unseen object categories. Different from them, RePro generalizes the model to recognize both object and predicate categories totally unrelated to the seen training ones.
Prompt Tuning for Open-vocabulary Visual Recognition. Prompt tuning (Liu et al., 2021) has been widely adopted in both image (e.g., open-vocabulary object detection (OV-Det) (Gu et al., 2021; Du et al., 2022; Ma et al., 2022)) and video (e.g., zero-shot video action recognition (Lin et al., 2022; Ni et al., 2022; Ju al., 2022; Nag et al., 2022)) domains. For OV-Det, recent works mainly focus on knowledge distillation from VLM and simply using handcrafted prompt (Gu et al., 2021; Ma al., 2022), or focus on prompt representation learning for object regions (Du al., 2022; Feng et al., 2022). For video action recognition, existing works mainly use fixed prompt (Nag et al., 2022), conventional learnable prompt (Ju et al., 2022), or prompt conditioned on the input video contexts (Ni et al., 2022), and they all focus on the cross-frame attention or feature interaction. In contrast, our RePro learns the compositional prompt by leveraging the motion cues of subject-object pairs, and has better cross-category generalization ability to detect novel visual relations in videos.
METHOD
To build the Open-VidVRD setting, we first divide the categories of dataset into base and novel splits. Specifically, we denote co and Co as the sets of hase and novel object categories, respec- tively. We use cj and CP to denote the sets of base and novel predicate categories, respectively. In the training stage, we use all visual relation triplet samples from CO x C₁ x CC. In the testing stage, we evaluate the model with the triplets sampled from all categories, i.e., CO U and Cj¹² UCC.
Based on this new setting, we first briefly introduce the preliminaries for open-vocabulary classifi- cation with pre- trained VLMs (Sec. 3.1). Then, we introduce the proposed Open- Vidn-iiiVDDDDD method RePro, illustrated in Figure 3, in which first extend opca-vocabulary object detection meth- ods (Gu et al., 2021; Du et al., 2022) to open- vecabubuyyl tracklet detection (Sec. 3.2), and then perform open-vocabulary relation classification for each tracklet pair (Sec. 3.3).
3

---

Published as a conference paper at ICLR 2023
fixed
dut-product
add
traininng
training & test
[8g
[sway] [hold]
suby
Texi
[away]
[bold]
[s] (video] [od]
[dog] [child]
Texi Essoder
Object Text Eimbds
[o:] [z]]
[lowards]
obj
Encoder
Concat
[kowands)
[Roor]
deg child flace
Multi-mode Pronpe Groupp
Selector
Video
Visual Embdds
Al
PK
Tracklte Detectee
Crop & Resize
Visual Encoder
Motion Ciucs FR
Relative Position
fij
ObjCk
Apps
Rol Festuros
Vieral Embhlk VL.)
towards
Concar
Rol
dog
Align
fi
ObyCls
Rol Festures fil
di
dog-tovards-child
Rol Festures
Vouul Eutbls (fe juin)
PredCts
(a) Tracklet Detection
(b) Relation Classification
Figure 3: The overall pipeline of RePro. The visual embeddings (v,, Vi,)) are only used at training time, in which train two V2L project modules (i.e., @h and Φp) to transfer the knowledge from the pre-trained VLM. At test time, only Rol features (fi, fi,)) and position features (fft,) are used. For detection (a), the knowledge is transferred by aligning vy to u, For relation classification (b), the knowledge is transferred by the prompt representations learned in the supervision of Ua.j.
3.1 PRELIMINARIES: OPEN-VOCABULARY CLASSIFICATION WITH PROMPT
Fixed Prompt. Pre-trained VLMs have a strong open-vocabulae vocabulary classification ability (Li et al., 2022a). They first extract text embeddings for all categories by feeding handcrafted prompt (e.g.. "a video of [CLASSJ") into the text encoder VLM, where [CLASS] can be replaced with the class name of an arbitrary object predicate. Their output text embedding to R for each class is
to VL (Wo), W. = [ww,..., Wn, w.)) Vc E CØ UC C Uc!."
(1)
where Wa is the prompt representation with context token vectors and the class token vector v.-- Then, for each object tracklet with cropped video region, the corresponding visual embedding can be extracted by the visual encoder of VLM, denoted as v, E R Similarly, the visual embedding of tracklet pair (i can also be extracted (e.g., based on the union region), denoted as Va Therefore, the i-th region (generally denoted as v;)) can be classified by the cosine similarities w.r.t (ta):
8 = arg max cos U. tz), Vc E Co U C or C₁ U C.P. where cos x. y) = x²' y (1 llll)
(2)
Learnable Prompt. Manually tuning the words in the prompt requires domain expertise and is time-consuming or not robust (Radford et al., 2021). The substitute method is to learn the prompt representations from the training data (Zhou et al., 2022b;a). Specifically, the context vector w, in We can be set as a learnable vector while w. is kept fixed. In the training stage, samples and {w:) are from base categories. In the testing stage, the L learned vectors in each W. are fixed and then the model performs classification in the same way Eq. (2).
3.2 OPEN-VOCARUTI OBJECT TRACKLET DETECTION
Tracklet Proposal Generation. Given a video, we first detect all the class-agnostic object tracklets using pre-trained tracklet detector, denote cally, each tracklet T; is characterized with bounding box sequence and the corresponding Rol Aligned (He al., 2017) visual feature, To reduce the computational overhead, we average the Rol features of all bounding boxes (i.e., along the temporal axis of the tracklet) following (Shang et al., 2021), 1), and denote as ʃi 2044
(r as shown in Figure 3(a). Specifi-
Tracklet Classification. Instead of directly classifying object tracklets using VLM Eq. (2), we train a visual-to-language (V2L.) projection module (.(()) to further utilize the annotations of base classes. In particular, do(.) maps the Rol Aligned feature fi of each tracklet to the same semantic space Rd, i.e. vy = (fi).. Let to be the text embedding of object class c E COO The probability of tracklet T; being classified as class
be calculated
exp(cos(0), 12)/T)
pa(e = Ee Eeeep exp(cos| (v), (2)/T)
Vc E CO,
(3)
where T is temperature parameter for softmax.
4

---

Published as a conference paper at ICLR 2023
Training Objectives. To train the object tracklet classification module, assign base category labels to detected tracklets according to the IoU w.n. ground- truth tracklets. We call those tracklets with assigned labels as positive tracklets, otherwise negative tracklets. Note that two cases can be recognized as negative tracklets: 1) the content is background; and 2) the content contains a novel object category. For these negative tracklets, the prediction (from any negative tracklet) on each base class to be 1/ 1/CC,), i.e. unlike any base category. Therefore, the classification loss for positive and negative tracklets can be calculated as:
follow the loss used by Du et al. (2022) that forces
Cels-pos
Σ Σ (o=o7) log Pi(c), Lebsneg (Tpl TTET, CEC
Σ T.E CECD
Σ
log Pi(c), (4)
[T
(
where Tp and Tn are the sets of positive and negative tracklets, respectively (i.e. Tp UT = T), and of is the ground-truth label for the i-th positive tracklet. We empirically found (in Sec. 3.2) that using the above negative tracklet loss works better than using the loss with a unique "background" class (Zareian et al., 2021; Gu et al., 2021). Besides, following (Gu et al., 2021), we distill the knowledge from a pre-trained visual encoder to (o(-) by aligning oy to vs using 4 loss, i.e.
disan = (1/N) -0:l1
(5)
Therefore, the overall loss for object tracklet classification is Lets = Lets-pos + Lels-nog = ACdistill. where À is syper-parameter to weight the classification and distillation.
3.3 OPEN- VOCABULARY VISUAL RELATION CLASSIFICATION
Based on the classified object tracklets, we perform open- greabuoccyyllryy relation classification for each tracklet pair, as shown in Figure 3(b). First, we learn the prompt representations based on the pre- extracted visual embeddings of tracklet pairs, for which we introduce the compositional prompt rep- resentations and the motion-based prompt groups. Second, we utilize the pre-extracted Rol Aligned features to train visual-to-langee to-language (V2L) projection module based the learned prompt repre- sentations. Finally, for testing, we extract all predicate text embeddings and classify the predicates of each tracklet pair by using the Rol Aligned features and the trained V2L projection module.
Compositional Prompt Representations. The compositional prompt consists learnable prompt representations S. and O₂ (of predicate class e) for subject and object, respectively:
S si...... LL we), O, = [o ........ . we],
(6)
where 8; and og are the learnable context vectors and ih the fixed class token for predicate e (for E Cf in training phase and for all in testing phase). Then, the predicate text embedding to is generated by concatenating the two outputs of VLM given two prompts (respectively) as inputs, i.e.
(S.,,. (LM),(O.)), and to E Rd
(7)
Motion-based based Prompt Groups. We vary the prompt contexts based on the motion cues, i.e., the relative spatio-temporal motion patterns, between each pair of subject and object. specific, we take the generalized IoU Rezatofighi et al. (2019) (i.e., GloU) the metric to calculate the motion patterns. For each tracklet pair (T;, TT), we use a vector to represent a motion pattern:
(8) where Gi Gi, are the GloU between subject-object for the start and end bounding boxes of their temporal intersection, respectively, and 7 is threshold for GloU. This definition considers two perspectives: 1) whether the two tracklets are near or far (i.e., the first two terms Eq. (8)), and 2) whether they move toward away to each other (i.e., the third term of Eq. (8). Overall, we have motion patterns (cf. Sec. A.2 for more details) and build 6 prompt groups correspondingly. Each group consists its own compositional prompt representations S. and O₂ as defined in Eq. (6).
Mii sign((G - 7, Gi - 7, G, GI,)),, and mij E (+,-}3,
It's worth noting that we aim to build a framework for learning motion- based multi-mode prompts. The used GloU-basee approach (in our framework) is simple and intuitive way to calculate motion cues. This approach is not perfect, e.g., it is poor to capture the motion pattern of tracklets moving back and forth. We leave other fancier (motion capturing) approaches as future work.
Training Objectives. Based on the above definition, we train the prompt representations with vi- sual embeddings and relative position features. For simplicity, we show the training process of
5

---

Published as a conference paper at ICLR 2023
single group (and in the end, we derive the final loss by averaging the losses across all groups). For each tracklet pair T)), we first calculate its motion cue, select the corresponding prompt group, and extract the class text embeddings to for each predicate class c E C₁¹ Then, we take the pre-extracted visual embeddings U; and Uj. and concatenate them the pair's visual embedding vi,j les, Øy] E R Following Shang et al. (2021), we additionally compute the relative position feature between bounding boxes of T; and Tj denoted as f(, E TR: (ef. Sec. A.2 in the Appendix for more details). The predicted probability of predicate class e in this tracklet pair is thus
(c) Sigmoid + Opes (fi (2)), Yc E EC...
(9)
where 0pos projects fī to the same dimension as V.jj. Going through all base classes (in Cf)), the probability vector of tracklet pair (T, Tj) is generated and can be denoted as Pt.j. i.e. each dimension is calculated by Eq. (9). In the training time, we assign the predicate labels according to the IoU for each tracklet pair w.r.t the ground-truth tracklet pair by following Shang et al. (2021). We denote the sets of positive and negative tracklet pairs as Pp and Pa. respectively. Due to the multi-label setting of VidVRD, we use binary cross-entropy loss for relation classification. The ground truth for positive tracklet pair in Pp is a binary vector of dimension [C,L), denoted as Pi.- For those negative tracklet pairs in Pu: we optimize the probability of each base class to zero, L.e, the ground truth is an all-zero vector. The classification loss is thus calculated as
Cpred (1/1P;)) (
(EEE(P..PPP))) + (1/1Pa
BCE(P 0).
(10)
Training V2I, Projection Module. Once the prompt representations are learned, we train a visual- to-language (V2L. projection module to use Rol Aligned features (fi) training data, and get rid of VLM's visual encoder at inference time. Given the learned prompt representations, we pre-extract the predicate class text embeddings (denoted as {E for each prompt group and fix them. Formally, for each tracklet pair (T,, T2), we concatenate their Rol features as füjj [f., fy] E R44096, Then, we use a V2L projection module &p to project it to the same dimension as text embeddings. Similar to Eq. (9), the probability of predicate class e is predicted as
Pr. (c) = Sigmoid(cos((pp(ffii) + (pon) Eg),, Ve C,..
(11)
where &oos is the learned -tmporal projection layer and is fixed. Then, we apply the same loss as defined in Eq. (10), and compute the final total loss by averaging across all groups.
Discussions. Intuitively, we can train the prompt representations together with the V2L projection module 6p. and use the loss align (f(.)) to v..j. i.e., distill the knowledge from the pre-trained visual encoder to the V2I. module. We name this variant as RePro". We justify that our RePro works better than ReProt due to: 1) directly using the teacher (i.e., vr to train the prompt is intuitively better than using student (i.e. projected visual embedding), and 2) the distillation makes the V2L module focus too much on the static visual alignment, rather than the dynamic relation information learned in the prompt. In experiments, we empirically show the superiority of RePro over ReProt.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
Datasets. We evaluated our method on the VidVRD (Shang et al., 2017) and VidOR (Shang et al., 2019) benchmarks: 1) VidVRD consists of 1,000 videos, and covers 35 object categories and 132 predicate categories. We used official splits: 800 videos for training and 200 videos for testing. 2) VidOR consists of 10,000 videos, which covers 80 object categories and 50 predicate categories. We used official splits: 7,000 videos for training, 835 videos for validation, and 2, 2,165 videos for testing. Since the annotations of VidOR-test are not released, we only evaluated models on validation set.
Evaluation Settings. To build the open-VidVRR VidVRD setting, we manually split base and novel cate- gories by selecting the common object and predicate categories as the base split, and selecting the rare ones as the novel split. The detailed splits are given in Sec. A.8 of the Appendix. We trained the model on the triplet samples of both base object and predicate categories in the training set. During testing, evaluated the model two settings: 1) Novel-split: triplet samples with all object cate- gories and novel predicate categories, and 2) All-splits: triplet samples with all object and predicate categories, in the testing set of VidVRD (or the validation set of VidOR).

---

Published as a conference paper at ICLR 2023
Table I: Performance (%) of tracklet detection on objects with novel categories.
VidVRD-test
VidOR-val
Methods
Distillation
BG-Embd
R@5
R@ 10
R@5
R@ 10
ALPro (Li et al., 2022a)
41.38
53.81
34.26
41.72
#1
x
X
2.17
3.71
2.33
3.48
#2
32.43
33.36
7.58
12.37
RePro #3
V
43.84
53.00
12.61
16.85
#4
46.34
50.42
31.62
37.08
Metrics. We follow three standard evaluation tasks in scene graph generation (Zellers et al., 2018): 8): scene graph detection (SGDet), scene graph classification (SGCIs), and predicate classification (PredCIs). We apply these metrics to VidVRD: detected triplet is considered to be correct if there the same triplet tagged in the ground truth, and both subject and object tracklets have a sufficient volume IoU (e.g., 0.5) with the ground truth. Following the standard setting (Shang et al., 2017), we use mAP and Recall @ K (R@K, K=50,100) as evaluation metrics.
4.2 IMPLEMENTATION DETAILS
Tracklet Detector & Pre-trained VLM. We used the Faster-RCNN (Ren et al., 2015)00155asee VL model (Zhang al., 2021) detect frame-level object bounding boxes and extracted corresponding Rol Alinged features, and then adopted Seq-NMS (Han et al., 2016) to generate class-agnostic object tracklets. The VLnVL model was trained on out-of-domain image data without seeing any VidVRD data. For pre-trained VLM, we used ALPro (Li et al., 2022a), which was pre-trained on a wide range of video-language data, and learned the fine-grained visual region to text entity alignment.
Relation Detection Details. Following the popular segment-based methods (Qian et al., 2019; Shang al., 2017; 2021), we first detected visual relations in short video segments, and then adopted greedy relation association algorithm (Shang et al., 2017) to merge the same relation triplets. The detailed hyperparameter settings are left in Sec. A.3 of the Appendix.
4.3 EVALUATE OPEN-VOCARULARE OBJECT TRACKLET DETECTION
We evaluated the tracklet detection part of RePro on novel object categories, as shown in Table
Comparison to ALPro. A straightforward baseline to achieve open-vocabulary tracklet detection is directly applying the pre-trained VI.M (ALPro in our case) by inputting the tracklet regions into its visual encoder to perform classification, as in Eq. (2). However, this has significant computational overhead due to the heavy pipeline of ALPro's visual encoder. In contrast, our RePro requires much less computational cost, since we only use projection layer (i.e., do). We thus compare our RePro with the above ALPro baseline. The results in row #4 show that RePro can achieve comparable performances on both datasets, with the projection layer do.
Negative Tracklet Classification. How to model the negative sample is a key challenge as widely discussed in many open- vocabuaayy object detection works. There are usually approaches: 1) using a unique background embedding (BG-Embd) in addition to the class text embeddings (Zareian et al., 2021; Gu al., 2021), and 2) only using the class text embeddings, and computing the loss of negative sample as Lels-neg in Eq. (4) (Gu et al., 2021). By comparing rows #3 and #4 of Table we find that without using background embedding (i.e., as Lets-neg) achieves better recall, and outperforms the other by large margin, especially on the more challenging VidOR benchmark. This because the tracklets recognized as negative may be due to the fact that they contain novel objects (rather than backgrounds), and aligning their embeddings (i.e., different novel class embeddings) to a unique background embedding hurts the model's 's recognition ability on novel objects.
Distillation. We verified the effectiveness of visual distillation (i.e,, Eq. (5)) by comparing rows #2 and #3 of Table 1. Obviously, the distillation helps RePro improve the detection recall by a large margin, especially for the more challenging VidOR benchmark. For row #1, we can observe that computing the negative tracklet classification loss as Les-nog without distillation has extremely low performance. This is because forcing the classification probability of negative tracklet to be 1/ CC,) (i.e., by Lets and without the guidance from the teacher (i.e., without distillation) make the model has poor generalize ability novel categories.
7

---

Published as a conference paper at ICLR 2023
Table 2: Performance (%) comparision to conventional methods on VidVRD-test. Relation Tagging
(RelTag) only considers the precision of relation triplets and ignores the localization of tracklets.
SGDet
RelTag P@5
Methods
Training Data
mAP
R@50
R@ 100
P@
P@ 10
Su et al. (2020)
base+novel
19.03
9.53
10.38
57.50
41,40
29.45
Liu et al. (2020)
base+novel
18.38
11.21
13.69
60.00
43.10
32.24
Li et al. (2021)
base+novel
22.97
12.40
14.46
68.83
49.87
35.57
Gao et (2022)
base+novel
17.67
9.63
11.29
56.00
43.80
32.85
RePro (Ours)
base
21.33
12.92
15,94
59.00
41.09
28.87
RePro (Ours)
base+
25.55
13.83
17.33
62.50
45.80
32.05
Table 3: Performance (%)
comparision of ViennDidVRDD methods on VidVRD-test.
SGDet
SGCIs
PredCIs
Split
Methods
mAP
R@50
R
100
mAP
R@ 50
R@ 100
mAP
R 50
R@ 100
ALPro
1.05
3.14
4.62
3.69
7.27
8.92
4.09
9.42
10.41
VidVRD-II
3.57
8.59
12.39
5.70
13.22
18.34
7.35
18.84
26.44
Novel
ReProt
2.56
8.26
11.73
8.63
15.04
18.84
9.34
18.67
24.13
RePro
6.10
13.38
16.52
10.32
19.17
25.28
12.74
25.12
33.88
ALPro
3.20
2.62
3.18
3.92
3.88
4.75
4.97
4.50
5.79
VidVRD-II
12.74
9.90
12.59
17.26
14.93
19.68
19.73
18.17
24.90
All
RePro
16.21
11.14
14.56
22.37
16.83
21.71
25.43
21.36
28.04
RePro
21.33
12.92
15.94
30.15
19.75
25.00
34.90
25.50
32.49
4.4
EVALUATE OPEN-VOCABULAR RELATION CLASSIFICATION
The relation classification part of our RePro was trained separately by keeping the results of tracklet detection fixed. All of our experiments for relation classification used the same tracklet detection results (which is row #4 in Table 1).
Comparison to Conventional VidVRD SOTA Methods. We compared our RePro with several SOTA methods in the conventional VidVRD setting, and showed the results in Table 2. The object tracklets and features used in SOTA methods are not uniform since VidVRD a very challenging task (see Sec. A.6 for details). We can observe that even when our RePro is trained with only base category samples (while others are with both base and novel category samples), our performance on SGDet tasks is comparable to others'. When trained with both base and novel category samples, our RePro outperforms all other SOTA methods in all SGDet tasks and most RelTag tasks.
Comparisons in the Setting of Viden-ViVVRD We compared the model performances in the set- ting of Open-VidVRD VidVRD and showed results in Table 3. Since our RePro is the first Open- VidVRD method, we compared to ALPro (implemented Eq. (2)). We also re-implemented the SOTA method VidVRD-II (Shang et al., 2021) and trained on base category samples. We replaced its classifier with text embeddings extracted by ALPro's text encoder. For both ALPro and VidVRD-III we used a fixed (handcrafted) prompt "a video of relation [CLASS]". In addition, we reported the results of RePro's intuitive variant RePro! as mentioned the "Discussion" of Sec. 3.3.
From the results in Table 3, we can observe that our RePro outperforms ALPro, VidVRD-II and ReProt by a large margin on both Novel-split and All-splits. By comparing RePro ALPro, we show that, unlike that in tracklet classification. directly applying pre-trained VLM to relation clas- sification sub-optimal and achieves poor performance. By comparing RePro to VidVRD-II, we demonstrate the superiority of our prompt tuning framework over the fixed prompt design. By com- paring RePro to ReProi, we validate the effectiveness of our training scheme for RePro.
4.5 ABLATION STUDIES
We conducted careful ablation studies shown in Table 4. Since the compositional and motion- based prompt design one of our main contributions, we conducted ablations w/o either of them (rows #1, #2 and #5). To further show the effectiveness of our motion pattern design, we designed two variants, i.e., rows #3 (Ens) and #4 (Rand). Their detailed settings are enumerated as follows: #1: It learns single prompt representation W. as in Eq. (1). The obtained predicate text embedding has the half dimensions of Eo Eq. (6). So we calculated the visual embeddings a tracklet pair Da.j v. Di (different from the concatenated vector Uijj in Eq. (9)). #2: Training with
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablations (%) for RePro with different prompt design in VidVRD-test, where C stands for Compositional, and M stands for Motion cues. Ens: ensemble all the learned prompts by averaging their representations. Rand: reandomly select prompt without considering motion cues.
SGDet
SGCIs
PredCIs
C
M
mAP
R@ 50
R@ 100
mAP
R@50
R
100
mAP
R@ 50
R@ 100
#1
3,50
9.91
13.88
7.21
14.54
19.83
8.63
20.33
27.43
Novel-split
#2
V
X
5.57
11.40
14.87
10.31
16.52
21.81
11.83
22.31
30.90
#3
Ens
6.24
11.57
15.20
10.77
16.03
21.98
12.36
21.32
29.91
#4 V
Rand
7.14
11.90
14.87
10.85
16.52
23.30
12.42
22.64
30.90
#5
6.10
13.38
16.52
10.32
19.17
25.28
12.74
25.12
33.88
#1
X
19.73
12.26
15.36
26.80
18.24
23.06
30.80
23.70
30.42
All-splits
#2
X
18.47
11.95
15.28
25.52
18.13
23.12
29.45
23.39
30.17
#3
Ens
20.15
12.38
15.61
27.93
18.61
23.55
31.68
23.61
30.29
#4 Rand
21.72
12.71
15.78
29.15
19.15
24.13
33.11
24.38
31.49
#5
21.33
12.92
15.94
30.15
19.75
25.00
34.90
25.50
32.49
compositional prompt but without motion cues. #3 & #4: Training with compositional prompt, but the prompt is randomly selected from the groups without considering motion cues. For testing, the prompts are ensembled by averaging (Ens) or randomly selected (Rand). #5: The proposed RePro.
Compositional Prompt. By comparing the results in #1 and #2, we can observe that the compositional prompt can effectively improve
Table 5: Ablations (%) on VidOR-val.
SGCIs
PredCIs
the performance Novel-split. Meanwhile,
C
M
R@50
R@ 100
R@50 R@ 100
the improvement on All-splits is not signifi-
cant. We conjecture that the base relations,
All-splits Novel-split
X
X
0.86
0.86
2.30
2,88
X
1.72
2.59
6.62
8.06
the majority of All-splits, require less compo- sitional semantic contexts for prompt learning.
Ens
2.30
2.59
7.20
8.35
Rand
2.01
2.30
5.76
7.20
Motion- based Prompt Groups. By compar- ing our RePro (#5) vs. #2, we can observe that with the help of motion cues, our RePro achieves significant improvements of recall on
V
2.01
2.30
7.20
8.35
X
X
9.49
12.85
25.62
34.83
X
10.06
13.40
27.00
36.73
Ens
9.49
12.68
25.66
35.16
all tasks, and also achieves considerable im- provements in mAP on most tasks. We can see Novel-split is more
Rand
10.03
13.13
26.94
36.48
10.03
12.91
27.11
35.76
that the improvement significant than that on All-splits, e.g., 14.87%-+16.52%% vs. 15.28%-+15.94%% on R@@ 100 of SGDet, showing that the motion-based prompt has a better generalizability for detecting novel relations. Be- sides, if comparing RePro (#5) to Ens (#3), we can see that RePro outperforms Ens in Novel-split on most tasks, and achieves considerable improvements for All-splits on all tasks. Compared to Rand (#4), RePro achieves clear improvements on most metrics for both Novel-split and All-splits.
Ablations VidOR. We conducted the same ablation studies on VidOR-val, shown in Table 5. Firstly, we can observe that the compositional prompt representation shows its efficiency on both Novel-split and All-splits, e.g., 1.86% and 90.06% motion-based prompt groups, the improvement of RePro is small due to the biased data distribu- tion (Li et al., 2021), i.e., the predicate categories strongly depend on the visual cues of subject and object tracklets, making the model predict relations simply based on object appearances without considering motion cues. More results on VidOR are left in the Appendix (Sec. A.7).
R@ of SGCIs. For the
5 CONCLUSIONS
In this paper, we introduced the challenging Open- Vide-VidVRD task. We analyzed two key charac- teristics, iee, compositional and motion-related, when applying prompt tuning in this new task. We proposed a novel method called RePro that learns compositional prompt representations while considering motion-based contexts. Our evaluations on both conventional and open-vocabulary datasets show a clear superiority of RePro for tackling video visual relation detection tasks.
Acknowledgement: This work was supported by the National Key Research & Development Project of China 2002ZD011070 the National Natural Science Foundation of China (U19B2043, 61976185), and the Fun- damental Research Funds for the Central Universities (22-2002200055).. This work was also supported by A STAR under its AME YTRG grant (Project No. A20F6c0101), and Singapore MOE Tier 2.

---

Published as a conference paper at ICLR 2023
ETHICS AND REPRODUCIBILITY STATEMENTS
Ethics statement. The open-vocabulary video visual relation detection (Open-VidVRD) that we introduced in this paper is a general extension of conventional VidVRD, and there are no known extra ethical issues in terms of the Open-VidVRR the pre-trained languae model (VLM) used for Been-VidRD,, the large-scale pre-training data might contain some videos and captions involved with discrimination/bias issues. When ap- plying pre-trained VLM to Open-VidVRD. VidVRD, the model tends to predict relations based more on the pre-trained knowledge, and focus less the visual cues. For example, when the pre-trained data involved with unethical videos and captions, model might predict "person-punch-dog" given a video of person caressing dog, which implies a person is abusing animals. To avoid the potential ethical issues, we can design algorithms to filter out those unethical training data for VLMs. For Open-VidVRD VidVRD models, rule-based methods to filter out those unreasonable relation triplets that involve ethical issues.
task and the proposed RePro model. As for
also introduce some common sense knowledge and design some
Reproducibility Statement. Our RePro is mainly implemented based the realsed code of AL- Pro (L.i et al., 2022a), VInVI. (Zhang et al., 2021), and VidVRD-III (Shang et al., 2021). We first modified the code of Vin VL to fit the video data and extract object tracklets in each video. Then We modified the code of VidVRD-II to be compatible with the visual and text encoder of ALPro, and setting. We provide the the detailed base/novel split information of object
fit the Open-VidVRR and predicate categories in the Appendix (Sec. A.8) to ensure all experiments can be reproduced. When training the RePro model and its variants, we manually set the random seed and fixed the seed for all experiments to ensure they can be reproduced. We also provide the code of our RePro model and the training/evaluate scripts the supplementary materials.
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, pp. 0299-6308, 2017.
Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counter- factual critic multi-agent training for scene graph generation. In ICCV, pp. 4613-4623, 2019.
Shuo Chen, Zenglin Shi, Pascal Mettes, and Cees G. M. Snoek. Social fabric: Tubelet compositions for video relation detection. In ICCV, 2021.
Siqi Chen, Long Chen, and Jun Xiao. Video scene graph generation from single-frame weak super- vision. In ICLR, 2023.
Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 16372-16382, 2021.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- open-vocabulary object detection with lasion-language model. In CVPR. pp. 14084-14093, 2022.
Chengjian Feng, Yujie Zhong, Zequn Jie, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards open- open-vocabulary detection using uncurated images. In ECCV, 2022.
Kaifeng Gao, Long Chen, Yifeng Huang, and Jun Xiao. Video relation detection via tracklet based visual transformer. In ACM MM, pp. 4833 4837, 2021.
Kaifeng Gao, Long Chen, Yulei Niu, Jian Shao, and Jun Xiao. Classffication-tee--rooooe formulating video scene graphs as temporal bipartite graphs. In CVPR, pp. 19497-19506, 2022.
Re-
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2021.
Wei Han, Pooya Khorrami, Tom Le Paine, Prajit Ramachandran, Mohammad Babaeizadeh, Honghui Shi, Jianan Li. Shuicheng Yan, and Thomas S Huang. Seq-nms for video object detection. arXiv preprint ar /602 160..00655. 2016.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In European Conference on Computer Vision, pp. 56- 73. Springer, 2022.
Peiliang Huang, Junwei Han, De Cheng, and Dingwen Zhang. Robust region feature synthesizer for zero-shot object detection. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 7622-7631, June 2022.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. In ACL, pp. 2763-2775, 2022.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, pp. 105-124. Springer, 2022.
Xuan Kan, Hejie Cui, and Carl Yang. Zero-shot scene graph relation prediction through common- sense knowledge integration. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 466-482. Springer, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv, 2014.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video-and lann-uuge pre-training with entity prompts. In CVPR, pp. 4953-4963, 2022a.
Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR, pp. 18869-18878, 20226.
Lin Li, Long Chen, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Jun Xiao. Nicest: Noisy label correction and training for robust scene graph generation. arXiv preprint arXiv:2207.133: 13316, 2022c.
Xingchen Li, Long Chen, Wenbo Ma, Yi Yang, and Jun Xiao. Integrating object-aware and interaction-aware knowledge for weakly supervised scene graph generation. In ACM MM, pp. 4204-4213, 2022d.
Xingchen Li, Long Chen, Jian Shao, Shaoning Xiao, Songyang Zhang, and Jun Xiao. Rethinking the evaluation of unbiased scene graph generation. In BMVC, 2022e.
Yicong Li, Xun Yang, Xindi Shang, and Tat-Seng Chua. Interventional video relation detection. In ACM MM, pp. 4091-4099, 2021.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. ECCV, pp- 388-404. Springer, 2022.
Chenchen Liu, Yang Jin, Kehan Xu, Guoqiang Gong, and Yadong Mu. Beyond short-term snippet: Video relation detection with spatio-temporal global context. In CVPR, pp. 10840-10849, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586 2021.
Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical languaga knowledge distillation. In CVPR, pp. 14074-14083, 2022.
Sauradip Nag, Xiatian Zhu, Yi-Zhe Song. and Tao Xiang. Zero-shot temporal action detection via vision-language prompting. In ECCV. Springer, 2022.

---

Published as a conference paper at ICLR 2023
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-iman pretrained models for general video recog- nition. In ECCV, pp. 1-18. Springer, 2022.
Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao. Video relation detection with spatio-temporal graph. In ACM MM, pp. 84-93, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, Advances in neural information processing systems, 28, 2015.
Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xindi Shang, Tongwei Ren, Jingfan Guo. Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In ACM MM, pp. 1300-1308, 2017.
Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. ICMR, pp. 279-287, 2019.
Xindi Shang, Yicong Li, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Video visual relation detection via iterative inference. In ACM MM, pp.
2021.
Zixuan Su, Xindi Shang, Jingjing Chen, Yu-Gang Jiang, Zhiyong Qiu, and Tat-Seng Chua. Video relation detection via multiple hypothesis association. In ACM MM, pp. 3127-3135, 2020.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3716-3725, 2020.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with deep association metric. In 20/7 IEEE international conference on image processing (ICIP), pp. 3645- 3649. IEEE, 2017.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, pp. 14393-14402 2021.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. CVPR 2021, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, pp. 1-12, 2022b.
A
APPENDIX
This Appendix has the following contents:
More details about the relative position feature are in Sec. A.I.
More details about the motion pattern in Sec. A.2.
12

---

Published as a conference paper at ICLR 2023
More details about the hyperparameters are given in Sec. A.3.
Analysis of the performance improvement in different predicate groups are in Sec. A.4.
Potential improvements of the motion pattern design are introduced in Sec. A.5.
The detailed experiment settings the compared SOTA methods are introduced in Sec. A.6.
More experiment results on VidOR are provided Sec. A.7.
The detailed basc/novel split information of object and predicate categories are in Sec. A.8.
A.1 RELATIVE POSITION FEATURE FOR TRACKLET PAIRS
We compute the relative position between bounding boxes of subject-object tracklet pair (T;; T3) by following Shang ct al. (2021). Specifically, we compute the relative position feature between subject-object for the beginning and ending bounding boxes of their temporal intersection. For the beginning bounding boxes of the subject and object, the position feature is calculated as:
2;
23
Yi
M/j
W.
h4g h
Wihh t, - tz
log
log
log
(12)
Ij
M/
wyht
Liseg
where (z,, 81) is the central coordinates of T''s beginning bounding box, (wy, h2) is its width and height, and f2 is the frame of this beginning bounding box. (I), My. wy. hy, ty) is defined similarly for T.. Legg is the number of frames in each video segment, and following Shang et al. (2021), we set Tseg = 30. The relative position feature between the ending bounding boxes of (T;, T,) is defined similarly as f.f and is denoted as f. The final relative position of (T; Tÿ) is concatenated as
fi. [f.. f.g]. and fig E R12
A.2 DETAILS ABOUT THE MOTION PATTERNS
We provide schematic of the motion patterns defined Eq. (8), as shown in Figure According to the definition in Eq. (8), mij has only 6 possible values, i.e., [+, -, and [-, are impossible.
frame
end frame
frame
frame
may [+, +, "lose, close, towards"
my [+, +,+
subject
"close, close, away"
object box
my [-, +, +]
[+---]
"far, close, towards"
"clase, far, away"
myj [-.--]] "far, far, away"
my [--,+]] towards"
Figure 4: The schematic the 6 motion patterns defined in Eq. (8).
A.3 HYPERPARAMETERS
We set do. &p and Opu all two-layer MLPs with hidden dimension 768. The À for weighting the distillation loss (Le., Eq. (5)) was set as 5.0. The prompt length was set as 10. The softmax temperature was set learnable. The GloU threshold was chosen based the statistics of the training set, by making the tracklet pairs evenly distributed w.r. different motion patterns. In our implementation, 7 was as -0.3 for VidVRD and -0.25 for VidOR. We trained our RePro using Adam (Kingma & Ba, 2014) with a learning rate le-4, and stopped the training when SGDet mAP drops.
A.4 ANALYSIS OF THE PERFORMANCE IMPROVEMENT IN DIFFERENT PREDICATE GROUPS
We evaluated the Recall@ @ 100 in the PredCls setting of some predicate groups (grouped by the prefix of predicate words) at the novel-split of the VidVRD dataset. We compared our RePro with the mean ensemble (Ens) and random select (Rand) variants of RePro (refer to Sec. 4.5). The results
13

---

Published as a conference paper at ICLR 2023
show that the improvements of motion-related predicates much larger than other context-related predicates. For example, we have 9.68% absolute improvements on "run" (e.g.. "run past", "run next to"). While for those predicates that can be roughly inferred by the context (e.g., "fly", "swim"), our approach has limited contributions. This indicates that the performance improvements of our RePro are largely attributed motion cues.
Table 6: Recall @ 100 (%) of PredCIs on the test set of VidVRD w.r. different predicate groups.
Methods Ens
move
sit
walk
stop
stand
fly
swim
34.48
50.92
12.90
18.30
37.03
35.51
37.50
15.38
Rand
37.93
51.85
16.12
18.30
44.44
36.44
50.00
15.38
RePro
44.82
55.55
25.80
18.95
40.47
41.12
50.00
12.82
A.5 POTENTIAL IMPROVEMENTS OF THE MOTION PATTERN DESIGN
The current proposed GloU-based motion pattern design can be further improved. Based on our pro- posed motion-pattem based prompt group learning framework, we can design other fancier motion capturing approaches, e.g. automatically learning the motion primitives from the training set. Then for each test sample, the motion pattern can be decomposed as the weighted combination of motion primitives. Consequently
the weighted combination of the prompt representations the desired prompt representation. We leave this as future work.
A.6 DETAILED EXPERIMENTAL SETTINGS OF THE COMPARED SOTA METHODS
Since VidVRD is very challenging task, The object tracklets and features used SOTA methods in Table 2 are not uniform. The object tracking algorithms for tracklet generation include Seq- NMS (Han al., 2016) and deepSORT (Wojke et al., 2017). The features include Rol Aligned features, I3D features (Carreira & Zisserman, 2017), and improved dense trajectory (iDT) fea- tures (Shang al., 2017) Here we enumerate their details as follows:
Su et al. (2020) uses Seq-NMS for tracklets generation, and improved dense trajectory (iDTT) feature and relative motion feature of tracklet pairs for relation classification.
Liu et al. (2020) uses deepSORT for tracklets generation, and uses Rol feature, I3D feature, and relative motion feature of tracklet pairs for relation classification.
Li at. (2021) uses Seq-NMS for tracklets generation, and uses Rol feature and relative motion feature of tracklet pairs for relation classification.
Gao ct al. (2022) uses deepSORT for tracklets generation, and uses Rol feature, 13D feature for relation classification.
Our RePro uses Seq-NMS for tracklets generation, and uses Rol feature and relative motion feature of tracklet pairs for relation classification.
A.7 MORE EXPERIMENT RESULTS ON VIDOR
Table 7: Performance (%) on the validation set of VidOR.
Novel-split
All-splits
Methods
SGCIs
PredCIs
SGCIs
PredCls
R@50
R@ 100
R@50
R@ 100
@@50
R@ 100
R@
R@ 100
ALPro
3.17
3.74
8.35
9.79
0.95
1.32
2.61
3.66
VidVRD-II
1.44
2.01
4.32
4.89
9.40
12.78
24.81
34.11
RePro!
1.72
2.30
6.62
8.06
8.88
11.52
23.84
31.57
RePro
2.01
2.30
7.20
8.35
10.03
12.91
27.11
35.76
We provide more experiment results for our RePro on the validation set of VidOR, shown in
Table 7. We first compare our RePro with using ALPro directly perform relation classification as
Eq. (2). We find that AL.rro performs slightly better than RePro novel-split, because Al.Pro
14

---

Published as a conference paper at ICLR 2023
doesn't have the trend of fitting base categories. However, ALPro performs much worse than Re- Pro in All-splits due to not trained on base categories. Furthermore, we also compare RePro with the VidVRD-II (Shang al., 2021) baseline and the variant RePro'. We can observe that RePro outperfoms both VidVRD-II and ReProt by large margin on both novel-split and all-split.
A.8 DETAILED BASE/NOVEL CATEGORIES OF OBJECT AND PREDICATE FOR VIDVRD AND VIDOR
We list the base/novel categories of object and predicate for training and evaluating our RePro and other baselines all experiments. We also provide more statistics information in the supplementary materials
VidVRD Object
25 base object categories:
"airplane", "bicycle", "bird", "bus", "car", "dog", "domestic.cat", "elephant", "hamster", "lion", "monkey", "rabbit", "sheep", "snake", "squirrel", "tiger", "train", "turtle", "whale", "zebra", "ball", "frisbee", "sofa", "skatcboard", "person"
10 novel object categories
"horse", "watercraft", "giant panda", "fox", "red panda", "cattle", "motorcycle", "bear", "antelope", "lizard"
VidVRD Predicate
71 base predicate categories:
"behind", "chase", "creep_behinc"" bereepbbnneath, "'reep_front", "creep left", right,, "fall_off", "faster", "fly_above", "fly _ext_to""" "fly "fly-past", "fly_towan ttwwrrd". "fly_with", "follow", "front", "jump_ beneath", "jump front", "jump left", "jump_next.to" next_to". "jump_front"
Soump-right" "jump "jump.toward" toward", "larger". "left", "lie_ behind", "lie front", "lie_ "lie next to", "lie right", "move behind", "move beneath". "move front", "move left", "move right", "move with", "next to", "play", "ride", "right", "run behind", "run front". "ruu_Jeft", "run past", "run_right", "run_with", "sit_above", "sit_front", "sit_left". riiht" "stand.behind"
fromtt, "eff" "stand_nextLo"" "stand "stop_ behind", "stop_front" "stopleftt, "stop_right", "swim_front", "swim_left", "swim right", "swim with,, "taller", "touch", "walk behind", "walk front", "walk left", "walk next. to", "walk right", with", "watch"
61 novel predicate categories:
"above", "away", "beneath", "bite", "creep_above", "creep_away" "crree_nexx_to" "creep-past", "creep_toward" "drive", "feed", "fight", "fly "way", behind", froot",,"" left_' "fly_right" "hold", "jump_above" "jump_away away". jjump_behind" "jump-past", "jump-with", "kick", "lie above"" "lie "lie beneath" "lie "lie inside", "lie_with", "move_above". "move_away", "moveenext.to". "move_past
"move_toward". "past", "pull", "rum above", "run away", "run beneath", "run next to". "run toward". "sit behind". "sit beneath", "sit inside", "sit next to", "stand above", "stand beneath", "stand "stand_inside". "stand_wing with"" "stop_above "swim behind", iswim_beneatee "swim nexttto. "toward", above, "walk beneath"" "walk_past" "walk toward"
sstopbenneat
"stop_withh
VidOR Object
50 base object categories:
"adult", "child" "toy", "dog", "baby". "car", "chair", "table", "sofa", "ball/sports. ball", "screen/monitor"" "cup". "bicycle". "guitar", "bottle". "backpack", "handbag", "baby seat", "camera", "cat", "cellphone", "bird", "sheep/goat", "laptop", "ski", "stool", "watercraftt. "duck", "bus/truck", "bench", "fruits" "baby walker", "horse". "bat". "dish",
15

---

Published as a conference paper at ICLR 2023
"electric_far "kangaroo", "motorcycle", "lion", "hamster/rat", "refrigerator", "elephant", "faucet", "cake", "penguin", "sink", "piano", "microwave", "cattle/cow", "aircraft"
30 novel object categories:
"antelope", "vegetables", "panda", "rabbit", "fish", "train", "snowboard", "suitcase", "squirrel", "leopard", "chicken", "skateboard", "traffic light", "surfboard", "camel", "racket", "bread", "bear", "oven", "scooter", "frishee", "stop_sign", "turtle", "stingray", "pig", "crab", "crocodile", "toilet", "tiger", "snake"
VidOR Predicate
30 base predicate categories:
"next_to", "in of "watch", "behind", "away", "towards", "beneath", "above", "hold", "lean_on" "speak_to", "ride", "hug", "touch", "carry", "hold_hand. "bite", "push", "pull", "play(instrument", "grab", "release", "pat", "inside", "lift", "caress", "point to",, "press", "hit", "use"
20 novel predicate categories:
"kick", "chase", "wave", "smell", "throw", "feed", "kiss", "wave hand to",, "shout at", "drive", "clean", "lick", "squeeze", "shake hand with", "get off", "knock", "cut", "open", "get_on", "close"
16


Reviews:
[{"review_id": "y33Xtdswomf", "reviewer": "Reviewer_QLPW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper considers the problem of video visual relation detection under an open-vocabulary scenario.\n\n- The main contribution is on prompt learning to adapt the strong visual-language model towards downstream tasks.\n\n- To enable the open-vocabulary, the prompt learning has exploited two techniques, namely, the compositional prompt representation, and motion-based prompt groups, which makes sense.\n\n- The idea has been evaluated on various datasets, and show clear performance improvements.", "summary_of_the_review": "Overall, I think I like this paper, I like the motivation for solving downstream video tasks with prompt learning, the idea of using composition prompt generation for predicate is good, and exploiting motions makes sense in video understanding tasks.\n\nHowever, the paper writing needs a lot improvement, now it's very hard to grasp the key ideas, too many notions, and unnecessary equations.\n", "strength_and_weaknesses": "Strength\n\n- The idea is simple, using compositional prompt and exploiting motion information do generate prompt do make sense in the video visual relation detection task.\n\n- The results have clearly validate the claims, demonstrating state-of-the-art performance under various settings.\n\nWeakness:\n\n- The writing is not satisfactory, it can surely be better.\n\n- Missing many references from ECCV: \n[1] https://arxiv.org/abs/2112.04478\n[2] https://arxiv.org/pdf/2208.03550\n[3] https://arxiv.org/abs/2208.02816\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bgaH2nMlmF", "reviewer": "Reviewer_Fty6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a new task called Open-vocabulary VidVRD, where the aim is to detect not only unseen relationships but also unseen objects and predicates. They propose a novel pipeline (i.e. RePro) to detect tracklets and classify the relations between tracklets. In particular, the proposed approach is a compositional and motion-based relation prompt learning framework. In the experiments, the authors show rich experiments, including various ablation studies. ", "summary_of_the_review": "The paper introduces an interesting task and a framework to solve the task.\nSome parts are still unclear, and this reviewer left the questions in the Weaknesses section. ", "strength_and_weaknesses": "**Strengths:**\n\n* The paper is well written. \n\n* The authors have introduced an interesting problem and proposed a novel solution for the problem. Further, they showed rich experimental results. \n\n**Weaknesses:**\n\n* What is the difference between visual embeddings ($\\mathbf{v}_i^O$) and RoI Aligned features ($\\mathbf{f}_i$) in Sections 3.1 and 3.2?  \n\n* Figure 3 mix-uses texts (e.g. \"dog\", \"child\") for both visual features and categories. \n\n* The experimental settings in Section 4.4 is not clear. Did the authors use the same tracklets for the comparisons in Tables 3 and 4? This reviewer is unsure if the comparisons are valid. \n\n* Typos:\n    * \"we distil the knowledge ...\" in Line 172\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VKBTmjOojRn", "reviewer": "Reviewer_K5Cm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an interesting prompt tuning method for video relation detection. It includes compositional and motion related cues. \n\nThe authors also propose a new task, called Open VIdVRD, which further considers unseen relationship combinations and unseen objects/motions.", "summary_of_the_review": "Overall the proposed method is novel and interesting. However, the design of motion cues seems limited in current stage. It still requires hand-craft effort or expert knowledge.\n\nThe introduced task is a good addition to VidVRD dataset. Results on open-vocabulary setting are good compared to baseline variants.", "strength_and_weaknesses": "Strengths:\n1. The authors propose to add the composition capability and motion cues to the prompt, making it more robust and less biased for video relation detection. The idea is new.\n2. The design of multi-mode (multiple different motion patterns) prompt groups is interesting.\n3. The paper is well-written and easy to follow. \n4. The authors conducted several experiments to verify the effectiveness of their method. It shows comparable results when comparing to VidVRD SOTA. In their ablation study, their open-vocabulary-based results are better than the baselines.\n\nWeaknesses:\n1. The motion pattern design seems to be complex and not easy to handle a variety of motions. While the use of GIoU is nice, the current design considered only toward/away and near/far. It is difficult to extend to generic motions with GIoU. Since the current design of motion group is very limited, the strong performance improvements are less convincing to be attributed to motion cues.\n2. In the comparison with traditional VidVRD, one may wonder the performance of the proposed model if adding novel samples.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8NQelnShjn", "reviewer": "Reviewer_Nj5q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach to video visual relation detection that is based on extension of compositional prompt tuning with motion cues. The Relation Prompt (RePro) is designed to address the technical challenges of Open-vocabulary Video Visual Relation Detection: the prompt tokens should respect the two different semantic roles of subject and object while the tuning should account for the diverse spatiotemporal motion patterns of the subject-object compositions. They achieves a new state-of-the-art performance on two VidVRD benchmarks of not only the base training object and predicate categories, but also the unseen ones.", "summary_of_the_review": "This paper proposes a new method for several technical challenges of Open-vocabulary Video Visual Relation Detection, and also provides useful prompt group that can be generalized. The paper shows the video visual relation detection is generalizable with the proposed framework. The proposed video visual relation detection task is very interesting, but I would appreciate if the author can clarify the evaluation and motivation behind the design. Overall, I lean toward acceptance.", "strength_and_weaknesses": "Strength\n- They proposes a novel setting for video visual relation detection task, Open-VidVRD.\n- They clearly describe interesting idea in this paper. The compositional prompt representation learning method that models the prompt contexts for subject and object separately. the motion-cue-based prompt groups shows good generalization ability on the tasks.\n\nWeakness\n- The novelty is limited. The paper claims that the first one to research on the the open vocabulary video visual relation detection, but the open vocabulary VRD has been widely studied in visual domain, which potentially could be compared.\n- The motivations in evaluation section, and ablations of the proposed approach is not easy to follow.\nIt's also hard to follow how the proposed approach improves the performance on the Open-VidVRD benchmark. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster