R^2-VOS: Robust Referring Video Object Segmentation via Relational Cycle Consistency
Authors: Xiang Li
Jinglu Wang
Xiaohao Xu
Xiao Li
Yan Lu
Bhiksha Raj
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Referring video object segmentation

Abstract: Referring video object segmentation (R-VOS) aims to segment the object masks in a video given a referring linguistic expression to the object. It is a recently introduced task attracting growing research attention. However, all existing works make a strong assumption: The object depicted by the expression must exist in the video, namely, the expression and video must have an object-level semantic consensus. This is often violated in real-world applications where an expression can be queried to false videos, and existing methods always fail in such false queries due to abusing the assumption. In this work, we emphasize that studying semantic consensus is necessary to improve the robustness of R-VOS. Accordingly, we pose an extended task from R-VOS without the semantic consensus assumption, named Robust R-VOS ($\mathrm{R}^2$-VOS). The $\mathrm{R}^2$-VOS task is essentially related to the joint modeling of the primary R-VOS task and its dual problem (text reconstruction). We embrace the observation that the embedding spaces have relational consistency through the cycle of text-video-text transformation which connects the primary and dual problems. We leverage the cycle consistency to discriminate and augment the semantic consensus, thus advancing the primary task. Parallel optimization of the primary and dual problems are enabled by introducing an early grounding medium. A new evaluation dataset, $\mathrm{R}^2$-Youtube-VOS, is collected to measure the robustness of R-VOS models against unpaired videos and expressions. Our method not only identifies negative pairs of unrelated expressions and videos, but also improves the segmentation accuracy for positive pairs with a superior disambiguating ability. The proposed model achieves the state-of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOS, and the novel $\mathrm{R}^2$-Youtube-VOS dataset.

R²-VOS: Robust Referring Video Object Segmentation via Relational Cycle Consistency
Anonymous Author(s)
Affiliation
Address
email
Abstract
Referring video object segmentation (R-VOS) aims to segment the object masks a video given a referring linguistic expression to the object. It is a recently introduced task attracting growing research attention. However, all existing works make a strong assumption: The object depicted by the expression must exist in the video, namely, the expression and video must have an object-level semantic consensus. This is often violated in real-world applications where an expression can be queried to false videos, and existing methods always fail in such false queries due to abusing the assumption. In this work, emphasize that studying semantic consensus is necessary to improve the robustness of R-VOS. Accordingly, we pose an extended task from R-VOS without the semantic consensus assumption, named Robust R- VOS (R2- -VOS. The R² VOS task is essentially related to the joint modeling of the primary R-VOS task and its dual problem (text reconstruction). We embrace the observation that the embedding spaces have relational consistency through the cycle of text-video-text transformation, which connects the primary and dual problems. We leverage the cycle consistency to discriminate the semantic consensus, thus advancing the primary task. Parallel optimization of the primary and dual problems are enabled by introducing an early grounding medium. A new evaluation dataset, Youtubue VOS, is collected measure the robustness of VOS models against unpaired vidcos and expressions. Extensive experiments demonstrate that our method not only identifies negative pairs of unrelated expressions and videos, but also improves the segmentation accuracy for positive pairs with superior disambiguating ability. Our model achieves the state-of-the-an of-the-art performance on Ref-DAVIS17, Ref-Youtube-VOR and the novel (2=Youttbe-Vor VOS dataset.
3
5
7
11
13
14
15
17
18
19
20
21
22
23
Introduction
24
25
Referring video object segmentation (R-VOS) aims to segment a referred object in a video sequence given linguistic expression. R-VOS has witnessed growing interest thanks to its promising potential in human-computer interaction applications such as video editing and augmented reality. Unlike
26
27
28 other video segmentation tasks 45
36
35
46
that only rely on visual cues, R-VOS [33 pairs
29
target video with linguistic expression referring to an object.
Previous works [ 44 tackle the R- R-VOS problem with strong assumption that the referred object exists in the video, i.e., there an object-level semantic consensus between the expression and the video. However, this assumption does not always hold in practice. As shown in Figure severe false-alarm problem experienced by previous methods when the semantic consensus does not exist, which may prevent those methods from being useful in various applications that cannot provide accurate vision-language pairs. We argue that the current R- R-VOS task is not completely defined with
31
notice
39
34
35
36 the assumption that the referred object always exists in the video.
37 Even when semantic consensus exists in the given video-lane Langu-la pairs, is still challenging to locate the correct object in the video due to the multimodal nature of the R-VOS task. Recently, MTTR [I]
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Query expression: "a kangaroo in the middle of two"
Positive
Retere objec
Negalive
Videos
Input frame
MITR
Beferformer
Ous
Illustration of the new R--VOS task. A linguistic expression is given to query set of videos without the semantic consensus assumption. Videos containing the referred object by the expression are positive, otherwise negative. Unlike the previous R-VOS setting that assumes all target videos are positive to the query expression, the new R task required to discriminate positive and negative text-video pairs, and further segment object masks for all frames in positive videos or treat entire negative videos as backgrounds. Compared to the previous state-of-the-ar R-VOS methods, MTTR [1 and ReferFormer [44], our method not only discriminates negative videos better but also shows a superior disambiguating ability between visually similar objects in positive vidcos.
Figure 1:
employs multimodal transformer encoder learn joint representation of the linguistic expression 40 and video, and then obtains the referred object by ranking all objects in the video. ReferFormer query to
follows the image-level method, ReTR [19]. to adopt the linguistic expression the transformer decoder to avoid redundant ranking of all objects. However, these latest methods suffer from semantic misalignment of the segmented object and the linguistic expression, even with sophisticated components employed. As shown in Figure [ the segmented objects by MTTR and ReferFormer are often not the object referred by the linguistic expression.
44
42
44
45
46 In this paper, we seek to investigate the semantic alignment problem between visual and linguistic modalities in referring video segmentation. We extend the current task definition of R-VOS 133 to accept both paired and unpaired video and language inputs. This new task, which we term Robust ROVOS (RS-VOS), overcomes the current limitation of the R-VOS task by additionally considering the semantic alignment of input video to referring expression. We reveal that this task is essentially related two problems that are interrelated [31] the R- R-VOS problem segmenting mask sequences from videos with referring texts, and its dual problem of reconstructing text expressions from videos with object masks. By linking the primary and dual problems, introduce a text-video-text cycle and a corresponding relational consistency constraint, which can enforce the semantic consensus between the text query and segmented mask to improve the primary task. In practice, naively conducting cyclic training of the text-vidco-text cycle will lead to two- stage regime and significantly increasing costs. We address this problem by incorporating an early grounding scheme, serving as proxy, to efficiently model the two tasks in parallel manner. In addition, we discriminate the semantic misalignment between the video and text by assessing the cycle consistency between the original and reconstructed texts, thus alleviating the false-alane problem. Our contributions can be summarized as:
48
50
51
the primary problem of
52
53
54
55
56
57
58
59
60
61
We notice severe aalse-alamn problem faced by previous R-VOS methods with unpaired inputs. To investigate the robustness of current referring segmentation models, we introduce the Ra-VOS task that accepts unpaired video and text as inputs.
63
64
We propose pipeline that jointly optimizes the primary referring segmentation and dual expression reconstruction task and introduces relational cycle consistency constraint to enhance the semantic alignment between visual and textual modalities.
86
68
Our method surpasses previous sfute-of-the-ae methods on Ref- Youtube- VOSe, Ref-DAVIS, and Youtube- VOS dataset in terms of both performance and speed.
89
2
Related Works
70
Vision and language representation learning. There have been a long line of studies on how to
72
learn better vision-language representation, e.g., multimodal attention
30
50
R 3, fusion scheme
7 14 15, 51), multi-step reasoning [47 [0 and pretraining [37 5
KAC Net (2) leverages

---

74
knowledge-aided consisteney constraints to enhance semantic alignment for weakly supervised phrase grounding. A structure- prreseriip constraint 442 proposed to preserve some intra-modal properties when learning vision-language representation for image-text retrieval.
75
76
Referring video object segmentation. R-VOS is novel task that aims segment object across frames given linguistic description. URVOS (39 is the first unified R-VOS framework with cross-modal attention and a memory attention module, which largely improves R- B-VOS performance. ClawCraneN [2 leverages cross-modal attention to bridge the semantic correlation between textual and visual modalities. ReferFormer 44 and MTTR are two latest works that utilize transformers
78
79
80
81
82
44 employs a linguistic prior to the transformer decoder to focus on the referred object. MTTR [ leverages a multimodal transformer encoder
to decode or fuse multimodal features. ReferFormer
83
84
to fuse linguistic and visual features. Different from other vision-language tasks, c.g., image-text 40], R-VOS needs to construct object-level
85
retrieval 25 32 and video question answering [8 multimodal semantic consensus in dense visual representation.
86
3
R--VOS
87
3.1
Task Definition
89 We introduce a novel task, robust referring video segmentation (R²-VOS), which aims to predict mask sequences {M,} for unconstrained video set {V} given language expression E, of an
90
81 object Different from the previous R-VOS setup, the queried video V is not required to contain
92 the referred object by expression E.. We define video V and an expression E, to have semantic
93 consensus if the object appears in V, and the video is positive with respect to E.. otherwise it is
94 negative. The R2-VOL task extended discriminate positive and negative videos, and predict
95 masks Mo of object for positive videos and treat all frames in the negative videos as background.
3.2
Problem Analysis
V
P: Referiin segmentation
Mo
V
N
M
Space 8
Space E
Space
Space E'
f
m
Eg
E
Eg
1
e
dec
E'
reconstraction
Cycle consislency
Cycle consislency
Primury and Dual problems
(b) Joint oplimization P.D problems
Point-wise consistency
Relational consistency
Figure 2: Problem analysis. (a) R²-VOS introduces the Primary problem of referring segmentation and the Dual problem of text reconstruction for positive videos. The P/D problems are connected in cycle path from original expression E, to reconstructed expression E,, (b) The cycle consistency between the original and reconstructed embeddings (e, and e¹,) can benefit to optimize the P problem. We enable the joint optimization for cycle consistency with a cross-modal proxy fn, defined between all mongle-modal operations (i.e., TT TT Ndee and ITdeo). (c) Point-wise consistency is not suitable R2-VOS because the mapping between & and E' are not necessarily bijective. An object be referred by various textual expressions. (d) Instead, apply relational consistency to preserve distances and angles.
97 Primary and dual problems for VOS. maximum a posteriori estimation problem of VV, F..). By applying the Bayes rule, we obtain:
The referring segmentation can be formulated as the
98
p(M,/V, E.) PVE.IV, (,)))) M.)p(M.IVV
99
As the prior V(M is not affected by the expression E,, we consider maximizing p( E. V, M.) as a dual problem of the referring segmentation (primary problem), which is to reconstruct the text expression given the video and object masks. We note that for negative videos, P(E./V/ V. M.) is undefined because the mask M, is empty. Thus, we only investigate the dual problem for positive videos. The primary problem and the dual problem can be connected a cycle path, i.e., from the original expression E,, to the reconstructed expression E through positive video queries, as shown 2 (a). We believe that the cycle constraint benefits optimize the primary problem by enhancing the semantic consensus.
100
101
102
105
105
in Figure
3

---

Early Grounding
V-I Projection
Skip
Visual
Vissal Decoder
Ensoder
Mult-scale feat.
Video V
Video fear.
Muliindil
Attention
Grounded Sent. Faarty
Text quecy C +
Instance emb. Mow
Sog. [M,]Fsn
cal lay by owl Text exp. E
Tesnua Encoder
cirri
Fool
Transurnazz Dosnder D2
Bex (B, Fes
Word-level lest.
g
Dynamic kermel 9
Score (Sp)evi
Sentence-
Relational Cycle Consistency
Reconstructed text emb. e'
F
Transfomer Decoder
text emb.
query' X
Product
Repeat
Concatenation
Pos. Neg.
Figure 3: Overview of the proposed model. Given video clip V (T) and textual expression E, referring object o, we first extract video feature and text feature separately, then fuse them in the early grounding module to obtain the visual representation fearty of the referred object o. Then project fearly to a textual space to be e' and add the relational cycle constraint with the original text embedding e. The final segmentation is obtained by dynamic convolutions with video features from the visual decoder and dynamic weights from the fused text embeddings. The semantic consensus of input pairs is discriminated be positive or negative by assessing the consistency between and e'.
107 In practice, we study the cycle consistency between the original textual embedding space & and the transformed textual embedding space E' induced by positive videos. By definition, the path from the original text embedding e, the reconstructed text embedding e is modulated with cross-modie interactions between video and text. Thus, to link the primary and dual problem and enable the joint optimization, we introduce cross-modal intermediate feature fm to convey information of both the input of the primary problem (V, F.) and the dual problem (V, Ma), as shown in Figure 2 (b). f. is defined between the encoder and decoder stages of single-modal operations, i.e., IIgns. Ieee Idde, only focus on the multi-modal interaction.
108
110
112
Idder 113
114
115
Relational cycle consistency. A key observation for cycle consistency between & and E` is that the mapping between them is not necessarily bijective, as there could be multiple textual descriptions for the same object. Thus, naively adding point-wise consistency, i.e., C2 c'2, Vc, E & will collapse the feature space to a sub-optimal solution. Instead, we take inspiration from relational knowledge distillation [33], and introduce relational cycle consistency for & and gr. The relational cycle
116
117
118
119
120
consistency is to preserve the structure of the feature space rather than exact point-wise consistency,
as illustrated in ( and (d). Mathematically, the structure- presterrig property is defined as isometric and conformal constraints to preserve pair-wise distance and angles for & and e' E E
122
e2 = lei e
(2)
Z(e), e2) = Z(e', e eb),
(3)
123
where
and Z(-) denote distance and angle metrics.
Method
124
125
In this section, we elaborate our Re-VOS framework with the relational consistency, which mainly consists of four parts: feature extraction, early grounding as a medium, video-text (V-T) projection for text reconstruction, and mask decoding for final segmentation, as shown in Figure 3 We first extract the video feature f, word-level text feature g, and sentence-level text embedding e. On the one hand, model the primary segmentation problem of maximizing p( P(M,l V, E.), we enable the multimodal interaction in the early grounding module to generate the grounded feature fearly. fearly coarsely medium linking the primary segmentation and dual text reconstruction problem. The final mask M, is obtained by dynamic convolution 4 on the decoded visual feature maps, with kernels learned from instance embedding {2, YT= On the other hand, to model the dual text reconstruction problem of maximizing pl [V, Ma). we utilize the grounded video feature fearty as the alternative of V and Ma. since fearly conveys contextual video clues of object In this way, we enable the parallel optimization of the primary and dual problem by relating them to fearly- Specifically, we employ a V-T projection module to project fearly onto reconstructed text embedding e'. We add relational constraint between
126
127
128
129
130
131
locates the referred object and filters out irrelevant features, which serves
132
133
134
135
137

---

139 e' and e to enforce the semantic alignment between the segmented mask and expression for positive videos. In addition, we introduce semantic consensus discrimination head H(e,, e') assess the consistency between original and reconstructed text embeddings, diseriminating the alignment of multimodal semanties and identifying negative videos.
140
142
143
4.1
Single-modal Feature Extraction
144
Visual encoder. Following previous methods [1 44 we build the visual encoder with a visual backbone and a deformable transformer encoder [2 top of it. The extracted features from the
145
146
backbone are flattened, projected to a lower dimension, added with positional encoding [12], and then fed into deformable transformer encoder [52) similar to the previous method 44 We denote the multi-scalc output of the transformer encoder as F and the low-resolution visual feature map from the backbone as f, where f E RTxC, x $ x C₁ is the feature channel, T is the video length and H and W are the original image size.
148
149
150
151
Textual encoder. We leverage a pre-trained linguistic model RoBERTa 27 to map the input textual expression E. to textual embedding space. The textual encoder extracts a sequence word-level text feature g E RC x and a sentence-level text embedding e E RC where Ce and L are the dimension of linguistic embedding space and the expression length respectively.
152
153
154
4.2
Early Grounding
156
We propose an early grounding module to coarsely locate the referred object and filter out irrelevant fea- tures. Then the grounded feature fearly encoding in- formation of segmentation problem, but also for the dual expression reconstruction task, which serves proxy connecting the two problems. Figure shows a visualization of fearly- Specifically, we utilize the power of dynamic convolution [4) to discriminate visual features in the first enable the multimodal interaction between video and text features, then apply the dynamic convolution with kernels learned from text feature discriminate the object-level semantics. In particular, multi-head cross-attention (MCA) 41 is
a black bear standing on a rock in stream
158
only be utilized for the primary
180
Frame
CAM of fearly
162
Figure 4: Visualization of channel activa- tion map (CAM) of fearly-
163
164
165
early stage. As shown in the blue part of Figure
3
186
187
188
leveraged to conduct the multimodal interaction:
hf LN(MCA(f, g) f) f' (N(FFN(hr) + hr)
(4)
169
hg LN(MMA(g.f) + +g) g' IN(FFN((h)) hg),
(5)
where MCA(X, Y) Attention (iteettionWx..... X W WVY.. W represents Iearnable weight. LN and FFN denote layer normalization and feed-forward network respectively. The text feature g' is further pooled fixed length, and followed by fully-connected layer to form the dynamic kernels
171
[0 is the kernel number and 0g E RCx( The dynamic kernels are applied separately 174 to video feature E CXTTH to form the fearly E RCCTHW
173
fearly BN @ @ OK
(6)
175
where L the concatenation channel dimension and yp() is convolution to reduce the feature dimension. BN denotes batch normalization.
177
4.3
Text Reconstruction
178
V-T projection. We leverage transformer decoder DD as textual decoder transform the visual representation of the referred object into the textual space. As shown in Figure a learnable text query eg E xxx1 is employed query the fearly. The output of the transformer decoder is the reconstructed text embedding e' DD (Feaaarlyy ep) x1
179
180
181
4.4 Referring Segmentation
185
Mask segmentation. Similar to previous methods decoders with dynamic convolution to segment the object masks, As shown in Figure 3 we first fusc
we leverage deformable transformer
5

---

185
the reconstructed text embedding e' to text embedding e. The fused text embedding e is then repeated
N times to form the instance query + Zo E R XN where C2 the dimension of instance query and N is the instance query number. We then use Tx deformable transformer decoders Dy with shared weights to decode the instance embeddings 2, E RRC for each frame, i.e., 24 = Dy (F 2o).
186
187
188
189
F, is the multiscale visual feature from visual encoder at time f. A dynamic kernel w, is further
learned from the instance embedding Zς. The final feature map fout.e [RC*#xW> is obtained by
190
fusing low-level features from the feature pyramid network (23) in the visual decoder. The mask prediction M, E R x xW can be computed by M. = w/ fout,e"
192
193
Auxiliary heads. We build a set of auxiliary heads to obtain the final object masks across frames. In particular, a box head, a scoring head and a semantic consensus discrimination head are leveraged to predict the bounding boxes B, E R RNx4, confidence scores S, E R RN and the alignment degree of multimodal semantics E TR. The box and scoring head are two fully-connected layers upon the instance embedding et. The semantic consensus discrimination head H(e, e') consists of two fully- connected layers upon the text embeddings e @ e'. Note that A represents the semantic alignment in the entire video rather single frame, since the expression is video-level description.
194
195
196
197
198
200
4.5
Loss Function
The loss function of our method can be boiled down three parts:
L Atext Asegm + >align
where Liezt, Leegm, and Lolign are losses for text reconstruction, referring segmentation and semantic consensus discrimination respectively. ground-truth semantic alignment Å {0, 1} utilized to discriminate positive and negative pairs. The Lalign is simply cross-entropy loss between the predicted alignment A and ground-truth Á. The other two terms are computed as follows:
203
204
205
206
Loss for text reconstruction. Given the text embedding e and reconstructed text embedding e', we employ a relational constraint to impose the cycle consistency between e and e'. We calculate the loss by
208
(8) where the indicator function (A) = 1 if the alignment indicates the referred object exists in the video, otherwise O, Aangle is hyperparameter balancing the distance loss Laist and angle loss
Lient = ((A) (Cdist + Aongle
209
210
Langle- We elaborate these two losses according to the relational eycle consistency Equation 2
211
212
}} denote a set of n-tuples, T {(x, x')[x x Am x' E X denote a set of pairs consisting of two n-tuples of distinct elements from two different sets X and X'. Specifically, the distance-based and angle-based relations relate text embeddings of 2-tupe and
Let X = { (1.; x,)),
213
214
3-tuple respectively, i.e., d² = {(x, x') xx = (e;, ey), = (e', ef), i + j} and & = [xx x x')xx
215
(e,, ej, xx), = (e/, c.., (,), $ k). Then the losses are given by:
218
Ldist
Σ
4(0n (n(x))) Φn(x)
n|x lle ej 11
(x,x))E42
Longle
Σ
x2(x)= = cos/iei, e,, ex),
(10)
(x, x')eaa
where (xx) = Ex=(e) ||æ1-2ul1a is the average distance function, and the Huber loss x
217
(z, x²) [x -x- if |z 2'1 S otherwise - -.
218
Loss for referring segmentation. Given a set of predictions y [y: and ground-truth 9. where y; (B,,, S.,,, M,,)+= and ý = (B., M, we search for an assignment or E PN with the highest similarity where PN is a set of permutations of N elements (ÿ is padded with t).
219
220
221
222
The similarity can be computed as
Liaatch 8) Abex Lbear + Aconf + >maask Cmask:
(11)
223
where Aboz: Aeonf. and Amask are weights to balance losses. Following previous works 6
43 we
224
leverage a combination of Dice [20] and BCE loss as Lmasks focal loss (4 as Leon and GloU
225
38 and LI loss as Lier. The best assignment ē is solved by Hungarian algorithm [I6] Given
the best assignment o, the segmentation loss between ground-truth and predictions is defined as
Leegm = 1(A) Emoteh(y (y, 8a()).

---

228
4.6
Inference
229
During inference, we select the candidate with the highest confidence to predict the final masks:
(M, {1(A) M Mg.)
S argmax + +... + St,T
(12)
where (M,) is the masks of referred object. i and Ma,1 represent the i-th candidate in S, and 231
M, respectively. s is the slot with the highest confidence to be the target object. We use ((A) to filter out predictions in negative videos to mitigate false alarm. ((A) 1 if A > 0.5, else 0.
Experiment
233
5.1
Dataset and Metries
235
Dataset. We conduct experiments on three datasets: Ref- Youtube- VOS, Ref-DAVIS and R
Youtube-VOS VOS. YoutYouuubb--o VOS [39] is large-scale benchmark that has 3,978 videos with about
237
15k language descriptions. There are 3,471 videos with 12,913 expressions in the training set and 507 videos with 2,096 expressions in the validation set. Ref-DAVIS-17 [3] contains 90 videos with 1,544 expressions, including 60 and 30 videos for training and validation respectively. R Youtubeee VOS our newly proposed evaluation dataset: it extends the Ref- Youtube- VOS validation set with each linguistic expression to query positive video (the same one as Ref- Youtube- VOS) and a negative video. To make each video can be picked as a negative video, we randomly shuffle the original video set and constrain all negative text-video pair unrelated.
239
241
242
243
Metrics We employ commonly region similarity I and contour accuracy F [3 for con- ventional Ref- Youtube- VOS and Ref-DAVIS- 17 benchmarks. For the proposed R² Youtube- VOS
245
task, we additionally introduce a new metric R = EMeMueg [M]
246
Exre,puu [M to evaluate the degree of object
247
false alarm in negative videos, where Mneg and Mpos are the sets containing segmented masks in negative and positive videos respectively. [M denotes the foreground area of mask M. The total foreground area of positive videos Σ E.MMp M serves as normalization term. Ideally, a model should treat all the negative videos as backgrounds with R =
249
250
251
5.2
Implementation Details
252
Following previous methods
our model is first pre-trained on Ref-COCO/+/g dataset 44
253
and then finetuned on Youtubee- VOS. The model is trained for 6 epochs with a learning rate multiplier of 0. at the 3rd and the 5th epoch. The initial learning rate is le-4 and a learning rate multiplier of 0.5 is applied to the backbone. We adopt batchsize of 8 and an Adam W (29] optimizer M. the evaluation on Ref-DAVIS directly
254
256
with weight decay 1 x 10-4, Following convention models trained on Ref- Youtube VOS without re-training. All images are cropped to have the
258
longest side of 640 pixels and the shortest side of 360 pixels during evaluation. The window size is
set to 5 for all backbones. We create negative pairs by shuffling positive pairs in each batch. Our method is implemented with PyTorch 34
280
261
5.3
Main Results
262
We compare our method with state-of-the-an R-VOS methods Youtube-VOS and Ref-DAVIS-
17 in Table 1 and R2-VOS task in Table 2
263
Comparison Ref- Bef-Youtube-VOS. In Table I we first compare method on Ref- Youtube
265
For results of ResNet 99) backbone, our method achieves 57.3 J &FF which outperforms the latest method ReferFormer
4 by 1.7 J &FF. In addition, our method runs at 30 FPS compared to 22 FPS ReferFormer (FPS is measured using single NVIDIA P40 with batchsize = 1). method achieves 60.2 J & and 61.3 J&FF with Swin-Timg and Video-Swin. Tiny backbones respectively, which outperforms ReferFormer 44
267
of raae-of-the-aa
For results of Swin- Tran-fTranfoommr [28] 28 backbones,
269
270
and MTTR by a clear margin. More analysis is available in the additional appendix A.
271
Comparison on Ref-DAVIS-17. Our method achieves 59.7 J&F on Ref-DAVIS-17 dataset, which outperforms ReferFormer by 1.2 J &.F.
7

---

Ref- Youtube-VOS
Ref-DAVIs-17
Method
Backbone
J&F
J&F
Spatial Visual
Backbone
CMSA 48
ResNet-50
34,9
33.3
36.5
34.7
32.2
37.2
CMSA RNN
48
ResNet-50
36.4
34,8
38.1
40.2
36,9
43.5
URVOS
39
ResNet-50
47.2
45,3
49.2
51.5
47.3
56.0
PMINet
ResNet-101
53.0
51.5
54.5
CTTD
22
ResNet-101
56.4
54.8
ReferFormer"
44
ResNet-50
55.6
54.8
56.5
58.5
55.8
61.3
Ours
Re
57.3
56.1
58.4
59.7
57.2
62.4
ReferFormer'
Swin-T
58.7
57.6
59.9
Ours
Swin-T
60.2
58,9
61.5
Spatio- temporal Visual Backbone
MITR
Video-Swin-
55.3
54.0
56.6
ReferFormer"
Video-Swin-T
58.0
60,9
Ours
Video-Swin-T
61.3
59.6
Table
Comparison state-of-the-art R- R-VOS methods Ref-Youtube--ie VOS and Ref-DAVEL 17 val set. indicates results imported from preprints.
Method
Backbone
J&FF & π
F
ReferFormer'
44
ResNet-50
47.3
54.8
56.5
30.6
Ours
ResNet-50
69.5
56.1
58.4
94.1
MITR" 1
Video-Swin-T
40.0
55.9
58.1
5.9
ReferFormer'
Vidco-Swin-T
49.1
58.0
60.9
28.5
Ours
Vidco-Swin-T
72.8
59.6
63.1
95.7
Table 2: Comparison to state-of-the-art R-VOS methods on R2-Yotuube-VOS.
Comparison on R²-VOS. As shown in Table the state-of-the-art R-VOS methods, ReferFormer and MTTR suffer from a low R metric which measures the false-alarm problem when the semantic consensus of the input text-video pair does not hold. Compared the severe false alarm of previous
274
276
R- VOS methods, our model successfully mitigates the false alarm the model, thanks to the proposed multimodal cycle consistency constraint and semantic consensus discrimination head.
278
Qualitative results. We compare the qualitative results of our method against state-of-the-art methods Figure R2. VOS. For positive videos: The result indi- cates that our method predicts accurate and temporally-consien
279
280
282
results, while Refer- Former [44) and MTTR (I] fail to locate the correct object. For negative videos: Both ReferFormer and MTTR suffer from severe false-alarm problem when the re- ferred object does not exist in the video. In contrast, with multi-modal cycle constraint and consensus discrinination, our method successfully filters negative videos and mitigates the false alarm. To further ex- plore how the consensus discrimination works, we visualize the text embedding and reconstructed text embedding spaces for both positive and negative videos. As shown Figure we
284
for Pos. Videos E' for Neg, Videos
Figure 5: Visualization of text embedding spaces. Dots represent original text embeddings in È, and triangles represent reconstructed ones & induced by positive and negative videos respectively. Elements in the same color belong to the same object. Note that an object have multiple text descriptions. The structure of E' is well preserved from E for positive videos (ellipses bound embeddings of same objects), while it is not preserved for negative videos.
286
288
290
282
293
294
295
296
notice that, for embeddings of positive videos, they preserve relative relations well, while for negative videos, the reconstructed embeddings have random pattern in the space.
297
298
5,4
Ablation Study
299
Module effectiveness. To investigate the effectiveness of different components in our method, we conduct experiments with the ResNet-50 backbone R2- Youtubee VOS dataset. We build transformer-basee baseline model and equip our proposed components step-by-step. As shown in 3 the baseline model achieves 52.4 J &.F. Then, we add our proposed components step-by- step to demonstrate the module effectiveness. After employing the early grounding module, the performance boosts to 55.5 J J&F and the cycle-consistency constraint brings another 1.4 J&F gain. Since the reconstructed text embedding is generated with visual features injected, we consider it can
300
302
304

---

Positive Video
Negative Video
Trane
MITR
Refee
Oues
Expression: the mirror in the bathroom is the right of the wood cabinet
Figure 6: Qualitative comparison to the state-of-the-art R-VOS method on the R R2-VOS task.
Constraint
J&rF
J
F
R
Components
J&F
F
R
None
55.5
54.4
56.5
32,9
Baseline
52.4
51.9
52.8
34.9
PW
54,4.
53.3
55.5
88.7
+BG
55.5
54.4
56.5
32.9
RA
56.7+12
55.5
57.9
93.6460.7
+CC
56.9-4.5
55.7
58.1
94.0+59.1
RD
56.440.9
55.2
57.6
90.4457.5
+FT
57.3
56.1
58.4
+5.1
RD+RA
56.9+14
55.7
58.1
94.0+
Table 3: Impact of different components in our method. EG: Early grounding. CC: Consis- tency constraint, FT: Fusing text embeddings.
Table 4:
Impact of the cycle consistency con-
straint. PW: Point-wise. RA: Relational angle.
RD: Relational distance.
Query Number J&FF
J
F
Window Size
J&F
J
F
R
54.9
54.2
55.6
94.7
53.5
53.0
54.0
89.2
57.3
56.1
58.4
94.1
3
56.8
56.5
57.1
92.1
57.0
56.8
57.2
93.5
5
57.3
56.1
58.4
94.1
Table 5: Impact of the query number.
Table 6: Impact of the window size.
encode some visual information, thus augmenting the original text embedding. By using the fused text embedding as instance query, we achieve our best performance of 57.3 J J&F.
307
308
Consistency constraint. We conduct experiments to ablate the influence of cycle-consistency constraints. As shown in Table[4 utilizing point-wise consistency constraint will lead a performance drop compared to the setting without cycle constraint. We consider the point-wise constraint may force an injective mapping from the textual domain to the visual domain. However, the mapping can be a many-to-one function for R- R-VOS, i.e., each object corresponds to multiple textual descriptions.
309
310
311
312
313
In addition, since the early grounding leverages the text feature to locate the referred object, if use
314
the direct point-wise constraint to form reconstructed text embedding, it will encourage the network to memorize the text feature in the fearly and result in a collapse for text reconstruction. Table shows that sole relational angle constraint can bring 1.2 J &FF gain, and it can be slightly improved with 1.4 J &FF gain by jointly using relational angle and distance constraint.
315
317
318
Instance query number. Although only one referral is involved for each frame in R-VOS task, to help the network optimization, we employ more than one instance query to each video. Table indicates that a query number of 5 brings the best result.
319
320
Frame number. Since R-VOS gives a text that describes an object over period of time, temporal information is vital to segment accurate and temporally-consistere results. We ablate on the best window size of input videos during training. As shown in Table 6 we notice that the performance improves as the window size increases and window size of 5 brings the best result of 57.3 J&F.
322
323
324
Conclusion
325
326
In this paper, we investigate the semantic misalignment problem in R-VOS task. A pipeline jointly models the referring segmentation and text reconstruction problem, equipped with a relational cycle consistency constraint, is introduced to discriminate and enhance the semantic consensus between visual and textual modalities. To evaluate the model robustness, we extend the R- &-V task to accept unpaired inputs and collect corresponding R²_ Rouuubee VOS dataset. We observe a severe false-alarm problem suffered from previous methods on 2 Youtube- VOS while ours accurately discriminates unpaired inputs and segments high-quality masks for paired inputs. Our method Youtueeu VOS, and R2-VOS dataset. We believe that, with unpaired inputs, V2- is a more general setting of referring video segmentation, which can shed light on a new direction to investigate the robustness of referring segmentation.
327
328
329
330
332
achieves state-of-the -fte----- performance on Ref-DAVIS17,
334

---

A
Additional Appendix
336
337
A.1
More Quantitative Result Analysis
338
Under the same ResNet-50 backbone, our method achieves 57.3 J J&F, 94.1 R. and 30 FPS compared to the 55.6 J&F, 30.6 RR and 22 FPS of ReferFormer. We will then point-to-point analyze reasons of improvements on J&F (for positive video), R. (for negative videos) and FPS (for inference speed).
339
340
J&F: (1) We introduce the early-grounding module which employs both pixel-wise and channel-wise attention to enable multimodal interaction. Different from the CM-FPN used in ReferFormer that solely fuses features from text to video in pixel-level, our early-grounding module first enables pixel-level bi-directional fusion and then generates dynamic kernels using the fused text feature g' to modulate the video feature f". The dynamic convolution (channel-wise attention) is commonly used to decode dense masks from visual features and is suitable to suppress irrelevant features. By equipping text-guided dynamic convolution in carly-stage, the pixel decoder can be more focused on the target object (as shown in Figure 4). (2) Our method leverages relational cycle consistency to constraint the intermediate feature fearly to contain correct object-level information to recover some properties of original text embedding. By applying this constraint, our method can better avoid interference and easier locate the correct object.
342
343
344
346
348
350
352
(3) Our instance query is composed of both original sentence embedding and the reconstructed one. Different from ReferFormer that only utilizes original sentence embedding as queries, the reconstructed embedding can encode visual information to facilitate the instance query decode the objects from visual features.
353
354
355
356
R: The newly introduced metric RR aims to measure the robustness of the model against unpaired inputs. video pairs with (object-level) semantic consensus can be assumed as in in-distribution for RVOS problem where semantic consensus can be kind of easily modeled. In contrast, unpaired text-video much more difficult to tackle because there can be unlimited out-of-distributie instead of directly detect the OOD of input pairs, we convert the problem to find semantic alignment between the input text embedding and reconstructed embedding and constraint the property of reconstructed space by introducing the cycle consistency. In this way, the comparison is conducted in the constraint original and reconstructed text spaces. For ReferFormer, it models the alignment of text to video by querying the visual features by text in the transformer decoder. In this way, the comparison is conducted in unconstrained text and video spaces thus results in a inferior performance.
357
358
359
(OOD) scenarios for the text-video pairs. In our method,
361
383
365
367
388
FPS: The speed improvement of our method mainly comes from our efficient multimodal fusion. Compared to the multi-scale CM-FPN, our carly-grounding module is only conduct at the high-level. In addition, our bi-direction multimodal fusion (Equ 4 & 5) only leverages cross-attention to avoid computational expensive video-to ceco-ttovideoo operations.
389
370
371
372
A.2
Limitations
An important challenge for video segmentation is that target object disappearance due occlusion, which can results in false positives on a per-frame level. In our method, we predict the video-level semantic alignment to handle the false positive in video-level resulted from unpaired text-video pairs. However, since only video-level object expression is available in RVOS task, our method can not address the frame-level false positives resulted from occlusion.
374
376
378
A.3
Additional Experiment on Negative Videos without Positive Text
R
Negative Video Source
ReferFormer
Ours
Ref-YYutube--OS Ref- Youtube- VOS & Ref-DAVIS
30.6
94.1
33.1
92.2
Table A: Impact of different negative video sources.
379 As shown in Table A, we test the robustness of our model on two settings. We generate negative videos from Ref- Youtube--VS and a combination of Ref-Youtube-VOS and Ref-DAVIS dataset.
10

---

381 In both settings, all videos in the validation set are leveraged. The results indicates that source of
382 negative videos has minor impact on the robustness of our model.

---

References
383
384
[1] Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. arXiv preprint arXiv:2111.1488 2021.
[2] Kan Chen, Jiyang Gao, and Ram Nevatia. Knowledge aided consistency for weakly supervised phrase grounding. In Proceedings of the IEFE Conference on Computer Vision and Puttern Recognition, pages 4042-4050, 2018.
387
[] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer, 2020.
390
382
[4] Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11000- 1039, 2020.
393
[5] Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, and Osamu Yoshie. Zerovl: A strong baseline for aligning vision-language representations with limited resources. arXiv preprint arXiv:2112.0933), 2021.
396
387
398
[6] Zihan Ding, Tianrui Hui, Shaofei Huang. Si Liu, Xuan Luo, Junshi Huang, and Xiaoming Wei. Progressive multimodal interaction network for referring video object segmentation. The 3rd Large-scale Video Object Segmentation Challenge. page 7, 2021.
389
400
401
[7] Akira Fukui, Dong Huk Park, Daylen Yang. Anna Rohrbach, Trevor Darrell, and Mareus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606. 01847, 2016.
402
403
404
[8] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6639-6648, 2019.
405
406
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
409
[10] Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. arXiv preprint 18X3v:1803.088 2018.
412
413
[11] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion.
detection for end-to-end multi-modal understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
415
416
[12] Guolin Ke, Di He, and Tie Yan Liu. Rethinking positional encoding in language pre-training. arXiv preprint arXiv:2006. /5595, 2020.
[13] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In Asian Conference on Computer Vision, pages 123-141. Springer, 2018.
419
420
[14] Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. Advances in neural information processing systems, 31, 2018
421
422
[15] Jin-Hwa Kim, Kyoung- -Won On, Woosang Lim, Jeonghee Kim, Jung- Woo Ha, and Byoung- Tak Zhang. Hadamard product for low-rank bilinear pooling. arXiv preprint 151.v.161.. 2016.
423
424
425
[16] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logisties quarterly, 2(1-2):83-9)) 97, 1955.
426
427
[17] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu, Less more: Clipbert for indo-and-languaee learning via sparse sampling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7331-7341, 2021.
428
[18] Jie Lei, Licheng Yu, Mohit Bansal, and Tamara I Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv: 180900696, 2018.
431
432
[19] Muchen Li and Leonid Sigal. Referring transformer: one-step approach to multi-task visual grounding. Advances in Neural Information Processing Systems, 34, 2021.
12

---

434
[20]
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv:1911.02855, 2019.
435
436
[21]
Chen Liang, Yu Wu, Yawei Luo, and Yi Yang. Clawcranenet: I Leveraging object-level relation for text-based video segmentation. arXiv preprint arXiv:2 10702, 2021.
437
438
[22] Chen Liang, Yu Wu, Tianfei Zhou, Wenguan Wang, Zongxin Yang, Yunchao Wei, and Yi Yang. Rethinking cross-modal interaction from top-down perspective for referring video object segmentation. arXiv preprint wxXiv21006000666 2021.
440
441
[23] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
442
443
[24] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
445
446
447
[25] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
449
450
[26] Yang Liu, Samuel Albanie, Arsha Nagrani, and Andrew Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint 19X7v 2019.
451
452
[27] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 197. 2019.
454
455
456
[28] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 132.30000 2021.
458
299
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05000 2017.
459
[30] Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.066:: 06333, 2020.
461
462
463
[31] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
465
466
[32] Antoine Miech, Ivan Laptev, and Josef Sivic. Learning a text-video embedding from incomplete and heterogeneous data. arXiv preprint arXiv:1804.00 02516, 2018.
487
468
[33] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
470
471
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, perf-peeformance deep learning library. arXiv preprint arXiv: 2019.
472
474
[35] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
475
478
477
[36] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Corkine-Horunn and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 1rXivv170.000655 2017.
479
480
481
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR, 2021.
483
13

---

485
[38] Hamid Rezatofighi, Nathan Tsoi, Jun JonYg Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 658-666, 2019.
486
487
488
489
[39]
Seonguk Seo, segmentation network with large-scale benchmark. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 208-223 Springer, 2020.
Lee, and Bohyung Han. Urvos: Unified referring video object
490
491
492
498
[40]
Xiaomeng Song, Yucheng Shi, Xin Chen, and Yahong Han. Explore multi-step reasoning in video question answering. In Proceedings of the 26th ACM international conference on Multimedia, pages 239-247, 2018.
494
495
496
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, undefinedukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS' 17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
498
499
[42] Liwei Wang, Yin Li, and Svetlana Lazebnik. Learning deep structure- presuurringeeegggggge image-text embeddings. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5005-5013. 2016.
500
501
502
[43] Yuqing Wang. Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8741-8750, 2021.
503
504
[44] Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. arXiv preprint 1.Xiv:2201.00533 2022.
506
507
[45] Ning Xu, Linjie Yang. Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint
509
orXiv:1809.0333 03327, 2018.
510
[46] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5188-5197, 2019.
511
512
[47] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21-29, 2016.
514
515
[48] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10502-10511, 2019.
516
517
518
[49] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer, 2016
519
520
[50] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 6281-6290, 2019.
522
523
[51] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 1821-1830, 2017.
525
526
527
[52] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 19..4.0000.04159 2020.
528
$29
14

---

Checklist
530
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
535
534
(b) Have you read the ethics review guidelines and ensured that your paper conforms to
535
them? [Yes]
536
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
537
(d) Did you describe the limitations of your work? [Yes]
538
2. If you are including theoretical results...
539
We are not including theoretical results.
540
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] The code are proprietary; most of our used dataset are public available; we plan to release code and novel dataset upon acceptance.
542
543
544
545
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
546
547
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] We conduct several experiments multiple times and find the results are very close.
549
550
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
551
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes]
555
(c) Did you include any new assets either in the supplemental material or as URL? [No]
556
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The data we used are public available datasets for academic purposes.
558
559
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The dataset that we used does not contain such content.
560
561
562
5. If you used crowdsourcing or conducted research with human subjects... We do not use crowdsourcing or conducted research with human subjects.
563
15


Reviews:
[{"review_id": "yjrei3MTm1y", "reviewer": "Reviewer_Fhve", "summary": "The authors introduce a \"new\" variant of referring video object segmentation. This variant considers the possiblity that the given referring expression does not correspond to any object in the given video. A method based on relational cycle consistency is introduced as well. The proposed method has an additional head that predicts whether there is an object that corresponds to the given expression or not. Experiments are mainly conducted on two standard benchmarks to validate the effectiveness of the proposed method.", "questions": "What is the range of A? When does the indicator function 1(A) equal to 1?", "limitations": "N.A.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The paper is well written is easy to understand. Notations and symbols are well defined.\n(2) The authors propose a method that can handle expression that does not correspond to any objects in the given video via a simple head, which is likely a simple fully connected layer.\n(3) The proposed method is evaluated on two standard benchmarks and achieves good performance on both datasets.\n\nWeakness:\n(1) While existing methods do not explicitly consider the possiblilty of being given an expression that does not correspond to any objects in the video, it is likely that this can be easily handled by adding a background class to ReferFormer [1] and other methods.\n(2) Researchers have already used cycle consistency (text reconstruction) to help referring expersion segmentation in images. To me, this work is mainly an extension of existing ideas to video.\n\n[1] Language as Queries for Referring Video Object Segmentation, CVPR'22", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tP02CGWOE7", "reviewer": "Reviewer_fwGH", "summary": "This paper presents a new method for referring video object segmentation (R-VOS). It points out an assumption that current R-VOS datasets make: that the object referred to in the text is present in the video. This assumption is not true in real-world text-video query applications. Current models rely on the assumption which results in false positives when the referred object is not in the video. The paper addresses this in two ways: by introducing a dataset $R^{2}$-YouTube-VOS and evaluation metric that measures the false positives made by a model, and by creating a method that enforces the semantic matching of visual and text features. The authors demonstrate that their method outperforms current methods on Ref-YouTube-VOS and Ref-DAVIS and has fewer false positives than previous methods on the proposed $R^{2}$-YouTube-VOS.", "questions": "Questions:\n- In the sentence in lines 242-243 can you clarify what \"constrain all negative text-video pair unrelated\" means?\n\nSuggestions:\n- There are many works on vision-language representation learning and it would be useful to the reader if the authors could include more of this background in Section 2.\n- Line 58 a better word instead of \"medium\" would be \"proxy\"\n- It would be more clear to call $f_{m}$ an intermediate feature rather than a medium in line 112.\n- Typo in line 109 \"textural\" -> \"textual\"\n- Figure 2c caption is a run-on sentence and would sound better split into two.\n- When discussing the window size in line 259 mention that Swin or VideoSwin is used as a backbone.\n- Refer to Figure 4 somewhere in the text.\n- Line 320 \"temporal-consistent\" -> \"temporally-consistent\"\n- The discussion of quantitative results is currently limited. It would benefit from more details on how the proposed method differs from the ReferFormer architecture and why the ReferFormer is not able to learn the necessary information to avoid false positives.", "limitations": "- The paper does not include a section on limitations/negative societal impact.\n- An important challenge for segmentation in videos is temporary object disappearance due to occlusion, which results in false positives on a per-frame level. Since the semantic alignment ground truth is per-video rather than per-frame the method would not able to address this challenge, but if the ground truth were collected per-frame then perhaps it would.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper points out an important limitation of current R-VOS datasets and evaluation metrics and takes the proper steps to address it by creating the metric R and augmenting Ref-YouTube-VOS with negative video-text alignment pairs.\n- The idea to enforce semantic consistency of the embedded textual space and generated embeddings is novel and interesting.\n- The experiments are carefully designed. They clearly show that the model improves upon the state of the art. The ablation results show the benefit of the proposed relational cycle consistency losses.\n\nWeaknesses:\n\n- The paper is missing some details on the $R^{2}$-YouTube-VOS dataset including how many negative text-video pairs are created.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "22deXEFa6Jv", "reviewer": "Reviewer_Gh1L", "summary": "This paper introduces the Robust R-VOS task that accepts unpaired video and text as inputs. In addition, a pipeline is proposed to optimize the primary referring and dual expression reconstruction task. A relational cycle consistency constraint is proposed to enhance the semantic alignment between different modalities. The proposed approach achieves state-of-the-art results on three datasets.", "questions": "1 In formula 8, l(A ̂) is used to measure semantic consensus between Reconstructed text feature and text feature, it should be explained clearly.\n2 It is not clearly stated in the article how to evaluate this model on the Robust R-VOS dataset. Is the model trained on the Ref-Youtube-VOS dataset and directly evaluated on the Robust R-VOS dataset?\n3 The ablation experiments in this paper are performed on the Ref-Youtube-VOS dataset, which does not need to consider this problem that unpaired video and text as inputs, so different modules need be used separately to verify their effectiveness.\n4 The authors should clearly introduce the source of the negative sample videos in the dataset.  If these videos are also from the Ref-Youtube-VOS dataset, each video in the entire dataset will correspond to a positive text message. In order to better verify the robustness of the model, video that do not correspond to text information in the entire dataset should be added.\n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1)This paper proposed the Robust R-Youtube-VOS dataset, is collected to measure the robustness of R-VOS models.\n2)The relational cycle consistency is applied to preserve the structure of the feature space rather than point-wise consistency. Through the experience, the cycle consistency achieves better results than point-wise consistency.\n3)The proposed approach achieves state-of-the-art on three datasets, and some ablation studies are conducted to validate the effectiveness of each component.\n4)The paper is generally well written.  \n\nWeaknesses:\n1) My main concern is that the main novelty. In this paper, for negative videos\nM_o is empty. For some R-VOS methods, they are able to align the semantic information of different modalities, and they also overcome the problem that unpaired video and text as inputs so that Robust R-VOS does not need to be studied separately.\n2) This paper proposed a new task and proposed a new dataset. However, the paper makes the ablation study based on the Ref-DAVIS17, and Ref-Youtube-VOS.\n3) This paper leverages text reconstruction to reconstruct the text feature to measure semantic consensus. It is commonly used for Video-Language tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject