Cross Modal Domain Generalization for Query-based Video Segmentation
Authors: Yan Xia
Zhou Zhao
Jieming Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain generalization
multi-modal
video segmentation

Abstract:   Domain generalization (DG) aims to increase a model's generalization ability against the performance degradation when transferring to the target domains, which has been successfully applied in various visual and natural language tasks. However, DG on multi-modal tasks is still an untouched field. Compared with traditional single-modal DG, the biggest challenge of multi-modal DG is that each modality has to cope with its own domain shift. Directly applying the previous methods will make the generalization direction of the model in each modality inconsistent, resulting in negative effects when the model is migrated to the target domains. Thus in this paper, we explore the scenario of query-based video segmentation to study how to better advance the generalization ability of the model in the multi-modal situation. Considering the information from different modalities often shows consistency, we propose query-guided feature augmentation (QFA) and attention map adaptive instance normalization (AM-AdaIN) modules. Compared with traditional DG models, our method can combine visual and textual modalities together to guide each other for data augmentation and learn a domain-agnostic cross-modal relationship, which is more suitable for multi-modal transfer tasks. Extensive experiments on three query-based video segmentation generalization tasks demonstrate the effectiveness of our method. 

Under review as a conference paper at ICLR 2023
CROSS MODAL DOMAIN GENERALIZATION FOR QUERY-BASED VIDEO SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Domain generalization (DG) aims to increase a model's generalization ability against the performance degradation when transferring to the target domains, which has been successfully applied in various visual and natural language tasks. However, DG multi-mode tasks is still an untouched field. Compared with traditional single-modal DG, the biggest challenge of multi-modal DG is that each modality has cope with its own domain shift. Directly applying the previous methods will make the generalization direction of the model in each modality inconsistent, resulting in negative effects when the model is migrated the target domains. Thus in this paper, we explore the scenario of query-based video segmentation study how better advance the generalization ability of the model in the multi-modal situation. Considering the information from different modalities often shows consistency, we propose query query-guided feature augmentation (QFA) and attention map adaptive instance normalization (AM-AdaIN) modules. Compared with traditional DG models, our method can combine visual and textual modalities together to guide each other for data augmentation and learn domain-agnostic cross-modal relationship, which is more suitable for multi-modal transfer tasks. Extensive experiments three query-based video segmentation generalization tasks demonstrate the effectiveness of our method.
INTRODUCTION
Query-based video segmentation is first introduced by Gavrilyuk et al. (0018), which aims to segment the queried actors or objects in video based on the given natural language query. Although these years have witnessed promising achievements in this field, the segmentation model trained on the source domain will degrade dramatically on the unseen target data due to the domain shift in real applications. As we see in Figure | even if both of the two sentences refer to standing guy, the visual context complexity, the background environment and the expression styles of the texts in these two cases quite different, which makes the performance of direct transfer far
unsatisfactory.
Domain adaptation (DA) Kim et al. (2021): Hoffman et al. (2018); Kim et al. (2019) and domain generalization (DG) Dou al. (2019): Volpi et al. (018); Choi
(2021) have been proposed to solve these problems. Different from DA that requires the acquisition of target domain data during training, which is usually difficult achieve, DG can learn domain invariant features and improve domain robustness without requiring target-domain data. Previous DG methods use kernel-based optimization to extract comain-agnosti features MMandet al. (0033; Li et (2018c b), or use meta-learning to simulate domain-shiff situations Li et al. (201 Sap; Liu et al. (2020). However, most of these methods are only suitable for multiple-soure source domains. Adversarial data augmentation based methods have been proposed to solve the single-domain generalization problem Volpi et al. (2018); Zhao et al. 22020), which use an adversarial loss to generate more realistic fictitious samples as much possible. However, it is difficult to generate effective meaningful samples that are largely different from the source distribution in semantic segmentation tasks. Thus recently some other methods, such as instance selective whitening Choi et al. (2021) and memory- guided network Kim et al. (022) have been proposed to handle this problem, and achieve good performance.
Although the above methods have achieved great success, few methods have been proposed specially for multi-mode modal domain generalization tasks. In query-basee video segmentation. domain shift not only exists the image level (e.g. light, weather, background, etc.) and the instance level (e.g.

---

Under review as a conference paper at ICLR 2023
Guy standing
aloo randing
Sovie
bepttoran
Video Stift
(a)
(b)
Figure 1: (a) The demo of our multi-modal domain generalization task. (b) The illustration of the necessity of introducing DG in this task. Source only means the model trained and tested on the same domain, while the target only means the model is trained another domain, and directly transferred to this domain. As we can see, the performances drop a lot in both two datasets.
size, shape, appearance, etc.)Lin et al. (021), but also on the natural language level (e.g. expression styles). A natural idea is to gradually augment the source domain data during training to simulate domain shift when facing new domains. However, directly applying the above methods to enhance the two modalities separately and then fuse them together may suffer from negatively affects, since it difficult to ensure that the generalization directions of these two modalities are consistent.
Thus in this paper, we combine the visual and textual modalities together to facilitate each other for data enhancement. According to our observations, actions belonging to the same type of actors in different domains share similar representations in the query-video latent space, the main reason for the performance degradation is that visual background and contextual complexity vary widely across different domains. Although exists diverse background information in source data, which can be used to augment the source domain. Therefore, we propose Query-guide Feature Augmentation (QFA) module, which can use the attention scores between query and video frames to distinguish query-related foreground regions from unrelated background regions. Then we keep the foreground regions be segmented unchanged, and synthesize novel visual features by gradually enhancing the background areas. To ensure the semantic consistency of the generated data, we introduce Moco-based contrastive learning perturbed video features. Besides, the expression styles of queries among different domains are different, which will lead to deviations in the attention map between visual and query features when migrating to the target domain. Hence we propose to use AdapIN Huang & Belongie (2017) on the siion-to--query attention map AdM-Adaa) to alleviate this issue. AdapIN can help the model remove the impact of style in attention map during training, and gradually introduce statistics from other samples to help the model learn robust cross-modal relationships. To be summarized, our main contributions
not have access to the target data during training, there
to force the model to maintain query-related information in the background-perturbee
follows:
We are the first to conduct domain generalization on the query-based video-segmentation task, which is also the early attempt to increase generalization ability cross-moda task.
We propose two novel QFA and AM-AdaIN modules. Compared with previous DG methods, our model is more suitable for multi-modal generalization task.
Extensive experiments three generalization tasks show that model can greatly enhance the model generalization ability, demonstrating the superiority of our method.
RELATED WORK
Domain Generalization. Domain generalization (DG) requires the model be robust without accessing the target domain when facing domain shift, which means the model trained only on single or multiple source domains should also perform well on unseen target domain. Early models focus on extracting domain-invarian features Li et al. (2018c b): Hu et al. (2020) using kemnl- based optimization to minimize the dissimilarity across domains Muandet et al. (013)) Dou et al.) (2019) propose domain-agnosti learning paradigm and encourage the model to learn semantically consistent features across training domains. Huang et al. (2021): b Xu et al. (022 use Fourier- based framework to enhance the domain robustness. Other models aim to enhance the source domains in data-level or feature-level to improve the robustness. Yue et al. (2019) use auxiliary dataset to enhance the source images and obtain different styles of images. Meta-learning is also an important method
2

---

Under review as a conference paper at ICLR 2023
to solve generalization problems, which typically partitions source domains into meta-train and meta-test splits to simulate the domain shifts Li et al. (2018a); Liu et al. (2020). As for single domain generalization, which is more challenger than multiple source situation, Volpi et al. (2018); Zhao et al.) (2020) use an adversarial data augmentation method which jointly performs domain generalization representation and new domain augmentation in an adversarial learning manner. However, for object or semantic segmentation task, it is difficult for these methods to generate effective fictitious target distribution. Thus instance selective whitening Choi al. (202I) and memory-guides network Kim et al. (022)) are used to extract domain- invariant features to enhance the generalization capability for semantic segmentation. Although these methods have achieved great success, there are still limit methods target for multi-modal domain generalization problem.
Query-based Video Segmentation. Query-based actor-action video segmentation aims to extract relevant regions from videos based on a given natural language query, which is first brought up in Gavrilyuk et al. (2018). Wann et al, (2019) propose an asymmetric cross-guided attention network to incorporate important information from natural language query and visual concepts for each other. Considering the traditional dynamic convolution may neglect spatial context information, Wang et al. (2020) propose context modulated dynamic convolutional operation, which can integrate natural language query and visual spatial context together to compute the segmentation mask. McInttoh et al. (2020) apply capsule network to encode both the video and textual inpur, which is more effective than standard convolution based models. Ning et al. (2021) think the spatial relations are also important for this task, thus they propose a polar attention module to make the sentence feature interact with positional embedding more directly. Hui et al. (222)) propose a tanguage-guider feature selection module and a cross-moda adaptive module select spatial- and temporal- relevant information dynamically and aggregate them comprehensively.
3 METHOD
3.1 PROBLEM SETUP AND MODEL OVERVIEW
Given a natural language query Q and its counterpart video frames V E RTXHXWxC where T, H. W. C are the frame number, height, width and channel number, respectively, the query-based video segmentation aims to generate accurate segment masks on the objects related to the input query. Our new proposed task aims enhance the generalization ability unseen target domains T E (Tr.... TN> } with the segmentation model trained only on single source domain S. The key idea is to improve the model robustness against -fitt--diittiuution shift among different domains. The procedure can be solved as a worst-case problem (inha et al. (201)); Qiao & Peng (202T):
min Φ
sup T:D(S.T)SP
TE| EE[Sng (d: T)},
where á are the model parameters for segmentation objective function Lengs D is similarity metric to measure domain distance between S and T, p is the largest domain distance.
Considering it is difficult to directly optimize the worst-case problem, we try to solve the query-based video segmentation generalization task by gradually generating a novel domain S with our proposed QFA and AM-AdalN 3.4 during training, and D (S,8) p. Besides, we apply moco-based contrastive learning to constrain the distance S p.
3.2 BASELINE MODEL
First we use an I3D layer Carreira & Zisserman (017) with stacked 3D convolution as our visual encoder and get visual representations as V E R & W XC then we apply average pooling over the temporal dimension to get V. Since the usage of vocabulary in different datasets is different, to enhance the generalization ability and learn some words with similar semantics but different expressions, we choose Bert as our text Encoder, the natural language query can be encoded as Q RLXC'' where L is the length of the query. Followed by Hui et al. (0021). we divide the visual features in the encoder and decoder layers into K different scales respectively, denoted as Vr E R",xW,,xC where Hg. W₁ and C are height, width and channel number of the i-th visual feature, respectively. To allow the visual features interacting with the query more comprehensively, design a query-vision interaction MVVMI).QVIM).
3

---

Under review as a conference paper at ICLR 2023
a
Video input
Generated Mask
WIND
MMM
Multi-acale Decoder
Visual Feature Px
Visual Feature P.
Query: a racing car moving the road
Visual Feature
b
Quaryy
Foreartund Regiors
Source Data
Attendion Map
Add&&Norm
Quers-guided
Bocigrourd Regions
Feature Aug
Comwer (031
Ax
Query-guided Average Pooling
Query: racing car moving on the road
Add&Norm
Actention Map
Augment Matrix
A
Self-Attention
A
Mazua
BDD Layer
Auxiliary Data
The overview of our proposed network. (a) The main pipeline of our model. (b) The architecture of query-vision interaction module (QVIM), notice the query-guided feature augmen- tation (QFA) only exists in one layer. (c) Our proposed QFA first uses the query to distinguish the query-related foreground and unrelated background regions in source video frames, then selects query-agnostic regions from auxiliary data. Finally we use the query-agnostic regions to enhance the background areas of the original visual features and obtain the augmented data. The Maug can avoid the interference of similar objects (e.g., truck) in the auxiliary data to the source data.
Figure 2:
Query- Query-Vision Vision Interaction Module Our QVIM aims to enhance the interaction between different scale visual features and natural language. Followed by |Wann et al. (2019), we adopt an 8-dimensional position encoding feature encode relative position information of each pixel. Then we apply self-attention layer on the visual feature to enable each pixel can perceive context-related information. Chen et al. (202T) have proved that attentional pooling can help model localize its attention to semantically more important regions, which can further enhance domain generalization ability. Thus in this task, employ query vectors obtain relevant spatial regions from visual features. The k-th layer attention scores Aş between query and each visual pixels can be defined as:
(ufg. (u (t' [1, E (1))
a
(2)
where q E RC is the RNN output from Q. We use A& to highlight the most query-related regions from input visual features, which can facilitate the subsequent segmentation task. Besides, we apply another cross-modality attention in Eq Fg/7btteed visual inputs and natural language query, which can select important information from query to enhance the representation of each pixel.
Multi-scale Decoder Layer Followed by Wang et al. (2019). we divide the visual features into K different scales, each scale obtained by I3D and QVIM layer. Then we combine average-pooling
H x
and fully convolutional layer to generate multi-scale response map ( } e [1) from the encoded visual features V3 x
3.3 QUERY-GUIDED FEATURE AUGMENTATION
Data augmentation is important technique to improve model generalization ability. However, previous works are not suitable for multi-modal generalization tasks. For example, Fourier-batt based methods Huang et al. (2021): Xu al. (2021) will cause the color of vital objects changed, conflicting
4

---

Under review as a conference paper at ICLR 2023
with the colors described in the question text. Also, is difficult for adversarial methods Volpi et al. (2018): Zhao et al. (2020): Qiao et al. (2200) to synthesize meaningful fictitious video frames in segmentation task. Thus this section, we take advantage of the consistency of information in multi- modality, and use the query as guidance to gradually make meaningful enhancements to the source domain in feature level. Concretely, given an arbitrary scale of visual features V E R X we distinguish important foreground regions Ry from background regions RG according the attention score af.; We set threshold 8, the pixels larger than 3 are considered as foreground regions R which is highly relevant to the question, while the others smaller than 3 are considered as irrelevant background parts Rg:
{
Ry. if > B Rs.
Ak
[T),, E [1,
if of <= B
then we convert Rs into a Bool matrix B, the pixels with values are set to I, and the pixels without values set to O. Next, we randomly select another video from the same batch as auxiliary data Viaw, Similarly, we the query and the auxiliary data calculate the attention scores Åg. and then subtract the attention matrix from the all-one matrix to get the augment matrix Maugi
Mang | -
The purpose of Eq 4 is to minimize the interference of query-related objects in auxiliary data to important regions of the original visual features. Finally, we combine the auxiliary background and the original visual features together to generate new augmented data Ve:
V = 1 Moug Viee + Ry Vi:
(5)
At the test time, we remove the QFA module and directly conduct inference on original visual features from target domains.
Progressive Manner: In order to make the query-guided feature augmentation procedure more robust, here adopt progressive expansion manner. At the initial few epochs, do not perform any data augmentation, allowing the model to learn a stable attention relationship between query and visual features. Then we gradually shift the threshold 3 from small to large during the training process, which means the replaced background area in the original video frames is progressively getting larger. To ensure that the augmented data still retains the key region features relevant to its counterpart query, we then introduce a semantic constraint objective function.
Constraints on Augmented Data: Previous works have proved that it is necessary to apply distance
constraints between source domain and augmented domains Qiao et al. (2020); Qiao & Peng (021) or using stochastic gradients to update the generated adversarial examples Volpi al. (2018) Zhao al. 2020). While in this paper, employ contrastive learning to pull the distance between query and its counterpart augmented visual features, and push the distance other mismatched data. To increase the diversity of negative samples, follow Moco He et al. (200)) by designing a dynamic dictionary with a queue to store negative samples. Given a query as q. its counterpart augmented visual data positive key k,,, and N samples from the dictionary as negative keys, apply InfoNCE loss Van den Oord et al. (2018) to measure the similarities between q and keys:
exply- k+/T) EN exply ka/T)²
Lcoust
llg
where 7 is a temperature haper-parameter. As the training progresses, the magnitude of data en- hancement is gradually increasing, and the ability of Lconst measure the distance between the query and positive/negative samples also improves accordingly. Compared with adversarial learning, our model can converge more stably, and our model can measure the semantic similarity between croas-modalities better than Wasserstein distance Qiao et al. (202)).
3.4
ADAPTIVE INSTANCE NORMALIZATION ON ATTENTION MAP
From Figure I we can see that the domain shift not only exists in visual level, but also exists in natural language description style. This will lead to deviation in the cross-attention map between the visual features and the query text when the model is migrated to the target domain. The model cannot pay attention to the key text positions, thus the unimportant text information is integrated into
5

---

(2) RVOS A2D Sentences/ R22 the types of objects in Under review as a conference paper at ICLR 2023
the visual pixels' representation, which will cause deviations in the segmentation results. To bridge this gap, we introduce style randomization on the cross attention map Ck between V/ R W, and Q RIxx, which can be computed as:
C₂ woftmarr0(([(( Q² RllaxWxL
H1 W
H
W
((C)) W
Σ how RR, o(Ck)
(ch p(Cx))2 +,, (8)
H&W
where is learning parameter, is the Relu activation function, e a small number to avoid zero After obtaining the feature statisties of source attention map Ck. normalization (AdaIN) Huang & Belongie (2017) to replace the statics of input with an arbitrary chosen C==
adaptive instance
AdaIN ()) = 8( 6( -C--MCC ((Ck) C(CK)
o(Cr) +
(9)
á = 11=1-o0 (1- (((C)) +a ((C)), == (1- - a) ((C)) a (
(10)
where a (O, 0.5). By introducing the mean and variance from other C as perturbed data, the model can learn Comain-invariant response regions from the visual-to-query attention map, which can extract important information from natural language query against expression style shift when transferring target domains. During inference time, we also remove the AdapIN layer.
3.5 TRAINING
Here we introduce how to incorporate our proposed QFA and AM-AdaIN modules into the baseline model. Although these two modules can be inserted into query-guided average pooling and query- vision cross attention in QVIM layer at any scale, respectively. Inserting into different layers will have different effects on the generalization ability of the model. We find that data augmentation is suitable for the largest scale of visual features, while AM-AdaIN is suitable for the smaller scale, the details will be analyzed in section
After obtaining the multi-scale response map {( " xw
[ks E [1,K]) in Section 3.2 and the H,, xW [k E []), vit E (0, 1}), we can compute the sulti-scale segment loss with binary cross cross-yntropy loss:
ground-trute -tuut pixel-level annotations (fut)liml.
H
1
Leeg
(og(s)>) + (1 (1-) log(1 s1,))
(11)
K
H
w=-==
The total loss function can be computed as Loeg + Looust.
EXPERIMENTS
4.1 DATASETS AND GENERALIZATION TASKS
A2D Sentences is first released by Gavrilyuk et al. (0018), they provide corresponding natural language descriptions for each video in Actor-Action Dataset Xu et al. (2013). It has total of 3,782 videos, which contains 8 action classes (e.g. walking, eating, running) and 7 actor classes (e.g. adult, dog, cat). Refer-Youtule Youtube- (RVOS) is first extended on the Youtube- VOS dataset Xu al.] (2018) by Seo et al. 2020). which contains 3975 high-resolution videos with 94 common object categories. RVOS is annotated with 27899 text expressions. J-HMDB Sentences is also brought up by Gavrilyuk et al. (2018) from Jhuang et al. (2013), which comprises 928 videos with 21 action classes, annotated with 928 sentences.
Generalization Tasks: In this paper, we evaluate our methods three generalization tasks: (1) A2D Sentences Sentences-RROSS, +RVOS (A2R) since the RVOS contains overall 94 object categories, we select the same video object types A2D, e.g. car, person, dog, cat, with total of 3528 video-query pairs. The domain generalization task requires that the types of objects tested on the target domain should exist in the source domain.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison with state-of-the-ar methods on three generalization tasks.
Precision
mAP
loU
Method
P@0.5
P@0.6
P@0.7
P@0.8
P@0.9
0.5:0.95
Overall
Mean
A2R
Baseline
38.46
30.61
21.00
10.29
1.96
18.54
40.66
35,36
SFA
39.17
32.26
22.56
10.97
2.18
19.37
40.74
35.64
AdaIN
40.31
32.71
23.16
12.24
2.61
20.12
42.78
36.57
RobustNet
40.90
33.02
23.30
11.99
2.60
20.56
43.21
37.26
Mixstyle
41.21
33.45
23.44
12.41
2.73
20.89
43,01
37.85
DSU
41.58
34.16
24.38
12.95
2.89
20.93
43.11
38.09
Ours
42.62
35.13
25.45
13.38
3.29
21.36
44.06
38.31
R2A
Baseline
48.65
40.95
30.27
16.11
2.29
25.12
52.17
43,81
SFA
49.96
42.65
31.63
17.23
2.31
26.06
51.76
43.76
AdalN
50.19
43.50
33.05
17.53
2.60
26.78
52.32
43.98
DSU
51.20
43.14
32.41
17.50
2.86
26.79
53.14
44.50
Mixstyle
50.50
42.50
32.59
19.02
3.27
27.12
53.27
44.23
RobustNet
51.64
44.30
35.93
18.22
2.52
27.35
53.08
44.55
Ours
53.02
45.87
35.93
20.69
3.66
29.09
54.73
46.05
A2J
Baseline
83.92
70.66
43.68
10.10
0.10
37.44
63.74
63.34
SFA
85.88
73.14
46.49
11.37
0.13
38.97
65.21
64.32
AdaIN
86.39
73.42
46.98
12.63
0.13
39.50
66.04
65.89
RobustNet
87.01
74.25
46.57
10.83
0.15
39.27
66.14
65.09
DSU
88.15
74.79
48.33
11.69
0.16
40.07
66.29
65.76
Mixstyle
87.91
75.48
49.50
12.65
0.17
40.58
67.20
65.62
Ours
89.25
76.17
49.86
12.90
0.20
41.23
67.87
67.19
A2D Sentences included in RVOS, the model can be directly migrated to A2D Sentences after training on RVOS. (3) A2D Sentences *J-HMDB Sentences following previous works Gavrilyuk al. (2018); Wang et al. (2019) 2020), we also use the model trained on A2D Sentences to evaluate the generalization ability on J-HMDB Sentences without any additional fine-tuning.
4.2 BASELINES AND EVALUATION METRICS
We the multi-layer QVIM encoder and decoder baseline model without any consideration of domain generalization. We also compare our method with five state-of-the-an DG models: Simple Feature Augmentation (SFA Li al. [2021), Adaptive Instance Normalization (AdaIN) Huang & Belongie (2017). Instance Selective Whitening (RobustNet) Choi et al. (202T). Mixstyle Zhou et al. (2020), Domain Shifts with Uncertainty (DSU) Li et al.) (2022). Following previous works, we adopt intersection-over ovveru-monn (IoU) to measure the model segmentation ability. more details can be found appendix A.1 The implementation details can be found appendix A.2
4.3
MAIN RESULTS
First we compare our full model with the existing domain generalization methods to verify the effectiveness our proposed methods. We reimplement these ofate-of-the-art DG modules on our bascline model. The main evaluation results on three generalization tasks are presented in Table I. From the results we can see that our method outperforms the baseline model by a large margin on all three generalization tasks, demonstrating the effectiveness of our model. Compared with other traditional DG methods that only consider single-modal domain enhancement, our model can combine vision and text modalities with each other to perform guided data enhancement respectively, which is more suitable for situations when both two modalities suffer from domain shift. Besides, the results in Table I and indicate that QFA achieve better performance than AdaIN Huang & Belongie (2017) and Mixstyle Zhou et al. (2020), demonstrating necessary to use natural language query guide the visual feature augmentation.

---

Under review as a conference paper at ICLR 2023
Table Analysis of the components on two generalization tasks.
Precision P@0.7
mAP
IoU
Method
P@0.5
P@0.6
P@0.8
P@0.9
0.5:0.95
Overall
Mean
A2R
Our Full Model
42.62
35.13
25.45
13.38
3.29
21.36
44.06
38.31
w/o QGAP
40.82
33.05
24.01
11.85
2.64
20.40
42.41
36.85
w/o cross-att
39.37
31.89
22.39
11.82
2.58
19.44
41.57
35.87
w/o Lconst
41.58
34.16
24.38
12.95
2.88
20.73
43.14
37.29
w/o QFA
40.16
32.00
21.97
11.03
2.24
19.39
41.76
36.35
w/o AM-AdaIN
41.72
33.59
24.23
12.81
2.66
20.88
42.64
37.67
R2A
Our Full Model
53.02
45.87
35.93
20.69
3.66
29.09
54.73
46.05
w/o QGAP
49.96
42.65
31.63
17.23
2.31
26.06
51.76
43.76
w/o cross-att
49.21
42.16
32.56
17.86
2.18
25.86
51.22
43.15
w/o Leonst
52.10
44.38
33.28
18.46
3.14
27.54
53.10
44.88
w/o QFA
50.45
43.55
33.56
18.58
2.83
27.33
53.15
44.52
w/o AM-AdaIN
51.76
44.27
33.87
18.71
3.35
27.72
53.31
45.09
Overa RGFA)
mAARIFFA mean@AM-AAaang
mesn)M4AAdaa))
meani
Progresstve Progresiive
67
Overll (oU%%)
(oU)))
38.75
Difirent Layara
Difiorett Layers
A2R
RBA
(a) R2A
(b) A2J
(c) Progressive manner
Figure Analysis of the QFA and AM-AdaIN in different layers (a) and (b). Analysis of the progressive manner (c).
4.4 ABLATION STUDIES
Model Components To investigate the effect of individual components, remove them from the full model and re-evaluate them on two generalization tasks, as shown in Table To demonstrate the effectiveness of the query-vision interaction module (QVIM), remove the query-guided average pooling (QGAP) and cross-attention module respectively, denotes as "w/o QGAP" and "w/o cross- both two tasks, which shows that
att". As we can see, the generalization performance drops the multi-layer QVIM plays an essential role in this task. To further verify our proposed query-guided feature augmentation (QFA) and whether it is necessary to apply constraints augmented data, we remove QFA and Lcoust, denotes as "w/o QFA" and "w/o Leonst". The results show that QFA can greatly advance the generalization ability of the model. Also without Cconst the performance drops, demonstrating the constraints on augmented data can guarantee the model generates more meaningful and query-preserving augmentations. Meanwhile, the AM-AdaIN can also improve the model's 's robustness by learning domain-agnostin vision-query relationship during training.
The Effect of QFA And AM-AdaIN in Different Layers We find through experiments that multi- layer QFA and AM-AdaIN will greatly reduce the performance of the model, thus we insert the two modules into only one QVIM layer respectively. In this subsection, we will study the effect of these two modules different scales. From the Figure (a) and (b) performance improvements on the model when K=I, and the worst when K=5, AM-AdalN has the best performance improvements when K=4. The results demonstrate the QFA is more suitable for
see that QFA has the best
8

---

Under review as a conference paper at ICLR 2023
Table 3: Analysis the bool matrix and augment matrix used QFA module.
A2R
R2A
A2J
Method
mAP
IoU
mAP
IoU
mAP
IoU
0.5:0.95
Overall
Mean
0.5:0.95
Overall
Mcan
0.5:0.95
Overall
Mean
w/o Bool Matrix
19.14
41.20
36.38
25.50
$2.28
43.09
39.15
65.78
65.32
w/o Augment Matrix
20.12
42.78
36.97
26.81
53.24
44.76
39.64
66.21
65.79
Full Model
21.36
44.06
38.31
29.09
54.73
46.05
41.23
67.87
67.19
black dog is walking on the left toddler yellow walking black lab
Query
black bird standing in the water
Video
Frame
Baseline
Ours
Figure 4: The visualization of segmentation results. The left is the result A2D Sentences of the R2A task. The right is the result on the RVOS of the A2R task.
enhancing visual features at larger scales, while AM-AdaIN is more suitable for deeper interactions between visual features and query representations.
The Effect of the Components in QFA To investigate the effectiveness of two matrices used in Eq 3.3 we remove the bool matrix B and augment matrix Mang respectively to test the model's performance on three tasks, as depicted in Table The results show that without bool matrix B, the performance will drop lot, which demonstrate it is important to the bool matrix to keep the query-related regions original visual features unchanged. The introduce augment matrix also guarantee that the query- guery-related objects in the auxiliary data will not interfere the original video features, which important for QFA module. Also we test whether is necessary to apply progressive manner in data augmentation, as shown in Figure 3). The results illustrate without progressive manner, it is difficult for the model to learn meaningful data augmentation, which will deteriorate the transferring performance.
4.5 QUALITATIVE ANALYSIS
To further qualitatively compare our method with baseline model, we visualized two segmentation results, shown Fig 4 From the results we can that compared with baseline model, the introduce of QFA and AM-AdaIN can help the model distinguish the query-related objects from the background more accurately. Notice that in the first demo, the baseline model can barely distinguish the dog from its surrounding objects. However, our full model can continuously replace the surrounding background regions during training, allowing the model to learn the difference between the objects and the surrounding environment more robustly. More qualitative results can be found in appendix material 6
5 CONCLUSION
In this paper, we introduce new task for domain generalization, generalizable query-based video segmentation, which aims to train a model on the source domain that can segment video objects according to the query text, and can also generalize to the unseen target domain. Previous methods are seldom designed for multi-modality tasks, especially when each modality has its own domain shift. To bridge this gap, we propose QFA and AM-AdaIN modules, which can combine the text and vision modalities together increase the holistic generalization ability of the model. We conduct extensive experiments on three datasets, and the results demonstrate that our methods can effectively promote the multi-modal domain generalization task.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Wuyang Chen, Zhiding Yu, Shalini De Mello, Sifei Liu, Jose M Alvarez, Zhangyang Wang, and Anima Anandkumar. Contrastive syn-to-real generalization. arXiv preprint arXiv:2104.02290 2021.
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selec- tive whitening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11580-11590 2021.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04800, 2018.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via mondel-agnostic learning of semantic features. Advances in Neural Information Processing Systems, 32, 2019.
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video Segmentation from sentence. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5958-5966, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexci Efros. and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. PMLR, 2018.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in Artificial Intelligence, pp. 292-302. PMLR, 2020.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomiza- tion for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6891-6902, 2021.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference on computer vision, pp. 1501-1510, 2017.
Tianrui Hui, Shaofei Huang, Si Liu, Zihan Ding, Guanbin Li, Wenguan Wang, Jizhong Han, and Fei Wang. Collaborative spatial-temporal modeling for language-queried video actor segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp.
4187-4196, 2021.
Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black. Towards understanding action recognition. In Proceedings of the IEEE international conference on computer vision, pp. 3192-3199, 2013.
Donghyun Kim, Yi-Hsuan Tsai, Bingbing Zhuang. Xiang Yu, Stan Sclaroff, Kate Saenko, and Manmohan Chandraker. Learning cross-modal contrastive features for video domain adaptation. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10- 2021, pp. 13598-13607. IEFF, 2021.
10

---

Under review as a conference paper at ICLR 2023
Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning to generalize semantic segmentation. arXiv preprint arXiv:2204.03609 2022.
Taekyung Kim, Minki Jeong, Seunghyeon Kim, Seokeon Choi, and Changick Kim. Diversify and match: domain adaptive representation learning paradigm for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 12456-12465, 2019.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018a.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8886-8895, 2021.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Yu Duan. Uncertainty modeling for -ut-t--distribution generalization. arXiv preprint arXiv:2202.0395.. 2022.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018b.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018c.
Chuang Lin, Zehuan Yuan, Sicheng Zhao, Peize Sun, Changhu Wang, and Jianfei Cai. Domain- invariant disentangled network for generalizable object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8771 8780, 2021.
Quande Liu, Qi Dou, and Pheng-Ann Heng. Shape-aware meta-learning for generalizing prostate mri segmentation to unseen domains. In International Conference on Medical Image Computing and Computer-Assister Intervention, pp. 475-485. Springer, 2020.
Bruce McIntosh, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. Visual-textual capsule routing for text-based video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9942-9951, 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pp. 10-18. PMLR, 2013.
Ke Ning, Lingxi Xie, Fei Wu, and Qi Tian. Polar relative positional encoding for ladou-lngguage segmentation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 948-954, 2021.
Fengchun Qiao and Xi Peng. Einertainty-guider model generalization to unseen domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6790-6800, 2021.
Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12556-12565, 2020.
Sconguk Sco, Joon-Yoone Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In European Conference on Computer Vision, pp. 208-223, Springer, 2020.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv: 10571, 2017.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. a887
2018. 2018.
11

---

Under review as a conference paper at ICLR 2023
Table Analysis of the parameter numbers of our proposed modules.
Baseline
QFA
Leonst
AM-AdalN
Parameters Numbers
109.71G
V
110.26G
V
V
112.36G
4
112.36G
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing unseen domains via adversarial data augmentation. Advances in neural information processing systems, 31, 2018.
Hao Wang, Cheng Deng, Junchi Yan, and Dacheng Tao. Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9939-3948, 2019.
Hao Wang, Cheng Deng, Fan Ma, and Yi Yang. Context modulated dynamic networks for actor and action video segmentation with language queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12152-12153 12159, 2020.
Chenliang Xu, Maao-Hang Hsieh, Caiming Xiong, and Jason Corso. Can humans fly? action understanding with multiple classes of actors. In Proceedings the Vision and Puttern Recognition, pp. 2264-2273 2015.
the IEEE Conference on Computer
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang. Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: siguence-to-sqquemee video object segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 585-601, 2018.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier- framework for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14383-14392, 2021.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- -innnenni-iiieettelli,, Kurt Keutzer, and Boqing Gong, Domain randomization and pyramid consistency: Simulation-to--eal generalization without accessing target domain data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2100-210 2110, 2019.
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entrropy adversarial data augmen- tation for improved generalization and robustness. Advances Neural Information Processing Systems,
2020.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In International Conference on Learning Representations, 2020.
A APPENDIX
A. EVALUATION METRICS
For IoU, we the "Overall IoU", which calculates the ratio of the total intersection divided by the total union area over the entire dataset, and the "Mean loU",, which first calculates the ratio of each sample and then obtains the average results the whole dataset. "P@ K" denotes compared with ground truth results, the IoU scores of testing samples are larger than K. We also measure the mean average precision different IoU thresholds from 0.50 0.95 with the step 0.05.
A.2 IMPLEMENTATION DETAILS
For natural language query inputs, set the maximum number of words in query as 20, and apply the Bevlin et al. (2018) as our text encoder. For video inputs, we employ the I3D network Carreira
Carreia & Zisserman 2017) pretrained on the Kinetics dataset extract the spatial and
12

---

Under review as a conference paper at ICLR 2023
The magle flying cver the
telling dag rolling on the ground
ball roiling man throwing boulling
green car is jumping and down
Figure 5: The visualization results of generalization task:RVOs
>A2D Sentences.
person wearing black jacket jumping on ramp
walking room past green
chid playing tennis on court
man attempting skareboarding trick
Figure The visualization results of generalization task:A2D Sentences RVOS part-
temporal features and we use the pre-trained ResNet-5(He et (2016) extract each video frame representations, the number of frames in one clip is 8. We divide the visual features into K 5 different scales, the sizes of them are 320 320, 160 x 160, 80 80, 40 40 and 20 20 separately, We set the hidden size of visual and query features as 512. Following Wang et al. 2019), the FCN network in deconvolutional layer contains three fully convolutional layers, where the kernel size is 3x3 for the first two layers and 1x1 for the remaining layer. For first 5 training epochs, we do not apply data augmentation and AdaIN in our baseline model, then we gradually increase the threshold from 0.05 to 0.30 by 0.05 every 10 training epochs. All experiments are implemented with Pytorch package on 4 NVIDIA V100 GPUs in this paper, the batch size is 16, and we use Adam optimizer with initial learning rate 1e-7.
For the implementation of other baseline methods, since the previous DG methods are all designed especially for single modality, here implement these models visual modality to test their performances, because we found that deploying them in visual modality works better than text modality. And if they are deployed in both modalities, the performances
worse
13

---

Under review as a conference paper at ICLR 2023
man runsing on the road
man black playing golf
top naked man coing pull up bar exardse
bay in gray shirt and black shorts swing baseball
Figure 7: The visualization results of generalization task:A2D Sentences JJ-HMDB Sentences.
A.3 DISCUSSIONS
The above experiments illustrate our proposed methods can effectively optimize the worst-case problem by generating novel domain S. At the initial training, S is close to original domain S. As the training progresses, / is gradually increasing, thus more background areas will enhanced, the distance D (S, is also getting larger. At the same time, the Lconst can guarantee the generated data has similar query-related regions to source data. Also our AM-AdaIN can introduce style randomization cross-attention map make the model learn query and visual feature. Furthermore, our methods consume very limited computing resources compared with the baseline model, the details be found in the next subsection.
robust relationship between
A.4 MODEL PARAMETERS
Our proposed modules only consume very limit computing resources compared with the baseline model, as depicted in Table The increases of parameters of QFA, Leonst and AM-AdalN are 0.55G, 2.1G, 0 respectively. Compared with traditional DG methods such meta-leariin and adversarial data augmentation, our model can effectively reduce parameters and can be casily deployed to multi-modal generalization tasks.
14


Reviews:
[{"review_id": "snRREJwSZs6", "reviewer": "Reviewer_yPJV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses the task of multi-modal DG where each modality has to cope with its own domain shift. Specifically, the specific scenario of query-based video segmentation is studied to better advance the generalization ability of the model in the multi-modal situation. The authors observe that actions belonging to the same type of actors in different domains share similar representations in the query-video latent space, and the main reason for the performance degradation is that visual background and contextual complexity vary widely across different domains. The authors propose a Query-guided Feature Augmentation (QFA) module that uses the attention scores between query and video frames to distinguish query-related foreground regions from unrelated background regions for synthesizing novel visual features. AdapIN is adopted to remove the impact of style in attention map during training, and gradually introduce statistics from other samples to help the model learn robust cross-modal relationships. Extensive experiments validate that the proposed model has better generalization ability.", "summary_of_the_review": "Please see the Strength and Weakness.\n\n---------\n\nAfter reading the author response and other fellow reviewers' comments, I see that some concerns have been addressed, while others still remains unsolved. Based on the technical content and the author response, I decide to revise the score.", "strength_and_weaknesses": "Strength:\n1.\tThis work addresses multi-modal DG problem in query-based video segmentation task, which is still an untouched field. Multi-modal DG is meaningful and has its specific challenge. The authors’ effort on handling the multi-modal DG problem is encouraged.\n2.\tThe authors observe that actions belonging to the same type of actors in different domains share similar representations in the query-video latent space, and the main reason for the performance degradation is that visual background and contextual complexity vary widely across different domains. Such observation is helpful.\n3.\tThe proposed QFA and AM-AdaIN are reasonable.\n4.\tExtensive experiments validate the effectiveness of the proposed model.\n\nWeakness:\n1.\tThe performances are over-claimed that “From the results we can see that our method outperforms the baseline model by a large margin on all three generalization tasks, demonstrating the effectiveness of our proposed QFA and AM-AdaIN modules.” In 4.3 Main result. The performance of A2R in Table I is just comparable to existing methods.\n2.\tThe proposed AM-AdaIN module is not illustrated in Fig. 2. It is hard to understand how it works in the proposed method.\n3.\tCan we replace the L_const into triplet loss, rather than Moco. Besides, the \\tau in L_const is not addressed.\n4.\tSome typos:\na)\tBold “c ” in c_{hw}^k of \\sigma(C_k)\nb)\tC’_K -> C’_k in Eqn. 9\nc)\tInconsistence between A2R and A2D Sentences to RVOSpart in Table I and Table II.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lVrR4bKk6Y", "reviewer": "Reviewer_XKxN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims at improving query-based video segmentation based on domain generalization. Specifically, this work focuses on solving the challenges of cross-modal settings, where domain shift exists at both the text and visual levels. To mitigate these challenges, this work proposes the query-guided feature augmentation method and uses AdaIN to modulate the text query.", "summary_of_the_review": "This work views query-based segmentation as a cross-domain generalization problem. It proposes a data augmentation scheme and uses AdaIN to improve model performance. The technical novelty of this paper is not up to the standard, and the motivations are not clear enough. In addition, the presentation needs further improvements.", "strength_and_weaknesses": "Strength\n+ A good attempt to improve domain generalization of the query-based video segmentation methods.\n+ The experimental results show that the proposed method outperforms the competing methods.\n\nWeaknesses\n+ This work proposes to augment training data with the proposed QFA. Only the background information is replaced during the augmentation. However, the foreground information also varies significantly for complex environments. For instance, as shown in Figure 1, the targets in the source and target domains have different scales. \n+ I am still confused about the detailed motivation for using AdaIN. It is originally used in style transfer, and this work seems to use it to mitigate the “style” of the query. However, AdaIN requires supervised training to present different styles, while the target domain data is not available during training for this task.\n+ The technical novelty of this paper is not up to the standard of ICLR. I believe that model performance is (marginally) improved but combining a data augmentation method and AdaIN and then applying them to the segmentation task is not appealing to me.\n+ The presentation should be improved. For instance, in the first sentence of the introduction, it seems that “byGavrilyuk” should be separated into two worlds by spacing. Similarly, “AdapINHuang” in the last paragraph of the introduction.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "eXEHtrThgK5", "reviewer": "Reviewer_HssY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. The paper attempts to introduce domain generalization to solve query-based video segmentation.\n2. The paper proposes QFA and AM-AdaIN modules to process query and video information, respectively.\n3. The authors conduct experiments on the A2D Sentences, Refer-Youtube-VOS (RVOS), and J-HMDB Sentences datasets.", "summary_of_the_review": "The paper lacks analysis and experimentation on research motivation. Some details of the method are unclear. I hope the author answers my question in the response. I'll change the score based on the author's reply.", "strength_and_weaknesses": "Strength:\n1. The research question is interesting.\n2. The author did some experimental comparisons, and the results show that the proposed method has improved on some baselines.\n\nWeaknesses:\n\n1 The paper experiment is not sufficient.\n\n-a. Does Table 1 compare other video segmentation methods? Does Table 1 compare with other domain adaption methods? Why does the title here compare with other generation methods? \n\n-b. From Table 2, the improvement of each module is minimal. Combining Tables 1 and 2, I doubt whether the motivation for introducing domain adaption is reasonable. I'm concerned that the author is simply doing a combination of methods and ignoring the rationale of the research motivation.\n\n2. Insufficient analysis of the rationality of domain adaption. \n\n-a. Does the paper have visualization results of domain adaption? I want to know what domain adaption means specifically for video segmentation tasks.\n\n3. The details of the method of the thesis are not clear. \n\n-a. How is Equation 1 trained? How does it relate to other loss functions (Equations 6 and 11)? \n\n-b. In Figure 2, what is the output of the augmented feature? Where did he enter the part of a? \n\n-c. Is the domain adaption of this paper merely from data augmentation?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bV2kOCsRXvP", "reviewer": "Reviewer_NKmv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies cross-modal domain gereralization on the query-based video-segmentation task. Two modules, i.e., Query-guided Feature Augmentation (QFA) and attention map adaptive instance normalization (AM-AdaIN) are proposed for the multi-modal generalization task. Experiments are conducted on three domain generalization tasks for query-based video segmentation.", "summary_of_the_review": "Novel domain generalization task but limited technical contrbution, good experimental results but insufficient analysis and insights behind the results, and generally fluent presentation.", "strength_and_weaknesses": "Strengths:\n1. The studied domain generalization task, i.e., multi-modal generalization for query-based video segmentation, is novel and has not been investigated before. The idea is interesting.\n2. Two modules are designed to address this challenging task, i.e. QFA and AM-AdaIN. This method achieves better results than the compared baslines. Ablation studies seem effective and sufficient.\n3. The presentation is generally good.\n\nWeaknesses:\n1. Motivation is not clear enough. Some source-only (baseline) examples would better show the requirement of domain generalization for query-based video segmentation.\n2. The technical contribution is relatively limited. Most components, e.g. both loss functions, are based on existing techniques.\n3. I expected there are at leaset two source domains so that domain-invariant representations can be learned. I do not understand why only one source domain is used. This is also domain generalization but not the typical setting.\n4. Baselines and some implementation details are not clear enough. For example, the compared baselines are mainly single-modal based methods. How to use them for the multi-modal generalization task?\n5. The analysis on why the proposed method performs better in the experiment part is weak.\n6. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject