Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text
Authors: Xiang Li
Jinglu Wang
Xiaohao Xu
Muqiao Yang
Fan Yang
Yizhou Zhao
Rita Singh
Bhiksha Raj
Venue: EMNLP 2023 Main (2023)
Keywords: Noisy Speech
Speech-Referring Video Object Segmentation
TL;DR: This paper presents STBridge, a novel method to enable noisy speech-referring video object segmentation.

Abstract: Linguistic communication is prevalent in Human-Computer Interaction (HCI). Speech (spoken language) serves as a convenient yet potentially ambiguous form due to noise and accents, exposing a gap compared to text. In this study, we investigate the prominent HCI task, Referring Video Object Segmentation (R-VOS), which aims to segment and track objects using linguistic references. While text input is well-investigated, speech input is under-explored. Our objective is to bridge the gap between speech and text, enabling the adaptation of existing text-input R-VOS models to accommodate noisy speech input effectively. Specifically, we propose a method to align the semantic spaces between speech and text by incorporating two key modules: 1) Noise-Aware Semantic Adjustment (NSA) for clear semantics extraction from noisy speech; and 2) Semantic Jitter Suppression (SJS) enabling R-VOS models to tolerate noisy queries. Comprehensive experiments conducted on the challenging AVOS benchmarks reveal that our proposed method outperforms state-of-the-art approaches.

Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text
Xiang Li
Jinglu Wang
Xiaohao Xu
Muqiao Yang' Bhiksha Raj*.
Fan Yang
Yizhou Zhao
Rita Singh'
Carnegie Mellon University 2 Microsooft 3 University of Michigan Ann Arbor
4 Ohio State University
Moohamed bin Zayed University of Artificial Intelligence
x x16@andrew. cmu, edu
Abstract
Video:
Linguistic communication prevalent in Human-Computer Interaction (HCI). Speech (spoken language) serves as convenient yet potentially ambiguous form due noise and accents, exposing gap compared to text. In this study, we investigate the prominent HCI task, Referring Video Object Segmentation (R- VOS), which aims to segment and track ob- jects using linguistic references. While text input is well-investigated, speech input under- explored. Our objective is to bridge the gap be- tween speech and lext, enabling the adaptation of existing text- input R- VOS models to accom- modate noisy speech input effectively. Specifi- seman-
R-VOS
(a)
Text:
Mask
"A pink car the grass"
Video:
R-VOS
(b)
Noisy Speech:
Mask
1. pink cvr on ill grapsl
STBridge
Figure 1: (a) Referring video object segmentation (R- VOS) employs text as a query for segmentation. (b) Speech-referring video object segmentation. Compared to written language (text), spoken language (speech) is a more noisy form, potentially involving greater infor- mation loss and disturbance due to background noises. We propose plug-and- ang--lay STBridge module, which seamlessly extends a frozen text-conditioned R- R-VOS model to accommodate noisy speech inputs.
cally, we propose method to align tic spaces between speech and text by incorpo- rating two key modules: 1) Noise-Aware Se- mantic Adjustment (NSA) for clear semantics extraction from noisy speech; and 2) Semantic Jitter Suppression (SJS) enabling R-VOS mod- els to tolerate noisy queries. Comprehensive periments conducted on the challenging AVOS benchmarks reveal that our proposed method outperforms state-of-the-art approaches.
However, more challenging scenario arises in the prevalent speech dialogue system, where the aim is to refer specific targets using spo- ken language, i.e., speech. The inherent nature of speech introduces vulnerabilities to disturbances from background noises (sound except for the re- ferring speech). Consequently, crucial information within the spoken content can be distorted or even lost, which poses extra challenges in maintaining effective segmentation when referring to targets ver- bally. Though previous R- R-VOS methods achieve remarkable performance with textual queries, their performance with real-world spoken language is rarely discussed. Hence, it is crucial to develop an effective approach to bridge the text and speech to adapt well tell-trained R- R-VOS methods (frozen) for speech inputs.
1
Introduction
Recent advances in vision- lision-language learning have significantly advanced Human-Compute Interac- tions (HCI). A demanding task within HCI is re- ferring video object segmentation (R-VOS), which involves segmenting and tracking objects in videos based on textual references. The successful devel- opment of R-VOS techniques has paved the way for diverse real-world applications such as video editing (Li et al., 2022d) and augmented reality (Huang et al., 2022). Notably, recent R VOS meth- ods (Luo et al., 2023) have shown unprecedented progress, propelled by the rapid advancement of multimodal foundation models such CLIP (Rad- ford et al., 2021). These R-VOS models enable various text-referred scenarios, allowing referring segmentation for generalized textual expressions even in complex visual scenes.
Yet, bridging speech R-VOS methods intro- duces new challenges. A straightforward solu- tion involves utilzing automatic speech recogni- tion (ASR) (Li et al., 2022a) to convert speech

---

text, followed by text-conditioned referring seg- mentation. However, this can result in suboptimal performance for two primary reasons. (1) Noise in language queries of R- R-VOS. Existing R- R-VOS mod- els rely on clean text-video pairs, wherein the tex- tual expression unambiguously identifies the target object. Nonetheless, information extracted from speech may be incomplete distorted due to back- ground noise and ASR errors, leading inadequate references to the target object. To maintain robust segmentation quality, is crucial to adapt R- E-VOS models to handle perturbed referring queries. (2) Noise in speech understanding. In practice, speech and background noise are closely intertwined, mak- ing it difficult to accurately comprehend semantic information from speech. Considering the diverse types of noise, effective noise-toleran speech understanding approach is vital for achieving ro- bust speech-referring video object segmentation.
applications. In summary, our contributions are follows:
We propose STBridge, a novel approach to bridge speech input to referring segmentation models, enabling segmenting objects with spo- ken language.
We introduce semantic-jitter suppression and noise-aware semantic adjustment modules to enable the noise-tolerant capability for speech queries.
We conduct extensive experiments on speech- referring segmentation benchmarks and the results of which show our approach performs favorably over prior arts.
2 Related Works
Video segmentation. Video segmentation (Wang et al., 2021c; Li et al., 2022b,c, 2023a; Yan et al.. 2023; Li et al., 2023c) is fundamental task enable video editing. Semi-supervised video ob- ject segmentation (VOS) which leverages first- frame mask to assign the target object among the most popular video segmentation tasks due to its high segmentation quality. Some recent works (Yang et al., 2020, 2021) propagate masks by exploring matches among adjacent frames. Space- Time-Memory networks (STM) (Oh et al., 2019) builds memory bank for matches. Several works follow the paradigm used in STM and improve the memory construction policy (Xie al., 2021; Liang et al., 2020; Wang et al., 202 or enhance the memory reading strategy (Cheng et al., 202 la; Scong et al., 2020; Hu et al.. 2021; Cheng et al.. 20216 lb; Seong et al., 2021; Yang et al., 2021). Re- cently, Yan et al. et al., 2023) introduced two-shot setting for VOS tasks which enables high- performance segmentation with limited annotated frames. Since the VOS task is primarily used for video editing which requires human involvement. to reduce the labor assigning the target object, referring video object segmentation (R-VOS) is in- troduced. Specifically, R-VOS aims to segment an object in a video sequence given linguistic description as the query. ReferFormer (Wu et al., 2022) and MTTR (Botach al., 2022) are two pio- neering works that utilize transformers to decode or fuse multimodal features. Recently, R2-VOS (Li et al., 2023b) introduces a cyclic structural consis- tency to enhance the robustness of R- VOS. And
In this paper, we present STBridge, novel approach that enables R-VOS models trained on clean text-video pairs to adapt to noisy speech as referring guidance, maintaining robust perfor- mance even amidst background noises. As illus- trated in Fig. 1, the proposed STBridge links the well-trained R-VOS model with speech input, in- corporating two core considerations improve the model's robustness: (1) enhancing the well- trained R-VOS model to accept incomplete guid- ance, and (2) providing the noise-tolerant capabil- ity for speech understanding. On the one hand, we introduce a semantic-jitter suppression (SJS) mod- ule to help the R-VOS model understand noisy information from referring guidance. The SJS module generates object queries with randomly jit- tered textual features, allowing the model to Iearn from incomplete referring guidance under proper supervision. On the other hand, we introduce a noise-aware semantic adjustment (NSA) module, which generates noise-adee aadstivee filters to enhance the speech representation. This differs from tra- ditional speech enbancement, as it focuses solely on encoded semantics during speech understand- ing, while discarding low-level information, i.e.. waveforms.
We further introduce a slack semantic alignment align text and speech queries, enabling the inte- gration of speech input with well-trained R-VOS models. Notably, our method incorporates addi- tional modules without any retraining of R- R-V models, which is essential for numerous roal-world

---

Video Encoder
Mask Decoder
Video V
Mask M
Text
perturb
Semantic Jitter Suppression
"A pink cut seating the grass" Text T
Encoder
9t
Semantic Alignment
"A pink seating on llle grass"
Speech Encoder
Ins
Noise-aware Semantic
Object Query
Adjustment
Training Only
Noisy Speech S
On
9s
Inference Only
Figure 2: Overview STBridge. We extend frozen VOS model (in blue) with trainable STBridge (in red) modules adapt to noisy speech input. (I) Training: A speech encoder is utilized to extract noisy speech Ins and noise 9n, embeddings from noisy speech S. On one hand, a noise-aware semantic adjustment (NSA) module is utilized to mitigate the noise influence, which derives cleaner speech embedding Is- On the other hand, enhance the noise-tolerant capability of R-VOS model, and then equip a semantic jitter suppression (SJS) module to suppress the noises. Moreover, semantic alignment constraints are introduced to align the text 9: and speech Is embeddings. speech embeddings during training, can directly discard the text branch and leverage speech embedding 9s the input to the SJS module.
first generate perturbation the text embedding
(II) Inference: After aligning the and
OnlineRefer (Wu et al., 2023) employs the query propagation module to enable the online R- VOS.
et al., 2023; Li et al., 2023c; Qu et al., 2023).
Method
Spoken language understanding. Spoken lan- guage, i... speech, enables a more natural way for humans to refer to certain object than us- ing text-based language. Thanks the emer- gence of datasets with paired images and speech, e.g., Flicker8K (Harwath and Glass, 2015) and AVOS (Pan et al., 2022), more works (Chrupata, 2022: Harwath et al., 2020; Kano et al., 2021; Seo et al., 2023) started to research on the repre- sentation of speech and explore the synergy be- tween speech and other modalities, e.g., image and video. For example, LAVISH (Lin et al., 2023) incorporates small set of latent tokens to align the visual and audio representation, and Visu- alVoiee (Gao and Grauman, 2021) conducts speech separation with the speaker's 's facial appearance as a conditional prior. Later, research on speech has also moved towards finer granularity tasks. Some works (Lei et al., 2021) focus on the mono-modal impact of speech to study the subtle semantic infor- mation of spoken language better understand hu- man speech, while others (Jiang et al., 2021) study how to introduce the knowledge of speech under- standing to create more natural human-compute interaction applications, e.g. talking head. (Hwang
To ground objects verbally, we start from a frozen text-referring video object segmentation (R-VOS) model (shown as blue modules in Fig. 2), includ- ing frozen video-text encoders and mask decoder. We introduce speech encoder, semantic jitter suppression (SJS) module, a noise-aware semantic adjustment (NSA) module, and a semantic align- ment constraint to bridge text and speech (shown as pink modules in Fig. 2). During training, STBridge leverages video V, text T, and noisy speech S triplets to align the query spaces between text and speech. Thereafter, and directly query the objects with speech during inference.
discard the text branch
3.1 Encoders
Frozen video and text encoders. We consider generic referring segmentation framework that equips video encoder &, and text encoder & to extract visual and textual features. Let de- the extracted visual feature as = (V) XC.XL.XXXIV and extracted text embeddings 9t = (T) CCL, where C₁ C and T.g, L, are the channel and length of visual and text embed- dings respectively. We freeze the video and text

---

encoders during both training and inference.
Fits
Ans
8x
Speech encoder. We leverage transformer- based speech encoder, Vec2 (Baevski et al.. 2020) to extract speech features. We addition- ally augment two linear layers top of the last hidden state of Wav2 to predict noise type. Thereby, each speech embedding corresponds to a noise embedding describe the noise informa- tion. We denote the extracted noisy speech em- bedding as Ins E RCxL, and noise embedding as In C CxL,, C and I, are the channel and length of embeddings.
Conv
Conv
Dyn Conv
Cross-Altn
S-N Cross-Altn
Conv
Conv
8
5n
Bi-directional Cross-attention
Nolse-aware Modulation
Figure 3: Illustration of Noise-aware Semantic Adjust- ment. Noisy speech and noise embeddings, i.e., Ins and 9n. first interact with each other via the bi-directional cross-attention mechanism. Then, the fused noise em- bedding 9,, is used to modulate the fused speech embed- ding Ins to make it more noise-aware.
3.2 Semantic Jitter Suppression
To equip the R-VOS model, which is typically trained on clean data samples, with the noise- tolerance capability, first mimic noisy text em- beddings of by applying semantic jitters to the orig- inal text embeddings Jt- After that, we introduce a learnable semantic jitter suppression block sp()) to suppress the jitter and generate proper object query q for the following mask decoding.
(3) where hn-s and his are outputs of N-S and S-N attention. K, Q, and V are derived by applying linear projections on the original speech or noise embedding. d is the dimension of K and Q. The hn-+s and harn are fused back to their paths with residual connections (He et al., 2016). We denote the fused embeddings as I's and gn.
hg = Softmax (ot kn/VV) Vn.
Specifically, we implement the semantic jitter with linear perturbation function where g = 9t + 8. Here, E {0, Cx is binary masking operation either word-level (along T, dimension) channel-level (along C' dimension); å e CC is a random noise; o denotes the Hadamard product. Besides, the jitter suppression block constructed by cascading a transformer encoder and a global average pooling layer which pools along the word dimension. Formally, the final object query q E RCx1 can be generated as
Noise-guided modulation (NGM). To incorpo- rate noise information into speech embeddings, we propose a noise-guided feature modulation with channel-wise attention (Tian et al., 2020). Differ- ent from attention in BCA acting along the time di- mension, channel-wise attention directly acting feature channels is more efficient to exploit seman- tically meaningful correlations (Wang al., 202 bb; Tian al., 2020), especially for instance-level cor- relations (Tian et al., 2020; Cao et al., 2020; Bolya et al., 2019). Given the speech-aware noise embed- ding Ixe. we first apply a fully connected layer on it to form the dynamic filters θ = [0 Here, each filter 02 E RCC1 represents the noise informa- tion for each timestep and modulates the speech embeddings according to their category and am- plitude. Then we utilize channel-wise attention modulate the noise-aware speech feature g', which is given by:
q = up o 9t +))
(1)
3.3 Noise-aware Semantic Adjustment
We introduce noise-aware semantic adjustment (NSA) adjust inaccurate semantics introduced by noises, which consists of two components: bi-directional cross-attention for noise-speech in- teraction and noise-guided modulation for speech embedding adjustment.
93hn θ ghs
(4)
Bi-directional cross-attention (BCA). In BCA, Noise-to-SS to-seech- (N-S) and Speech-to-Noise (S- N) attennattnn layers are involved to compute noise-aware speech embeddings g/ and speech- aware noise embedding g',- Formally, they take the form:
where Isin is the modulated speech embeddings and represents Hadmard product. We fuse the J back to g's with a residual connection, which derives the final output 9s = g's + 9s/n-
3.4 Frozen Mask Decoder
Referring segmentation methods typically leverage a query-based mask decoder D) (Wu et al.,
hy Softmax (T.k./Vd) V.

---

2022: Botach al., 2022) that takes an object query q E RCx1 encoding the object information and a video feature ʃ as inputs to predict the ob- ject masks M E IRNxxaxXoxxu, object bound- ing boxes B E NR.xx and confidence scores S E L across video frames. N is the ob- ject candidate number. Here, we omit the detailed structure (available in Appendix) for simplicity. It is worth mentioning that the object query for the decoder simply an averaged text embedding for recent popular R- R-VOS methods, which takes the form: 4 = pool(g:). The well-trained mask de-
Object segmentation. Following the object seg- mentation methods (Wu al., 2022: Wang et al., 2021c), we assign each mask prediction with a ground-truth label and then apply a set of loss functions between them to optimize the segmenta- tion mask quality. Given a set of predictions { and ground-truth y (Bi, St, Mi) where y {B.,, St,, M we search for an assignment o E Pv with the highest similarity where PN is a set of permutations of N elements. The similarity can be computed as
coder keeps frozen in our method.
Lenatch 0)
Aboar baau
(8)
Amask Lmask
3.5
Training Objectives
We utilize a semantic alignment loss Xalign to align speech and text queries, noise classification loss Froise to facilitate speech understanding, and a segmentation loss Lmatch to segment objects:
where Abwar Acenf and Amask are constant numbers to balance the losses. Following previous works (Ding et al., 2021; Wang et al., 2021c), we leverage combination of Dice (Li al., 2019) and BCE loss as Lmask: focal loss (Lin et al., 201 20) as Loouf and GloU (Rezatofighi al., 2019) and L1 loss as Lbor. The best assignment á is solved by the Hungarian algorithm (Kuhn, 1955).
C Malignhatigg + Anoise + Cmatch (5)
where Xatign and Anoise are constants.
Semantic alignment. To bridge the text and speech queries, we conduct semantic alignment between text and speech embeddings. As the ob- ject query requires sentence-level semantics (each sentence describes one object), it is not necessary to enforce a tight sequence- seweence-t--squeeee to-sequence alignment between text and speech embeddings (Tan et al., 2023). Instead, we align text and speech embed- dings with a loose constraint. Specifically, given a text embedding Or and a speech embedding Js. we first pool them among word and time dimen- sions correspondingly. After that, an alignment constraint is applied between them
3.6 Inference
During inference, we only keep speech and video as inputs. We first pool the speech embedding g's into a fixed size and then utilize to replace the text embedding gt- Thereby, noisy speech can replace text to query the visual object. Please note that the text branch stays functional as we froze the R-VOS model during the training of STBridge.
Experiment
4.1 Datasets and Metries
Datasets. We conduct experiments on the large- scale speech-referring video object segmentation dataset, AudioGuided-VO VOS (AVOS) (Pan al., 2022) which augments three R-VOS benchmarks with speech guidance: Ref-YoutubeVOSs (Seo et al., 2020), A2D-sentences (Xu et al., 2015) and JHMDB-sentencee (Jhuang et al., 2013). Specifi- cally, it involves 18,811 pairs of video sequences and speech audio, which is divided into the training, validation, and test set in a ratio of 0.75, 0. and 0.15, respectively. The AVOS test set only contains Ref- Youtube VOS samples. The A2D-sentences and JHMDB-sentences test sets are evaluated on their original test splits with speech as queries. Based on the AVOS dataset, we synthesize noisy speech by combining randomly picked audio from
Calign [[poo(g:) po))
(6)
where || |12 is the L2-Norm.
Noise classification. We augment the clean speech with different categories of audio, e.g., dog barking, noise. We apply noise classification head on top of noise embedding In to predict noise categories. Let us denote the predicted probabil- ities as E N.11 and the ground truth class as e, where N. is the noise type number. The noise classification loss Lnoise can be computed as
Lnoise logpld]
where pic] denotes the probability of class c.

---

No Noise
Noise (30 dB)
Noise (20 dB)
Noise (10 dB)
Method
Query
J
F
J
F
J
F
F
Text-referring Video Object Segmentation (for reference)
ReferFormer (Wu et al., 2022)
Text
66.3
68.1
MTTR (Botach al., 2022)
Text
63.2
64.7
Noisy Speech-refering Video Object Segmentation
ReferFormer (Wu et al., 2022)
Text (ASR)
56.
60.7
55.3
58.9
53.3
58. I
50.3
56.4
MTTR (Botach et al., 2022)
Text (ASR)
52.4
56.8
51.8
56.5
49.4
55.4
47.8
53.4
STBridge (Ours)
Speech
63.7
67.4
63.5
67.1
62.1
66.1
59.9
64.8
Table I:
Quantitative results for noisy speech-referring object segmentation in videos. The noise loudness (dB) is measured by the signal-noise ratio (SNR) to the clean speech.
Audioset (Gemmeke et al., 2017). Specifically, noise ranging from O to 40 dB signal-noise ratio (SNR) to the clean speech is sampled during train- ing. For validation and testing, we create noisy speech under 10 dB, 20 dB, and 30 dB SNR for comprehensive evaluation.
Method
Query
J&JF
J
J
Text-referring Video Object Segmentation (for reference)
ReferFormer
Text
67.2
66.3
68.
MTTR
Text
64.0
63.2
64.7
Speech-referring Video Object Segmentation
ReferFormer
Text (ASR)
58.4
56.1
60.7
MTTR STBridge (Ours)
Text (ASR)
54.6
52.4
56.8
Speech
65.5
63.7
67.4
Metries. We leverage the region similarity J and contour accuracy JF (Pont-Tuset et al., 2017) met- rics for the evaluation of speech-referring video object segmentation. The overall evaluation metric I &&F is the average of I score and F score. Both the J and F scores are the larger the better.
Table 2: Quantitative results for speech-referring object segmentation in videos. Both the J t and F t scores are the larger the better. J&F is the average of J and F convention.
4.2 Implementation Details
et al., 2020) to conduct ASR to adapt them to speech input. We notice that ASR-converted text will degrade the baseline models' performance without noise impact. We consider this result from word errors in the converted text from speech. For example, if the target object 'cat' is wrongly recognized as 'cap' by ASR, the R-VOS model will inevitably segment the wrong object.
We implement our method in PyTorch. Without losing generality, we leverage ReferFormer (Wu et al., 2022) as our frozen R-VOS model (can be replaced with any query-based model). We train our model for 2 epochs with learning rate of le-4. on 8 NVIDIA V100 GPUs. We adopt batchsize 8 and an AdamW (Loshchilov and Hutter, 2017) optimizer with weight decay x 10-4, Images are cropped to have the longest side 640 and the shortest side 360 during training and evaluation. In Eq. 1, we utilize the random noise 8 ~ Uniform - 0.5. 0.5) and a masking ratio of 0.1 for as default. Please refer to the Appendix for more details.
All experiments
Segmentation with noisy speech. As shown Table 1, we compare the performance of STBridge to previous text-queried methods with noisy speech as inputs. We modify the signal-noise ratio (SNR) of noisy speech to comprehensively evaluate the noise influence. We notice that -ased meth- ods suffer severe performance drops compared to clean speech. In contrast, STBrigde shows more robust performance with only slight degradation when noise becomes loud.
4.3 Quantitative Results
Segmentation with clean speech. Table 2 com- pares the proposed STBridge with previous meth- ods using the ResNet-50 (He et al., 2016) backbone. To better analyze the performance of STbridge, we introduce two popular R- R-VOS baselines (with text query), i.e., ReferFormer (Wu et al., 2022) and MTTR (Botach et al., 2022), and leverage Wav2Vec (same as our speech encoder) (Baevski
4.4 Visualization
In Fig. 4, we show the qualitative comparisons be- tween our method, i.e., STBridge, and cascade of Wav2Vec2 (Baevski et al., 2020) (ASR model) and ReferFormer (Wu et al., 2022) (RVOS model). Note that the ASR model is fairly chosen to have

---

Speech
young women seating on top of brown horse in water on the far right
goose standing in the second from the right
GT
Referf ormer
Ours
Figure 4: Qualitative comparison between STBridge (our referrine VOS model) and ASR-assisted Refer-
Former (an assembly of ASR (Baevski et al., 2020) and text-referring VOS (Wu al., 2022) models).
equipping of NSA module benefits the performance with both clean and noisy speech queries. We con- sider the reason is that the two types of attention in NSA better filter out irrelevant features and help the speech embedding focus on the target object. With all modules, STBridge achieves 65.6 and 62.4 I &FF for clean and noisy speech correspondingly.
(a) Original
(b) Jittered
(c) After SJS
Figure S: (a) Original embedding. (b) Semantic-jitte injected embedding. (c) Jitter-suppressed embedding.
Semantic jitter type. During training. STBridge generates semantic jitter and then learns to sup- press it using the SJS module to enhance the noise- tolerant capability of the R R-VOS model. We con- duct ablation study to investigate the influence of different semantic jitter types. Specifically, the implemented semantic jitter the text embedding Jt has form of Jt + where m and 8 are binary mask and random noise respectively. As shown in Table 4, we notice masking among both the word-level and channel-level shows an improve- ment in performance. Random noise brings an additional 0.5 J &.F to the final performance.
the same speech encoder as STBridgge Notably, our method successfully refers to the correct ob- ject while ASs-assisted ReferFormer fails to un- derstand the speech input and predicts wrongly. Fig. 5 demonstrates the function of the SJS mod- ule. By comparing Fig. (b) and Fig. 5 (c), we can notice that SJS module effectively suppresses noises in the jittered embeddings.
4.5 Ablation Experiments
We conduct ablation studies to show the impact of different modules. Unless otherwise specified, all experiments are conducted on the AVOS test set.
Design choices. We conduct experiments to ab- late the design choices STBridge and their im- pacts on the segmentation performance. (I) We first study the effect of frame window size selected from the entire video sequence during training. We notice that the window size only shows a marginal impact the performance, which can be due to the visual encoder and mask decoder being frozen during training. As shown in Table 5a, notice window size of achieves the best performance. (2) After that, we ablate on the loss type for semantic alignment Table 5b. We leverage T.1, T-2, and Co- sine loss to align the text and speech embeddings. We notice that La loss achieves the best perfor-
Module effectiveness. We conduct experiments to validate the effectiveness of our proposed mod- ules. We add the proposed modules step-by-step as shown in Table 3. (1) With semantic alignment be- tween textual and speech representation, STBridge achieves 61.0 and 55.7 J &-F with clean and noisy speech queries, respectively.
(2) After equipping the SJS module, STBridge can better handle noises thus boosting the performance with noisy speech to 59.7 J &FF while only marginal improvement is achieved with clean speech queries. (3) The

---

Module
Clean
Noisy
Semantic Jitter Type
J&F
SA
SJS
NSA
J&F
J&F
Word Mask
Channel Mask
Random Noise
61.0
55.7
61.2
61.3
59.7
61.4
63.9
61.4
61.9
65.5
62,4
62.4
Table 3: Module effectiveness with clean and noisy speech as queries. SA: semantic alignment. SIS: semantic jitter suppression. NSA: noise- semantic adjustment.
Table 4: Ablation on semantic jitter types. We conduct abla- tions on the impact created semantic jitter types in the SIS module. We conduct this experiment with noisy speech (10 dB).
L
3
5
7
Type
L L2 Cosine
Ratio 0.05 0.1 0.2
Amp. 0.1 0.5
J &
59.4 61.7
62.4 62.3
J & 62.1 62.4 60.9
J&F 61.7 62.4 60.3
J J&F 61.1 62.4 62.0
(a) Window size.
(h) Alignment loss.
(c) Masking ratio.
(d) Noise amplitude.
Table Design choices for STBridge. report the performance with the noisy speech queries on AVOS test set.
(a) We ablate the window size (input frame number) during training. (b) We ablate the semantic alignment loss types. (c) We ablate the making ratio for in creating semantic jitters. The ratio is calculated by sum(m) (d)
We ablate the amplitude of the random noise d. The noise ~ Uniform(-Ane Amp).
by mixing clean speech from AVOS test set with different categories of audio recordings from Au- dioset (Gemmeke al., 2017). As shown in Fig. 6, we illustrate the results of queries with different noise categories. We notice that sustained and loud noises, e.g., ambulance siren, can lead to a severe performance drop compared to short-lived and faint noises, e.g., horse clip-clop.
60
J &
30
Lan gon chateeceel oeyale aubolaace stres
pooel Cong
deving hole Raliaae forn goo
plano
ealsa stolde viets tywag keyhoard
5 Conclusion
In conclusion, this paper presents STBridge, a novel approach that enables R- R-VOS models trained on clean text-video pairs to adapt to noisy speech as referring guidance, maintaining robust perfor- mance. The approach incorporates semantic jitter suppression (SJS) and noise-aware semantic adjust- ment (NSA) modules to enhance noise tolerance in speech queries. Experimental results demonstrate the effectiveness of STBridge, outperforming pre- vious methods on three benchmarks. STBridge ex- pands the applicability of R- VOS models, enabling robust speech-referred video object segmentation in real-world scenarios.
Figure 6: Analysis of the impact of noise categories.
mance among them. (3) Semantic jitter suppression is an essential component in STBridge for noise tolerance. We conduct ablation studies to demon- strate the impact of different masking ratios and random noise amplitude. Table 5c demonstrates the performance with different masking ratios of E [0, CL (calculated as 1 sum(m)
Cxl, Small masking ratios cannot provide enough perturbation to the inputs while large ratios may lose the seman- tics to the target object. We find masking ratio of 0.1 is good trade-off as shown in Table 5c. (4) We ablate the amplitude of noise d added as semantic jitter in Table 5d. We notice that amplitude of 0.5 leads to the best performance.
Limitation. In spite of STBrdge's high perfor- mance on existing benchmarks, we only consider the scenario that text and speech queries are in the same language. Bridging text and speech in dif- ferent languages can impose more challenges as the semantic spaces may suffer more divergence, which will be our future focus.
Noise category. We conduct an experiment to in- vestigate the impact of different noise categories. We additionally synthesize noisy speech queries

---

References
In 2015 IEEE Workshop on Automatic Speech Recog- nition and Understanding (ASRU), pages 237-244, IEEE.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for supe-supe learning of speech representations. Advances in neural information processing systems, 33: 12:449--12440.
David Harwath, Wei-Ning Hsu, and James Glass. 2020. Learning hierarchical discrete linguistic units from hisually-grounder speech. In International Confer- ence Learning Representations.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. 2019. Yolact: Real time- instance segmentation. In Proceedings of the TEEE/CVF international con- ference on computer vision, pages 9157-9166.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. 2022. End-to-end referring video object segmentation with multimodal transformers. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
Li Hu, Peng Zhang. Bang Zhang, Pan Pan, Yinghui Xu, and Rong Jin. 2021. Learning position and target consistency for memory-basee video object segmen- tation. arXiv preprint arXiv:2104.044
Jiale Cao, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. 2020. Sipmask: Spatial information preservation for fast image and video instance segmentation. In Computer -ision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XIV 16, pages 1-18. Springer.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. 2022. Multi-view transformer for 3d visual ground- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15524-15533.
Geumbyeol Hwang, Sunwon Hong. Seunghyun Lee, Sungwoo Park, and Gyeongsu Chae. 2023. Disco- head: Audio-and-video-diieee tion by disentangled control of head pose and facial expressions. In ICASSP 2023-2023 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. 2021a. Modular interactive video ob- ject segmentation: Interaction-to-mask, propaga- tion and difference-aware fusion. arXiv preprint MrXiv:2103.07941.
talking head genera-
Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. 2021b. Rethinking space-time time-timemne networks with im- proved memory coverage for efficient video object segmentation. arXiv preprint arXiv:2106.0522
Hueihan Jhuang, Juergen Gall, Silvia Zuffi. Cordelia Schmid, and Michael J Black. 2013. Towards under- standing action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3192-3199.
Grzegorz Chrupata. 2022. Visually grounded models of spoken language: A survey of datascts, architec- tures and evaluation techniques. Journal of Artificial Intelligence Research, 73:673-707.
Yuming Jiang, Ziqi Huang. Xingang Pan, Chen Change Loy, and Ziwei Liu. 2021, Talk-to-edit: fine-ggraned facial editing via dialog. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13799-13808.
Zihan Ding, Tianrui Hui, Shaofei Huang. Si Liu, Xuan Luo, Junshi Huang, and Xiaoming Wei. 2021. Pro- gressive multimodal interaction network for refer- ring video object segmentation. The 3rd Large-scale Video Object Segmentation Challenge, page 7.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. 2021. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790.
Ruohan Gao and Kristen Grauman, 2021. Visualvoice: Audio-visual speech separation with cross-modal consistency. In 2021 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 15490-15500. IEEE
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. 2021. Transformer-bace direct speech-to-speech translation with transcoder. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 958- 965. IEEE
Jort F Gemmeke, Daniel PW Bllis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore. Manoj Plakal, and Marvin Ritter. 2017. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780. IEEF.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceed- ings of the 2014 conference empirical methods in natural language processing (EMNLP), pages 787- 798.
David Harwath and James Glass. 2015. Deep multi- modal semantic embeddings for speech and images.

---

Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):33-97.
Tsung- Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017b. Focal loss for dense object detection. In Proceedings of the IEFF international conference on computer vision, pages 2980-2988.
Yi Lei, Shan Yang, and Lei Xie. 2021. line-grained emotion strength transfer, control and prediction for emotional speech synthesis. In 2021 IEEE Spoken Language Technology Workshop (SLT", pages 423- 430. IERE.
Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, and Gedas Bertasius. 2023. Vision transformers are aurameter-efficiee audio-visual learners. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2299-2309.
Jinyu Li al. 2022a. Recent advances in end-to-end automatic speech recognition. APSIPA Transactions on Signal and Information Processing, 11(1).
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711.0510:
Xiang Li, Haoyuan Cao. Shijie Zhao, Junlin Li, Li Zhang, and Bhiksha Raj. 2023a. Panoramic video salient object detection with ambisonic audio guid- ance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1424-1432.
Zhuoyan Luo. Yicheng Xiao, Yong Liu, Shuyan Li. Yi- tong Wang, Yansong Tang, Xiu Li, and Yujuu Yang. 2023. Soc: Semantic-assisted object cluster for re- ferring video object segmentation. arXiv preprint arXiv:2305.1 17011.
Xiang Li, Jinglu Wang, Xiao Li, and Yan Lu. 2022b. Hy- brid instance-awar temporal fusion for online video instance segmentation. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 36, pages 1429-1437.
Joon-Young Seoung Wug Oh, Joon- Young Lee, Ning Xu, and Scon Joo Kim. 2019. Video object segmentation using space-time memory networks. In Proceedings of the TEEE/CVF International Conference on Com- puter Vision, pages 9226-9235.
Wenwen Pan, Haoman Shi, Zhou Zhao, Jieming Zhu, Xi- uqiang He, Zhigeng Pan, Lianli Gao. Jun Yu, Fei Wu, and Qi Tian. 2022. Wnet: Audio-guided video object segmentation via wavelet-based cross-modal denois- ing networks. In Proceedings of the IEFF/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 1320-1331.
Xiang Li, Jinglu Wang, Xiao Li, and Yan Lu. 2022c. Video instance segmentation by instance flow assem- bly. IEEE Transactions on Multimedia.
Xiang Li. Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj. and Yan Lu. 2023b. Robust referring video ob- ject segmentation with eyclic structural consensus. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. 2017. The 2017 davis challenge on video object segmentation. arXiv preprint 1r.ivv:104.0000 0067000
Xiang Li. Jinglu Wang, Xiaohao Xu, Xiulian Peng. Rita Singh, Yan Lu, and Bhiksha Raj. 2023c. Re- thinking audiovisual segmentation with semantic quantization and decomposition. arXiv preprint arXiv:2310.00132.
Liao Qu, Xianwei Zou, Xiang Li, Yandong Wen, Rita Singh, and Bhiksha Raj. 2023. The hidden dance of phonemes and visage: Unveiling the enigmatic link between phonemes and facial features. arXiv preprint 2rXiv:2307.13953
Xiaoya Li, Xiaofei Sun, Yuxian Meng. Junjun Liang, Fei Wu, and Jiwei Li. 2019. Dice loss for data-imbalanced nlp tasks. arXiv preprint arXiv: Seti::9911.0285.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR.
Zhen Li, Ch Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng, 2022d. Towards an end-to-end framework for flow-guided video inpainting. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17562-17571.
Hamid Rezatofighi, Nathan Tsoi, Jun YunYoun Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 458-666.
Yongqing Liang, Xin Li, Navid Jafari, and Jim Chen. 2020. Vidco object segmentation with adaptive fea- ture bank and uncertain-region refinement. Advances in Neural Information Processing Systems, 33.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017a. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125.
Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. 2023. Avformer: Injecting vision into frozen speech models for zero-shot av-asr. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22922-22931.

---

Seonguk Sco, Joon- Young Lee, and Bohyung Han. 2020. Urvos: Unified referring video object segmentation network with large-scale benchmark. In Com- puter Vision-ECCV 2020: 16th European Confer- ence, Glasgow. UK, August 23-28, 2020, Proceed- ings, Purt XV 16, pages 208 223. Springer.
Chenliang Xu, Shao-Hang Hsich, Caiming Xiong, and Jason Corso. 2015. Can humans fly? action under- standing with multiple classes of actors. In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition, pages 2264-2273.
Kun Yan, Xiao Li, Fangyun Wei, Jinglu Wang, Chenbin Zhang, Ping Wang, and Yan Lu. 2023. Two-shot video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2257-2267.
Hongje Seong, Junhyuk Hyun, and Buntai Kim. 2020. Kernelized memory network for video object seg- mentation. European Conference on Computer Vision, pages 629 645. Springer.
Zongxin Yang. Yunchao Wei, and Yi Yang. 2020. Col- laborative video object segmentation by foreground- background integration. In European Conference on Computer Vision, pages 332-348. Springer.
Hongje Seong, Seoung Wug Oh, Toon-Young Lee, Seongwon Lee, Suhyeon Lee, and Euntai Kim. 2021. Hierarchical memory matching network for video object segmentation.
Zongxin Yang. Yunchao Wei, and Yi Yang. 2021. As- sociating objects with transformers for video object segmentation. Advances in Neural Information Pro- cessing Systems, 34.
Yi Xuan Tan, Navonil Majumder, and Soujanya Poria. 2023. Sentence embedder guided utterance encoder (segue) for spoken language understanding. Proc. Interspeech 2023.
Zhi Tian, Chunhua Shen, and Hao Chen. 2020. Con- ditional convolutions for instance segmentation. In Computer Vision-ECCo 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part 16, pages 282-298, Springer.
Haochen Wang. Xiaolong Jiang, Haibing Ren, Yao Hu, and Song Bai. 2021a. Swiftnet: Real-time video ob- ject segmentation. arXiv preprint S0tiv:2212..46611
Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chich Chen. 2021b. Max-deeplab: End- to-end panoptic segmentation with mask transform- ers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5463-5474.
Yuging Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2021c. End-to-end video instance segmentation with transformers. In Proceedings of the IEEF/CVF Con- ference on Computer Vision and Pattern Recognition, pages
Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen. 2023. Onlinerefer: A simple online baseline for referring video object seg- mentation. In Proceedings of the IEEE/CVF Interna tional Conference Computer Vision (ICCV), pages 2761-2770.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. 2022. Language as queries for referring video object segmentation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 4974-4984.
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Sheng- ping Zhang, and Wenxiu Sun. 2021. Efficient re- gional memory network for video object segmenta- tion. arXiv preprint arXiv:2108 12934.

---

A More Implementation Details
Mr for each time step is derived by dynamic convolution between protolype mask Pt and dy- namic weights which are learned from instance embedding é by two fully connected layers. The prototype masks { (P:) is generated by feature pyramid network (FPN) (Lin et al., 2017a) with visual feature (1)))
Training details. Following the previous refer- ring segmentation methods (Wu et al., 2022; Bo- tach et al., 2022; Kamath et al., 2021), we leverage loss weight coefficients Adice and Xfocal to bal- ance Dice (Li al., 2019) and focal (Lin et al., 2017b) losses in maskk- And Agion and λL to balance GIoU (Rezatofighi et al., 2019) and LI losses Lioz. During training, we set Anoise = Aatign = 1, Aconf = Agion = Adice = 2, and ALI = Afooal = 5. We set the layer number in the transformer encoder in yp 3. The frozen ReferFormer (Wu et al., 2022) is pre-trained on Ref-COCO/+/g (Kazemzadeh et al., 2014) for 12 epochs and then finetuned the AVOS training set while with text queries as input for 6 epochs. For the text, speech, and video triplets used in STBridge training, we only ensure the text and speech describe the same object while the words in the text and speech may differ slightly.
C Inference Details of R-VOS models
To obtain the final segmentation result, we select the mask (among N candidates) with highest confi- dence throughout time as:
Mj Ma
(9)
8 = arg max (Si,1 + +... S
where {M, Y- is the masks referred object. Sa, and Mi, represent the i-th slot in St and M, respec- tively. á is the slot with the highest confidence to be the target object. Box predictions can help training procedures but are not used during inference.
During STBridge training, the noisy speech in- put generated by mixing the original clean speech with randomly picked audio, which is injected noise, from AudioSet with an SNR ranging from to 40. The noise categories include 'am- bulance siren',: baby laughter', 'gun shooting'. 'cat meowing'. "chainsawing trees', 'coyote howling". 'dog barking'. 'driving buses', 'helicopter', 'horse clip-clop'. 'lawn mowing', 'lions roaring'. 'bird singing', 'guitar', "glockenspiel,, 'piano", 'tabla', 'ukulele', 'violin', 'race car', 'typing keyboard'. The sampling probabilities for each category the same.
Frozen J&F
J
F
65.5
63.7
67.4
58.9
57.3
60.4
Table A: Ablation study on updating R-VOS parameters during training.
D Training with Trainable R- R-VOS Model
We conduct additional ablation studies to show the results of training with updating parameters in R- VOS model. As shown in Table A, we notice that updating R-VOS parameters during the adaptation to noisy speech inputs will result in severe perfor- mance degradation. We consider this because 1) the information in noisy speech input not enough to accurately refer to the object resulting in noises in the training process, and 2) the S- V-SOS dataset is smaller than the R-VOS dataset leading to overfit- ting.
B Detailed Structure of Mask Decoder
We demonstrate the detailed mask decoding pro- cess in Figure A. Except for the visual feature fe and object query q as defined in the main paper, additionally enroll the prototype masks J and instance embedding {e,) Given the object query g E CCx1 from STBridge, we first repeat N times to form the input to the transformer de- coder TrD where N is the object candidate number (the final output is selected from object candidates based on confidence score). After that, we gen- crate instance embedding {Er)_] for each time step separately using a shared transformer decoder TrD with encoded memory {f:)T= from visual encoder. The instance embedding here encodes the instance information and is leveraged to guide the mask decoding process. The mask prediction
E More Visualization.
As shown in Fig. B, demonstrate more visual- izations of the proposed method. We notice that our method can correctly refer to the target object and help the R- R-VOS model segment temporally consistent object masks across frames.

---

M
M
M
Pi
Pc
Prototype Masks {P:)/=1
FC
FC
FC
e
FPN
Transformer Decoder TrD
Transformer Decoder TrD
Transformer Decoder TrD
f
Video Feature (fi)e=1
Repeat
X Dot Product
Object Query
Figure A: Illustration of mask decoder, which derives mask predictions {Mr) from the visual features (ft) the object query q. Given the query feature y from STBridge, we first repeat it N times to form the input to the transformer decoder TrD where N is the object candidate number (the final output is selected from object candidates based confidence score). After that, generate instance embedding {er (= for each time step separately using shared transformer decoder TrD with visual feature (ft)== from visual encoder. The mask prediction M, for each time step is derived by a dynamic convolution between prototype masks P; and dynamic weights which are learned from instance embedding e, by two fully connected layers. The prototype masks (P.)?- is generated by feature pyramid network (FPN) (Lin et al., 2017a 7a) with visual feature [fe]T=

---

Speech
a white and brown owl standing next to a white cat
GT
Ours
Speech
the giraffe walking around
GT
Ours
GT Speech
a lizard in small enclosure with two others outside and eat
Ours
GT Speech
a person grabbing a crocodile
Ours
Speech
a zebra is standing to the left of view
GT
Ours
Figure More visualization our method on AVOS test set.


Reviews:
[{"review_id": "yvTPGBGzGY", "reviewer": "Reviewer_uQBw", "summary": "", "questions": "1. What are the essential difference between noise in query and noise in ASR (line 067 vs line 076). From my understanding, the first noise is due to ASR errors which is mostly caused by the second noise.\n2. Following the above question, can you leverage the background noise from audio signals as contextual information for better queries?\n3. Is it possible to extend your idea to ASR-free queries, i.e., taking raw audio signal as input to the R-VOS model?\n4. Can you use entity/keywords extraction from speech instead of ASR?\n5. The approach relies on training data with the triplet set of video, text and audio, all synced. Is there enough data like this? How could you leverage the data with only aligned video & audio, or text and audio, etc?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. What is the essential difference between \"noise in query\" and \"noise in ASR\" (Line 067/Line 076)? From my understanding, the first noise is due to ASR errors which are mostly caused by the second noise.**\n\nWe agree that the \"noise in query\" is mostly caused by \"noise in ASR\". We separately analyze them because they occur in different components of the model and can be addressed differently. We summarize the differences as follows:\n\n|Noise|Module|Our Solution|\n|:----| :----| :----|\n|in language query|Frozen R-VOS model|Adapt R-VOS model to tolerate noisy queries (Line 075 -> SJS module)|\n|in speech understanding|Speech encoder|Enhance speech encoder to extract more accurate embedding (Line 082 -> NSA module)|\n\nWe will highlight the differences in the revision.\n\n---\n\n**2. Can you leverage the background noise from audio signals as contextual information for better queries?**\n\nWe consider that the background noise has been incorporated as contextual information in the noise-aware semantic adjustment module (NSA). As shown in Fig 3, we first extract a noise embedding $g_n$ which encodes the background noise information and then learn dynamic kernels from it to modulate the noisy speech embedding $g_{ns}$. In this way, we can incorporate the background noise as contextual information to enhance speech query learning. As shown in Tab 3, NSA module brings 2.9 $\\mathcal{J}$\\&$\\mathcal{F}$ and 5.7 $\\mathcal{J}$\\&$\\mathcal{F}$ gain with clean and noisy speech queries respectively.\n\n---\n\n**3. Is it possible to extend your idea to ASR-free queries? Can you use entity/keyword extraction from speech instead of ASR?**\n\nWe would like to clarify that STBridge is an ASR-free method that enables end-to-end speech-referring VOS. As shown in Fig 2, we apply the semantic alignment constraint between speech embedding $g_s$ and text embedding $g_t$ during training. In this way, we can directly obtain the object query $q$ from the aligned speech embedding during inference. \n\nWe recognize that it might be misleading to use an example of an ASR-based solution to introduce the challenges (Line 063). We will revise the wording to enhance clearance in the revision.\n\n---\n\n**4. The approach relies on a triplet set of video, text, and audio. Is there enough data like this? How could you leverage the data with only aligned video \\& audio or text and audio?**\n\nIn this work, we utilize three popular datasets containing 18,811 videos with more than 150,000 frames. For each video, a text and a speech describing the target object are included. Given this extensive dataset, we consider it sufficient to demonstrate the effectiveness of the proposed method.\n\nTo further scale up for practical consideration, data augmentation is necessary to create the required data triplet. When only video & audio pairs are available, we consider the text can be first generated by off-the-shelf ASR model and then train STBridge with the video-audio-text triplets. For the scenario with only audio & text pairs, we consider the proposed semantic alignment constraint can still be leveraged. Specifically, the speech encoder can be trained by distilling knowledge from the frozen text encoder. We notice several multimodal foundation models in a variety of fields (such as captioning, text-to-speech, and segmentation) have been proposed recently. By leveraging those powerful foundation models to create pseudo labels, training with limited data access becomes feasible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method provides solution to the important field of speech driven video object segmentation with many application scenarios.\n2. The novelty is sufficient though the solution is a modularized approach which provides step-by-step functions to deal with a challenging task. Hopefully this could lead to a more end-to-end solution from audio signals to VOS.\n3. The experiments are thorough with good results and ablation studies.\n4. The reference looks comprehensive to me.\n5. The writing is clear and easy to follow.", "reasons_to_reject": "Please refer to my questions below for possible improvement, mostly in the approach, or at least in the writing to clarify a few details.", "questions_to_authors": "1. What are the essential difference between noise in query and noise in ASR (line 067 vs line 076). From my understanding, the first noise is due to ASR errors which is mostly caused by the second noise.\n2. Following the above question, can you leverage the background noise from audio signals as contextual information for better queries?\n3. Is it possible to extend your idea to ASR-free queries, i.e., taking raw audio signal as input to the R-VOS model?\n4. Can you use entity/keywords extraction from speech instead of ASR?\n5. The approach relies on training data with the triplet set of video, text and audio, all synced. Is there enough data like this? How could you leverage the data with only aligned video & audio, or text and audio, etc?", "paper_topic_and_main_contributions": "This paper proposed a novel approach that adapts R-VOS models trained on clean text-video pairs to noisy speech input.  To align the semantic spaces between speech and text by incorporating two key modules: 1) Noise-Aware Semantic Adjustment for clear semantics\nextraction from noisy speech; and 2) Semantic Jitter Suppression enabling R-VOS models to tolerate noisy queries. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "No concerns."}, "extra_text": {}}, {"review_id": "XDr3eDjgH4", "reviewer": "Reviewer_A74t", "summary": "", "questions": "A. How does the proposed method handle cases where the spoken descriptions are ambiguous or inconsistent with the video content? Are there any strategies or techniques to handle such cases?\n\nB. Have you considered using noise reduction techniques to first reduce the noise in speech and then extract text through ASR to perform the R-VOS task? If so, how effective is this approach?\n\nC. How sensitive is the proposed method to variations in speech quality or the levels of noise in the speech? Did you observe any significant performance degradation with extremely noisy speech (e.g. SNR higher than 40 dB) or low-quality recordings?\n\nD. Have you tested the proposed method on languages other than English? If so, what were the challenges and how did the method perform in those cases?\n\nE. Have you considered incorporating other modalities (e.g. visual cues or lip movements) along with speech to improve the performance of the proposed method? If so, what were the challenges and potential benefits of such an approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. Lack of analysis of failure cases.**\n\nThanks for the valuable suggestion. We observe that STBridge faces challenges in robustly segmenting target objects within crowded scenes, particularly when dealing with multiple objects belonging to the same class. We attribute this suboptimal performance to the fact that STBridge relies on a sentence-level embedding (same as the underlying R-VOS model), to capture both entity and contextual information (such as color, position, and action of the object). Consequently, in the case of complex sentences, certain aspects of contextual information might be overlooked during the process of feature extraction.\n\nTo improve, we consider it could be beneficial to encode the entity and each contextual information (e.g., color, position, and action) into separate embeddings to better handle complex descriptions. Nevertheless, applying this approach requires retraining the currently frozen R-VOS model to adjust the object query generation, which may impair the current plug-and-play usability of STBridge. \n\nWe will consider the potential improvement in our future work. We will include the failure case analysis in our revision.\n\n**2. The evaluation is conducted on benchmark datasets, and it would be beneficial to include real-world scenarios or scenarios with more complex visual scenes.**\n\nWe appreciate the reviewer for raising the concern regarding real-world scenarios. We concur that a more extensive evaluation with more complex scenes would be valuable. Nevertheless, due to the substantial manual effort required for collecting ground-truth masks for evaluating video segmentation, we were regrettably unable to carry out this evaluation during the rebuttal period. We will keep the constructive suggestion in mind and include the suggested evaluation in our future work.\n\n---\n\n**3. How does the proposed method handle cases where the spoken descriptions are ambiguous or inconsistent with the video content? Are there any strategies or techniques to handle such cases?**\n\nAs STBridge aims to extend a frozen R-VOS model with noisy speech-referring capability, its behavior with ambiguous or inconsistent referring guidance can depend on the underlying R-VOS model. \n\nWe notice that a recent study $\\mathrm{R}^2$-VOS proposes a cyclic structure consistency to detect and filter out the false positives caused by inconsistent text-video pairs in R-VOS task (no output when the referred object does not exist in the video). By applying STBridge to the $\\mathrm{R}^2$-VOS, we believe that ambiguous and inconsistent spoken descriptions can be successfully tackled.\n\n[R1] Robust Referring Video Object Segmentation with Cyclic Structural Consensus, ICCV 2023\n\n---\n\n**4. Have you considered using noise reduction techniques to first reduce the noise in speech and then extract text through ASR to perform the R-VOS task? If so, how effective is this approach?**\n\nThanks for your suggestion. We conduct an additional comparison between STBridge and ASR baseline equipped with a popular speech enhancement method, SepFormer [R2]. As shown in Tab A, we notice that even when assisted with speech enhancement, the ASR baseline is still suppressed by STBridge. As shown in Tab 2, when no noise is added, STBridge still outperforms ASR baselines which indicates the upper bound of utilizing speech enhancement. \n\n|Method|$\\mathcal{J}$\\&$\\mathcal{F}$|\n|:----:|:----:|\n|ASR| 53.4|\n|ASR+SE| 58.0|\n|STBridge|62.4|\n\nTable A. Comparision with ASR and ASR+speech enhancement (SE) baselines under 10dB noise.\n\n[R2] https://huggingface.co/speechbrain/sepformer-wham16k-enhancement\n\n---\n\n**5. How sensitive is the proposed method to variations in speech quality or the levels of noise in the speech? Did you observe any significant performance degradation with extremely noisy speech or low-quality recordings?**\n\nWe ablate on the different SNR of noise as shown in Tab 2. The lower the SNR the larger the noise. Compared to no noise, the performance with 10 dB noise drops 3.2 $\\mathcal{J}$\\&$\\mathcal{F}$. We conduct an additional experiment with more noisy speech (5 dB), and the results drop significantly by 23.5 $\\mathcal{J}$\\&$\\mathcal{F}$ compared to no noise which indicates the speech encoder fails to understand the content in such a noise level and the model randomly segment objects without a clear guidance. \n\n---\n\n**6. Have you tested the proposed method on languages other than English? If so, what were the challenges and how did the method perform in those cases?**\n\nRegrettably, we have not been able to conduct such tests due to the absence of publicly available datasets catering to other languages. It's important to note that STBridge is not tailored exclusively to English, and its design allows for potential applicability in different languages.\n\nNonetheless, we recognize that if the R-VOS model is primarily trained in English (as is commonly the case) while the target speech is in a different language, it could present challenges in aligning the distinct speech and text spaces. The divergence between the English text space and the speech characteristics of other languages may pose hurdles to effective alignment.  \n\n---\n\n**7. Have you considered incorporating other modalities (e.g. visual cues or lip movements) along with speech to improve the performance of the proposed method? If so, what were the challenges and potential benefits of such an approach?**\n\nThe integration of other modalities, such as visual cues or lip movements, in conjunction with speech, represents a promising avenue for enhancing the performance of speech-referring VOS. Here, we outline the potential benefits and challenges associated with this approach:\n\n- Benefits:  Visual cues, encompassing facial expressions, gaze, and lip movements, can provide supplementary information alongside speech, aiding in disambiguation during dialogue, particularly in noisy environments. We believe that the inclusion of visual cues could benefit speech-referring VOS by enhancing speech comprehension, thereby enabling the extraction of more precise object queries for segmentation.\n\n- Challenges: While visual cues offer complementary data, effectively leveraging them to facilitate speech comprehension remains an open challenge. Specific challenges include:\n   \n    - Handling Incomplete Visual Observations: Given that facial features, including lips, can be partially obscured or occluded, devising strategies to deal with incomplete or partial visual observations is crucial when extracting visual features.\n\n    - Balancing Speech and Visual Information: Striking the right balance between relying on speech and incorporating visual cues without overemphasizing one over the other is a key challenge to address.\n\n    - Temporal Alignment: Given that temporal information is critical for speech comprehension, aligning visual and audio features presents an important consideration.\n\nWe will include the discussion in our revision.\n\n---\n\nWe would like to thank the reviewer for the valuable comments which deeply inspire us for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n1. Novel Approach: The paper proposes a new approach, STBridge, to bridge the gap between speech and text in R-VOS tasks. The incorporation of two key modules, NSA and SJS, effectively addresses the challenges of noisy speech input and aligning the semantic spaces between speech and text.\n2. Robust Performance: The comprehensive experiments conducted on three challenging benchmarks demonstrate that the proposed method outperforms state-of-the-art approaches. STBridge achieves significant improvements in both clean and noisy speech input scenarios, highlighting the effectiveness of the proposed modules.\n\nBenefits to the NLP Community:\n1. Expanded Scope: The paper addresses a gap in the existing literature by exploring the underexplored area of speech-input R-VOS models. By adapting text-input R-VOS models to accommodate noisy speech input effectively, the proposed method expands the scope of R-VOS research and opens up new possibilities for multimodal HCI tasks.\n2. Noise-Tolerant Speech Understanding: The robustness of the proposed method to handle noisy speech queries provides valuable insights for researchers working on speech understanding tasks. This can contribute to the development of more accurate and reliable speech-based models, benefiting various NLP applications beyond video object segmentation.\n3. Practical Applications: The practical applications of the proposed method, such as video editing and augmented reality, have significant implications for the NLP community. The successful adaptation of text-input R-VOS models to handle noisy speech input enables the development of more user-friendly and natural human-computer interaction systems, enhancing the user experience in real-world scenarios.", "reasons_to_reject": "Weakness:\nLack of analysis of failure cases: The paper focuses on the performance improvements achieved by the proposed method but does not analyze the failure cases where the method might not perform well. Understanding the limitations and failure cases of the approach would provide a more comprehensive evaluation.\n\nRisk:\nLack of real-world scenarios: The evaluation is conducted on benchmark datasets, and it would be beneficial to include real-world scenarios or scenarios with more complex visual scenes. This would provide a more realistic evaluation of the approach.", "questions_to_authors": "A. How does the proposed method handle cases where the spoken descriptions are ambiguous or inconsistent with the video content? Are there any strategies or techniques to handle such cases?\n\nB. Have you considered using noise reduction techniques to first reduce the noise in speech and then extract text through ASR to perform the R-VOS task? If so, how effective is this approach?\n\nC. How sensitive is the proposed method to variations in speech quality or the levels of noise in the speech? Did you observe any significant performance degradation with extremely noisy speech (e.g. SNR higher than 40 dB) or low-quality recordings?\n\nD. Have you tested the proposed method on languages other than English? If so, what were the challenges and how did the method perform in those cases?\n\nE. Have you considered incorporating other modalities (e.g. visual cues or lip movements) along with speech to improve the performance of the proposed method? If so, what were the challenges and potential benefits of such an approach?", "paper_topic_and_main_contributions": "The paper addresses the problem of referring video object segmentation (R-VOS) using speech input, which is more challenging compared to using text input due to noise and potential information loss. The paper makes several contributions to the field of R-VOS and speech-referring segmentation:\n\nFirstly, it addresses the gap between text and speech input, which is an underexplored area in the field. It introduces a method to align the semantic spaces between speech and text, enabling the adaptation of text-input R-VOS models to handle noisy speech input effectively. This allows for a more natural and convenient way for humans to refer to objects in videos using speech.\n\nSecondly, the paper proposes the Noise-Aware Semantic Adjustment (NSA) module, which enhances the R-VOS models to handle incomplete and distorted information from noisy speech. It focuses on extracting accurate semantics from noisy speech, enabling effective segmentation even with perturbed referring queries.\n\nThirdly, the paper introduces the Semantic Jitter Suppression (SJS) module, which helps the R-VOS models tolerate noisy queries by generating perturbations and suppressing noise in the textual features. This module allows the models to learn from incomplete referring guidance and adapt to noisy speech input.\n\nThe paper provides comprehensive experiments on three challenging benchmarks, demonstrating the superiority of the proposed method compared to state-of-the-art approaches. The experiments cover both clean and noisy speech input scenarios, showcasing the noise-tolerant capabilities of the proposed method.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4bpRycQnwk", "reviewer": "Reviewer_zoVN", "summary": "", "questions": "Another benefit of this end2end approach compared to using ASR, is inference speed. I'm curious of the quantitative comparison.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.\n\n---\n\n**1. The type of noise is limited to only background noise but the definition of noise in speech is much broader. I suggest the authors modify the paper to be clearer on what kind of noise is being tackled here.**\n\nWe appreciate your valuable suggestion. Indeed, STBridge is specifically designed to mitigate the impact of background noise in speech. We will make necessary revisions in the paper to explicitly clarify this focus on background noise.\n\n---\n\n**2. Only a limited pool of noise categories (21 kinds in total) are used as injected noise, and therefore not sure how this approach generalizes when OOD background noise appears in the speech query.**\n\nTo evaluate the performance of OOD background noise, we train a model with 21 seen noise categories and test it on 10 randomly picked unseen noise categories from AudioSet. As shown in Tab A, we notice the performance with unseen noise is slightly lower than that with seen noise while still showing obvious superiority compared to training with clear speech (no noise). This indicates the noise-tolerant capability of STBridge can be generalized to unseen noise categories.\n\n|Method|Train Noise| Inference Noise |$\\mathcal{J}$\\&$\\mathcal{F}$|\n|:----|:----:|:----:|:----:|\n|STBridge|NA| Unseen| 55.7 |\n|STBridge|Seen | Seen| 62.4 |\n|STBridge|Seen| Unseen| 61.0 |\n\n\n\nTab A. STBridge with unseen noise type during inference.\n\n---\n\n**3. The info regarding the ASR model (size, pretraining/finetuning data) is missing. It might be worth considering whether the proposed approach is still competitive compared to a stronger ASR model, e.g. Whisper.**\n\nWe leverage the Wav2Vec2-Base model pre-trained and finetuned on Librispeech 960h as the speech encoder for ASR baselines which shares the same architecture used in STBridge. To investigate the performance with a stronger ASR model, we conduct additional experiments to utilize Whisper-Large as the ASR models as shown in Tab B. We notice that the model with Whisper-Large shows a slightly superior performance than STBridge. However, it's important to highlight that Whisper-Large demands significant computational resources, which in turn negatively affects inference speed, as detailed in Tab C. \n\n|Method|$\\mathcal{J}$\\&$\\mathcal{F}$|\n|:----|:----:|\n|ASR baseline (Wav2Vec2-Base)| 58.4 |\n|ASR baseline (Whisper-Large)| 66.6 |\n|STBridge (Wav2Vec2-Base)| 65.5 |\n\nTab B. Comparison of models with different ASR models.\n\n---\n\n**4. Comparison of inference speed between end2end STBridge and baseline using ASR.**\n\nWe present the inference speed results for both our end-to-end STBridge and the ASR baseline models in Tab C. Notably, STBridge demonstrates a superior inference speed compared to the ASR baselines. Specifically, when employing the Wav2Vec2-Base model, STBridge outperforms the ASR baseline by 5.0 FPS. Remarkably, in a comparison against the computationally demanding Whisper-Large, STBridge exhibits a substantial improvement of 26.1 FPS.\n\n|Method|FPS|\n|:----|:----:|\n|ASR Baseline (Whisper-Large) | 5.2 |\n|ASR Baseline (Wav2Vec2-Base) | 26.3 |\n|STBridge (Wav2Vec2-Base) | 31.3 |\n\nTab C. Comparision of model speeds with batchsize=1 on a single NVIDIA P40 GPU. FPS: frames per second.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "two modules - noise-aware semantic adjustment, and semantic jitter suppression are proposed to handle the noisy speech input for semantic information extraction. Extensive ablation studies showed that the proposed modules are effective. The proposed approach is effective even when no noise is injected (table1). The impact of different noise types are also investigated, and they show that \"sustained and loud noises can lead to a severe performance drop compared to short-lived and faint noises\"", "reasons_to_reject": "1. The type of noise is limited to only background noise (and from AudioSet specifically), but the definition of noise in speech is much broader - reverberation, far-field, low bandwidth, missing segments, accents. Accents in particular appeared in the abstract as a motivating factor, but never mentioned in the main text of the paper. \nThe approach contains a background noise class prediction module, which could mean that the approach only works for background noise. If this is the case, I suggest the authors modify the paper to be clearer on what kind of noise is being tackled here.\n\n2. The generalization ability. Only a limited pool of noise categories (21 kinds in total) are used as injected noise, and therefore not sure how this approach generalize when OOD background noise appear in the speech query.\n\n3. The info regarding the ASR model are largely missing. The only info the author presented in the paper is that it's W2V2, but the size, pretraining/finetuning data are missing. It might be worth considering whether the proposed approach is still competitive compared to a stronger ASR model, e.g. Whisper", "questions_to_authors": "Another benefit of this end2end approach compared to using ASR, is inference speed. I'm curious of the quantitative comparison.", "paper_topic_and_main_contributions": "This paper tackles the task of speech query based referring video object segmentation (R-VOS). To handle the noisy nature of speech queries, the authors proposed two modules, namely noise-aware semantic adjustment, and semantic jitter suppression. Experiments show that the proposed modules are effective, and their approach achieves the state-of-the-art performance on three referring video object segmentation benchmarks, outperforming both speech R-VOS models, and ASR + text R-VOS models, closing the gap between speech based models and text based models using ground truth text.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach that enables referring video object segmentation (R-VOS) models trained on clean text-video pairs to adapt to noisy speech as referring guidance and maintain robust performance. The authors achieve this by using Noise-Aware Semantic Adjustment for precise semantics extraction from noisy speech without using ASR and Semantic Jitter Suppression, which makes R-VOS models more robust to noisy queries. Experiments on three challenging benchmarks demonstrate the superiority of the proposed method compared to state-of-the-art approaches.

The reviewers praised the novelty of the proposed solution, many practical applications, clear writing and robust experimental analysis. The authors successfully answered all of the reviewers' questions during the rebuttal.
Decision: Accept-Main