Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models

## Abstract

Abstract We introduce the first zero-shot approach for Video Semantic Segmentation (VSS) based on pre-trained diffusion models.
A growing research direction attempts to employ diffusion models to perform downstream vision tasks by exploiting their deep understanding of image semantics.
Yet, the majority of these approaches have focused on image-related tasks like semantic correspondence and segmentation, with less emphasis on video tasks such as VSS.
Ideally, diffusion-based image semantic segmentation approaches can be applied to videos in a frame-by-frame manner.
However, we find their performance on videos to be subpar due to the absence of any modeling of temporal information inherent in the video data.
To this end, we tackle this problem and introduce a framework tailored for VSS based on pre-trained image and video diffusion models.
We propose building a scene context model based on the diffusion features, where the model is autoregressively updated to adapt to scene changes.
This context model predicts per-frame coarse segmentation maps that are temporally consistent.
To refine these maps further, we propose a correspondence-based refinement strategy that aggregates predictions temporally, resulting in more confident predictions.
Finally, we introduce a masked modulation approach to upsample the coarse maps to the full resolution at a high quality.
Experiments show that our proposed approach outperforms existing zero-shot image semantic segmentation approaches significantly on various VSS benchmarks without any training or fine-tuning.
Moreover, it rivals supervised VSS approaches on the VSPW dataset despite not being explicitly trained for VSS.
We released our code in https://github.com/QianWangX/VidSeg_diffusion .

## 1 Introduction

Diffusion models [, , , , ] have showcased remarkable capabilities in learning real data distributions effectively, allowing them to generate high-quality images and videos with soaring diversity and fidelity.
This was achieved by exploiting their scalability to train on large-scale datasets [, ].
Interestingly, those models were able to learn profound abstractions of images and their semantics as an indirect consequence of their large-scale training.
This entitled them to be considered as *Foundation Models* that possess high degrees of generalizability and understanding of images.
As a result, a growing direction of research attempts to use the internal representations of diffusion models to perform various downstream image vision tasks.
For instance, pre-trained diffusion models were used to perform image semantic correspondence [, ], keypoints detection [], and image semantic segmentation [, ].

Video vision tasks, on the other hand, have not received the same attention compared to their image counterparts.
A simple approach is adopting image-based approaches to solve video tasks.
For example, the diffusion-based image semantic segmentation approach EmerDiff [] can be employed to perform Video Semantic Segmentation (VSS) in a frame-by-frame manner.
However, our initial investigations show that it performs poorly in segmenting videos in terms of temporal consistency, as shown in \Creffig:teaser.
This can be attributed to the lack of modeling of video temporal information, causing inconsistent predictions across frames.

An intuitive solution for enhancing the temporal consistency of these approaches is employing a video diffusion model,   e.g., Stable Video Diffusion (SVD) [] as a backbone.
The SVD model is initially trained on images, then expanded with temporal layers and fine-tuned on videos.
Ideally, the temporal features from SVD should exhibit better temporal consistency.
To investigate this hypothesis, we visualize the temporal features of a pre-trained SVD in \Creffig:svd_feat to examine their temporal consistency.
The figure shows that the temporal features are surprisingly unstable and tend to change significantly between video frames.
On the other hand, the spatial features encode the structure of the scene, similar to Image Stable Diffusion [] (SD).
Based on these observations, we capitalize on the spatial features of either SD or SVD and attempt to enhance them temporally.

Figure: Figure 2: A visualization of the first three PCA components for the features of two video frames extracted from the most semantically-rich blocks in SD (Block 7) and SVD (Bock 8). In the second row, we show the $x$-$t$ slice of an image row (highlighted in the red line in the leftmost image) horizontally across the PCA visualization ($x$-axis) and stack it chronologically across the full batch of video frames ($t$-axis). The plot shows that the spatial features of both SD and SVD are temporally more consistent between video frames compared to the features of temporal layers in SVD.\cref@constructprefixpage\cref@result
Refer to caption: x2.png

In this paper, we introduce a diffusion-based zero-shot approach for VSS.
First, we build a *scene context model* based on the features of a pre-trained image (SD) or video (SVD) diffusion model.
This context model produces per-frame coarse segmentation maps and is autoregressively updated to accommodate scene changes throughout the video.
To further enhance the temporal and spatial consistency of these coarse maps, we propose a correspondence-based refinement (CBR) strategy encompassing a pixel-wise voting scheme between the video frames.
Finally, we propose a masked modulation process to reconstruct the full-resolution segmentation maps that is more stable and less noisy than that of [].
Experiments show that our proposed approach significantly outperforms zero-shot image semantic segmentation methods on various VSS benchmarks.
More specifically, we improve mIOU over image semantic approaches by at least 29% on VSPW, CityScapes, and Camvid datasets.
Remarkably, our approach performs comparably well as supervised VSS approaches on the diverse VSPW dataset.
We also show that for currently released models, SD features lead to a higher quality result than SVD features, but this trend may reverse when SVD training considers larger datasets in the future.

## 2 Related Work

### 2.1 Diffusion Models’ Features

The large-scale training of diffusion models on the LAION-5B dataset [] with 5 billion images allowed them to learn semantically rich image features.
As a result, a growing direction of research attempts to employ these features to perform downstream vision tasks.
Several approaches [, , ] investigated using these features to perform semantic correspondence.
They observed that the features are semantically meaningful and generalize well across different objects and styles of images.
For example, a human head in any arbitrary image will have similar features to any other human head in other images, regardless of the scene variations.
Those features even generalize across similar classes of objects like animal heads.
At the same time, the features will differ from those of unrelated object classes like buildings, landscapes, vehicles, *etc*.
EmerDiff [] capitalized on this observation to perform image semantic segmentation.
Since the diffusion features are distinct for different objects, they can easily be clustered to separate those objects and produce a coarse segmentation map.
Then, they proposed a modulation strategy to produce fine segmentation maps at a remarkable quality.
However, we observed that the produced segmentation maps by EmerDiff are not temporally consistent, as illustrated in \Creffig:teaser, making it unsuitable for Video Semantic Segmentation (VSS).
Therefore, we propose a diffusion-based pipeline tailored to improve the temporal consistency in VSS.

### 2.2 Video Semantic Segmentation

Video semantic segmentation (VSS) [, , , , , , ] is a spatiotemporal variation of image segmentation on videos that aims to predict a pixel-wise label across the video frames.
Those predictions should be temporally consistent under object deformations and camera motion, making VSS more challenging than its image counterpart.
Recent approaches attempted to exploit the temporal correlation between video features to produce temporally consistent predictions.
Several approaches [, ] incorporated optical flow prediction to model motion between frames.
Other approaches [] proposed a temporal consistency loss on the per-frame segmentation predictions as an efficient replacement for optical flow.
TMANet [] utilized a temporal attention module to capture the relations between the current frame and a memory bank.
DVIS [] further improved the efficiency by treating VSS as a first-frame-segmentation followed by a tracking problem.
Recent work UniVS [] proposed a single unified model for all video segmentation tasks by considering the features from previous frames as visual prompts for the consecutive frames.
Despite their remarkable performance, these supervised approaches do not generalize well on unseen datasets [].
Therefore, it is desired to have an approach that generalizes well across datasets.
Inspired by the success of EmerDiff [] on image semantic segmentation, we attempt to exploit the diffusion features to propose the first temporally consistent zero-shot VSS approach.

## 3 Preliminaries

### 3.1 Stable Diffusion architecture

Stable Diffusion (SD) [, ] is one of the prominent latent diffusion models that achieves a good tradeoff between efficiency and quality.
It is trained to approximate the real data distribution by adding noise to the latents of data samples until they reach pure Gaussian isotropic noise.
During sampling, it performs a series of Markovian denoising steps starting from pure noise to recover a noise-free latent that is decoded to produce a synthetic image.
SD utilizes a UNet architecture to predict either the noise or some other signal at each time step.
This UNet encompasses multiple blocks for the encoder and the decoder at different resolutions ranging from $8\times 8$ to $64\times 64$, where every block has residual blocks, self-attention, and cross-attention modules.
The attention is computed as:

$$ $\cref@constructprefix{page}{\cref@result}f\left(\sigma\left(\frac{QK^{T}}{ \sqrt{d}}\right)\cdot V\right)$ (1) $$

where $Q,K,V$ are the query, key, and value vectors in the attention layers, $\sigma$ is the Softmax activation, and $f$ denotes a fully connected layer.
The query is always computed from the image features, while the key and the value are computed from the image in self-attention and a conditional signal (*e.g.*  textual prompt) in cross-attention.
Since the semantically rich features are located in the decoder [, ], We only consider the decoder blocks.
The decoder has 12 blocks over 4 resolutions, where We refer to the first block as 0, with a resolution of $8\times 8$, and the last block as 11, with a resolution of $64\times 64$.

### 3.2 Emerging Image Semantic Segmentation from Diffusion Models

page\cref@result
EmerDiff [] observed that it is possible to extract semantically rich features from some of the UNet blocks and use them to produce coarse semantic segmentation maps.
Given an RGB image $X$ with a resolution of $H\times W$, the spatial resolution of the low-dimensional UNet features becomes $H/S_{i}\times W/S_{i}$, where $S_{i}$ is the scale factor of block $i$ determined by both the size of the latent representation and the downsampling factor of that block.
By applying K-Means clustering on the low-dimensional feature maps from Block $b_{k}$ at timestep $t_{k}$, we obtain a set of binary masks $\mathcal{M}=\{M_{1},M_{2},...,M_{L}\}$, where $M\in\mathbb{R}^{H/S_{i}\times W/S_{i}}$, and $L$ is the number of distinct clusters.

A *modulation* strategy is used to obtain fine-grained image-resolution segmentation maps.
This is achieved by modulating the attention module at block $b_{m}$ and denoising timestep $t_{m}$ for each binary mask $M_{l}$ based on the following formula:

$$ $\cref@constructprefix{page}{\cref@result}f\left(\sigma\left(\frac{QK^{T}}{ \sqrt{d}}\right)\cdot V\pm\lambda M_{l}\right),$ (2) $$

where $\lambda$ controls the degree of modulation.
The intuition behind this process is to add or subtract a certain amount of perturbation $\lambda$ on the region specified by mask $M_{l}$ and then continue the denoising process to reconstruct a modulated image.
By applying $+\lambda$ and $-\lambda$, we get two different modulated images denoted as $I_{l}^{+}$ and $I_{l}^{-}$ respectively.
Then, a difference map is computed as $D_{l}=\|I_{l}^{+}-I_{l}^{-}\|^{2}$, where $D_{l}\in\mathbb{R}^{H\times W}$.
This is repeated for all masks in $\mathcal{M}$ to get a set of difference maps $\mathcal{D}=\{D_{1},D_{2},...,D_{L}\}$.
Finally, the full-resolution segmentation map is computed as $Y=\arg\max_{l}\mathcal{D}\ .$

## 4 Method

Figure: Figure 3: Our Video Semantic Segmentation (VSS) approach encompasses three stages. In Stage 1, we initialize a Scene Context Model $\Omega$ as a KNN classifier with the aggregated diffusion features $\widetilde{F}^{1}$ of the first frame and a coarse mask $M^{1}$ produced by K-Means clustering. In Stage 2, we use the context model $\Omega$ to predict coarse masks for the remaining frames in the batch $M^{2\dots B}$. We refine the coarse maps $M^{1\dots B}$ using our correspondence-based refinement (CBR). In Stage 3, we use the refined coarse masks to modulate the attention layers of the diffusion process with factor $\pm\lambda$ to obtain a modulated latent $\hat{z}_{t}$. Then, we blend $\hat{z}_{t}$ with the original unmodulated latent $z_{t}$ using the coarse masks to obtain a less noisy latent $\widetilde{z}_{t}$. Finally, the latent $\widetilde{z}_{t}$ is decoded to obtain images $I^{+},I^{-}$ that are used to compute a set of difference maps per segment $l\in L$. The final predictions are made by applying an $\arg\max$ operation over the difference maps similar to []. The process is repeated for the following batch of frames where the context model is updated in an autoregressive manner using the coarse masks $M^{1\dots B}$ and their corresponding features $\widetilde{F}^{1\dots B}$. \cref@constructprefixpage\cref@result
Refer to caption: x3.png

Our method encompasses three main components, as illustrated in \Creffig:workflow.
First, we construct a *scene context model* to produce coarse segmentation maps based on the diffusion features (\Crefsec:scm).
Then, we introduce a *correspondence-based refinement* strategy to curate the coarse segmentation maps (\Crefsec:cbr).
Finally, we propose a *masked modulation* approach that produces less noisy and more stable full-resolution segmentation maps (4.3).
We also provide some details on adapting our approach to employ features from Stable Video Diffusion (SVD) in \Crefsec:svd.

### 4.1 Scene Context Model

page\cref@result
Image segmentation algorithms are designed for segmenting individual images and can only process videos in a frame-by-frame manner.
This is not ideal for videos, as the per-frame predictions will be completely independent and consequently temporally inconsistent.
To address this limitation, we propose to create a scene context model that is initialized at the first frame and then updated throughout the video in an autoregressive manner.

Given a video sequence $\mathcal{X}=\{X^{1},X^{2},\dots,X^{N}\}$ with $N$ frames, we extract diffusion features $F^{n}_{i}$ for all frames in $[1,N]$, where $i$ is the decoder block.
Since different blocks have different information, we aggregate features from multiple blocks by averaging to produce an aggregated feature $\widetilde{F}^{n}$.
We found that aggregating blocks 6, 7, and 8 attains the best results (see \Crefsec:abl).
Note that these blocks share the same resolution and number of channels.
Then, we process the video in batches of length $B$.

For the first batch, we use K-Means to extract the initial coarse segmentation map for the first frame ${M}^{1}$ based on the aggregated features $\widetilde{F}^{1}$.
Given the diffusion features $\widetilde{F}^{1}$ and the coarse map ${M}^{1}$ as labels, we train a KNN classifier $\Omega({M}^{1},\widetilde{F}^{1})$ as a context model to discriminate between different clusters.
Then, we use the context model $\Omega$ to predict the coarse segmentation maps for the remaining frames in the first batch $\{{M}^{2},{M}^{3},\dots,{M}^{B}\}$.
We refine the coarse maps further using the correspondence-based refinement (\Crefsec:cbr) and use them alongside their aggregated diffusion features to update the context model as $\Omega([\widetilde{M}^{1},\widetilde{M}^{2},\dots,\widetilde{M}^{B}],[
\widetilde{F}^{1},\widetilde{F}^{2},\dots,\widetilde{F}^{B}])$.
The context model is then used for the next batch.
This strategy ensures that the context model $\Omega$ adapts to changes in the video in an auto-regressive manner.

### 4.2 Correspondence-Based Refinement

page\cref@result
Since the context model operates purely in the feature space, it is unaware of the spatial arrangement of clusters or how they develop temporally.
This might cause inconsistencies between clusters, especially across borders between objects.
To alleviate this, we propose a refinement strategy based on the semantic correspondence between consecutive frames.
We compute per-pixel correspondences (in the coarse map resolution) similar to [] based on the diffusion features of block $c$ between images $j$ and $j+1$ to produce a correspondence-based coarse segmentation $\hat{M}^{j}$ :

$$ $\hat{M}^{j}[p]=\begin{cases}\arg\max_{q}\Gamma(F^{j}_{c}[p],F^{j+1}_{c}[q])\ , &\text{if}\parallel p-q\parallel^{2}\leq\mathcal{T}\\ &\\ {M}^{j}[p]\ ,&\text{Otherwise}\\ \end{cases},$ (3) $$

where $\Gamma$ is a distance metric that we choose to be the cosine similarity.
The threshold $\mathcal{T}$ discards faulty matches that are spatially unplausible.
These correspondences are computed for all pixels $p$ and over all frames within the batch.
Finally, we perform a majority voting for all pixels over the temporal axis to produce the final coarse segmentation map $\widetilde{M}^{j}$:

$$ $\widetilde{M}^{j}[p]=\arg\max_{l\in L}\sum_{j=0}^{B-1}\mathbf{1}(\hat{M}^{j}[p ]=l)\enspace.$ (4) $$

Here, $\mathbf{1}(\hat{M}^{j}[p]=l)$ is an indicator function that equals 1 if $\hat{M}^{j}[p]=l$ and 0 otherwise.
This interplay between the context model and the correspondence-based refinement leads to more accurate predictions.
The context model encodes the feature space of the video batch, while the refinement strategy spatially and temporally regularizes the predictions.
This setup is illustrated in \Creffig:cbr.

Figure: Figure 4: A detailed illustration of (a) the scene context model and (b) correspondence-based refinement.\cref@constructprefixpage\cref@result
Refer to caption: x4.png

### 4.3 Masked Modulation

page\cref@result
The modulation process aims to upsample the coarse segmentation masks to the full-resolution of the video frames.
When applying the modulation process, it is only the modulated regions that are expected to change, as explained in \Crefsec:emerdiff.
However, in practice, the modulation process produces noise outside that region, causing discrepancies when computing the final segmentation labels.
Therefore, we propose a masked modulation process that employs the coarse segmentation map to mask out both the latents and the difference maps outside the modulated region.
For a denoising timestep $t$, we blend the latents as follows:

$$ $\cref@constructprefix{page}{\cref@result}\widetilde{z}_{t}=z_{t}*(1-M_{l})+ \hat{z}_{t}*M_{l},$ (5) $$

where $z_{t}$ is the latent from the unmodified sampling step, $\hat{z}_{t}$ is the latent from the modulated sampling process, and $M_{l}$ is the low-resolution mask we are modulating.
Despite the fact that the modulation is only performed at timestep $t_{m}$, we apply latent blending after $t_{m}$ until timesteps $t_{f}$, as once the attention map of a certain timestep is modified, it will influence all the following denoising timesteps.

To further suppress the noise in the difference map, we can apply the same blending strategy to the difference maps.
We compute the filtered difference map $\widetilde{D}_{l}$ as:

$$ $\widetilde{D}_{l}=D_{l}*M_{l}+s\cdot D_{l}*(1-M_{l}),$ (6) $$

where $s$ is a scaling hyperparameter that controls the filtering strength.

### 4.4 Adapting SVD for Video Semantic Segmentation

page\cref@result
As it is natural to explore applying video models for video tasks, we further investigate the possibility of using Stable Video Diffusion (SVD)[] features to perform VSS.
Unlike SD, which is text-conditioned, SVD is image-conditioned.
SVD adapts the SD 2.1 architecture and finetunes on a highly-curated large video dataset.
Moreover, it employs a video VAE encoder to encode and decode the videos.
SVD extends the SD architecture design mainly in two parts:
1) A video residual network that consists of a set of convolutional blocks that handle every frame of the video latent independently.
2) A temporal attention layer is applied on top of the output of the spatial attention layer, which computes the full attention of the features along the temporal axis on each spatial location.
The output of the temporal attention and the output of the spatial attention are then mixed with a learnable weight.
We use SVD as a backbone model to extract the features as well as do modulation following similar steps described in previous sections.

## 5 Experiments

To the best of our knowledge, no zero-shot diffusion-based Video Semantic Segmentation (VSS) approaches exist.
Therefore, we compare against existing zero-shot image segmentation by adapting them to the VSS setting.

**Table 1: Quantitative performance comparison.\cref@constructprefixpage\cref@result**
| Method | Backbone | Training | VSPW | Cityscapes | Camvid |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU | mVC8 | mVC16 | mIoU | mIoU | mVC8 | mVC16 |  |  |  |
| TMANet[] | ResNet-50 | Supervised | – | – | – | 80.3 | 76.5 | – | – |
| UniVS[] | Swin-T | Supervised | 59.8 | 92.3 | – | – | – | – | – |
| DVIS++[] | VIT-L | Supervised | 63.8 | 95.7 | 95.1 | – | – | – | – |
| CLIPpy | T5 + DINO | zero-shot | 17.7 | 72.4 | 68.4 | 4.7 | 2.6 | 44.4 | 35.6 |
| EmerDiff (SVD) | SVD | zero-shot | 39.7 | 82.1 | 78.5 | 11.0 | 7.3 | 71.7 | 64.4 |
| EmerDiff (SD) | SD 2.1 | zero-shot | 43.4 | 68.9 | 64.3 | 21.5 | 6.9 | 39.8 | 32.9 |
| Ours(SVD) | SVD | zero-shot | 53.2 | 89.3 | 88.0 | 36.2 | 16.6 | 87.4 | 85.8 |
| Ours(SD) | SD 2.1 | zero-shot | 60.6 | 90.7 | 89.6 | 37.3 | 20.6 | 92.3 | 91.9 |

### 5.1 Experimental Setup

Implementation details.
We evaluate our method with both SD 2.1 and SVD backbones, and we denote them as *Ours (SD)* and *Ours (SVD)*.
Given the video frames, we encode them using VAE, and we add a certain level of noise that corresponds to timestep $t_{inv}$ and start the denoising process at this noise level.
As we need to perform modulation at timestep $t_{m}$, the value of $t_{inv}$ must be larger than $t_{m}$.
Initially, we set the modulating coefficient $\lambda$ to 10 as in [].
However, we observed that latent blending suppresses the modulating strength; thus, we increase $\lambda$ to 50 when latent blending is enabled.
We set the batch size $B=14$, which is also the original training batch size of SVD.

We fix all hyperparameters for all videos and datasets, and we do not apply any post-processing methods like a conditional random field (CRF) on the output segmentation maps.
All experiments were conducted on a single NVIDIA A100 40G GPU.
We provide detailed parameter settings for SD and SVD backbones in Appendix.

Evaluation protocol.
The clusters generated by K-Means in the first frame are class-agnostic.
To be able to compare our predicted segmentation maps against the groundtruth, we use the labels from the groundtruth of the first frame to assign labels to our clusters.
This is similar to the evaluation protocol in Video Object Segmentation, but we differ in that we only use the labels from the groundtruth and keep the segmentation masks from K-Means.

Baselines.
We compare against zero-shot image segmentation approaches CLIPpy [] and EmerDiff [].
For EmerDiff, we adapt it to our zero-shot VSS setup.
We use the same label assignment strategy for the first frame, but then we train a KNN classifier to predict the next frame in an autoregressive manner, similar to our approach.
We also provide results of the supervised approaches DVIS++ [], UniVS [], and TMANet [] to showcase where our zero-shot approach stands compared to them.

Dataset.
We evaluate on the validation set of three commonly used VSS datasets: VSPW[] with diverse videos, Cityscapes[] and CamVid [] with driving videos.
VSPW has a resolution of 480p, while CamVid is 360p.
For Cityscapes, to fit the video frames into GPU memory, we generate the segmentation maps at the resolution of $512\times 256$ and upsample them to $2048\times 1024$ for evaluation.
We provide details of the settings for individual datasets in the Appendix.

Metrics.
We report mean Intersection-over-Union (mIoU) and mean Video Consistency (mVC) [] as quantitative metrics similar to existing VSS approaches [, , ].
mIoU describes the mean intersection-over-union between the predicted and ground truth pixels, while mVC computes the mean categories’ consistency over the long-range adjacent frames.
We denote mVC evaluated under 8 and 16 video frames as mVC8 and mVC16, respectively.
We use both metrics together to showcase the segmentation quality on individual images as well as the overall temporal consistency.

Figure: Figure 5: Qualitative comparison of different zero-shot methods. Note that the color of a segmentation cluster only represents the relative index of the clusters when the video is processed. The color itself does not map to an absolute label.\cref@constructprefixpage\cref@result
Refer to caption: extracted/5622794/figures/quali_comp/clippy/287/00001032.jpg

### 5.2 Quantitative results

We provide the quantitative results in \Creftab:quantitative.
Our approach with both SD and SVD backbones performs the best in terms of all evaluation metrics on all datasets amongst the zero-shot approaches.
More specifically, it improves over EmerDiff for both SD and SVD backbones in terms of mIoU with 33%, 29% on the VSPW dataset, 54%, 106% on the CityScapes dataset, and 99% and 78% on the Camvid dataset.
Furthermore, our approach performs similarly to the supervised method UniVS and DVIS++ in terms of mIoU despite not being explicitly trained for this task.
On the CityScapes and Camvid datasets, our approach outperforms the zero-shot methods by a huge margin.
However, there is still a performance gap compared to the supervised approaches.
We attribute this to two main reasons.
First, CityScapes and Camvid datasets are for driving scenarios and have small objects and challenging lighting conditions, which poses a challenge when inverting the video frames (see Appendix).
Secondly, we downsample their video frames of CityScapes by a factor of $16$ to match the resolution of the diffusion models.
This makes it difficult to segment small objects such as pedestrians and traffic poles, contrary to the VSS-specialized approaches that consider small objects when designing their solutions. We leave it for future work to adopt a tiled approach for high-resolution video segmentation.

### 5.3 Qualitative results

page\cref@result
We show some qualitative results of different zero-shot methods in \Creffig:quali_comp.
CLIPpy struggles to locate the boundaries accurately and produce coarse segments.
EmerDiff can segment the first frame accurately but struggles to preserve the masks temporally.
Our method with SD backbone produces the best segmentation maps in terms of segmentation quality and temporal consistency.
The maps have sharper boundaries and clean clusters compared to other methods.
Ours (SVD) performs better than its EmerDiff counterpart.
However, the overall segmentation quality obtained by the SVD backbone is worse than that of SD.
This can be attributed to the degraded feature representation of SVD compared to SD as a result of training on a relatively small video dataset compared to SD.
We provide video results on our project website [https://qianwangx.github.io/VidSeg_diffusion/](https://qianwangx.github.io/VidSeg_diffusion/).

### 5.4 Ablation Analysis

page\cref@result

**Table 2: Ablation study. The videos we use here are the first 30 videos from VSPW validation set. CBR refers to Correspondence-Based refinement.\cref@constructprefixpage\cref@result**
| Batch size $B$ | Masked Modulation | Feature Aggregation | CBR | SD2.1 | SVD |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU | mVC8 | mVC16 | mIoU | mVC8 | mVC16 |  |  |  |  |
| 1 | ✗ | ✗ | ✗ | 33.4 | 70.6 | 60.2 | 26.6 | 82.2 | 78.3 |
| 14 | ✗ | ✗ | ✗ | 43.4 | 76.9 | 73.8 | 38.9 | 90.2 | 88.7 |
| 14 | ✓ | ✗ | ✗ | 45.6 | 87.5 | 85.5 | 38.6 | 90.5 | 89.0 |
| 14 | ✓ | ✓ | ✗ | 46.6 | 87.6 | 85.7 | 39.4 | 90.2 | 89.2 |
| 14 | ✓ | ✗ | ✓ | 47.4 | 89.2 | 87.5 | 37.1 | 91.5 | 90.3 |
| 14 | ✓ | ✓ | ✓ | 46.5 | 89.8 | 88.4 | 38.2 | 92.3 | 91.3 |

We show an ablation of our newly proposed components in \Creftab:ablation.
Incorporating more frames in training our context model greatly improves the mIoU as well as mVC.
Enabling the masked modulation and the feature aggregation improves all metrics further.
The best results in terms of mIoU are attained by enabling the correspondence-based refinement and disabling the feature aggregation.
The feature aggregation can negatively impact the mIoU on some occasions due to the coarse groundtruth of the VSPW dataset.
\Creffig:feature_fusion_comp shows an example where our approach remarkably predicts the fine details of the window as one class and the background as another class, while the groundtruth annotates the whole region as a window.
Finally, the best trade-off between the segmentation quality (mIoU) and temporal consistency (mVC) is achieved with all components.

Feature Aggregation.
To validate the efficacy of feature aggregation, we show a visual comparison of the segmentation maps produced by features in different blocks in \Creffig:feature_fusion_comp.
The aggregated features can encode more spatial details, which could enhance the coarse masks and, consequently, the high-resolution segmentation maps.

Masked Modulation.
We show the qualitative comparison with or without the latent blending in \Creffig:latent_blending_comp.
The figure shows that without the latent blending, the difference map in (c) contains high activations outside of the modulated sub-region indicated by the coarse binary mask.
The existence of activation in these regions can lead to a false assignment of the segmentation labels, as shown in (e).
For example, a part of the lab table is classified as a wall.
After applying latent blending, we remove activations outside of the mask region and obtain a cleaner segmentation mask, as shown in (f).

Figure: Figure 6: High-resolution segmentation map generated by aggregated feature has more details than features from a single block. We omit the low-resolution segmentation map for a better visual comparison.\cref@constructprefixpage\cref@result
Refer to caption: x5.png

Figure: Figure 7: We show that given a low-resolution mask of a sub-region (b), the corresponding difference map without masked modulation (c) can have high activation outside the masked sub-region, which will result in spatial inconsistency on the final segmentation map (e). By applying masked modulation on the intermediate latents, the high activation on the irrelevant regions of the difference map will be removed (d), therefore producing a cleaner segmentation map (f). \cref@constructprefixpage\cref@result
Refer to caption: extracted/5622794/figures/ablate_diff_map/exp_1/input_image.jpg

## 6 Limitations and Future Work

page\cref@result
One of the limitations of our approach is its dependency on the quality of the image inversion method.
Moreover, fine image details are likely to be discarded due to the compression from the VAE encoder.
Therefore, our approach can benefit from future research improving both VAE encoding and image inversion.
It is also beneficial to investigate if other video diffusion models have semantically higher quality features than SVD.
Another limitation of our approach is that it is instance-agnostic, *i.e.*  it groups all objects of the same class into the same cluster.
This is due to the inherent nature of diffusion features that group similar semantics.
For future work, our approach can be extended to perform Video Instance or Panoptic segmentation.

## 7 Conclusions

In this work, we introduced the first zero-shot method for Video Semantic Segmentation (VSS) using pre-trained diffusion models.
We proposed a pipeline tailored for VSS that leverages image and video diffusion features, and attempts to enhance their temporal consistency.
Experiments showed that our proposed approach significantly outperforms zero-shot image semantic segmentation methods on several VSS benchmarks, and performs comparably well to supervised methods on VSPW dataset.

## Appendix A supplemental material

### A.1 Codebase and website

Our anonymized code is available at [https://anonymous.4open.science/r/VidSeg_Anonymous-827E](https://anonymous.4open.science/r/VidSeg_Anonymous-827E).
We also provide video results on the anonymous project page [https://anonymous.4open.science/w/VidSeg_Anonymous-E341-website/](https://anonymous.4open.science/w/VidSeg_Anonymous-E341-website/).

### A.2 Broader impact

Video semantic segmentation has important applications in autonomous driving and surveillance security.
However, the source videos used for segmentation may contain private information such as human faces and driving plates. Guidelines for responsible usage need to be made to prevent privacy invasion.
Also, diffusion models can inherit and propagate biases from their training data, leading to unfair treatment of certain groups.

### A.3 Settings

#### Dataset.

VSPW (Video Scene Parsing in the Wild) is a large-scale video semantic segmentation dataset that consists of a wide range of real-world scenarios and categories.
It has 124 categories in total.
The resolution of this dataset is $480\times 853$.
Since each video consists of less than 10 classes, we set the number of clusters for KMeans as 20.
Cityscapes is a large-scale urban streets video sequence dataset.
The objects are grouped into 30 classes in total.
Each video clip has 30 frames, and only the 20th frame has dense annotations.
We use its validation set, which contains 15000 frames from three cities.
As the original resolution of the frames is $1024\times 2048$, which is too big to fit into the GPU memory with SVD, we downsample the frames to $256\times 512$ for all the experiments and evaluate at the original resolution by upsampling the segmentation maps.
As each video clip may contain classes of more than 10, we set the number of clusters as 30 in order to capture the small objects.
CamVid (Cambridge-driving Labeled Video Database) is a road scene dataset with dense segmentation annotations with 11 classes in total.
The resolution of this dataset is $360\times 480$.
We use its validation set, which has one video clip and contains 100 frames.
We set the number of clusters for KMeans as 20.

#### Implementation details.

We build our method on top of SD 2.1 and SVD code repository [https://github.com/Stability-AI/generative-models](https://github.com/Stability-AI/generative-models).

#### Computational resources.

We use one NVIDIA A100 GPU to conduct all our experiments.
As the running time depends on the number of clusters and spatial resolution of the video frames, generally, it will take around 2 minutes to process a batch of video frames using SD 2.1 and 5 minutes using SVD.
The main cost of the time comes from the modulating process, which involves several modified forward passes of the backbone model.

### A.4 Hyperparameters

We provide all hyperparameters for SD 2.1 and SVD in \Creftab:hyperparam.

**Table 3: Hyperparameters settings for SD and SVD.\cref@constructprefixpage\cref@result**
|  | Ours (SD 2.1) | Ours (SVD) |
| --- | --- | --- |
| $t_{f}$ | 25 | 25 |
| $t_{m}$ | 20 | 17 |
| Sampling timesteps | 25 | 25 |
| Sampler | EDM | EDM |
| Block $b_{k}$ | Block 6, 7 and 8 | Block 6, 7 and 8 |
| Block $b_{m}$ | Block 7 | Block 8 |
| Block $c$ | Block 7 | Block 8 |
| Spatial threshold $\mathcal{T}$ | 1 | 1 |
| Filtering strength $s$ | 0.7 | 0.7 |
| Modulating factor $\lambda$ | 50.0 | 50.0 |
| Modulating attention type | cross attention | self attention |
| Injected features | spatial attention | spatial & temporal attention |

### A.5 Failure cases

We identify several typical failure cases in our method.
The first one is the inaccurate diffusion inversion process (\Creffig:failure_inversion).
In adjacent frames, the texture and shape details of the small objects can be different after inversion.
Segmentation maps will also inherit this discrepancy between input frames and reconstructed frames, which will result in flickering on the frames.
The second one is flickering between similar regions (\Creffig:failure_temporal_inconsist).
Some objects are originally assigned with one cluster, while in the later frames, they are grouped into different clusters.
The third one is dealing with unseen objects (Fig. 10).
As we only use an anchor frame’s ground truth to guide the class-agnostic clusterings, we cannot handle well the unseen objects in the anchor frame.

Figure: Figure 8: Failure case: inversion. In adjacent frames, the car and the sign of the car experience shape changes, which finally result in obvious shape changes on the segmentation maps. We highlight the main areas of discrepancy in green and blue circles.\cref@constructprefixpage\cref@result
Refer to caption: x6.png

Figure: Figure 9: Failure case: temporal inconsistency. The sidewalk is clustered into the yellow region in the first frame, and later the same sidewalk is clustered into another region denoted as purple region.\cref@constructprefixpage\cref@result
Refer to caption: x7.png

Figure: Figure 10: Failure case: unseen objects. The door and albums, which do not appear in the previous frames, are assigned with the wrong clusters.\cref@constructprefixpage\cref@result
Refer to caption: x8.png

### A.6 Ablation study

We provide additional ablation experiments here.
We ablate the modulating Block $b_{m}$ in \Crefablation:modulate_block_idx for both SD and SVD. Modulating Block 7 and Block 8 for SD and SVD, respectively, can give the most spatial details as well as maintain the semantics.
These blocks are, at the same time, the most semantically-riched blocks in SD and SVD.
In \Crefablation:latent_blending, we show more examples of how latent blending and difference map filtering help with removing spatial noises.
In \Crefablation:sd_svd_diff, we show comparisons between the difference maps and segmentation maps produced by SD and SVD.
We show that the difference maps of SD contain finer details, which bring sharper boundaries to the final segmentation maps.
We further show PCA visualization of a $64\times 64$ resolution block to complement \Creffig:svd_feat in the main paper.
We show that in high-resolution blocks, SD features have more semantic information and spatial details, as well as more temporally stable than SVD spatial features and temporal features.
We also show that segmentation maps under different numbers of K-Means clusters and after GT labels reassignment in \Crefablation:kmeans_num_clusters.
Increasing the number of clusters can help segment more small objects (from 5 to 20).
However, further increasing the number of clusters may not necessarily segment out the clusters that are aligned with defined clusters in ground truth (from 20 to 30).
Although K-Means originally generated 30 clusters, most of them are merged to the same classes after GT label reassignment.

Figure: Figure 11: Ablation: modulating different block will result in different segmentation maps. Modulating Block 7 and Block 8 give the best results for SD and SVD, respectively (highlighted in shadow).\cref@constructprefixpage\cref@result
Refer to caption: x9.png

Figure: Figure 12: Ablation: latent blending and difference map filtering.\cref@constructprefixpage\cref@result
Refer to caption: x10.png

Figure: Figure 13: Ablation: Difference maps comparison between SD and SVD.\cref@constructprefixpage\cref@result
Refer to caption: x11.png

Figure: Figure 14: Ablation: number of K-Means clusters. In the first row, we show the segmentation maps generated by different numbers of clusters in K-Means. In the second row, we show the segmentation maps generated by the same numbers of clusters followed by GT labels reassignment.\cref@constructprefixpage\cref@result
Refer to caption: x12.png

Figure: Figure 15: A visualization of the first three PCA components for the features extracted from the most semantically-rich blocks of Block 9 in both SD and SVD of the first and last video frames in a batch. In the second row, we show the $x$-$t$ slice of a set of pixels (highlighted in the red line in the leftmost PCA visualization) horizontally across the PCA visualization ($x$-axis) and stack it chronologically across the full batch of video frames ($t$-axis). \cref@constructprefixpage\cref@result
Refer to caption: x13.png

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: Questions:
1. In l. 107 what evidence is there that training dataset diversity and size will affect the trend that SD features leads to higher quality results than SVD features?
2. Is L in l. 177 set to the number of classes in the dataset? If so how does the model deal with segmenting separate instances of the same class in the same mask?
3. The exposition of the scene context model would benefit greatly from a set of equations showing clearly what the input and output of $\Omega$ is.
4. Why was cosine similarity chosen in l. 267 and not L1 or L2?
5. Can you give an intuition on the two cases in parametric Equation 4?
5. What is $B$ in Equation 5? If it's the number of batches why does the sum exclude the last batch?
6. In l. 429-430 why doesn't the masked modulated upsampling address this?

Minor typos/Writing Suggestions:
- Define what modulation is in the beginning of the paper.
- l. 166 "2024), We"-> "2024), we"
- l. 167 "where We"-> "where we"
- l. 267 "unplausible"->"implausible"
- Figure 4 caption would benefit from a more in depth description of the scene context model.
- l. 428 "their"->"the"
- l. 458 "produce"->"produces"
soundness: 2
strengths: 1. The method's contribution in terms of technical novelty is strong: the masked modulation strategy, correspondence algorithm and scene context model are novel.
2. The method is well-presented so that the method's components are clear. For the most part the paper is clearly written.
summary: This paper introduces a zero-shot video semantic segmentation (VSS) method based on diffusion models. The method consists of 3 main components: 1. a scene context model that models the classes in a video based on diffusion model decoder features 2. a correspondence algorithm that produces a coarse segmentation map for each frame and 3. an upsampling strategy that applies masking to modulated segmentation masks. The method is implemented using either Stable Diffusion and Stable Video Diffusion as backbones. Experiments are conducted to compare the method's performance to the baselines on 3 VSS datasets. Ablations are carried out to asses the contribution of the 3 components above on a subset of a VSS dataset.
weaknesses: 1. The label setting and motivation are not clear. In Section 5.1, Evaluation Protocol, it says that the labels from the ground truth of the first frame are used to assign labels to the method's clusters (l. 359-360). However, the paper claims that the method is zero-shot [1], leading to confusion on the task the method is designed for. If the method is using the first frame of the video to assign classes then it should be categorized as Video Object Segmentation [2] and compared to the baselines for this task, rather than Video Semantic Segmentation.
2. The method is not well-validated and deserves further analysis. In Table 2, it is not clear based on the difference in mIOU and mVC whether all components are necessary to achieve the best performance. The analysis does not delve into the contributions of different components (e.g. when the feature aggregation can fail: in l. 472 it says "some occasions" but what occasions?) Other aspects of the model were not validated, such as choosing the features from blocks 6-8. 
3. The computational efficiency (training time, inference time and memory use on the GPU) of the non-diffusion-based baseline methods are missing. This is important in assessing the ease-of-use of the method and contributes to one's decision to apply the method.

[1] "Zero-Shot Semantic Segmentation". Maxime Bucher, Tuan-Hung Vu, Matthieu Cord and Patrick Pérez. NeurIPS, 2019.
[2] "The 2017 DAVIS Challenge on Video Object Segmentation". Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Luc Van Gool. Arxiv:1704.00675, 2018.

---

confidence: 4
contribution: 4
initial_rating: 5
presentation: 2
questions: Questions were noted in the above weaknesses.

- Also it is not clear how the context model operates, how KNN is trained exactly? It is a nonparametric technique so it is not clear how in their figure they refer to it as undergoing some fine-tuning or adaptation when there are no parameters. An explanation of how this is done would be more suitable for that.

- In their evaluation protocol they do use the labels in the first frame to assign labels, so is this really considered zero shot? Are these labels used during the correspondence refinement. Or this is solely some form of Hungarian matching during evaluation only? Why not just do this Hungarian matching per image why only first frame? I am wondering if something is wrong in the evaluation protocol that lead to the inconsistency above between VSPW vs. CamVid.
soundness: 2
strengths: - The novelty of their contribution as the first to mine diffusion models for video semantic segmentation
- Consistent strong gains on three acknowledged benchmarks Cityscapes, CamVid and VSPW.
summary: Authors propose a technique for unsupervised video semantic segmentation that is relying on diffusion models. They propose a context model that relies on a KNN classifier and a correspondence-based refinement module that that ensures the temporal consistency of the labels. The method is evaluated on VSPW benchmark and compared to single image models that mine diffusion models for segmentation.
weaknesses: - There is an inconsistent behaviour in the results that needs to be explained. Their results show better performance on VSPW than Cityscapes, when it is widely acknowledged that VSPW is much harder dataset with more categories. For example refer to OCRNet[1] results on VSPW at 36.68% while on Cityscapes at 81.8%. While these are fully supervised models it is still not explained why unsupervised ones will act differently? It seems something is wrong in the setup. Even if the claim this has to do with the dataset resolution, that wouldn't be the case on CamVid and still why fully supervised methods face more challenges in VSPW than Camvid/Cityscapes.

- Is there a way to setup their baseline for EmerDiff to just be evaluated per image? is there a reason why they are only reporting the adaptation to the video segmentation with their proposed context model. Currently, it seems their baseline is still dependant on their own adaptation so a simple comparison to image based baselines that rely on diffusion models will serve the purpose of confirming the benefit from their video based approach.

- It would be interesting to see its performance w.r.t DiffSeg[2] another image based segmentation mining diffusion models with focus on attention maps.
 

- While not sure but seems their ablation is conducted on part of the VSPW val set, why 30 videos only? os is this the full number of videos in the val set?

[1] Yuan, Yuhui, et al. "Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation. 2021." 1909.

[2] Frick, Raphael Antonius, and Martin Steinebach. "DiffSeg: Towards Detecting Diffusion-Based Inpainting Attacks Using Multi-Feature Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to "Weaknesses" section for further details.
soundness: 2
strengths: 1. The proposed paper is well-written and easy-to-follow. The figure demonstration is very clear.
2. The problem of interest, i.e. (zero-/few- shot) Video Semantic Segmentation is practically meaningful. I agree the authors that VSS is less explored compared to its image (single-frame) counterpart.
3. The proposed paper provides a feasible solution to the proposed problem, given the experimental results shown in Section 5 in the main paper, though certain important aspects are omitted (please refer to the next section for further details).
summary: The proposed paper focus on the problem of zero-shot Video Semantic Segmentation (VSS) based on Diffusion models, Stable Diffusion (SD) and Stable Video Diffusion (SVD) models. Specifically, the authors introduce a zero-shot VSS approach leveraging spatial features from pretrained diffusion models (SD or SVD) learned from large-scale image/video data and a scene context model that adapts segmentation maps over video frames. 
This approach includes a correspondence-based refinement (CBR) strategy, based on the semantic correspondence between consecutive frames to enforce better temporal consistency and a masked modulation process to upsample the coarse segmentation masks to the full resolution of the video frames for improved segmentation accuracy. The authors perform experiments on three commonly used VSS datasets: VSPW, Cityscapes and CamVid, to evaluate their proposed zero-shot VSS pipeline.
weaknesses: 1. Technical contributions. The proposed paper seems to heavily rely on existing works, i.e. Emerdiff [Namekata et al., 2024] to extract per-frame semantic features and generate coarse segmentation maps, with additional correspondence-based refinement (CBR) and mask modulation to refine and upsample the coarse segmentation maps. On the other hand, the proposed segmentation map refinement strategy, CBR which employ pixel-wise correspondence has been widely studied in previous work, especially video segmentation related vision tasks, e.g. Tang et al. (2023) as the authors pointed out in the paper and [a-c] etc. While the mask modulation operation is essentially a weighted summation operation to mask out latents and difference map, the overall technical contributions c.f. video semantic segmentation seems incremental and lacking significance.
2. Sufficiency of Experiments. While the focus of the proposed work is zero-shot VSS, several aspects are missing to convincingly support authors' claims. a) It is important to show the performance gain of the proposed pipeline versus naive frame feature aggregation to prove the effectiveness of the proposed CBR and mask modulation. I.e., what is the performance like with SD spatial features + average/max pooling of the features for the final video-level segmentation? Does row 1 and 2 in Table 2 refer to a similar case? If so, it would be better to be clarified in the paper. b) While "zero-shot" being one of the major contributions claimed in the paper, it would be important to compare the proposed pipeline with other foundation models like SAM/SAM2. One additional minor point is that, the proposed pipeline requires GT labels to assign class labels to the segmentation map for the first frame, whether the proposed pipeline is fully "zero-shot" may be argueable.
3. Computation Efficiency. The proposed pipeline for processing videos requires to process per-frame features, while also computing pixel-level correspondence (in the proposed CBR module), which is quite computationally intensive. This has also validated by the authors in the supplementary material. There has been works explored object-level correspondence for more efficient computation, e.g. [b], I wonder if the authors can provide more insights from this aspect.
4. Minor: many references are missing the conference information.

[a] Cheng et al., Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation, NeurIPS 2021.
[b] Cheng et al., Putting the Object Back into Video Object Segmentation, CVPR 2024.
[c] Sun et al., Alignment before aggregation: trajectory memory retrieval network for video object segmentation, ICCV 2023.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: It is hard to understand why SD features would be more stable than SVD in both spatial and temporal dimensions (as depicted in Appendix A.10). From my perspective, Table 1 shows that EmerDiff (SVD) performs better than EmerDiff (SD) in mVC, suggesting that the feature in SVD should have better temporal consistency. However, SVD underperforms compared to SD in other results, so I agree that, compared to SD, SVD may have a more degraded feature representation (as depicted in Sec 5.3). Meanwhile, can you provide more convincing evidence for this?
soundness: 2
strengths: S1: This paper is well written and easy to follow.

S2: This paper explores the characteristics of SD and SVD on video tasks and proposes a novel zero-shot video semantic segmentation method based on Emerdiff. The proposed CBR strategy can effectively enhance the temporal and spatial consistency of coarse segmentation maps.
summary: This paper introduces a zero-shot video semantic segmentation approach using pre-trained diffusion models. The proposed methods, which involve a CBR strategy, scene context model, and others, enhance spatial and temporal consistency. Experiments demonstrate its good performance on VSS benchmarks.
weaknesses: W1: The comparisons in experiments are weak and can’t show the superiority of these methods. The baselines that are surpassed are all focused on image segmentation, so it does not effectively demonstrate the method’s strengths. Additionally, although the paper compares its approach with other supervised video segmentation methods, it does not achieve state-of-the-art performance. Furthermore, there are no comparisons with other zero-shot video segmentation methods. Therefore, I suggest adding baselines of other zero-shot video segmentation methods to highlight the advantages of the proposed approach.

W2: I understand one of the main innovations of this paper lies in the CBR strategy, but the ablation study does not provide quantitative results for the CBR ablation and doesn’t have enough analysis for it.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The inputs and outputs of Latent Blending in the Fig.3 are not very easy to understand. Could you pls explain more?


2. What is the distance metric used in the pixel-level voting mechanism during the cross-frame matching process?
soundness: 3
strengths: 1. Using diffusion to model the temporal dimension is a very interesting idea. Predicting a coarse segmentation map for each frame and adapting to scene changes in the video through autoregressive updates is highly intuitive.

2. The writing is good and the proposed method is easy to follow.

3，The experimental results are solid.
summary: To address the challenge of existing methods lacking temporal modeling in video data, this paper uses a diffusion model with autoregressive updates to adapt to scene changes, handling the issue of temporal continuity. Its effectiveness has been validated on datasets such as VSPW.
weaknesses: 1. The goal of VSS (Video Semantic Segmentation) is to predict the semantic category for each pixel in each frame based on predefined classes. EmerDiff has shown good performance in image segmentation tasks. Would its approach, with the addition of a certain level of temporal modeling, serve as a more suitable baseline or comparison method to validate the effectiveness of the proposed approach?

2. VSS is a highly complex task, and balancing the effectiveness of capturing temporal and spatial information with computational efficiency is key. An important concern is whether the proposed method might increase the computational burden, leading to excessively long training times.

3. Lack comparison and discussion with recent important work on video scene segmentation (VSS) task, General and Task-Oriented Video Segmentation, in ECCV 2024.

4. How was the KNN classifier chosen, and is it the best clustering model?