End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning
Authors: Xin Yang
YAN WENDING
Michael Bi Mi
Yuan Yuan
Robby T. Tan
Venue: NeurIPS 2024 poster (2024)
Keywords: optical-flow free
video semantic segmentation
adverse conditions
Primary area: machine_vision

Abstract: Adverse weather conditions can significantly degrade the video frames, causing existing video semantic segmentation methods to produce erroneous predictions. In this work, we target adverse weather conditions and introduce an end-to-end domain adaptation strategy that leverages a fusion block, temporal-spatial teacher-student learning, and a temporal weather degradation augmentation approach. The fusion block integrates temporal information from adjacent frames at the feature level, trained end-to-end, eliminating the need for pretrained optical flow, distinguishing our method from existing approaches. Our teacher-student approach involves two teachers: one focuses on exploring temporal information from adjacent frames, and the other harnesses spatial information from the current frame. Finally, we apply temporal weather degradation augmentation to consecutive frames to more accurately represent adverse weather degradations. Our method achieves a performance of 25.4 and 33.0 mIoU on the adaptation from VIPER and Synthia to MVSS, respectively, representing an improvement of 4.3 and 5.8 mIoU over the existing state-of-the-art method.

End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning
Xin Yangb
Yan Wending
Yuan Yuan² National University of Singapore
Michael Bi MI²
Robby T. Tan'
Huawei International Pte Ltd e06746120u, nus. edu, yan. wend inghhawwe... com, mi cicchblling bathaelbim@@yaho..ccommm yuanyuani 0thuawei com, robby. tan@nus. edu. sg
Abstract
Adverse weather conditions can significantly degrade video frames, leading to erro- neous predictions by current video semantic segmentation methods. Furthermore, these methods rely on accurate optical flows, which become unreliable under ad- verse weather. To address this issue, we introduce the novelty of our approach: the domain-adape -ndativve video semantic segmentation
first end-to-end, aptical-flow-ff method. This is accomplished by enforcing the model to actively exploit the tempo- ral information from adjacent frames through a fusion block and temporal-spatial teachers. The key idea of our fusion block is to offer the model way merge information from consecutive frames by matching and merging relevant pixels from those frames. The basic idea of our temporal-spatial teachers involves two teachers: one dedicated to exploring temporal information from adjacent frames, the other harnesses spatial information from the current frame and assists the temporal teacher. Finally, we apply temporal weather degradation augmentation consecutive frames Our method achieves a performance of 25.44 5..4% and 33.0% mloU on the adaptation from VIPER (28) and Synthia [29] to MVSS (18), respectively, representing an improvement of 4.3% and 5.8% mloU over the existing state-of-the-at method.
accurately represent adverse weather degradations.
Introduction
Unsupervised domain adaptation (UDA) is gaining attention in video semantic segmentation, offering solution to the challenge of annotations by adapting models from labeled synthetic datasets to unlabeled real-world scenarios. However, existing video-based UDA methods often falter under the assumption of ideal conditions, neglecting the drastic impact of adverse weather conditions like nighttime and fog. These weather conditions can lead to significant degradation in video quality and result inaccurate predictions.
The existing UDA methods often rely on two components: pretrained optical flow and pseudo- labels, where the optical flow is used warp adjacent frames, and the pseudo-labels are used for unsupervised training on the target domain [10 30 36 24 99. However, when it comes to adverse weather conditions, the reliability of these components diminishes for two main reasons. Firstly, adverse weather conditions introduce significant degradation in low-level features, including issues such as noise and glare effects during nighttime, as well as occlusions in rainy and foggy conditions. Since existing methods are not inherently designed to handle such low-level degradations, they can
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

(a) Image
(h) TPS 36
Ours
(d) Image
(c) [6]
(f) Ours
Figure I: Our model demonstrates enhanced robustness compared to TPS (36 in semantic segmenta-
tion tasks under foggy and snowy conditions. It notably excels by significantly reducing inaccuracies in the segmented areas.
easily be misled by these adverse effects, leading to inaccurate predictions 200 19
Secondly, as highlighted in 2 37, adverse weather conditions have distinct styles, which magnify the domain gaps between synthetic datasets and real-world datasets in adverse weather scenarios.
To address these challenges, the novelty of our method lies in introducing the first end-to-end, aptical-flow-free video domain adaptation strategy, tailored for conditions. Unlike the existing methods, we avoid relying on potentially erroncous optical flows from pretrained models. Instead, we design fusion block that merges the feature-level information from adjacent frames. We simultaneously train the segmentation model and the fusion block, guided by segmentation losses. Hence, our fusion block learns combine temporal information which can benefit the semantic segmentation task, from different frames.
videos in adverse weather
We have developed a temporal-spatial teacher-student learning approach effectively train the fusion block and enhance the quality of pseudo-labels. This approach encompasses two teachers, a temporal teacher and spatial teacher, who collaboratively instruct student model. Temporally, the teacher network receives consecutive frames, including the current frame and its adjacent frames. We use the predictions of the current frame as our pseudo-labels. The student network also receives the same adjacent frames, but for the current frame, we provide it with a cropped segment. Then, we enforce consistency between the student network's prediction and the pseudo-label. This compels the student network to actively incorporate temporal information from adjacent frames, enabling it to perform outpainting on the cropped segment and produce the same prediction as the pseudo-label. Spatially, the teacher network benefits from high-resolution version of the cropped segment to create the pseudo-label, a proven method for enhancing pseudo-label quality, as suggested
in [13] To the best of our knowledge, integrating temporal and spatial modeling using two teachers and one student to achieve an optical-flow-free model is novel. Additionally, the fusion block and its integration into our temporal teacher-student framework have not been explored before.
Augmentation plays crucial role in enhancing the effectiivneess of UDA methods 37 20] In the context of adverse weather conditions, certain weather-specin degradations exhibit temporal patterns. For instance, areas with low light in one frame during nighttime are likely to persist in adjacent frames, albeit with potentially varying intensity due to vehicle movement. Similarly, the presence of fog and the accumulation of rain effects also span consecutive frames, with intensity changes influenced by shifts in depth [31 37 To effectively capture these characteristies of adverse weather conditions, we introduce temporal weather degradation augmentation strategy. This strategy involves applying correlated augmentations to either the same closely positioned locations in consecutive frames, with each undergoing gradual changes in intensity.
Fig. I compares our method with TPS [36, illustrating our method's 's enhanced robustness in adverse weather conditions, achieved independently of pretrained optical flow. In summary, our contributions are as follows:
We present an end-to-end, optical-flow-fie domain adaptation strategy. by incorporating a fusion block that merges feature-level temporal information. This enables us to bypass the reliance on potentially erroneous optical flows from pretrained models under adverse weather conditions. To the best of our knowledge, this is the first strategy of its kind.
We introduce a temporal-spe spatial reacher-student learning method, wherein a temporal teacher guides the student model in gathering information from adjacent frames, and spatial teacher concentrates on the current frame. These teachers train the fusion block to actively explore the temporal information while effectively harnessing spatial information.
We develop a temporal augmentation strategy that applying weather degradation augmen- tations to corresponding closely positioned locations across consecutive frames. This

---

approach, featuring gradual intensity variations, effectively captures the dynamic nature of adverse weather degradations.
Our method achieves a performance of 25.4% and 33.0% mloU on the adaptation from VIPER (28) and Synthia [29] to MVSS [18], respectively, representing an improvement of 4.3% and 5.8% mloU over the existing late-of-theeart method.
Related work
Video semantic segmentation Video semantic segmentation aims to label each pixel in video frames while maintaining temporal consistency. Unlike image segmentation, it must address the challenges of temporal coherence and efficiency across sequences. For instance, methods like those in 17 35 B2 [22] capture temporal information from consecutive frames by leveraging supervision from existing labels.
Domain adaptive video semantic segmentation
UDA techniques are extensively applied in 13
various computer vision tasks 33
39]. Techniques such as 31 and pscudo-label-based self-learning 20 are commonly employed in these methods. The primary function of these approaches is to adapt models from labeled source domain (for instance, under clear weather conditions) an unlabeled target domain (like adverse weather conditions). These techniques enable the model to perform impressively in the target domain, despite the absence of ground truth labels.
31
23
12
37
20
adversarial training involving domain discriminators [33
approach 8
Recent studies have sought expand UDA methods from image-based to video- based tasks means to circumvent the labor-intensive and costly process of labeling videos 10 30
36
9]] 24).
These works typically utilize synthetic datasets like VIPER 28 and Synthia (29] their source domains, where semantic segmentation ground truths are automatically generated due to their synthetic nature. As for the target dataset, they use real-world urban scene dataset, Cityscapes-Seq 77. These studies successfully develop models capable of making predictions on both synthetic and real-world datasets, thus eliminating the need for manual labeling of the real-world data.
Among these methods, DA-VSK [00] employs a temporal domain discriminative loss to minimize the differences between source and target domains and improve the accuracy of less confident target predictions. VAT-VST 30 introduces a two-stage UDA method, initially utilizing sequence domain discriminator bridge domain gaps, followed by second stage that employs a pseudo-label-bat predictions from several preceding frames to create pseudo-labels for the current frame. TPS (36) presents a cross-frame augmentation and pseudo-labeling technique, where predictions from adjacent frames serve as pseudo-labels for the current frame. SFC (9] develops Segmentation-to-Fiin Module (SFM) to involve optical flow in the training of the semantic segmentation model. Random augmentation applied the current frame are then reconciled with these pseudo-labels through a consistency loss, training the model to become robust to these augmentation. It's important to highlight that all these methods depend on pretrained optical flow estimations: DA-VSN uses it for intra-domate consistency loss, VAT-VST for aggregating predictions, TPS for warping pseudo-labels, and SFC for additional supervision.
intra-domain consistency loss to
self-learning approach. This approach aggregates
Adverse weather degradation Current video-based UDA methods are mainly developed for adapting models from synthetic to real-world datasets under ideal conditions. However, they fall short in adverse weather conditions. This limitation is largely due to two key factors in the domain gap between synthetic and weal-wrrdd scenes under such conditions: style-related differences, and significant low-level degradations 25
21
19 5)
The style-related gap refers the stylistic disparities between synthetic and real-world datasets, which UDA typically addresses by training models to recognize both styles. In scenarios like Cityscapes-Sey with ideal conditions, low-level degradations are minimal, allowing existing methods to primarily tackle the style-related gap. But in adverse weather, as [99 discusses, these degradations can severely distort features. For example, a car might be obscured by glare from headlights, leading to erroneous feature extraction. Such challenges render pseudo-labele and pretrained optical flows unreliable. Therefore, our research focuses on overcoming these obstacles by proposing a video-based UDA method specifically designed for adverse weather conditions.
3

---

Temporal Augment
Fasion Black
Tantaa TAIRE
Student
Currest Frame
Temporal Augment
Teacher
Tusion
Groand Trudf
Supervised Low-
Tomporal
Fusion Block
(b) Source Pipeline
Student
Offect
version
Previous Fasture
Caro
Spatial Loss
Teacher
(a) Target Pipeline
(c) Fusion Block
Our network comprises two pipelines: the source and the target. (a) Target Pipeline: The upper teacher (temporal) takes both the current and adjacent frames to create temporal pseudo-labels. The student, on the other hand, receives a cropped segment of the current frame and a complete adjacent frame, with loss function enforcing its predictions align with the temporal teacher. The lower teacher (spatial) uses the same segment as the student, but from the original image and at a higher resolution. Similarly, consistency loss is applied make the student's predictions consistent with the spatial teacher's pseudo-labels (b) Source Pipeline: The student model undergoes supervised learning with consecutive frames inputs. (c) Fusion Block: This component integrates multiple offset layers, which adjust pixels from adjacent frames relative to the current frame, and convolutional layers merge these pixels.
Figure 2:
Proposed method
Our proposed method is designed train video semantic segmentation model capable of handling adverse weather conditions using an UDA approach. Distinguishing itself from existing methods, ours operates efficiently without the requirement of optical flows. In the source pipeline, we leverage synthetic datasets and their corresponding ground truths for supervised training. This pipeline takes two inputs: the current frame and an adjacent frame. Upon applying temporal weather degradation augmentation to both the current and adjacent frames, they are then input into our network. Subsequently, the network processes the two inputs individually, producing separate of feature maps for each frame. These feature maps are then fused by the fusion block, resulting in the final prediction for the current frame. A supervised loss is computed based on the prediction and the ground truth for the current frame.
H
W
Coup
Yije log(pize),
N
where, H and W are the height and width of the image, respectively. C is the number of classes. Vije from the ground truths indicating whether the class label e is the correct classification for the pixel at position j). Pije is the predicted probability of the pixel at position (i.j) belonging class N is the total number pixels considered in the calculation.
As for the target pipeline, we use real-world video frames captured from adverse weather conditions, without ground truths. Within this pipeline, implement temporal-spatial teacher-student system involving two teacher models: a temporal teacher and a spatial teacher, and a student model. The temporal teacher processes two complete frames and uses its predictions as temporal pseudo-labels. For the student model, we employ the complete adjacent frame and generate a cropped segment from the current frame. Similarity augmentation to both the cropped segment of the current frame and the complete adjacent frame. These augmented frames are then fed into student model. temporal consistency loss ensures that the student's 's predictions, derived from various augmentations, align with the pseudo-labels provided by the temporal teacher. Therefore, using the cropped segment the current frame compels the student to actively extract temporal information from the adjacent frame. Meanwhile, the temporal
the source pipeline, we apply temporal weather degradation

---

An illustration of optical flows generated using a pretrained FlowNet2 model [27] The optical flows are generated by utilizing information from the corresponding frame and its previous frame. The left two columns display frames and optical flows under ideal conditions, while the right two columns depict frames and optical flows under adverse weather conditions, with nighttime as an illustrative example. Under ideal conditions, the optical flows accurately capture vehicle details, traffic signs, and poles. In contrast, optical flows under nighttime conditions exhibit significant failures, with missed detection of the middle poles, and erroneous predictions for the bus.
Figure 3:
weather degradation augmentation equip the student model to handle real-world conditions where weather-specific degradations often extend across consecutive frames.
Furthermore, the spatial teacher is provided with a high-resolution version of the cropped segment, without any augmentation. The predictions made by this teacher before the fusion block act as spatial pseudo-labels. A spatial consistency loss is then applied, comparing the student model's predictions before the fusion block with the spatial pseudo-labels. This process is designed to direct the student model effectively harness spatial information from the current frame.
3.1 End-to-end training with fusion block
In video-based UDA, where ground truths are unavailable for the target pipeline, existing methods rely on pseudo-labels. A common practice in these methods involves using predictions from the previous frame as pseudo-labels (30 10, 36 24 9). These pseudo-labels are then warped onto the current frames based optical flows generated from pretrained models, providing pseudo-labels for the current frame. Subsequently, various techniques are employed to leverage these pseudo-labels for
unsupervised training on the current frame [10 30 36 9].
While this approach has shown promise adapting from synthetic to real-world domains, it faces challenges in adverse weather conditions. Severe weather conditions can significantly distort visual appearance, leading to incorrect pseudo-label generation. The optical flow models, originally pretrained for ideal weather, also suffers from the substantial domain gap between ideal and adverse weather conditions
40 6
An example in Fig. highlights this difference. In the example, we utilize the same pretrained optical flow model (FlowNet2) [27] used in existing works. Optical flow predictions under ideal conditions precisely depict object details in images, such as vehicles, poles, and traffic signs, enabling accurate warping of pseudo-labels from adjacent frames to the current frame. Conversely, optical flow predictions under adverse weather conditions, such nighttime, exhibit the model's 's inability to identify the movements of distant cars and middle poles, as well as imprecise tracking of the bus.
To overcome this challenge, we propose an end-to-end approach that eliminates the reliance on pretrained optical flow. This training approach comprises a fusion block and temporal teacher model. The fusion block is specifically designed to merge feature-level information from both the current frame and its adjacent frames, thereby incorporating temporal information for refining predictions on the current frame.
5

---

Fusion block The fusion block provides the model an alternative to merge the information from consecutive frames. This is achieved by matching the relevant pixels from adjacent frames, then fusing the matched information. We use deformable convolutional layers as offset layers for matching pixels. We first obtain features from the current frame, denoted as Feur. The offset layers then map information from adjacent frames to the current frame, resulting in adj- Subsequently, both features are concatenated and fused with a convolutional layer to form a new Feur- This process is repeated several times. Offset and fuse layers are trained end-to-end with segmentation losses, enabling the fusion block to merge beneficial information from adjacent frames for semantic segmentation.
3.2 Temporal-Spatial Teacher-Student learning
The teacher-student learning paradigm has been increasingly utilized in image-based UDA g 16 37
13 20
In this approach, the teacher and student models share identical architectures. The teacher model's parameters are updated using the Exponential Moving Average (EMA) of the student model's weights, while the student model is refined through backpropagation with custom loss functions.
Within our proposed methodology, we introduce a dual-teacher system to collaboratively steer the student model. This system comprises temporal teacher, tasked with enhancing the model's s ability to harness temporal information across consecutive frames, and a spatial teacher, focused on extracting and utilizing spatial details from the current frame. It is noteworthy that the temporal teacher's architecture mirrors that of the student model, while the spatial teacher differs by excluding the fusion block. This architectural distinction is clearly illustrated in Fig,
In the temporal dimension, our model is designed to self- elff-suficiently extract temporal information from consecutive frames, diverging from traditional methods that rely on pre-trained optical flows for information warping. This is achieved by presenting the student model with a randomly cropped rectangular segment comprising 25% of the current frame alongside its fully intact neighboring frames. The locations of the rectangle is selected randomly in each iteration of the training process. Consequently, throughout the entire training process, the model encounters different scenarios where the locations and content of the cropped regions vary. In contrast, the temporal teacher processes the entire current frame. The fusion block then combines the feature maps from two complete frames, utilizing them to generate pscudo-labels. These pscudo-labels further guide the student in compensating for the missing information in the cropped frame segment, following a temporal loss:
H
W
Lremp
ΣΣΣ 1=1
Mije
temp log(pije),
N
where, temp is derived from the pseudo-labels. Since the student model must derive the missing information solely from its adjacent frame, the fusion block is specifically trained to harness temporal cues from these frames to reconstruct complete prediction for the current frame. Consequently, our model demonstrates proficiency in synthesizing temporal information in an end-to-end manner. The entire process is steered by a semantic task-specific loss, ensuring that the fusion block is precisely tailored to this task. It selectively merges information from adjacent frames that is beneficial for semantic segmentation. This targeted fusion, guided by the semantic segmentation loss, distinguishes our approach from existing methods by focusing the training on semantically relevant features rather than indiscriminate information amalgamation.
Spatially, our approach within the target pipeline integrates an established method, as delineated in [3]. We adopt this method to ensure the student model can incorporate information from the current frame with fidelity; it is particularly included to preserve and possibly improve the model's spatial accuracy while it learns to integrate temporal information. For the student model, feature maps are extracted from the cropped segment of the current frame prior to their introduction to the fusion block. Conversely, the spatial teacher is provided with a high-resolution variant of the same cropped segment, from which we also derive feature maps before they reach the fusion block. We apply spatial consistency loss directly to the feature maps to align the student's learning with that of the spatial teacher:
H
W
Lepas
1
Σ
spat Fstudy2
(3)
N
where, Frspss are the features from the spatial teacher, and Fatud are the features from the student. Jespat is resized to match the dimensions of the cropped segment for loss computation. The efficacy

---

HA
(a) Previous Frame
(b) Previous with Augs.
(c) Current Frame
(d) Current with Augs.
Figure 4: This illustration demonstrates the temporal weather degradation augmentation technique. For enhanced visualization, we have utilized Cityscapes -Se as an example. Frames (a) and (b) are consecutive frames captured from real-world scene under ideal conditions. Frames (c) and (d) show the same frames, but with applied augmentation, including random noise, a moving glare, rectangle "foggy" area with intensity change, and a changing illumination.
of this technique for guiding the student model learning spatial information from unlabeled target images (in the context of our work. the current frames) has been validated in [13] Thus, this loss safeguards the spatial precision of the model and may also enhance it.
As such, our model innovatively integrates fusion block, trained using insights from a temporal teacher, to weave together information from consecutive frames without depending on pretrained optical flows. This enables the model to inherently learn and apply temporal details for better current frame predictions. Meanwhile, the spatial teacher ensures the model's spatial accuracy not compromised and even enhances its capability to extract spatial information.
3.3 Temporal weather degradation augmentation
Adverse weather conditions introduce two primary categories of degradation in vision tasks: random disturbances, such as noise and occlusions, and specific reeater-related degradations, like low-light, glare, and fog. These degradations occur in similar locations across consecutive frames but exhibit varying intensities due the movement of objects and the camera.
Our model is strategically developed to counteract such degradations by leveraging the temporal information from consecutive frames. Our objective is train it to accurately discern the real scene obscured by weather-specific degradations, through a detailed analysis of the variations in their intensity. To achieve this, we simulate weather-induced impairments, including blur, glare, and changes in illumination and chromaticity to both the source images in the source pipeline and the target images used for the student model in the target pipeline. These augmentations are consistently applied to corresponding regions in consecutive frames, with incremental variations in intensity to mimic the dynamic nature of weather-related visual degradations. An illustration of the augmentation are presented in Fig. 4
Further, consistency loss is employed to ensure that predictions from the augmented frames align with their corresponding ground truths or pseudo-labels. Hence, these augmentation strategically trains the model discern the authentic scene behind weather-induced visual distortions by leveraging the variability of degradation intensities across frames.
Overall, our pipelines be described as follows: Let the input image at frame be denoted as X,, with the student encoder as S and the teacher encoder as T. We define the student fusion block as Fs and the teacher fusion block as Fy. Thus, for the temporal pipeline, we impose the following consistency,
Es Fx(S(ArwD(Xx-i)) ((Crop(ArwD(X.))) = (x.
(4)
where, ATwD represent the temporal weather degradations, and Crop indicates that the model is provided with only a cropped segment of the current frame. By enforcing this consistency, we encourage the student model to align with the teacher's performance. As result, the student model learns to be robust against weather degradation while effectively utilizing information from X,- K to compensate for missing details in the cropped current frame.
For the spatial pipeline, we enforce the following,
((Cro)(ArwD(xi))) = T(X,),
(5)
7

---

Table I: Quantitative results of our method compared existing UDA methods, with both image- based and video-based, evaluated against MVSS 18]. Bold numbers are the best scores, and underline numbers the second best scores. The IoU (%) of all classes and the average mloU (%)
are presented. Our method outperforms the best existing method by 4.3 mloU (%) in average, even with the absence of pretrained optical flows (NOOF).
UDA from Synthetic to Real under adverse weather condition: VIPER MVSS
Method
Design
car
bus
moto.
bey..
pers.
light
sign
sky
road
side.
vege.
teer.
buil.
mloL
Source-only
Image
39.4
2.6
0.0
27.3
13.6
0.6
39.4
6.6
4.2
46.2
20.2
38.2
AdvEnt 33
Image
38.3
4.8
0.5
0.0
26.3
14.6
0.7
40.7
18.2
4.4
45.5
20.7
39.1
19.5
FDA
Image
38.5
2.2
0.3
0.5
21.3
10.7
0.8
41.8
29.4
4.5
51.4
22.5
39.7
20.3
Image
33.3
5.9
8.2
43.1
37.3
5.1
49.9
22.8
42.6
DA-VSN T0
Video
36.0
0.6
0.2
0.0
21.1
0.6
0.9
45,5
34,4
4,0
50.2
23,4
49.0
20.4
SFC 9
Video
41.2
4.0
0.0
0.0
21.3
5,6
0.7
41.4
36.2
4,5
47.2
20.4
38.8
20.1
TPS
Video
45.8
5.1
0.3
18.9
0.0
0.0
39.6
39.7
3.0
20.8
39.1
20.2
26
Video
41.7
5.7
0.0
1.3
14.2
0.2
1.4
36.3
43.3
3.4
46.0
24.7
52.4
20.8
Ours
NOOF
46.0
8.6
0.0
0.5
30.9
1.1
2.3
46.4
60.2
2.7
56.4
20.7
54.3
25.4
where, X, represents the same cropped image segment at higher resolution. By enforcing this consistency, we ensure that the student model remains robust to weather degradation while preserving spatial precision.
3.4 Overall loss
The overall loss of the network is defined as:
L= Laup a a< + Lspat);
(6)
where, Loup denotes the supervised loss used in the source pipeline. The temporal loss and spatial loss in the target pipeline represented by Ltemp and Lspat. respectively. The parameter o, empirically to 0.1, ensures that the losses in the target pipeline do not become overly dominant.
Experiments
In this part of the paper, undertake extensive analysis of our video semantic segmentation approach. The evaluation starts with an overview of the datasets utilized, along with detailed break- down of the models and settings implemented. Subsequently, of our approach, showcasing its capabilities and robustness against a range of challenging weather conditions through both quantitative metrics and qualitative examples. To conclude, we engage in ablation studies to discern the impact and necessity of the distinct components integral to our method.
delve into detailed examination
Datasets For our source datasets in the video semantic segmentation work, we select VIPER 28 and Synthia [29] for their extensive collections of labeled, synthetic urban landscape frames. For our target dataset, we have chosen MVSS [1], which is characterized by its diverse collection of real-world urban scenes captured under various adverse weather conditions. Since VIPER, Synthia, and MVSS have different class protocols, we evaluate only the common classes, following existing UDA methods. We assess target domain segmentation performance using Intersection over Union (IoU%), with higher percentages indicating better performance.
Baseline models In our experiments, compare method with range of UDA techniques, encompassing both image-based and video-based approaches. To ensure equitable comparison, adopt the DeeplabV2 architecture (34] across all methods. The image-based and video-based methods are configured and trained according to their standard settings. For our method, in line with recommendations from [36), we use the same optimization strategy. This includes consistent parameters across all methods, such as the number of epochs, batch sizes, learning rates, and the pretrained backbone, Accel
4.1 Quantitative results
As shown in Tabs. 22 12 our models outperform other methods on the real-world dataset under adverse
weather, MVSS [8] Our model surpasses the second best method by 4.3% and 5.8% in mloU,

---

Table 2: Quantitative results of our method compared existing UDA methods, with both image- based and video-based, evaluated against MVSS 18]. Bold numbers are the best scores, and underline numbers the second best scores. The IoU (%) of all classes and the average mloU (%) are presented. Our method outperforms the best existing method by 5.8 mloU (%) in average, even with the absence of pretrained optical flows (NOOF).
UDA from Synthetic Real under adverse weather condition: Synthia - MVSS
Method
Design
car
bicy.
pers.
pole
light
sign
sky
road
side.
vege.
mloU
Source-only
Image
29.0
0.5
14.5
0.7
0.2
25.2
15,8
10.0
37.2
38.6
17.2
AdvEnt 33
Image
37.6
2.4
27.0
0.5
0.2
36.1
56,4
12.9
32.2
41.7
24.7
FDA (38
Image
40.0
2.5
30.9
1.7
0.1
38.1
59.5
14.9
34.8
43.7
26.6
RDA 15
Image
42.3
2.3
38.4
0.0
0.1
34.0
68.3
13.1
39.7
37.0
27.5
DA VO
Video
41.9
1.2
35.7
0.0
64.6
35.1
40.5
SFC
Video
42.7
0.5
33.0
0.0
0.0
27.2
60.6
16.2
37.1
39.4
25.7
TPS 36
Video
36.4
0.7
40.3
0.0
0.1
34.0
65.7
16.0
42.0
42.5
27.8
MoDA 26
Video
35.2
0.5
23.5
0.3
0.0
41.3
64.9
15.7
41.4
47.3
27.0
Video,
Ours
45.1
1,5
43.1
1.2
0.0
51.1
70.7
19.5
47.4
50.6
33.0
NOOF
Image
TPS 36
Ours
Ground Truth
Figure 5: Comparisons the semantic segmentation performance with TPS 336, Ours, and ground truths on MVSS under rainy and nighttime conditions.
adapting from VIPER 28 and Synthia [29 to MVSS, respectively, our model mark substantial advancements. These consistent gains in loU across most classes highlights our models' robustness against various adverse weather conditions.
It is worth noting that all video-bases methods outperform image- based ones in ideal conditions. However, this advantage does not hold in adverse weather, indicating a failure to effectively use temporal information due unreliable pseudo-labels and optical flows. In contrast, end-to-end designed models consistently leverage temporal information under any condition, showcasing their versatility.
4.2 Qualitative results
Building the qualitative insights from Fig. under foggy and snowy conditions, we extend our performance showcase to include rainy and nighttime scenarios in Fig. 5 Our method is evaluated alongside TPS (36) and compared ground truth segmentation maps. The results highlight that, while TPS tends to yield substantial inaccuracies, our method significantly reduces such errors. This clearly demonstrates our model's robustness in the face of adverse weather conditions.
4.3 Ablation studies
evaluate the effectiveness of each component we implemented on VIPER - MVSS, with the results detailed in Tab. 3 The table reveals that omitting the pretrained optical flow leads to a decrease in performance for the Accel baseline. However, this loss performance mitigated once we incorporate fusion block, underscoring its efficacy alternative to pretrained optical flow,

---

Table Ablation studies of our proposed techniques. We can observe that each component indepen- dently contributes to the overall improvement in performance.
Baselines
Fus. Blk
Tem.
Spa.
Tem. Augs.
mloU (%)
Accel
TPS
Tea.
Tea.
(NOOF)
20.2
18.6
7
/
21.7
/
/
/
23.8
/
/
/
24.3
7
25.4
especially in adverse weather conditions. Furthermore, gradual improvement in mIoU (%) evident as more techniques are incorporated, affirming the positive contribution of each component to the overall semantic segmentation performance under adverse weather conditions.
5 Conclusion
In conclusion, our novel end-to-end video-based method significantly enhances video semantic segmentation in adverse weather conditions, notably achieving this improvement without the reliance on pretrained optical flows. This method includes a fusion block, temporal-spatial teacher-student learning system, and a strategy for temporal weather degradation augmentation. Our fusion block effectively merges temporal information from adjacent frames, eliminating the reliance on pretrained teachers:
optical flows seen in existing works. The teacher-student learning approach temporal teacher for guiding the student to explore the temporal information from adjacent frames, and a spatial teacher train the student harness spatial information from the current frame. Additionally, we apply temporal weather degradation augmentation to accurately simulate and respond to weather-related degradations in consecutive frames. Upon evaluating our models on MVSS dataset featuring real-world adverse weather conditions, we observed that our approach surpasses many existing image-based and video-based methods in performance.
References
Qi Bi, Jingjun Yi, Hao Zheng, Wei Ji, Yawen Huang, Yuexiang Li, and Yefeng Zheng. Learning generalized medical image segmentation from decoupled feature queries. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 810-818, 2024.
Qi Bi, Shaodi You, and Theo Gevers. Interactive learning of intrinsic and extrinsic properties for all-day semantic segmentation. IEEE Transactions on Image Processing, 32:3821-3835, 2023.
[3] Qi Bi, Shaodi You, and Theo Gevers. Generalized foggy-scene semantic segmentation by frequency decoupling. In Proceedings the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1389-1399, 2024.
[4] Qi Bi, Shaodi You, and Theo Gevers. Learning content-cnhanced mask transformer for domain generalized urban-scene segmentation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 819-827, 2024.
[5] Qi Bi. Shaodi You, and Theo Gevers. Learning generalized segmentation for foggy-scents by bi-directional wavelet guidance. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 801-809, 2024.
[6] Jun Chen, Hui DuanStudent Member, Yuanxin SongStudent Member, Zemin Cai, and Guangguang Yang. Optical flow computation for video under the dynamic illumination. IEEE Transactions on Multimedia, 2022.
[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213 3223,
2016.
10

---

[8] Jinhong Deng, Wen Li, Yuhua Chen, and Lixin Duan. Unbiased mean teacher for cross-domain object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4091-4101. 2021.
[9] Yuan Gao, Zilei Wang. Jiafan Zhuang. Yixin Zhang, and Junjie Li. Exploit domain-robust optical flow in domain adaptive video semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[10] Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Domain adaptive video segmentation via temporal consistency regularization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8053-8064, 2021.
[11] Amirhossein Habibian, Haitam Ben Yahia, Davide Abati, Efstratios Gavves, and Fatih Porikli. Delta distillation for efficient video processing. In European Conference on Computer Vision, pages 213-229.
Springer, 2022.
[12] Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Daformer: Improving network architectures and training strategies for domain domain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9924 9935, 2022.
[13] Lukas Hoyer, Dengxin Dai, and Lue Van Gool. Hrda: Context-award high-resolution domain- addatiee semantic segmentation. In European Conference on Computer Vision, pages 372-391. Springer, 2022.
[14] Yubin Hu, Yuze He, Yanghao Li, Jisheng Li, Yuxing Han, Jiangtao Wen, and Yong-Jin Liu. Efficient semantic segmentation by altering resolutions for compressed videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22627-22637, 2023.
[15] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8988-8999 2021.
[16] Takashi Isobe, Xu Jia, Shuaijun Chen, Jianzhong He, Yongjie Shi, Jianzhuang Liu, Huchuan Lu, and Shengjin Wang. Multi-target domain adaptation with collaborative consistency learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8187 8196, 2021.
[17] Samvit Jain, Xin Wang, and Joseph E Gonzalez. Accel: A corrective fusion network for efficient semantic segmentation video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8866-8875, 2019.
[18] Wei Ji, Jingjing Li, Cheng Bian, Zongwei Zhou, Jiaying Zhao, Alan L Yuille, and Li Cheng. Multispectral vidco semantic segmentation: A benchmark dataset and baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1094-1104, 2023.
[19] Tobias Kalb and Jürgen Beyerer. Principles of forgetting in domain-incremennal semantic segmentation in adverse weather conditions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 99508-1958,, 2023.
[20] Mikhail Kemnerley. Jian-Gang Wang, Bharadwaj Veeravalli, and Robby T Tan. 2penet: Two-phase consistency training for day-to-night unsupervised domain adaptive object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1144-11493, 2023.
[21] Mingjia Li, Binhui Xie, Shuang Li, Chi Harold Liu, and Xinjing Cheng. Vble: visibility boosting and logit- constraint learning for domain adaptive semantic segmentation under adverse conditions. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 8605-8613, 2023.
[22] Chen Liang, Qiang Guo, Chongkai Yu, Chengjing Wu, Ting Liu, and Luoqi Liu. Semantic segmentation on vspw dataset through masked video consistency. arXiv preprint arXiv:2406.04979 2024.
[23] Yen-C Yen-Cheng Liu, Chih-Yao Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.0 09480, 2021.
[24] Shao Yuao-YuanLL Lo. Poojan Oza. Sumanth Chennupati, Alejandro Galindo, and Vishal M Patel. Spatio-temporal pixel-level contrastive learning based source-free domain adaptation for video semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10534- 10543, 2023.
11

---

[25] Xianzheng Ma. Zhixiang Wang. Yacheng Zhan. Yingiang Zheng, Zheng Wang, Dengxin Dai, and Chia-Wen Lin. Both style and log matter: Cumulative domain adaptation for semantic foggy scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18922-
18931, 2022.
[26] Fei Pan, Xu Yin, Senkju Lee, Axi Niu, Sungeui Yoon, and In So Kweon. Moda: Leveraging motion priors from videos for advancing unsupervised domain adaptation in semantic segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2649-2658, 2024.
[27] Fitsum Reda, Robert Pottorff, Jon Barker, and Bryan Catanzaro. flownet--pytorch: Pytorch implementation of flownet 2.0: Evolution of optical flow estimation with deep networks. httpa://github. com/ /OVNNIDIA//// flownet2- -pytorch 2017.
[28] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. In Proceedings of the IEEE International Conference on Computer Vision, pages
2017.
[29] German Ros. Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3234-3243, 2016.
[30] Inkyu Shin, Kwanyong Park, Sanghyun Woo, and In So Kweon. Unsupervised domain adaptation for video semantic segmentation. arXiv preprint arXiv:2107. 1052, 2021.
[31] Vishwanath A Sindagi, Poojan Oza, Rajecv Yasarla, and Vishal M Patel. Frior-based domain adaptive object detection for hazy and rainy conditions. In Computer Vision ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 763-780. Springer, 2020.
[32] Guolei Sun, Liu, Henghui Ding, Thomas Probst, and Luc Van Gool. Coarse-to-fite feature mining for video semantic segmentation. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3126-3133 2237,3137,001 2022.
[33] Muan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 2517-2526. 2019.
[34] Mark Weher, Huiyu Wang, Siyuan Qiao, Jun Xie, Maxwell D Collins, Yukun Zhu, Liangzhe Yuan, Dahun Kim, Qihang Yu, Daniel Cremers, et al. Deeplab2: tensorflow library for deep labeling. arXiv preprint arXiv:2106.00 09748, 2021.
[35] Yuctian Weng, Mingfei Han, Haoyu Hc, Mingjie Li, Lina Yao, Xiaojun Chang, and Bohan Zhuang. Mask propagation for efficient video semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[36] Yun Xing, Dayan Guan, Jiaxing Huang, and Shijian Lu. Domain adaptive video segmentation via temporal pscudo supervision. In European Conference on Computer Vision, pages 621-639. Springer, 2022.
[37] Xin Yang, Michael Bi Mi, Yuan Yuan, Xin Wang, and Robby T Tan. Object detection in foggy scenes by embedding depth and reconstruction into domain adaptation. In Proceedings of the Asian Conference on Computer Vision, pages 1093-1108, 2022.
[38] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4085-4095. 2020.
[39] Jingjun Yi, Qi Bi, Hao Zheng, Haolan Zhan, Wei Ji, Yawen Huang, Yuexiang Li, and Yefeng Zheng Learning spectral decerraldddoomottteeed tokens for domain generalized semantic segmentation. In ACM Multimedia 2024.
[40] Yinqiang Zheng, Minglang Zhang, and Feng Lu. Optical flow in the dark. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6749-6757, 2020.
12

---

Table 4: Comparison mloU and Inference Time for Different Models
Model
mloU (%)
Inference Time (s)
FlowNee2+DD--
20.4
0.35
FlowNe12+TPS
20.2
0.17
Ours
25.4
0.11
A
Supplemental material
Inference time analysis
Developing efficient video semantic segmentation models has posed persistent challenge. While numerous accurate architectures exist, their computational demands hinder real-time video frame processing, limiting their usability, particularly in scenarios like autonomous driving under ever- changing conditions.
To address this critical issue, researchers have been exploring time-efficient solutions for video semantic segmentation [17 IT 14] For the existing video-based UDA semantic segmentation methods, TPS (36) has made strides by improving the processing speed threefold compared to its predecessor, DA-V ASN 10]. Nevertheless, these methods rely on pretrained optical flows, introducing additional time overhead during execution.
In contrast, our proposed approach eliminates the need for this extra step, granting us a notable advantage in terms of execution time. To substantiate this claim, we provide a detailed comparison of inference times in Tab. 4 Inference time is computed by averaging the results from processing 1,000 images on one RTX3090 GPU.
In comparison existing methods, approach distinguishes itself by replacing the optical flow generation process with a lightweight fusion block. This substitution not only reduces inference time but also enhances semantic segmentation performance. As result, method stands out promising candidate for practical deployment scenarios such as autonomous driving. It excels in streamlining the inference process, aligning with the demand for efficient real-time video semantic segmentation.
Analysis on ideal conditions
To demonstrate the generalization ability of our methods, further assess our approach using Cityscapes-Seq (7), a dataset comprising real-world urban scenes captured under ideal conditions. As shown in Tabs. 5 where we adapt the models from VIPER and Synthia, Cityscapes-Seq (7]. Despite being primarily designed for adverse weather, our models demonstrate effective generalization in ideal conditions, achieving comparable performance other methods specifically designed for such conditions, even without using the informative optical flow.
Network configurations
The detailed network structures of the Fusion Block can be found in Tab. 7 C' represents the number of channels, which is defined to be the same as the number of classes. This Fusion Block fuses information from adjacent frames into the prediction of the current frame by matching relevant information from the surrounding pixels of the adjacent frame through the offset layers, and then combining information from different frames. Thus, temporal knowledge is incorporated without the need for optical flows.
13

---

Table 5: Quantitative results of our method compared existing UDA methods, with both image- based and video-based, evaluated against Cityscapes-Sed [7]. Bold numbers are the best scores, and underline numbers are the second best scores. The IoU (%) of all classes and the average mloU (%) are presented.
Under ideal condition: VIPER Cityscapes-Sey
Method
Design
mloU
Source-only
Image
37.1
AdvEnt[33]
Image
44.5
FDA 38
Image
44.4
RDA
Image
44.4
10
Video
47.8
VAT-VST
Video
48.7
SFC
Video
51.7
TPS 36
Video
48.9
Video,
Ours
51.2
NOOF
Table 6: Quantitative results of our method compared to existing UDA methods, with both image- based and video-based, evaluated against Cityscapes-Seq (7]. Bold numbers the best scores, and underline numbers are the second best scores. The loU (%) of all classes and the average mloU (%) presented.
Under ideal condition: Synthia Cityscapes-Sed
Method
Design
mloU
Source-only
Image
38.3
AdvEnt 33
Image
44.0
FD8
Image
45,2
RDA
Image
45.1
DA-VSN
Video
49.5
VAT-VST
Video
47.1
SFC
Video
55.3
TPS
Video
53.8
Video.
Ours
51.0
NOOF
NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information how each question in the checklist:
these questions. For
You should answer [Yes], , [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
14

---

Table 7: Network Structure of Fusion Block
Fusion Block
Bottleneck
Conv, 3 x 3, 2C, stride 1, padding 0
Offset1
Conv, 3 x 3, C, stride 1, padding 1, Sigmoid
Fusel
DeformConv, 3 x 3, C, stride 1, padding 0
Offset2
Conv, x 3, C, stride 1, padding Sigmoid
Fuse2
DeformConv, 3 x 3, C, stride 1, padding 0
Offset3
Conv, x C, stride 1, padding Sigmoid
Fuse3
DeformCCnv, 3 x 3, C, stride 1, paddingdggggggg 0
Offset4
Conv, 3 x 3, C, stride I, padding 1, Sigmoid
Fuse4
DetormConv, 3 3, C, stride I. padding 0
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] " or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
All the claims are supported by the related experiments, evidences, or refer-
ences.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [NA]
Justification: Compared to the baselines, the method robust to different adverse conditions, while the framework does not introduce additional overhead.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
15

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
All the details are included in the paper, we will also release our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
16

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We will release our code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/guides//odeee SodedddeeeeeeSSubbbbocooe isnPoolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines tpt
/ /niis.cccpubliic gwwp://giiissse.eu CodeeSmmeiiiaeccc SinSSuuubiiiiie ssi on Policyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
17

---

Answer: [Yes]
Justification:
The paper includes all the experiment settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The paper does not include statistical significance experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгrоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification:
The paper does not include resource-related experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
18

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics ht httiiigtttssss//aappp...pppppppp
Answer: [Yes]
Justification: The research conforms the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
19

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. www. con/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
20

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "V33v47Y30l", "reviewer": "Reviewer_G5E9", "summary": "This paper studies an important task of video semantic segmentation. Specifically, it focuses on adverse weather scenes and proposes an end-to-end, optical-flow-free, and domain-adaptive algorithm by using fusion blocks and temporal-spatial teachers. Extensive experiments are conducted on VIPER, Synthia and MVSS. This is an interesting paper. However, some issues need to be addressed.", "questions": "Please address the weaknesses above", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper studies an important task of video semantic segmentation under adverse weather conditions. \n2. Good results are obtained. It shows clear improvements over compared methods (SFC and TPS).\n3. In general, this paper is easy to follow and the proposed modules are easy to understand.", "weaknesses": "1. The fusion module, one of the main contributions, does not have much novelty. In my opinion, it is just concatenation of features. \n2. Temporal-Spatial Teacher-Student learning seems not novel. \n3. It is interesting that the proposed method does not use optical flow but achieve much better performance than those using optical flows. In my opinion, optical flows provide more relevant information and could help a lot in video segmentation. So are the comparisons between optical-flow-based methods and the proposed one fair? Are there factors which could boost performance used in this paper, but not used in previous methods?\n4. The discussions about related works are not enough. This task is very close to video semantic segmentation. However, many methods are not discussed and compared with the proposed one. The use of temporal information is largely explored in video segmentation domain and it is necessary to explain how this paper is \"technically\" different in terms of using temporal information. Those works include, but not limited to: a. Mask Propagation for Efficient Video Semantic Segmentation; b. Coarse-to-Fine Feature Mining for Video Semantic Segmentation; c. Semantic Segmentation on VSPW Dataset through Masked Video Consistency", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer G5E9 for recognizing the importance of our task, our good results, and the clarity of our paper.\n\n__Weakness-1 and Weakness-2__\n\nThank you for your valuable feedback. We will address both Weakness-1 and Weakness-2 together, as they are closely related. Additionally, the novelty of our fusion block should be considered an integral part of our end-to-end framework for temporal modeling, rather than a standalone module.\n\nThe innovation of our Temporal-Spatial Teacher-Student learning approach lies in providing a solution for temporal modeling without the use of optical flow, particularly in scenarios where ground truths for the target domain are unavailable. In our case, since a significant part of the current frame is completely erased, there are no clues left for the student model to learn from. Consequently, the student model must extract temporal information from adjacent frames to fill in the erased parts and produce a consistent prediction with the temporal teacher’s pseudo labels. Unlike existing methods that use optical flow to warp temporal information, our approach enables the student model to learn to gather temporal information through this teacher-student learning mechanism. During the learning process, the spatial teacher ensures the spatial precision of the model and may also enhance it, preventing the network from focusing too much on temporal information and neglecting the spatial information in the current frame.\n\nSince we intend to avoid using optical flows under adverse weather conditions, we need to design an alternative solution to merge temporal information for the student network. Therefore, our fusion block is designed to \"offset\" and \"fuse\" information from different frames. Objects in consecutive frames can move to different locations due to their motion. For instance, a car in the bottom-left corner of frame t might move to the top-right corner in frame t+1. Directly concatenating features from such frames can cause misalignment due to large displacements. Existing methods use optical flow to \"offset\" objects between frames before concatenating features. Instead of using optical flow, we integrate deformable convolutional layers with standard convolutional layers. The deformable convolutional layers learn an \"offset\" to relocate features, followed by standard convolutional layers to \"fuse\" the relocated features, as illustrated in Figure 2(c). This combination allows our network to gather temporal information without any pretrained optical flows and enables end-to-end training with the Temporal-Spatial Teacher-Student learning approach under adverse weather conditions.\n\nTo the best of our knowledge, integrating temporal and spatial modeling using two teachers and one student to achieve an optical-flow-free model is novel. Additionally, the fusion block and its integration into our temporal teacher-student framework have not been explored before. To avoid any misunderstanding, we will clarify this in our paper\n\n\n__Weakness-3:__\n\nRegarding the fairness of our comparison, the backbone for all the models is the same, using AdvEnt, as stated in the experiment section. Apart from the components discussed in our paper, we did not add any other elements to our experiments.\n\nUnder clear weather conditions, accurate optical flow can indeed provide strong prior information for generating better predictions. However, under adverse weather conditions, existing optical flow methods can be erroneous, leading to degraded performance. Existing methods use optical flows generated from pretrained FlowNet 2.0 [20], which is trained on clear synthetic datasets and thus become erroneous when applied to adverse weather scenes. Although we could augment clear data with synthetic adverse weather conditions, it is well known that there are significant gaps between synthetic and real weather conditions. Figure 3 in the main paper shows that under clear daytime conditions (left columns), the optical flows are accurate and useful for semantic segmentation. However, under adverse weather conditions (right columns), the optical flow predictions are erroneous. Consequently, using generated optical flows can introduce incorrect information and decrease the performance of existing models.\n\n__Weakness-4:__\n\nFollowing the suggestion, we will expand the related work section to include a video semantic segmentation subsection, covering [a-c] and other relevant works.\n\nOur use of temporal information differs from that in methods [a-c], which are conventional video semantic segmentation tasks trained with ground-truth labels from datasets like VSPW. In those methods, models learn to capture temporal information from consecutive frames using supervision from these labels. In contrast, our paper focuses on self-supervised domain adaptive video semantic segmentation, which does not rely on ground-truth labels.\n\nWithout ground-truth supervision, domain adaptive methods cannot directly learn temporal information. As discussed in our paper, existing methods [4, 5, 18, 23, 27] use optical flow to warp predictions from consecutive frames to generate pseudo labels, thereby “forcing” the model to learn temporal information. However, generated optical flow can be erroneous under adverse weather conditions, as evidenced in Figure 3. Therefore, we develop an optical-flow-free approach to compel the model to learn temporal information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BHq8fT4eAr", "reviewer": "Reviewer_L8p7", "summary": "In this paper, a end-to-end domain-adaptive video semantic segmentation method without optical flow estimation is proposed to address the problem of video frame quality degradation under adverse weather conditions. The proposed method uses the temporal information of adjacent frames through fusion blocks and spatiotemporal teacher models to enhance the model's robustness to video semantic segmentation under adverse weather conditions. The fusion block combines information by matching and fusing relevant information pixels from adjacent frames. The spatiotemporal teacher model includes a temporal teacher and a spatial teacher to guide the student model from the temporal dimension and the spatial dimension, respectively.", "questions": "Please refer to the weaknesses.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. For the first time, an end-to-end video semantic segmentation method without optical flow estimation is proposed, which is suitable for adverse weather conditions.\n2. The model's adaptability to real scenarios is enhanced by simulating dynamic weather degradation in consecutive frames.\n3. The article achieves significant performance improvements on multiple datasets, surpassing existing state-of-the-art methods.", "weaknesses": "1. The paper does not have any comparison or explanation on the amount of calculation and the number of parameters, which makes me worry about the practicality of the method.\n2. There are few baselines selected for visual comparison, which makes it difficult to reflect the effectiveness of the proposed method.\n3. The paper may lack in-depth discussion and justification of the theoretical basis of the proposed method.\n4. Is there some newer sota method that can be compared? The method in the table doesn't seem to be up to date.\n5. The work in this article is carried out under severe weather degradation conditions. Therefore, I think that the article should add a section on image restoration in the related works for discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer L8p7 for recognizing the importance and performance of our work. Here is our response to the feedback:\n\n__Weakness-1:__\n \nThank you for the feedback. Our model has an inference time of 0.11 seconds, which is faster than the state-of-the-art baselines: DA-VSN's 0.35 seconds and TPS's 0.17 seconds. In terms of parameters, our model’s size is 173MB, whereas DA-VSN is 619MB (FlowNet part) + 169MB (segmentation part), and TPS is 619MB (FlowNet part) + 169MB (segmentation part). Note that, although TPS is a faster and more accurate version of DA-VSN, both share the same number of parameters.\n\n__Weakness-2 and Weakness-4:__\n\nThank you for pointing this out. In the main paper, we selected TPS for visual comparison because it has the second-best quantitative performance. In addition to this, we recently found a CVPR’24 paper (published after the NeurIPS’24 submission deadline), but its code has not yet been released. We implemented it ourselves following the motion training provided by the authors. We will include this CVPR’24 paper and the related discussion stated in this rebuttal in our main paper. Additionally, we will add a more thorough survey of this year’s publications to our main paper.\n\nHere is the qualitative comparison. For IoU, higher values are better.\n\nFor Table 1, Viper to MVSS domain adaptation,\n\n|Method|Design|car|bus|moto.|bicy.|pers.|light|sign|sky|road|side.|vege.|terr.|buil.|mIoU|\n|------|------|---|---|-----|-----|-----|-----|----|---|----|-----|-----|-----|-----|----|\n|MoDA [1]|Video|41.7|5.7|0.0|1.3|14.2|0.2|1.4|36.3|43.3|3.4|46.0|24.7|52.4|20.8|\n|Ours|Video|__46.0__|__8.6__|0.0|0.5|__30.9__|__1.1__|__2.3__|__46.4__|__60.2__|2.7|__56.4__|20.7|__54.3__|__25.4__|\n\n\nFor Table 2, Synthia to MVSS domain adaptation,\n\n|Method|Design|car|bicy.|pers.|pole|light|sign|sky|road|side.|vege.|mIoU|\n|------|------|---|-----|-----|----|-----|----|---|----|-----|-----|----|\n|MoDA [1]|Video|35.2|0.5|23.5|0.3|0.0|41.3|64.9|15.7|41.4|47.3|27.0|\n|Ours|Video|__45.1__|__1.5__|__43.1__|__1.2__|0.0|__51.1__|__70.7__|__19.5__|__47.4__|__50.6__|__33.0__|\n\nWe have also included the qualitative comparison in the attached rebuttal PDF.\n\nPan et al., “MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation”, CVPR’24\n\n__Weakness-3:__\n\nWe have made every effort to address the reviewer’s concerns regarding the lack of in-depth discussion and theoretical justification, which we will elaborate below. However, with all due respect, we are unsure which specific parts of our method require deeper analysis and further theoretical justification. More detailed feedback on this would be much appreciated.\n\nExisting optical flow methods often fail under adverse weather conditions. To address this issue, we use a temporal teacher to guide the student network in learning from adjacent frames, instead of relying on optical flow. Here is our basic idea. We input the current frame into the temporal teacher, which generates predictions used as pseudo-labels. Concurrently, we mask out part of the current frame and feed it into the student network. We enforce a consistency loss between the student’s predictions and the pseudo-labels from the temporal teacher. This approach encourages the student network to reconstruct the masked-out information by leveraging temporal data from adjacent frames, so that it can align its predictions with the pseudo-labels.\n\nTo ensure that the student network does not overlook important spatial information while focusing on temporal data, we incorporate a spatial teacher. This integration helps the student model learn and utilize relevant spatial information as well.\n\nAs objects move, their features can appear in different locations across frames, making feature alignment challenging as we don’t have optical flow information. To address this, we use a fusion block with deformable convolutional layers to align features from different frames, followed by standard convolutional layers for merging. This block is trained end-to-end with the temporal teacher, enabling the network to effectively integrate temporal information from various frames and improve the model’s semantic segmentation performance.\n\n\n__Weaknesses-5:__\nThank you, we will follow the suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j1zuu5T4u9", "reviewer": "Reviewer_1Ffg", "summary": "This paper proposes a video segmentation method for adverse weather conditions by using the unsupervised domain adaptation paradigm. \n\nIts general idea is to introduce the temporal information from adjacent video frames by the proposed spatial-temporal teacher-student learning scheme. \n\nNotably, the proposed method is end-to-end, and does not rely on optical flows from pretrained models.\n\nIn addition, a temporal weather degradation augmentation is introduced to mimic the real-world weather variations across consecutive frames.\n\nExperiments show the state-of-the-art performance of the proposed method.", "questions": "- Q1: Line 264, could the authors explain what types of adverse weather conditions are in the MVSS dataset?\n\n- Q2: The module design and configuration of the teacher-student pipeline is missing. Please provide accordingly.\n\n- Q3: The related work in this paper is not extensive. Discuss some more recent works, such as [a,b,c].\n\n- Q4: More visual segmentation results should be provided.\n\n- Q5: Multiple presentation issues. \n\n- Q6: The figures in this paper can be significantly polished.", "limitations": "The authors do not provide a limitation discussion.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The proposed method is the first work for video semantic segmentation under the adverse weather conditions, which can benefit the vision community on a variety of tasks and applications.\n\n+ The proposed method is moderately novel and rationale. Most importantly, it is end-to-end, and does not rely on optical flows from pretrained models.\n\n+ A simple but effective temporal augmentation strategy is proposed to augment the weather degradations among consecutive frames.\n\n+ The proposed method significantly outperforms the state-of-the-art methods under a variety of settings.", "weaknesses": "- The presentation of this work may focus too much on the high-level conceptual clarity, which leads to the miss of some important experiment and implementation details. For example:\n\n(1) Line 264, could the authors explain what types of adverse weather conditions are in the MVSS dataset?\n\n(2) The module design and configuration of the teacher-student pipeline is missing. Please provide accordingly. \n\n- The related work in this paper is not very extensive. Some important references are missing. For example:\n\n(1) Please cite and discuss STPL [a]. It is a subsequent work following DA-VSN and TPS.\n(2) Some more recent works in 2023-2024 on video segmentation [b] and adverse conditions [c] can be discussed. \n\n[a] Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free Domain Adaptation for Video Semantic Segmentation. CVPR 2023.\n\n[b] Multispectral video semantic segmentation: A benchmark dataset and baseline. CVPR 2023.\n\n[c] Learning generalized segmentation for foggy-scenes by bi-directional wavelet guidance. AAAI 2024.\n\n\n\n- More visual segmentation results should be provided. Currently, only Fig.1 and 5 have several visual results.\n\n- There are multiple presentation issues, especially inconsistency, in this submission. For example:\n\n(1) The title of the submission is not consistent with the title in OpenReview. Please unify it. \n\n(2) Line 78-79, mIoU should be presented in percentage, not number. E.g., 4.3% mIoU.\n\n(3) The caption of Fig.3 is difficult to understand, and need to be simplified. \n\n(4) Table 1, 2, 5 and 6. When reporting the performance of ours, ‘Video,’ should be ‘video’.\n\n(5) Inconsistency between ‘Flow2Net’ and ‘FlowNet2’. Please unify it.\n\n- The figures in this paper can be significantly polished. For example:\n\n(1) Fig.2. The teacher/student net and the fusion block in (c) can be more specified.\n\n(2) Fig.3 is not very informative, as it is not the important results or design. Maybe it can be incorporated into Fig.2. \n\n(3) Fig.4. It would be much better to place one type of augmentation on one image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 1Ffg for recognizing that our work can benefit the community, novel, effective, and with a significant performance gain.\n\n__Q1:__ The MVSS dataset consists of a total of 52,735 RGB images, with 3,545 of these images annotated. The dataset includes a variety of adverse conditions such as overexposure, nighttime, rain, fog, and snow.\n\n__Q2:__ In the teacher-student pipeline, the weights of both teachers are updated using the following equation: \n$W_t(i+1) = 0.9995W_t(i) + (1 - 0.9995)W_s(i)$, where $W_t(i)$ represents the weight of the teacher models at iteration $i$, and $W_s(i)$ represents the weight of the student model at iteration $i$. For the weighing factors of the teacher-student losses, we use the ratio of pixels exceeding a threshold $\\tau \\ (0.968)$ of the maximum softmax probability, as suggested in [7]. For all other configurations, we use the same settings as those in the existing methods described in [5, 27] to ensure a fair comparison.\n\n__Q3__: We appreciate the suggestion. We will expand the related work and discuss all the suggested papers.\n\n__Q4__: We have included a more recent method for more qualitative comparison in the attached rebuttal.pdf.\n\n__Q5:__ Thank you for pointing out the presentation issues in the paper, we will revise all of them accordingly.\n\n__Q6:__ We appreciate the suggestion. We will further improve the presentation of the images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "91bTmSNIVR", "reviewer": "Reviewer_dY49", "summary": "This paper provides a practical solution for video-based semantic segmentation under adverse weather conditions. Existing methods mainly focus on domain adaptation from synthetic to real data (Viper/Synthia to Cityscapes), but this is the first paper to address videos under adverse weather conditions.\n\nCurrent approaches heavily depend on optical flows generated from pretrained models to gather temporal information from adjacent frames for pseudo label-based learning. However, this paper highlights that the generated optical flow can be significantly inaccurate due to the degradations caused by adverse weather conditions.\nTo solve this problem, the authors proposed an end-to-end semantic segmentation model that directly incorporates temporal information from adjacent frames through a temporal and spatial teacher-student learning approach.\nTo further enhance the model's robustness under adverse weather conditions, the authors introduced a temporal weather degradation augmentation, which mimics real-world scenarios where degradation is similar but varies in intensity across consecutive frames.\n\nThe paper conducted experiments on Viper/Synthia to MVSS domain adaptation and achieved state-of-the-art performance even without using optical flow information.", "questions": "Please refer to the weakness.", "limitations": "I do not see any potential negative societal impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Video domain adaptation under adverse conditions is an important and practical task.\n\n2. Identifying the issue of optical flows under adverse conditions and proposing a self-learning solution to gather temporal information is novel and contributes to the community.\n\n3. The method achieves good performance even without relying on optical flow information from pretrained models.", "weaknesses": "1. According to your method, the spatial loss should be computed between the teacher's output and the corresponding segment of the student's output. This is unclear in Figure 2, as the arrow points to the entire image, including the white area. Is the white area also involved in computing spatial loss?\n\n2. In the caption of Figure 3, it should state, \"the left two columns display the frames and optical flows under ideal conditions\" and \"the right two columns,\" not \"top two rows\" or \"bottom two rows.\"\n\n3. For the teacher-student approach: \n     - What is the weight smooth coefficient parameter alpha for the EMA updating? \n     - How are the pseudo labels selected? Is it done using a threshold?\n\n4. Compared to image-based domain adaptation papers using datasets such as Cityscapes [1], the qualitative results in your paper on MVSS [2] seem poor for both TPS [3] and your method.\n\n5. The inference speed comparison in Table 4 should be highlighted and included in the main paper. Inference speed is crucial for video semantic segmentation, making it a key reason why Accel [4] is preferred over the latest transformer-based methods in this domain. I suggest discussing Accel in the related work section, as it is used in all your video-based benchmarks.\n\n6. Details of the temporal augmentations are missing. For instance, the implementation of \"foggy\" areas and glare effects should be explained.\n\n7. It would be beneficial to include an overall loss equation, Loverall, that describes how Lsup, Ltemp, and Lspat​ are integrated and weighted.\n\n[1] The cityscapes dataset for semantic urban scene understanding\n\n[2] Multispectral video semantic segmentation: A benchmark dataset and baseline\n\n[3] Domain Adaptive Video Segmentation via Temporal Pseudo Supervision\n\n[4] Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that our task is important, the idea is novel and contributes to the community, and with a good performance.\n\n__Weakness-1:__ The white area is not involved in computing the spatial loss, we will adjust the arrow to make it correctly pointing to the right-bottom part.\n\n__Weakness-2:__ Thank you for pointing out the issue. Your understanding is correct, and we will revise it accordingly.\n\n__Weakness-3:__ The weight smooth coefficient is 0.9995. Regarding the pseudo-label selection for the teacher-student losses, we use the ratio of pixels exceeding a threshold τ (0.968) of the maximum softmax probability, as suggested in [7].\n\n__Weakness-4:__ Unlike the Cityscapes dataset, which contains images captured under clear conditions using an automotive-grade 22 cm baseline stereo camera and produces high-quality images, the MVSS dataset is more suited to real-world scenarios. The MVSS images are captured from different cameras under various conditions and are of relatively lower quality. The Cityscapes dataset features images with a resolution of 2048 x 1024, whereas the MVSS dataset images have a resolution smaller than 630 x 460. Due to this lower quality, the qualitative results from MVSS inputs do not appear as good as those from Cityscapes.\n\n__Weakness-5:__\nThank you for the suggestion, we will include Accel in the related work and emphasis the importance of inference speed.\n\n__Weakness-6:__\nFor the “foggy” and glare effects, we first randomly select the affected area in the current frame. To create the “foggy” area, we adjust the gamma and chromaticity. The glare effect is produced by using a Gaussian kernel at the selected area, with the center having the highest values that gradually decrease with distance from the center. Once the augmentations in the current frame are completed, we add the augmentations to the adjacent frames. However, the affected area will randomly move around, and the intensity of the augmentations will also vary, mimicking real-world scenarios.\n\n__Weakness-7:__\nThank you for the suggestion and we will follow accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful feedback. We are encouraged by their recognition of the importance of our task (G5E9, L8p7, dY49) and its potential benefit to the community (1Ffg, dY49). We are also pleased that they acknowledged the novelty of our method (1Ffg, dY49). Additionally, we appreciate that all the reviewers recognized the effectiveness of our method and noted its clear improvement compared to existing methods.


We have recently found a CVPR’24 paper, MoDA (published after the NeurIPS’24 submission deadline). We include it here for further comparison and will also incorporate it into the main paper.

Below are the quantitive comparisons. For IoU, higher values are better.

For Table 1, Viper to MVSS domain adaptation,

|Method|Design|car|bus|moto.|bicy.|pers.|light|sign|sky|road|side.|vege.|terr.|buil.|mIoU|
|------|------|---|---|-----|-----|-----|-----|----|---|----|-----|-----|-----|-----|----|
|MoDA [1]|Video|41.7|5.7|0.0|1.3|14.2|0.2|1.4|36.3|43.3|3.4|46.0|24.7|52.4|20.8|
|Ours|Video|__46.0__|__8.6__|0.0|0.5|__30.9__|__1.1__|__2.3__|__46.4__|__60.2__|2.7|__56.4__|20.7|__54.3__|__25.4__|


For Table 2, Synthia to MVSS domain adaptation,

|Method|Design|car|bicy.|pers.|pole|light|sign|sky|road|side.|vege.|mIoU|
|------|------|---|-----|-----|----|-----|----|---|----|-----|-----|----|
|MoDA [1]|Video|35.2|0.5|23.5|0.3|0.0|41.3|64.9|15.7|41.4|47.3|27.0|
|Ours|Video|__45.1__|__1.5__|__43.1__|__1.2__|0.0|__51.1__|__70.7__|__19.5__|__47.4__|__50.6__|__33.0__|

We have also included the qualitative comparison in the attached rebuttal PDF.

Pan et al., “MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation”, CVPR’24

Meta-review: All reviewers highlight the strong contributions. The rebuttal has alleviated the remaining concerns, and all reviewers vote for accepting the paper.
Decision: Accept (poster)