Title: End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: optical-flow free

video semantic segmentation

adverse conditions
Abstract: Adverse weather conditions can significantly degrade the video frames, causing existing video semantic segmentation methods to produce erroneous predictions. In this work, we target adverse weather conditions and introduce an end-to-end domain adaptation strategy that leverages a fusion block, temporal-spatial teacher-student learning, and a temporal weather degradation augmentation approach. The fusion block integrates temporal information from adjacent frames at the feature level, trained end-to-end, eliminating the need for pretrained optical flow, distinguishing our method from existing approaches. Our teacher-student approach involves two teachers: one focuses on exploring temporal information from adjacent frames, and the other harnesses spatial information from the current frame. Finally, we apply temporal weather degradation augmentation to consecutive frames to more accurately represent adverse weather degradations. Our method achieves a performance of 25.4 and 33.0 mIoU on the adaptation from VIPER and Synthia to MVSS, respectively, representing an improvement of 4.3 and 5.8 mIoU over the existing state-of-the-art method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a practical solution for video-based semantic segmentation under adverse weather conditions. Existing methods mainly focus on domain adaptation from synthetic to real data (Viper/Synthia to Cityscapes), but this is the first paper to address videos under adverse weather conditions.\n\nCurrent approaches heavily depend on optical flows generated from pretrained models to gather temporal information from adjacent frames for pseudo label-based learning. However, this paper highlights that the generated optical flow can be significantly inaccurate due to the degradations caused by adverse weather conditions.\nTo solve this problem, the authors proposed an end-to-end semantic segmentation model that directly incorporates temporal information from adjacent frames through a temporal and spatial teacher-student learning approach.\nTo further enhance the model's robustness under adverse weather conditions, the authors introduced a temporal weather degradation augmentation, which mimics real-world scenarios where degradation is similar but varies in intensity across consecutive frames.\n\nThe paper conducted experiments on Viper/Synthia to MVSS domain adaptation and achieved state-of-the-art performance even without using optical flow information.\n\nSTRENGTHS:\n1. Video domain adaptation under adverse conditions is an important and practical task.\n\n2. Identifying the issue of optical flows under adverse conditions and proposing a self-learning solution to gather temporal information is novel and contributes to the community.\n\n3. The method achieves good performance even without relying on optical flow information from pretrained models.\n\nWEAKNESSES:\n1. According to your method, the spatial loss should be computed between the teacher's output and the corresponding segment of the student's output. This is unclear in Figure 2, as the arrow points to the entire image, including the white area. Is the white area also involved in computing spatial loss?\n\n2. In the caption of Figure 3, it should state, \"the left two columns display the frames and optical flows under ideal conditions\" and \"the right two columns,\" not \"top two rows\" or \"bottom two rows.\"\n\n3. For the teacher-student approach: \n     - What is the weight smooth coefficient parameter alpha for the EMA updating? \n     - How are the pseudo labels selected? Is it done using a threshold?\n\n4. Compared to image-based domain adaptation papers using datasets such as Cityscapes [1], the qualitative results in your paper on MVSS [2] seem poor for both TPS [3] and your method.\n\n5. The inference speed comparison in Table 4 should be highlighted and included in the main paper. Inference speed is crucial for video semantic segmentation, making it a key reason why Accel [4] is preferred over the latest transformer-based methods in this domain. I suggest discussing Accel in the related work section, as it is used in all your video-based benchmarks.\n\n6. Details of the temporal augmentations are missing. For instance, the implementation of \"foggy\" areas and glare effects should be explained.\n\n7. It would be beneficial to include an overall loss equation, Loverall, that describes how Lsup, Ltemp, and Lspat​ are integrated and weighted.\n\n[1] The cityscapes dataset for semantic urban scene understanding\n\n[2] Multispectral video semantic segmentation: A benchmark dataset and baseline\n\n[3] Domain Adaptive Video Segmentation via Temporal Pseudo Supervision\n\n[4] Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Video\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nI do not see any potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a video segmentation method for adverse weather conditions by using the unsupervised domain adaptation paradigm. \n\nIts general idea is to introduce the temporal information from adjacent video frames by the proposed spatial-temporal teacher-student learning scheme. \n\nNotably, the proposed method is end-to-end, and does not rely on optical flows from pretrained models.\n\nIn addition, a temporal weather degradation augmentation is introduced to mimic the real-world weather variations across consecutive frames.\n\nExperiments show the state-of-the-art performance of the proposed method.\n\nSTRENGTHS:\n+ The proposed method is the first work for video semantic segmentation under the adverse weather conditions, which can benefit the vision community on a variety of tasks and applications.\n\n+ The proposed method is moderately novel and rationale. Most importantly, it is end-to-end, and does not rely on optical flows from pretrained models.\n\n+ A simple but effective temporal augmentation strategy is proposed to augment the weather degradations among consecutive frames.\n\n+ The proposed method significantly outperforms the state-of-the-art methods under a variety of settings.\n\nWEAKNESSES:\n- The presentation of this work may focus too much on the high-level conceptual clarity, which leads to the miss of some important experiment and implementation details. For example:\n\n(1) Line 264, could the authors explain what types of adverse weather conditions are in the MVSS dataset?\n\n(2) The module design and configuration of the teacher-student pipeline is missing. Please provide accordingly. \n\n- The related work in this paper is not very extensive. Some important references are missing. For example:\n\n(1) Please cite and discuss STPL [a]. It is a subsequent work following DA-VSN and TPS.\n(2) Some more recent works in 2023-2024 on video segmentation [b] and adverse conditions [c] can be discussed. \n\n[a] Spatio-Temporal Pixel-Level Contrastive Learning-based Source-Free Domain Adaptation for Video Semantic Segmentation. CVPR 2023.\n\n[b] Multispectral video semantic segmentation: A benchmark dataset and baseline. CVPR 2023.\n\n[c] Learning generalized segmentation for foggy-scenes by bi-directional wavelet guidance. AAAI 2024.\n\n\n\n- More visual segmentation results should be provided. Currently, only Fig.1 and 5 have several visual results.\n\n- There are multiple presentation issues, especially inconsistency, in this submission. For example:\n\n(1) The title of the submission is not consistent with the title in OpenReview. Please unify it. \n\n(2) Line 78-79, mIoU should be presented in percentage, not number. E.g., 4.3% mIoU.\n\n(3) The caption of Fig.3 is difficult to understand, and need to be simplified. \n\n(4) Table 1, 2, 5 and 6. When reporting the performance of ours, ‘Video,’ should be ‘video’.\n\n(5) Inconsistency between ‘Flow2Net’ and ‘FlowNet2’. Please unify it.\n\n- The figures in this paper can be significantly polished. For example:\n\n(1) Fig.2. The teacher/student net and the fusion block in (c) can be more specified.\n\n(2) Fig.3 is not very informative, as it is not the important results or design. Maybe it can be incorporated into Fig.2. \n\n(3) Fig.4. It would be much better to place one type of augmentation on one image.\n\nQUESTIONS:\n- Q1: Line 264, could the authors explain what types of adverse weather conditions are in the MVSS dataset?\n\n- Q2: The module design and configuration of the teacher-student pipeline is missing. Please provide accordingly.\n\n- Q3: The related work in this paper is not extensive. Discuss some more recent works, such as [a,b,c].\n\n- Q4: More visual segmentation results should be provided.\n\n- Q5: Multiple presentation issues. \n\n- Q6: The figures in this paper can be significantly polished.\n\nLIMITATIONS:\nThe authors do not provide a limitation discussion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a end-to-end domain-adaptive video semantic segmentation method without optical flow estimation is proposed to address the problem of video frame quality degradation under adverse weather conditions. The proposed method uses the temporal information of adjacent frames through fusion blocks and spatiotemporal teacher models to enhance the model's robustness to video semantic segmentation under adverse weather conditions. The fusion block combines information by matching and fusing relevant information pixels from adjacent frames. The spatiotemporal teacher model includes a temporal teacher and a spatial teacher to guide the student model from the temporal dimension and the spatial dimension, respectively.\n\nSTRENGTHS:\n1. For the first time, an end-to-end video semantic segmentation method without optical flow estimation is proposed, which is suitable for adverse weather conditions.\n2. The model's adaptability to real scenarios is enhanced by simulating dynamic weather degradation in consecutive frames.\n3. The article achieves significant performance improvements on multiple datasets, surpassing existing state-of-the-art methods.\n\nWEAKNESSES:\n1. The paper does not have any comparison or explanation on the amount of calculation and the number of parameters, which makes me worry about the practicality of the method.\n2. There are few baselines selected for visual comparison, which makes it difficult to reflect the effectiveness of the proposed method.\n3. The paper may lack in-depth discussion and justification of the theoretical basis of the proposed method.\n4. Is there some newer sota method that can be compared? The method in the table doesn't seem to be up to date.\n5. The work in this article is carried out under severe weather degradation conditions. Therefore, I think that the article should add a section on image restoration in the related works for discussion.\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies an important task of video semantic segmentation. Specifically, it focuses on adverse weather scenes and proposes an end-to-end, optical-flow-free, and domain-adaptive algorithm by using fusion blocks and temporal-spatial teachers. Extensive experiments are conducted on VIPER, Synthia and MVSS. This is an interesting paper. However, some issues need to be addressed.\n\nSTRENGTHS:\n1. This paper studies an important task of video semantic segmentation under adverse weather conditions. \n2. Good results are obtained. It shows clear improvements over compared methods (SFC and TPS).\n3. In general, this paper is easy to follow and the proposed modules are easy to understand.\n\nWEAKNESSES:\n1. The fusion module, one of the main contributions, does not have much novelty. In my opinion, it is just concatenation of features. \n2. Temporal-Spatial Teacher-Student learning seems not novel. \n3. It is interesting that the proposed method does not use optical flow but achieve much better performance than those using optical flows. In my opinion, optical flows provide more relevant information and could help a lot in video segmentation. So are the comparisons between optical-flow-based methods and the proposed one fair? Are there factors which could boost performance used in this paper, but not used in previous methods?\n4. The discussions about related works are not enough. This task is very close to video semantic segmentation. However, many methods are not discussed and compared with the proposed one. The use of temporal information is largely explored in video segmentation domain and it is necessary to explain how this paper is \"technically\" different in terms of using temporal information. Those works include, but not limited to: a. Mask Propagation for Efficient Video Semantic Segmentation; b. Coarse-to-Fine Feature Mining for Video Semantic Segmentation; c. Semantic Segmentation on VSPW Dataset through Masked Video Consistency\n\nQUESTIONS:\nPlease address the weaknesses above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}