Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark

Authors: wei  ji
jingjing  li
wenbo  li
yilin  shen
li  cheng
hongxia  jin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-27 10:12:24

Abstract: Thanks to the rapid progress in RGB & thermal imaging, also known as multispectral imaging, the task of multispectral video semantic segmentation, or MVSS in short, has recently drawn significant attentions. Noticeably, it offers new opportunities in improving segmentation performance under unfavorable visual conditions such as poor light or overexposure. Unfortunately, there are currently very few datasets available, including for example MVSeg dataset that focuses purely toward eye-level view; and it features the sparse annotation nature due to the intensive demands of labeling process. To address these key challenges of the MVSS task, this paper presents two major contributions: the introduction of MVUAV, a new MVSS benchmark dataset, and the development of a dedicated semi-supervised MVSS baseline - SemiMV. Our MVUAV dataset is captured via Unmanned Aerial Vehicles (UAV), which offers a unique oblique bird’s-eye view complementary to the existing MVSS datasets; it also encompasses a broad range of day/night lighting conditions and over 30 semantic categories. In the meantime, to better leverage the sparse annotations and extra unlabeled RGB-Thermal videos, a semi-supervised learning baseline, SemiMV, is proposed to enforce consistency regularization through a dedicated Cross-collaborative Consistency Learning (C3L) module and a denoised temporal aggregation strategy. Comprehensive empirical evaluations on both MVSeg and MVUAV benchmark datasets have showcased the efficacy of our SemiMV baseline.

Main paper: 
 introduction: 
 Semantic segmentation is the process of categorizing each pixel in an image/video to a specific class label, which plays a vital role in visual scene understanding [1, 2] . Remarkable progresses has been made in the past several years, particularly in RGB-based semantic segmentation [3, 4, 5, 6, 7, 8, 9, 10, 11] . With the increasing accessibility of thermal sensors in capturing thermal radiation of objects with temperature above absolute zero, multispectral semantic segmentation (MSS) [12, 13, 14, 15] , where both RGB and infrared thermal cameras are engaged, starts to gain notable traction. An exemplar illustration is shown in Fig. 1 . This multispectral imaging setup excels specifically in scenes with unfavorable visual conditions, such as low-light or overexposure. On the other hand, the dynamic and ever-changing nature of real-world scenarios has propelled interests in video semantic segmentation (VSS) [16, 17, 18, 19, 20] with impressive segmentation performance. The task of multispectral (aka RGB-Thermal or RGB-T) video semantic segmentation (MVSS) [21] has emerged recently as a promising visual segmentation setup that has the best of both worlds.Nevertheless, as being a relatively new task, there are still a number of hurdles in MVSS research. The most prominent one is the lack of quality datasets. One major MVSS benchmark presently available is MVSeg [21] , a recently introduced dataset. It is worth mentioning that the multispectral videos in existing MVSS datasets, including MVSeg, are all taken from eye-level views, which clearly suggests the lack of data source diversity in MVSS. Even after securing good quality of RGB-T video footage, it is still a demanding exercise to make annotations. In practice, this is achieved by furnishing pixel-wise semantic labels on a selected subset of frames or key-frames 1 . Consequentially, there typically exists only sparse annotations in the MVSS datasets.This paper aims to address the above-mentioned challenges. Specifically, we introduce MVUAV, a new MVSS dataset containing a diverse range of RGB-T videos captured by Unmanned Aerial Vehicles (UAVs) from an oblique bird's-eye viewpoint. As depicted in Fig. 1 , this viewpoint provides a broader, holistic perspective free from the constraints of eye-level capture adopted by existing MVSS datasets such as MVSeg. In detail, our MVUAV dataset comprises 413 RGB-Thermal videos with 53,828 frame pairs in total. A subset of 2,183 key-frame pairs are meticulously annotated with pixel-wise semantic labels across 36 different classes of objects and stuffs, where the pixel annotation rate is 99.18%. Our dataset captures diverse real-world scenarios such as roads, streets, bridges, parks, seas, beaches, courts and schools; it also spans different lighting conditions from daytime to low-light and even pitch-dark scenarios. The MVUAV also presents some challenges, such as large scale variation, fine-grained scene parsing, moving object segmentation, and adverse illumination conditions, making it a valuable asset for evaluating various MVSS algorithms.Meanwhile, we seek to tackle the MVSS task from a relatively new semi-supervised perspective. As illustrated in Fig. 2 , our strategy utilizes a small number of sparsely labeled RGB-Thermal videos, alongside massive amount of unlabeled videos. In the closely-related domain of semi-supervised RGB-based semantic image segmentation, the idea of consistency regularization has played important role in the eventual performance. Its efficacy stems from perturbation-invariant training, by enforcing consistent predictions despite of varied perturbations presented in unlabeled RGB images at different processing levels: input [22] , feature [23, 24] , or network [25] . There motivates us to explore consistency regularization in RGB-Thermal videos. In fact, MVSS seems to be an ideal setting, since RGB and thermal images essentially capture the same scenes from different sensory perspectives, that offers innate input perturbations. Further, processing the multimodal data through two parallel networks with distinct parameters also leads to valuable feature-level and network-level perturbations.Building on these insights, we introduce SemiMV, a novel semi-supervised MVSS framework. At its core is a Cross-collaborative Consistency Learning (C3L) module, where the RGB and thermal streams mutually offer pseudo-supervisions to each other. A pixel-wise reliability map is also generated, based on the learned cross-modal consistency, to guide the temporal fusion process and mitigate potential noise.The main contributions of this paper are summarized as follows:• A new benchmark dataset, MVUAV, is introduced. The RGB-T videos in the new dataset present complementary perspectives to the the existing MVSS datasets such as MVSeg that are typically from eye-level views, by capturing from an oblique bird's-eye viewpoint. It also provides pixel-level dense annotations with a rich set of 36 visual semantic categories. • We propose a simple yet effective semi-supervised MVSS baseline, SemiMV. Our baseline is to our knowledge the first in employing the consistency regularization idea tailored for the semi-supervised MVSS task. Our experimental evaluation on the MVSeg and MVUAV benchmark datasets demonstrates the efficacy of the SemiMV baseline.2 Related Work 
semantic segmentation in diverse modalities: 
 Semantic segmentation, a crucial task in computer vision, has evolved significantly over recent decades. This evolution spans four primary data modalities: RGB images, multispectral images, RGB videos, and multispectral videos, each uniquely contributing to the field's advancement and expanding its practical applications [26, 27, 28, 29, 19, 30, 31, 32] .RGB semantic segmentation (RSS) has undergone remarkable advancements. FCN [33] , as a milestone work, uses a fully convolutional network for per-pixel prediction. Subsequent innovations, such as dilation convolution [34] , pyramid pooling [35] , and attention mechanisms [36, 37, 38] , have enriched segmentation by integrating contextual information and capturing global contexts. Recently, transformer-based methods [39, 40, 41, 42] have gained prominence with the advent of Visual Transformer [43] . For a detailed overview, readers are advised to refer to comprehensive survey papers [44] . Multispectral semantic segmentation (MSS) is an emergent field that aims to enhance model robustness in adverse lighting conditions by integrating RGB and thermal imagery [45] . The primary challenge in MSS lies in effectively merging RGB and thermal data. Various strategies have been developed, such as concatenation [46, 47] , element-wise summation [48, 49] , bridgingthen-fusing [14] , attention-weighted fusion [50] , explicit extract and fusion [13] , and cross-modal fusion [15] . Additionally, there is a related area of research focused on multimodal RGBD-based segmentation [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66] , which addresses some limitations of purely RGB-based segmentation. On the other hand, video semantic segmentation (VSS) is gaining traction with the growing importance of dynamic video processing. VSS models [18, 67, 16, 68, 69, 70] have focused on leveraging temporal contexts of video frames. Notably, some studies [71, 72, 73] utilize optical flow [74] to warp features from neighbouring frames for feature alignment and aggregation. Recently, attention-based methods [17, 75, 76, 77] have been proposed to selectively retrieve information from past frames for improving current frame segmentation, yielding promising results. Multispectral video semantic segmentation (MVSS) is a nascent area that combines the strengths of multispectral and temporal contexts. A pioneering study [21] introduces the largescale MVSeg dataset to benchmark this field, and proposes a baseline model for learning a joint representation from multispectral video input. Our paper contributes a new MVUAV dataset to enrich the dataset diversity in MVSS.Semantic segmentation has a major challenge in real-world scenarios where only limited pixellevel labels are available due to high expense of human labor. Recently, semi-supervised semantic segmentation has surged in popularity to utilize a plethora of unlabeled data.Semi-supervised RSS has gained extensive research interest by extending the powerful semi-supervised learning (SSL) techniques. Notably, consistency regularization and self-training have demonstrated great success. Consistency regularization enforces the consistency of the predictions with various perturbations, e.g., input perturbation via augmenting input images [22] , feature perturbation using multiple decoders [23] or feature dropout [24] , and network perturbation through different initialization [25] . Meanwhile, self-training methods [78, 79] generate pseudo segments for the unlabeled images and train subsequent models with both human-annotated and pseudo-labeled data.To leverage temporal video information, several semi-supervised VSS methods [78, 80] have been devised. For example, Zhuang et al. [80] propose inter-frame feature reconstruction to leverage the ground-truth labels to supervise the model training on unlabeled frames. Despite their effectiveness, these semi-supervised models are limited to single-mode RGB inputs. To our best knowledge, our work is the first to deal with semi-supervised MVSS problem with RGB-Thermal video inputs.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations are discussed in the supplementary materials. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
the mvuav dataset: 
 This section describes the construction of the MVUAV dataset and analyzes its statistical results. Dataset Collection. The main principle of data acquisition is to provide a comprehensive collection of calibrated RGB and thermal infrared video sequences from a new UAV perspective, and furnish high-quality semantic annotations. Toward this objective, we initially gather about 500 RGB-T video sequences from a latest aerial tracking dataset VTUAV [81] (altitudes ranging from 5-20m). The RGB-Thermal videos are captured at diverse environments such as parks, streets, and beaches, under various lighting conditions and seasons. To ensure the quality of our dataset, we remove unqualified videos or frames that are blurry, misaligned, or repetitive. After this selection process, the MVUAV dataset consists of 413 high-quality multispectral UAV videos, with 53,828 paired frames in total. Dataset Annotation. We then provide pixel-wise semantic labels to the multispectral UAV videos. We employ Labelme toolkit to annotate the multispectral UAV videos. The annotation process presents many challenges. First, the dataset has many complex scenes at adverse conditions, e.g., nighttime, darkness, overexposure, making it difficult to identify target objects from RGB images alone. Second, it is crucial to maintain annotation consistency across video frames, otherwise objects might be misidentified in different frames. Third, the UAV-view presents a more complex and crowded visual field compared to the eye-level view in MVSeg, making object identification and silhouette distinction more challenging. To overcome these challenges, we first overlay thermal heatmaps onto corresponding RGB images to visually aid annotators in identifying objects in complex scenes. Meanwhile, video frames from the same video are annotated by the same person to ensure temporal consistency. Three inspectors review and correct the annotations to make sure temporal consistency, with the assistance of video-level annotation visualization. Due to these challenges, the annotation and quality control process takes about 90 minutes per frame on average. We visualize some representative examples from the MVUAV dataset in Fig. 3 . Dataset Statistics. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The authors will make the newly proposed dataset and source code, along with detailed instructions, publicly available on our project website. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The details are given in Sec. 5.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
problem definition: 
 The training setting of the semi-supervised MVSS task is illustrated in Fig. 2 , where we have a small set of labeled multispectral videos with sparse annotations and a larger corpus of unlabeled multispectral videos. Following the common practice in VSS [75, 17] and MVSS [21] , we use video clips as input units. Each video clip contains a sequence of t frame pairs and only the final frame pair in the labeled video clip has semantic annotations. Formally, we denote a multispectral video clip asV = {(I R i , I T i )} t i=1, where (I R i , I T i ) represents the i-th frame pair with spatial resolution of H × W . The labeled set is denoted asD L = {(V L n , y n )} n L n=1, which comprises n L clips, and y n is the pixel-level semantic labels for the final frame pair of each clip, in a space of C classes. The unlabeled set is denoted asD U = {V U n } n U n=1 , including n U unlabeled multispectral video clips. Additionally, we use an evaluation set, D V = {(V V n , y n )} n V n=1 .The goal of Semi-MVSS is to develop a segmentation model that can effectively learn from both D L and D U , and exhibit robust generalization to D V .Fig. 4 depicts the overall architecture of our SemiMV. The network takes a multispectral video clip as input, which contains a Query frame pair at time step t, and M Memory pairs selected from past frames. In the semi-supervised MVSS setting, only the Query pairs from the labeled video set have ground-truth semantic annotations. Supervised Training. We first feed the RGB and Thermal pairs into two parallel segmentation networks (N et R and N et T ), e.g., DeepLabv3+, which generate initial segmentation predictions P R i andP T i (i ∈ [t -M, • • • , t]).For the labeled Query images, common supervised training is employed on the outputs of two networks using ground-truth segmentation maps, represented by:EQUATIONHere (I R t , I T t ) is a Query pair in the labeled set and y is the corresponding ground-truth map. l ce denotes the cross-entropy loss function. Cross-collaborative Consistency Learning. The key challenge in semi-supervised MVSS lies in how to mine effective supervision from unlabeled RGB-Thermal videos to complement label-guided training. In semi-supervised RSS, consistency regularization has achieved notable success, benefiting from perturbation-invariant training to enforce consistent predictions across various perturbations of unlabeled RGB images at different processing levels-input [22] , feature [23, 24] , or network [25] .This success motivates us to consider applying consistency regularization to RGB-thermal videos. To explore this, a simple yet effective Cross-collaborative Consistency Learning (C3L) module is devised, which leverages the unique properties of RGB-Thermal videos to perform perturbation-invariant training. Our intuition is that: RGB and thermal images inherently capture the same scene from distinct sensory perspectives, i.e., visible light and thermal infrared, which provide innate input perturbations; further, processing these multimodal data through two parallel networks with distinct parameters introduces valuable feature-level and network-level perturbations. With these insights, our C3L is devised to apply mutual pseudo supervision between RGB and thermal streams to effectively utilize unlabeled RGB-Thermal frame pairs. Specifically, we first compute a pair of one-hot pseudo-labels from the initial probabilistic segmentation predictions, P R i and P T i , using the argmax function:Y R i , Y T i = argmax(P R i ), argmax(P T i ).Then, the pseudo-labels are exploited to provide pseudo supervisions to the other stream, defined as:EQUATION! ! " ! ! # "#$ $ "#$ % % ! " % ! # ℛ ! Denoised Memory Bank C3L ' & " ' & # Denoised Memory Read Seg Head ' ! " ' ! # Memory frames (Unlabeled) Query frame (Un-/ labeled) RGB Seq. TIR Seq. collaboration f ! " f ! # ℒ !"# C3L f & " f & # F & " F & # Dual-C3L f ! " , f ! # , ℛ ! !∈ %&', •••,%&* p * * ∈ ,,- cross-modal F & " F & # % & '!()* % %Figure 4 : Overview of proposed method. For simplicity, the supervised losses are omitted. The C3L loss L c3l (Eq. 2) aims to learn from unlabeled RGB-Thermal pairs. The DMR is responsible for integrating temporal information from the denoised memory bank to update query features. A dual-C3L loss (Eq. 7) is further applied to regularize updated query features. Finally, a segmentation head predicts the final mask P f inal t . The dotted \\ means stop gradient.Ideally, the C3L loss is expected to enhance the robustness of the model by fostering crossmodal consistency between unlabeled RGB-Thermal pairs. However, our experiments indicate a decrease in segmentation performance with this approach, as discussed in Sec. 5.4. We conjecture this decrease is primarily due to the inherent limitations of particular sensors, where the pseudo label generated from either RGB or thermal images alone might be incomplete, causing confused training and error accumulation. To counteract this, we need to consider effective cross-modal collaboration in the C3L framework. In terms of the design of cross-modal collaboration, a lot of fusion strategies [48, 49, 50, 13, 15] have been proposed in the MSS field as discussed in related works. Therefore, our work does not claim a new fusion strategy, but introduces the insights to highlight the importance of cross-modal collaboration for semi-supervised consistency regularization in MVSS. In C3L, we employ the Cross-modal Fusion block (CMF)2 from [13] in our implementation and discuss other design choices in the supplements. By introducing cross-modal collaboration, the pseudo labels are refined by engaging the complementary information from the alternate stream, leading to significant improved performance. Denoised Memory Read. Next we consider how to integrate temporal information from past video frames. Among solutions in the related VSS and MVSS fields, attention-based memory read methods [21, 75, 17] have yield promising results, which selectively retrieve information from past (Memory) frames for improving current (Query) frame segmentation. In semi-supervised MVSS, due to the absence of ground-truth supervisions for past frames, Memory features are prone to be unreliable.To deal with this issue, we further introduce a reliability estimation strategy, which can be easily integrated into existing temporal aggregation modules [21, 75] to mitigate potential noise. Here we utilize prototypical memory read [21] for efficient temporal aggregation. Our intuition is that reliable RGB and thermal features tend to yield consistent predictions. Conversely, discrepancies in these predictions can, to a certain extent, suggest potential unreliability. To quantify this, we design a normalized bidirectional Kullback-Leibler (KL) divergence function to estimate the pixel-wise reliability map as:EQUATIONHere, the KL divergence function is performed pixel-wisely (we omit the pixel scripts for simplicity); N (•) is a min-max normalization function performed spatially to normalize divergence values to the range of 0 to 1. The final reliability map R i is with the dimension of H × W × 1.To efficiently store reliable memory features with minimal memory usage, we then establish a denoised prototype-based memory bank [21] . Concretely, for each memory feature f * i ∈ R H×W ×D extracted from N et R and N et T , where * ∈ {R, T } indicates the image modality and D is the channel dimension, we generate C denoised class-level prototype features by spatially aggregating denoised features belonging to each category:EQUATIONHere × means pixel-wise multiplication to down-weight unreliable memory features based on the reliability map R i . G is the aggregation operation, which spatially averages the features belonging to each class based on pseudo-label Y i . With this process, we obtain a condensed and denoised prototype-based memory bank {p * ∈ R M C×D } * ∈{R,T } . Subsequently, we use the attention mechanism to selectively retrieve relevant semantic information from the denoised memory bank, thereby refining query features. Taking RGB query feature f R t ∈ R H×W ×D as an example, the updated RGB query feature F R t is derived as follows:EQUATIONEQUATIONHere, fR t and p * indicate L 2 normalized features, ⊗ denotes matrix multiplication, [•, •, •] means feature concatenation, and ϕ(•) is a convolutional operation to adjust channel size.The denoised memory read module finally outputs two enhanced query features F R t , F T t ∈ R H×W ×D , which are enriched with useful denoised temporal contexts from unlabeled past frames. Dual-C3L. In order to make full use of the unlabeled data and to further regularize the memoryupdated features, we add C3L loss on the updated query features as well, called Dual-C3L loss:EQUATIONwhere { P R t , P T t } are updated predictions inferred from updated query features {F R t , F T t }, and { Ŷ R t , Ŷ T t } are corresponding pseudo labels. Accordingly, for the labeled query pairs, an additional supervision loss, Lsup , is also applied on the updated predictions, similar to Eq. 1.Final Prediction and Training Objective. To infer the final output, the updated query features {F R t , F T t } are concatenated together, followed by a 3 × 3 convolutional layer as segmentation head to predict the final mask P f inal t . A supervised cross-entropy loss is also applied to P f inal t , as:EQUATIONThe overall training objective of the proposed SemiMV framework is thus defined as:EQUATIONwhere λ is the trade-off weight to balance the supervised losses and pseudo losses from C3L. 
datasets and evaluation metric: 
 Our experiments are conducted on both MVSeg and MVUAV datasets. MVSeg [21] is a street scene dataset with 26 semantic classes. It consists of 452, 84, and 202 videos in its training, validation, and testing subsets, respectively. Annotations are provided for every 15 frames, resulting in 2,241 training, 378 validation, and 926 testing semantic masks. MVUAV provides bird's-eye view scenes with 36 semantic classes. The dataset splits are mentioned in Sec. 3. We follow the partition protocols of [25] and divide the whole training set via randomly sub-sampling 1/2, 1/4, 1/8, and 1/16 training videos as the labeled set, and treat the remaining videos as the unlabeled set. The numbers of annotated videos/frames for each training partition, denoted as (#videos, #frames), are included in Table 2 and Table 3 . By convention [21] , we adopt the mean Intersection over Union (mIoU) for evaluation.The model is implemented on the Pytorch and trained using two NVIDIA A100 GPUs. To be consistent with previous work [25, 21] , we adopt DeepLabv3+ [34] with ResNet50 as backbone, for both RGB and thermal streams. For the RGB stream, we initialize the network parameters using weights pretrained on ImageNet [92] . For the thermal stream, we randomly initialize the network parameters, and generate 3-channel thermal images as inputs by repeating the 1-channel thermal images. All training images are resized to 320 × 480. The model is optimized by Adam with batch size of 2, and the learning rate is 2e-4 which is annealed following the poly LR policy. The network converges around 200 epochs. We follow [21] to adopt three reference frames (M = 3) at a sample rate of 3 as Memory. λ is empirically set as 1. The network training involves two-stages: the first stage is backbone warming-up trained with only annotated query frames (50 epochs), and the second stage is main-training of SemiMV trained with all videos (150 epochs). During testing, the SemiMV processes each frame sequentially, inferring results in just 19.1 ms per frame.Since our SemiMV is the first work to address semi-supervised MVSS, to provide a reference level, we reimplement some related approaches in semi-supervised RSS (MT [22] , CCT [23] , CPS [25] , and UniMatch [24] ), semi-supervised VSS (IFR [80] ), VSS (Accel [77] ), and MVSS (MVNet [21] ) fields, using their official codes. These models use ResNet50 [93] as feature extractor. Results on MVSeg Dataset. Table 2 lists the segmentation results on the MVSeg dataset. We show the fully-supervised performance achieved by training only on the labeled data using different modalities, in the first row of each block. We observe that the semi-supervised models consistently surpass their supervised baselines across all data partitions, highlighting the significance of semi-supervised learning in semantic segmentation. In particular, our SemiMV improves the SupOnly (RGBT) baseline by large margins, i.e., +2.22%, +5.67%, +6.16%, +5.98%, under 1/16, 1/8, 1/4, 1/2 partition protocols, respectively. This demonstrates the effectiveness of our SemiMV in engaging unlabeled multispectral videos to enhance the generalization capabilities of segmentation models. Moreover, compared to MVNet [21] , which utilizes sparsely labeled videos alone, our SemiMV consistently performs better. This is attributed to the simultaneous usage of both unlabeled past frames and extensive unlabeled multispectral videos in our SemiMV framework. tral and temporal contexts can better identify objects in low-light places, for example, the pedestrians within the yellow boxes, while other methods struggle. Moreover, the results of our method are closer to the ground truths than MVNet, attributing to the combined benefits of semi-supervised learning and multispectral video data. More results are provided in the supplementary materials.In this section, we conduct ablation studies on the MVSeg dataset under 1/4 partition setup. As the proposed C3L and DMR are gradually incorporated, increased performance is consistently observed. In particular, C3L enhances the supervised baseline by a remarkable amount of +3.85% (36.88% → 40.73%), demonstrating its efficacy in harnessing unlabeled RGB-Thermal frame pairs. The addition of our DMR further boosts performance by +1.66% mIoU, benefiting from the exploitation of denoised and context-rich unlabeled past frames. Notably, our Dual-C3L loss added on updated query features further elevates the mIoU score to 43.04%, amplifying the combined strengths of our C3L and DMR modules in leveraging unlabeled multispectral videos for Semi-MVSS. Analysis on C3L. We then delve into the designs of C3L in Table 5 .To effectively utilize unlabeled RGB-Thermal frame pairs, our C3L leverages cross pseudo labels as supervision signals and introduces crossmodal collaboration to enable the generation of reliable pseudo labels. To verify their significance, we develop two variants, C3L - 1 w/o cross supervision and C3L - 2 w/o cross collaboration. In C3L - 1 , the mean-teacher output of each stream is used to generate pseudo label for itself without using cross supervisions. This leads to a reduction of 1.61% mIoU compared to the standard C3L. We conjecture this drop is due to the accumulation of self-errors, whereas our C3L, with its cross supervision, has the ability to mutually correct potential errors, thereby enhancing overall accuracy. Moreover, it is worth noting that the C3L - 2 variant, which applies cross pseudo supervision without the benefit of cross-modal collaboration, leads to a deterioration in result. This indicates that cross-modal collaboration is critical and indispensable for the effective functioning of our Semi-MVSS framework. Analysis on DMR. Table 6 presents our DMR-Proto and an alternative module -DMR-STM as in [75] . DMR-STM performs an all-to-all attention mechanism for matching between query and memory frames.Our results reveal that, 1) engaging temporal contexts from unlabeled past frames is indeed useful, as both DMR modules yield increased mIoU scores; 2) the proposed reliability estimation strategy effectively filters out unreliable features, enhancing the performance of both DMR variants. Considering computational efficiency, DMR-Proto is used for temporal integration in our SemiMV network.Additional model analyses are provided in the supplementary materials. 
conclusion: 
 This study introduces MVUAV, a new multispectral video semantic segmentation dataset obtained through UAVs from oblique bird's-eye viewpoint. The dataset, accompanied with precise semantic annotations, serves as a complementary resource to the MVSeg dataset, offering a broader perspective for evaluating MVSS models. Additionally, this paper pioneers the development of SemiMV, the first semi-supervised MVSS framework tailored to utilize both labeled and unlabeled multispectral videos effectively. Comprehensive empirical results affirm the effectiveness of our approach and highlight the promising potential of semi-supervised leaning in MVSS.Broader Impacts. The proposed MVUAV dataset offers significant value in enhancing the performance of semantic segmentation and propelling further research in the field of MVSS. We envisage that the most proximate impacts of this dataset will be positive, providing a valuable asset for researchers and developers in the field. Our semi-supervised MVSS method makes robust scene parsing available without intensive human annotation efforts, which saves a lot of costs. Note that the proposed MVUAV dataset is strictly prohibited from being used to identify or invade the privacy of any individual and is made available solely for academic purposes. In addition, we discuss potential limitations with some feasible solutions in the supplements. Reproducibility Statement. Our source code and dataset, along with easy-to-follow instructions, are publicly available on our project website. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have discussed the potential societal impacts in Sec. 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We prohibit people from using our MVUAV in any manner to identify or invade the privacy of any person. Additionally, our MVUAV dataset is made freely available solely for academic purposes.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the original papers for all assets used in our research, which are publicly available to the research community.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The newly proposed MVUAV dataset is detailed in Sec. 3. We will also release it along with user-friendly usage guidelines.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing experiments and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing experiments and research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We outline our main contributions at the end of Sec. 1. The claims made match the dataset description in Sec. 3, methodology in Sec. 4, and experimental results in Sec. 5. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We have reviewed the NeurIPS Code of Ethics and confirmed that we conform the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In video-based segmentation datasets (e.g., Cityscapes[1] and MVSeg[21]), it is a common practice to sparsely annotate video frames, given similar content of consecutive frames and substantial cost saving in the amount of annotation effort.CMF[13] is built on the gating mechanism that enables the model to emphasize useful features in one modality and compensate its missing information from the other. 


Decision: This paper proposes a new benchmark dataset, MVUAV, and a semi-supervised learning approach for multispectral video semantic segmentation (MVSS) for UAV images. The initial ratings are mixed. The authors provided a strong rebuttal, and all reviewers increased their rating after the discussion, reaching an all-positive consensus. The authors should improve the representation and explanations, clarify the motivation and significance of research on UAV-view images, address the ethical concerns, and incorporate the additional feedback provided during the rebuttal and discussion period.

Reviews: review: paper_summary: paper_summary: The paper addresses multispectral video semantic segmentation (MVSS) and proposes a new semi-supervised learning approach. It introduces the SemiMV framework, which utilizes a Cross-collaborative Consistency Learning (C3L) module and denoised temporal aggregation strategy. Additionally, the paper establishes the MVUAV benchmark dataset, captured by UAVs, offering unique bird’s-eye views and various semantic categories.
strength_weakness: strength_weakness: W1: Compared to UAVid, the resolution seems somewhat low, especially considering the UAV perspective where scenes are typically larger and require higher resolution for detailed analysis.

W2: The focus on UAV-captured data might limit the dataset’s applicability to ground-level or other perspectives, potentially reducing its versatility.
questions: questions: Q1: The paper should address whether the RGB and thermal (TIR) images in the MVUAV dataset are precisely aligned. Some existing RGB-T datasets [1]  have issues with modality misalignment, which serveral previous research [2][3] has tried to address. Therefore I wonder if this dataset exist same misalignment issue.

Q2: The author should clarify the relationship between MVUAV and MVSeg. Is MVUAV merely a complementary perspective, or does it offer other different scene?

Q3: Given the sparse annotations, can this dataset be used for general segmentation tasks? Or if it is only suitable for semi-supervised  methods?


[1] Multispectral Pedestrian Detection: Benchmark Dataset and Baseline
[2] Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection
[3] Attentive Alignment Network for Multispectral Pedestrian Detection
limitations: limitations: The authors acknowledge that existing MVSS datasets, including MVUAV, are small due to high labeling costs, and while semi-supervised methods help, they don't fully meet real-world demands. They also mention that the SemiMV baseline and MVUAV dataset, though promising, face challenges such as small targets and scale variation, necessitating further research and enhancements.
paper_summary: paper_summary: This paper introduces a new multi-spectral aerial-view semantic segmentation dataset called MVUAV, which consists of 413 video sequences and 53K frames with sparsely annotated pixel-level segmentation labels. To provide a way of using this sparsely annotated dataset, it also introduces a new semi-supervised semantic segmentation method that takes RGB and thermal image pairs as input. The newly designed key component in this method is cross-collaborative consistent learning, which uses the segmentation predictions of one modal network as pseudo-labels for training another modal network. This paper demonstrates the effectiveness of this method with better performance than other baselines on a multi-spectral semi-supervised semantic segmentation task on two benchmark datasets including MVUAV.
strength_weakness: strength_weakness: 1. Immature presentation
--

Overall, the presentation in the paper is not mature for publication.

a. Section 4.2 is not easy to follow due to missing description of some terminology.

- What is the memory feature  the memory feature $f^*_i$ and how is it obtained in eq 4?

- In eq 5, the dimensions of $f_t$ and $transpose(p)$ are H$\times$W$\times$1 and MC$\times$D, respectively. Here, the dimension of $f_t$ is guessed from eq 4, in which $f$ is element-wise multiplied with $\mathcal{R}$ of the dimension H$\times$W$\times$1. How can the matrix multiplication be applied to these two variables? 

b. Figures do not deliver any crucial and complicate information.

- Figure 2 is not necessary to understand what it conveys.

- Figure 4 is not referred in the contents, so it is unclear what this figure exists to support.


2. Insufficient contribution
--

Contribution is not sufficient to meet NeurIPS standards.

a. As this paper also mentioned, the proposed network does not have any novel component. C3L is very simple and not new as already used in various previous works (works in Ln 187). 

b. Dataset is somewhat novel as it contains aerial-view images. However, no experiment is provided to demonstrate why acquiring this characteristics in dataset is important. 

3. Insufficient experiments
--

a. All baseline methods compared in table 2 and 3 are developed not for the purpose of semi-supervised learning. Without comparing other semi-supervised learning method, it is difficult to figure out the effectiveness of the proposed method.

b. Ablation studies for several parameters which are crucial in the method are needed, e.g., M and lambda.
questions: questions: Please address weaknesses I pointed out.
limitations: limitations: .Limitation and potential societal impact are not mentioned in the main manuscript but included in supplementary material.
paper_summary: paper_summary: The paper proposes advancements in multispectral video semantic segmentation (MVSS) through two key contributions: the creation of a new benchmark dataset, MVUAV, captured via UAVs, and the development of SemiMV, a semi-supervised learning baseline designed to optimize sparse annotations using Cross-collaborative Consistency Learning (C3L).
strength_weakness: strength_weakness: 1. The paper presents a new dataset for semantic segmentation from a high-altitude perspective, yet lacks a detailed comparison with existing aerial-view datasets [1, 2, 3, 4] which is necessary to establish the dataset's relevance and uniqueness.

2. The paper fails to articulate the motivation and significance behind the introduction of the UAV-View dataset, leaving readers uncertain about the necessity and potential contributions of this new dataset to the field.

3. There is an absence of a thorough analysis of limitations and broader impacts, which is a concern as it may not meet submission requirements that typically expect such discussions to understand the full implications and potential drawbacks of the research.

4. The paper does not adequately address privacy concerns regarding the UAV-view multimodal dataset, such as the potential capture of private information like pedestrians and storefronts, and fails to clarify whether appropriate privacy measures are in place or if there is official and public approval for data collection in the relevant regions.

[1] Vision Meets Drones: A Challenge

[2] The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking

[3] Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmark

[4]  LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
questions: questions: Please see the weakness
limitations: limitations: Please see the weakness

Comments: comment: Dear Reviewer XBRU, we sincerely appreciate your feedback and are delighted to see the raised score. We will carefully revise our paper according to your suggestions. Our dataset, source code, and project website will also be released to the public. Thank you once again for your valuable time and effort in strengthening our paper.
As the authors' rebuttal addresses most of my questions, I am raising my initial rating.
Dear Reviewer w3ua, we appreciate your encouragement and positive feedback. We are pleased that your concerns have been addressed. We will improve our work according to your suggestions. In addition, our dataset, source code, and project website will be released publicly. Thank you once again for your valuable time and effort in enhancing our paper.
I appreciate your efforts in addressing my concerns in the rebuttal. Based on your responses, I am increasing my score to accept.
Dear Ethics Reviewer Xe58, thank you for conducting the ethics review. We sincerely appreciate your valuable time and consideration.
Dear Ethics Reviewer WRY9, we are sincerely grateful for the time and effort you invested in reviewing our paper and providing thoughtful feedback and recommendations for improving our paper.

Firstly, regarding the **data privacy** and **human rights** issues you raised, we will release the de-identified videos using defacing and storefront detection tools to mitigate potential privacy risk. We have also obtained official approvals from the original authors of VTUAV [56] to process and re-annotate their videos with semantic labels, thereby advancing the development of the MVSS field. Additionally, we will clearly state that the MVUAV dataset is strictly prohibited from being used to identify or invade the privacy of any individual and will be made available solely for academic purposes. We highly value the recommendation from both you and reviewer Mx7S regarding the privacy and human rights issues, as they significantly enhance the ethical considerations of our work. **These safeguards will be clearly outlined in the revised paper to ensure controlled use of the MVUAV dataset.**

Secondly, we appreciate your recommendation regarding **the comparison with existing aerial-view datasets.** We discussed the differences and advantages between existing aerial-view datasets [a, b, c, d] and our proposed MVUAV. For detailed information, please kindly refer to our response to Reviewer Mx7S's W1. Meanwhile, in the PDF file of general response, we also provided a table listing the detailed statistics of these datasets. **Following the suggestion, we will incorporate these discussions and detailed analyses in our paper.**

Thank you again for your valuable input. We hope our response addresses your concerns. Please let us know if there are any further questions.

---

[a] Vision Meets Drones: A Challenge   
[b] The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking   
[c] Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmark  
[d] LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
Dear Reviewer Mx7S, we are delighted to see that your questions and concerns have been addressed. We will carefully revise our paper, incorporating the necessary changes and references based on your suggestions. Additionally, our dataset, source code, and project website with easy-to-follow guidelines, will be made publicly available to the research community. Thank you once again for your valuable feedback and efforts in helping us improve our paper.
Thanks for addressing my concerns in the rebuttal. I raise my score to weak accept. Please make the necessary changes and references as noted in the rebuttal.