Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark
Authors: Wei Ji
Jingjing Li
Wenbo Li
Yilin Shen
Li cheng
Hongxia Jin
Venue: NeurIPS 2024 poster (2024)
Keywords: Computer Vision
Deep Learning
Semantic Segmentation
Primary area: machine_vision
TL;DR: In this work, we propose the SemiMV baseline, the first approach to apply semi-supervised learning specifically for multispectral video semantic segmentation, and introduce a new UAV-view dataset to advance research in this field.

Abstract: Thanks to the rapid progress in RGB & thermal imaging, also known as multispectral imaging, the task of multispectral video semantic segmentation, or MVSS in short, has recently drawn significant attentions. Noticeably, it offers new opportunities in improving segmentation performance under unfavorable visual conditions such as poor light or overexposure. Unfortunately, there are currently very few datasets available, including for example MVSeg dataset that focuses purely toward eye-level view; and it features the sparse annotation nature due to the intensive demands of labeling process. To address these key challenges of the MVSS task, this paper presents two major contributions: the introduction of MVUAV, a new MVSS benchmark dataset, and the development of a dedicated semi-supervised MVSS baseline - SemiMV. Our MVUAV dataset is captured via Unmanned Aerial Vehicles (UAV), which offers a unique oblique bird’s-eye view complementary to the existing MVSS datasets; it also encompasses a broad range of day/night lighting conditions and over 30 semantic categories. In the meantime, to better leverage the sparse annotations and extra unlabeled RGB-Thermal videos, a semi-supervised learning baseline, SemiMV, is proposed to enforce consistency regularization through a dedicated Cross-collaborative Consistency Learning (C3L) module and a denoised temporal aggregation strategy. Comprehensive empirical evaluations on both MVSeg and MVUAV benchmark datasets have showcased the efficacy of our SemiMV baseline.

Unleashing Multispectral Video's Potential in Semantic Segmentation: A Semi-supervised Viewpoint and New UAV-View Benchmark
Wei Ji " Li"!,
Wenbo Li Yilin Shen', Li Cheng', Hongxia Jin
Project website:
https: github. 10/MVUAV
Abstract
Thanks to the rapid progress in RGB & thermal imaging, also known as multispec- tral imaging, the task of multispectral video semantic segmentation, MVSS in short, has recently drawn significant attentions. Noticeably, it offers new opportu- nities in improving segmentation performance under unfavorable visual conditions such as poor light or overexposure. Unfortunately. there are currently very few datasets available, including for example MVSeg dataset that focuses purely toward eye-level view; and it features the sparse annotation nature due to the intensive demands of labeling process. To address these key challenges of the MVSS task, this paper presents two major contributions: the introduction of MVUAV, a new MVSS benchmark dataset, and the development of dedicated remi-supervised MVSS baseline SemiMV. Our MVUAV dataset is captured via Unmanned Aerial Vehicles (UAV), which offers a unique oblique bird's-eye view complementary to the existing MVSS datasets; it also encompasses broad range of day/night lighting conditions and over 30 semantic categories. In the meantime, to bet- ter leverage the sparse annotations and extra unlabeled semi-supervised learning baseline, SemiMV, is proposed to enforce consistency regularization through dedicated cross-collaborative Consistency Learning (C3L) module and a denoised temporal aggregation strategy. Comprchensive empirical evaluations both MVSeg and MVUAV benchmark datasets have showcased the efficacy of our SemiMV baseline.
vidcos,
1
Introduction
Semantic segmentation is the process of categorizing each pixel in an image/video to specific class label, which plays vital role visual scene understanding made in the past several years, particularly in RGB- based semantic segmentation [3 5 6 g [0
Remarkable progresses has been
II. With the increasing accessibility of thermal sensors in capturing thermal radiation of objects with temperature above absolute zero, multispectral semantic segmentation (MSS) 22
13 [4. [5], where both RGB and infrared thermal cameras are engaged, starts to gain notable traction. An exemplar illustration is shown in Fig. D This multispectral imaging setup excels specifically in scenes with unfavorable visual conditions, such as low-light or overexposure. On the other hand, the dynamic and ever-changing nature of real-worrd world scenarios has propelled interests in video semantic segmentation (VSS) [6 [7 [8]
19 20 with impressive segmentation performance. The task of multispectral RGB-T) video semantic segmentation (MVSS) has emerged recently promising visual segmentation setup that has the best of both worlds.
(aka EGBB-Thrmm
Nevertheless, as being relatively new task, there are still number of hurdles in MVSS research. The most prominent one is the lack quality datasets. One major MVSS benchmark presently
University of Alberta Yaae University "Samsung Al 'Equal contribution. Corresponding author: Jingjing Li <jingjinl @ @ualberta.ca>.
View (work done here). 'Intern mentor.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

MVUAV
bird's eye view
long
(b) MVSeg Dataset
eye-lovel
(a) Data Acquisiion
(c) MVUAV Dataset
Figure I: (a) Viewpoint diversity of the existing MVSeg dataset [21 and the new MVUAV dataset. (b) & (c) Representative samples from the MVSeg & MVUAV datasets, where RGB videos, thermal videos, and the corresponding semantic annotations are visualized.
available is MVSeg [11]. recently introduced dataset. is worth mentioning that the multispectral videos existing MVSS datasets, including MVSeg, are all taken from eye-level views, which clearly suggests the lack of data source diversity in MVSS. Even after securing good quality of RGB-T video footage, it is still a demanding exercise to make annotations. In practice, this is achieved by furmishing pixel-wise semantic labels a selected subset of frames key-frames Consequentially, there typically exists only sparse annotations in the MVSS datasets.
This paper aims to address the above-mentioned challenges. Specifically, introduce MVUAV, a new MVSS dataset containing a diverse range of RGB-T videos captured by Unmanned Aerial Vehicles (UAVs) from oblique bird's-eye viewpoint. As depicted in Fig. [ this viewpoint provides a broader, holistic perspective free from the constraints of eye-level capture adopted by existing MVSS datasets such as MVSeg, In detail, our MVUAV dataset comprises 413 RGB-Thermal videos with 53,828 frame pairs in total. A subset of 2, 183 key-frame pairs are meticulously annotated with pixel-wise semantic labels across 36 different classes of objects and stuffs, where the pixel annotation rate 19% Our dataset captures diverse real-world scenarios such as roads, streets, bridges, parks, seas, beaches, courts and schools; it also spans different lighting conditions from daytime to low-light and even pitch-dark scenarios. The MVUAV also presents some challenges, such as large scale variation, fine-grained scene parsing, moving object segmentation, and adverse illumination conditions, making it a valuable asset for evaluating various MVSS algorithms.
Meanwhile, seek to tackle the MVSS task from relatively new cemi-super perspective. As illustrated in Fig. our strategy utilizes a small number of sparsely labeled -GG-Thermal videos, alongside massive amount of unlabeled videos. In the deeated domain of semi-supervised RGB-based semantic image segmentation, the idea consistency regularization has played important role in the eventual performance. Its efficacy stems from cntturbatioo-invarian training, by enforcing consistent predictions despite of varied perturbations presented in unlabeled RGB images at different to explore consistency regularization in RGB-Therme videos. In fact, MVSS seems to be an ideal setting, since RGB and thermal images essentially capture the same scenes from different sensory perspectives, that offers innate input perturbations. Further, processing the multimodal data through two parallel networks with distinct parameters also leads to valuable feature-level and network-leve perturbations. Building on these insights, we introduce SemiMV, a novel semi-supervisee MVSS framework. At its core
processing levels: input (22) feature [23] (24,, network (25]. There motivates
collaborllee Consistency Learning (C3L) module, where the RGB and thermal streams mutually offer pscudo-supervisions to each other. A pixel-wise reliability map is also generated, based on the learned cross-moda consistency, to guide the temporal fusion process and mitigate potential noise.
The main contributions of this paper are summarized follows:
A new benchmark dataset, MVUAV, is introduced. The RGB-T videos in the new dataset present complementary perspectives to the the existing MVSS datasets such as MVSeg that are typically from eye-level views, by capturing from oblique bird's-eye viewpoint. It also provides pixel-level dense annotations with a rich set of 36 visual semantic categories.
In video-basee segmentation datasets (e.g., Cityscapes i and MVSeg 21)). it is a common practice to sparsely annotate video frames, given similar content of consecutive frames and substantial cost saving in the amount of annotation effort.

---

with annotation
without annotation
Sampled Labeled RGBT Videos
Remaining Unlabeled RGBT Videos
Figure 2: Illustration of training examples in semi-supervised MVSS setting, where a limited amount
of sparsely labeled RGB-Thermal (RGBT) videos and massive unlabeled ones are utilized.
We propose a simple yet effective semi-supervised MVSS baseline, SemiMV. Our baseline is knowledge the first in employing the consistency regularization idea tailored for the semi-supervised MVSS task. Our experimental evaluation on the MVSeg and MVUAV benchmark datasets demonstrates the efficacy of the SemiMV baseline.
2 Related Work
2.1 Semantic Segmentation in Diverse Modalities
Semantic segmentation, a crucial task in computer vision, has evolved significantly over recent decades. This evolution spans four primary data modalities: RGB images, multispectral images. RGB videos, and multispectral videos, each uniguely contributing to the field's advancement and
expanding its practical applications [26 27
28
29
19
31 32).
RGB semantic segmentation (RSS) has undergone remarkable advancements. FCN 33 as a milestone work, uses fully convolutional network for per-pixel prediction. Subsequent innovations, such as dilation convolution [4 pyramid pooling [35), and attention mechanisms [36 37 38 have
enriched segmentation by integrating contextual information and capturing global contexts. Recently, transformer-based methods [39 40
41 42 have gained prominence with the advent of Visual Transformer 43 For detailed overview, readers are advised to refer to comprehensive survey papers 44 Multispectral semantic segmentation (MSS) is an emergent field that aims to enhance model robustness in adverse lighting conditions by integrating RGB and thermal imagery [45). The primary challenge in MSS lies in effectively merging RGB and thermal data. Various strategies
have been developed, such concatenation 46
47 element-wise summation 48 49 bridging-
then-fusing [14), attention-weighhe
fusion [0 explicit extract and fusion [13, and cross-modal
fusion [5]. Additionally, there related area research focused on multimodal RGBD-based segmentation [51] [22 53 54
65 66). which addresses some the other hand, video semantic segmentation (VSS) is gaining traction with the growing importance of dynamic video processing. VSS models [18] have focused on leveraging temporal contexts of video frames. Notably, some 73 utilize optical flow [74 to warp features from neighbouring frames for feature
55
56,
57
58
59
60,
61
62
63
64
limitations of purely RGB-based segmentation.
67
16
68
69
70
studies 71
72
alignment and aggregation. Recently, diention-based methods [17 75 76 77] have been proposed to selectively retrieve information from past frames for improving current frame segmentation, yielding promising results. Multispectral video semantic segmentation (MVSS) a nascent area that combines the strengths of multispectral and temporal contexts. A pioneering study (1] introduces the large- scale MVSeg dataset to benchmark this field, and proposes a baseline model for learning a joint representation from multispectral video input. Our paper contributes new MVUAV dataset enrich the dataset diversity in MVSS.
2.2 Semi-supervised Learning in Segmentation
Semantic segmentation has major challenge in real-world scenarios where only limited pixel- level labels are available due to high expense of human labor. Recently, semi-supervised semantic segmentation has surged in popularity to utilize plethora of unlabeled data.
Sermi-sueervised RSS has gained extensive research interest by extending the powerful semi-supervised learning (SSL) techniques. Notably, consistency regularization and self-training have demonstrated great success. Consistency regularization enforces the consistency of the predictions with vari- ous perturbations, e.g., input perturbation via augmenting input images [22]. feature perturbation
3

---

CT
9949
Elrvniste
Nighttiee
Blyykb
MVUAY
Beach
hhitic light
Trafi
HT
1000
Nighttime
Baakiing
Oeiona
Mesdgg
(a) Taxonomic system
(b) Dataset visual examples
Figure 3: Illustrations of the proposed MVUAV dataset. (a) Taxonomic system and its histogram
distribution showing the number of annotated frames across different categories. (b) Examples of
multispectral UAV videos and corresponding annotations in both daytime and nighttime scenarios.
using multiple decoders [23 or feature dropout 24 and network perturbation through different
initialization [25]. Meanwhile, self-training methods 778 79] generate pseudo segments for the unlabeled images and train subsequent models with both human- -anaann-d and pseudo-labeled data. To leverage temporal video information, several semi-supervised VSS methods (78] 80 have been devised. For example, Zhuang et al. [80] propose inter-frame feature reconstruction to leverage the ground-truth labels to supervise the model training on unlabeled frames. Despite their effectiveness, these semi-supervised models are limited to single-mode RGB inputs. To our best knowledge, our work is the first to deal with sumi-super MVSS problem with GGB-Thermal video inputs.
3 The MVUAV Dataset
This section describes the construction of the MVUAV dataset and analyzes its statistical results. Dataset Collection. The main principle of data acquisition is to provide a comprehensive collection of calibrated RGB and thermal infrared video sequences from
UAV perspective, and furnish high-quality semantic annotations. Toward this objective, we initially gather about 500 RGB-T video sequences from latest aerial tracking dataset VTUAV [81] (altitudes ranging from 5-20m). The RGB-Thermal videos are captured at diverse environments such as parks, streets, and beaches, under various lighting conditions and seasons. To ensure the quality of our dataset, we remove unqualified videos or frames that are blurry, misaligned, or repetitive. After this selection process, the MVUAV dataset consists of 413 high-quality multispectral UAV videos, with 53,828 paired frames total.
Dataset Annotation. We then provide pixel-wise semantic labels to the multispectral UAV videos. We employ Labelme toolkit annotate the multispectral UAV videos. The annotation process presents many challenges. First, the dataset has many complex scenes at adverse conditions, e-g., nighttime, darkness, overexposure, making it difficult to identify target objects from RGB images alone. Second, it is crucial to maintain annotation consistency across video frames, otherwise objects might be misidentified in different frames. Third, the UAV-view presents more complex and crowded visual field compared to the eye-level view in MVSeg. making object identification and silhouette distinction more challenging. To overcome these challenges, we first overlay thermal heatmaps onto corresponding RGB images to visually aid annotators in identifying objects in complex scenes. Meanwhile, video frames from the same video are annotated by the same person to ensure temporal consistency. Three inspectors review and correct the annotations to make sure temporal consistency, with the assistance of video-level annotation visualization. Due to these challenges, the annotation and quality control process takes about 90 minutes per frame on average. We visualize some representative examples from the MVUAV dataset in Fig. 3
Dataset Statistics. Table [ outlines some critical attributes of the MVUAV dataset and related semantic segmentation datasets with different modalities. Our MVUAV dataset comprises 413 RGB- Thermal videos at a frame rate of 25 fps, including 54k image pairs in total and 2, 183 annotated image pairs. The new MVUAV dataset can act as a valuable asset to complement the existing MVSeg dataset for more thorough evaluations of various MVSS models. Additionally, the MVUAV dataset

---

Table 1: Statistics of various semantic segmentation datasets in diverse modalities. 'Surv.' "#Cls' and
'Anno.' are the shorthand for surveillance, the number of classes and annotation density, respectively.
Dataset
Year
Color Infrared Video
UAV
Capture
#Vids( rames)
#GTs
Resolution
#CIs %cnno.
Cityscapes
2016
(150k)
5,000
2048 1024
97.10%
Cam Vid 82
2009
V
Car 5 (40k)
701
960x
32
96.20%
UAVid 83
2020
/
X
/
Drone
42 (38k)
420
3840x2160
8
82.69%
SODA
2020
7
Pedestrian
2,168
640
21
79.73%
SCUT-Seg 85
2021
x
/
X
X
Car
2,010
x70x576
56,50% 10
MFNet 46
2017
X
X
Car
1,569
x40)
7.86%
PST900 47
2020
/
Robot
x
3.02%
SemanticRT 45
2023
X
X
Surv.
11.371
1280 1024
13
21.27%
PMB 86
2023
X
Car
1,500)
x00
15
98, 16%
CART [8
2024
Drone
2,282
x60x600
99.98%
MVSeg [21
2023
V
Car/Surv.
738 (533)
3,545
480x640
26
98.96%
MVUAV
Drone 413 (54k)
2,183
1920 1080
36
99.18%
features high-resolution imagery (1920x 1080), which is helpful for fine-grained scene parsing. This dataset also provides detailed annotations for a rich set of semantic categories, covering 8 root categories and 36 sub-classes (including background) as shown in Fig. rate of 99.18%. This can facilitate detailed and comprehensive scene understanding. Additional dataset details and discussions with related UAV-view datasets 88 supplementary material.
3 at a high pixel annotation
90 197] are provided in the
Dataset Splits. The dataset is partitioned into training. validation, and test sets, which contain 275, 35, and 103 videos respectively, with 1, 1,464, 171, and 548 annotated masks. One frame is annotated for every 25 frames. Additionally, the test set data involves both daytime and nighttime scenes, consisting of 72 videos with 351 annotated frames and 31 videos with 197 annotated frames, respectively.
Methodology
4.1 Problem Definition
The training setting of the semi-supervised MVSS task is illustrated in Fig. 2 where we have small set of labeled multispectral videos with sparse annotations and a larger corpus of unlabeled multispectral videos. Following the common practice in VSS (75 17 and MVSS (21. we use video clips as input units. Each video clip contains a sequence of t frame pairs and only the final frame pair in the labeled video clip has semantic annotations. Formally,
denote multispectral video clip as V = {(1,7 I.") I where (4, IT") represents the i-th frame pair with spatial resolution of H x W. The labeled set is denoted as D = {(V), yn }n)) which comprises nt clips, and yn is the pixel-level semantic labels for the final frame pair of each clip. in a space of C' classes. The unlabeled set is denoted as D ( jn iincluding ny unlabeled multispectral video clips. Additionally, we use an evaluation set, DV = {(V), ) The goal of Semi-MVSS is to develop a segmentation model that can effectively learn from both D.. and DV, and exhibit robust generalization to DV
ja
4.2 Proposed SemiMV Framework
Fig. 4 depicts the overall architecture of our SemiMV. The network takes multispectral video clip as input, which contains a Query frame pair at time step f, and M Memory pairs selected from past frames. In the semi-supervised MVSS setting, only the Query pairs from the labeled video set have ground-truth semantic annotations.
Supervised Training. We first feed the RGB and Thermal pairs into two parallel segmentation networks (Net and Nel"),, e.g., DeepLabv3+, which generate initial segmentation predictions P. For the labeled Query images, common supervised training is employed on the outputs of two networks using ground-truth segmentation maps, represented by:
and PT E [t- M, ..)....).. ,t)).
Sou B ..ee (P. v) + lea (Pf .)))
Here (7,4, is Query pair in the labeled set and is the corresponding ground-truth map. lee denotes the cross-entropy loss function.
Cross-collaborative Consistency Learning. The key challenge in supervise MVSS lies in how to mine effective supervision from unlabeled RGB-Thermal videos to complement label-guided training. In semi-supervised RSS, consistency regularization has achieved notable success, benefiting from perturbation-invanl unlabeled RGB images at different processing levels-input input [22), feature [23] 24], network [25].
training to enforce consistent predictions across various perturbations of
5

---

This success motivates us to consider applying consistency regularization to RGB-thermal videos. To explore this, simple yet effective cross-collaborative Consistency Learning (C3L) module devised, which leverages the unique properties of RGB-Therma videos to perform perturbation-invariat training. Our intuition is that: RGB and thermal images inherently capture the same scene from distinct sensory perspectives, iee, visible light and thermal infrared, which provide innate input perturbations; further, processing these multimodal data through two parallel networks with distinct parameters introduces valuable feature-level and network-level perturbations. With these insights, our C3L is devised to apply mutual pseudo supervision between RGB and thermal streams to effectively utilize unlabeled KGB-Thermal frame pairs.
Specifically, we first compute pair of one-hot labuls from the initial probabilistic segmen- tation predictions, Pft and Pi", using the argmax function: Y," Y argmax( (, argmax (PT)). Then, the pseudo-labels are exploited to provide pseudo supervisions to the other stream, defined as:
UD L = (1 (1 U (P. Y?) + Lee (P. ))
Ideally, the C3L loss is expected to enhance the robustness of the model by fostering cross- modal consistency between unlabeled RGB- Thermal pairs. However, our experiments indi- cate decrease in segmentation performance with this approach, discussed in Sec. [5.4] conjecture this decrease is primarily due to the inherent limitations of particular sen- sors, where the pscudo label generated from either RGB thermal images alone might be incomplete, causing confused training and er- accumulation. To counteract this, need to consider effective mooss-modal collaboration in the C3L framework. In terms of the design of cross-modal collaboration, a lot of fusion have been pro- posed in the MSS field as discussed in related works. Therefore, our work does not claim new fusion strategy, but introduces the insights highlight the importance of cross-modal col- laboration for semi-supervised consistency reg- ularization in MVSS. C3L, we employ the Cross-modal Fusion block (CMF) from [13] our implementation and discuss other design choices in the supplements. By introducing cross-modal collaboration, the pseudo labels are refined by engaging the complementary information from the alternate stream, leading to significant improved performance.
Memory frames (Unlabeled)
Query frame (Un-/labeled)
1
RGB Seq.
R
TIR Seq.
i
Ner
moss-modal
Net
C3L
collaboration
P
R
P
Dencised
Memory Read
F
Feeat
y
YIT
C3L
Dual-C3L
FR
P
Denoised
Seg Head Ppriina FIRal
strategies [48 49
50
13 15)
Memory Bank
(P' (P'EECERT
Figure 4: Overview of proposed method. For sim-
plicity, the supervised losses omitted. The C3L loss Lest (Eq. (2) aims to learn from unlabeled RGB- Thermal pairs. The DMR is responsible for integrat- ing temporal information from the denoised memory bank update query features. A dual-C3L loss (Eq. 7 is further applied to regularize updated query fea- tures. Finally, segmentation head predicts the final mask Pf P.fimel The dotted! \\ means stop gradient.
Denoised Memory Read. Next we consider how to integrate temporal information from past video frames. Among solutions the related VSS and MVSS fields, attention- based memory read methods 21
75 17 have yield promising results, which selectively retrieve information from past (Memory) frames for improving current (Query) frame segmentation. In semi-supervised MVSS, due to the absence of ground-truth supervisions for past frames, Memory features are prone to be unreliable.
To deal with this issue, we further introduce reliability estimation strategy, which can be easily integrated into existing temporal aggregation modules [2] 775 to mitigate potential noise. Here we utilize prototypical memory read [2 for efficient temporal aggregation. Our intuition that reliable RGB and thermal features tend to yield consistent predictions. Conversely, discrepancies in these predictions can, to a certain extent, suggest potential unreliability. To quantify this, we design a normalized bidirectional Hellback-Leibbee (KL) divergence function to estimate the pixel-wise
²CMF 3 is built on the gating mechanism that enables the model to emphasize useful features in one modality and compensate its missing information from the other.

---

reliability map as:
PR(O) R 1 (V 1 (c) log PF
PT(C) P.T(C)
I (c) CEC
(3)
-EC
Here, the KL. divergence function performed pixel-wisely (we omit the pixel scripts for simplicity); is (-()) range of O to 1. The final reliability map R, is with the dimension H W X
min-max normalization function performed spatially to normalize divergence values to the
To efficiently store reliable memory features with minimal memory usage, we then establish a denoised prototype-based memory bank (2T]. Concretely, for each memory feature f E R D extracted from Net and Net". where * (R.T) indicates the image modality and D is the channel dimension, we generate C denoised class-level prototype features by spatially aggregating denoised features belonging to each category:
(4) Here x means pixel-wise multiplication to down-weight unreliable memory features based on the reliability map R2 G is the aggregation operation, which spatially averages the features belonging to each class based pseudo-label Y, With this process, obtain condensed and denoised prototype- prototype-based based memory bank {p" E RMMCDD ) (R,TT.
p = g(f x Y,) E RCDD
Subsequently, we use the attention mechanism to selectively retrieve relevant semantic information from the denoised memory bank, thereby refining query features. Taking RGB query feature r. R H x an example, the updated RGB query feature F. is derived follows:
w Softmax (T," 2 transpose [p" * E (R,T),
(5)
F = " w I.)
(6)
Here, I. and p' indicate L2 normalized features, @ denotes matrix multiplication, [s.) means feature concatenation, and ((-) is convolutional operation to adjust channel size.
The denoised memory read module finally outputs two enhanced query features F., FT E ER which are enriched with useful denoised temporal contexts from unlabeled past frames.
Dual-C3L. In order to make full use of the unlabeled data and to further regularize the memory- updated features, add C3L loss on the updated query features as well, called Dual-C3L loss:
Lest Buf .If' JED" U' (PoR Y?)) PPT )),))
where [P.R, }.T are updated predictions inferred from updated query features {F F?) and (Ý;R, 877) are corresponding pseudo labels. Accordingly, for the labeled query pairs, an additional supervision loss, Lsup. is also applied on the updated predictions, similar Eq. Final Prediction and Training Objective. To infer the final output, the updated query features LFF,F") are concatenated together, followed by a x convolutional layer as segmentation head to predict the final mask pf finaa A supervised cross- cross-entropy loss is also applied to P pfinal as:
Cfinal B ae) (P. (P.final y).
(8)
The overall training objective the proposed SemiMV framework is thus defined as:
Liotal = Seup + Loup + sup + X(Cear + Lea))
where X is the trade-off weight to balance the supervised losses and pseudo losses from C3L
5
Experiments
5.1 Datasets and Evaluation Metric
Our experiments are conducted on both MVSeg and MVUAV datasets. MVSeg (2 is a street scene dataset with 26 semantic classes. It consists of 452, 84, and 202 videos in its training, validation, and testing subsets, respectively. Annotations are provided for every 15 frames, resulting in 2,241 training. 378 validation, and 926 testing semantic masks. MVUAV provides sred's view scenes with 36 semantic classes. The dataset splits are mentioned in Sec. We follow the partition protocols of (25) and divide the whole training set via randomly sub-sampling 1/2, 1/4, 1/8, and 1/16 training videos as the labeled set, and treat the remaining videos as the unlabeled set. The numbers of annotated videos/frames for each training partition, denoted as (#vdd,ss, # #frames), are included in Table 2 and Table 3 By convention (1I. we adopt the mean Intersection over Union (mloU) for evaluation.
7

---

5.2 Implementation Details
The model is implemented on the Pytorch and trained using two NVIDIA A10 GPÜs. To be consistent with previous work [25 21]. adopt DeepLabv3+ (34] with ResNet50 as backbone, for both RGB and thermal streams. For the RGB stream, we initialize the network parameters using weights pretrained ImageNet [92]. For the thermal stream, parameters, and generate 3-channel thermal images as inputs by repeating the 1-channel thermal images. All training images are resized to 320 x 480. The model is optimized by Adam with batch size of 2, and the learning rate is 2e-4 which is annealed following the poly LR policy. The network converges around 200 epochs. We follow [2] to adopt three reference frames (M = 3) at sample rate of as Memory. À is empirically set as 1. The network training involves two-stages: the first stage is backbone warming-up up trained with only annotated query frames (50 epochs), and the second stage is main-training of SemiMV trained with all videos (150 epochs). During testing, the SemiMV processes each frame sequentially, inferring results in just 19.1 ms per frame.
randomly initialize the network
5.3 Comparison with the State-of-the-Arts
Since our SemiMV is the first work to address semi-supervised MVSS, to provide reference level, we reimplement some related approaches in semii-sueeriised RSS (MT [22), CCT [23] CPS 25
and UniMatch (24) ) semi-supervised VSS (IFR [80), VSS (Accel (77). and MVSS (MVNet 21
fields, using their official codes. These models use ResNet50 [93] as feature extractor.
Results on MVSeg Dataset. Ta-
Table 2:
Quantitative evaluation
the MVSeg dataset.
ble 2 lists the segmentation results on the MVSeg dataset. We show the fully-supervised performance achieved by training only on the la- beled data using different modalities, the first row of each block. We ob- serve that the semi- supervised mod- els consistently surpass their super- vised baselines across all data parti- tions, highlighting the significance of semi- superviee learning in se- mantic segmentation. In particular,
SupOnly stands for the model trained on the labeled data.
1/16
1/8
1/4
1/2
Method
(26,140)
(54,282)
(111,561)
(228,1119)
SupOnly (RGB)
21.95
27.09
35.79
42.37
MT [22
23.39
29.45
38.75
44.51
CCT 23
23.81
29.66
39.04
44.89
CPS 25
23.88
30.05
39.27
45.34
UniMatch 24
24.73
30.47
39.39
45.42
Accel 777 (Video)
23.16
28.41
37.31
43.75
IFR 80
24.79
30.97
40.69
46.21
SupOnly (RGBT)
23.26
28.45
36.88
43.75
MVNet 21
24.70
30.32
39.89
46.08
SemiMV improves the SupOnly (RGBT) baseline by large margins, i.e., +2.22%, +5.67%, 16.16% +5.98%, under 1/16, 1/8, 1/4, 1/2 partition protocols, respectively. This demonstrates the effectiveness of our SemiMV in engaging unlabeled multispectral videos to enhance the generalization capabilities of segmentation models. Moreover, compared to MVNet (2], which utilizes sparsely labeled videos alone, our SemiMV consistently performs better. This is at- tributed to the simultaneous usage of both unlabeled past frames and extensive unlabeled multispectral videos in our SemiMV framework.
SemiMV (Ours)
25.48
34.12
43.04
49.73
Results MYUAV Dataset. Ta-
Table 3: Quantitative evaluation on the MVUAV dataset.
ble 3 reports the comparison re-
1/16
1/8
1/4
1/2
sults on the MVUAV dataset. Ben- efiting from the integration of rich multispectral video information and the powerful capability of semi- supervised learning utilize unla- beled data, our SemiMV achieves the highest performance. The im- provement of our SemiMV over the supervised baseline SupOnly (RGBT) are +3.82%, +5.16%, +5,21%, +4.63% under 1/16, 1/8, 1/4, 1/2 partition protocols, respectively. Overall, our architecture adapt to various settings and scenarios by achieving superior performance on both MVSeg and MVUAV datasets. This showcases the robustness of our SemiMV framework.
Method
(23,91)
(40,184)
(70,365)
(141,732)
SupOnly (RGB)
10.09
13.47
20.07
26.25
MT 22
11.33
15.89
23.02
27.83
CCT 23
11.75
16.11
23.72
28.71
CPS 25
12.55
16.70
24.01
29.09
UniMatch
22
13.36
17.21
24.10
29.21
Accel [77 (Video)
11.23
14.69
21.45
27.70
IFR 80)
13.11
17.03
24.91
29.87
SupOnly (RGBT)
11.28
14.88
21.31
27.60
MVNet 21
13.07
16.86
23.36
29.77
SemiMV (Ours)
15.10
20.04
26.52
32,23
Visual Comparison. Fig. illustrates the qualitative results of different methods on the MVSeg dataset under 1/4 partition protocol. As seen, the MVNet and our SemiMV reconciling both multispec-

---

RGB Frames TIR Frames
SupOnly
UniMatch
IFR
MVNet
Ours
GTs
Figure 5: Qualitative results on MVSeg dataset. We highlight the details with the yellow boxes.
tral and temporal contexts can better identify objects in low-light places, for example, the pedestrians within the yellow boxes, while other methods struggle. Moreover, the results of our method are closer to the ground truths than MVNet, attributing to the combined benefits cemi-supervised learning and multispectral video data. More results are provided in the supplementary materials.
5.4 Ablation Studies
In this section, we conduct ablation studies on the MVSeg dataset under 1/4 partition setup.
Effect of each component. In Ta-
Table 4: Ablation study of the proposed SemiMV framework.
ble 4 we evaluate the performance improvements achieved by systemati- cally integrating key components into our framework. Initially, we compare two supervised baselines: RGB vs. RGB-Thermal, each trained with la-
Information
Methods
RGB Thermal Labeled Unlabeled Video
mloU
RGB
35.79
RGB-Thermal
/
36.88
+C3L
/
40.73
+DMR
/
42.39
+Dual-C3L
/
43.04
beled RGB frames or RGB-Thermal frame pairs, respectively. It is observed that incorporating thermal infrared branch brings notable performance gain of 1.09%, verifying the value of multispectral information in semantic segmentation. As the proposed C3L and DMR are gradually incorporated, increased performance is consistently observed. particular, C3L enhances the supervised baseline by a remarkable amount of +3.85 % (36.88% - 40.73%), demonstrating its efficacy in harnessing unlabeled RGB-Thermal frame pairs. The addition of our DMR further boosts performance by + +1.66% mIoU, benefiting from the ex- ploitation of denoised and context-rich unlabeled past frames. Notably, our Dual-C3L. loss added on updated query features further elevates the mloU score to 43.04%, amplifying the combined strengths of our C3L and DMR modules in leveraging unlabeled multispectral vidcos for Semi-MVSS.
Analysis on C3L We then delve into the designs of C3I in Table 5 To effectively utilize unlabeled RGB- Thermal frame pairs, our C3L lever- ages cross pseudo labels as super- vision signals and introduces cross- modal collaboration to enable the generation of reliable pseudo labels. To verify their significance, we develop two variants, C3L w/o cross supervision and C3L w/o cross collaboration. In C3L the mean-teacher output of each stream is used to generate pseudo label for itself without using cross supervisions. This leads to reduction of 1.61% mloU compared to the standard C3L. We conjecture this drop is due to the accumulation of self-errors, whereas our C3L, with its cross supervision, has the ability to mutually correct potential errors, thereby enhancing overall accuracy. Moreover, it is worth noting that the C3L variant, which applies cross pseudo supervision without the benefit of cross cross-modal modal collabora- tion, leads to a deterioration in result. This indicates that cross-modal collaboration is critical and indispensable for the effective functioning of our Semi-MVSS framework.
Table Ablation analysis of our C3L module.
Ablation analysis on C3L Labeled
Unlabeled
mloU
Baseline
36.88
C3L w/o cross supervision
/
39.12
C3L w/o cross collaboration
/
36.67
C3L
40.73
Analysis on DMR. Table presents our DMR-Prom and an alternative module DMR-STM as in (75). DMR-ST STM performs an all-to-all at- tention mechanism for matching be- tween query and memory frames.
Table 6: Ablation analysis of our DMR module. DMRSSTM
DMR-Proto
Methods mloU
w/o temporal
wlo denoised
with denoised
w/o denoised
with demoised
40.73
41.47
42.25
41.85
42.39
A
(40.74)
(+1.5
(+1.12)
(+1.66)
Our results reveal that, 1) engaging temporal contexts from unlabeled past frames indeed useful, as both DMR modules yicld increased mloU scores; 2) the proposed reliability estimation strategy

---

effectively filters out unreliable features, enhancing the performance of both DMR variants. Consid- ering computational efficiency, DMR-Proto used for temporal integration in our SemiMV network. Additional model analyses are provided in the supplementary materials.
6
Conclusion
This study introduces MVUAV, a new multispectral video semantic segmentation dataset obtained through UAVs from oblique bird's-eye s-eee viewpoint. The dataset, accompanied with precise semantic annotations, serves as a complementary resource the MVSeg dataset, offering broader perspective for evaluating MVSS models. Additionally, this paper pioneers the development of SemiMV, the first semi-supervised MVSS framework tailored utilize both labeled and unlabeled multispectral videos effectively. Comprehensive empirical results affirm the effectiveness of our approach and highlight the promising potential of ceni-supervised leaning in MVSS.
Broader Impacts. The proposed MVUAV dataset offers significant value in enhancing the perfor- mance of semantic segmentation and propelling further research in the field of MVSS. We envisage that the most proximate impacts of this dataset will be positive, providing a valuable asset for researchers and developers in the field. Our semi-supervised MVSS method makes robust scene parsing available without intensive human annotation efforts, which saves a lot of costs. Note that the proposed MVUAV dataset strictly prohibited from being used to identify or invade the privacy of any individual and is made available solely for academic purposes. In addition, we discuss potential limitations with some feasible solutions the supplements.
Reproducibility Statement. Our source code and dataset, along with casy-to-follow instructions, are publicly available on our project website.
7
Acknowledgements
This work was partially supported by the the Alberta Innovates CASBE NSERC Alliance and the NSERC Discovery (RGPIN--0199045555) grants, and Samsung AI Center-Mountain View.
References
[1] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3313300 3223,222,22222 2016.
[2] Wei Ji. Deep learning- based segmentation for complex scene understanding, University of Alberia, 2024.
[3] Qi Bi, Shaodi You, and Theo Gevers. Generalized foggy-scene semantic segmentation by frequency decoupling. In CVPR, pages 1389-1399 ,000-119,,,00000 2024.
[4] Qi Bi, Shaodi You, and Theo Gevers. Learning generalized segmentation for foggy-scenes by bi-directional wavelet guidance. In AAAL, pages 801-80 1809, 2024.
[5] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Decplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4)8334884, 2017.
[6] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(10):3349 3364, 2020.
[7] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV, pages 173-190, 2020.
[8] Jingjun Yi, Qi Bi, Hao Zheng, Haolan Zhan, Wei Ji, Yawen Huang, Yuexiang Li, and Yefeng Zheng. Learning spectral-decomposinee tokens for domain generalized semantic segmentation. In ACM MM, 2024.
[9] Qi Bi, Shaodi You, and Theo Gevers. Interactive learning of intrinsic and extrinsic properties for all-day semantic segmentation. IEEE Transactions on Image Processing, 32:3821-38333 2023.
[10] Wei Ji, Jingjing Li, Qi Bi, Tingwei Liu, Wenbo Li, and Li Cheng. Segment anything is not always perfect: An investigation of sam on different real-world applications. Machine Intelligence Research, 21:617-633 2024.
10

---

[11] Munan Ning, Donghuan Lu, Yujia Xie, Dongdong Chen, Dong Wei, Yefeng Zheng, Yonghong Tian, Shuicheng Yan, and Li Yuan. Madav2: Advanced multi-anchor based active domain adaptation segmenta- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45((1): 2023.
[12] Wujie Zhou, Xinyang Lin, Jingsheng Lei, Lu Yu, and Jeng-Neng Hwang. Mffenet: Multiscale feature fusion and enhancement network for rgb-thermal urban road scene parsing. IEEE Transactions on Multimedia, 25338, 2021.
[13] Wei Wu, Tao Chu, and Qiong Liu. Complementarit--wwarr cross-modal feature fusion network for rgb-1 semantic segmentation. Pattern Recognition, 131: 101:108881, 2022.
[14] Qiang Zhang, Shenlu Zhao, Yongjiang Luo, Dingwen Zhang, Nianchang Huang, and Jungong Han. Abmdmet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation. In CVPR, pages 2633-2644 2021.
[15] Wujie Zhou, Shaohua Dong, Caie Xu, and Yaguan Qian. Edge-aware guidance fusion network for rgb thermal scene parsing. In AAAI, pages 3571 2571-3579, 2022.
[16] Ping Hu, Fabian Caba, Oliver Wang, Zhe Lin, Stan Sclaroff, and Federico Perazzi. Temporally distributed networks for fast video semantic segmentation. In CVPR, pages 8818-8827, 2020.
[17] Matthieu Paul, Martin Danelljan, Luc Van Gool, and Radu Timofte. Local memory attention for fast video semantic segmentation. In IROS, pages 1102- 1109, 2021.
[18] Jiangtong Li, Wentao Wang, Junjie Chen, Li Niu, Jianlou Si, Chen Qian, and Liqing Zhang. Video semantic segmentation via sparse temporal transformer. In ACM MM, pages 59-68, 2021.
[19] Jingjing Li, Tianyu Yang, Wei Ji, Jue Wang, and Li Cheng. Exploring denoised cross- contrast for weakly-supervised temporal action localization. In CVPR, pages 19914-19924, 2022.
[20] Xiaoqi Zhao, Youwei Pang, Jiaxing Yang. Lihe Zhang, and Huchuan Lu. Multi-source fusion and automatic predictor selection for zero-shot video object segmentation. In ACM MM, pages 2645-2653, 2021.
[21] Wei Ji, Jingjing Li, Cheng Bian, Zongwei Zhou, Jiaying Zhao, Alan L Yuille, and Li Cheng. Multispectral vidco semantic segmentation: A benchmark datasct and baseline. In CVPR, pages 1094-1104, 2023.
[22] Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consistency targets improve semi- supervised deep learning results. NeurlPS, pages 1195-1204, 2017.
[23] Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross- consistency training. In CVPR, pages 12674-12684, 2020.
[24] Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong consistency in supi-super semantic segmentation. In CVPR, pages 7236-7246, 2023.
[25] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi supervupe semantic segmentation with cross pseudo supervision. CVPR, pages 2613-2622, 2021.
[26] Qi Bi, Jingjun Yi, Hao Zheng, Wei Ji, Yawen Huang. Yuexiang Li, and Yefeng Zheng. Learning generalized medical image segmentation from decoupled feature queries. In AAAI, pages 810-818, 2024.
[27] Wei Shuang Yu. Junde Wu, Kai Ma, Cheng Bian, Qi Bi, Jingjing Li, Hanruo Liu, Cheng, and Yefeng Zheng. Learning calibrated medical image segmentation via multi-rater agreement modeling. In CVPR, pages 12341-12351, 2021.
[28] Junde Wu, Wei Ji, Huazhu Fu, Min Xu, Yueming Jin, and Yanwu Xu. Medsegdiff-r2: Diffusion-based medical image segmentation with transformer. In AAAI, pages 6030-6033 00006 2024.
[29] Miao Zhang, Jingjing Li. Wei Ji. Yongri Piao, and Huchuan Lu. Memory-oriented decoder for light field salient object detection. NeurlPS, pages 898-900 9088908,444 2019.
[30] Miao Zhang, Wei Ji, Yongri Piao, Jingjing Li, Yu Zhang, Shuang Xu, and Huchuan Lu. Lfnet: Light field fusion network for salient object detection. IEEE Transactions on Image Processing, 29:6276-6287, 2020.
[31] Xiaoqi Zhao, Youwei Pang, Wei Ji, Baicheng Sheng, Jiaming Zuo, Lihe Zhang, and Huchuan Lu. Spider: 2024, A unitied framework for context-dependent concept segmentation. In ICML, pages
[32] Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Automatic polyp segmentation via multi-scale subtraction network. In MICCAI, pages 120- 130, 2021.
11

---

[33] Jonathan Long, Evan Shelhamer. and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR. pages 3431-3440, 2015.
[34] Liang-Chiet Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[35] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.
[36] Liang-Chich Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan I. Yuille. Attention to scale: Scale-aware semantic image segmentation. In CVPR. pages 3640-3649, 2016.
[37] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, pages 3146-3154, 2019.
[38] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, pages 603-612, 2019.
[39] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[40] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Josc M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34:12077- 2021.
[41] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Lao, and Ling Shao. Pyramid vision transformer; A versatile backbone for dense prediction without convolutions. In ICCV, pages 368-578, 2021.
[42] Qi Bi, Shaodi You, and Theo Gevers. Learning content-enhanced mask transformer for domain generalized urban-scene segmentation. In AAAI, pages 819-827, 2024.
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
[44] Irem Ulku and Erdem Akagündüz. A survey on deep lcarning-based architcctures for semantic segmentation on 2d images. Applied Artificial Intelligence, pages 1 1-45, 2022.
[45] Wei Ji. Jingjing Li, Cheng Bian. Zhicheng Zhang, and Li Cheng. Semantiert: A large-scale dataset and method for robust semantic segmentation in multispectral images. In ACM MM, pages 3307-3316, 2023.
[46] Qishen Ha, Kohei Watanabe, Takumi Karasawa, Yoshitaka Ushiku, and Tatsuya Harada. Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In IROS, pages 5108-5115, 2017.
[47] Shreyas S Shivakumar, Neil Rodrigues, Alex Zhou, Ian D Miller, Vijay Kumar, and Camillo J Taylor. Ps1900: Rgb-thermal calibration, dataset and segmentation network. In ICRA, pages 9441-9447, 2020.
[48] Yuxiang Sun. Weixun Zuo, and Ming Liu. Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes. IEEE Robotics and Automation Letters, 4(3):2576 20066-2533222 2019.
[49] Yuxiang Sun, Weixun Zuo, Peng Yun, Hengli Wang, and Ming Liu. Fuseseg: semantic segmentation of urban scenes based on rgb and thermal data fusion. IEEE Transactions on Automation Science and Engineering, 18(3):1000-100 10000,0-011111001110111 2020.
[50] Fuqin Deng, Hua Feng, Mingjian Liang, Hongmin Wang. Yong Yang. Yuan Gao, Junieng Chen, Junjie Hu, Xiyue Guo, and Tin Lun Lam. Feanet: eratune-enhanced attention network for rgb-thermal real-time semantic segmentation. In IROS, pages 4467-4473, 2021.
[51] Wei Ji, Ge Yan, Jingjing Li, Yongri Piao, Shunyu Yao, Miao Zhang, Li Cheng, and Huchuan Lu. Dmra: Depth-induced multi- scale recurrent attention network for rgb-d saliency detection. IEEE Transactions on Image Processing. 31:2321-2336, 2022.
[52] Yongri Piao, Wei Ji, Jingjin Li, Miao Zhang, and Huchuan Lu. Depth-induced multi-scale recurrent attention network for saliency detection. In ICCV, pages 7254-7263 2019.
[53] Wei Ji. Jingjing Li, Qi Bi, Chuan Guo, Jie Liu, and Li Cheng. Promoting saliency from depth: Deep unsupervised rgb-d saliency detection. ICLR, 2022.
12

---

[54] Jingjing Li, Wei Ji, Qi Bi, Cheng Yan, Miao Zhang, Yongri Piao, Huchuan Lu, et al. Joint semantic mining for weakly supervised rgb-d salient object detection. NewrIPS, 34:11945-1195,, 2021.
[55] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang. Suppress and balance: A simple gated network for salient object detection. In ECCV, pages 35-51, 2020.
[56] Xiaoqi Zhao, Lihe Zhang, Youwei Pang, Huchuan Lu, and Lei Zhang. A single stream network for robust and real-time rgb-d salient object detection. In ECCV, pages 646-662. 2020.
[57] Jingjing Li, Wei Ji, Miao Zhang. Yongri Piao, Huchuan Lu, and Li Cheng. Delving into calibrated depth for accurate rgb-d salient object detection. International Journal of Computer Vision, 131(4):55--87, 2023.
[58] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Xiang Ruan. Self-supe pretraining for rgb-d salient object detection. In AAAI, pages 3463 04013,37,,200 2022.
[59] Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, and Huchuan Lu, Accurate rgb-d salient object detection via collaborative learning. In ECCV, pages 52-69. Springer, 2020.
[60] Jingjing Li, Wei Ji, Size Wang. Wenbo Li, and Li Cheng. Dvsod: Rgh-d video salient object detection. In NewrIPS, pages 8774-8787, 2023.
[61] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, and Huchuan Lu. Joint learning of salient object detection, depth estimation and contour extraction, IEEE Transactions on Image Processing, 31:7350-7362, 2022.
[62] Wei Ji, Jingjing Li, Shuang Yu, Miao Zhang, Yongri Piao, Shumyu Yao, Qi Bi, Kai Ma, Yefeng Zheng, Huchuan Lu, and Li Cheng. Calibrated rgb-d salient object detection. In CVPR, pages 9471 2021.
[63] Youwci Pang. Lihe Zhang, Xiaoqi Zhao, and Huchuan Lu. Hicrarchical dynamic filtering network for rgb-d salient object detection. In ECCV, pages 235-2522 252,, 2020.
[64] Youwei Pang. Xiaoqi Zhao, Lihe Zhang, and Huchuan Lu. Caver: Cross-modal view-mixed transformer for bi-modal salient object detection. IEEE Transactions on Image Processing, pages 892-904, 2023.
[65] Xiaoqi Zhao, Youwei Pang, Lihe Zhang, Huchuan Lu, and Lei Zhang. Towards diverse binary segmentation via a simple yet general gated network. International Journal of Computer Vision, pages 1-78, 2024.
[66] Miao Zhang, Shumyu Yao, Beiqi Hu, Yongri Piao, and Wei Ji. C2dfnet: Criss-cross dynamic filter network for rgb-d salient object detection. IEEE Transactions Multimedia, 25:5142-51555 2022.
[67] Guolei Sun, Yun Liu, Hao Tang, Ajad Chhatkuli, Ie Zhang, and Lue Van Gool. Mining relations among cross-frame affinities for video semantic segmentation. In ECCV, pages 522-539, 2022.
[68] Yule Li, Jianping Shi, and Dahua Lin. Low-latency video semantic segmentation. In CVPR, pages 1997-6005 2018.
[69] Xiaoqi Zhao, Shijie Chang, Youwei Pang, Jiaxing Yang, Lihe Zhang, and Huchuan Lu. Adaptive multi- source predictor for zero-shot video object segmentation. International Journal of Computer Vision, pages 1-19, 2024.
[70] Miao Zhang, Jie Liu, Yifei Wang. Yongri Piao, Shunyu Yao, Wei Ji, Jingjing Li, Huchuan Lu, and Zhongxuan Luo. Dynamic context-sensitive filtering network for video salient object detection. ICCV, pages 1553-1563, 2021.
[71] Raghudeep Gadde, Varun Jampani, and Peter Gehler. Semantic video cnns through representation warping. In ICCV, pages 4453-4462, 2017.
[72] David Nilsson and Cristian Sminchisescu. Semantic video segmentation by gated recurrent flow propaga- tion. In CVPR, pages 6819-6828, 2018.
[73] Yu- Syuan Xu, Tsu-Jui Fu, Hsuan Hsuan-Kung Yang, and Chun- Yi Lec. Dynamic vidco segmentation network. In CVPR, pages 6556-6565, 2018.
[74] Alexey Dosovitskiy, Philipp Fischer, Eddy 1lg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, pages 2758-2766 2015.
[75] Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, pages 9226 92066,92555222 2019.
13

---

[76] Guolei Sun, Yun Liu, Henghui Ding. Thomas Probst, and Luc Van Gool. Coarse-to-iin fine feature mining for video semantic segmentation. In CVPR. pages 3126-3137, 2022.
[77] Samvit Jain, Xin Wang, and Joseph E Gonzalez. Accel: A corrective fusion network for efficient semantic segmentation video. In CVPR, pages 8866-888 8875, 2019.
[78] Liang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D Collins, Ekin D Cubuk, Barret Zoph, Hartwig Adam, and Jonathon Shlens. Naive-student: Leveraging semi-supervised learning in video sequences for urban scene segmentation. In ECCV, pages 695 714, 2020.
[79] Yuchao Wang, Haochen Wang. Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In CVPR, pages 4248 4257, 2022.
[80] Jiafan Zhuang, Zilei Wang, and Yuan Gao. Semi- supervised video semantic segmentation with inter-frame feature reconstruction. In CVPR, pages 3263-3271, 2022.
[81] Pengyu Zhang, Jie Zhao, Dong Wang. Huchuan Lu, and Xiang Ruan. Visible-thermal uav tracking: large-scale benchmark and new baseline. In CVPR, pages 8886-8895, 2022.
[82] Gabriel Brostow, Julien Fauqueur, and Roberto Cipolla. Semantic object classes video: high- definition ground truth database. Puttern Recognition Letters, 30(2):88-97, 2009.
[83] Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS Journal of Photogranmetry and Remote Sensing, 165:108- 119, 2020.
[84] Chenglong Li, Wei Xia, Yan Yan, Bin Luo, and Jin Tang. Segmenting objects in day and night: Edge- conditioned cnn for thermal image semantic segmentation. IEEE Transactions on Neural Networks and Learning Systems, (11)30069-30333 2020.
[85] Haitao Xiong. Wenjie Cai. and Qiong Liu. Menet: Multi-level correction network for thermal image semantic segmentation of nighttime driving scene. Infrared Physics & Technology, 113:103628, 2021.
[86] Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi-interaccivee feature learning and a full-time multi-modality benchmark for image fusion and segmentation. In ICCV, pages 8115 8124, 2023.
[87] Connor Lee, Matthew Anderson, Nikhil Raganathan, Xingxing Zuo, Kevin Do Georgia Gkioxari, and Soon-Jo Chung. Cart: Caltech aerial rgb-thermal dataset the wild. arXiv preprint arXiv:2403.0890 2024.
[88] Dawei Du, Yuankai Qi, Hongyang Yu. Yifan Yang. Kaiwen Duan, Guorong Li. Weigang Zhang, Qingming Huang, and Qi Tian. The unmanned aerial vehicle benchmark: Object detection and tracking. In ECCV, pages 370-386, 2018.
[89] Deyi Ji, Feng Zhao, Hongtao Lu, Mingyuan Tao, and Jieping Ye. Ultra Ultra-high resolution segmentation with ultra-rich context: A novel benchmark. In CVPR, pages
2023.
[90] Junjue Wang, Zhuo Zheng, Ailong Ma, Xiaoyan Lu, and Yanfei Zhong. Loveda: A remote sensing land-cover dataset for domain adaptive semantic segmentation. NewrIPS, 2021.
[91] Pengfei Zhu, Longyin Wen, Xiao Bian, Haibin Ling, and Qinghua Hu. Vision meets drones: A challenge. arXiv preprint arXiv:1804.07437 2018
[92] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, pages 248 255, 2009.
[93] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We outline main contributions at the end of Sec. 0 The claims made match the dataset description in Sec. methodology in Sec. and experimental results in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in the supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification: Sec. provides a detailed explanation of the proposed SemiMV.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The proposed method is illustrated in Sec. 4 and our implementation details are provided Sec. [.2] The MVSeg dataset used the experiments are public available from previous works. Besides, we will release the code and our new dataset on our website.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The authors will make the newly proposed dataset and source code, along with detailed instructions, publicly available on our project website.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Coaee//ddeeSSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details given in Sec. 3 and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We show the reliability of the proposed method by conducting multiple rumning experiments the SemiMV, in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details are given in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp/:/eerrppp...c//b///////t
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirmed that we conform the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the potential societal impacts in Sec. 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
We prohibit people from using our MVUAV in any manner to identify or invade the privacy of any person. Additionally, our MVUAV dataset made freely available solely for academic purposes.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers for all assets used in our research, which are publicly available to the research community.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The newly proposed MVUAV dataset is detailed in Sec. We will also release it along with user-friendly usage guidelines.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "QOMaiyM2xO", "reviewer": "Reviewer_w3ua", "summary": "The paper addresses multispectral video semantic segmentation (MVSS) and proposes a new semi-supervised learning approach. It introduces the SemiMV framework, which utilizes a Cross-collaborative Consistency Learning (C3L) module and denoised temporal aggregation strategy. Additionally, the paper establishes the MVUAV benchmark dataset, captured by UAVs, offering unique bird’s-eye views and various semantic categories.", "questions": "Q1: The paper should address whether the RGB and thermal (TIR) images in the MVUAV dataset are precisely aligned. Some existing RGB-T datasets [1]  have issues with modality misalignment, which serveral previous research [2][3] has tried to address. Therefore I wonder if this dataset exist same misalignment issue.\n\nQ2: The author should clarify the relationship between MVUAV and MVSeg. Is MVUAV merely a complementary perspective, or does it offer other different scene?\n\nQ3: Given the sparse annotations, can this dataset be used for general segmentation tasks? Or if it is only suitable for semi-supervised  methods?\n\n\n[1] Multispectral Pedestrian Detection: Benchmark Dataset and Baseline\n[2] Weakly Aligned Cross-Modal Learning for Multispectral Pedestrian Detection\n[3] Attentive Alignment Network for Multispectral Pedestrian Detection", "limitations": "The authors acknowledge that existing MVSS datasets, including MVUAV, are small due to high labeling costs, and while semi-supervised methods help, they don't fully meet real-world demands. They also mention that the SemiMV baseline and MVUAV dataset, though promising, face challenges such as small targets and scale variation, necessitating further research and enhancements.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1: The paper presents a dataset that clearly must have required a lot of effort and resources to gather. It has the potential to contribute significantly to the community.\n\n\nS2: The authors also provide a semi-supervised MVSS baseline, which demonstrates the practical application and effectiveness of their data.\n\n\nS3: The dataset offers annotations and has advantages over existing datasets in terms of resolution, modality, annotation quality, and dataset size.", "weaknesses": "W1: Compared to UAVid, the resolution seems somewhat low, especially considering the UAV perspective where scenes are typically larger and require higher resolution for detailed analysis.\n\nW2: The focus on UAV-captured data might limit the dataset’s applicability to ground-level or other perspectives, potentially reducing its versatility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer w3ua, we greatly appreciate the time and effort you have dedicated to providing constructive suggestions on ways to strengthen our paper. We are also grateful for the positive comments and recognition. Below, we make a point-by-point response to all the comments.\n\n---\n\n***W1**: \"Compared to UAVid, the resolution seems somewhat low, ... .\"*      \n**Response**: Thank you for your constructive comment. We agree that higher resolution are often desirable for detailed analysis, especially from the UAV perspective. Compared to UAVid, **the resolution difference in our MVUAV arises from the inherent characteristics of thermal infrared cameras compared to RGB cameras.** While our MVUAV dataset has a relatively lower resolution than the RGB-based UAVid dataset, it uniquely **offers both RGB and thermal UAV videos, providing essential complementary information to enhance low-light vision**, which UAVid does not offer. Moreover, **compared to existing *RGB-thermal datasets*** such as image-only UAV dataset CART [23] and video dataset MVSeg [22], **our MVUAV features a relatively higher resolution** (1920×1080 in MVUAV vs. 960×600 in CART and 640×480 in MVSeg). For higher resolution needs, we could employ off-the-shelf super-resolution tools to enhance the image quality. We will include this discussion in our paper.\n\n---\n\n***W2**: \"The focus on UAV-captured data might limit the dataset’s applicability to ground-level or other perspectives, ... .\"*      \n**Response**: Thanks for your valuable comment. We humbly believe that *UAV-view data and ground-level data have different characteristics and advantages, making them suited to different application scenarios.* From a complementary perspective, **our MVUAV dataset offers a distinct bird's-eye viewpoint that complements existing ground-level datasets** like MVSeg. The presence of both datasets could **enrich the diversity of perspectives available in the field of MVSS, enabling more comprehensive analysis and validation of algorithms** across various scenarios. \nFurthermore, **UAV-captured data provide a broader, holistic view, free from the constraints of ground-level capture.** This characteristic is **advantageous for applications that require comprehensive coverage in challenging conditions,** such as aerial nighttime search and rescue, sea patrols, firefighting response support, traffic management, and UAV delivery services.\n\n---\n\n***Q1**: \"The paper should address whether the RGB and thermal (TIR) images in the MVUAV dataset are precisely aligned. ... .\"*      \n**Response**: Thank you for your insightful question. **We concur with the importance of well-aligned RGB-T pairs for segmentation tasks, and have proactively taken efforts to ensure the quality of our dataset. Specifically, this awareness has been taken into account during the dataset collection and preparation stages of both the sourced VTUAV dataset [56] and our MVUAV dataset.** In the VTUAV dataset, the authors manually identified corresponding feature points on both RGB and thermal images and calculated an affine transformation matrix from these points. Using this matrix, one image was warped to align with the other, and the common overlapping regions were extracted and resized to a consistent resolution while maintaining the aspect ratio. *This ensures that most frames are well-aligned. Additionally, we performed a visualization screening process* by overlaying thermal heat maps onto paired RGB images. This made it easier for our inspectors to verify alignment and allowed us to filter out low-quality samples (e.g., similar content, blurred, or misaligned images), *thus further enhancing the overall quality of the MVUAV dataset.* We will include these discussions and add related works in our final paper.\n\n---\n\n***Q2**: \"The author should clarify the relationship between MVUAV and MVSeg. ... .\"*      \n**Response**: Thanks for your constructive suggestion for improving our paper.\n**Our MVUAV dataset not only offers a distinct bird's-eye viewpoint that complements existing ground-level MVSeg dataset, but also includes additional scenarios.** Thanks to the unique characteristics of UAVs, which provide a broader and more holistic view free from the constraints of ground-level capture, **MVUAV encompasses extra challenging scenes such as** rivers, boats, bridges, and playgrounds, as shown in Figure 3. It also covers a diverse set of 36 semantic classes. **Various visual scenes can be accessed on *our project website*.**\nWe will discuss this in the revised paper.\n\n---\n\n***Q3**: \"Given the sparse annotations, can this dataset be used for general segmentation tasks? ... .\"*      \n**Response**: Thanks for your question. **Yes, our dataset can be used for general fully-supervised segmentation tasks,** in addition to the semi-supervised setting. **For example,** researchers can directly utilize our labeled RGB-Thermal samples in the MVUAV dataset for multispectral (RGB-thermal) semantic segmentation (MSS) task. **In Table 9 of the Appendix, we provide comprehensive benchmarking results of various segmentation methods on the new MVUAV dataset under the fully-supervised setting.** *These results, along with our dataset and source code, will be made publicly available*. We hope this will support researchers in addressing their specific needs.\n\n---\n\nThanks again for your encouragement and insightful suggestions. If there are other questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EfxolEnuag", "reviewer": "Reviewer_XBRU", "summary": "This paper introduces a new multi-spectral aerial-view semantic segmentation dataset called MVUAV, which consists of 413 video sequences and 53K frames with sparsely annotated pixel-level segmentation labels. To provide a way of using this sparsely annotated dataset, it also introduces a new semi-supervised semantic segmentation method that takes RGB and thermal image pairs as input. The newly designed key component in this method is cross-collaborative consistent learning, which uses the segmentation predictions of one modal network as pseudo-labels for training another modal network. This paper demonstrates the effectiveness of this method with better performance than other baselines on a multi-spectral semi-supervised semantic segmentation task on two benchmark datasets including MVUAV.", "questions": "Please address weaknesses I pointed out.", "limitations": ".Limitation and potential societal impact are not mentioned in the main manuscript but included in supplementary material.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. MVUAV, a new dataset, is very valuable in the field of multi-spectral semantic segmentation. I admit that the aerial-view has more and smaller semantics than the ground-view (called eye-level view in the paper) due to its wider viewing angle, so labeling takes significantly more time.\n\n2. SemiMV, a new method introduced in the paper, showed better performance than other baselines on two benchmark datasets evaluating multi-spectral semantic segmentation tasks.", "weaknesses": "1. Immature presentation\n--\n\nOverall, the presentation in the paper is not mature for publication.\n\na. Section 4.2 is not easy to follow due to missing description of some terminology.\n\n- What is the memory feature  the memory feature $f^*_i$ and how is it obtained in eq 4?\n\n- In eq 5, the dimensions of $f_t$ and $transpose(p)$ are H$\\times$W$\\times$1 and MC$\\times$D, respectively. Here, the dimension of $f_t$ is guessed from eq 4, in which $f$ is element-wise multiplied with $\\mathcal{R}$ of the dimension H$\\times$W$\\times$1. How can the matrix multiplication be applied to these two variables? \n\nb. Figures do not deliver any crucial and complicate information.\n\n- Figure 2 is not necessary to understand what it conveys.\n\n- Figure 4 is not referred in the contents, so it is unclear what this figure exists to support.\n\n\n2. Insufficient contribution\n--\n\nContribution is not sufficient to meet NeurIPS standards.\n\na. As this paper also mentioned, the proposed network does not have any novel component. C3L is very simple and not new as already used in various previous works (works in Ln 187). \n\nb. Dataset is somewhat novel as it contains aerial-view images. However, no experiment is provided to demonstrate why acquiring this characteristics in dataset is important. \n\n3. Insufficient experiments\n--\n\na. All baseline methods compared in table 2 and 3 are developed not for the purpose of semi-supervised learning. Without comparing other semi-supervised learning method, it is difficult to figure out the effectiveness of the proposed method.\n\nb. Ablation studies for several parameters which are crucial in the method are needed, e.g., M and lambda.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer XBRU, thank you for your recognition that our MVUAV dataset is valuable and our SemiMV method shows better performance. We hope to address each of your questions point-by-point and clarify some misunderstandings.\n\n---\n\n***Q1-a**: \"(**a1**) What is the memory feature* $f_i^*$, ...*\" and \"(**a2**) In eq 5, ... How can the matrix multiplication be applied to these two variables?\"*  \n**Response to Q1-a**:  \n(**a1:**) $f_i^*$ **represents the decoded feature extracted from segmentation nets (i.e., $Net^R$ and $Net^T$ in Fig. 4) at the $i_{th}$ past frame** ($i \\in [t-M, \\cdots, t-1]$), where $* \\in\\\\{R,T\\\\}$ indicates the image modality and $M$ is the number of past frames stored in memory for temporal utilization. Here we can adopt common segmentation networks such as DeepLabv3+ [6] and SegFormer [50] to obtain $f_i^*$ $\\in \\mathbb{R}^{H\\times W\\times D}$. A summary of notation definitions (including $f_i^*$) was presented in Table 7. **In Eq. 4**,  $f_i^*$ **is used to generate denoised prototype feature** $p_i^* \\in \\mathbb{R}^{C\\times D}$.  \n\n(**a2:**) Similar to $f_i^*$, $f_t^*$ **has the dimension of $H\\times W\\times D$, not $H\\times W\\times 1$.** Thus, the matrix multiplication can be directly applied on their two $L_2$ normalized variables, $\\bar{f}_t^R \\in \\mathbb{R}^{H\\times W\\times D}$ and $transpose(\\bar{p}^*) \\in \\mathbb{R}^ {D\\times MC}$, thereby obtaining the attention weight $\\textbf{w} \\in \\mathbb{R}^ {H\\times W\\times MC}$ with softmax function in Eq. 5.\n\n**For better understanding, we provide a diagram (Figure A) in the PDF file of the general response**, illustrating the feature transformation process and corresponding dimension changes of Eqs. 4 & 5 & 6. If you have any other questions, please let us know.\n\n---\n\n***Q1-b1**: \"Figure 2 is not necessary ...\"*  \n**Response**: We respectfully believe Figure 2 is valuable as our work is the first to address the MVSS task from a semi-supervised perspective. It helps readers quickly understand the use of training data in this context, especially for those who may be unfamiliar with semi-supervised MVSS.\n\n---\n\n***Q1-b2**: \"Figure 4 is not referred in the contents ...\"*  \n**Response**: Figure 4 is referred to at the beginning of Sec. 4.2 (line 173). It depicts the overall architecture of our proposed SemiMV.\n\n---\n\n***Q2-a**: \"As this paper also mentioned, the proposed network does not have any novel component ...\"*   \n**Response**: **We respectfully believe that our proposed method is valuable and introduces new insights compared to previous works, including: 1) Cross-collaborative Consistency Learning:** Previous works [7,36,43,53] are specifically designed for single-modality RGB images, while our SemiMV excels in processing unlabeled multispectral videos by engaging their dual-perspective characteristic and cross-modal collaboration. Our experiments (lines 359-374) show that directly adapting [7,43] to semi-supervised MVSS is ineffective because they overlook the importance of cross pseudo-supervision and cross-modal collaboration. **2) Denoised temporal strategy:** We introduce a pixel-wise reliability map based on the learned cross-modal consistency to guide the temporal fusion process and mitigate noise. This addresses the noisy memory feature issue not covered by previous works [22,35,37]. **3) Extensive experiments:** Tables 4-6 & 12 verify the superiority of our SemiMV through investigating various design choices for semi-supervised MVSS. Tables 10 & 11 show that our method performs well with different backbones (CNN and transformer) and achieves consistent performance improvements when integrated with existing semi-supervised schemes. These results indicate that our SemiMV can serve as a scalable baseline for future work. **We will make our source code publicly available with easy-to-follow guidelines.**\n\n---\n\n***Q2-b**: \"Dataset is somewhat novel as it contains aerial-view ...\"*  \n**Response**: Thank you for recognizing the novelty of our dataset. Regarding why acquiring these characteristics in the dataset is important, please kindly refer to our response to Reviewer Mx7S's W2, which articulates the significance of MVUAV through three aspects. Note that we are not claiming that UAV-view data are superior to ground-level counterparts. We aim to highlight their unique benefits. Our MVUAV dataset offers a distinct bird's-eye view that complements the existing ground-level MVSeg, thereby enriching the diversity of perspectives in MVSS. Given the extensive literature on UAV-view research [33,45,56] and the aforementioned advantages, we believe this perspective is beneficial for the MVSS task. We hope this clarifies your concern.\n\n---\n\n***Q3-a**: \"All baseline methods compared in table 2 and 3 are developed not for the purpose of semi-supervised learning. ...\"*  \n**Response**: We respectfully remind the reviewer that MT, CCT, CPS, UniMatch, and IFR in Tables 2 and 3 are specifically developed for semi-supervised learning, as mentioned in lines 293-296. Additionally, Tables 4-6 and 9-14 present extensive ablation studies, various design choices, different backbones, and integration with other semi-supervised schemes, demonstrating the scalability and effectiveness of our method.\n\n---\n\n***Q3-b**: \"Ablation studies for several parameters ... e.g., M and lambda..\"*  \n**Response**: Following your suggestion, we investigated the impact of $M$ and $\\lambda$, **with results in Tables A & B of the one-page PDF**. Adding memory frames consistently improves mIoU scores, with a noticeable increase from  40.73% to 43.04% when $M=3$. Raising $M$ further beyond 3 gives marginal returns. Thus, we set $M=3$ for a better trade-off between accuracy and memory cost. For $\\lambda$, we found that $\\lambda=1$ balances supervised and pseudo losses effectively. These results will be included in our final paper.\n\n---\n\nThanks again for dedicating your valuable time and effort to our paper. If there are other questions, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gvh3v5EU9", "reviewer": "Reviewer_Mx7S", "summary": "The paper proposes advancements in multispectral video semantic segmentation (MVSS) through two key contributions: the creation of a new benchmark dataset, MVUAV, captured via UAVs, and the development of SemiMV, a semi-supervised learning baseline designed to optimize sparse annotations using Cross-collaborative Consistency Learning (C3L).", "questions": "Please see the weakness", "limitations": "Please see the weakness", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The MVUAV dataset introduces an oblique bird’s-eye view for multispectral video semantic segmentation, providing rich and diverse data that include a wide array of lighting conditions and over 30 semantic categories.\n\n2. The SemiMV framework uses semi-supervised learning for MVSS tasks, employing a Cross-collaborative Consistency Learning (C3L) module and a denoised temporal aggregation strategy, offering a solution to utilize sparse annotations and unlabeled data.\n\n3. The paper's empirical evaluations confirm that the SemiMV baseline enhances the multispectral video semantic segmentation.", "weaknesses": "1. The paper presents a new dataset for semantic segmentation from a high-altitude perspective, yet lacks a detailed comparison with existing aerial-view datasets [1, 2, 3, 4] which is necessary to establish the dataset's relevance and uniqueness.\n\n2. The paper fails to articulate the motivation and significance behind the introduction of the UAV-View dataset, leaving readers uncertain about the necessity and potential contributions of this new dataset to the field.\n\n3. There is an absence of a thorough analysis of limitations and broader impacts, which is a concern as it may not meet submission requirements that typically expect such discussions to understand the full implications and potential drawbacks of the research.\n\n4. The paper does not adequately address privacy concerns regarding the UAV-view multimodal dataset, such as the potential capture of private information like pedestrians and storefronts, and fails to clarify whether appropriate privacy measures are in place or if there is official and public approval for data collection in the relevant regions.\n\n[1] Vision Meets Drones: A Challenge\n\n[2] The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking\n\n[3] Ultra-High Resolution Segmentation with Ultra-Rich Context: A Novel Benchmark\n\n[4]  LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Mx7S, we sincerely appreciate the time and effort you spent reviewing our paper and your positive feedback. Your comments are insightful, and we look forward to addressing each of your concerns point-by-point.     \n\n---\n\n***W1**: \"The paper presents a new dataset for semantic segmentation from a high-altitude perspective, yet lacks a detailed comparison with existing aerial-view datasets [1, 2, 3, 4] which is necessary to establish the dataset's relevance and uniqueness.\"*      \n**Response**: Thanks for your valuable suggestion to include a detailed comparison with the mentioned aerial-view datasets [1, 2, 3, 4], which will make our work more comprehensive. In comparison, **VisDrone2018 [1] and UAVDT [2] are two large-scale datasets designed primarily for object detection and tracking tasks** in UAV-view RGB videos and/or images, providing bounding box annotations for target objects. In contrast, **our MVUAV dataset is focused on the semantic segmentation task** in UAV-view RGB-thermal videos, offering dense pixel-wise semantic annotations. **URUR [3] and LoveDA [4] are two high-resolution segmentation datasets** collected by high-quality satellite or Spaceborne images. **The key advantage of our MVUAV dataset compared to URUR and LoveDA is the inclusion of complementary multispectral (RGB-thermal) videos.** This feature aids in detecting target objects at nighttime or in adverse lighting conditions, thereby enhancing low-light vision capabilities. **In the PDF file of the general response, we present the detailed statistics of these datasets [1, 2, 3, 4] and our MVUAV dataset in Table C.** Following your suggestion, we will include these discussions and detailed analyses in our paper.  \n\n---\n\n***W2**: \"The paper fails to articulate the motivation and significance behind the introduction of the UAV-View dataset, leaving readers uncertain about the necessity and potential contributions of this new dataset to the field.\"*      \n**Response**: We greatly appreciate the opportunity to better articulate the motivation and significance of our UAV-view multispectral video semantic segmentation dataset (MVUAV). **The significance of MVUAV can be illustrated through the following points: (1) Importance of UAV-View Characteristics:** UAV-view data provide a broader, holistic perspective free from the constraints of ground-level capture. This characteristic has demonstrated to be advantageous for many applications in computer vision, such as detection [45], tracking [56], and segmentation [33]. \n**(2) Capability of Low-Light Vision:** Compared to existing UAV-view RGB segmentation datasets like UAVid [33], our dataset offers a unique combination of RGB and thermal infrared videos, enhancing low-light vision capabilities that existing works do not cover. \n**(3) Advancement of MVSS Task:** From a complementary perspective, our MVUAV dataset offers a distinct bird's-eye viewpoint that complements existing ground-level datasets like MVSeg. The presence of both datasets enriches the diversity of perspectives available in the field of MVSS, enabling more comprehensive analysis and validation of algorithms across various scenarios. This is particularly advantageous for applications requiring comprehensive coverage in challenging conditions, such as aerial nighttime search and rescue, sea patrols, firefighting response support, traffic management, and UAV delivery services. Thanks for your valuable suggestion. **We will add this motivation and discussion in our final paper to make it more clear.**  \n\n---\n\n***W3**: \"There is an absence of a thorough analysis of limitations and broader impacts, which is a concern as it may not meet submission requirements that typically expect such discussions to understand the full implications and potential drawbacks of the research.\"*      \n**Response**: Thanks for your valuable feedback. We respectfully remind the reviewer that a detailed discussion on limitations and broader impacts is included in Appendix A.6. We apologize for any confusion caused due to content layout. In the revised paper, we will ensure that proper reference is added to the main text to guide readers to our discussion.  \n\n---\n\n***W4**: \"The paper does not adequately address privacy concerns regarding the UAV-view multimodal dataset, such as the potential capture of private information like pedestrians and storefronts, and fails to clarify whether appropriate privacy measures are in place or if there is official and public approval for data collection in the relevant regions.\"*      \n**Response**: Thank you for raising the important issue of privacy concerns. In our MVUAV dataset, we provide rich pixel-level semantic segmentation annotations for selected multispectral UAV videos from VTUAV [56] and we do not create new source data. We have obtained official approvals from the original authors and institutions to further process and re-annotate their videos with semantic labels to advance the development of the MVSS field.\n**Following your suggestions, we will release de-identified videos using defacing and storefront detection tools to mitigate potential privacy issues, and we will also claim that we prohibit people from using our MVUAV in any manner to identify or invade the privacy of any person**. Additionally, **our MVUAV dataset will be made freely available solely for academic purposes.**      \n\n---\n\n***Ethics Review***: Please kindly refer to our responses to W3 and W4.   \n\n---\n\nOnce again, thank you for your valuable time and comments for enhancing the quality of our paper. We hope our response can address your concerns. If there are any further questions, please feel free to share, and we are happy to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chairs, 

We would like to thank you for your valuable time and efforts in providing these insightful questions and suggestions for improving our paper. We are also pleased that the reviewers have generously appreciated our new MVUAV dataset and the semi-supervised MVSS baseline - SemiMV.

In the individual response, we have provided detailed, point-by-point responses to each reviewer's comments. A one-page PDF file is also attached that contains all relevant figures and tables used in the response, including a figure illustrating the feature transformation process and corresponding dimension changes of Eqs. 4 & 5 & 6 (Reviewer XBRU), two ablation tables verifying the impact of hyperparameters (Reviewer XBRU), and a table comparing our MVUAV dataset with related UAV datasets (Reviewer Mx7S). We hope this could provide a more comprehensive presentation of our work.

If there are any further questions, please let us know. We would be happy to discuss and answer them in the reviewer-author discussion phase.

Best Regards,     
Authors of Paper 906

Meta-review: This paper proposes a new benchmark dataset, MVUAV, and a semi-supervised learning approach for multispectral video semantic segmentation (MVSS) for UAV images. The initial ratings are mixed. The authors provided a strong rebuttal, and all reviewers increased their rating after the discussion, reaching an all-positive consensus. The authors should improve the representation and explanations, clarify the motivation and significance of research on UAV-view images, address the ethical concerns, and incorporate the additional feedback provided during the rebuttal and discussion period.
Decision: Accept (poster)