We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline
Authors: Simar Kareer
Vivek Vijaykumar
Harsh Maheshwari
Judy Hoffman
Prithvijit Chattopadhyay
Viraj Uday Prabhu
Venue: Accepted by TMLR ()

Abstract: There has been abundant work in unsupervised domain adaptation for semantic segmentation (DAS) seeking to adapt a model trained on images from a labeled source domain to an unlabeled target domain. While the vast majority of prior work has studied this as a frame-level Image-DAS problem, a few Video-DAS works have sought to additionally leverage the temporal signal present in adjacent frames. However, Video-DAS works have historically studied a distinct set of benchmarks from Image-DAS, with minimal cross-benchmarking. In this work, we address this gap. Surprisingly, we find that (1) even after carefully controlling for data and model architecture, state-of-the-art Image-DAS methods (HRDA and HRDA+MIC)} outperform Video-DAS methods on established Video-DAS benchmarks (+14.5 mIoU on Viper$\rightarrow$CityscapesSeq, +19.0 mIoU on Synthia$\rightarrow$CityscapesSeq), and (2) naive combinations of Image-DAS and Video-DAS techniques only lead to marginal improvements across datasets. To avoid siloed progress between Image-DAS and Video-DAS, we open-source our codebase with support for a comprehensive set of Video-DAS and Image-DAS methods on a common benchmark. Code available at https://github.com/SimarKareer/UnifiedVideoDA

Published in Transactions on Machine Learning Research (01/2024)
We're Not Using Videos Effectively: An Updated Domain Adaptive Video Segmentation Baseline
Simar Kareer, Vivek Vijaykumar, Harsh Maheshwari, Prithvijit Chattopadhyay, Judy Hoffman, Viraj Prabhu
Georgia Institute of Technology
Correspondence: skareer@gatech, edu
Reviewed on OpenReview: https: openreviee. formm 10R6X666.
Abstract
There has been abundant work in unsupervised domain adaptation for semantic segmenta- lion (DAS) seeking to adapt a model trained on images from a labeled source domain to an unlabeled target domain. While the vast majority of prior work has studied this as a frame- level Image-DAS problem, a few Video-DAS works have sought to additionally leverage the temporal signal present adjacent frames. However, Video-DAS works have historically studied distinct set of benchmarks from Image-DAS, with minimal cross-benchmarkingg In this work, address this gap. Surprisingly, find that (1) even after carefully controlling for data and model architecture, state-of-the-art Image-DAS methods (HRDA and HRDA+MMCC outperform Video-DAS methods on established Video-DAS benchmarks (+14.5 mIoU on Viper- Sitssscppcyccceseeg +19.0 mIoU on Sauthia-Seq--Chtyycaapp (2) naive combinations of Image-DAS and Video- DDS techniques only lead to marginal im- provements across datasets. To avoid siloed progress between Image-DAs DAS and Video-DAS open-source codebase with support for comprehensive set of Video-DAS, and Image- DAS methods common benchmark. Code available at this link.
and
1 Introduction
Semantic Segmentation mloU
Deep learning systems perform remarkably well when training and test data are drawn from the same distribution. However, this assumption is of- ten violated in the real world e-g- under changing weather (Hoffman et al., 2018), geographies (Prabhu et al., 2022; Kalluri et al., 2023), or curation strategies (Saenko et al., 2010), wherein annotat- ing additional data may be prohibitively expen- sive. To address this, unsupervised domain adap- tation (UDA) (Saenko et al., 2010; Ganin & Lem- pitsky, 2015; Tzeng et al., 2017) attempts to train model which performs well on the target do- main given access to labeled source data and un- labeled target data. UDA has seen successful ap- plication to computer vision tasks such as semantic segmentation (Hoffman et al., 2018; Hoyer et al., 2022b;c), object detection (Chen et al., 2018; Li et al., 2022; Vibashan et al., 2023) and person re- identification (Mekhazni et al., 2020; Zhao et al., 2021). In this work, we focus on the complex task of semantic segmentation adaptation due to its high labeling cost and ubiquity in practical applications such as self-driving (Wu et al., 2021).
80
Viper to Cityscapes
+19.0 mioU
75
Synthla to Cityscapes
70
65
mloU
+14.5 mloU
60
55
50
45
40
DAVSN 100V2021
TPS ECOV 2022
STPL CVPR 2023
MOM WACV 2023
HRDA MIC
CVPR20023
Image Method
Video Methods
Figure 1: Overview. Recent domain adaptive video segmentation mothods do not compare against state- of-the-art baselines for Image-DAS. We perform the first such cross- benchmrrkhn,,,, and find that even af- ter controlling for data and model architecture, Image- DAS methods strongly outperform Video-DAS meth- ods on the two key Video-DAS benchmarks.
Since data from simulation can be cheaply anno- tated (Richter et al., 2016), several works for se-

---

Published in Transactions on Machine Learning Research (01/2024)
mantic segmentation adaptation have attempted to leverage labeled simulated data (Hoyer et al., 2022b) to adapt to unlabeled real data. Benchmarks for this task are commonly in the context of autonomous driving, where the source data are videos from driving simulator, and the target data are videos from a real dashcam. This problem has seen considerable progress: for instance, on the popular GTAV (Richter et al., Ciyysapes (Cordts et al., 2016) benchmark, absolute performance has jumped by >45 points since its inception (Hoffman et al., 2016). Surprisingly though, the works driving this progress have typically ignored the sequential nature of the data and studied this as an Image-DAS, problem.
A few isolated works have pioneered the problem of Video- MAso and attempt to leverage temporal dynamics in domain aligmment for semantic segmentation (Guan et al., 2021; Shin et al., 2021; Xing et al., 2022; Wu et al., 2022; Cho et al., 2023; Lo et al., 2023). However, these works employ a different set of benchmarks and methods than the Image-Doe DAS methods. As a result, progress on these two related problems has been siloed, with no recent cross-benchmrkinng We bridge this gap and present updated baselines for Video-DAS.
Concretely, we benchmark the performance of HRDA (Hoyer et al., 2022b) and HRDA+MIC (Hoyer et al., 2022c), state-of-the-art methods for Image-DAS DAS, on Video-DAs DAS benchmarks. We find that even after care- fully controlling for model architecture and training data, HRDA+ +MI outperforms otate-of-the-art Video- DAS methods, e.g. by 14.5 mIoU on Cheer-Cityscapes-Seg and 19.0 mIoU on (Figure 1), the two established benchmarks for this task. We perform an ablation study to identify the source of this improvement and find restoreesolution fusion (Hoyer et al., 2022b) to be the most significant factor. In essence, multi-resolution fusion enables segmentation models to utilize full resolution images rather than down-samplinn sampling,, while maintaining a manageable memory footprint.
The strong performance of recent Image-DAS methods (e.g. HRDA+MIC) Vidco-DAS benchmarks raises the question of whether current Video-DAS methods are still useful given this updated baseline. To study this, we benchmark a comprehensive set of Video-DAS techniques from the literature, including video-level domain discriminator, the ACCEL video segmentation architecture (Jain et al., 2019), temporally consistent mix-up (Cho et al., 2023), and a suite of pseudo-label refinement strategies. We exhaustively test combi- nations of these methods with HRDA and do not observe significant gains with any single combination. Tn contrast to prior work which only reports on Viper--Cityscapess Seq and propose and corroborate these findings two additional Vidco-DAS shifts in order to draw stronger conclu- sions. Specifically, we propose Viper->BDDVid and Synthia-Seq-BBDDY where BDDVid is constructed from BDD10k (Yu et al., 2020).
we
In the process, we develop and open-source UnifiedVideoDA, the first modern codebase built on top of a popular semantic segmentation library, MMSSegmentation (Contributors, 2020), which supports a range of Image-DAS and Video-DAS techniques out-of-the-box. We hope that this codebase will bridge the two communities and bolster an increased exchange of ideas.
We make the following contributions:
We discover that even after controlling for model architecture, HRDA+MIC (a recently proposed Image-DAS method) sets the state-of-the-art on existing Vidco-DAS benchmarks, outperforming specialized Video-DAS methods, by as much as +14.5 mloU on Viper-CClyscape-See and +19.0 mIoU on SeaubiiaSSe---iiyscaaaa
We perform an ablation study to explain this performance gap, and discover multi-resolution fu- sion (Hoyer et al., 2022b), which enables utilization of full-resolution images with a manageable memory footprint, to be the largest contributing factor.
We explore hybrid UDA strategies which add popular Video-DAS techniques to updated baseline. While a few selectively improve performance, e-g. pseudo-label refinement. (+4.6 mloU, Table 6), and targeted architectural modification (+2.2 mloU, Table 4), we do not find any strategy with significant improvements across existing benchmarks and two additional shifts we propose.
We open-source our codebase UnifiedVideoDA, built on top of MMSegmentation, a commonly used library for Image-DAS. We enable Vidco-DAS rescarch in this codebase by enabling it to load consecutive frames, optical flow, and implementing baselines for key video techniques. We hope this codebase can help the community push the frontier of Video-DAS and Image-DAS simultaneously.
2

---

Published in Transactions on Machine Learning Research (01/2024)
2 Related work
2.1 Image-level UDA for Semantic Segmentation (Image-DAS)
Approaches for unsupervised domain adaptive semantic segmentation typically leverage one (or both) of two techniques: distribution matching and self-training. Distribution matching-basee methods attempt to align source and target domains in a shared representation space via techniques such as domain adversarial learning (Hoffman et al., 2016; Tzeng et al., 2017; Tsai et al., 2018; Vu et al., 2019). Some works additionally perform style transfer from source to target and ensure predictive consistency (Murez el al., 2017; Hoffman et al., 2018). In contrast, self- traininiining approaches iteratively generate pseudo-labels for unlabeled target examples from a source model and retrain on these. A number of techniques are used to improve the quality of these pseudo-labels, such as using prediction confidence (Prabhu et al., 2021b; Tan et al., 2020; Mei et al., 2020), augmentation consistency (Prabhu et al., 2021a; Melas-Kyriazi & Manrai, 2021). student-teacher learning (Tarvainen& & Valpola, 2018) and cross-domain mixing (Tranheden et al., 2020). The state-of-the- art method for this problem (Hoyer et al., 2022c;b;a) combines such pseudo-label-based self-training, with a SegFormer (Xie et al., 2021) backbone, in addition to auxiliary objectives for feature distance regularization and rare class sampling (Hoyer et al., 2022a), multi-rcsolution fusson (Hoyer et al., 2022b) and masked prediction (Hoyer et al., 2022c). In this work, we bross-benchmark these methods on Video-DAS datasels.
2.2 Video-level Unsupervised Domain Adaptation
Most prior works for video UDA are designed for the tasks of semantic segmentation and action recognition, and we now survey both these lines of work.
Semantic Segmentation. Given the strong progress in UDA for semantic segmentation, few works have explored how to integrate video information to improve performance, which we refer to as Video-DAS. DAVSN (Guan et al., 2021) was the first work to tackle this problem, and established two benchmarks: Viper to Cityscapes-Seg and Synthia-Seq to Cityscapes-Ser (Richter et al., 2017; Ros et al., 2016; Cordts et al., 2016), along with a method based on temporal consistency regularization realized by combining adversarial learning and self-training. A few works have since built upon this. Xing et al. (Xing et al., 2022) encourages spatio-temporal predictive consistency via a cross- frame pseudo-labeling strategy. Wu et al. (Wu et al., 2022) also encourages temporal consistency, and includes an image level domain discriminator. Shin et al. (Shin et al., 2021) uses two-stage objective of video adversarial learning followed by video self-training, whereas Cho et al. (Cho et al., 2023) optimizes temporally consistency cross-domain mixing objective, which aligns representations between the mixed and source domains. Finally, Lo el al. (Lo el al., 2023) uses spatio-temporal pixel-level contrastive learning for domain alignment. We summarize the differences of key mothods in Table 1, and benchmark against all methods in Table 2.
Action Recognition. Video UDA for action recognition has been studied extensively, and broadly falls under four categories (Xu et al., 2022): Adversarial, discrepancy- semantic-based, and reconstruction- based. Adversarial methods such as (Chen et al., 2019; 2022b; Pan et al., 2019) use video-level domain discriminators to align source and target domains, and aligns features at different levels by focusing key parts of the videos, for instance by introducing temporal co-attention to attend over key video frames (Pan et al., 2019). Other works explore alignment across additional modalities, such as optical flow (Munro & Damen, 2020) and audio (Yang et al., 2022). In contrast, discrepancy- based methods (Jamal et al., 2018; Gao et al., 2020) explicitly minimize feature distances between source and target video clips, while sermantic-baeed mothods employ contrastive objectives by crcating positive pairs from fast and slow versions of the same video, cutmix augmentations of a given clip (Sahoo el. al., 2021), or by leveraging auxiliary modalities (Song et al., 2021) (negatives are randomly sampled from other other videos or by shuffling frames). Finally, reconstruction-based approaches (Wei et al., 2023) learn to reconstruct video clips and separate task-specific features from domain-specific ones in the learned latent space.
Unfortunately, progress in Video-DAS and Image-DAS has been siloed, with each using a different and incomparable set of benchmarks, backbones, and algorithms. We address this gap by cross-benchmarking state-of the-art Image-DAS techniques on Video-DAS benchmarks, and additionally investigate the effec- tiveness of algorithms that combine techniques from both lines of work.
3

---

Published in Transactions on Machine Learning Research (01/2024)
Table 1: Characteristics of prior works in Video-DAS. While these works have additional components, we highlight the components which utilize sequential video frames.
Approach
Video Discrim. ACCEL
Consis. Mixup
PL. Refine
DAVSN (Guan et al., 2021)
MaxConf
UDA-VSS (Shin et al., 2021)
Consis
TPS (Xing et al., 2022)
Warp Frame
I2VDA (Wu al., 2022)
Custom
Moving Object Mixup (Cho et al., 2023)
/
Consis
3 Preliminaries
3.1 Video UDA for Semantic Segmentation (Video-DAS)
In unsupervised domain adaptation (UDA), the goal is to adapt. a model trained labeled source domain to unlabeled target domain. Formally, given labeled source examples (æs, (ss.ys) Ds and unlabeled target examples E Dr, we seek to learn model he with maximal performance on unseen target data. Further, we focus on the task of K-way semantic segmentation, where for input image x E IR xxxxWxx x3 we seek to make per-pixel predictions 9 E N#XWxK
In Video-DAS, the input data a series of Ns source videos and Ny target videos of length given by and Dy {[ 2rls | L...., Nr.. As additional input, we can estimate the optical flow Ot-t-& between frames xa and F*+* using trained flow prediction model. Pixels mapped logether by optical flow represent the same object at different points in time, so they should intuitively be assigned the same class. In the absence of target labels, Video-DAS methods seek to leverage this temporal signal.
Ds {[( (1)...)),..., Br)le i=
3.2 Overview of Video- DAS and Image- DAS
Review of Vidco-DAS methods
1) DA-VSN (Guan et al., 2021). Domain Adaptive Video Segmentation Network (DA-VSN) one of the first Video-DAS approaches, relying on a standard image discriminator and an additional video-level discriminator which acts on stacked features from consecutive frames. Further, optimizes unconfident predictions from frame H, to match confident predictions in the previous frame Tt-1-
2) TPS (Xing al., 2022). Temporal Pseudo Supervision (TPS) improves upon DAVSN via pseudo- label based self-training, which jointly optimizes source cross entropy and temporal consistency losses. The temporal consistency loss warps predictions made for the previous frame forward in time (via optical flow), and uses the warped predictions a pseudo-label for the augmented current frame.
3) STPL (Lo et al., 2023). STPL tackles the Video-DAS problem with an additional source-free constraint. Notably, they employ contrastive losses on However, we are unable to compare against their method as their code not public.
video features to obtain strong performance.
4) I2VDA (Wu et al., 2022). I2VDA only leverages videos in the target domain, via temporal consis- tency regularization: predictions ha(x)) and he(#++1) are merged in intermediate space, which then propagated to t+1 1 and supervised against the corresponding pseudo-label it+1 via cross entropy loss.
5) MOM (Cho et al., 2023). Moving Object Mixing (MOM) sets the current SOTA in Video-DAS and relied primarily on a custom class mix operation, which blends images from source and target domain. Coneretely, they extend class-mix vidcos with consistent mix-up which ensures that classmix is tempo- rally consistent. Further they, employ consistency-basee pseudo-label refinement, and image-level feature alignment between source and mixed domain images.
4

---

Published in Transactions on Machine Learning Research (01/2024)
Pred
Label
Image
Image t
Source
Data Aug
hg
Supervised Source Loss
Time:
Time: t+k
Optical
Imaget
Data Aug
hg
Target
Adaptation Loss
Consis Mixup
ACCEL
Time:
Time: t+k
Time:
Time: t*k
Pscudo-label Refinement
Pg
Seg. Network
Video Discriminator
Figure 2: Simplified Training Pipeline. Temporally separated frames are first augmented and then passed through a model hg to produce source and target predictions, which produce supervised and adap- tation losses. In this standard self-training pipeline, Video-DAS methods typically add one or more of the following techniques: consistent mixup, ACCEL, pseudo-label refinement, and video discriminators. We further elaborate on each of these techniques in Figure 3.
All of the works described above use Resnet-101 (He et al., 2016) backbone with DeepLabV2 (Chen et al., 2017) segmentation head, and most use the ACCEL (Jain et al., 2019) architecture. The ACCEL architecture is a wrapper, which separately generates sogmentations for 2e and Je+k: propagates 24t - xa via optical flow, and then fuses the two sets of predictions with a convolutional layer. Table provides summary of the primary components used in prior Video-DAS works: video discriminators, architectural modifications (ACCEL), consistent mixup, and pseudo-label refinement. For more details on each approach see Appendix A.2.
Image-DAS baseline. For our updated baseline, employ HRDA+MIC (Hoyer et al., 2022c), the culmination of a line of work (Hoyer et al., 2022a;b;c), which substantially improves the state-of-thh-art in Image-DAS. We choose this line of work due to its substantive gains over previous Image-Dad DAS methods, and note that numerous following Image-DAS works have also chosen to add on top of (Chen et al., 2022a; Vayyat et al., 2022; Xie al., 2023; Zhou et al., 2023; Ettedgui et al., 2022).
Concretely, HRDA+MIC combines i) teacher self-troe leff-training with rare class sampling and an ImageNet feature-distance-bese regularization objective, ii) nulti-Resolution Fusion (MRFusion) which enables mod- cls to train on full resolution images while maintaining manageable memory footprint by fusing predictions from high and low resolution crops, and iii) an auxiliary masked image consistency (MIC) objective which generates segmentation masks from a masked version of an image. For details see Appendix A.1.
3.3 Overview of Video-DAS techniques
In Table 1, we summarized key techniques from prior Video-DAS works, specifically, video domain discrim- inators, the ACCEL architecture (Jain et al., 2019), temporally-consiitent mix-up (Cho et al., 2023), and pseudo-label refinement. Now, we explain each of these techniques in depth (Fig, 3), as well how they are integrated into our overall pipeline (Fig. 2).
Preliminaries: We first define the optical flow based propagation operation (prop) that both ACCEL and pseudo-label refinement rely on. Intuitively, prop( 0(-4+*) aligns image Ii+* with based on the optical flow between the two frames 0 Intuitively, o a mapping from each pixel in o, to a unique pixel in 21tk- Mathematically, for cach pixel [i,]] we have
prop( 0 or- = letali + oy.j + oz]
where og and oy the and y components of or-tth- For simplicity, we will denote this with prop(*+**).

---

Published in Transactions on Machine Learning Research (01/2024)
(a) Consistent Mixup Accel
Seg. Network
Propagation via Optical flow
Car
Vegetation Sky Sign
t+k
Prediction
cony
Select Class
Source
Target Mixed
Score Fusion Layer
Copy Source Segments
Paste Source
Segments
Consis Mixup
ACCEL
(b) Video Discriminator
(c) Pseudo-label Refinement
Video Discrim
concat
Refined t
refine
hg
Optical
Prop
concal
Target
Video Discrim
Pseudo-label Refinement
Figure 3: Key Video Techniques. a) Consistent mix-up ensures that paired frames receive the same class mix-up, in this case "car" and "truck". The ACCEL architecture makes predictions ûs and Do+k, aligns them via prop( 0t-r+*), then fuses them with to distinguish whether temporally stacked features belong to the source or target domain, in conjunction with a feature encoder that is updated adversarially to make the two indistinguishable. c) Pseudo-label refinement improves predictions by fusing dt and Di+ based on one several criteria.
1 convolution. b) Video discriminators learn classifier
Video discriminator: Following (Tsai et al., 2018), jointly train auxiliary domain discriminator network D (to classify whether intermediate features belong to source or target sample), and fcature encoder to make source and larget features indistinguishable. Following (Guan el al., 2021), we feed in con- catenated features for temporally separated frames P loss (Tzeng et al., 2017):
and optimize a domain adversarial
min CD -->
Σ log D(P(as))) -E Σ log(1 D(P((T))
DcE
ET
min C, -
Σ log D(P((T))
xTEDríj
ACCEL: ACCEL (Jain et al., 2019) is video segmentation architecture which fuses predictions for consccutive frames. Specifically, we compute individual frame logits ûr ho(x;) and dL+* = -ig (*++)). The logits are then aligned via S = propp(r+a and stacked along the channel dimension (H, W, 2C). This is fused into a unified prediction (H, W, C) via a Ixl convolutional layer LACCEL(X,, (x,, xx+x) = convixt sonvix1f0 (fe During training, gradients are only backpropagated through the f(x,) branch. Further, note that this architecture assumes access both frames It and Lith at evaluation time. Following prior works in Video- Video-DAS, we share weights across hg and Ng-

---

Published in Transactions on Machine Learning Research (01/2024)
(a) Consistency
(b) Max Confidence
Pred g
Pred
Max Confidence Predictions
consistent
Condistent pirels
Refined Pred
Refined Pred
D.66
Pred t+k
-t
Pred t*k
max
0.2
0.9
0.9
Masked
incorsistent
Inconsistent
pizels
Pixel Confidence Score
(c) Warped Frame
(d) Oracle
Predi
Pred
Ignore Prediction
X
correct
Correct pixels
Oracie
Pred t+k .
Use Warped Prediction
Truth
incerrect pixels
incorrect
Figure 4: Each pseudo-label refinement strategy takes input. the current prediction ût, as well prediction for future frame Dr+z warped via optical flow the current frame, and merges them together to make refined prediction v'.- a) Consistency discards predictions that are inconsistent across frames. b) Max confidence selects the more confident prediction between frames. c) Warped frame uses the warped prediction instead of the current prediction. d) Oracle performs consistency based refinement, but with the ground truth label.
Consistent Mixup: Class-mixing has emerged as popular component of several UDA methods. Specif- ically, the mix operation mix(x1, 22, k) copies all pixels belonging to class k in x1 to the same position in ø2. Many UDA approaches utilize the class-mix augmentation across source and target images mix(a.s, (7, k), and correspondingly across source ground truth and target pseudo-labels mix(ys, 0r, k). Further, in order to correctly apply the class-mix augmentation on models which take 24 and Titk as input (such as ACCEL), (Cho et al., 2023) propose consistent mix-up, wherein they pick the same set of classes for both frames x, F++h so break optical flow correspondence.
Pseudo-label Refinement: Several Video-DAS methods utilize the redundancy in information between two consecutive frames to refine pseudo-labels. Broadly speaking, these strategies take pseudo-labed labels for two timesteps dt. dt+k. along with their optical flow 0--4th; and output an improved pscudo-label 8. Below we list a few common strategies (see Figure for an overview). For simplicity, we drop per-pixel indexing (+.1);
a) Consistency simply discards any pixel which inconsistent after optical flow propagation between ②t (rop((t+*)
te, lr*k and was used in (Cho et al., 2023), as follows: 06
NAN
Dr + prop (rop()r+*)
b) Max confidence selects the more confident prediction between St. Desk, similar to (Guan et al., 2021). Pr < prop(P:+k)
Let f. and Prok denote the corresponding top-1 confidence values, we set 8/ prop((e+*)
else

---

Published in Transactions on Machine Learning Research (01/2024)
c) Warp frame simply sets 8/ = prop((++)), intuitively using a future past frame's propagated prediction as a pseudo-label for the current frame. For k -1, this strategy matches (Xing et al., 2022). In our experiments, we test both k = 1 (forward warp) and & = -1 (backward warp).
d) Oracle is an upper bound strategy for pseudo-label refinement, which filters based on consistency with the ground truth y, given by: 8//
0t NAN
01
V/
Dt + y
4 Experiments
We now present our experimental setup. First, we compare the efficacy of Video-DAS methods with state- of-the-art Image-DAS methods (Sec. 3.2). Next, we explore hybrid UDA strategies that combine techniques from both lines of work (Sec. 4.3). To do so, we benchmark representative approaches for TPS (Xing et al., 2022), DAVSN (Guan et al., 2021), UDA-VSS (Shin et al., 2021) and MOM (Cho et al., 2023), as well as each key Vidco-DAS component individually. Finally, we explore psoudo-label refinement strategics across datasets, architectures, and frame distance (Sec. 4.5).
4.1 Experimental Setup
Datasets. Prior work in Video-DAS (Guan et al., 2021; Xing et al., 2022; Cho et al., 2023) studies adaptation on two popular benchmarks: Viper (Richter et al., 2016) to Cityscapes-Seg (Cordts et al., 2016) and Synthia- Seq (Ros et al., 2016) to Cityscapes-Seq.
Viper contains 13367 training clips and 4959 validation clips, each of length 10. Cityscapes-Seq contains 2975 training clips and 500 validation clips, each of length 30. Performance is measured over the 20th frame in each Cityscapes- See clip for which labels are available. Note that Viper is similar to the GTAV dataset (Richter et al., 2017) popularly used in the Image- DAS literature, as both are curated from the GTAV video game. However unlike the GTAV dataset, Viper contains sequentially labeled frames. Similarly, Cityscapes-Seq is a strict superset of the Cityscapes split used in Image-DAS works: while Cityscapes- Seq contains complete video clips from the original dataset and is largely unlabeled, Cityscapes only contains every 20th frame for which labels are available. Viper contains a total of 31 classes, of which 15 overlap with Cityscapes, so calculate mIoU over these 15 classes.
Tiper--Cityscapes-See
Synthia-Seq is considerably smaller synthetic dataset, with a total of 850 sequential frames. To mitigate the small size of the dataset, we train on every frame, consistent with prior work. Synthia-Seq contains total of 13 classes, and we calculate mloU over the 11 classes shared with Cityscapes.
Santhie-Seg--Citycappsss
BDDVid. We introduce support for a new target domain dataset derived from BDD10k (Yu et al., 2020), which to our knowledge has not been studied previously in the context of Video-DAS. BDD10k is series of 10,000 driving images across variety of conditions with semantic segmentation annotation. Of these 10,000 images, we identify 3,429 with valid corresponding video clips in the BDD100k dataset, making this subset suitable for Video-DAS. We refer to this subset as BDDVid. Next, we split these 3,429 images into 2,999 train samples and 430 evaluation samples. In BDD10k, the labeled frame is generally the 10th second in the 40-second clip, but not always. To mitigate this, we ultimately only evaluate images in BDD10k that perfectly correspond with the segmentation annotation, while at training time we use frames directly extracted from BDD100k video clips.
Metrics. Following prior work, report mean intersection-over-une validation set (which is identical to the Cityscapes validation set) and over the BDDVid validation set. We employ two evaluation paradigms popular in the literature: the first predicts y1 given a single frame ot, while the second does so given two frames (t-)) The dual frame approach is employed by subset of approaches that we describe in more detail in Sec. 3.3.
(mloU) over the Cityscapes-See Seq
Implementation Details. To run these experiments, we make number of contributions building off of the MMSegmentation (MMSeg) codebase (Contributors, 2020). We extend MMSeg to load consecutive frames
8

---

Published in Transactions on Machine Learning Research (01/2024)
Table 2: Existing approaches in Video-DAS fail leverage the recent advances in Image-DAS. Even Moving Object Mixing, the state of the art in Video-DAS, underperforms HRDA+MIC which is an image only method. All approaches use DeepLabV2 architecture with Resnet-101 backbone. Experiments that we ran are marked with a
Approach
Synthaa-Sq--CS--eg
Source"
36.7
30.1
DAVSN (Guan et al., 2021)
47.8
49,5
TPS (Xing et al., 2022)
48.9
53.8
Vidco-DAS
I2VDA (Wu et al., 2022)
51.2
53.0
STPL (Lo et al., 2023)
52.5
51.8
Moving Object Mixing (Cho et al., 2023)
53.8
56.3
HRDA* (Hoyer et al., 2022b)
65.5
76.1
Image-DAN
HRDA + MIC (Hoyer et al., 2022c)
68.3
75.3
Target* (upper bound)
83.0
84.9
To understand why HRDA+MIC (Hoyer al., 2022c) greatly outperforms many Vidco-DAS methods, we perform ablation study. All experiments a DeepLabV2 archilecture with Resnet-101 backbone.
Table 3:
Approach
Viper + CS-Seq Synthia-Ser + CS-Seq
Full approach (Hoyer et al., 2022c)
68.3
75.3
MIC
65.5
76. 76.14a.s
MRRusion
53.3 16.0
67.3 . 8.0
Rare class sampling
52.4 15.9
66.2
ImgNeet feature distance reg.
50.7 17.6
66.4 5.9
with their corresponding optical flows. The optical flows are generated by Flowformer (Huang al., 2022) Viper (Richter et al., 2017), Synthia-Seq (Ros et al., 2016), BDDVid, and Cityscapes-Seq (Cordts et al., 2016). In addition, provide implementations of many Video-DAS techniques such video discriminators, consistent mix-up, and various pseudo-label refinement strategies.
MMSeg enables researchers to easily swap out backbones, segmentation heads etc., which will help keep Vidco-DAS techniques up to date in the future. Throughout the paper we deploy two base architectures following prior work: 101= (He al., 2016; Chen et al., 2017) and Segformer (Xie et al., 2021). After running learning rate sweep we find the parameters used in HRDA+MIC (Hoyer al., 2022c) work best and retain the AdamW optimizer (Loshchilov & Hutter, 2019) with learning rate of 66-5 (encoder), 6c-4 (decoder), batch size of 2, 40k iterations of training, and linear decay schedule with warmup of 1500 iterations. For all experiments, we fix the random seed to 1.
4.2 How do Video-DAC methods compare to updated Image-DAS baselines?
Table presents results. We find that:
SoTA Image-DAS methods outperform specialized Video-DAS methods. Specifically, HRDA+MIC beats the state-of-the-art in Vidco-DAS (MOM) by 14.5 mloU on Viper-Cityscapee-See and 19.0 mloU on SymlriaaSeey--iiyyccppsss does so without explicitly leveraging the temporal nature of the data, while the Video-DAS approaches explicitly exploit it. Further, the improvements of HRDA+MIC are most pronounced on rare classes like traffic light (+33.7 mloU Viper, +46.3 mloU Synthia-Seq), traffic sign (+25.8 mloU Viper, +37.3 mloU Synthia-Seq) and bicycle (+24.0 mloU Viper). We present all class-wis wise ToUs in Tables 10 and 11. Note
even after controlling for model archilecture. Importantly, it

---

Published in Transactions on Machine Learning Research (01/2024)
Table We test adding existing Video-DAS methods to our Image-DAS baseline (HRDA with Dee)LabV2). as well as additional custom combinations of the key tochniques used in prior works. We bold the best performing method and underline the second-best. Vid=Video Discriminator, ACCEL= Mixup, PL=Pseudo-label Refinement, Syn=Syuthia-Ser Viper and BDD=BDDVid
Name
Vid ACCEL
PL
Vip
Syn CS
Vip Vip-BDD
Syn
Source
None
36.7
30.1
36.6
25.4
HRDA (baseline)
None
65.5
76.1
49.1
48.1
+ TPS
Warp Frame
64.9
72.9
49.3
51.0
+ DAVSN
MaxConf
65.3
75.2
49.7
52.1
UDA-VSS
Consis
64.3
76.9
50.5
50.7
MOM
Consis
65.9
75.2
47.6
51.8
Custom
None
63.4
75.9
50.6
49.6
Custom
None
66.9
76.6
50.0
50.3
+ Custom
Consis
65.8
76.5
49.7
51.6
Custom
None
63.7
76.1
48.9
49.7
+ Custom
Consis
65.9
75.2
47.6
51.8
+ Custom
Consis
64.3
76.9
50.5
50.7
+ Custom
Consis
66.0
76.7
49.6
51.5
Target (Upper Bound)
86.1
87.2
55.8
59.9
that in Table 2 which compares against prior work, we evaluate two established benchmarks, but in Table 4 we construct additional shifts to draw stronger conclusions.
Multi-rcsolution fusion (MRFusion) is the biggest contributor improved performance. To explain the strong gains obtained by HRDA+MIC, systematically strip away components from the full approach. A large portion of the performance gap is due to HRDA's 's MRFusion strategy (+12.2 mIoU Viper-CCtyscapes-SSg, 8.8 mIoU Synthia-CCityscapos-Soo This component allows HRDA to utilize full-resolution images whereas prior Video-DAS works are forced to down-sample their images to satisfy memory constraints. Additionally, the remaining components (MIC, RCS and ImageNet Feature Distance) each contribute a few points performance improvement.
perform an ablation study in Table 3 wherein we
A natural question that arises from our analysis is whether we can merge the key techniques from Image- DAS and Video-DAS to further boost performance. We study this in the next subsection.
4.3
Can we combine techniques from Image-DAS and Video-DAS to improve performance?
We now evaluate whether the state-of-the-art image baseline (Hoyer et al., 2022c;b) can be further improved by incorporating the key techniques from Video-DAS. To reduce the computational burden, we use HRDA (rather than HRDA + MIC) as our starting point. Specifically, we try:
(a) Adding a video discriminator to the HRDA objective, total optimizing MLade + CHRDA (we find to work the best).
(b) Swapping out our architecture with ACCEL (Jain et al., 2019).
(c) Adding consistent cross-domain mix-up by ensuring that 2i, Tz4 are given the same classmix aug- mentation. Note that consistent mixup is meaningful only when using multi-frame input architec- ture such as ACCEL, and so we always use the two in conjunction.
(d) A modified version of HRDA that uses refined pseudo-labels for self-training, where refinement is performed using of the strategies described above.
10

---

Published in Transactions on Machine Learning Research (01/2024)
Temporal Consistency of Predictions
100
with MR Fusion
80
PL-PredConsit-mlll
80
40
Building
Vege
Sky
Bicycle M.sycle
Avg
Light
Sign
Class
Figure 5: Temporal Consistency of Predictions (PL-PredConsis--oU) with and without MRFusion. DeeplabV2 backbone trained on Viter--Citysrapps-SS
Results. In Table 4, we benchmark combinations of HRDA with prior Video-DAS works, addition to other valid combinations the common techniques identified in Video-DAS. Note that most prior Video- DAS methods can be constructed using a combination of one or more of the techniques mentioned above. These approaches
meant to be naive way to updale prior Video-DAS work to leverage the strong HRDA baseline. As seen, technique outperforms the rest all datasets. However, do consistent gains across datasets by leveraging ACCEL (0.5-2.2 mloU) and consistency based pscudo-label refinement (0.3-3.5 mloU), with the largest improvements on the Synthia-Seg-BDDVas shift. While these initial improvements are promising, neither pseudo-label refinement. nor ACCEL dominate across datasets. addition, while at the upper bound we see an improvement 3.5 mloU, on many shifts this doesn't 't hold, and we see improvements of just 0.3 mloU. And for perspective, all these gains are relatively small compared the image level interventions observed in Table 3. Thus, we encourage future work to take these learnings to craft approaches with more significant improvements across multiple datasets.
4.4 Why is it difficult to combine Image-DAt DAS and Video-DAS?
We now propose redundancy hypothesis to explain why Video-DAS techniques do not further improve performance on top of state-of-the-as Image-DAS methods. We hypothesize that HRDA's multi-resolution fusion (MRRFusion)-based self-training strategy leads to predictions that are not only highly accurate but also highly temporally consistent, obviating the need for further temporal refinement. To validate this, we first define a custom metric PL-PredConsis, which estimates the temporal predictive consistency of pseudolabels by computing mloU between pseudolabels for the current frame ih and warped pseudolabels for future frame 8% prop((t4x). as:
PL-PreaCCosiis-mIollfuuu 0r) mIoU(H 9.)
In Figure 5, and Table 9 of the appendix, we report per-class and mean PL-PredConsis values for HRDA models trained without (lop row) and with (bottom row) MRFusion as seen, models trained with MRFusion exhibit significantly higher temporal consistency than those trained without (89.9 vs 78.4 on Viper->Cityscapee-Seg and 86.7 vs 83.3 on gies typically seek to leverage temporal signals from adjacent frames to improve model predictions for the current frame however, in the presence of MRFusion temporal consistency is already so high that there is
Recall that Vidco-DAS strate-
11

---

Published in Transactions on Machine Learning Research (01/2024)
Pseudo-label Refinement Over Increasing Frame Distances
Refinement
Acc After Refinement
100
PL Warped mloU
PL Warped Acc
PL Consistency
PL Consistency Acc
% Pixels Recained
96 Pixels Retained
85
80
75
As frame distance increases, warped metrics decrease, consistency metrics stay same, and fewer pixels remain
70
65
60
0
6
2
4
10
Frame Distance
Figure 6: Pseudo-label refinement. over larger frame distances on Viper-CCttsccpee-SSer At larger frame distances, we observe drop in warped metries and percentage of pixels retained, while con- sistency metrics remain stagnant. This suggests that pscudo-label refinement worse over larger frame distances.
simply not much temporal signal to leverage. Further, MRFusion is designed to improve recognition small objects by leveraging detail from high-resolution inputs accordingly we find that that temporal consistency is greatly improved across these classes like 'eence', 'traffic sign', 'traffic light!.
4.5 Exploring pseudo-label refinement strategies
In light of the limited improvements offered by consistency-based pseudo-label refinement as found in the previous subsection, we now perform a comprehensive analysis of pseudo-label refinement strategies. Notably, perform analysis stronger architectures like Segformer (Xie et al., 2021), because pseudo-label refinement is less memory intensive than ACCEL video discriminators, which benchmark in Table 4.
We report results across architecture in Tables 5-8 of the appendix. We find that:
Pscudo-label refinement is more effective on Synthia-Seq than Viper. We find that on Synthia- Seq, pseudo-label refinement improves performance in 3/4 settings (0.7-2.4 mIoU improvement), in fact, improving upon the state-of-the-art using backward flow and max confidence (77.9 vs 76.6 mIoU, Table 5). On Viper, we observe less consistent improvements, with +4.6 mloU HRDA Segformer, and +0.3 mloU HRDA De One possible explanation is that pseudo-label refinement is more effective when there is less labeled data as is the case with Synthia-Seq, but we leave further analysis to future work.
Oracle pseudo-label refinement lags behind fully supervised performance. Across settings, we observe a considerable gap between the HRDA baseline and the "target only" oracle approach (on average +14.3 on Siper--Ctyyccape-SSee and +11.0 on refinement, this gap shrinks considcrably (on average +4.9 on Viper-CCitssapee-SSer and +2.7 on Synthia- Beg-+Cilysccpes--Seq) suggesting that while more precise pseudo-label refinement methods can improve performance, perfect refinement is insufficient to match supervised performance.
With oracle pseudo-label
Refining pscudo-labels via traditional methods across larger frame distances hurts. Finally, we investigate whether refinement over larger frame distances produces stronger pseudo-labels. All prior Video-DAt DAS works use frames It, Tt-1, Zt-2 for pseudo-label refinement, whereas we consider the impact of using Z, Te4k for (1,3,6, 6, 10). To investigate whether using larger frame distances beneficial, we take trained HRDA Segformer model and run evaluation with two new custom metrics, PL-Warped and PL-Consistency. PL-Warped takes future prediction Ôet and current, ground truth y4- We warp Ôet*
12

---

Published in Transactions on Machine Learning Research (01/2024)
onto the current frame to generate 0/ prop(Or+k) We then define:
PL-Warped (oo M.) mIoU( yr)
PL-Warped yo) Acc yr)
A warped metric measures whether future frames provide more accurate pseudo-labels than the current frame, assuming perfect propagation. This intuitively tests if getting closer to object improves predictions it. Note Acc is class-wist wise average accuracy. Similarly, consistency metric helps us measure whether temporally consistent predictions are likely to be correct.
PL-CoosiistencyymIIl00- Dt, ye) = mIoU(( = Dc). yc) PL-Consistencyy-Ac((-t Dz, Ut) = Acc((f/ == (z), ve)
As shown in Figure 6, we find that pseudo-label refinement becomes less effective over larger frame distances: PL PL-Warped quickly decreases with increasing frame distances, while PL-Consistency remains stagnant even as the proportion of pixels retained after filtering decreases.
5 Discussion and Future Work
The field of Vidco-DAS is currently in an odd place. Image-based approaches have advanced so much that they simply outperform video methods, largely because of techniques such as multi-resolution fusion. However, certain Video-DAS techniques still show occasional promise, such as ACCEL and pseudo-label refinement. Further, while we find that no single pscudo-label refinement strategy performs well across settings, such approaches are generally more effective on Synthia suggesting suitability to the the low data setting. However, initial exploration leads us to believe that static, hand-crafted refinement strategies based on heuristics are likely too brittle to consistently improve performance, and adaptive, learning-basee approaches (like ACCEL) perhaps require more consideration. Further, while we show Image-DAS methods to be far superior to Video-DAS methods for sim-to-real semantic segmentation adaptation, we leave to future work an investigation into whether this holds more generally. Perhaps Video-DAS will only lead to consistently significant improvements for segmentation on harder domain shifts (such as BDDVid). Further, in tasks like action recognition which necessitate model to understand series of frames, video may be necessary.
To accelerate research in Video-DAS, we open source code built off of MMSegmentation, and provide multi-frame support I/O, optical flow, new domain shifts, and baseline implementations of key Video-DAS mothods. We believe that the lack of a standardized codebase for Vidco-DAS is one of the primary reasons for the siloed progress in the field, and hope that with our codebase future work can both easily cross- benchmark methods and develop specialized Video-DAS methods built. on top of the latest and greatest. image-level semantic segmentation.
References
Liang-Chis Chen, George Papandrcou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Decplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
Min- Hain-Hnng Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Woo, Ruxin Chen, and Jian Zheng. Temporal atten- tive alignment for large-scale video domain adaptation. In IEFE International Conference on Computer 12743.
Vision (ICCV), 2019. URL httpa////axxvv.oa//e
Mu Chen, Zhedong Zheng, Yi Yang, and Tat-Seng Chua. Pipa: Pixel- and patch-wise self-supervised learning for domain adaptative semantic sogmentation, 2022a.
Peipeng Chen, Yuan Gao, and Andy J. Ma. Multi-level attentive adversarial learning with temporal dilation for unsupervised video domain adaptation. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 776-785, 2022b. doi: BAA.1///WA555155..000000
13

---

Published in Transactions on Machine Learning Research (01/2024)
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cmn for object detection in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3339 3348, 2018.
Kyusik Cho, Suhyeon Lee, Hongje Scong, and Euntai Kim. Domain adaptive video semantic segmentation via cross-domain moving object mixing. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 489-498, 2023.
MMSSggmentatio Contributors. Mmsegmentation: Openmmlab semantie segmentation toolbox and bench- mark, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. In CVPR, 2009.
Shahaf Ettodgui, Shady Abu-Husscin, and Raja Giryes. progressive cyclic style-transler, 2022.
Procst: Boosting semantic segmentation using
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- lional conference on machine learning, pp. 1180- 1189. PMLR, 2015.
Zan Gao, Leming Guo, Tongwei Ren, An-An Liu, Zhi-Yong Cheng, and Shengyong Chen. Pairwise two- stream convnets for cross-domain action recognition with small data. IEEE Transactions on Neural Networks and Learning Systems, PP:1 15, 12 2020. doi: 10.1109/TNNLS 2020 30440.0.
Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian La. Domain adaptive video segmentation via tempo- ral consistency regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8053 8064, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Judy Hoffman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens the wild: Pixel-level adversarial and musstraint-based adaptation, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, pp. 1994-2003, 2018.
Lukas Hoyer, Dengxin Dai, and Lue Van Gool. Daformer: Improving network architectures and training strategies for admain-adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognátion, pp. 9924-99299999 9935, 2022a.
Lukas Hoyer, Dengxin Dai, and Lue Van Gool. Hrda: Context-aware high-resolution domain-adaptive semantic segmentation. In Computer Vision-ECCT" 2022: 17th European Conference, Tel Aviu Israel, October 23 27, 2022, Proceedings, Purt XXX, pp. 372 391. Springer, 2022b.
Lukas Hoyer, Dengxin Dai, Haoran Wang, and Lue Van Gool. Mic: Masked image consistency for context- enhanced domain adaplation. arXiv preprint arXiv:2212.01322 2022c.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer archilecture for optical flow, 2022.
Samvit Jain, Xin Wang, and Joseph Gonzalez. Accel: corrective fusion network for efficient semantic segmentation video, 2019.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Arshad Jamal, Vinay P. Namboodiri, Dipti Deodhare, and K. S. Venkatesh. Deep domain adaptation in action space. In British Machine Vision Conference, 2018. URL https://apl wwmanticschollattee org/
Tarun Kalluri, Wangdong Xu, and Manmohan Chandraker. Geonet: Benchmarking unsupervised adapla- tion across goographics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15368 15379, 2023.
Yu-Jhe Li, Xiaoliang Dai, Chih-Yao Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He, Kris Kitani, and Peter Vajda. Cross-domain adaptive teacher for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7581-7590, 2022.
Shao-Y Yuan Lo, Poojan Oza, Sumanth Chenmupati, Alejandro Galindo, and Vishal M. Patel. Spapio-temporal pixel-level contrastive learning-based source-free domain adaptation for video semantic segmentation, 2023.
Rya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Ke Mci, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self- self-training for unsupervised domain adaptation. 2020.
Djebril Mekhazni, Amran Bhuiyan, George Ekladious, and Eric Granger. Unsupervised domain adaptation in the dissimilarity space for person re-identification. In Computer Fision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16, pp. 174-174. Springer, 2020.
Luke Kyras-Kyriazi and Arjun K. Manrai. Pixmatch: Unsupervised domain adaptation via pixelwise con- sistency training, 2021.
Jonathan Munro and Dima Damen, Multi-modal Domain Adaptation for Fine- Fine-grained Action Recognition. In Computer Vision and Pattern Recognition (CVPR), 2020.
Zak Murez, Soheil Kolouri, David Kriegman, Ravi Ramamoorthi, and Kyungnam Kim. Image to image translation for domain adaptation, 2017.
Boxiao Pan, Zhangjie Cao, Ehsan Adeli, and Juan Carlos Niebles. Adversarial cross-domain action recogni- tion with co-attention. CoRR, abs/ Abs/1912.10405,, 2019. URL http://arxiv. http://arrdv.or//bb////a http://///111..1////
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Augco: augmentation consistency-guided self-training for sourco-free domain adaptive semantic segmentation. arXiv preprint arXw::2107.10140, 2021a.
Viraj Prabhu, Shivam Khare, Decksha Kartik, and Judy Hoffman. Sentry: Selective entropy optimiza- tion via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558 8867,5677 2021b.
Viraj Prabhu, Ramprasaath R Selvaraju, Judy Hoffman, and Nikhil Naik. Can domain adaptation make object recognition work for everyone? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3981-3988 2022.
Stephan Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pp. 102-118. Springer, 2016.
Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun. Playing for benchmarks. Tn Proceedings of the IEEE International Conference on Computer Vision, pp. 2213 2222, 2017.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia datasct: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 3234 3243, 2016.
Kate Sacnko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213 226. Springer, 2010.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Aadarsh Sahoo, Rutav Shah, Rameswar Panda, Kate Saenko, and Abir Das. Contrast and mix: Temporal contrastive video domain adaptation with background mixing. Advances in Nearal Information Processing Systems, 34:23386 23400, 2021.
Inkyu Shin, Kwan Yong Park, Sanghyun Woo, and In So Kweon. Unsupervised domain adaptation for video semantic segmentation. CoRR, abs abs/2217.11152. 2021. URL https: httt:://aaxiv..rrg/bb///// 11052.
Xiaolin Song, Sicheng Zhao, Jingyu Yang, Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio- temporal contrastive domain adaptation for action recognition. In 2021 IEEE/CVE Conference on Com-
puter Vision and Pattern Recognition (CVPR), pp. 9782-9790, 2021. 00966.
doi: 10. 10.109/CVPPRA6:3:20221
Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbllanced domain adaptation: An empirical odyssey, 2020.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, 2018.
Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross-domain mixed sampling, 2020.
Y.-H. Tsai, W.-. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic sogmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 7167-7176, 2017.
Midhun Vayyat, Jaswin Kasi, Amuraag Bhattacharya, Shuaib Ahmed, and Rahul Tallamraju. Cluda Contrastive learning in unsupervised domain adaptation for semantic segmentation, 2022.
VS Vibashan, Poojan Oza, and Vishal M Patel. Instance relation graph guided source-free domain adaptive object detection. In 2023 IEEE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3520 3530. IEEE, 2023.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 2517-2526, 2019.
Pengfei Wei, Lingdong Kong, Xinghua Qu, Yi Ren, Zhiqiang Xu, Jing Jiang, and Xiang Yin. Unsupervised video domain adaptation for action recognition: A disentanglement perspective, 2023.
Xinyi Wu, Zhenyao Wu, Hao Guo, Lili Ju, and Song Wang. Dannet: A one-stage domain adaptation network for unsupervised nighttime semantic segmentation. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 15769 15778, 2021.
Xinyi Wu, Zhenyao Wu, Jin Wan, Lili Ju, and Song Wang. Is it necessary to transfer temporal knowledge for domain adaptive video semantic segmentation? In European Conference on Computer Vision, pp. 357-373. Springer, 2022.
Binhui Xie, Shuang Li, Mingjia Li, Chi Harold Liu, Gao Huang, and Guoren Wang. Sepico: Semantic-guided pixel contrast for domain adaptive semantic segmentation, 2023.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34:12077-12090, 2021.
Yun Xing, Dayan Guan, Jiaxing Huang, and Shijian Lu. Domain adaptive vidco segmentation via temporal pseudo supervision. In Computer Vision-ECCS ECCon 2022: 17th European Conference, Tel Aviu, Isruel, October 23-27, 27, 2022, Proceedings, Part XXX, pp. 221-639. Springer, 2022.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Yuecong Xu, Haozhi Cao, Zhenghua Chen, Xiaoli Li, Lihua Xie, and Jianfei Yang. Video unsupervised domain adaptation with deep learning: A comprehensive survey, 2022.
Lijin Yang, Yifei Huang, Yusuke Sugano, and Yoichi Sato. Interact before align: Leveraging cross-mode modal knowledge for domain adaptive action recognition. In Proceedings of the IEFE/CVE Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 14722-14732, June 2022.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: diverse driving dataset for heterogencous multitask learning. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, pp. 2636 2645, 2020.
Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe. Learning to generalize unseen domains via memory-basee based multi-source meta-learning for person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6277-6286, 2021.
Qianyu Zhou, Zhengyang Feng, Qiqi Gu, Jiangmiao Pang, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Context-aware mixup for domain adaptive semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 33(2):804-88) 817, feb 2023. doi: 10. 11110//csvt.2022.33000 20t.::tcc..t.22.3300.0.. URL https: / //do/////oi/o/ owg//o.////. 110..2Ftatttt.t......... 2022. 3206476...
17

---

Published in Transactions on Machine Learning Research (01/2024)
A Appendix
A.1 Image-DAS Methods
HRDA+-MIC falls under the self training style of UDA. The core training objectives are to minimize cross entropy loss on source domain images with ground truth labels, as well as cross entropy on target domain images with pseudo-labels. At each iteration pseudo-labels are generated via a teacher network 6 which is defined as an EMA update of the student following (Tarvainen & Valpola, 2018).
HRDA introduced Multi Resolution Fusion (MRFusion); which fuses high res and low image crops, allowing the model to use high resolution details with manageable GPU footprint. The context and detail predictions dc, Da are resized with scale then fused with learnable attention module a.
" = ((1 a a) o (e.8) + + (a, 8) o Ba.
MIC additionally introduced a masked reconstruction objective, which attempts to predict segmentation maps from masked target images M. Given pseudo-label p" Φ(x), pseudo-weight q, and masked prediction gM ((*
Line gLoc ,P""
ImageNet feature distance regularizes souree domain predictions based on ImageNet (Deng et al., 2009) representations. Finally, Rare Class Sampling encourages drawing source domain examples containing rare classes. A class c is drawn with frequency P(c) where fe the proportion of pixels with label c
e0--///T
P(c)
Ee=1 el1-fo//T
A.2 Video-DAS Methods
An early work in Vidco-DAS presents DAVSN (Guan et al., 2021), which broadly falls under the domain discriminative style of UDA. Their approach has an image discriminator, video discriminator, and a di- vergence loss between these discriminators. The image discriminator simply tries to distinguish source vs target features. Similarly the video discriminator tries to distinguish source video features and target video features. These vidco features are simply stacked features from two consecutive frames. The divergence loss helps each discriminator focus on different information. In addition to these losses, DAVSN explicilly optimizes for unconfident predictions in the current frame to match confident predictions from the previous frame (G) 5 (E (pT) E (fT IoT pT-1l-
Another early work Video-DAS presents UDA-VSS (Shin et al., 2021), an approach which relies on a standard video discriminator, and self training algorithm. The self training algorithm notably aggregates predictions from the future and previous frames, to improve pscudo-label generation. And these pscudo- labels are further refined by ignoring pseudo-labels predictions which are inconsistent between neighboring frames, via optical flow alignment. Note UDA-VSS reports on Viper to Cityscapes- VPS and Synthia-Seq to Cityscapes- VPS, which is slightly different that Cityscapes-Seq. Thus we cannot directly compare to the reported numbers in Table 2.
DAVSN is improved upon TPS (Xing et al., 2022) by adopting pscudo-label based self training. There are only two losses, source cross entropy, and temporal consistency. The temporal consistency loss simply uses the previous frame warped forward in time via optical flow as pseudo-label for the augmented current frame.
STPL (Lo et al., 2023) tackles the Video-DAS problem, but operates in the source free DA setting. Notably they employ rastive losses on spatio-temporal video features. However, they do not have code available, so we do not benchmark their approach.
I2VDA (Wu et al., 2022) tackles the same Video-DAS problem, but designs an approach which only leverages videos in the target domain. Their key technique temporal consistency regularization. First predictions
18

---

Published in Transactions on Machine Learning Research (01/2024)
Table 5: A comparison of pseudo-label refinement strategies with Segformer backbone and HRDA MIC.
We bold the best refinement strategy and underline the second bost. Refinement helps Synthia-Seg but not Viper.
Name
Flow Direction
PL Refine
Synthia-Seq- Viper-+CSSeq
Source
46.1
52.0
MMC
76.4
76.6
Custom
Backward
warp frame
74.4
75.5
Custom
Backward
consis
75.0
76.0
Custom
Backward
maxconf
74.0
77.9
Custom
Forward
warp frame
73.2
72.6
Custom
Forward
consis
74.0
73.6
Custom
Forward
maxconl
75.3
74.2
Oracle
80.0
84.2
Target
86.1
87.2
Table 6: comparison of pseudo-label refinement strategies with Segformer backbone and HRDA (No MIC). We bold the best refinement strategy and underline the sccond best. Refinement improves both Viper and Synthia-Seq benchmarks.
Name
Flow Direction
PL Refine
Viper->CSSoq
Synthia-Seq
Source
46.1
52.0
HRDA
70.7
72.4
Custom
Backward
warp frame
71.5
69.0
Custom
Backward
consis maxconf
74.8
74.8
Custom
Backward
70.7
74.4
Custom
Forward
warp frame
74.3
69.4
Custom
Forward
consis
75.3
74.5
Custom
Forward
maxcon!
73.9
73.2
Oracle
82.3
83.8
Target
86.1
87.2
fxx)) and (x++1) are merged in an intermediate space, then they are propagated to + and compared against the corresponding pseudo-label Ūc+1.
The current state of the art is set by Moving Object Mixing (MOM) (Cho al., 2023), which differentiates itself via consistent mixup and contrastive objective. Classmix is augmentation in which pixels from a source image with labels in Gelassmix are pasted onto a target image Consistent mixup simply ensures that the same Celassmix is chosen for both It and Ti+k, where is the frame distance. The contrastive objective pulls together representations of given class across source and mixed domain images. Given a mixed domain image, MOM filters out pixels which are inconsistent between two frames using optical flow. Then a feature centroid fo is calculated for each instance the mixed image. Similarly, in the source domain bank of feature centroids is stored for each class. MOM then performs L1 loss between fe, and the most similar feature in the bank of source domain features for class
A.3 Pseudo-label Refinement Results
We explore pseudo-label refinement across number of dimensions: architecture, dataset, flow direction and refinement strategy. Tables 5, 6, and 8 correspond to different architectures: HRDA+MIC on Segformer, HRDA on Segformer, -IRDA-MIC on DIV2, HRDA on DLv2.
19

---

Published in Transactions on Machine Learning Research (01/2024)
Table 7: A comparison pseudo-label refnement strategies with DeepLabV2 backbone and MIC. We bold the best refinement strategy and underline the second best. In this refinement is worse than the bascline.
Name
Flow Direction
PL Refine
Viper->CSSeq Synthia-Seq -
Source
36.7
30.1
HRDA MIC
68.3
75.3
Custom
Backward
warp frame
62.9
71.1
Custom
Backward
consis maxconf
67.7
74.7
Custom
Backward
65.0
73.8
Custom
Forward
warp frame
67.2
72.3
Custom
Forward
consis
66.8
74.5
Custom
Forward
maxconf
68.2
71.9
Oracle
78.9
82.8
Target
83.0
84.9
Table 8: A comparison of pseudo-label refinement strategies with DeepLabV2 backbone and HRDA (no MIC). We bold the best refinement strategy and underline the second best. Refinement helps slightly on both benchmarks.
Name
Flow Direction
PL Refine
Viper CSSee Synthia-Seg CCSeq
Source
36.7
30.1
HRDA
65.5
76.1
Custom
Backward
warp frame
62.9
73.8
Custom
Backward
consis
65.8
76.8
Custom
Backward
maxconf
63.9
75.7
Custom
Forward
warp frame
63.7
71.7
Custom
Forward
consis
64.5
74.9
Custom
Forward
maxconf
64.1
73.9
Oracle
77.3
82.7
Target
83.0
84.9
A.4 Temporal-Conssitenny Results
Here we include the temporal consistency results on the Sputhaa-ee---CiyycappsSeeg shift as well
Table 9:
Temporal Consistency of Predictions (PL-PredConmsis-IoO)) with and without MRFusion.. DeeplabV2 backbone trained on Syethia-Seq SeyssappssssssseSSSSess
Method road side buil pole light sign veg sky per rider car PL-PredConsii-mmol
No MRFusion 87.6 83.0 88.3 66.1 72.4 77.3 92.6 94.4 79.6 81.3 93.6
83.3
HRDA
96.4 88.1 94.0 67.6 75.3 81.2 95.0 96.7 81.0 83.7 95.0
86.7
A.5 Class-wise loUs
In Tables 10 and 11 we provide the classwisc IoUs for the HRDA+MIC bascline and each prior work in MASe controlling for architecture.
20

---

Published in Transactions on Machine Learning Research (01/2024)
Table 10: Classwise loUs on VSSSe. All methods use Resnet101+DDLY
Method
road side buil fen light sign veg
ter
sky
per
truck bus mot bic
mloU
HRDA+MIC
95.5 71.5 91.2 21.1 66.2
73.1
89.8 47.3
92.3 79.6 89.7 48.6
54.9 38.5 64.6
68.3
MOM
89.0 53.8
86.8
31.0 32.5
47.3
85.6 25.1
80.4
65.1 79.3 21.6
43.4
25.7
40.6
53.8
STPL
83.1
38.9
81.9
48.7
32.7
37.3
84.4
23.1
64.4
62.0
82.1 20.0
76.4
40.4
12.8
52.5
I2VDA
84.8 36.1 84.0
28.0
36.5
36.0
85.9
32.5
74.0
63.2
81.9 33.0
51.8
39.9
0.1
51.2
TPS
82.4
36.9
79.5
9.0
26.3
29.4
78.5
28.2
81.8
61.2
80.2
39.8
40.3
28.5
31.7
48.9
DAVSN
86.8 36.7
83.5
22.9
30.2
27.7
83.6
26.7
80.3
60.0
79.1 20.3
47.2
21.2
11.4
47.8
Table 11: Classwise ToUs on Saathha-Seq--SSSSSe All methods use Resnet 101 DLV2.
Method
road
side
buil
pole
light
sign
veg
sky
per
rider
car
mIoU
HRDA+MIC
94.5
65.6
91.4
49.7
62.6
66.9
90.2
91.8
78.2
45.8
91.2
75.3
MOM
90.4
39.2
82.3
30.2
16.3
29.6
83.2
84.9
59.3
19.7
84.3
56.3
STPL
87.6
42.5
74.6
27.7
18.5
35.9
69.0
55.5
54.5
17.5
85.9
51.8
I2VDA
89.9
40.5
77.6
27.3
18.7
23.6
76.1
76.3
48.5
22.4
82.1
53.0
TPS
91.2
53.7
74.9
24.6
17.9
39.3
68.1
59.7
57.2
20.3
84.5
53.8
DAVSN
89.4
31.0
77.4
26.1
9.1
20.4
75.4
74.6
42.9
16.1
82.4
49.5
A.6 Impact Multiple Frames on Image-DAS
In order compare fairly between an Image-DAS method which uses only one frame per clip, and a Video- DAS method which multiple, we should allow the Image-DAS method to access the additional frames well. Interestingly in Table A.6 we find that the Image-DAs DAS methods actually perform worse when we naively add these additional frames to the training set. We anticipate this is because of the strong redundancy between frames. Regardless, throughout the paper we restrict our Image-DAe DAS methods to only using single frame since it performs better.
We evaluated the effect including additional unlabelled frames as part of the target dataset, to fairly compare MIC to approaches which utilize both o, and Zz44. We find that simply including It, I in our target dataset actually hurts performance.
Table 12:
Approach
Target Frames
mIoU
HRDA DLV2
only
65.5
HRDA DLV2
1,1+1
64.4
HRDA+MIC DLV2
only
68.3
HRDA+MIC DLV2
t,t ++1
65.0
21


Reviews:
[{"review_id": "obsJ0ZUzYE", "reviewer": "Reviewer_P6Qq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Although video recognition has been shown before to be challenging to benefit from video-specific signals significantly beyond still-image signals of the individual frames, the margin in the results are indeed surprising and regardless this can be quite informative for the development of the important subfields of both machine learning (domain adaptation of temporal data) and computer vision (video semantic segmentation) \n- there is an open-source code released for the benchmarking of Video-DA on image and video semantic segmentation using the implementation of the various components of the state-of-the-art techniques.\n- several recurring components of Image-DA are ablated to find the most effective part of the state-of-the-art Image-DA for Video-DA with clear take-away regarding the high-resolution processing being a key factor.\n- studies the effectiveness of adding video-specific components to state-of-the-art Image-DA.\n\n**Weaknesses**\n- there is only one line of work for Image-DA that is tested on videos. Those line of works already clearly outperform the prior methods on Image-DA by a relatively large margin which could, to some extent, explain the observation. For various reasons including better understanding of with what method the temporal information might not be as useful it would be good to include other image-DA baselines and/or components.\n- the benchmark test-bed is quite limited: only semantic segmentation as the task, only autonomous vehicles as the domain, only synthetic-to-real as the source-target setup, and only GTA as the synthesizer and city-escapes as the real-world target. Despite this clear limitation, the results are probably already interesting but the statements throughout the paper should be accordingly delimited so it doesn’t read as if it is the general video domain adaptation that is the subject of the study. Perhaps action recognition could be used and/or other datasets such as CamVid, LaRS, or VSPW to remain more general.\n- there is always a conclusivity problem when it comes to “negative results”, in this case on improving image-DA with ideas transferred from video-DA. One can always posit there exists a better adaptation of those techniques. But the paper seems to have conducted the naive adaptation properly. \n- I am also missing side/ablation studies shedding light into why the video domain adaptation is not effective beyond the specific image domain adaptation. For instance in what way the improvements brought about by multi-resolution possibly overshadow any possible improvement from temporal information? Do the remaining errors, after multi-resolution modeling, in the target domain have characteristics that make them difficult which are independent of the temporal information? A discussion of, at least possible hypotheses based on the results and the experiences of the benchmarking, is missing in general.\n\nMinor points:\n- The start of the abstract implicitly assumes all domain adaptations, at least for computer vision, have their images coming from video, undermining the cases where images are still and do not come as frames of videos. It is better to reword to disambiguate.\n- Following are some suggested edits regarding the style of writing which although might make the paper less exciting but is probably important to follow to keep the narrative as objective as possible in an academic writing. \n- - exclamation mark in the abstract and then again in 3rd paragraph of page 2 and elsewhere\n- - adjective “massive” in the caption of Figure 1 and then again in 3rd paragraph of page 2 and elsewhere\n- - adjectives such “startling” and “astonishing” performance in page 2 and other similar terms elsewhere.\n- in sec 3.1: why should the number $N$ of source and target series in Video-DA be the same?\n- in page 7, $h_\\theta$ is used for both feature representation, logits, and then earlier for outputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the effectiveness of recent video domain adaptation (Video-DA) techniques beyond that of the state-of-the-art image domain adaptation (Image-DA) methods applied to video frames, i.e., without acknowledging any temporal and continuity aspect of video frames. The question is interesting and rightly-posed since video recognition, despite the theoretical promise, has then and again proved challenging to benefit from aspects of video beyond frame-based recognitions. \n\nThe paper focuses on semantic segmentation as the recognition task, synthetic-to-real as the domain adaptation task, with synthetic datasets as the source data (GTA-based) and real-world Cityscapes as the target data.\n\nOn the Image-DA side, it takes a recent thread of works which benefits from high resolution images and in several recent works has clearly achieved state-of-the-art performance. For the video-DA side it takes several recent works published in 2022 and 2023 computer vision venues.\n\nThe main take-away is that all the several recent Video-DA works are largely outperformed by the state-of-the-art Image-DA method on the selected tasks.\n\nThe paper then moves on to ablate which Image-DA component contributes most the performance which is found to be, by far, the high-resolution processing.\n\nFinally, the paper studies whether any of the techniques used in the several Video-DA methods can bring additional improvement to the state-of-the-art Image-DA applied on video DA but is found to not be the case; at least neither consistently nor significantly despite occasional small improvements.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The statements throughout the paper (possibly including the title) should be accordingly delimited so it doesn’t read as if it is the general video domain adaptation that is the subject of the study,or the scope of conclusion statements corroborated by the empirical results. (important for claims and evidence to match)\n\n- A discussion regarding the reasons behind the Video-DA techniques not improving over the state-of-the-art Image-DA method.\n\nThe paper has clear strengths, the other mentioned negative aspects do not fall within the criteria of TMLR but could be good to address regardless.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9GJtvNawky", "reviewer": "Reviewer_Ep2P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have found this work interesting from the perspective of providing the community a study on image-DA vs. video-DA techniques on tasks of common interest, which is the case of segmentation, but also of classification, and of action recognition, even if for the latter some major revision of the image-DA methods should have been carried out.\nGood points of the paper regard the comparison of several video-DA methods for segmentation task, the comparisons with essentially a recent image-DA method (and its variant), and an ablation analysis. It is also addressed the hybrid case in which image-DA and video-DA algorithms are integrated, showing not so consistent better results. \nAs a whole, the experimental analysis is fine even if it is not always comfortable to follow the work given the large number of methods considered, most of the times shortly described. For example, HRDA and HRDA+MIC are not easy to catch from only the description reported in the paper (including the one in Appendix).\n\nBut, my main concern is related to significance of the work so organized.\nIn particular, first, I don't deem so interesting for the community the findings claimed in the paper unless the reasons why image-DA methods are working better for videos with respect to video-DA techniques are provided. Claiming that the MRFusion and MIC are the major responsible for the performance improvement is not sufficient, it'd be interesting to figure out why video-DA methods are not exploiting similar mechanisms, or how the latter can be integrated in video-DA methods for boosting performance. In other words, one would like to see why temporal information is not so relevant for the task at hand. \nMoreover, using only one image-DA method is also a limiting factor: HRDA is a specific method, but there a number of other techniques that can be compared, which are not following the same mechanisms of HRDA and that may deserve some comparisons.\n\nSecond, just 2 types of experiments having Cityscapes as target is too limiting to me. Probably, the literature addressed only these 2 cases, but drawing conclusions from these experiments only is insufficient to my opinion. Maybe one can provide all possible combinations of the 3 datasets as source/target.\n\nThird, as also quoted above, it would be more interesting to provide a comparison on image- vs. video-DA methods for an extended set of tasks, not only for segmentation, but I understand that this would likely lead to another paper. Indeed, it'd be interesting to figure out the differences and similarities of exploiting temporal/multiple frames information in a DA setting for a set of tasks.\n\nFourth, when combining image- and video-DA methods, several strategies and parameters tuning have been applied, and it is unclear whether these variations affects (and how much) the final performance. It is also strange that results on the methods' combination are presented only for Viper to Cityscapes and not for Synthia to Cityscapes (Table 4).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates domain adaptation (DA) for video (semantic) segmentation task. Authors start from the assumption that video-DA methods use a set of benchmarks different from those used by image-DA methods, and realized that the latter are much better performing on video benchmarks as compared to the former techniques supposedly designed for such benchmarks and leveraging temporal information. In addition, it is also reported that combinations of image-DA and video-DA methods do not even lead to consistent performance improvements.\nPerformances are reported for essentially 2 use cases (Viper to Cityscapes and Synthia to Cityscapes), and show that the most performing image-DA method (HRDA+MIC, CVPR 2023) outperforms state-of-the-art video-DA algorithms by a large margin (+14.5 and +19.0) in mIoU (mean Intersection over Union).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see such work revised in the following way.\n\nProvide experiments with more datasets or more combinations of the considered datasets.\n\nAblation analysis should be better detailed, addressing why image-DA method is working better and what is missing in the video-DA methods for not reaching similar performance.\n\nExplore more thoroughly the design of a method combining image-DA and video-DA mechanisms, or better, explore how to integrate in video-DA methods some strategies of image-DA techniques in order to assess whether (video-DA) performance can improve.\n\nPossibly addressing a larger set of DA tasks, not only related to segmentation.", "broader_impact_concerns": "I don't see ethical concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nttu0Bb657", "reviewer": "Reviewer_RrVf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1: The ablation studies of advanced image-based and popular video-based techniques for domain adaptive video semantic segmentation are comprehensive and convincing.\n\n2: The constructed strong baseline is solid and conveys insights into the necessary techniques and strategies for the domain adaptive video semantic segmentation.\n\n3: The whole work is easy to understand and follow up on. The open-source codebase helps other researchers reproduce the results and develop their methods.\n\n**Weaknesses:**\n\n1: The usage of **\"video domain adaptation\"** and **\"domain adaptive video semantic segmentation\"**. To the best of my knowledge, the \"domain adaptation\" is generally specific to the classification task (both image-based and video-based [1] areas) or serves as a collective term, including many domain adaptation tasks. However, this paper only considers the \"(domain adaptive) video semantic segmentation\" task as mentioned in previous literature [2, 3]. Therefore, the authors are encouraged to rethink using the sentence \"video domain adaptation\" in the whole main text.\n\n2: Is the setting of **batch size** correct? In the Implementation Details section, the authors say the batch size is 2. I want to know what the \"2\" means. Does it mean the total batch size among all GPUs or the number for each GPU?\n\n3: The **\"Preliminaries\"** and **\"Experiments\"** sections are encouraged to be **re-organized**. First, the \"Preliminaries\" section generally introduces the background knowledge for the task, and that's what Section 3.1 does. But Section 3.2 introduces the \"Experimental Setup,\" which generally belongs to a part of \"Experiments.\" Similarly, Section 4.2 mainly discusses the domain adaptation techniques, while Section 4 is called \"Experiments.\" I think it would be better if this paper consolidates the method part and merges Section 3.2 into the experiment part.\n\n4: Verifying the proposed simplified baseline on other benchmarks. The developed strong baseline shows promising results. However, the authors only test the methods on two scenarios. Following MIC [4], testifying the model's effectiveness on DarkZurich and ACDC can make the results more comprehensive. \n\n\n[1] Chen, Min-Hung, et al. \"Temporal attentive alignment for large-scale video domain adaptation.\" ICCV, 2019.\n\n[2] Hu, Ping, et al. \"Temporally distributed networks for fast video semantic segmentation.\" CVPR, 2020.\n\n[3] Guan, Dayan, et al. \"Domain adaptive video segmentation via temporal consistency regularization.\" CVPR, 2021.\n\n[4] Hoyer, Lukas, et al. \"MIC: Masked image consistency for context-enhanced domain adaptation.\" CVPR. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper comprehensively examines the impact of cutting-edge image-based domain adaptation techniques on the domain adaptive video semantic segmentation task, thereby establishing a stronger baseline. Additionally, it delves into the applicability of widely used video-based domain adaptive techniques for this task and conveys insights about the necessary components. Furthermore, the authors have made the codebase open-source to facilitate further research in this area.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the \"Strengths And Weaknesses\" part.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The AE sides with acceptance. The work is informative and the review and revision process has brought the claims and evidence into better agreement. The chosen methods and benchmarks are state-of-the-art (or near state-of-the-art) and popular, so these results can guide further examination of image and video DA, because the experimental study is thorough within its chosen breadth. Nevertheless, the AE must echo the feedback from two of three reviewers that a general claim along the lines of "not using video effectively" would ideally be justified by evidence from more datasets and tasks. The degree of generality or lack thereof was a key point in the official recommendations, and the cause for diverging scores in leaning toward acceptance vs. rejection. While the AE respects the disagreement raised, this is not an obstacle to acceptance, because the intended scope of the paper is well-signaled and gaps have been adequately addressed by the revision. In summary, the positives for informing the community with the experimental comparison and analysis overcome the negatives w.r.t. a potentially even broader paper. The paper can be accepted as is.

Regarding certifications, 2/3 reviewers have voted for the reproducibility certification. Given the level of detail in the submission and the linked code, the AE agrees it is reproducible. To quote the [editorial policies](https://jmlr.org/tmlr/editorial-policies.html#certifications):

> Reproducibility Certification. This is awarded to papers whose primary purpose is reproduction of other published work. Beyond simple verification, the paper must contribute significant added value through additional baselines, analysis, ablations, or insights.

While I would say the _primary_ purpose of this paper is analysis and benchmarking, rather than reproduction, this work does reproduce a number of methods and benchmark settings and certainly provides added value. Therefore I will leave this to the Editors to confirm.

The AE thanks the authors and reviewers for engaging in the TMLR process to deliver an improved paper with agreement between its claims and evidence and a clear audience.

As a last note, happy winter holidays to you all!
Decision: Accept (poster)