Learning from Labeled Images and Unlabeled Videos for Video Segmentation
Authors: Cristina Mata
Michael S Ryoo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Video
Segmentation
Representation

Abstract: Performance on video object segmentation still lags behind that of image segmentation due to a paucity of labeled videos. Annotations are time-consuming and laborious to collect, and may not be feasibly obtained in certain situations. However there is a growing amount of freely available unlabeled video data which has spurred interest in unsupervised video representation learning. In this work we focus on the setting in which there is no/little access to labeled videos for video object segmentation. To this end we leverage large-scale image segmentation datasets and adversarial learning to train 2D/3D networks for video object segmentation. We first motivate the treatment of images and videos as two separate domains by analyzing the performance gap of an image segmentation network trained on images and applied to videos. Through studies using several image and video segmentation datasets, we show how an adversarial loss placed at various locations within the network can make feature representations invariant to these domains and improve the performance when the network has access to only labeled images and unlabeled videos. To prevent the loss of discriminative semantic class information we apply our adversarial loss within clusters of features and show this boosts our method's performance within Transformer-based models.

Under review as a conference paper at ICLR 2023
LEARNING FROM LABELED IMAGES AND UNLABELED VIDEOS FOR VIDEO SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Performance on video object segmentation still lags behind that of image segmen- tation due paucity of labeled videos. Annotations are time-consuming and laborious to collect, and may not be feasibly obtained certain situations. How- ever there is growing amount of freely available unlabeled video data which has spurred interest in unsupervised video representation learning. In this work we focus on the setting in which there is no/little access labeled videos for video object segmentation. To this end we leverage large-scale image segmen- tation datasets and adversarial learning to train 2D/3D networks for video object segmentation. We first motivate the treatment of images and videos as two sepa- rate domains by analyzing the performance gap of an image segmentation network trained on images and applied to videos. Through studies using several image and video segmentation datasets, we show how an adversarial loss placed at various locations within the network domains and improve the performance when the network has access to only la- beled images and unlabeled videos. To prevent the loss of discriminative semantic class information we apply our adversarial loss within clusters of features and show this boosts method's performance within Transformer-based models.
make feature representations invariant to these
INTRODUCTION
Video object segmentation is attracting more attention due to its importance in many applications such as robotics and video editing. Much progress has been made on general video understanding tasks such as action recognition, but is lagging on dense labelling tasks such video segmentation. This is mainly due to the laborious and time-consuming nature of pixel annotation collection, re- sulting in small, sparsely annotated video datasets. To deal with this problem, researchers often use large-scale image segmentation datasets to learn semantic representations through pretraining.
While image segmentation datasets are useful souree of labeled data for pretraining, the repre- sentations learned do not translate well to videos containing video artifacts such as motion blur, low lighting and low resolution. Figure shows example of the domain differences arising in common datasets, where the boundaries and small spokes of the moving bicycle are blurred. When image-pretrained models are applied directly to videos, performance drop observed in object de- tection (Kalogeiton et al.) 2015; Tang et al. 2012) and in our own video segmentation experiments. Further, Kalogeiton et al. find that addition motion blur, the location of objects in the frame, diversity of appearance and aspects, and camera framing all have an effect on the performance of image-prette penge-pperraneed models on videos. Thus supervised training on images is insufficient for pixel- pixel-wise video understanding, producing need for convenient alternative representation learning method that uses unlabeled videos.
Unsupervised video representation learning has mainly focused on the classification task rather than on segmentation. Some methods learn useful features from unlabeled videos with data augmenta- tions (Behrmann et al. 2021), contrastive losses (Han et al., 2021), and pretext tasks such as frame shuffling (Xu et al., 2019). The learned networks are successful on video classification, but do not translate well to segmentation due to loss of detail that occurs in the spatial bottleneck. Applying the same self-supervised methods for segmentation leads to poor performance because the features learned do not contain enough local information for the decoder to reconstruct the output.

---

Under review as a conference paper at ICLR 2023
Figure I: A single image from the COCO dataset (left) and a frame from Davis 2019 (right) shows domain differences such as motion blur arising from video artifacts.
In this paper, we propose an approach to video segmentation that takes advantage of both labeled image segmentation data and unlabeled videos. Taking inspiration from (Tang al.) (012)) we use unlabeled videos to minimize the domain difference between image representations and the spatial component of video representations. We train our video segmentation networks to be invariant to properties specific to video (motion blur, viewpoints, etc.) so that we can train on labeled images and apply them to videos without a performance drop.
To achieve this invariance propose inserting domain discriminator within video segmentation network to discourage it from learning image or video-specific features. We train the network for segmentation using labeled images while using unlabeled videos to adversarially train the discrimi- nator predicting the domain of the sample, shown in Figure 2
We experiment with two different segmentation backbones: convolutional neural networks (CNNs) and Transformer-based networks. To take advantage of temporal information in unlabeled videos while retaining spatial information from labeled images, also apply our method to VideoSwin (Liu et al. 2021b) with spatiotemporal window size. In our CNN we place the discriminator at the end of the encoder prior the decoding stage. our Transformers we experiment with placing the discriminator at either the end of the encoder or after the patch embedding layer. We find that placing the discriminator after the patch embedding to target low-level features boosts the contribution of the adversarial loss. We conduct experiments using the video object segmentation (VOS) datasets Davis 2019 and FBMS and show that in our target setting with no access to labeled videos, our method improves segmentation performance over models supervised with images.
2 RELATED WORK
For video object segmentation network must generate pixel-wise classification for one more moving target objects in a video. Labels must be consistent across frames but not necessarily across videos. In the unsupervised VOS track no annotations are provided during inference. Without the target object annotation from the first frame, unsupervised VOS methods rely on knowledge from image pretraining optical flow. Li et al. (2018) leverage embeddings from instance segmenta- tion network trained on still images to generate an embedding for each object in a scene and then semantie score and motion features from optical flow to select foreground object embeddings for a track. Similarly, RTNet (Ren et al.] (02T) distinguishes foreground objects from moving dis- tractors using module that computes similarities between pairs of motion and appearance features from different objects. Recently, TransportNet (Zhang et al. 2021) aligns RGB and flow features in two-stream network by optimizing Wasserstein distance with Factorized Sinkhorn method. Yang et al. (2021c) do this alignment using co-attention between RGB and flow. Yang et al. (2021a) rely solely on optical flow the input their segmentation network. Optical flow features can be cumbersome to generate prior to segmentation so in our work we focus on learning strong semantic object representations from labeled images and unlabeled videos.
Much work has been done on self/un-supervised learning for classification, but video segmentation representations have not been as well explored. Inspired by MoCo (He et al., 2020), VideoMoCo (Pan et al. 202T) adopts a momentum encoder but drops frames from videos to improve temporal robustness. V12CLR (Diba et al., 202T) learns features from unlabeled videos within separate 3D and 2D CNN encoders by clustering latent frame and clip features then applying a contrastive loss between clusters. Our method differs from these because it targets learning detailed segmentation features. Self-training Zoph al. (2020) is complementary to method, are self-supervised
2

---

Under review as a conference paper at ICLR 2023
Images Video Frames
L seg
Encoder
Decoder
Losc
Discriminator
Figure 2:
Our general adversarial method takes the output of the encoder and uses its feature repre- sentation to classify the training samples' domain. The discriminator's parameters optimized by the adversarial loss.
objectives for video correspondence learning in video instance segmentation Fu ct al. 2021) Our method differs from these in the way uses video representations.
Many works have addressed domain adaptation for semantic segmentation in order to solve the synthetic-to-real problem or transfer representations. [Shin et al. (202I) use distillation to tackle synthetic-to-real domain adaptation for video semantic segmentation. Guan al. (2021) enforce similar temporal consistency between consecutive real frames and consecutive synthetic frames. Hong al. (2017) use unlabeled videos for image segmentation by generating pseudo-labels. [Tang] et al. (2012) address image-to-video domain adaptation in object detection by retraining an image- trained detector on target video samples. Our work is most similar that of Yang et al. (202Ib), which uses domain adversarial learning on the class tokens of a vision transformer for image classi- fication.
METHODS
3.1
DOMAIN ADVERSARIAL LOss
Our main approach relies on treating images and videos as separate domains and applying a domain adversarial loss to learn robust object representations. Many image and video segmentation net- works use encoders that learn low-level features in the shallow layers and abstract features in deeper layers. Based observations by Kalogeiton al., (2015), hypothesize that the object repre- sentations learned by an encoder trained only on images differs from the representations learned on videos. This is due video artifacts, especially motion blur, camera framing, appearance and aspect diversity. We further hypothesize that this makes it more difficult for segmentation decoder recover fineggrined details videos. Our goal is to bring spatial features learned from images closer to those learned from videos using an adversarial loss propagated through the encoder. In real-world scenario the most likely setting access small amount of labeled images (and per- haps labeled videos) and a large amount of unlabeled videos. Our method makes it straightforward to use all available data.
To make the feature representations invariant to the image and video domains, we add domain dis- criminator that distinguishes between the domains at different points in the encoder. During training maximize the discriminator's loss, which encourages the encoder to leam feature representations that fool the discriminator by containing as little information as possible about the domain.
Formally, are given samples from source dataset D {(x y²) } where each x= is single image and y is its pixel-wise segmentation label. In the unsupervised domain adaptation setting have access to samples from unlabeled target dataset D' {x*) Jt where each x² is video frame. Given a sample T; E (D*, D'}, our discriminator estimates its domain D, through
arg Pr(D; D Mxx)) DE(D"DDY
where D; is the domain r; and h(-) is a function mapping I; to some latent feature space. In case, h(x;)) is the encoder's feature representation of æ;. Using binary cross entropy loss, the

---

Under review as a conference paper at ICLR 2023
Patch Embedding
Palch Embedd
Merge
Swin Blook
Patch Morge
Swin Blook
Decader
Padding
Conv
Norm
Patch level: Apply per- token
Global
Concatenate tokens
Concatenation
Discrimnator
Dicor Iisorminttor
12
T
Token
Figure 3: We place the adversarial loss after the patch embedding layer the end of the coder. The patch embedding consists of convolutions applied per-patch. Here it is feasible to apply the discriminator per-token, allowing the discriminator to notice video artifacts moving object boundaries which may not be present at deeper layers. At the global encoder level the tokens are concatenated then passed to the discriminator. Each blue block indicates token i,, with n tokens in the sequence. + indicates token concatenation.
discriminator's training objective is to minimize
Cosc
-d; (1- d:) log(1 1 p.) (DDD))ED
Σ
(2)
where Ode are the parameters of the discriminator, D D*UD', d: is binary indicator 1 [xx D*). and p; is the network's s estimate of Pr(D. = D*)h(=)))
During joint training with the rest of the segmentation network, Gradient Reversal Layer (Ganin et al. 2016) between the encoder and discriminator. The layer outputs the identity on the forward pass and reverses the sign on the discriminator's gradients during the backward pass.
The encoder and decoder are trained for segmentation through a standard cross entropy loss:
Le...
Σ ΣΣ Yi. log Ps...
(3)
where j iterates over each pixel in the image, c ranges over the number of object classes and Pijik the decoder's estimate of the probability that pixel j in image belongs to class k. In case of spatiotemporal input, ý ranges over the temporal dimension as well. The final network is trained end-to-end by minimizing the segmentation loss and maximizing the adversarial loss
L = Le.c, - -uoo.
where ya is constant scaling factor chosen through grid search to ensure the discriminator does not collapse. Our joint training procedure ensures that the network learns feature representations that are useful for segmentation and do not contain domain information. The general architecture of our method with loss components shown in Figure 2
3.2 TOKEN COMBINATION
We consider two types of segmentation backbones: CNNs and Transformers. In the CNN backbone, we place the domain discriminator at the end of the encoder, and flatten the spatial dimensions of the features for the discriminator's 's input. In a Transformer- individually to the discriminator or combined through concatenation. The discriminator can also be applied at any layer within the transformer whereas in the CNN the feature size in earlier layers is computationally expensive. The number of tokens at the end of the Transformer encoder is large so applying the discriminator pok-token at this location would increase the loss's magnitude and dominate training. Instead we batch tokens within a sample and flatten their spatial dimensions. Figure illlsttras this token concatenation.
encoder the tokens may be passed
4

---

Under review as a conference paper at ICLR 2023
Some video artifacts such motion blur are most apparent around moving object boundaries. When applied to tokens at the global level, the adversarial loss operates over the entire spatial resolution and may not target these features well. To address this place adversarial loss directly after the patch embedding layer to operate over low-level features containing detailed spatial information. We apply the loss per-token right after the patch embedding layer.
3.3 CLUSTERED ADVERSARIAL LOSS
We want our adversarial loss make network features invariant to image-video domain differences while retaining discriminative class information. However our general adversarial loss formulation agnostic to semantic class. This becomes an issue with sparse annotations: different image and video datasets may not be labeled with the same classes, but we want to take advantage of all labeled data regardless of annotation consistency.
To this end propose grouping tokens from batch containing image and video frames into class clusters then applying the adversarial loss within each cluster. At the patch embedding level, each token is likely to be dominated by information from single class because tokens are formed from local patches. Thus an unsupervised clustering method, k-means, is likely to contain image and video features from single class. Applying the loss within cluster ensures the loss removing discriminative class information.
We modify the discriminator take pair of features and estimate the probability that they belong to the same domain. Formally our clustered adversarial loss becomes
Lat Σ Σ (BL.)))BB
bb log(() (1 b:) log(1 9:)
where e ranges over each cluster, Be is a set of pairs of image and video features within a cluster, b; is binary indicator of whether 2; and n belong to the same domain D, i.e. [[;; E D E D), and qi is the network's 's estimate of Pr(x; E D. a E D). While the k-means clustering step is the most time-consuming operation, it is only carried out during training and inference time remains the same.
4 EXPERIMENT SETTINGS
4.1 DATASETS
To understand how our method changes object representations learned from videos we conduct experiments on subsets of Davis 2019 (Perazzi et al. 2016) and on FBMS (Ochs et al. 2014). We use COCO Stuff (Lin et al.] 2014) source of labeled images, which has many overlapping classes with Davis and FBMS.
Davis is unsupervised video object segmentation dataset with annotations for 60 training and 30 validation videos. The original annotations label foreground target objects with a unique ID that is consistent within a video but not across videos. For our experiments we process Davis annotations so that they are semantically consistent across videos in order to better measure our network's 's un- derstanding of semantics rather than identification of moving objects. We focused 10 classes in common with COCO that appear in both training and validation splits. Having an object appear in the training set allows us to analyze how its object representation changes when using our method: any difference in performance results from changes in the object representations as opposed to the model's s ability to generalize to unseen classes. breakdown of the number of video frames each of the shared classes in Davis can be found in the supplementary material. The final Davis training set consists of 3,401 frames and 1,558 frames for validation. The COCO training set used with Davis has 75,377 frames and the validation set has 3,176 frames.
As an additional source of unlabeled video data, we use YouTube- Bounding (YTBB) RReal et al. 2017). It features bounding box annotations for 23 objects in about 380,000 videos sourced from public YouTube URLs. When decoding the videos we use 30 fps and skip videos that have been taken down by the uploading user. Similar to COCO use classes shared with Davis. Vi-
5

---

Under review as a conference paper at ICLR 2023
sual inspection of YTBB reveals that the videos often feature motion blur, low lighting, and low resolution.
FBMS (Ochs al. 2014) contains 59 video sequences with 353 annotated training frames and 367 annotated test frames. The content ranges across 19 object classes most which are in common with COCO. We process annotations to be semantically consistent across videos. When training COCO with FBMS we draw images containing classes in common with FBMS, resulting in 80,279 training images.
4.2 IMPLEMENTATION
To analyze the domain shift between images and videos, we train segmentation networks jointly on images and videos. For our CNN we use Deeplabv3 (Chen et al.) (017) with a ResNet- 101 backbone. The domain discriminator consists of 2D convolution reduce the feature size, then fully connected layer, dropout, ReLU and the classification layer.
the Swin-T Transformer backhone with UperNet decoder (Liu et al., 202 with patch size (4, 4) and window size (7.7). The discriminator is implemented as sequence of alternating linear and leaky ReLU layers. When the discriminator is placed at the end of the encoder the tokens are concatenated and spatial dimensions flattened. At the patch embedding level each token's spatial dimensions are flattened and the token and batch dimensions are combined to apply the discriminator per-token.
For our 2D Transformer
Our spatiotemporal model is VideoSwin with a Swin-T backbone. It uses patch size (1,4,4) and window size (4,7,7). The discriminator is implemented as a PatchGAN discriminator (Isola et al., 2017] with four layers.
In all our experiments we use the Py Torch learning framework. Images and video frames are resized to 256 x 256 and normalized.
our experiments with VideoSwin we resize inputs to 512x512 and clip length 4 for videos, zero-padding single images to the same temporal resolution. The segmentation cross entropy loss uses loss weighting calculated from DAVIS or FBMS. The 2D CNN is initialized with COCO peco-pprtr weights from the PyTorch model zoo. The Swin Transformer is initialized using ADE20k weights provided by the model's 's authors. We initialize the VideoSwin Transformer with the same ADE20K weights and replicate them across the temporal dimension where appropriate.
The adversarial loss's coefficient μ is set to constant 1.0 that performed better than slow increase. When training with the clustered adversarial loss the number of clusters is pre-set to the number of classes in the video dataset. On one GeForce GTX Titan X GPU, our 2D Transformer with the general adversarial loss takes 365 seconds to perform an epoch training on Davis with batch size 16 and 60.1 GFLOPS for inference. Under the same settings the clustered loss takes 1312 seconds and 60.0 GFLOPS. Our 2D CNNs are trained with batch size 8 and learning rate 0.001 and our 2D Transformer is trained with batch size 32 and learning rate 0.0032. VideoSwin is trained using batch size 16 and learning rate 0.0001. We will publicly release code upon publication acceptance.
RESULTS
5.1 2D CNNS ON DAVIS
We focus on the setting in which we have access to labeled images and unlabeled videos due to the challenging nature of collecting video annotations. To test the adversarial loss' contribution to learning detailed semantic representations, we place discriminator at the end of 2D CNN encoder which acts on global spatial feature representations. We train on video frames from Davis, images from COCO and also experiment with training random unlabeled YouTube videos from YTBB. Unlabeled frames are used in the adversarial loss but not the segmentation task loss. For our baseline use the 2D CNN trained only on labeled images.
Table shows our results on Davis measured using mean Jaccard index or Intersection over Union (IOU) over all semantic classes. We find that in our target setting with no access to labeled videos, the adversarial loss boosts the performance of the baseline CNN. The adversarial loss even boosts performance on Davis when using random unlabeled YouTube videos train the discriminator. Our

---

Under review as a conference paper at ICLR 2023
Table 1: Results Davis from training 2D CNN Davis, COCO and YTBB. Adv indicates whether or not an adversarial loss was included during training. Labeled and Unlabeled columns indicate the label setting for the training set. Results are measured on Davis by averaging IOU over 10 classes plus background. The adversarial loss boosts performance over the baseline trained on labeled images and no videos.
Adv
Labeled
Unlabeled
mIOU
A
COCO
32.3
COCO
Davis YTBB
36.9
+ 4.60
COCO
37.7
+ 5.40
Davis
32.2
X
COCO+Davis
46.2
A
B
C
D (ours)
E F (ours)
Figure 4: Qualitative results on Davis videos. Column A: The original frame. Column B: The ground truth. Column C: The 2D CNN results when trained COCO (image data). Column D: The 2D) CNN trained on labeled COCO and unlabeled Davis data with the adversarial loss. Column E: Swin results when trained on COCO images. Column F: Swin trained with the clustered adversarial loss on labeled COCO and unlabeled Davis frames. When the Swin transformer is trained with the clustered adversarial loss recovers fine-grained details and has better separation of semantic classes, whereas the models trained only on images show spurious and missing predictions.
method outperforms the supervised setting in which the model is trained only on labeled videos and labeled images. Training with our adversarial loss on images and unlabeled videos performs better than training just labeled videos, highlighting the usefulness of our method in settings where labeled videos are difficult to collect and object diversity is limited.
5.2 TRANSFORMERS
In the following experiments test whether adversarial loss boosts the performance of Transformer- based model trained only on labeled images. We use Swin as our 2D Transformer model and VideoSwin with spatiotemporal window size our 3D Transformer.
2D Transformer on Davis We train a 2D Swin Transformer on videos from Davis and images from COCO and show results Table 2 We find that placing the discriminator right after the patch embedding as described in section 3.2 yields better performance than applying it globally, showing that
features are better targeted at shallower layers. Training with the adversarial loss is able to improve performance in the unlabeled video setting as well as with access to labeled videos. We test the performance of our clustered adversarial loss and show that clustering tokens at the patch embedding level before applying the adversarial loss leads to a boost in performance.
VideoSwin on Davis We also explore how our method performs model that learns spatiotem- poral features, namely, VideoSwin with image patch tokens and spatiotemporal window extending over the entire clip length. 3 the results of training this network using the general ad- versarial loss the patch level. In the unlabeled video setting the adversarial loss improves the

---

Under review as a conference paper at ICLR 2023
Table 2: Results Davis from training 2D Transformer Davis and COCO. Adv indicates
whether or not an adversarial loss was included during training. Cluster indicates whether the clus- tered or general adversarial loss was used. The Patch/Global column indicates the discriminator's
location. Results are measured on Davis by averaging IOU over 10 classes plus background.
Adv
Cluster
Patch/Global
Labeled
Unlabeled
mIOU
A
x
COCO
29.98
x
Global
COCO
Davis
25.27
+.71
X
Patch
COCO
Davis
32.70
+ 2.72
/
V
Patch
COCO
Davis
38.53
8.55
Davis
25.97
X
COCO + Davis
29.01
+ 3.04
J
X
Global
COCO + Davis
28.15
+2.18
X
Patch
COCO + Davis
32.55
+ 6.58
Patch
COCO + Davis
38.27
+ 12.30
Table 3: Results on DAVIS of VideoSwin with the general adversarial loss at the patch embedding level.
Adv
Labeled
Unlabeled
mIOU
A
COCO COCO
19.35
DAVIS
30.48
+ 11.13
DAVIS
25.75
COCO+DAVIS
32.46
+ 6.71
COCO+DAVIS
x 33.65
7.90
image-supervised model, showing that our method can be applied to spatiotemporal models that rely on mmage-pretrained weights.
FBMS We test whether method achieves boost in performance different video dataset FBMS and show results in Table 4 In the unlabeled video setting, the general adversarial loss improves over the model supervised by images. Because the annotations for FBMS do not include all instances each class in every video, there is discrepancy between the annotations for FBMS and COCO which believe prevents the model from learning strong semantic class representations, leading the model to perform worse when trained with both labeled videos and images than training with our adversarial loss.
ABLATION ON UNLABELED IMAGES
We conduct ablation study test whether the performance boost with our adversarial loss in the previous experiments is due to feature invariance in the image-video domains or invariance to dataset biases. To test this we train a 2D model on labeled images from COCO and unlabeled images from another large-scale image segmentation dataset Pascal VOC (Everingham et al. 2010) to train our adverarial loss. We use the entire Pascal dataset as a source of unlabeled data.
Table 4: Results on FBMS of training Transformer with our general adversarial loss the patch level.
Adv
Labeled
Unlabeled
mIOU
A
X
COCO
33.98
COCO
FBMS
34.47
0.49
X
FBMS
26.6
COCO+FBMS
34.40
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study whether the adversarial loss boosts improvement when training la- beled COCO images and unlabeled Pascal images. The adversarial loss is placed at the end of the encoder in 2D CNN or Transformer. Adding unlabeled data from another image dataset does not significantly affect test IOU measured on COCO or Davis. This is in contrast to our image-video adversarial learning where training on unlabeled videos increased performance over the supervised image baseline
Model
Adv
Labeled
Unlabeled
Test
mIOU
CNN
X
COCO
DAVIS
32.3
CNN
COCO
Pascal
DAVIS
33.7
CNN
X
COCO
COCO
44.5
CNN
COCO
Pascal
COCO
45.0
Transformer
COCO
DAVIS
10.35
Transformer
V
COCO
Pascal
DAVIS
10.21
Transformer
COCO
COCO
11.31
Transformer
COCO
Pascal
COCO
10.10
Our results in Table that using unlabeled images when training with the adversarial loss does not significantly improve results regardless the backbone. Additionally the same observations hold when testing COCO. We believe the adversarial loss' improvements due to features becoming invariant to video artifacts, not object appearance diversity.
7 DISCUSSION
In experiments with the adversarial loss within 2D Transformer, we found placing the patch embedding and applying the loss per-token increases performance over concatenating tokens at the end of the encoder and applying the loss over the concatenated feature. Each token at the end of the encoder contains information from the entire image. This makes it more difficult for the discriminator to target low level video features such as motion blur moving object boundaries. Features at the beginning of the network retain fine-grained details that still have some separation of video-specific features and class information. While the patch-level loss can target low level feature differences in the domains, more abstract features present in either images videos may be embedded in intermediate features. The model may benefit from placing the domain discriminator at intermediate blocks within the transformer. We leave this for future work.
Joint training of segmentation models on multiple datasets requires a significant overlap in semantic labels. Otherwise, missing information confuses the network and degrades performance. Our clus- tered adversarial loss retains discriminative class information by first clustering low level appearance features, typically aligned along class. Applying the discriminator within cluster removes domain- specific but not class information. Additionally the loss does not rely on features specific to the image/video domain gap and could be applied to differrntt adaptation tasks.
Finally, we note that while spatiotemporal learning has been critical for good performance in video classification, fully 3D end-to-end trainable networks for VOS have had limited success without relying on optical flow. Here we take step towards developing standalone 3D Transformer for vOS and show that our adversarial loss can improve on the supervised baseline.
8 CONCLUSION
In this paper we propose an unsupervised method to learn fine-grained representations from labeled images and unlabeled videos for video segmentation. It addresses a gap in the video representation learning literature where the focus is typically video classification rather than segmentation. We leverage image segmentation datasets to learn strong semantic features then use an adversarial loss to address the performance drop that ensues when applying these networks videos. Our findings indicate the general formulation of the loss improves CNN and 2D/3D Transformer's performance and our clustered adversarial loss improves 2D Transformer performance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nadine Behrmann, Mohsen Fayyaz, Juergen Gall, and Mehdi Noroozi. Long short view feature decomposition via contrastive video representation learning. ICCV, 2021.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CVPR, 2017.
Ali Diba, Vivek Sharma, Reza Safdari, Dariush Lotfi, Saquib Sarfraz, Rainer Stiefelhagen, and Luc Van Gool. Vi2clr: Video and image for visual contrastive learning of representation. ICCV, 2021.
M. Everingham, L. Van Gool, C. K. Williams, Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010.
Yang Fu, Sifei Liu, Umar lqbal, Shalini De Mello, Humphrey Shi, and Jan Kautz. Learning to track instances without video annotations. CVPR, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarte training of neural net- works. JMLR, 2016.
Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Domain adaptive video segmentation via temporal consistency regularization. Accepted, ICCV, 2021.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representa- tion learning. NeurIPS, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised representation learning. CVPR, 2020.
Seunghoon Hong, Donghun Yeo, Suha Kwak, Honglak Lee, and Bohyung Han. Weakly supervised semantic segmentation using web-crawled videos. CVPR, 2017.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-ine tomgg translation with conditional adversarial networks. CVPR, 2017.
Vicky Kalogeiton, Vittorio Ferrari, and Cordelia Schmid. Analysing domain shift factors between videos and images for object detection. TPAMI, 2015.
Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C. C.Jay Kuo. Instance embedding transfer to unsupervised video object segmentation. CVPR, 2018
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. ECCV, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: hierarchical vision transformer using shifted windows. ArXiv preprint, 2021a.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021b.
Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term video analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. CVPR, 2021.
Perazzi, Pont-Tuset, B. McWilliams, Van Gool, M. Gross, and benchmark dataset and evaluation methodology for video object segmentation. CVPR, 2016.
A
Esteban Real, Jonathon Shiens. Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: a large high-precision human- CVPR, 2017.
human-annotated annotated dataset for object detection in video.
10

---

Under review as a conference paper at ICLR 2023
Sucheng Ren, Wenxi Liu, Yongtuo Liu, Haoxin Chen, Guoqiang Han, and Shengfeng He. Recipro- cal transformations for unsupervised video object segmentation. CVPR, 2021.
Inkyu Shin, Kwanyong Park, Sanghyun Woo, and In So Kweon. Unsupervised domain adaptation for video semantic segmentation. ArXiv preprint, 2021.
Kevin Tang, Vignesh Ramanathan, Li Fei-fei, and Daphne Koller. Shifting weights: adapting object detectors from image to video. NeurIPS, 2012.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. ICCV, 2021a.
Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. TVT: Transferable vision transformer for unsupervised domain adaptation. ArXiv preprint, 2021b.
Shu Yang, Lu Zhang, Jinqing Qi, Huchuan Lu, Shuo Wang, and Xiaoxing Zhang. Learning motion- appearance co-attention for zero-shot video object segmentation. ICCV, 2021c.
Kaihua Zhang, Zicheng Zhao, Dong Liu, Qingshan Liu, and Bo Liu. Deep transport network for unsupervised video object segmentation. ICCV, 2021.
Barret Zoph, Golnaz Ghiasi, Tsung-Y Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc
Le. Rethinking pre-training and self- traf-igai NeurlPS, 2020.


Reviews:
[{"review_id": "wSc-yq7QOd", "reviewer": "Reviewer_79eQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to improve semantic segmentation performance on single images from videos, from a network trained on still images. The argument is that videos typically have several distribution shifts from standard images, such as framing and motion blur. Therefore, applying a pre-trained segmentation network trained on still images results in sub-par performance on videos. The proposed method is to apply an adversarial loss on the features from the segmentation network, where the discriminator tries to discern whether the input is from a still image or a video. This allows the network to learn the image distribution of videos. Experiments are performed with networks trained on COCO, and tested on the Davis dataset, using both CNN and transformer architectures. Ablations also show that the improvement is not evident when training and testing on still image datasets.", "summary_of_the_review": "Overall, this is a nice simple method which shows strong improvements on a perhaps novel problem. I would be interested in knowing if this method would generalize to other distributional shifts (and perhaps more interested in why not, if so).", "strength_and_weaknesses": "Overall, the motivation of this work is quite interesting, and the idea that we can improve inference on videos due to the distributional shift is quite novel. The overall method is quite simple, at least in theory, although adversarial losses can be difficult to implement and train. The experiments show strong improvements over the baseline, and also demonstrate that this is not a general method for all segmentation networks (there must be a distributional shift), via the ablation.\n\nOverall, this would be more compelling if more datasets were evaluated on. In addition to number, there are also 'video' datasets such as Cityscapes, where the images were recorded in a video manner, but do not seem to exhibit the same issues one might see in Davis. Would we expect to see similar improvements for those datasets? Or is there more clarification needed on the exact class of issues that we would expect this method to improve upon?\n\nAnother topic of interest is that this does not seem like a method that is limited to image -> video transfer, but should, in theory, work on any two sets of inputs from different domains (E.g. daytime -> night time). Would this method generalize to those cases?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LeSZiHJauT", "reviewer": "Reviewer_KyqK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an adversarial loss function between image and video domain to unsupervisely learn the fine-grained representations from labeled images and unlabeled videos for video segmentation. The authors show that their proposed method can improve the performance of both the CNN and transformer based methods on video segmentation.", "summary_of_the_review": "The idea of this paper is interesting, but it is not well demonstrated.", "strength_and_weaknesses": "Strength:\nThe idea of narrowing representation variance between image and video data is interesting.\n\n\nWeakness:\n1.The experimental results are not convincing.\n1)Selection of the dataset is strange. The authors require large-scale video segmentation dataset with common classes in COCO, but they use the subset of the DAVIS 2019 dataset and the small-scale FBMS dataset, ignoring large ones like the Youtube-Objects Dataset which would be better qualified.\n\n2)Performance for baseline methods on the DAVIS dataset is rather low, comparing with state-of-the-art results, e.g. 65.6% J mean on 2019-val by [1]. Effectiveness of the proposed loss function on such algorithms does not guarantee the effectiveness on top-performing ones.\n[1]Luiten, J., Zulfikar, I.E., & Leibe, B. (2020). UnOVOST: Unsupervised Offline Video Object Segmentation and Tracking. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 1989-1998.\n\n2.The authors claim that ”Some video artifacts such as motion blur are most apparent around moving object boundaries. When applied to tokens at the global level, the adversarial loss operates over the entire spatial resolution and may not target these features well. To address this we place our adversarial loss directly after the patch embedding layer to operate over low-level features containing detailed spatial information. We apply the loss per-token right after the patch embedding layer.” in Section 3.2, but I cannot find the demonstration of this statement.\n\n\n3.Experiment, performance not convincing.\nThere are many typos in this manuscript, some frequent errors are:\n1)inconsistent verbal forms, e.g. sometimes ‘labelled’, sometimes ‘labeled’;\n2)lack of commas, e.g. “To prevent the loss of discriminative semantic class information  (short of a comma) we apply our ...” in the abstract section, “To deal with this problem  (short of a comma) researchers often use ...” in the introduction section;\n...", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LHDPV_sc__z", "reviewer": "Reviewer_SNu6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for training a video segmenter by combining image segmentation datasets with unlabeled video datasets. The main innovation is to treat the problem effectively as a domain adaptation problem. As such, the paper uses an adversary to ensure that features from images and features from video frames are indistinguishable. An additional innovation is to perform this adversarial loss on combined clusters of features from videos and images so that the differing frequency of categories in image and video does not cause a problem", "summary_of_the_review": "The paper proposes to use domain adaptation for video segmentation. While it has some clever ideas, I don't find this framing of video segmentation as domain adaptation particularly compelling or interesting enough to justify acceptance.\n\nAs a recommendation, I would suggest that the authors consider expanding the idea of clustered adversarial loss to more domain adaptation applications. I think that might make for a more valuable contribution.", "strength_and_weaknesses": "Strengths:\n- The idea of using clustering of features to avoid penalizing differing class frequencies in the two domains is a clever one. This is a nice idea for other domain adaptation problems as well.\n- The end result is quite good performance on video segmentation.\nWeaknesses:\n- The novelty of this work is somewhat limited. It is mainly about applying existing domain adaptation approaches to this problem. This might be enough if the application itself was innovative or surprising, but I am not sure that is the case; the motivation seems very much to be that video frames and images are different domains.\n- Is video segmentation only about image frame segmentation? I feel that motion-based cues should also play a role, and the fact that they don't seems to be more an issue with the dataset rather than the problem. In any case, I find the prospect of framing video segmentation as solely a different domain for image segmentation somewhat limiting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0-_EpLVn1_i", "reviewer": "Reviewer_AueS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper performs video segmentation via labeled images and unlabeled videos, in which models in the image domain is adapted to the video domain via an adversarial loss. Ablations are conducted to determine at which level(s) the loss can be added to optimize the performance.", "summary_of_the_review": "Based on the weaknesses and the various concerns discussed in the Clarity, Quality, Novelty And Reproducibility, I would not recommend acceptance per its current states.", "strength_and_weaknesses": "Strength:\n\n+ The paper is in general easy to understand, despite the ambiguity in task definition (see Clarity).\n+ Ablations are performed and interesting founding is presented.\n\nWeaknesses:\n\n- Using adversarial losses to adopt image model to video domain is not new. The idea of clustered loss is also incremental.\n- The claim on most self-supervised video representations focus on the recognition level, isn’t accurate. There are quite a lot of self-supervised video correspondence learning works that highly related to video segmentation task (i.e., most evaluate on the video segmentation task). [1] leverages self-supervised video correspondence technique to semi-supervised VIS. However, all above is not mentioned at all.\n- Seems no STOA approaches are compared, and the baseline's performance is rather low.\n- Davis is not a ideal dataset for semantic segmentation since most of the sequences contain very few objects, and almost one single object per category.\n\n[1] Fu et al. Learning to Track Instances without Video Annotations. CVPR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject