Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes an approach for training a video segmenter by combining image segmentation datasets with unlabeled video datasets. The main innovation is to treat the problem effectively as a domain adaptation problem. As such, the paper uses an adversary to ensure that features from images and features from video frames are indistinguishable. An additional innovation is to perform this adversarial loss on combined clusters of features from videos and images so that the differing frequency of categories in image and video does not cause a problem
strength_and_weaknesses: Strengths:
- The idea of using clustering of features to avoid penalizing differing class frequencies in the two domains is a clever one. This is a nice idea for other domain adaptation problems as well.
- The end result is quite good performance on video segmentation.
Weaknesses:
- The novelty of this work is somewhat limited. It is mainly about applying existing domain adaptation approaches to this problem. This might be enough if the application itself was innovative or surprising, but I am not sure that is the case; the motivation seems very much to be that video frames and images are different domains.
- Is video segmentation only about image frame segmentation? I feel that motion-based cues should also play a role, and the fact that they don't seems to be more an issue with the dataset rather than the problem. In any case, I find the prospect of framing video segmentation as solely a different domain for image segmentation somewhat limiting.
clarity,_quality,_novelty_and_reproducibility: As above, I think the novelty is limited. On the positive side it is quite clear.
summary_of_the_review: The paper proposes to use domain adaptation for video segmentation. While it has some clever ideas, I don't find this framing of video segmentation as domain adaptation particularly compelling or interesting enough to justify acceptance.

As a recommendation, I would suggest that the authors consider expanding the idea of clustered adversarial loss to more domain adaptation applications. I think that might make for a more valuable contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper performs video segmentation via labeled images and unlabeled videos, in which models in the image domain is adapted to the video domain via an adversarial loss. Ablations are conducted to determine at which level(s) the loss can be added to optimize the performance.
strength_and_weaknesses: Strength:

+ The paper is in general easy to understand, despite the ambiguity in task definition (see Clarity).
+ Ablations are performed and interesting founding is presented.

Weaknesses:

- Using adversarial losses to adopt image model to video domain is not new. The idea of clustered loss is also incremental.
- The claim on most self-supervised video representations focus on the recognition level, isn’t accurate. There are quite a lot of self-supervised video correspondence learning works that highly related to video segmentation task (i.e., most evaluate on the video segmentation task). [1] leverages self-supervised video correspondence technique to semi-supervised VIS. However, all above is not mentioned at all.
- Seems no STOA approaches are compared, and the baseline's performance is rather low.
- Davis is not a ideal dataset for semantic segmentation since most of the sequences contain very few objects, and almost one single object per category.

[1] Fu et al. Learning to Track Instances without Video Annotations. CVPR 2020.
clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: The description is unclear at many points. 1. The problem definition is not well presented. It took me a long time to figure out whether the task is for semantic, or instance segmentation (i.e., it is figured out from (3)). This ambiguity leads to many questions when reading this draft: for instance segmentation, transferring from image to video domain needs to consider instance association, which is not mentioned anywhere; for semantic segmentation, using DAVIS is not ideal.  2. Many figures are blurry with too tiny texts, figure 1 is almost uninformative. 

Novelty: This paper falls into the large category of adopting an adversarial loss to mitigating image-video domain gaps, that is investigated by tons of works in recent years. The idea is not new.

Reproducibility: the work seems reproducible according to the description.
summary_of_the_review: Based on the weaknesses and the various concerns discussed in the Clarity, Quality, Novelty And Reproducibility, I would not recommend acceptance per its current states.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method to improve semantic segmentation performance on single images from videos, from a network trained on still images. The argument is that videos typically have several distribution shifts from standard images, such as framing and motion blur. Therefore, applying a pre-trained segmentation network trained on still images results in sub-par performance on videos. The proposed method is to apply an adversarial loss on the features from the segmentation network, where the discriminator tries to discern whether the input is from a still image or a video. This allows the network to learn the image distribution of videos. Experiments are performed with networks trained on COCO, and tested on the Davis dataset, using both CNN and transformer architectures. Ablations also show that the improvement is not evident when training and testing on still image datasets.
strength_and_weaknesses: Overall, the motivation of this work is quite interesting, and the idea that we can improve inference on videos due to the distributional shift is quite novel. The overall method is quite simple, at least in theory, although adversarial losses can be difficult to implement and train. The experiments show strong improvements over the baseline, and also demonstrate that this is not a general method for all segmentation networks (there must be a distributional shift), via the ablation.

Overall, this would be more compelling if more datasets were evaluated on. In addition to number, there are also 'video' datasets such as Cityscapes, where the images were recorded in a video manner, but do not seem to exhibit the same issues one might see in Davis. Would we expect to see similar improvements for those datasets? Or is there more clarification needed on the exact class of issues that we would expect this method to improve upon?

Another topic of interest is that this does not seem like a method that is limited to image -> video transfer, but should, in theory, work on any two sets of inputs from different domains (E.g. daytime -> night time). Would this method generalize to those cases?
clarity,_quality,_novelty_and_reproducibility: The method is well motivated and very clear. The motivation is perhaps a little unintuitive at first, but the method is simple (not a bad thing), and the results defend the story well.
summary_of_the_review: Overall, this is a nice simple method which shows strong improvements on a perhaps novel problem. I would be interested in knowing if this method would generalize to other distributional shifts (and perhaps more interested in why not, if so).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an adversarial loss function between image and video domain to unsupervisely learn the fine-grained representations from labeled images and unlabeled videos for video segmentation. The authors show that their proposed method can improve the performance of both the CNN and transformer based methods on video segmentation.
strength_and_weaknesses: Strength:
The idea of narrowing representation variance between image and video data is interesting.


Weakness:
1.The experimental results are not convincing.
1)Selection of the dataset is strange. The authors require large-scale video segmentation dataset with common classes in COCO, but they use the subset of the DAVIS 2019 dataset and the small-scale FBMS dataset, ignoring large ones like the Youtube-Objects Dataset which would be better qualified.

2)Performance for baseline methods on the DAVIS dataset is rather low, comparing with state-of-the-art results, e.g. 65.6% J mean on 2019-val by [1]. Effectiveness of the proposed loss function on such algorithms does not guarantee the effectiveness on top-performing ones.
[1]Luiten, J., Zulfikar, I.E., & Leibe, B. (2020). UnOVOST: Unsupervised Offline Video Object Segmentation and Tracking. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 1989-1998.

2.The authors claim that ”Some video artifacts such as motion blur are most apparent around moving object boundaries. When applied to tokens at the global level, the adversarial loss operates over the entire spatial resolution and may not target these features well. To address this we place our adversarial loss directly after the patch embedding layer to operate over low-level features containing detailed spatial information. We apply the loss per-token right after the patch embedding layer.” in Section 3.2, but I cannot find the demonstration of this statement.


3.Experiment, performance not convincing.
There are many typos in this manuscript, some frequent errors are:
1)inconsistent verbal forms, e.g. sometimes ‘labelled’, sometimes ‘labeled’;
2)lack of commas, e.g. “To prevent the loss of discriminative semantic class information  (short of a comma) we apply our ...” in the abstract section, “To deal with this problem  (short of a comma) researchers often use ...” in the introduction section;
...
clarity,_quality,_novelty_and_reproducibility: The ideas are clear, novel and reproducible.
summary_of_the_review: The idea of this paper is interesting, but it is not well demonstrated.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers shared common concerns on 1) the limited technical novelty and 2) insufficient experimental results. 

On the novelty side, two reviewers commented that image-to-video domain adaptation using adversarial learning has been explored before. The authors rebutted that this criticism is incorrect, claimed their work is the first to do so, and asked the reviewers to provide references. Unfortunately, the reviewers did not provide the references but this meta reviewer was able to find a few relevant papers [A, B, C]. (This meta reviewer provided these references to the authors during the discussion phase, and later they admitted that this has been done but their application to segmentation task is unique. This is true, but it significantly undermines -- and invalidates -- their original claim which wasn't specific to segmentation.)

* [A] Andrew Kae and Yale Song. "Image to Video Domain Adaptation Using Web Supervision." WACV 2020. https://arxiv.org/abs/1908.01449
* [B] Jin Chen, Xinxiao Wu, Yao Hu, and Jiebo Luo. "Spatial-temporal causal inference for partial image-to-video adaptation." AAAI 2021
. https://ojs.aaai.org/index.php/AAAI/article/view/16187
* [C] Wei Lin, Anna Kukleva, Kunyang Sun, Horst Possegger, Hilde Kuehne, and Horst Bischof. "CycDA: Unsupervised Cycle Domain Adaptation from Image to Video." ECCV 2022. https://arxiv.org/abs/2203.16244

On the experiments side, the reviewers pointed out that the paper does not report SOTA performance on the dataset and compared to rather weak baseline approaches only. They also commented that evaluation on more compelling/larger benchmarks are necessary. Further, the reviewers acknowledged that the proposed approach seems general (not limited to image-to-video setting) and suggested evaluating on a more general domain adaptation setting. 

Given these concerns, one reviewer downgraded their score from 8 to 6 during the discussion phase.

This meta reviewer carefully read all the reviews and the rebuttal, and found no strong reasons to overturn the reviewers' recommendation. We are recommending rejection in its current form of this submission.
justification_for_why_not_higher_score: The concerns area critical and shared by the reviewers. No strong reason was found to overturn their recomendation.
Decision: Reject