Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a method on generalization of segmentation from synthetic data to real street scene data. To adapt the model pre-trained with synthetic source domain data such as GTA and SYNTHIA to target domain such Cityscapes, BDD and IDD, the authors proposes Instance-adaptive Batch Normalization (IaBN). In addition, as a method to learn from a single sample, TT-SEG was proposed
in which a pseudo GT mask is estimated from augmented images with the initial model and the last parts of the network are finetuned with the pseudo GT mask. In the experiments, the proposed method which employs both IaBN and TT-SEG successfully outperformed all the baselines, which indicated the effectiveness of the proposed method.


main_review: Pros)

The paper is well written and easy to understand. The experiments including comparison to baselines and ablation studies seem to be comprehensive.

Cons)

The idea on combining BN and IN has been proposed before. The novelty is limited. In addition, in this work, the hyperparameter, alpha, has to be decided by validation data.

TT-SEG is a simple method for test-time finetuning. Although the effectiveness was shown by the experiments, the novelty seems to be limited.

The description on baselines such as ASG and CSG in Related Work is too brief to understand the principle difference to the proposed method and among them. 

Taking six seconds is too much from practical point of view. As the authors mentioned, the proposed method is unpractical for road scene analysis.

Comment)

Why was ECE used only for Table1(b) without showing MIoU ? Is MIoU also improved by introducing MC-Dropout ?
 
summary_of_the_review: The novelty of the proposed method is limited, since IaBN is an existing method and the method on one-sample adaptation is straight-forward. Although the effectiveness of the proposed method regarding performance was shown by the experiments, the required processing time was too much. From these points, this paper is regarded as being around the border.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper contributes two techniques to improve generalization of semantic segmentation networks. The first technique is an adaptation of the test-time behaviour of batch normalization, where the statistics of the sample under consideration are blended into the training-time statistics of the batch-norm layer. The second technique is to use test time augmentations to first derive a pseudo-labeling from the sample and then to update parts of the weights to maximize the probability of the pseudo-labels. The paper additionally contributes a multi-dataset evaluation procedure and shows favorable performance of the proposed techniques on this evaluation protocol.
main_review: Strengths

- Test-time training is an interesting concept and the paper spans a nice logical bridge from the first, technical rather minor conribution, that improves calibration of the prediction to leverage areas that are likely predicted correctly to further improve th inference results.

- Interesting, but a bit heuristic, combination of test-time training and pseudo-labeling.

- Good results and broad evaluation.

Weaknesses:

- Inference speed: Test-time training is by definition rather costly and the proposed approach increases runtime by more than an order of magnitude when compared to the baseline. It is unclear how to improve the practicality of the approach.

- When comparing to Wang 2021a it is not clear to me how much improvement actually comes from using multi-scale predictions and augmentations. It would be great to have an experiment along these lines.

- No comparison to (offline) pseudo-labeling approaches (a recent example would be [A]). The overall strategy would also be applicable without test-time training in principle. I.e. one could derive pseudo-labels for the whole target dataset and adapt the classifier once on the full target dataset. While the assumption here is that only a single image from the target domain is available, the setting seems to be rather artificial as a few unlabeled images from the target domain should almost always be available (and test-time training might help on top of such a transfer procedure).

Other questions:

- It would be interesting to see some failure cases (if any). Does test-time training always improve results?

[A] Zou et al., PseudoSeg: Designing Pseudo Labels for Semantic Segmentation, ICLR 2021


summary_of_the_review: I think the overall approach is interesting, but ad-hoc in its design. It would be great to see some comparisons to existing pseudo-labeling approaches as I think that the setting with only a single image from the target domain artificial.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The article proposes two techniques for test-time adaptation. The first is instance-adaptive bn, to combine statistics from source data with each single test sample.  The second is test-time training, based on pseudo labeling of the test sample. The two parts lead to promising results.
main_review: Pros:
1.	The writing is good, making the paper very easy to understand.
2.	Both the two modules are conceptually simple. 
3.	Extensive experiments have been performed for evaluation.
4.	Code is provided.

Cons:
1.	My major concern goes to the novelty of the paper. The proposed normalization technique is incremental. Instance-level normalization is a smart way for test-time adaptation, which has been well studied [1][2][3]. For the test-time training part, it is also hard to claim a unique contribution since it mostly follows a pseudo labeling-based segmentation paradigm. 

  [1] Test-time adaptable neural networks for robust medical image segmentation
  [2] Autoencoder based self-supervised test-time adaptation for medical image analysis
  [3] Tent: Fully Test-Time Adaptation by Entropy Minimization

2.	It is not clear to me the advantage to learn \alpha using a validation set, rather than online learning during test-time finetuning. 
3.	The necessity to use an independent development set for model/parameter selection needs more studies. It is also not clear to me how to select a universe development set that can fit any target domain, or should we find a unique dev set for each target domain.
4.	The model is currently evaluated using relatively lightweight backbones (Res50, HRNet18), and it is not clear whether the method can work well for larger models (e.g., Res101, HRNet48) and lead to consistent improvements.

summary_of_the_review: Overall, the paper has some merits (e.g., good writing, promising results). However, the technical contributions seems to be minor. I am concerning that the paper may be still under the level of ICLR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies an existing problem - domain generalization for semantic segmentation of urban scenes. The technique keys of this work has two main points: instance-adaptive batch normalization and testing-time training using pseudo labels. Instance-adaptive batch normalization aims to bias to the data distribution of individual testing examples, which is a tradeoff of previous t-BN and p-BN. This paper generate pseudo labels for test-time training, which is not quite novel. To evaluate the effectiveness of the proposed method, this paper conducts experiments on GTA5/SYNTHIA -> Cityscapes, BDD100k and IDD, and shows better results.

main_review: There are strengths of this paper: 
+ The paper is overall well-written and the reviewer can easily follow the whole draft and get the key points of this paper. 

+ This paper shows consistent better than previous method. Besides, this paper also analyze two components of this work to demonstrate the effectiveness of their work. According to the experiments  and ablation study using GTA5, Cityscapes, BDD100k, IDD, their method achieves more favorable results than some previous methods including t-BN and p-BN, where their method is a tradeoff between them.  Besides, this paper also shows better results than previous test-time training method. 

The weakness of this paper are as follows:
- The novelty of this work is the main drawback. The first point instance-adaptive normalization is a trivial solution, which is the tradeoff of previous t-BN and p-BN. Second, there are a lot of work applying pseudo labels in semantic segmentation, such as Naive-students [ECCV2020]. Even though this work leverages pseudo labels for a different application, the technique contribution is not strong.

- Comparison to more previous work [a,b,c] in domain generalization in semantic segmentation is needed. 
[a] RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening, CVPR, 2021.
[b] FSDR: Frequency Space Domain Randomization for Domain Generalization, CVPR, 2021.
[c] Domain Randomization and Pyramid Consistency: Simulation-to-Real Generalization without Accessing Target Domain Data, ICCV, 2019.

- In instance-adaptive batch normalization paragraph, the authors use \head{u}_{c}^{s}, but in Eq (4), the authors use u_{c}^{s}. Need to confirm the notations in the paragraph.

- In the experiments, the paper uses 0.1, 0 and 1 for the alpha in Eq 4, where 0 and 1 are the baseline method. Is the method sensitive to alpha in acheving good results, more analysis is needed. For various target datasets, they have different levels of domain gap, it is unclear how to choose the parameter for \alpha.


Question:
1. What is the motivation to test on the expected calibration error with a combination  of MCDropout ?
2. How to choose alpha for Eq. 4? 



summary_of_the_review: This paper presents a new model and pipeline for domain generalization in semantic segmentation. Although this work shows better results than previous work. Lacking of novelty and insights is the main drawback of this work. Besides, the experiments can be improved, that more comparisons with previous work is helpful. Therefore, the reviewer rates this paper marginally below the acceptance threshold.
review_title: null

Meta-review: This submission presents a technique to improve generalization of urban scenes segmentation.  Based on a pre-trained deep net on synthetic data, the approach aims at adapting statistics on real target domain such as Cityscapes, BDD or IDD datasets using an Instance-adaptive Batch Normalization (IaBN) at test time. Results are reported on several synthetic to real scenarios.

Most of the reviewers were not convinced by the approach and have raised several issues. After rebuttal and discussion, no one really changed her/his mind. The novelty of the proposed method is limited to the use of the existing IaBN in this context except the one-sample adaptation.  Although the proposed method is effective on some benchmarks, the extra processing time may be a significant limitation. Additional comparisons are necessary. We encourage the authors to consider the reviewers feedbacks for future publication.
Decision: Reject