Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a self-supervised approach for target domain adaption specifically for 360 room layout estimation. The idea is to leverage the consensus of the predicted layout from multiple panorama of the same scene. A weighted boundary consistency loss is proposed to learn from pseudo ground truth with uncertainty. A multi-view consistency metric is proved to be strongly correlated with the room layout metric and thus can be used to tune hyper-parameters even with no access to the ground truth.
strengths_and_weaknesses: - The method is technically sound. The self-supervised learning is an important and effective solution for domain adaptation.
- Evaluation on multiple datasets show positive contribution by using the proposed method.
- Ablation study is provided to show the contribution of individual technical components.

questions: - How is the camera pose on the target dataset obtained? Is it calculated from color images or relies on external sensors?
- In Tab 4, it seems that disable uncertainty loss, some of the metrics get better. I'd like to hear authors' comments on this.
- Is the SSLayout360 in Tab 1 and 2 trained only on pre-trained dataset and never sees images from target domain?

limitations: - It seems that the 6dof camera poses of panoramas are required to do the projection. Hence, precisely speaking, the method is not fully self-supervised but requires camera pose ground truth. This is usually accessible, easier compared to the ground truth layout, but may also cause error for the layout projection and thus hurts the overall finetuning performance.

- The experiment could be stronger to demonstrate the effectiveness of the method from two aspects: 1) a stronger baseline. It seems SSLayout360 is in general outperforming HorizonNet. It would be convincing to show that this method is able to improve powerful backbones. 2) analyze the domain gap. It would be nice to add some discussions about the gap between datasets. Some datasets are closer to each other thus the adaption may not be a big issue. Also, if the method is able to finetune a pre-trained model on synthetic data, then the value of the approach would be much higher.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a framework for finetuning monocular indoor layout estimation models to target domain without ground truth labels. The framework consists of a self-supervise loss using output layouts and uncertainty from multiple views, as well as a metric for evaluating the finetuned model and for hyper-parameter tuning. The paper evaluates against baseline methods both qualitatively and quantitatively in the aforementioned settings, and demonstrates the effectiveness of the proposed scheme.
strengths_and_weaknesses: Strengths:

[1] Novelty. The task setting and the solution is novel, in that a purely self-supervised loss and a metric for finetuning are proposed, which is shown as both intuitive and effective. 
[2] Evaluation. The evaluation is sound and comprehensive.

Weakness:
[1] Limited improvements quantitively. On the one hand, limited qualitatively results and comparisons are included in the paper, making it difficult to interpret the effectiveness of the proposed other than the benchmarks. Given the two samples in Fig. 4, the improvement upon baseline methods and pretrained model is limited, where the proposed method shows improvements in some regions but misses out in a few others. The quantitative numbers show improvements, but it is difficult to interpret how the improvement in numbers translate to cross-view consistency (e.g. from BEV). Also limited number of baselines are proposed. I wonder if it is possible to look for or adapt more previous methods to this setting for evaluation.
questions: Please see the Weakness section for questions.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method to transform a model that estimates room layout boundaries from multiple views from a source to a target dataset. This allows finetuning an existing model on a new dataset without ground truth labels. 

strengths_and_weaknesses: Strengths:
* The idea to finetune a model based on multi-view boundary consistency is interesting, as it allows finetuning a model without additional labeled data on the target dataset. The proposed method only requires calibrated camera views, and the proposed loss is effective, as shown by the experiments.
* The proposed ablation studies properly motivate the design choices taken for the loss. 
* The paper is very well written and easy to follow. The paper also contains a supplementary document that adds missing details for reproducibility and explanations for the used camera projection model.
* The qualitative evaluations show that the model finetuned with the proposed loss cleans up the noisy predictions of the pretrained model.

Weaknesses:
Overall, I do not have many points of criticism with this paper. The paper is well-motivated, the idea is simple yet effective, and the paper has been thoroughly polished. 
* In my opinion, the experiments should show a comparison to a model that uses labels from the target dataset as an upper bound of the achievable performance. This will help put the proposed method into perspective. 

* Only comparing against SSLayout360 undermines the claim that the method has been extensively tested against other state-of-the-art methods. Consider adding more baselines, such as [4, 22]. 

questions: * I am wondering what was the reasoning of only comparing against the SSLayout360 method and not against other methods mentioned in the literature review, such as [4, 22]? 

* How does this method compare against approaches that have access to large amounts of training data? Is labeling a lot of data worth the effort or not?
limitations: I can see that limitations and ethical considerations are not discussed in this paper. Consider adding them.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method to predict room layouts from a set of registered 360 panorama given an arbitrary single-image room layout prediction model trained on a different distribution of data. The main idea is to use reprojected estimates to compute multi-view consistency variance, and then sample from that variance to create pseudo-labels for training. The paper also proposes an entropy metric for evaluating geometric consistency across views. This approach requires no additional ground truth for adapting pre-trained models to new datasets, thus providing an effective way to transfer room layout prediction models to new data.
strengths_and_weaknesses: __Technical Contribution__

This paper presents a reasonably straightforward yet very practical solution for fine tuning a model with multi-view constraints without the need for additional data. The weighted boundary consistency loss in Section 3.2 is an effective way to synthesize the multiple predictions from other views, and the visualizations in Figure 1 and Supplementary Figure 1 provide an intuitive illustration of how it performs. Additionally, the entropy metric seems like an elegant answer to the challenging problem of evaluating results without ground truth data.

This paper provides an in-depth experimental analysis looking at performance improvements on a target dataset when using three different widely-used datasets for training. The quantitative results in Tables 2 and 3 demonstrate clear improvement on standard benchmark metrics compared to the most relevant prior work [34]. Moreover, the paper provides an ablation study that clearly justifies the design decisions of the reprojection loss computation and adds insight to the question of diminishing returns with more images. To the latter question, the results in Table 4(a),(b), and (e) seem to justify the intuition that more reprojection would provide a more stable distribution for pseudo-labels.


__Presentation__

The paper is altogether well written and effectively organized. The writing is clear, and all equations are sufficiently explained. The figures add explanatory value to the paper as well, and the tables are laid out in a clear way that make the conclusions obvious to the reader.

__Summary__

This is a strong paper that provides a simple but seemingly effective method to adapt pre-trained models to new datasets without extensive data labelling. The positive indicators in the quantitative results and the practicality of the solution suggest that this paper will have a reasonable impact on future research into the problem. There is some question about the usefulness of the proposed entropy metric as an evaluation metric in lieu of labels data, but the other hard evaluation metrics show strong gains, justifying the core methodology. This is a somewhat niche application in computer vision--360 room layout estimation--so the paper might benefit from some discussion about the generality of this approach to other problems. However, the relative size of the research community working on this problem should not hinder this paper's acceptance to NeurIPS. I recommend this paper for acceptance.
questions: * Although this paper presents the multi-view consistency, pseudo-label approach for the task of room layout estimation, the methodology described appears to be more generally applicable. Could the authors share their insight into whether this approach would be transferrable for other tasks in a multi-view setting? For example, object detection, segmentation, or depth estimation.

* One result stands out as curious. While generally the results in Tables 2 and 3 support the proposed method, it is interesting to observe that the entropy metric, $H_{MLC}$ does not particularly favor the approach proposed in the paper. In fact, those tables report similar or even lower entropy (i.e. better consistency) using the simple pre-trained model or the student-teacher approach of [34]. On the other hand, the IoU and depth metrics show consistent significant improvements with the proposed multi-view, self-training approach. Does this indicate that $H_{MLC}$ is simply not an effective metric for evaluating the quality of the model? If so, it would seem to stand counter to the effectiveness of the hyper-parameter tuning shown in Figure 3.
limitations: This paper does not explicitly address any limitations or social impact of this work.
review_title: null

Meta-review: metareview: The reviewers were all supportive of this paper and commended the author’s efforts to clarify their questions with follow up responses and additional experiments. This area chair agrees and recommends acceptance.   

The authors are encouraged to take the reviewer comments on board in the final camera ready paper. This includes the promise to discuss the limitations raised by the reviewers (e.g. z7v1 and CFmm). The additional results with a different backbone and non-GT poses can be included in the supplementary material if there is no space in the main paper.  

Minor comments:  
Suggested text change in abstract:   
“This comes in handy in …” -> “This can be valuable in …”   

confidence: Less certain
Decision: Accept