360-MLC: Multi-view Layout Consistency for Self-training and Hyper-parameter Tuning
Authors: Bolivar Enrique Solarte
Chin-Hsuan Wu
Yueh-Cheng Liu
Yi-Hsuan Tsai
Min Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-training
scene understanding
omnidirectional vision

Abstract: We present 360-MLC, a self-training method based on multi-view layout consistency for finetuning monocular room-layout models using unlabeled 360-images only. This can be valuable in practical scenarios where a pre-trained model needs to be adapted to a new data domain without using any ground truth annotations. Our simple yet effective assumption is that multiple layout estimations in the same scene must define a consistent geometry regardless of their camera positions. Based on this idea, we leverage a pre-trained model to project estimated layout boundaries from several camera views into the 3D world coordinate. Then, we re-project them back to the spherical coordinate and build a probability function, from which we sample the pseudo-labels for self-training. To handle unconfident pseudo-labels, we evaluate the variance in the re-projected boundaries as an uncertainty value to weight each pseudo-label in our loss function during training. In addition, since ground truth annotations are not available during training nor in testing, we leverage the entropy information in multiple layout estimations as a quantitative metric to measure the geometry consistency of the scene, allowing us to evaluate any layout estimator for hyper-parameter tuning, including model selection without ground truth annotations. Experimental results show that our solution achieves favorable performance against state-of-the-art methods when self-training from three publicly available source datasets to a unique, newly labeled dataset consisting of multi-view images of the same scenes.

360-MLC: Multi-view Layout Consistency for Self-training and Hyper-parameter Tuning
Bolivar Solarte"! Chin-Hsuan Wu Yueh-Cheng Liu', Yi-Hsuan Tsai Min Sun
National Tsing Hua University, PPhiar Technologies
https://earriquesolarte htwpollrrtte...ttbbbee.e 10/36000311mm m
Abstract
We present 360-MLC, a self-training method based on multi-view layout consis- tency for finetuning monocular room-layout models using unlabeled 360-images only. This can be valuable in practical scenarios where a pre-trained model needs to be adapted new data domain without using any ground truth annotations. Our simple yet effective assumption is that multiple layout estimations in the same scene must define consistent geometry regardless of their camera positions. Based on this idea, we leverage pre-trained model to project estimated layout boundaries from several camera views into the 3D world coordinate. Then, we re-project them back to the spherical coordinate and build a probability function, from which we sample the pseudo-labels for self-training. To handle unconfident pseudo-labels we evaluate the variance in the re- ee-proeected boundaries as an uncertainty value to weight each pseudo-label our loss function during training. In addition, since ground truth annotations are not available during training nor in testing, we leverage the entropy information in multiple layout estimations as quantitative metric to measure the geometry consistency of the scene, allowing to evaluate any layout estimator for hyper-paramete tuning, including model selection without ground truth annotations. Experimental results show that solution achieves favorable performance against state-of-the-aa methods when self-training from three publicly available source datasets unique, newly laheled dataset consisting of multi-view images of the same scenes.
1 Introduction
Room-layout geometry one of the fundamental geometry representations for an indoor scene, which can be parameterized with points and lines describing corners and wall boundaries. Therefore, this geometry has been largely used as a primary stage for challenging tasks like robot localization [5, 42], scene understanding [24], floor plan estimation [29], etc. Several methods have been proposed for estimating layout geometry from imagery [12, 13, 45], while the current state-of-the-es methods [19, 10, 17, 47, 15] leverage deep learning approaches regress the wall-ceiling and floor boundaries directly from monocular 360-images in supervised manner.
However, deploying layout model using 360-images in new target domain remains challenging problem. For instance, a pre-trained layout model may estimate inconsistent geometry, due to novel view positions, different lighting conditions, severe object occlusions in the scene, especially for large and complex rooms. handle these issues, ground truth annotations the new target domain are usually required to finetune the model, which involves cumbersome data labeling process. Moreover, considering the large variety of indoor scene styles, a room-layout model may
"The authors contribute equally to this paper. CCurrently at Google.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(c) Pseudo Label Y
Set of 360-images
(a) Set of Layout Boundaries
(b) Multi-view Layout Re-projection
Room-Layout Model
Uncertainty Ø
Weighted Boundary Consistency Loss
LWBC
Figure I: 360-MLC pipeline. (a) Our method uses a pre-trained room-layout model and collects the layout estimations from multiple views in the same scenes; (b) we re-project multi-view estimates to the target view (the image with the orange bounding box); (c) 360-MLC then generates the pseudo- label and the uncertainty to compute the consistency loss self-supervisions without requiring ground truth labels in the target scene.
demand more new labeled data to adapt to another target domain. In this paper, aim to make pre-trained room-layou model self trainable in the new target domain without using any annotated data during model finetuning, in which such setting is practical but has not been studied widely for room-layout estimation.
To this end, we present 360-MLC, self- self-training method based on Multi-view Layout Consistency (MLC), capable of adapting a pre-trained layout model into new data domain using solely registered 360-images. Our main assumption is that multiple layout estimates must define together a consistent scene geometry regardless of their camera positions. Based on this idea, we project several estimated layout boundaries into the world coordinate with the Euclidean space and re-project them back to target view (spherical coordinate) for building a probability function, from where we sample the most likely points pscudo-labels. In addition, we evaluate the variance in the re-projected boundaries as an uncertainty measure to weight our pscudo-labels for unconfident boundary regions. We combine both our pseudo-label and its uncertainty into the proposed weighted boundary consistency loss (CwBc), which allows us to finetune a layout model in a self-training manner without requiring any ground truth annotations. Therefore, pseudo-labels generated via our MLC have higher quality than pseudo-labele predicted from a single view, which is crucial for achieving satisfactory performance when no annotations are available in the new domain.
We further propose the multi-view layout consistency metric based on entropy information (Hvn.c) that quantitatively evaluates any layout predictor without requiring ground truth labels. This metric can be used to monitor the quality of the predicted layouts in both training and testing. As a result, our HMLC metric can be used for ground truths are available in the new domain. Our key idea behind HMLC is to evaluate the entropy information from multiple layouts projected into discrete grid as 2D density function, where better geometry alignment of layouts would yield a lower entropy evaluation. We leverage our HMLC metric across different experiments and show its versatility and reliability for the multi-view layout setting.
tuning and model selection, which is practical when no
In experiments, we leverage the MP3D-FPE [29] multi-view dataset as our unlabeled new domain. Since layout ground truths are not available in this dataset, we manually annotate 7 scenes with 700 frames for only the performance evaluation purpose. For the dataset used in model pre- tre-training,
validate our framework on three real-world room-layout estimation datasets: MatterportLayout [36,
48], Zillow Indoor Dataset (ZInD) [9], and the dataset used in LayoutNet [47]. We show that method is able to handle two practical settings during training: 1) having both the labeled pre- pre-trained data and the unlabeled data in the target domain, 2) only providing the unlabeled data with the pre-traine trained model. Moreover, we demonstrate the usefulness of the proposed modules, including the weighted boundary consistency loss with pseudo-labels, and the multi-view layout consistency metric that facilitates the hyper-parameter tuning without the need for ground truth labels.
Our contributions based on the idea of Multi-vien view Layout Consistency are summarized as follows:

---

1. Based on multi-view geometry consistency, we propose a self-training framework for room- layout estimation, requiring only registered 360-images as the input.
2. We propose the weighted boundary consistency loss function (CwBc) that uses multiple estimated layout boundaries with their uncertainty to improve the self- foff-traaiing process using pseudo-labels.
3. We introduce the multi-view layout consisteney metric (HMLc) for measuring multi-view layout geometry consistency based on entropy information. This metric allows us to evaluate any layout estimator quantitatively without requiring ground truth labels, which enables hyper-parameee tuning and model selection.
2 Related Work
Indoor room layout estimation. Estimating the layout structure of room for cluttered indoor environments is a challenging task. Early methods estimate plane surfaces and their orientations based on points or edge features to construct the spatial layouts for perspective images [12, 13, 35] or panorama images (i.e. 360-images) [45, 41]. On the other hand, some approaches [3, 14, 7, 40] leverage semantic cues in the scene, such as objects or humans, to improve the layout estimation.
Deep learning approaches [19, 10, 17, 47, 15] leverage convolutional neural networks (CNN) to extract the geometric cues (e.g., corners or edges) and semantic cues (e.g.. pixel-wise segmentation), which largely enhance the performance. Recent state-of-the-art methods [43, 30, 37] are able to robustly estimate the layout of the whole room from single panorama image. CFL [11] and AtlantaNet [21] avoid the commonly-used Manhattan world assumption [8], enabling the ability to handle complex room shapes. In addition to monocular approaches, MVLayoutNet [16] and PSMNet [38] explore the usage of multi-view 360-images as inputs to further improve the layout estimations. However, training deep neural networks to perform layout estimation requires large-scale datasets with manual annotations. In this paper, we aim to mitigate this problem by learning from unlabeled data.
Self-training. To incorporate unlabeled data during training, self-training [27, 44, 23] uses a pre- trained teacher model to generate pseudo-labels for the data without ground truth annotations. Then, both the labeled data and the unlabeled data with pseudo-labels are used to train better model (i.e.. student model). Due to its simplicity, many attempts have been made for the field of semi-supervised learning [18, 4, 28] and unsupervised domain adaptation [49, 50]. Moreover, recent methods [39, 46] show that self- self-training can surpass state-of-the fully-supervised models on large-scale datasets such as ImageNet.
However, most of these works focus on self-training for classification or object detection tasks, while self-training for tasks considering geometric predictions (e.g.. room-layout estimation studied in this paper) is rarely explored. SSLayout360 [34] trains a layout estimation model with pseudo- labels produced by the Mean Teacher [33]. Yet, the simple extension from classification to layout estimation considers each image independently, ignoring the geometry information coming from other camera views. In addition, another challenge that pseudo-labels are usually noisy. Previous methods attempt to reduce the noise by ensembling multiple predictions for an image under different augmentations [4, 22] or by selecting only the pseudo-labels with high confidence [28]. this paper, we leverage multi-view consistency from the layout estimations and measure their uncertainty to construct reliable lseudo-labels.
Unsupervised model validation. Without ground truth annotations, how to evaluate a machine learning model remains an open issue. Traditional unsupervised learning algorithms (c.g., clustering)
can be evaluated without external labels through computing cohesion and separation [32]. For unsupervised domain adaptations, where the problem assumes no labeled data is available in the target domain, unsupervised validation is more practical for hyper-pane parameter tuning [20, 25]. [20] evaluates the confidence of the predictions of the classifier using entropy. [25] proposes the soft neighborhood density, measuring the local similarity between data samples (the higher, the better). However, these metrics are designed for classification or segmentation tasks and cannot easily be adopted by our tasks. Therefore, in this paper, we study unsupervised validation from another perspective: evaluating the geometry consistency between multiple views, requiring no ground truth annotations.
3

---

3 Our Approach: 360-MLC
Our primary goal is to deploy model pre-trained on source domain into new dataset (target domain), where the data distribution may differs from the one used in the pre-trained model. We assume that several images in the new scene are captured and registered by their camera poses, but their layout ground truth annotations are not available. Under this practical scenario, we present 360-MLC, self-trainie self-training method that is based multi-view layout consisteney.
For illustration purposes, Fig. presents the overview of our method. First, we begin with a set of 360-images and a pre-trained room-layout model that can generate a set of estimated layout boundaries (see green lines in Fig. I-(a)). Then, by leveraging the related camera poses of every image, all layout boundaries are projected into the world coordinate and then re- re-projected them back into a target view (see yellow lines in Fig. I-(b). Details of these projections are described $..1.
Assuming that the projected boundaries from all the views describe the same scene geometry, we can compute the most likely layout boundary positions as pseudo-labels for self-training along with their uncertainty based on the variance (see Fig. I-(c)). Upon the estimated pseudo-labels, we define our weighted boundary consistency loss LWBC. allowing us to define reliable regularization used for self- self-training. More details are presented in $3.2.
A challenging step towards our goal is the lack of metric for evaluating layout estimator when no ground truth annotations are available. To tackle this issue, $3.3 describes our multi-view layout consistency metric HMI.C. which allows to evaluate multiple layout estimates without requiring any ground truth. We leverage the proposed metric throughout all the experiments described in $4 as the quantitative metric for model selection and hyper-parameter tuning.
3.1 Multi-view Layout Re-projection
In this section, describe the multi-view layout re-projection process of all estimated layout boundaries from different cameras positions in the scene. To this end, we define the set of 360-images and boundaries in the scene as follows:
(L, YihLEN,, I, LERHxW YERW,
(1)
where I is the i-th 360-image with the size of W columns times H rows pixels, Y, is the layout boundary of image I,. and N is the number of images. Y, R R is a vector of boundaries at all columns, where (())) = 6 specifies that the layout boundary is at row & for column @ in the pixel coordinate.
For supervised training. Y; is given as ground truth label. However, in our proposed self-training framework, we aim to ensemble pseudo-labels through geometry re-projection of multiple views. To begin with, we describe the process to project the boundary Y, into the j-th target view as the re-projected boundary Y- The projection of Y, into the world coordinate using the Euclidean space described as follows:
X; = (Y, Tr, h4)
where X; is the projected layout boundary in the world coordinate; h; is the camera height; 7; E SE(3) is the camera pose with respect the world coordinate; Proj(-) is the layout projection for spherical cameras that maps spherical coordinates [0, ø) E R² into the 3D world coordinate [x,, y, x] R3 Details this projection function Proj(-) and how we handle unknown camera heights with estimated poses are presented in the supplementary material.
Then, X; in the world coordinate can be re-projected into a j-th target view (Yg=z) as follows:
Y = Proj [ooo--(TT,,,,
(3)
where P-)j-1(-) is the inverse of the projection function presented in Eq. (2). We collect all re- projected boundaries into Y, = [Yi-ji...; YN-z] E W where Y; E R is a vector of row positions on boundaries from N views at column 0. As shown in the yellow lines in Fig. 1-(b), the visualization of Y, reveals the underlying geometry of the scene. More visualization results are presented in $4.2.

---

(a) Pre-trained model
(b) After our self-training
Figure 2: 2D density function of multi-view layouts. We project multiple layout boundaries into a top-view 2D density map that reveals the geometry consistency in the scene. (a) We present the projection of multi-view layout using pre-trained model before self-training. (b) We observe better layout consistency of the scene after our self- self-training trainiiin process. In $3.3, we use such a top-view map for calculating our layout consistency metric HMLC.
3.2 Weighted Boundary Consistency Loss
In this section, we describe how multiple re- re-projected boundaries can be used to estimate pseudo-label boundaries and its uncertainty as a reliable supervision for our self-training formulation. First, we obtain a set of predicted boundaries (V; = MMIa)) for all views using the pre-trained model M. Following the process in $3.1, we re-project the estimated boundaries into Y, for each view. We define the pseudo-label (Y₁) and its uncertainty (a,) as follows:
Y,- [edian Y (
F. oj (Y (0)) RWW.
(4)
where Median(.) and STD(-) are the median and standard deviation both applied Y (0) (i.c., N re-projections of row positions on boundary at column target view. By leveraging Eq. (4), we define our geometry loss as follows:
0). The pscudo-labels are computed for each
N W |Y; - Y,00
LWBC ΣΣ
(5)
Lwic is our proposed weighted boundary consistency loss for finetuning the room-layout model M. Note that the denominator weights the pseudo-label accordingly as the uncertainty at each column This design aims reduce the effect of unstable predictions in the scene (e.g., drastic occlusions) that generally incur a larger variance in the re-projection due to the inconsistency between multiple views.
3.3 Multi-view Layout Consistency Metric
In practice, one challenge for adapting a pre-trained model into a completely unlabeled new data domain is that there no labelled hold-out dataset for tuning typer-parameters such as learning rate, etc. Hence, we need metric to measure the performance based unlabeled data. To this end, propose a the HMLC metric to measure the multi-view layout consistency that does not rely on ground truth labels but on model outputs only.
First, we collect all estimated layout boundaries in the scene projected by Eq. (2) build top-view 2D density map. This can be described as follows:
x (X+)i=NN
(XX) RUV
where X is the set of all estimated layouts in the scene, and 4(-) is top-down projection function that maps X into a discrete 2D-grid with the size U x as a normalized histogram. Our key idea is evaluate the entropy in this discrete grid follows:
HMLC = Σ - (X) log +b (X).
The intuition behind this evaluation comes from the fact that better alignment of layout boundaries yield in lower entropy evaluation, while higher entropy value would reflect the poor alignment of
5

---

Table I: Dataset statistics.
Source Dataset
Number of Frames
Target Dataset
Number Frames
MatterportLayout [48]
2094
MP3D-FPE [29]
2094
ZInD [9]
2094
MP3D-FPE
2094
LayoutNet [47]
817
MP3D-FPE
817
layout geometry between multiple views. As result, we can use HMIC to select hyper-parameters and early stop the model training, without using any ground truth annotations. For illustration purposes, Fig. 2 presents two projected scenes that show how the geometry consistency correlates with less disordered 2D projection. More details are presented in the supplementary material.
4 Experiments
4.1 Experimental Setup
Datasets. We conduct extensive experiments using publicly available 360-image layout datasets: Matterport3D Floor Plan Estimation (MP3D-FPE) [29] as the target dataset, and three real-world datasets as the pre-training datasets, including MatterportLayout [36, 48], Zillow Indoor Dataset (ZInD) [9], and the dataset used in LayoutNet [47] that combines PanoContext [45] and Stan- ford2D3D [1] (referred to as the LayoutNet dataset for simplicity). MP3D-FPE is collected using the MINOS simulator [26] to render sequences of 360-images within each scene/room from the Matterport3D dataset [6]. Note that is the only dataset containing multi-view 360-images and thus we consider it as the target dataset.
In experiments, we aim to pre-train the layout model using each of the pre-trained dataset, and then we utilize the target MP3D-FPE dataset for self-training and performing evaluation. We follow the standard training split released in each pre-trained dataset, For the target dataset MP3D-FPE, we use 2094 frames as the training set and 700 frames as the testing set, from 32 and 7 scenes, respectively, where those scenes are not included in the training set of the Matterport3D dataset. On average, there are 10.46 views within each room. Note that the target MP3D-FPP dataset is challenging dataset since includes many complex scenes that do not follow the Manhattan assumption [8], and we carefully label the ground truth 2D) and 3D layout for the testing set. As result, compared to other datasets, the performance scores on MP3D-FPE are lower when the model evaluated using state-of-the-ar methods.
Evaluation metrics. We follow Zou et al. [48] to construct the four standard protocols for evaluation. To evaluate layout boundary, we use 2D and 3D ontersection-of-ree union (IoU). To evaluate layout depth, we use root-mean-soue suuun- error (RMSE) by setting the camera height as 1.6 meters, and 81, which describes the percentage of pixels where the ratio between the estimation and the ground truth depth within the threshold of 1.25. In addition, we introduce a new metrie HMLC outlined in $3.3 to analyze the consistency between multiple layout estimations. We show in $4.3 that HaLe is highly co-related to 2D/3D loUs on the hold-out dataset. Hence, HMLC suitable for syper-parameter tuning when ground truths are not available.
Implementation details. We adopt HorizonNet [30] as our layout estimation backbone due to its stateoof-the-at performance. Note that, different from the original HorizonNet, our model is trained without the supervision on the corner channel since our pseudo-label contains only the boundary information. Common data augmentation techniques for 360-images are also applied during training, including left-right flipping, panoramic horizontal rotation, and luminance augmentation. We use the Adam optimizer to train the model for 300 epochs by setting the learning rate as 0.0001 and the batch size as 4. We save the model every epochs and the early-stopped model selected based on the lowest HMLC score on the hold-out test dataset for all methods. All models are trained on single NVIDIA TITAN X GPU with 12 GB of memory. We will make our models, codes, and dataset available to the public.

---

Table 2: Evaluation results of Setting on MP3D-FPE [29].
Pre-trained Dataset
Method
2D IoU (%)
3D IoU (%)
RMSE
HMLC' 4
Pre-trained
65.38
62.28
0.58
0.78
8.18
Matterport ayout [48]
ayouu3360 [34]
70.53
66.74
0.48
0.82
8.15
Ours
71.50
67.70
0.46
0.82
8.10
Pre-trained
45.43
42.17
1.02
0.61
8.32
7InD [9]
SSI SSlaayou3600
62.62
58.27
0.66
0.74
8.34
Ours
65.60
60.70
0.56
0.75
8.19
Pre-trained
64.34
58.92
0.61
0.70
8.50
LayoutNer [47]
SSI ayyou3366
67.48
62.89
0.56
0.77
8.29
Ours
69.40
65.22
0.55
0.72
8.32
Table 3: Evaluation results of Setting 2 MP3D-FPE [29].
Pre-trained Dataset
Method
2D (%)
3D IoU (%)
RMSE
81 f
HMLLC
SSLayou1360-ST
70.58
66.64
0.52
0.81
8.10
MatterportLayout [48]
Ours
71.50
67.20
0.49
0.78
8.14
SSLayout360
56.52
$2.12
0.72
0.74
8.20
ZlnD 19]
Ours
64.45
59.53
0.59
0.75
8.17
SSI ayyut33066-
66.14
61.51
0.57
0.76
8,30
LayoutNet [47]
Ours
69.12
64.62
0.57
0.69
8.31
4.2 Experimental Results
Baselines. The proposed framework is compared against our re-implementation of SSLay- out360 [34]. We ensure that our re-implemented SSLayout360" has similar performance compared to the reported results in [34]. Note that MLC and SSLayout360 and we train models using only the boundary supervision as stated $4.1. We implement extended self-training version based SSI ayout336""" namely SSLayout360-STT. in which we disable the supervised loss and train without any ground truth annotations. We initialize all models with the different from our approach, both SSLayout360* and SSLayou(360-ST are trained on two NVIDIA TITAN X GPUs due to the requirement of larger memory for their teacher-student architecture.
the same backbone, i.e., HorizonNet, for both our 360-
pre-trained weights from the official HorizonNet' release. In addition,
Setting I: labeled pre-training data unlabeled target data. In this setting, sample the same amount of training data from both the pre-training dataset and the target dataset, as shown in Table 1. We start from the pre-trained model, and then generate pscudo-labels for unlabeled data. Then, during model finetuning, we include both labeled pre- traitrained data and pscudo-labeled data from MP3D-FPE. In Table 2, we show that our method consistently performs favorably against SSLayout360* across most metrics, in which SSLayout360" does not consider the usage of multi-view layout consistency as our approach does.
Setting 2: pre-trained model + unlabeled target data. more practical and challenging setting is that only the pre-trained model and unlabeled data in the target domain are available during model finetuning. In Table 3, similar to Setting 1, our method has consistent performance improvement against SSLayout360-ST Note that we observe that our performance gains are larger (especially on ZInD and LayoutNet) compared to Setting For Matterport Layot, since its data distribution is closer to the MP3D-FPE (both are from Matterport3D), the performance difference is less.
Moreover, when remove the labeled pre-trained data from Setting 1, we do not observe significant performance drop in Setting on all dataset settings, while SSI SSLayout360-ST more sensitive to the labeled pre-trained data, e.g., on ZInD, 2D/3D IoU is decreased by 6.1% This demonstates the
'tttps:/ htaps:///itthu wwt.///on///////eeee
http:///ggggt..oee////////////// /orizznnet

---

Figure 3: Multi-view pseudo-labeling. We show the qualitative visualization of our proposed 360-MLC. In (a), all re-projected layout boundaries from different camera views are presented as yellow lines. In (b), the corresponded pseudo-labels the pseudo labels is shown as 2D maps. We can appreciate that our proposed 360-MCL can estimate plausible pseudo-labels from estimations along the scene.
depicted magenta. In (c), the uncertainty in
Ground Truth
Ours
SSLayouu3600 [34]
Pre-trained
Figure 4: Qualitative results in the 2D top-view. We project the estimated layouts registered by the corresponding camera poses into 2D. Comparing to our baseline SSLayou1360* [34] and the pre-trained model, our model produces more consistent layouts (highlighted with red circles).
effectiveness of considering multi-view layout consistency that generates more reliable aseudo-labels. even when no labeled data is provided during training.
Qualitative results. For illustration purpose, we visualize the proposed multi-view pseudo-labeling Fig 3. Fig. 4 we show qualitative results of multiple layouts projected into the 3D world coordinate for two test scenes evaluated MP3D-FPE [29]. following the Setting 2. It can be observed that our proposed 360-MI.C presents sharper and clearer layout boundaries for those scenes, demonstrating better performance compared with the baselines. Lastly, we analysis the qualitative layout predictions on 360-images in Fig 5.
4.3
Ablation Study
In this section, we present our ablation study validate the effectiveness of the proposed components. We conduct experiments using the ZInD pre-trained weights under the Setting 2 mentioned in $4.2.

---

Figure Qualitative comparisons on 360-images. We compare our model with SSLayou1360" in 360-images. The cyan, yellow, and magenta lines are ground truth, SSLayout360", and 360-MLC, respectively. We observe that our model predicts layout boundary more closely to the ground truth than SSLayout360, and is more robust towards furniture (e.g., tables, couch) and large rooms. The dashed white bounding boxes highlight the error predictions from the baseline.
Table 4: Ablation study with Setting on MP3D-FPE [29], pre-trained ZInD [9].
Frames (%)
Median
Mean
Lwac
2D ToU (%)
ToU (%) t
RMSE +
HMLC 4
(a)
57.29
53.44
0.71
0.68
8.23
(b)
50
$9.55
55.46
0.65
0.69
8.22
(c)
100
46.17
42.63
0.82
0.63
8.22
(d)
100
62.81
58.05
0.58
0.76
8.18
(c) 100
64.45
59,53
0.59
0.75
8.17
Results are presented in Table 4, describing four main experiments. First, we investigate how the number of views used to generate our pseudo-labels may affect the performance of our proposed solution. Second, we replace the Median function in Eq. (4) with Mean, aiming to demonstrate the effects of outliers in the quality our pseudo-labels. Then, we analyze the proposed LWbC. with vanilla L1 loss function. Lastly, we show how HMILC could be used in hyper-parameter tuning, such as selecting learning rates and models.
Multi-view pseudo-labeling. In this experiment, we aim to verify whether the more views of estimations we use, the better quality of the pseudo-labels we can acquire. Therefore, we experiment three models trained with the same amount of data but using pseudo-labels created by 10%, 50% and 100% of frames, as shown in row (a), (b), and (e), respectively in Table 4. The result demonstrates the contribution of our proposed method: using the layout estimations from more views for self- teffiiii can help ensemble more robust training signals.
Median and mean in Eq. (4). In this experiment, investigate the impact of the median operator for pseudo-label generation. We compare the mean and median functions in row (c) and (e) in Table 4. It can be observed that the median function has better performance since it is capable of ignoring outliers in the re-pro re-projected boundaries from multiple views, increasing the robustness of pseudo-labels.
LWBC and C1. The results in row (d) and (c) of Table show that adding the uncertainty a in CWBC performs better than the simple L1 loss. This is because proposed loss function dewghweights the part of the layout boundaries that are noisy (i.e., high variance), avoiding the effect unreliable pseudo-labels during self-training.
HMLCC for hyper-parameter tuning. We present an example of using our proposed metric HMLC for hyper-parameee tuning. In Fig. 6, we test three different settings using the same model training

---

(c)
8.45
60
2D YoU (%6
8.40
55
MNLC
3D lou
55
8.35
50
8.30
45
0
5
10 epochs
15
20
10 epochs
15
20
0
10
15
20
epochs
Ir=1e-S, data=1k
Ir=1e:4, data=1k
Ir=1e data=2k
Figure 6: An example of hyper-parameter tuning. We show that the layout evaluation metries, (a) 2D IoU and (b) 3D IoU, evaluated using ground truth labels, are inversely correlated with the proposed metric (c) HMLC under three different settings. In this way, we are able to select better models with the lower value of HMLC. We conduct the experiment under two conditions: different learning rates and different amount of training data.
Table Evaluation results of Setting 2 on MP3D-FPE [29] using estimated poses.
Pre-trained Dataset
Method
2D IoU (%)
3D loU (%)
RMSE
61
Ours ground truth poses
71.50
67.20
0.49
0.78
MatterpontLayou [48]
Ours + estimated poses
70.85
66.91
0.48
0.78
Ours noisy poses
66.05
61.41
0.71
0.65
process. Among them, using learning rate x 10-5 and 1K training samples (green lines) performs the worst, while adopting learning rate 1 10-5 and 2K training samples (blue lines) performs the best. We show that the trend of our proposed unsupervised metric (Fig. 6-(c)) is consistent with the two supervised metrics, 2D/3D loU (Fig. 6-(a) and (b)). Therefore, without ground truth labels, our metric can be served as a robust indication for validation.
Limitations
Several views from the same scene with their registered camera poses are required to formulate the proposed 360-MLC. Although the registration of multiple camera poses be accomplished accu- rately by external sensors, structure from motion (SfM), or Simultancous Localization and Mapping (SLAM) solutions, any еиrог in this registration may lead to poor performance. To complement the experiments depicted in $4, Table shows that our proposed method can keep similar performance using estimated poses under mild noise conditions. However, under severe noise conditions, appreciate a lower performance for our proposed solution.
6 Conclusions
We present 360-MLC, self-training method based on multi-view layout consistency for finetuning monocular 360-layout models using unlabeled data only. Our method tackles practical scenario where a pre-trained model needs to be adapted to new data domain without using any ground truth annotations. In addition, we leverage the entropy information multiple layout estimations as quantitative metric to measure the geometry consistency of the scene, allowing us to evaluate any layout estimator for param-aarrettrr tuning and model selection in an unsupervised fashion. Experimental results show that our self-training solution achieves favorable performance against state-of-the-ard methods from three publicly available source datasets to the newly labeled multi-view MP3D-FPE dataset.
7 Acknowledgements
This work is supported in part by Ministry of Science and Technology of Taiwan (MOST 110-2634- F-002-051). We thank National Center for High-performance Computing (NCHC) for computational and storage resource.
10

---

References
[1] Tro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv: 100..07220000.. 2017. 6
[2] Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural
information processing systems, 27, 2014. 3
[3] Sid Yingze Bao, Min Sun, and Silvio Savarese. Toward coherent object detection and scene layout understanding. Image and Vision Computing, 29(9)569-557, 2011. 3
[4] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Paperno,, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances Neural Information Processing
Systems, 32, 2019. 3
[5] Federico Boniardi, Abhinav Valada, Rohit Mohan, Tim Caselitz, and Wolfram Burgard. Robot localization in floor plans using room layout edge extraction network. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pages 5291-5297. IEEE, 2019.
[6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments.
International Conference on 3D Vision (3DV), 2017.
[7] Yu-Wei Chao, Wongun Choi, Caroline Pantofaru, and Silvio Savarese. Layout estimation of highly cluttered indoor scenes using geometric and semantic cues. In International Conference on Image Analysis and Processing, pages 489-499. Springer, 2013.
[8] James M Coughlan and Alan L. Yuille. Manhattan world: Compass direction from a single image by bayesian inference. In Proceedings of the seventh IEEE international conference on computer vision, volume 2. pages 941 141- IEEE, 1999. 3, 6
[9] Steve Cruz, Will Hutchcroft, Yuguang Li, Naji Khosravan, Ivaylo Boyadzhiev, and Sing Bing Kang. Zillow indoor dataset: Annotated floor plans with 360deg panoramas and 3d room layouts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2133-2143, 2021. 2. 6, 7, [10] Saumitro Dasgupta, Kuan Fang, Kevin Chen, and Silvio Savarese. Delay: Robust spatial layout estimation for cluttered indoor scenes. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 616-624, 2016. 1,3
[11] Clara Leraadez-LLabrddon Jose M Facil, Alejandro Perez-Yus, Cédric Demonceaux, Javier Civera, and Jose J Guerrero. Corners for layout: End-to-end layout recovery from 360 images. IEEE Roboties and Automation Letters, 5(2):1255 1262, 2020.
[12] Alex Flint, Christopher Mei, David Murray, and Ian Reid. A dynamic programming approach to recon- structing building interiors. In European conference computer vision. pages 394 407. Springer, 2010.
1.3
[13] Alex Flint, David Murray, and Ian Reid. Manhattan scene understanding using monocular, stereo, and 3d features. In 2011 International Conference on Computer Vision. pages 2228-2235.02 IEEE, 2011. 1.3
[14] Abhinav Gupta, Martial Hebert, Takeo Kanade, and David Blei. Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. Advances in neural information processing systems, 23,
2010. 3
[15] Martin Hirzer, Vincent Lepetit, and PETER ROTH. Smart hypothesis generation for efficient and robust room layout estimation. In Proceedingy of the IEEE/CVF winter conference on applications of computer vision. pages 2912 2220090,, 2020. 1.3
[16] Zhihua Hu, Bo Duan, Yanfeng Zhang, Mingwei Sun, and Jingwei Huang. Mylayoutnet: 3d layout reconstruction with multi-view panoramas. arXiv preprint arXi::2112.0673, 20210063312...
[17] Chen Lee, Vijay Badrinarayanan. Tomasz Malisiewicz, and Andrew Rabinovich. Roomnet: End-to-end
room layout estimation. In Proceedings of the IEEE international conference on computer vision, pages
4865-4874 2017. 1,3
[18] Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3. page 896, 2013.
[19] Arun Mallya and Svetlana Lazebnik. Learning informative edge maps for indoor scene layout prediction. In Proceedings of the IEEE international conference on computer vision, pages 336-944, 2015. 1,3
[20] Pietro Morerio, Jacopo Cavazza, and Vittorio Murino. Ainimal-entropp correlation alignment for unsuper- vised deep domain adaptation. International Conference on Learning Representations. 2018.
[21] Giovanni Pintore. Marco Agus, and Enrico Gobbetti. Atlantanet: Inferring the 3d indoor layout from a single 360" image beyond the manhattan world assumption. In European Conference Computer Vision, pages 432-448. Springer, 2020.
[22] Iiiaa Radosavovic, Piotr Dollár, Ross Girshick. Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni- supervised learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 41 19-4128, 2018.
[23] Ellen Riloff. Automatically generating extraction patterns from untagged text. In Proceedings of the national conference on artificial intelligence, pages
1996. 3
11

---

[24]
Antoni Rosinol, Arjun Gupta, Marcus Abate. Jingnan Shi, and Luca Carlone. 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans. In Proceedings of Robotics: Science and Systems, July 2020.
[25] Kuniaki Saito. Donghyun Kim, Piotr Teterwak. Stan Sclaroff, Trevor Darrell, and Kate Saenko. Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9184-911889 9193, 2021. 3
[26] Manolis Savva, Angel X Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun. Minos: Multimodal indoor simulator for navigation in complex environments, arXiv preprint wxXiv1712.039.
2017. 6
(27] Henry Scudder. Probability of error of some adaptive cetteen-rccognttion machines. IEEE Transactions on Information Theory, 11(3):363-377 1965. 3
[28] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang. Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems, 33:596-608, 2020. 3
[29] Bolivar Solarte, Yueh-Cheng Liu, Chin-Hsuan Wu, Yi-Hssun Tsai, and Min Sun. 360-dfpe: Leveraging monocular 360-layouts for direct floor plan estimation. IEEE Robotics and Automation Letters, pages 1-1,
2022. 2, 6, 7, 8, 9, 10
[30] Cheng Sun, Chi- Wei Hsiao, Min Sun, and Hwann-Tzowe Chen. Horizonnet: Learning room layout with Id representation and pano stretch data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1047-1056, 2019. 3, 6
[31] Cheng Sun, Min Sun, and Hwann- Tzong Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pateen Recognition, pages 2573-2582. 2021. 3
[32] Pang-Ning Tan, Micahel Steinbach, and Vipin Kumar. Introduction to data mining, pearson education. Inc.,
New Delhi, 2006. 3
[33] Antti Tarvainen and Harri Valpola. Mcan teachers better role models: Weight-average consisteney targets improve semi- supe-super deep learning results. Advances neural information processing systems,
30, 2017.
[34] Phi Vu Tran. Sslayout360: Semi-supervised indoor layout estimation from 360 panorama. In 202/ IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 15348-15357. IEEE
Computer Society, 2021. 3, 7,
[35] Grace Tsai, Changhai Xu, Jingen Liu, and Benjamin Kuipers. Real-time indoor scene understanding using bayesian filtering with motion cues. In 201 International Conference Computer Vision, pages 121-128.
IEEE, 2011.
[36] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. Layoutmp3d: Layout annotation of matterport3d. arXiv preprint arXiv:2003, 13516, 2020. 2,
[37] Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. Led2-net: Monocular 360deg layout estimation via differentiable depth rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021. 3
[38] Haiyan Wang. Will Hutchcroft, Yuguang Li, Zhiqiang Wan, Ivaylo Boyadzhiev, Yingli Tian, and Sing Bing Kang. Psmnet: Position-and -wwrr stereo merging network for room layout estimation. arXiv preprint
arXiv:2203. 2022. 3
[39] Qizhe Xie, Minh Thanh-Thag Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020. 3
[40] Jiu Xu, Björn Stenger. Tommi Kerola, and Tony Tung. Pano2cad: Room layout from a single panorama image. In 2017 IEEE winter conference on applications of computer vision (WACV), pages 354-362. IEFE,
20 3
[41] Hao Yang and Hui Zhang. Efficient 3d room shape recovery from a single panorama. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5422-5430, 2016. 3
[42] Shichao Yang and Sehastian Scherer. Monocular object and plane slam in structured environments. IEEE Robotics and Automation Letters, 4(4):314531152, 2019.
[43] Shang-Ta Yang, Fu-En Wang. Chi-Han Peng, Peter Wonka, Min Sun, and Hung-Kuo Kuo Chu. Dula-net: A dual-projection network for estimating room layouts from single rgh panorama. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3363-3372, 2019.
[44] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196, 1995.
[45] Yinda Zhang. Shuran Song, Ping Tan, and Jianxiong Xiao. Panocontext: A whole-room 3d context model for panoramic scene understanding, In European conference on computer vision, pages 668-686. Springer, 2014. 3,
[46] Barret Zoph, Golnaz Ghiasi, Tsung- Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self self-training. Advances in neural information processing systems, 33:3833- 3845, 2020. 3
12

---

[47] Chuhang Zou, Alex Colburn, Qi Shan. and Derek Hoiem. Layoutnet: Reconstructing the 3d room layout from single rgb image. In Proceedings of the IEEE Conference Computer Vision and Pattern
Recognition, pages 2051-2059, 2018. 1, 1, 3, 6,
[48]
Chuhang Zou, Jheng-Wee Wei Su. Chi-Han Peng. Alex Colburn, Qi Shan. Peter Wonka, Hung-Kuo Kuo Chu, and
Derek Hoiem. 3d manhattan room layout reconstruction from a single 360 image. 2019. 6, 7, 10
[49]
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference computer vision (ECCV), pages 289-305, 2018.
[50] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5982-5991, 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] proprictary.
The code and the data
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No] It is clear from the problem definition.
(c) Did you discuss any potential negative societal impacts of your work? [No] There is no such issue.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] will do after the publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] please referred the cited papers.
(c) Did you include any new assets either the supplemental material or as URL?? [No]
will do after the publication.
13

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [No] please referred to the cited papers.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] please referred to the cited papers.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "keuaC6KYKmo", "reviewer": "Reviewer_zKQV", "summary": "The paper proposes a framework for finetuning monocular indoor layout estimation models to target domain without ground truth labels. The framework consists of a self-supervise loss using output layouts and uncertainty from multiple views, as well as a metric for evaluating the finetuned model and for hyper-parameter tuning. The paper evaluates against baseline methods both qualitatively and quantitatively in the aforementioned settings, and demonstrates the effectiveness of the proposed scheme.", "questions": "Please see the Weakness section for questions.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[1] Novelty. The task setting and the solution is novel, in that a purely self-supervised loss and a metric for finetuning are proposed, which is shown as both intuitive and effective. \n[2] Evaluation. The evaluation is sound and comprehensive.\n\nWeakness:\n[1] Limited improvements quantitively. On the one hand, limited qualitatively results and comparisons are included in the paper, making it difficult to interpret the effectiveness of the proposed other than the benchmarks. Given the two samples in Fig. 4, the improvement upon baseline methods and pretrained model is limited, where the proposed method shows improvements in some regions but misses out in a few others. The quantitative numbers show improvements, but it is difficult to interpret how the improvement in numbers translate to cross-view consistency (e.g. from BEV). Also limited number of baselines are proposed. I wonder if it is possible to look for or adapt more previous methods to this setting for evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_r-jbIAZCCQ", "reviewer": "Reviewer_z7v1", "summary": "This paper proposes a method to predict room layouts from a set of registered 360 panorama given an arbitrary single-image room layout prediction model trained on a different distribution of data. The main idea is to use reprojected estimates to compute multi-view consistency variance, and then sample from that variance to create pseudo-labels for training. The paper also proposes an entropy metric for evaluating geometric consistency across views. This approach requires no additional ground truth for adapting pre-trained models to new datasets, thus providing an effective way to transfer room layout prediction models to new data.", "questions": "* Although this paper presents the multi-view consistency, pseudo-label approach for the task of room layout estimation, the methodology described appears to be more generally applicable. Could the authors share their insight into whether this approach would be transferrable for other tasks in a multi-view setting? For example, object detection, segmentation, or depth estimation.\n\n* One result stands out as curious. While generally the results in Tables 2 and 3 support the proposed method, it is interesting to observe that the entropy metric, $H_{MLC}$ does not particularly favor the approach proposed in the paper. In fact, those tables report similar or even lower entropy (i.e. better consistency) using the simple pre-trained model or the student-teacher approach of [34]. On the other hand, the IoU and depth metrics show consistent significant improvements with the proposed multi-view, self-training approach. Does this indicate that $H_{MLC}$ is simply not an effective metric for evaluating the quality of the model? If so, it would seem to stand counter to the effectiveness of the hyper-parameter tuning shown in Figure 3.", "limitations": "This paper does not explicitly address any limitations or social impact of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Technical Contribution__\n\nThis paper presents a reasonably straightforward yet very practical solution for fine tuning a model with multi-view constraints without the need for additional data. The weighted boundary consistency loss in Section 3.2 is an effective way to synthesize the multiple predictions from other views, and the visualizations in Figure 1 and Supplementary Figure 1 provide an intuitive illustration of how it performs. Additionally, the entropy metric seems like an elegant answer to the challenging problem of evaluating results without ground truth data.\n\nThis paper provides an in-depth experimental analysis looking at performance improvements on a target dataset when using three different widely-used datasets for training. The quantitative results in Tables 2 and 3 demonstrate clear improvement on standard benchmark metrics compared to the most relevant prior work [34]. Moreover, the paper provides an ablation study that clearly justifies the design decisions of the reprojection loss computation and adds insight to the question of diminishing returns with more images. To the latter question, the results in Table 4(a),(b), and (e) seem to justify the intuition that more reprojection would provide a more stable distribution for pseudo-labels.\n\n\n__Presentation__\n\nThe paper is altogether well written and effectively organized. The writing is clear, and all equations are sufficiently explained. The figures add explanatory value to the paper as well, and the tables are laid out in a clear way that make the conclusions obvious to the reader.\n\n__Summary__\n\nThis is a strong paper that provides a simple but seemingly effective method to adapt pre-trained models to new datasets without extensive data labelling. The positive indicators in the quantitative results and the practicality of the solution suggest that this paper will have a reasonable impact on future research into the problem. There is some question about the usefulness of the proposed entropy metric as an evaluation metric in lieu of labels data, but the other hard evaluation metrics show strong gains, justifying the core methodology. This is a somewhat niche application in computer vision--360 room layout estimation--so the paper might benefit from some discussion about the generality of this approach to other problems. However, the relative size of the research community working on this problem should not hinder this paper's acceptance to NeurIPS. I recommend this paper for acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_IWfpGgbKnC", "reviewer": "Reviewer_eCgj", "summary": "The paper presents a self-supervised approach for target domain adaption specifically for 360 room layout estimation. The idea is to leverage the consensus of the predicted layout from multiple panorama of the same scene. A weighted boundary consistency loss is proposed to learn from pseudo ground truth with uncertainty. A multi-view consistency metric is proved to be strongly correlated with the room layout metric and thus can be used to tune hyper-parameters even with no access to the ground truth.", "questions": "- How is the camera pose on the target dataset obtained? Is it calculated from color images or relies on external sensors?\n- In Tab 4, it seems that disable uncertainty loss, some of the metrics get better. I'd like to hear authors' comments on this.\n- Is the SSLayout360 in Tab 1 and 2 trained only on pre-trained dataset and never sees images from target domain?\n", "limitations": "- It seems that the 6dof camera poses of panoramas are required to do the projection. Hence, precisely speaking, the method is not fully self-supervised but requires camera pose ground truth. This is usually accessible, easier compared to the ground truth layout, but may also cause error for the layout projection and thus hurts the overall finetuning performance.\n\n- The experiment could be stronger to demonstrate the effectiveness of the method from two aspects: 1) a stronger baseline. It seems SSLayout360 is in general outperforming HorizonNet. It would be convincing to show that this method is able to improve powerful backbones. 2) analyze the domain gap. It would be nice to add some discussions about the gap between datasets. Some datasets are closer to each other thus the adaption may not be a big issue. Also, if the method is able to finetune a pre-trained model on synthetic data, then the value of the approach would be much higher.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The method is technically sound. The self-supervised learning is an important and effective solution for domain adaptation.\n- Evaluation on multiple datasets show positive contribution by using the proposed method.\n- Ablation study is provided to show the contribution of individual technical components.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z8Si4MD-Qv", "reviewer": "Reviewer_CFmm", "summary": "The paper proposes a method to transform a model that estimates room layout boundaries from multiple views from a source to a target dataset. This allows finetuning an existing model on a new dataset without ground truth labels. \n", "questions": "* I am wondering what was the reasoning of only comparing against the SSLayout360 method and not against other methods mentioned in the literature review, such as [4, 22]? \n\n* How does this method compare against approaches that have access to large amounts of training data? Is labeling a lot of data worth the effort or not?", "limitations": "I can see that limitations and ethical considerations are not discussed in this paper. Consider adding them.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The idea to finetune a model based on multi-view boundary consistency is interesting, as it allows finetuning a model without additional labeled data on the target dataset. The proposed method only requires calibrated camera views, and the proposed loss is effective, as shown by the experiments.\n* The proposed ablation studies properly motivate the design choices taken for the loss. \n* The paper is very well written and easy to follow. The paper also contains a supplementary document that adds missing details for reproducibility and explanations for the used camera projection model.\n* The qualitative evaluations show that the model finetuned with the proposed loss cleans up the noisy predictions of the pretrained model.\n\nWeaknesses:\nOverall, I do not have many points of criticism with this paper. The paper is well-motivated, the idea is simple yet effective, and the paper has been thoroughly polished. \n* In my opinion, the experiments should show a comparison to a model that uses labels from the target dataset as an upper bound of the achievable performance. This will help put the proposed method into perspective. \n\n* Only comparing against SSLayout360 undermines the claim that the method has been extensively tested against other state-of-the-art methods. Consider adding more baselines, such as [4, 22]. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept