Geometric Exploitation for Indoor Panoramic Semantic Segmentation

Authors: duc  cao
dinh  seok
joon  kim
kyusung  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 08:51:12

Abstract: PAnoramic Semantic Segmentation (PASS) is an important task in computer vision,
as it enables semantic understanding of a 360° environment. Currently,
most of existing works have focused on addressing the distortion issues in 2D
panoramic images without considering spatial properties of indoor scene. This
restricts PASS methods in perceiving contextual attributes to deal with the ambiguity
when working with monocular images. In this paper, we propose a novel
approach for indoor panoramic semantic segmentation. Unlike previous works,
we consider the panoramic image as a composition of segment groups: oversampled
segments, representing planar structures such as floors and ceilings, and
under-sampled segments, representing other scene elements. To optimize each
group, we first enhance over-sampled segments by jointly optimizing with a dense
depth estimation task. Then, we introduce a transformer-based context module
that aggregates different geometric representations of the scene, combined
with a simple high-resolution branch, it serves as a robust hybrid decoder for
estimating under-sampled segments, effectively preserving the resolution of predicted
masks while leveraging various indoor geometric properties. Experimental
results on both real-world (Stanford2D3DS, Matterport3D) and synthetic (Structured3D)
datasets demonstrate the robustness of our framework, by setting new
state-of-the-arts in almost evaluations, The code and updated results are available
at: https://github.com/caodinhduc/vertical_relative_distance.

Main paper: 
 introduction: 
 In recent years, 360°camera images have garnered significant attention from learning systems and practical applications, including holistic sensing in automated vehicles [6, 8, 11, 14, 17, 20] and immersive experiences in augmented reality (AR) and virtual reality (VR) devices [1, 29] . Unlike traditional pinhole cameras with their limited Fields of View (FoV), panoramic images offer an expansive 360°× 180°FoV, providing a more comprehensive perception of both indoor and outdoor environments. This wide FoV enhances numerous fundamental computer vision tasks by enabling a richer understanding of scenes, thus benefiting many fundamental computer vision tasks. One such task, Panoramic Semantic Segmentation (PASS) is a pivotal task that generates dense, pixel-wise class maps, significantly improving high-level understanding of complex environments. By harnessing the wide field of view and unique properties of panoramic images, PASS enables more comprehensive scene analysis, delivering valuable insights across a range of applications.Most current Panoramic Semantic Segmentation (PASS) approaches rely on 2D panoramas transformed through equirectangular projection [24, 33] . However, these methods face two major challenges: limited annotated data and significant image distortions. In terms of data scarcity, existing Figure 1 : Panoramic segmentation problem can be re-formulated to the estimation of over-sampled segments: floor, ceiling and under-sampled segments: chair, table, bookcase, window, etc. datasets are small and lack scene diversity due to the labor-intensive process of manually labeling and verifying segments in each image. For example, the widely-used Stanford 2D3DS dataset [2] contains only 1,413 multi-modal equirectangular images spanning 13 object categories, which is insufficient for robust deep learning model development-posing a substantial challenge for PASS tasks. To address this challenge, aside from unsupervised domain adaptation approaches [19, 35, 38] , which are effective but predominantly applied to outdoor scenes, the development of a large-scale synthetic dataset [37] , featuring detailed 3D structural annotations and photo-realistic 2D renderings of indoor environments, presents a promising solution. As for the second issue, the conversion of panoramic images from spherical to rectangular coordinates results in the oversampling of regions near the poles (floor and ceiling) compared to those near the equator in the original 360°data. This distortion leads to an imbalance in the relative sizes of different classes within each image. Supporting this observation, a quantitative analysis of the Stanford2D3DS dataset shows that, on average, ceilings and floors occupy 18% and 20% of a panoramic image, respectively. Consequently, larger classes such as floors and ceilings are easier to predict, while smaller, more intricate classes like chairs and tables pose greater challenges, as reflected in the disparity of quantitative results between these groups.In this work, to tailor optimization for different regions of panoramic images more effectively, instead of designing a deep learning network which performs to entire image, we divide the equirectangular RGB input into two subgroups of segments. As shown in the Figure 1 , we reformulate the problem of panoramic semantic segmentation as the estimation of over-sampled segments (floor and ceiling) and under-sampled segments (chair, table, window, etc.). We then exploit indoor scene geometric properties to tailor optimization for each group with different strategies as follows: 1) We propose a collaborative study on semantic segmentation of the over-sampled segments and dense depth estimation, this approach enables each task to leverage the benefits of the other, not only through implicit cross-task representation but also by enforcing consistent geometric losses. 2) To fully leverage the rich geometric information in panoramic images, we go beyond the common approaches to introduce a novel concept of vertical relative distance which indicates the relative positions of 3D points with respect to the key components of indoor scenes. These relative distances, combined with image features and other geometric representations are incorporated by a designated transformerbased context module, along with a simple high resolution branch, it can be served as a hybrid decoder optimized for the under-sampled segments estimation.In this paper, we present a novel approach to Indoor Panoramic Semantic Segmentation (PASS). Our key contributions are as follows:• We propose a new method for PASS that decomposes the task into sub-problems and optimizes them by integrating geometric information through distinct strategies. • We introduce the vertical relative distance, a new geometric representation that captures the spatial relationships between planar surfaces (ceilings and floors) and other object pixels in 3D space. • We design a hybrid decoder combining a simple high-resolution branch with a transformerbased context module, which integrates scene representations and exploits relationships among geometric components. • Our framework achieves state-of-the-art performance, demonstrating robustness, accuracy, and efficiency on publicly available panoramic semantic segmentation datasets.To evaluate the effectiveness of our proposed techniques, we benchmark our framework against baseline models and prior methods on three widely-used panoramic semantic segmentation datasets: the real-world Stanford2D3DS, Matterport3D, and the synthetic Structured3D. On the Stanford2D3DS evaluation (fold 1), our method achieves a new state-of-the-art performance with 56.8% mIoU. On the Matterport3D dataset, we surpass previous methods under the same input conditions, reaching 33.06% mIoU on testing set. A similar trend is observed on the Structured3D test set, where our model attains 71.66% mIoU, demonstrating its robustness across diverse scenarios.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: main claims made in the abstract and introduction accurately reflect the paper's contribution, there are three main contributions (check introduction for more detail). Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Panoramic Semantic Segmentation. With recent advances in deep learning, numerous neural network-based methods have emerged for panoramic semantic segmentation. Deng et al. [9] were among the first to convert a pinhole urban traffic scene dataset into wide-angle (fisheye) images, introducing a pioneering framework for panoramic semantic segmentation. Yang et al. [32] later proposed a method for semantic segmentation on 360-degree panoramic annular images, captured with a single panoramic camera, to improve full-field environmental perception. Building on this, they developed DS-PASS [31] , which enhances their earlier work by incorporating attentional mechanisms to improve efficiency in panoramic segmentation.To address distortion in equirectangular images, Tateno et al. [25] introduced distortion-aware convolutions, where the convolutional filter adapts its shape based on the level of distortion in the projected image. Similarly, Zhuang et al. [39] proposed Adaptively Combined Dilated Convolution (ACDNet), which enhances the Field of View near the poles of spherical images by using dilated convolutions as a direct replacement for standard convolutions. Su and Grauman [23] introduced Spherical Convolution, a technique that adjusts kernel sizes dynamically based on spherical coordinates. Coors et al. [7] developed SphereNet, which handles Equirectangular Projection (ERP) by modifying the sampling grid positions of convolution filters to achieve distortion invariance, allowing end-to-end training. Along the same lines, Zhao et al. [36] proposed a distortion-aware CNN for 360-degree spherical images, incorporating both distortion-aware convolutional and pooling layers. Khasanova and Frossard [15] took a different approach by replacing traditional convolutional filters with graphbased filters that adjust dynamically according to the position within the omnidirectional image. Finally, Jiang et al. [13] introduced a novel convolution kernel for CNNs on arbitrary manifolds and topologies, utilizing parameterized differential operators discretized via an unstructured mesh.Sharing a similar vision but employing different approaches, Zhang et al. [34, 35] introduced Trans4PASS and Trans4PASS+, which address spherical distortions and image deformations using Deformable Patch Embedding (DPE) and Deformable Multi-Layer Perception (DMLP) modules. Li et al. [18] proposed a Spherical-Geometry-Aware method to handle distortions when converting 360-degree data into 2D panoramic images. Building on the strengths of DPE and DMLP, we adopt Trans4PASS+ as the baseline model for this paper.Multi-task approach for the Panoramic Semantic Segmentation. A Panoramic Semantic Segmentation network can be trained jointly with other computer vision tasks to leverage additional 3D geometric information from the scene, helping to resolve ambiguities that are challenging for purely 2D approaches. HoHoNet [24] introduced a framework for jointly predicting layout structure and performing dense per-pixel tasks, such as depth estimation and semantic segmentation, based on a 1D horizontal feature representation. Similarly, Berenguel-Baeta et al. [4] proposed a method to jointly perform semantic segmentation and depth estimation from a single equirectangular panorama, utilizing Fourier convolution (FFC) to expand the receptive field and enhance feature extraction. Following a similar intuition, MultiPanoWise [21] extends vision transformers to jointly infer multiple pixel-wise estimation tasks along with signals from intrinsic decomposition.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We mention two limitations on the conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 This section details the architecture of our proposed network, a novel approach for panoramic semantic segmentation from an RGB equirectangular image. As illustrated in the Figure 2 , the network consists of three key modules: (1) an encoder that generates both high-resolution coarse features and low-resolution fine features, (2) a branch dedicated to the concurrent estimation of the over-sampled segments (floor and ceiling) along with dense depth estimation, and (3) a hybrid decoder that integrates a deformable MLP with a novel transformer-based context module to produce semantic masks for the under-sampled segments. The proposed framework consists of three main modules: an encoder for extracting image features, a branch that estimates over-sampled segments alongside dense depth estimation, and a hybrid decoder for estimating under-sampled segments before a merging process to obtain the final segmentation result.Given an input image of size H × W × 3, we adopt the encoder architecture from Trans4PASS+ [35] to generate multi-level feature maps, denoted as f = {f 1 , f 2 , f 3 , f 4 }. These feature maps correspond to resolutions of {1/4, 1/8, 1/16, 1/32} of the original image size, with channel dimensions C = {64, 128, 320, 512}, respectively. Following the approach from the baseline [35] , we apply Deformable Patch Embedding (DPE) modules to the image features at each level before the decoding stage. This transforms the feature hierarchy into embedded feature maps, represented as e = { e 1 , e 2 , e 3 , e 4 }, while maintaining hierarchical resolution, the channel dimensions are unified to C emb . In our network, we set the number of embedding channels to C emb = 128.We design a simple decoder to jointly estimate dense depth and over-sampled segments. To preserve these planar properties, we leverage the Plane Surface Normal Loss (PSN) from Xie et al. [27] to enforces geometric constraints on the predicted depth map by utilizing ground truth floor-ceiling masks. This facilitates accurate plane equation fitting in subsequent processing steps.After obtaining the dense depth for the entire image and the predicted floor-ceiling masks, we introduce several approaches to extract indoor geometric representations. These representations, combined with the global image feature map, form the input for the proposed transformer-based context module, which enhances the scene understanding for the under-sampled segments estimator.Global Image Feature. We consider the embedded feature map e = e 1 , e 2 , e 3 , e 4 to form the global feature of image. To streamline the patch embedding process in the transformer-based context module, we upsample low-resolution embedded feature maps (e 2 , e 3 , e 4 ), then stack them with e 1 along the channel dimension to create the global image feature denoted as F img ( H 4 × W 4 × 4 * c emb ). Cross-Task Feature Distillation. An intuitive way to help the segmentation model understand scene context is to distill representations from the hidden features of other tasks within a multi-task network, as proven by the effectiveness shown in [4, 28, 30] . In this work, we propose distilling information from the hidden feature map F hid ( H 4 × W 4 × C emb ), introduced in Section 3.2. This map not only captures early depth features but also implicitly encodes the estimation of floor-ceiling planes. 3D Point Cloud. Another approach to enhance the segmentation estimator's awareness of 3D context is by first predicting a depth map from the input RGB image and incorporating it as input to the segmentation network. However, we argue that using point cloud data in Cartesian coordinates offers a more robust geometric understanding than depth maps. Point clouds represent 3D data directly, making them better suited for capturing the spatial layout and geometry of a scene. Specifically, given the predicted depth map as 3.2 at resolution of H × W , we apply an Equirectangular to Cartesian projection (detail in appendix) to generate a point cloud set S ∈ R N ×3 , where N = H × W . We then resize and format this point cloud into F pc ( H 4 × W 4 × 3) for further processing. Vertical Relative Distance. Both cross-task feature distillation and 3D point clouds efficiently capture geometric information about the surrounding environment, but neither fully leverages the distinctive characteristics of indoor scenes (e.g., chairs typically rest on the floor, while lights hang from the ceiling). As a key observation, floors and ceilings are prominent features that dominate most indoor panoramic images, acting as critical constraints in these scenes. In 3D coordinates, the floor and ceiling usually form parallel planes, creating the upper and lower boundaries of the space. Thus, the total distance between a point to the floor and ceiling tends to remain constant. These distances not only reflect a point's position relative to the floor-ceiling planes but also describe its spatial relationship to the key constraints derived from the panoramic image. Based on this observation, we introduce a novel concept called vertical relative distance, which utilizes prior information such as point cloud data (S ∈ R N ×3 ) and predicted floor-ceiling masks. This new metric is constructed as follows: We define S f ∈ R N f ×3 and S c ∈ R Nc×3 as the sets of floor and ceiling point clouds, respectively. For the floor set S f , the equation of the floor plane e f in Cartesian coordinates is given by:a f x + b f y + c f z + d f = 0 (1)We apply the least squares method (detail in appendix) to S f to determine the plane coefficients a f , b f , c f , and d f . The distance from each point p(x, y, z) ∈ S to the floor plane e f is then calculated as follows:EQUATION)By applying this process to the entire set S, we obtain d f ∈ R H×W , representing the distances of points in S to the floor plane. Similarly, using this procedure, we compute the distances of points in S to the ceiling plane, denoted as d c ∈ R H×W . The resulting pair of tensors, d f and d c , are stacked and resized to formF dist ∈ R H 4 × W 4 ×2, which encodes the vertical relative distance of the indoor scene (as shown in Figure 3 ). Additionally, we leverage the predicted floor-ceiling maskF m ∈ R H 4 × W 4 ×2, which provides prior positional information, highlighting the precise locations of the constraint components within the panoramic images.While estimating over-sampled segments is relatively straightforward, predicting under-sampled segments poses a greater challenge due to the dense appearance of objects at varying scales and the inherent ambiguities of working with monocular images. Our goal is to design a robust decoder that efficiently leverages image features while capturing the intrinsic relationships between contextual and geometric elements within the indoor scene. To achieve this, we propose a hybrid decoder with the following structure:Transformer-based Context Module. Given a single panoramic image, our objective is to improve the extraction of representations that capture both global image features and diverse geometric structures. As illustrated in the Figure .4, the global image feature F img is combined with the upper branch hidden feature map F hid , 3D point cloud F pc , vertical relative distance F dist , and prior predicted floor-ceiling masks F m . These concatenated features serve as the input to the context module:EQUATIONThe context module is composed of patch and positional embeddings, followed by six stacked transformer encoder layers [26] . Each layer consists of a multi-head self-attention (MHSA) mechanism and a feed-forward network (FFN). MHSA, a key component of the transformer architecture, allows the model to simultaneously attend to information from multiple representation subspaces. In the self-attention module, the input embedding Z is passed through three projection matrices (W Q , W K , W V ) to generate the query (Q), key (K), and value (V ) embeddings.EQUATIONThe output of self-attention is the aggregation of the values that are weighted by the attention weights:EQUATION)Where d is the dimension of query embedding. Multiple self-attention layers are stacked and their concatenated outputs are fused by weighting matrix W h , to form MHSA:EQUATIONAfter exploiting the relationship between global image feature with different geometric information through a sequence of transformer encoder layers, the output tokens can be reshaped into an image-like and upsampled to a size of H 4 × W 4 by strided 3 × 3 transpose convolution for the next processing. Combine with a high-resolution branch. While transformer encoder layers are effective at capturing long-range dependencies, they typically produce lowresolution outputs ( H 16 × W 16 before upscaling in our case), which hinders the decoder's ability to generate high-resolution, dense predictions. To address this limitation, we introduce a high-resolution branch that directly extracts image features c1 by applying a Deformable MLP module, generating features at a resolution of H 4 × W 4 . These features are then fused with the context module output via element-wise summation. Finally, a lightweight head is employed to accurately estimate fine-grained masks for under-sampled segments at a resolution of H 4 × W 4 ×N cls , where N cls represents the number of classes for under-sampled segments, with an additional class for the unknown.Since the over-sampled and under-sampled segments are estimated independently, a straightforward merging mechanism combines them to produce the final segmentation for the entire image. First, the two sets of predicted masks are unified. For overlapping segments, priority is given to floor or ceiling predictions, as they tend to be more accurate and stable. Finally, the merged segments is upscaled to match the original image resolution.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We will make it publicly (weight, code, pipeline) via provided github. Guidelines: Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment settings: 
 Datasets: We utilize three publicly available datasets for comparison: Stanford2D3DS [3] , Struc-tured3D [37] , and Matterport3D [5] . The Stanford2D3DS dataset contains 1,413 real-world panoramic images with annotations for 13 semantic classes, organized into three official folds. We follow the fold-splitting scheme established in previous works [4, 18, 34, 35] . The Structured3D dataset, on the other hand, provides 21,835 synthetic equirectangular images with diverse lighting setups and annotations for 40 semantic classes. Consistent with [12, 37] , we define training, validation, and test splits as follows: scenes 00000-02999 for training, scenes 03000-03249 for validation, and scenes 03250-03499 for testing. For all evaluations, we use raw rendered images under full lighting and furniture configurations. Meanwhile, the Matterport3D dataset introduces an additional challenge with its 10,800 panoramic images capturing 40 semantic classes in complex indoor scenes. Following the processing and split protocol of Guttikonda and Rambach [12] , we create training, validation, and test subsets for consistency in our experiments.Implementation details We train our model on a single NVIDIA GeForce RTX 3090 GPU, starting with an initial learning rate of 5e-5, adjusted using a poly decay strategy with a power of 0.9 over the training epochs. For the Stanford2D3DS, Structured3D, and Matterport3D datasets, we train for 100, 50, and 100 epochs, respectively. The AdamW optimizer [16] is used with an epsilon of 1e-8, a weight decay of 1e-4, and a batch size of 4. Image augmentations include random horizontal flipping, random cropping, and resizing to 512 × 1024. In the testing phase, images are also processed at a resolution of 512 × 1024. Other settings and hyperparameters match those of Tran4PASS+ [35] . For the segmentation and depth estimation tasks, we use Focal and Huber losses, respectively, with the final training loss computed as a combination:EQUATIONHere, L over-sampled-segment and L under-sampled-segment represent the segmentation losses for the over-sampled and under-sampled segments estimation, respectively. In our experiments, the weights α 1 , α 2 , and α 3 are set to [1, 5, 1].Table 1 : Comparison with state-of-the-art methods on the Stanford2D3DS dataset. Consistent with recent work, we report performance as the average mIoU across all three official folds (Avg mIoU) and on fold 1 specifically (F1 mIoU). Our approach demonstrates substantial improvements over both the baseline and existing methods.Method Venue Validation Avg mIoU (%) Validation F1 mIoU (%)Tangent [10] CVPR 2019 45.6 -FreDSNet [4] ICRA 2022 -46.1 PanoFormer [22] ECCV 2022 48.9 -SFSS-MMSI [12] WACV 2024 -52.9 HoHoNet [24] CVPR 2021 52.0 53.9 Trans4PASS [34] CVPR Results on the Stanford2D3DS dataset: Table 1 presents a quantitative comparison of our framework with various panoramic semantic segmentation methods on the Stanford2D3DS validation and test sets. Our method demonstrates significant robustness, achieving a 1.8% and 3.1% mIoU improvement over the baseline Trans4PASS+, respectively, surpassing the current state-of-the-art performance by a small margin. The qualitative results in the Figure 5 further report the advantages of our approach in handling diverse geometric representations. Our model consistently outperforms previous methods, particularly in regions where similar RGB features lead to confusion. For instance, in the first row, the board and wall share similar RGB colors, making it challenging for previous methods to differentiate between them. In contrast, our method accurately distinguishes these classes. Overall, prior methods that estimate all segments using a single network tend to yield results optimized for large segments (such as the floor and ceiling) but struggle with segments near the image equator. In contrast, our proposed learning strategy mitigates the effects of imbalanced distortion across indoor panoramic images, resulting in clearer estimations of smaller segments such as chairs, boards, and tables. Performance of the depth estimation task on the Stanford2D3DS dataset: Since our work integrates depth estimation within the network, we also report the performance of the depth estimation task. We compare our framework to FreDSNet [4] , a multi-task learning model for joint panoramic semantic segmentation and depth estimation. The evaluation metrics include Mean Relative Error (MRE), Mean Absolute Error (MAE), Root-Mean Square Error (RMSE), logarithmic RMSE (RM-SElog), and three relative accuracy measures defined as the fraction of pixels with a relative error within thresholds of 1.25 n for n = 1,2,3 (δ1,δ2,δ3). As result shown in the Table 2 , our network outperforms FreDSNet at all of quantitative metrics, demonstrate the powerfulness of our framework not only on segmentation but also depth estimation.We further conduct the experiment on Structured3D synthetic dataset, which provides a greater variety of images and classes. As the illustration in the Table 3 , on both validation and test sets, our method proves the effectiveness, outperforms previous works to mark a new state of the art performance with 72.86% mIoU and 71.66% mIoU, respectively. Qualitative comparisons in the Figure 6 highlight the robustness of our framework.Results on the Matterport3D dataset: Lastly, we evaluate the proposed approach on the challenging Matterport3D dataset, which features diverse classes within complex indoor scenes. As shown in Table 3 , given only RGB input, our network slightly outperforms previous methods, achieving new peaks with quantitative results of 36.42% mIoU on the validation set and 33.06% mIoU on the test set. These metrics are competitive with methods that utilize additional dense depth (RGBD) input. However, due to the inherent difficulty of the Matterport3D dataset, the results across all methods are generally constrained, as illustrated in the qualitative visualization in Figure 7 . Model performance with and without the involvement of depth. We also conducted ablation studies on the Stanford2D3DS dataset (fold 1) with the baseline( [35] ), our method, and additional two configurations to analyze the role of depth estimation task. In the first setting, we retained the segment partitioning and optimization strategy but removed both the depth branch and the features F pc and F dist from the input of the transformer-based context module. In the second setting, we maintained joint learning for over-sampled segments with depth estimation, but excluded F pc and F dist from the transformer-based context module, detail of the quantitative comparison is shown in the Table 5 .Figure 7 : Qualitative comparison of semantic segmentation results from Trans4PASS+ [35] , SFSS-MMSI [12] , and ours using the Matterport3D dataset. Our method generally performs better than previous approaches (highlighted in white boxes). However, some challenges remain in regions with ambiguous textures (highlighted in black boxes). Zoom in for better view. Model complexity. Table 6 presents the model complexity, comparing the number of parameters and TFLOPs of our approach against previous methods. Trans4PASS+ [35] 0.039 0.131 HoHoNet [24] 0.070 0.125 SFSS-MMSI [12] 0.040 0.079 PanoFormer [22] 0.020 0.081 Ours 0.053 0.135 
conclusion: 
 In this paper, we introduce a novel approach for Indoor Panoramic Semantic Segmentation that decomposes the task into sub-problems, optimizing each by leveraging geometric information in distinct ways. We treat the floor and ceiling as constrained components of the panoramic image and propose vertical relative distance as a new geometric representation of the indoor scene. Additionally, we design a hybrid decoder with a transformer-based context module to aggregate diverse geometric properties effectively. Our framework demonstrates both efficiency and robustness, achieving new state-of-the-art performance on three public datasets. Despite its strong performance, our method has some limitations. First, since the two groups of segments are estimated separately, some unknown areas can emerge during the merging process. Second, the addition of a depth estimation branch increases the model's complexity considerably (as report in the Table 6 ). Besides, the performance on the Matterport3D dataset remains limited. Future work should focus on exploring robust methods that incorporate a deeper understanding of scene structure to further enhance the panoramic semantic segmentation task. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the original papers and public datasets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not release new assets Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide the appendix, including the detailed algorithm explanations from the main paper, color bars for segmentation labels, examples of distance to ceiling, floor masks, and 2D/3D visualizations of our approach. Additional qualitative/quantitative results will be updated on github, please follow the repository at: https://github.com/caodinhduc/vertical_relative_distance.A.1 Projection of equiretangular image to 3D points in Cartesian coordinates using depths. A pixel from equirectangular image can be projected to spherical coordinate, then be converted to Cartesian coordinates (x, y, z) using a depth map estimated as described in Section 3.2. Pixels in equirectangular images can be defined: u, v, where u ranges from 0 to w and v ranges from 0 to h. If we define the horizontal unit angle as ϑ = 2π w and the vertical unit angle as φ = π h , each pixel (u, v) in the ERP can be mapped to the spherical coordinate (θ, ϕ) as shown in Figure 8 :(θ, ϕ) = (u • ϑ, v • φ)Here, ρ is the radius or depth, θ ∈ [0, 2π] is the longitude angle, and ϕ ∈ [0, π] is the latitude angle.To convert from spherical coordinates (θ, ϕ) to Cartesian coordinates (x, y, z) using depth maps, the following formulas are used: x = ρ sin(θ) sin(ϕ), y = ρ cos(ϕ), and z = ρ cos(θ) sin(ϕ).A.2 Least Square Method for plane fitting from point cloud.The least squares method is a statistical technique used to find the best-fitting plane to a set of three-dimensional points (x i , y i , z i ). The equation of a plane in three-dimensional space can be written as:ax + by + c = zwhere a, b, and c are the plane parameters to be determined.To apply the least squares method, we aim to minimize the sum of the squared distances between the observed points and the plane. The objective function to minimize is:S(a, b, c) = n i=1 (ax i + by i + c -z i ) 2where n is the number of data points. These equations can be written in matrix form as:EQUATIONDenoting the matrix on the left as A and the vector on the right as B, we obtain a system of linear equations: Aw = B, where w = a b c.The least squares solution is given by:w = (A T A) -1 A T BThis provides the optimal parameters a, b, and c for the plane equation ax + by + c = z. Assuming the z-axis is aligned with gravity, the plane equation can be initially defined as c = z, where x and y are zero. Here, c represents the average z-coordinate value of the point cloud.In images where the floor or ceiling is not pre-estimated, the plane is defined as a surface parallel to another, separated by a predefined distance.A.3 Color bars for segmentation labels. A.5 2D/3D visualization of our approach step by step. NeurIPS Paper Checklist 


Decision: After the discussion phase all reviewers lean toward acceptance, noting that the manuscript is well written and has novelty. Concerns raised were primarily about the evaluation (e.g., requesting additional ablations, performance on a missing benchmark, study of computational complexity). A detailed rebuttal was submitted that helped to address many of the reviewer concerns, including adding comparisons on Matterport3D and an ablation study related to depth estimation. After the reviewer-author discussion phase, three of the four reviewers raised their rating to borderline accept. As all reviewers are in agreement, and the authors demonstrated commitment to improving the manuscript in the rebuttal phase, the AC reached a decision to accept the paper.  Please take the reviewer feedback into account when preparing the camera ready version.

Reviews: review: paper_summary: paper_summary: The authors decompose the indoor panoramic semantic segmentation task into two sub-tasks: segmentation and depth estimation and design to enhance the geometric information. Specifically, the method first introduces the vertical relative distance to demonstrate the relationships between planar objects (ceiling and floor) and other object pixels in 3D coordinates. Then it aggregates various representations into a transformer-based context module to learn the geometric context. The experimental results partially prove the efficiency of the method proposed by the authors.
strength_weakness: strength_weakness: 1.	The authors claimed in Line 54 that the method is proposed to deal with the various distortions present in panoramic images. However, the proposed method improved the performance mainly by incorporating more representations rather than aiming for distortions, and the distortion problem is not highlighted in the qualitative results.
2.	Although the input of the proposed model is only RGB images, the method utilizes the depth ground truth for the supervision of the depth estimation task, thus the comparisons with the state-of-the-art methods that utilize only RGB in Table 2 are unfair.
3.	The ablation studies without the depth estimation task are needed, for instance, the authors can supplement in Table 4 the experimental results of “F_img+F_h+F_m”.
4.	The qualitative comparisons and analysis without the depth estimation task are suggested to supplement in the main paper.
5.	It is suggested to conduct on more datasets such as Matterport3D to verify the general ability of the proposed method.
6.	Minor: The DPE in Figure 2 is suggested to specify.
questions: questions: Please the weakness section.
limitations: limitations: The authors mentioned the limitations in the conclusion section.
paper_summary: paper_summary: This paper introduces a novel approach to panoramic semantic segmentation. The work views panoramic segmentation from two perspectives including over-sampled segmentation and under-sampled segmentation. The rich geometric depth information is exploited using a transformer-driven context module. The experiments on two public datasets demonstrate the effectiveness of the proposed model.
strength_weakness: strength_weakness: 1. The section 3.5 should be better formalized to help the readers understand the merging process.
2. The proposed framework contains a lot of steps, which could lead to increased running time. It would be nice if the authors could present the computation complexity of different steps in the framework.
3.  Matterport3D is also an important benchmark of panoramic indoor segmentation. It would be nice to evaluate the proposed method on the Matterport3D as well.
4. Regarding the vertical relative distance, would you directly compare the proposed representation against more recent state-of-the-art representations or distance measures to verify the superiority of your proposed method?
5. The proposed transformer-based context module should be compared against existing state-of-the-art decoders and context aggregation methods like ASPP, PSP, SegFormer, UperNet, FPN, Mask2Former, etc.
6. Recent panoramic segmentation methods like MultiPanoWise and DATR could be discussed and compared.
questions: questions: 1. How about the current occlusion-aware seamless segmentation, which is relevant for panoramic scene understanding?
2. Following SFSS-MMSI, would you consider providing more multi-modal ablation results for analysis? For example, when using different representations, how would the performance change?
3. Would you consider visualizing some feature maps or attention maps to help better understand the effectiveness of the proposed method?
4. Is it possible to evaluate on outdoor panoramic segmentation benchmarks? For example, some outdoor panorama datasets could be enriched with geometric information by using large models like depth anything. This could be discussed.
limitations: limitations: This paper has clearly discussed the limitations.
paper_summary: paper_summary: The method divides the indoor panorama semantic segmentation problem into the prediction of over-sampled segmentation (like ceiling, floor, and planar objects) and under-sampled segmentation (like objects in indoor scenery like furniture, windows, door, etc.) subtasks. 

The paper utilizes over-sampled segment prediction with multi-task semantic and depth estimation, to provide spatial relationships of the 3D scene objects concerning the planar constraints in the form of vertical relative distances. The method presents a transformer-based attention mechanism to aggregate the obtained geometric feature representations from the over-sampled segment prediction branch to estimate the under-sampled segments. The prediction from both subtasks is then merged to produce the final panorama semantic segmentation result. Their method produces better results than the current SOTA on the mentioned standard datasets.
strength_weakness: strength_weakness: •	The current method is computationally complex compared to the current methods.
•	The process produces unknown segments due to the merging of the two separate groups of segment predictions.
•	The performance improvement of the proposed heavy capacity model does not seem significant
•	The ablation study of the effect of different utilized geometric representations might require qualitative comparisons highlighting the effect of each module as the performance improvement seems limited 
•	Lack of reproducibility
questions: questions: •	Network details of deformable MLP, segmentation, and depth head module need to be discussed.
•	Detailed loss function equation missing.  
•	Line 125: HoHoNet [22] constructed a framework ‘for of’ layout structure joint per-pixel dense prediction tasks, e.g., depth estimation, and semantic segmentation based on features of 1D horizontal representation seems mistaken.

•	The paper misses mentioning the concept and/or gaps of the current compared methods called tangent [10], SFSS [12] and Panoformer [20] in the related work.
limitations: limitations: Overall, the idea of dividing the segmentation task into planar and object segmentation subtasks while utilizing easily obtained planar geometric representation to provide spatial relationships to help segment challenging objects seems distinctive but seems to show limited performance improvement using the heavy capacity model. Also, the paper misses the mentioned module, and related work details which need to be addressed.
paper_summary: paper_summary: The goal of the paper is panoramic semantic segmentation.  Given a panoramic image, produce a panoptic segmentation.

Main ideas:
1) Separate panoramic image into two sets of segments:
  over-sampled, representing planar objects such as ceiling and floor, and
  under-sampled, representing for other elements
2) Tailor optimization to expected properties of two sets:
  For oversampled:
   joint semantic segmentation and dense depth estimation
  For undersampled:
    add hand-crafted feature representing vertical relative distances.
  For combining:
    Transformer-based Context Module

Specific contributions:
1) separation of ceiling and floor from rest of scene,
2) vertical relative distances features
3) transformer-based context module

These contributions are different from, but conceptually related to, past work as follows:
1) The separation of ceiling and floor seems related in concept to previous work that separates the ground in lidar scans
2) The vertical relative distances features seems related to the HHA geocentric coordinate frame in "Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images" (Gupta et al. 2013).
strength_weakness: strength_weakness: It seems that there is an error in the calculation of the improvement provided by the method in table 1: SGAST4PASS gets 55.5 mIoU, and Trans4PASS++ gets 53.7 -- the difference is 1.8, not 2.8 as reported in the table (bottom row) and text (line 289).   This should be corrected.

It is not clear why the authors chose Trans4PASS++ as the baseline for this study.   There are newer methods that get better results (e.g., SFSS-MMSI, which appears in table 2).   In particular, it seems weird to have a table where one method performs a little bettern in val mIoU that the proposed one, and then list a +5.18 gap versus a midling method next to the result in the bottom row of the table.   Shouldn't the baseline be the current SOTA method?

It would have been nice for the authors to show results on Matterport3D, which has a wide variety of buildings with panoramic images and semantic segmentations (see the SFSS-MMSI paper).   S2D3D has only 13 classes, and all scenes are from similar buildings with low ceilings, and so the benchmark is not as varied and difficult.

The related work in section 2 lists and describes previous papers, but it does not describe their shortcomings or contrast them with the ideas in the submission.  As a result, the authors do not help the reader place the submission in the context of how it addresses an unsolved research problem.   I think it would be much better if the last sentence or two of each paragraph in section 2 lists why the listed methods do not solve the PASS problem, and how those limitations related to the approach taking in this paper.
questions: questions: I understand the motivation for separating the ceiling and floor from the rest of the scene, partly because they are oversampled and partly because they are planar.   Does the same logic apply for walls?   I understand that they will not be as oversampled as the ceiling and floor in a panoramic image, but still they will be oversampled.   Plus, they are planar.   Did you consider separating them too?   If not, why not?   If so, why didn't it work?
limitations: limitations: Limitations are discussed in one sentence in the conclusion.   There is no discussion of handling more complex indoor environments, for example like a hotel lobby, or spa, or church, or ... where ceilings and floors are not so well identified.

Comments: title: Looking for the responses from the reviewers
Appreciate the authors' rebuttal
Difference between vertical relative distance and HHA feature
Comment
comment: Dear reviewer QfCk,

We sincerely thank the reviewer for the discussion. Regarding the concern as the reviewer DACe, we have provided the explanation to address and claim our novelty, it will be more clarified in the revised version.
Dear all reviewers,

We would like to thank all the reviewers for the valuable feedback and recommendations during the discussion phase.

Sincerely
Thanks for the efforts on the experiments on the ablation study of the use of depth and Matterport3D dataset. Nonethless, I have the same concern on the contribution with Reviewer DACe. Therefore, I raise my rating to borderline accept.
Dear Area Chairs and Reviewers,

There are 10 hours left until the end of discussion phase, but we still did not get the the responses from all reviewers even though we solved many concerns in the rebuttal phase, this comment is to express that we really appreciate if the responses is updated, it will make the fair for our paper evaluation.

Sincerely
We thank the reviewer for the appreciation and reconsideration of our work, we agree with the reviewer that the impact of the segment dividing strategy for reducing distortions across segments on each group should be illustrated in the qualitative visualization. Furthermore, the, the gap between indoor PASS (Supervised, Multimodal) and outdoor PASS (UDA) could be discussed in the revised version. Again, we thank the reviewer for the constructive feedback and discussion.
We thank the reviewer for the appreciation and reconsideration of our work, we understand the reviewer's concern as the similar motivation can be seen before, but for lidar scanning. During the rebuttal phase, we also provided some arguments to support why our idea can be seen as novel in indoor PASS, we believe that our reformulated concept for indoor PASS, as well as the distance-based method, will be beneficial to the community, especially the holistic understanding method for indoor scenes. Again, we thank the reviewer for the constructive feedback and discussion.
I appreciate all the work the authors put into the rebuttal.   The results on Matterport3D are particularly appreciated.   I also appreciate the new experiments aimed at teasing apart the effect of depth input/estimation on the results.

I am willing to raise my review to borderline accept (i.e., "not going to argue if others want to accept").  Yet, I do not feel inclined to advocate strongly for acceptance myself because the core method (separating floor and ceiling and handling them separately) does not seem like a big conceptual jump, since people have done things like it for years -- especially separating out the ground from objects in outdoor environments.
Dear reviewer DACe,

We assume that the mentioned concept is **Geocentric Pose: height above the ground** which was presented in the paper "Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images" (Gupta et al. 2013)." 
We would like to provide further explanation to clarify the difference between our proposed concept **vertical relative distance** and the **height above the ground**:

**Similarity:** 

Both "vertical relative distance" and "height above the ground" share the same intuition about measuring the distance between points in image to a constraint component of the images

**Difference:**
"height above the ground" take the 2D depth image as input and consider height above the lowest point in
the 2D image as a surrogate for the height from the supporting ground plane. Since it works with 2D images, in case camera intrinsic is not use, it will sometimes be incorrect in reflecting the relative height between different points due to the ill-posed problem; if camera intrinsic is used, it can be treated as a point-to-point distance rather than a distance to the ground. Meanwhile, our "vertical relative distance" is modeled in 3D coordinates with pre-defined ceiling-floor, which facilitate the concept to reflect the relative distance from 3D points to the pre-defined plane accurately. 

In summary, both "vertical relative distance" and "height above the ground" share the motivation of distance-based concept, but because of different input, pre-defined component, we think that it can be considered as different representation.
The rebuttal helps to solve many of the concerns. The improvement of the proposed model over the baseline is significant. The direct comparison of the Transformer-based Context Module against existing decoders shows the gains of the proposed method.

The reviewer would like to elevate the rating of borderline acceptance. The reviewer suggests that the benefit of the proposed method for tackling distortions could be better illustrated in the results of qualitative visualization. Besides, the gap between indoor PASS (Supervised, Multimodal) and outdoor PASS (UDA) would be worth discussing in the paper.

Sincerely,