Geometric Exploitation for Indoor Panoramic Semantic Segmentation
Authors: Duc Cao Dinh
Seok Joon Kim
Kyusung Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: indoor panoramic semantic segmentation
vertical relative distance
Primary area: machine_vision
TL;DR: We propose a new approach for Indoor Panoramic Semantic Segmentation

Abstract: PAnoramic Semantic Segmentation (PASS) is an important task in computer vision,
as it enables semantic understanding of a 360° environment. Currently,
most of existing works have focused on addressing the distortion issues in 2D
panoramic images without considering spatial properties of indoor scene. This
restricts PASS methods in perceiving contextual attributes to deal with the ambiguity
when working with monocular images. In this paper, we propose a novel
approach for indoor panoramic semantic segmentation. Unlike previous works,
we consider the panoramic image as a composition of segment groups: oversampled
segments, representing planar structures such as floors and ceilings, and
under-sampled segments, representing other scene elements. To optimize each
group, we first enhance over-sampled segments by jointly optimizing with a dense
depth estimation task. Then, we introduce a transformer-based context module
that aggregates different geometric representations of the scene, combined
with a simple high-resolution branch, it serves as a robust hybrid decoder for
estimating under-sampled segments, effectively preserving the resolution of predicted
masks while leveraging various indoor geometric properties. Experimental
results on both real-world (Stanford2D3DS, Matterport3D) and synthetic (Structured3D)
datasets demonstrate the robustness of our framework, by setting new
state-of-the-arts in almost evaluations, The code and updated results are available
at: https://github.com/caodinhduc/vertical_relative_distance.

Geometric Exploitation for Indoor Panoramic Semantic Segmentation
Duc Cao Dinh
Seok Joon Kim Laboratory Department MAXST Seoul, Korea
Kyusung Cho
{caodinhduc, seokjoon, tscho)/maxste com
Abstract
PAnoramic Semantic Segmentation (PASS) is an important task in computer vi- sion, as it enables semantic understanding of 360 environment. Currently, most of existing works have focused on addressing the distortion issues 2D panoramic images without considering spatial properties of indoor scene. This restricts PASS methods in perceiving contextual attributes to deal with the ambi- guity when working with monocular images. In this paper, we propose novel approach for indoor panoramic semantic segmentation. Unlike previous works, we consider the panoramic image as composition of segment groups: over- sampled segments, representing planar structures such as floors and ceilings, and under-sampled segments, representing other scene elements. To optimize each group, we first enhance over-sampled segments by jointly optimizing with a dense depth estimation task. Then, we introduce transformer- based context mod- ule that aggregates different geometric representations of the scene, combined with simple high-resolution branch, serves as robust hybrid decoder for estimating under-samples segments, effectively preserving the resolution of pre- dicted masks while leveraging various indoor geometric properties. Experimental results on both real-world (Stanford2D3DSS Matterportt3D) and synthetic (Struc- tured3D) datasets demonstrate the robustness of our framework, by setting new state-of-the-arts in almost evaluations, The code and updated results available distance,
at: htttt:://gtthtti com http:///woo.nniiin///rrrrrraaaaa
Introduction
In recent years, 360" camera images have garnered significant attention from learning systems and practical applications, including holistic sensing in automated vehicles [6 8 immersive experiences augmented reality (AR) and virtual reality (VR) devices [ 29 Unlike traditional pinhole cameras with their limited Fields of View (FoV), panoramic images offer expansive 360 180 FoV, providing a more comprehensive perception both indoor and outdoor environments. This wide FoV enhances numerous fundamental computer vision tasks by enabling richer understanding of scenes, thus benefiting many fundamental computer vision tasks. One such task, Panoramic Semantic Segmentation (PASS) is pivotal task that generates dense, pixel-wise class maps, significantly improving high-level understanding of complex environments. By harnessing the wide field of view and unique properties of panoramic images, PASS enables more comprehensive scene analysis, delivering valuable insights across a range of applications.
14
17
20 and
Most current Panoramic Semantic Segmentation (PASS) approaches rely on 2D panoramas trans- formed through equirectangular projection [24] 33). However, these methods face two major chal- lenges: limited annotated data and significant image distortions, In terms of data scarcity, existing
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Inpat
Over-sampled segments
Under-sampled segments
Merged sogmentation
Figure 1: Panoramic segmentation problem can be re-formulated to the estimation of over-sampled segments: floor, ceiling and under-sumpled segments: chair, table, bookcase, window, etc.
datasets are small and lack scene diversity due to the labor-intensive process manually labeling and verifying segments in each image. For example, the widely-used Stanford 2D3DS dataset [2] contains only 1,413 multi-modal equirectangular images spanning 13 object categories, which is insufficient for robust deep learning model development-posse address this challenge, aside from unsupervised domain adaptation approaches [9 35 38, which
a substantial challenge for PASS tasks. To
are effective but predominantly applied to outdoor scenes, the development of a large- large-scale synthetic
dataset (37], featuring detailed 3D structural annotations and photo-realiste 2D renderings of indoor environments, presents promising solution. As for the second issue, the conversion of panoramic images from spherical to rectangular coordinates results in the oversampling of regions near the poles (floor and ceiling) compared to those near the equator in the original 360 data. This distortion leads to an imbalance in the relative sizes of different classes within each image. Supporting this observation, quantitative analysis of the Stanford2D3DS dataset shows that, average, ceilings and floors occupy 18% and 20% of a panoramic image, respectively. Consequently, larger classes such as floors and ceilings easier to predict, while smaller, more intricate classes like chairs and tables pose greater challenges, as reflected in the disparity of quantitative results between these groups.
In this work, to tailor optimization for different regions of panoramic images more effectively. instead of designing a deep learning network which performs to entire image, we divide the equirectangular RGB input into two subgroups of segments. As shown in the Figure I we reformulate the problem of panoramic semantic segmentation as the estimation over-sampled segments (floor and ceiling) and under-samplee segments (chair, table, window, etc.). We then exploit indoor scene geometric properties to tailor optimization for each group with different strategies as follows: 1) We propose collaborative study on semantic segmentation of the over-sampled segments and dense depth estimation, this approach enables each task to leverage the benefits of the other, not only through implicit cross-task representation but also by enforcing consistent geometric losses. 2) To fully leverage the rich geometric information panoramic images, we go beyond the common approaches to introduce novel concept of vertical relative distance which indicates the relative positions of 3D points with respect to the key components of indoor scenes. These relative distances, combined with image features and other geometric representations based context module, along with a simple high resolution branch, it can be served as a hybrid decoder optimized for the under-sanee sampled segments estimation.
incorporated by designated transformer-
In this paper, we present a novel approach to Indoor Panoramic Semantic Segmentation (PASS). Our key contributions are follows:
We propose new method for PASS that decomposes the task into sub-problems and optimizes them by integrating geometric information through distinct strategies.
We introduce the vertical relative distance, a new geometric representation that captures the spatial relationships between planar surfaces (ceilings and floors) and other object pixels in 3D space.
We design hybrid decoder combining simple high-resolution branch with transformer- based context module, which integrates scene representations and exploits relationships among geometric components.
Our framework achieves state-of-the-are performance, demonstrating robustness, accuracy, and efficiency on publicly available panoramic semantic segmentation datasets.
To evaluate the effectiveness of our proposed techniques, we benchmark our framework against base- line models and prior methods on three widely-used panoramic semantic segmentation datasets: the real-world Stanford2D3DS. Matterport3DD and the synthetic Structured3D. On the Stanford2D3DDS evaluation (fold 1), our method achieves state-of-ther thrt--rrttt performance with 56.8% mloU. On

---

the Matterport3D dataset, we surpass previous methods under the same input conditions, reaching 33.06% mloU on testing set. A similar trend is observed on the Structured3D test set, where our model attains 71 71.66% mloU, demonstrating its robustness across diverse scenarios.
2 Related Work
Panoramic Semantic Segmentation. With recent advances in deep learning, numerous neural network-based methods have emerged for panoramic semantic segmentation. Deng et al. [ were among the first to convert pinhole urban traffic scene dataset into wide-angle (fisheye) images, introducing a pioneering framework for panoramic semantic segmentation. Yang et al. [32 later proposed a method for semantic segmentation on 360-degree panoramic annular images, captured with a single panoramic camera, to improve full-field environmental perception. Building on this, they developed DS-PASS [31,, which enhances their earlier work by incorporating attentional mechanisms to improve efficiency in panoramic segmentation.
To address distortion in equirectangular images, Tateno et al. [25] introduced distortion-awar convo- lutions, where the convolutional filter adapts its shape based on the level of distortion in the projected image. Similarly, Zhuang al. [39] proposed Adaptively Combined Dilated Convolution (ACDNet), which enhances the Field of View near the poles of spherical images by using dilated convolutions a direct replacement for standard convolutions. Su and Grauman [23] introduced Spherical Convolution, a technique that adjusts kernel sizes dynamically based on spherical coordinates. Coors al. 7 developed SehereNet, which handles Equirectangular Projection (ERP) by modifying the sampling grid positions of convolution filters to achieve distortion invariance, allowing end-to-end training. Along the same lines, Zhao al. 36 proposed distortion-aware CNN for 360-degree spherical images, incorporating both distortion-aware convolutional and pooling layers. Khasanova and Frossard (15) took a different approach by replacing traditional convolutional filters with graph- based filters that adjust dynamically according to the position within the omnidirectional image. Finally, Jiang et al. [3 introduced novel convolution kernel for CNNs on arbitrary manifolds and topologies, utilizing parameterized differential operators discretized via an unstructured mesh.
Sharing similar vision but employing different approaches, Zhang et al. [34 35 introduced Trans4PASS and Trans4PASS+, which address spherical distortions and image deformations using Deformable Patch Embedding (DPE) and Deformable Multi-Layer Perception (DMLP) modules. Li al. 18] proposed a Spherical- Geometry-Aware method to handle distortions when converting 360-degree data into 2D panoramic images. Building on the strengths of DPE and DMLP, we adopt Trans4PASS+ as the baseline model for this paper.
Multi-task approach for the Panoramic Semantic Segmentation. A Panoramic Semantic Seg- mentation network can be trained jointly with other computer vision tasks to leverage additional 3D geometrie information from the scene. helping to resolve ambiguities that are challenging for purely 2D approaches. HoHoNet (24) introduced a framework for jointly predicting layout structure and performing dense per-pixel tasks, such as depth estimation and semantic segmentation, based ID horizontal feature representation. Similarly, Berenguel- -Beta et al. 4 proposed a method to jointly perform semantic segmentation and depth estimation from single equirectangular panorama, utilizing Fourier convolution (FFC) to expand the receptive field and enhance feature extraction. Following similar intuition, MultiPano Wise (21) extends vision transformers jointly infer multiple pixel-wise estimation tasks along with signals from intrinsic decomposition.
Method
This section details the architecture of our proposed network, a novel approach for panoramic semantic segmentation from an RGB equirectangular image. As illustrated in the Figure 2 the network consists of three key modules: (1) an encoder that generates both high-resolution coarse features and low-resolution fine features, (2) a branch dedicated to the concurrent estimation of the over- sumpled segments (floor and ceiling) along with dense depth estimation, and (3) hybrid decoder that integrates deformable MI.P with novel transformer-bused context module to produce semantic masks for the under-sampled segments.
3

---

Input image
Final segmentation
Fau
F
Dacoder
Opensampled
Point cloud
Pp
Giobal image
Feu Hidden feature map
cloud
Depth
Under-sampled segments
Verticll relative distances
Ceiling- floor predicted marks
Transformer Context Module
Masks merging
Seg ilead
Distance to ceiling
Distance floor
High
Figure 2: The proposed framework consists three main modules: an encoder for extracting image features, branch that estimates over-sampled segments alongside dense depth estimation, and hybrid decoder for estimating under- sampled segments before a merging process to obtain the final segmentation result.
3.1 Encoder
Given an input image of size H W 3, we adopt the encoder architecture from Trans4PASS+ 35
to generate multi-level feature maps, denoted as (1.L2.fa:L4)- These feature maps corre- spond to resolutions of 11/4,1/8, 1/16, 1/32] of the original image size, with channel dimensions C' [64, 128, 512), respectively. Following the approach from the baseline [55] we apply Deformable Patch Embedding (DPE) modules to the image features at each level before the de- coding stage. This transforms the feature hierarchy into embedded feature maps, represented as CI, é3, ). while maintaining hierarchical resolution, the channel dimensions are unified to Cemb- In our network,
the number of embedding channels to Cemb 128.
3.2 Over-sampled segments estimation
We design simple decoder to jointly estimate dense depth and over-sampled segments. Given the C1 d22 43, 64, we apply deformable MILPs at each level and upscale embedded feature maps W. These upscaled features are then fused using point-wise summation to spatial resolution to T 4.
obtain a hidden feature map (Fhia) of size H. W
4 Cemb.
Next, lightweight prediction heads process the hidden features estimate both dense depth ( T x 1) and over-sampled segments (# x T x Nes),, where Net 3, corresponding to floor, ceiling, and unknown, represents the union of all other classes. Unlike regular objects, ceilings and floors are typically planar surfaces in 3D environments.
To preserve these planar properties, we leverage the Plane Surface Normal Loss (PSN) from Xie al. (27) enforces geometric constraints on the predicted depth map by utilizing ground truth floor-ceiling masks. This facilitates accurate plane equation fitting in subsequent processing steps.
3.3 Contextual Information Exploitation
After obtaining the dense depth for the entire image and the predicted floor-ceiling masks, we introduce several approaches to extract indoor geometric representations. These representations, combined with the global image feature map, form the input for the proposed transformer- context module, which enhances the scene understanding for the under-sampled segments estimator.
Global Image Feature. We consider the embedded feature map e = C1, 62, C3, 64 to form the global feature of image. To streamline the patch embedding process in the transformer-but -basedd context module, we upsample low-resolution embedded feature maps (e2, C3 C4), then stack them with e along the channel dimension create the global image feature denoted Fing
4 Cemb).
Cross- Task Feature Distillation. intuitive way to help the segmentation model understand scene context is distill representations from the hidden features of other tasks within multi-task network.

---

as proven by the effectiveness shown in 14] 28 30 In this work, we propose distilling information from the hidden feature map Fiid (
Cemb). introduced in Section 3.2 This map not only captures early depth features but also implicitly encodes the estimation of floor-ceiling planes.
3D Point Cloud. Another approach enhance the segmentation estimator's awareness of 3D context is by first predicting a depth map from the input RGB image and incorporating it as input to the segmentation network. However, we argue that using point cloud data in Cartesian coordinates offers a more robust geometric understanding than depth maps. Point clouds represent 3D data directly, making them better suited for capturing the spatial layout and geometry a seene. Specifically, given the predicted depth map as 3.2 at resolution of H X W, we apply an Equirectangular to Cartesian projection (detail in appendix) generate point cloud set S IR Nx3, where N H W. We W
then resize and format this point cloud into Fp (4 x T x 3) for further processing.
Vertical Relative Distance. Both cross-task feature distillation and 3D point clouds efficiently capture geometric information about the surrounding environment, but neither fully leverages the distinctive characteristics of indoor scenes (e.g.. chairs typically rest on the floor, while lights hang from the ceiling). As key observation, floors and ceilings are prominent features that dominate most indoor panoramic images, acting as critical constraints in these scenes. In 3D coordinates, the floor and ceiling usually form parallel planes, creating the upper and lower boundaries of the space. Thus, the total distance between point the floor and ceiling tends to remain constant.
These distances not only reflect a point's position relative to the floor-ceiling planes but also describe its spatial relationship to the key constraints derived from the panoramic image. Based this observation, we introduce a novel concept called vertical relative distance, which utilizes prior information such as point cloud data (S IR Nx3) and predicted floor-ceiling masks. This new metric is constructed as follows:
Image
cciling-floor masks
dense depth
di distance to ceiling
distance to floor
Figure 3: Visualization of an example showing distance to floor and distance to ceiling masks. The gradient from light to dark represents the transition from greater to shorter distances.
We define S E RgNyxx and S. E R-Nxx3 as the of floor and ceiling point clouds, respectively. For the floor set Sy, the equation of the floor plane e in Cartesian coordinates is given by:
0 + byy + cy + dy =
(I)
We apply the least squares method (detail in appendix) to Sy to determine the plane coefficients as, by, Cf. and dy. The distance from each point p(x, y, E S to the floor plane e is then calculated as follows:
d(p es) lafx + byy + cjz dyl
|) 63 c
By applying this process to the entire set S, we obtain d, E #x representing the distances of points in S to the floor plane. Similarly, using this procedure, we compute the distances of points S the ceiling plane, denoted as d, RHxW The resulting pair of tensors, d and de, are stacked and resized to form Faist E R2 x 16 x2 which encodes the vertical relative distance of the indoor scene (as shown in Figure 3) Additionally, we leverage the predicted floor-ceiling mask Fm E R x # x2 which provides prior positional information, highlighting the precise locations of the constraint components within the panoramic images.
3.4 Under-sampled segments estimation
While estimating over-sampled segments is relatively straightforward, predicting under-sampled segments poses greater challenge due to the dense appearance of objects at varying scales and the inherent ambiguities of working with monocular images. Our goal design robust decoder that efficiently leverages image features while capturing the intrinsic relationships between contextual
5

---

and geometric elements within the indoor scene. To achieve this, we propose a hybrid decoder with the following structure:
brassodmeer-bsedd Context Module. Given a single panoramic image, our objective is to improve the extraction of representations that capture both global image features and diverse geometric structures. As illustrated in the Figure 4 the global image feature Fimg is combined with the upper branch hidden feature map l'hid. 3D point cloud F'pe. vertical relative distance Faist, and prior predicted floor-ceiling masks Fm. These concatenated features serve as the input to the context module:
Z = [Fimg: Fhidd Fpc: Faist; Fmn
The context module is composed of patch and positional embeddings, followed by six stacked trans- former encoder layers [26]. Each layer consists of a multi-head self-attention (MHSA) mechanism and feed-forward network (FFN), MHSA, a key component of the transformer architecture, allows the model to simultaneously attend to information from multiple representation subspaces. In the self-attention module, the input embedding Z is passed through three projection matrices (W², W WV) to generate the query (Q), key (K), and value (V) embeddings.
Q = Z K ZW V= ZWV
The output of self-attention is the aggregation of the values that are weighted by the attention weights:
SA(Q, V) softmar QKT
)
(5)
Vd
Where d is the dimension of query embedding. Multiple self-attention layers stacked and their concatenated outputs are fused by weighting matrix Wh, to form MHSA:
H
(HSA(Q.K.)) = Σ M.
(6)
After exploiting the relationship between global image feature with different geometric information through sequence of transformer encoder layers, the output tokens can be reshaped into an image-like
and upsampled to a size of H W by strided 3 x 3 transpose convolution for the next processing.
I 4
Combine with high-resolution branch. While transformer encoder layers are effective at capturing rangrange dependencies, they typically produce low- resolution outputs 11 x To before upscaling in our case), which hinders the decoder's ability to generate high-resolution, dense predictions. To address this limitation, we introduce a high-resolution branch that directly extracts image features cl by applying a De- formable MLP module, generating features lution of H x W These features are then fused with the context module output via element- wise summa- tion. Finally, a lightweight head is employed to accu- rately estimate fine-grained masks for conde--saplle segments at a resolution of I! x 4 x Nets, where Nets represents the number of classes for under-sampled segments, with an additional class for the unknown.
Transformer Encoder Layer
W.
Feed-Forward Network
Transformer Encoder Layer
Norm
reso-
Transformer Encoder Layer
Multi-Head Attention Q+ K V
Patch Position Embedding
W
Norm
F Fing Friit F Fpe F Fdist
3.5 Merged segments estimation
Figure 4: Transformer based context module
Since the over-sampled and under-sampled segments are estimated independently, a straightforward merging mechanism combines them to produce the final segmentation for the entire image. First, the two sets of predicted masks are unified. For overlapping segments, priority is given to floor or ceiling predictions, as they tend to be more accurate and stable. Finally, the merged segments is upscaled to match the original image resolution.

---

Experiments
4.1 Experiment Settings
Datasets: We utilize three publicly available datasets for comparison: Stanford2D33DS (3. Struc- tured3D (37]. and Matterport3D [5]. The Stanford2D3DS dataset contains 1,413 real-world panoramic images with annotations for 13 semantic classes, organized into three official folds. We follow the fold-splitting scheme established in previous works 4 18 34 33]. The Structured3D dataset, on the other hand, provides 21,835 synthetic equirectangulat lar images with diverse lighting se- tups and annotations for 40 semantic classes. Consistent with [12 37,, we define training, validation, and test splits as follows: scenes 00000-000000000000 -2000002999900 for training, scenes 03000 000049 for validation, and scenes 03250-03499 for testing. For all evaluations, we use raw rendered images under full lighting and furniture configurations. Meanwhile, the Matterport3D dataset introduces an additional challenge with its 10,800 panoramic images capturing 40 semantic classes in complex indoor scenes, Following the processing and split protocol of Guttikonda and Rambach [12]. create training, validation, and test subsets for consistency in our experiments.
Implementation details We train our model on a single NVIDIA GeForce RTX 3090 GPU, starting with an initial learning rate of 5e-5, adjusted using poly decay strategy with power of 0.9 over the training epochs. For the Stanford2D3DS, Structured3D, and Matterport3D datasets, we train for 100, 50, and 100 epochs, respectively. The AdamW optimizer
[16] is used with an epsilon of 1e-8, weight decay of le-4, and batch size of 4. Image augmentations include random horizontal flipping, random cropping, and resizing to 512 1024. In the testing phase, images are also processed at resolution of 512 x 1024. Other settings and hyperparameters match those of Tran4PASS+ [35]. For the segmentation and depth estimation tasks, we use Focal and Huber losses, respectively, with the combination:
final training loss computed
Lioal = 01 a1 Lover- -ove---ors-- swwpted- segment + (un-Lunder-sanan samplee--semeeeeeeeeeeeeee + Org. deephh
Here, Lover sumpled- segment and somple-- segment represent the segmentation losses for the over-sampled and under-sampled segments estimation, respectively. In our experiments, the weights 01, (22, and 0g are 51 1].
4.2 Experiment Results
Table 1: Comparison with state-of-the-arr methods on the Stanford2D3DS dataset. Consistent with recent work, report performance as the average mloU across all three official folds (Avg mloU) and on fold specifically (Fl mloU). Our approach demonstrates substantial improvements over both the baseline and existing methods.
Method
Venue
Validation Avg mloU (%)
Validation Fl mloU (%)
Tangent
10
CVPR 2019
45.6
FreDSNet
4
ICRA 2022
46.1
PanoFormer
22
ECCV 2022
48.9
SFSS-MMSI
12
WACV 2024
52.9
HoHoNet
24
CVPR 2021
52.0
53,9
Trans4PASS
34
CVPR 2022
52.1
53.3
Trans4PASS+
35
Arxiv 2022
53.7
53.6
SGAT4PASS
18
IJCA 2023
55.3
56.4
Ours
55.5
56.8
Results on the Stanford2D3DS dataset: Table presents quantitative comparison of our framework with various panoramic semantic segmentation methods on the Stanford2D3DS validation and test sets. Our method demonstrates significant robustness, achieving 1.8% and 3. 1% mloU improvement over the baseline Trans4PASS+, respectively, surpassing the current tfte-off-the-rr performance by small margin. The qualitative results in the Figure further report the advantages of our approach in handling diverse geometric representations. Our model consistently outperforms previous methods, particularly in regions where similar RGB features lead to confusion. For instance, in the first row, the board and wall share similar RGB colors, making it challenging for previous methods to differentiate between them. In contrast, our method accurately distinguishes these classes. Overall, prior methods
7

---

U
Image
TrassPASSS
SGAT4PASS
Ours
Ground Truth
Figure 5: Qualitative comparison of semantic segmentation results from Trans4PASS+ (35) SGAT4PASS [18) and ours using the Stanford2D3D dataset. Black boxes highlight the improve- ments. Zoom for the better view.
that estimate all segments using a single network tend yield results optimized for large segments (such as the floor and ceiling) but struggle with segments near the image equator. In contrast, our proposed learning strategy mitigates the effects of imbalanced distortion across indoor panoramic images, resulting in clearer estimations of smaller segments such as chairs, boards, and tables.
Table 2: Quantitative comparison of depth estimation task.
Method
MREG
MAE]
RMSE
RMSElogl
0: 1
82 f
FreDSNet
35
0.095
0.133
0.518
0.208
0.843
0.958
0.986
Ours
0.074
0.120
0.390
0.760
0.865
0.988
0.991
Performance of the depth estimation task on the Stanford2D3DS dataset: Since our work inte- grates depth estimation within the network, we also report the performance of the depth estimation task. We compare framework to FreDSNet 4 multi-task learning model for joint panoramic semantic segmentation and depth estimation. The evaluation metrics include Mcan Relative Error (MRE), Mean Absolute Error (MAE), Root-Mean Square Error (RMSE), logarithmic RMSE (RM- SElog), and three relative accuracy measures defined as the fraction of pixels with a relative error within thresholds of 1.25" for 1,2.3 (61,52,53). As result shown in the Table 2 network outperforms FreDSNet at all of quantitative metrics, demonstrate the powerfulness of our framework not only segmentation but also depth estimation.
Results on the Structured3D dataset: dataset, which provides a greater variety of images and classes. As the illustration in the Table 3 on both validation and test sets, our method proves the effecctiveness, outperforms previous works to mark new state of the art performance with 72.86% mloU and 71.66% mloU, respectively. Qualitative comparisons in the Figure 6 highhight the robustness of our framework.
We further conduct the experiment on Structured3D synthetic
Results on the Matterport3D dataset: Lastly, we evaluate the proposed approach on the challenging Matterport3D dataset. which features diverse classes within complex indoor scenes. As shown in Ta given only RGB input, our network slightly outperforms previous methods, achieving new peaks with quantitative results of 36.42% mloU on validation set and 33.06% mloU on the test set. These metrics are competitive with methods that utilize additional dense depth (RGBD) input. However, due to the inherent difficulty of the Matterport3D dataset, the results across all methods are generally constrained, as illustrated in the qualitative visualization in Figure] 7

---

Image
Trans4PASSt
SFSS-MMSI
Ours
Ground Troth
Figure 6: Qualitative comparison of semantic segmentation results from Trans4PASS+ 35), SFSS-
MMSI [1]. and ours using the Structured3D dataset. Black boxes highlight the improvements.
Table Comparison with the SOTA methods on the Structured3D and the Matterport3D validation and test sets. Our method marks new state of the arts on both datasets given the same input.
Structured3D
Matterport3D
Methods
Modal
Val mloU (%)
Test mloU (%)
Val mloU (%)
Test mloU (%)
PanoFormer
22
RGB
55.57
54.87
30.04
26.87
Trans4PASS+
35
RGB
66.74
66.90
33.43
29.21
SFSS-MMSI
12
RGB
71.94
68.34
35.15
31.30
PanoFormer
22
RGBD
60.98
$9.27
33.99
31.23
SFSS-MMSI
12
RGBD
73.78
70.17
39.19
35.92
Ours
RGB
72.86
71.66
36.42
33.06
4.3
Ablation study
Impact of each contextual representation. We conduct the ablation study on the Stanford2D3DS dataset [3] (fold 1) measure the influence different geometric representation the performance of network. We consider a baseline setting with global image feature is input of transformer-basee context module. After that, model train with additional geometric properties as input of the transformer-bot based context module, detail of the performance is reported in the Table
Table 4: Impact of each geometric representation to the model performance.
Geometric properties
mloU (%)
Pixel Ace (%)
Fing Fing Fhid
54.41
82,06
54.83
82.46
Fing Fhid + Fpc
55.32
82.87
Fing Fiid + Epe + Faist
56.68
83.32
Fimg + Fisd + Ege + Faist + Fm
56.80
83.45
Model performance with and without the involvement of depth. We also conducted ablation studies on the Stanford2D3DS dataset (fold 1) with the baseline( (35)). our method, and additional two configurations to analyze the role of depth estimation task. In the first setting, we retained the segment partitioning and optimization strategy but removed both the depth branch and the features Fpe and Faist from the input of the trunsformer-base maintained joint learning for over-sampled segments with depth estimation, but excluded pe and Faist from the transformer-basel context module, detail of the quantitative comparison is shown in the Table
context module. the second setting, we

---

Image
Trans4PASS
SFSS-MMSI
Ours
Ground Truth
Figure 7: Qualitative comparison of semantic segmentation results from Trans4PASS+ B5 SFSS- MMSI [22] and ours using the Matterport3D dataset. Our method generally performs better than previous approaches (highlighted in white boxes). However, some challenges remain in regions with ambiguous textures (highlighted in black boxes). Zoom in for better view.
Table 5: Impact of depth estimation involvement the model performance.
Settings
Avg mloU (%)
Fold mloU (%)
Baseline
53.7
53.6
Without depth involvement (depth estimation, both Fyee and Faca))
54.3
54.7
With depth estimation but no Fpe and Faist
54.6
55.0
With full depth involvement
55.5
56.8
Model complexity. presents the model complexity, comparing the number of parameters and TFLOPs of our approach against previous methods.
Table 6: Computational complexity comparison with input size: 512 x 1024 3.
Input
Params(G)
TFLOPs
Trans4PASS+ 35
0.039
0.131
HoHoNet 24
0.070
0.125
SFSS-MMSI 12
0.040
0.079
PanoFormer
0.020
0.081
Ours
0.053
0.135
5
Conclusion
In this paper, we introduce a novel approach for Indoor Panoramic Semantic Segmentation that decomposes the task into sub-problems, optimizing each by leveraging geometric information in distinct ways. We treat the floor and ceiling as constrained components of the panoramic image and propose vertical relative distance new geometrie representation of the indoor scene. Additionally, we design hybrid decoder with transformer-bat based context module to aggregate diverse geometric properties effectively. Our framework demonstrates both efficiency and robustness, achieving new state-of-the ffate-of-the-ree of-the-art art performance on three public datasets. Despite its strong performance, our method has some limitations. First, since the two groups of segments are estimated separately, some wnknown areas can emerge during the merging process. Second, the addition of a depth estimation branch increases the model's complexity considerably (as report in the Table Besides, the performance on the Matterport3D dataset remains limited. Future work should focus exploring robust methods that incorporate a deeper understanding of scene structure further enhance the panoramic semantic segmentation task.
10

---

Acknowledgments and Disclosure of Funding
We thank MAXST Co. Ltd for their support in providing time and infrastructure for this project.
References
[1] Hao Ai, Zidong Cao, Jin Zhu, Haotian Bai, Yucheng Chen, and Ling Wang. Deep learning for omnidirectional vision: survey and new perspectives. ArXiv, abs/2205.10468, 2022.
[2] Armeni, A. Sax, A. R. Zamir, and Savarese. Joint 2D-3D-Semantic Data for Indoor Scene Understanding. ArXiv e-prints, 2017.
[3] Iro Armeni, Sasha Sax, Amir Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. ArXiv, abs/1 105, 2017.
[4] Bruno Berenguel-Bacte Jesus Crrmudez-Cameo and Josechu J. Guerrero. Fredsnet: Joint monocular depth and semantic segmentation with fast fourier convolutions from single panora- mas. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 6080- 6086, 2022.
[5] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[6] Yanzhi Chen, Xi Li, Anthony Dick, and Rhys Hill. Ranking consistency for image matching and object retrieval. Pattern Recognition, 47(3):1349-133 1360, 2014.
[7] Benjamin Coors, Alexandru Condurache, and Andreas Geiger. Spherenet: Learning spheri- cal representations for detection and classification in omnidirectional images. In European Conference on Computer Vision, 2018
[8] Greire Payen de La Garanderie, Amir Atapour Abarghouei, and Toby Breckon. Eliminating the blind spot: Adapting 3d object detection and monocular depth estimation to 360 panoramic imagery. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[9] Liuyuann Deng, Ming Yang, Yeqiang Qian, Chunxiang Wang, and Bing Wang. Cnn based semantic segmentation for urban traffic scenes using fisheye camera. 2017 IEEE Intelligent Vehicles Symposium (IV), pages 231-236, 2017.
[10] Mare Eder, Mykhailo Shvets, John Lim, and Jan-Michael Frahm. Tangent images for mitigating spherical distortion. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1222311431, 2019.
[11] Shaohua Gao, Kailun Yang, Haowen Shi, Kaiwei Wang, and Jian Bai. Review on panoramic imaging and its applications in scene understanding. IEEE Transactions on Instrumentation and Measurement, 71:1-34, 2022.
[12] Suresh Guttikonda and Jason Raphael Rambach. Single frame semantic segmentation using multi-modal spherical images. 2024 JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3210-3219, 2023.
[13] Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Niessner. Spherical CNNs on unstructured grids. In International Conference on Learning Representations, 2019.
[14] Xiaoheng Jiang. Li Zhang, Pei Lv, Yibo Guo, Ruijie Zhu, Yafei Li, Yanwei Pang. Xi Li, Bing Zhou, and Mingliang Xu. Learning multi-level density maps for crowd counting. IEEE Transactions Neural Networks and Learning Systems, 31:2705-22 2715, 2020.
[15] Renata Khasanova and Pascal Frossard. Geometry aware convolutional filters for omnidirec- tional images representation. In International Conference on Machine Learning, 2019.
[16] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference Learning Representations (ICLR), San Diega, CA, USA, 2015.

---

[17] Xi Li, Anthony R. Dick, Hanzi Wang, Chunhua Shen, and Anton van den Hengel. Graph mode-based contextual kernels for robust svm tracking. 2011 International Conference on Computer Vision, pages 1 1156-1 1633, 2011.
[18] Xuewei Li, Tao Wu, Zhongang Qi, Gaoang Wang, Ying Shan, and Xi Li. Sgat4pass: Spherical geometry-aware transformer for panoramic semantic segmentation. In International Joint Conference on Artificial Intelligence, 2023.
[19] Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2502-2511, 2018.
[20] Chaoxiang Ma, Jiaming Zhang, Kailun Yang, Alina Roitberg, and Rainer Stiefelhagen. Densep- ass: Dense panoramic semantic segmentation via unsupervised domain adaptation with attention- augmented context exchange. 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2766-2776777 2772, 2021.
[21] Uzair Shah, Muhammad Tukur, Mahmood Alzubaidi, Giovanni Pintore, Enrico Gobbetti, Mowafa Househ, Jens Schneider, and Marco Agus. Multipanowise: Holistic deep architecture for multi-task dense prediction from single panoramic image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 1311-1321, 2024.
[22] Zhijie Shen, Chunyu Lin, Kang Liao, Lang Nie, Zishuo Zheng, and Yao Zhao. Panoformer: Panorama transformer for indoor 360 depth estimation. In European Conference on Computer Vision, 2022.
[23] Yu-Chuan Su and Kristen Grauman. Learning spherical convolution for fast features from 360 imagery. In Neural Information Processing Systems, 2017.
[24] Cheng Sun, Min Sun, and SwaannTzzog Chen. Hohonet: 360 indoor holistic understanding with latent horizontal features. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2573-2583 2020.
[25] Keisuke Tateno, Nassir Navab, and Federico Tombari. Distortion-awant convolutional filters for dense prediction in panoramic images. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[26] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
[27] Yaxu Xie, Fangwen Shu, Jason Raphael Rambach, Alain Pagani, and Didier Stricker. Planerec- net: Multi-task learning with cross-task consistency for picce-wise plane detection and recon- struction from single rgb image. In British Machine Vision Conference, 2021.
[28] Dan Xu, Wanli Ouyang, Xiaogang Wang, and N. Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 675-684, 2018.
[29] Mai Xu, Yuhang Song, Jianyi Wang, Minglang Qiao, Liangyu Huo, and Zulin Wang. Pre- dicting head movement in panoramic video: A deep reinforcement learning approach. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:2693- 2708, 2017.
[30] Xiaogang Xu, Hengshuang Zhao, Vibhav Vineet, Ser Nam Lim, and Antonio Torralba. Mtformer: Multi-task learning via transformer and cross-task reasoning. In European Conference on Computer Vision. 2022.
[31] Kailun Yang, Xinxin Hu, Hao Chen, Kaite Xiang, Kaiwei Wang, and Rainer Stiefelhagen. Ds-pass: Detail-sensitive panoramic annular semantic segmentation through swaftnet for sur- rounding sensing. 2020 IEEE Intelligent Vehicles Symposium (IV), pages 457-464, 2019.
12

---

[32] Kailun Yang, Xinxin Hu, Luis Miguel Bergasa, Eduardo Romera, and Kaiwei Wang. Pass: Panoramic annular semantic segmentation. IEEE Transactions on Intelligent Transportation Systems, 11:4171-4185 2020.
[33] Kailun Yang, Jiaming Zhang, Simon ReiB, Xinxin Hu, and Rainer Stiefelhagen. Capturing omni- range context for omnidirectional segmentation. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1376-1386, 2021.
[34] Jiaming Zhang, Kailun Yang, Chaoxiang Ma, Simon Reiß, Kunyu Peng, and Rainer Stiefelhagen. Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmenta- tion. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16896-16906, 2022.
[35] Jiaming Zhang, Kailun Yang. Hao Shi, Simon ReiB, Kunyu Peng, Chaoxiang Ma, Haodong Fu, Kaiwei Wang, and Rainer Stiefelhagen. Behind every domain there is shift: Adapting distortion-awart vision transformers for panoramic semantic segmentation. arXiv preprint arXiv:2207. 1860, 2022.
[36] Qiang Zhao, Chen Zhu, Feng Dai, Yike Ma, Guoqing Jin, and Yongdong Zhang, Distortion- aware cnns for spherical images. In International Joint Conference Artificial Intelligence, 2018.
[37] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In European Conference Computer Vision, 2019.
[38] Xueye Zheng, Tianbo Pan, Yuan Luo, and Lin Wang. Look at the neighbor: Distortion- aware unsupervised domain adaptation for panoramic semantic segmentation. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 18641-18652. 2023.
[39] Chuanqing Zhuang, Zhengda Lu, Yiqun Wang, Jun Xiao, and Ying Wang. Acdnet: Adaptively combined dilated convolution for monocular panorama depth estimation. ArXiv, abs/2112.1 14440, 2021.
13

---

A Appendix
We provide the appendix, including the detailed algorithm explanations from the main paper, color bars for segmentation labels, examples of distance to ceiling, floor masks, and 2D/3D visualizations of our approach. Additional qualitativequuanitativee results will be updated on github, please follow the repository at:
http///aaddnidddceeeeet.aal.a..e distancee
A.1 Projection of equiretangular image to 3D points in Cartesian coordinates using depths.
(u.v)
60.0, 0,
Equirectangular Image
Spherical Image
Figure 8: Convert equirectangular image spherical image. Image is adjusted from Ai et al. [
A pixel from equirectangular image can be projected to spherical coordinate, then be converted to Cartesian coordinates (Ir, y, =) using depth map estimated as described in Section 3.2 Pixels in equirectangular images can be defined: и, v, where u ranges from 0 to w and ranges from 0 to h. we define the horizontal unit angle as 1 2n and the vertical unit angle as p a. each pixel (u, v)
in the ERP can be mapped to the spherical coordinate (0, Φ) as shown in Figure
(0, () (u 0, v sp
Here, p is the radius or depth, E [0, 2T] is the longitude angle, and ø E [0, || is the latitude angle. To convert from spherical coordinates (0, Φ) to Cartesian coordinates (x, v. =) using depth maps, the following formulas are used: (s) sin (Ø). y = cos( (), and cos( sin(
A.2 Least Square Method for plane fitting from point cloud.
The least squares method is statistical technique used to find the best-fitting plane to set of three-dimensional points (Ii; Vis Zi). The equation of a plane in three-dimensional space can be written as:
+
where b, and are the plane parameters to be determined.
To apply the least squares method, we aim minimize the sum of the squared distances between the observed points and the plane. The objective function to minimize is:
S(a, b,
Σ (ax; by:
2z)²
where is the number of data points. These equations can be written in matrix form
yo
20
//1
Fa
z1
(8)
:
:
Cc_
Un
2m
14

---

Denoting the matrix on the left as A and the vector on the right as B, we obtain a system of linear
equations: Aw = B, where
The least squares solution is given by:
w -
This provides the optimal parameters a, b, and for the plane equation by Assuming the 2-axis is aligned with gravity, the plane equation can be initially defined as c = 2, where and zero. Here, represents the average z-coordinate value of the point cloud.
In images where the floor ceiling is not pre-estimated, the plane defined as a surface parallel to another, separated by predefined distance.
A.3
Color bars for segmentation labels.
Void
Beam
Board
Bookcase
Ceiling
Chair
Clutter
Column
Door
Floor
Sofa
Table
Wall
Window
Figure 9: Stanford2D3DS dataset label colors.
Wall
Floor
Cabinet
Bed
Chair
Sofa
Table
Door
W Indoow
Booksbelf
Shalves
Curtain
Cluckis
Celling
Books
Redislgerutor
Televiaiin
Papse
Towal
Sho. Curiain
Bux
Whiteeborrd
Stand
Slak
Bathtub
Otb. Furniture
Prop
Figure 10: Structured3D dataset label colors.
Vold
Viall
Floor
Tabbee
Picture
Cattirtt
Custion
Window
Sofa
Bed
Curtain
Chass of
drawers
Plant
Sink
Stairs
Calling
Toilet
Steel
Towel
Fireplace
Lighttng
Railing
Shevinn
Blinds
Gym Equip.
Seating
Board Panel
Furniture
Appliances
Clothes
Objects
Misc
Figure Matterport3D dataset label colors.
15

---

A.4 More examples of distance to ceiling and distance to floor masks.
Image
ceiling-floee floor masks
Dense predicted depth
distance to ceiling
distance to floor
Figure 12: More examples of distance to ceiling and distance to floor masks, where light to dark colors represent distances from far to near.
A.5 2D/3D visualization of our approach step by step.
b
OD
Figure 13: a) Input image. b) Predicted depth. c) Ceiling mask before softmax. d) Floor mask before softmax. e) Different views of pointcloud constructed from predicted depth. Ceiling and floor 3D visualization. g) Planes of ceiling and floor in 3D coordinates after applying least square method. h) Distance of 3D points to ceiling plane. i) Distance points to floor plane, k) Final segmentation.
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: main claims made in the abstract and introduction accurately reflect the paper's contribution, there are three main contributions (check introduction for more detail).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We mention two limitations the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
There are no theoretical result in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We will make it publicly (weight, code, pipeline) via provided github.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
We will make it publicly (weight, code, pipeline) via provided github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.cc/pulli//guiii wwwtii/gguie///eeeeeeoocc CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We mention it clearly in the implementation detail and provided github.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: There are no Experiment Statistical Significance in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We mention the GPU, computational complexity in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: stttps://nott /ttppris///uurpss...o cctpriissccccpcccccccooo http://s................ccccccaa
Answer:
[Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original papers and public datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
21

---

Answer:
[NA]
Justification:
We do not release new assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "QcXaWYm50b", "reviewer": "Reviewer_QfCk", "summary": "The authors decompose the indoor panoramic semantic segmentation task into two sub-tasks: segmentation and depth estimation and design to enhance the geometric information. Specifically, the method first introduces the vertical relative distance to demonstrate the relationships between planar objects (ceiling and floor) and other object pixels in 3D coordinates. Then it aggregates various representations into a transformer-based context module to learn the geometric context. The experimental results partially prove the efficiency of the method proposed by the authors.", "questions": "Please the weakness section.", "limitations": "The authors mentioned the limitations in the conclusion section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors consider segmenting the planar objects (which make up about 40% of a panoramic image) and other objects in separate strategies, which alleviate the negative impact caused by the imbalance category distribution.\n2.\tThe authors consider implicitly modeling the spatial relationships by defining vertical relative distance, which is adaptive to the indoor scenes.", "weaknesses": "1.\tThe authors claimed in Line 54 that the method is proposed to deal with the various distortions present in panoramic images. However, the proposed method improved the performance mainly by incorporating more representations rather than aiming for distortions, and the distortion problem is not highlighted in the qualitative results.\n2.\tAlthough the input of the proposed model is only RGB images, the method utilizes the depth ground truth for the supervision of the depth estimation task, thus the comparisons with the state-of-the-art methods that utilize only RGB in Table 2 are unfair.\n3.\tThe ablation studies without the depth estimation task are needed, for instance, the authors can supplement in Table 4 the experimental results of “F_img+F_h+F_m”.\n4.\tThe qualitative comparisons and analysis without the depth estimation task are suggested to supplement in the main paper.\n5.\tIt is suggested to conduct on more datasets such as Matterport3D to verify the general ability of the proposed method.\n6.\tMinor: The DPE in Figure 2 is suggested to specify.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QfCk,\nThank you for appreciating our approach. We will address your comments below.\n\n**Q1: About the wrong claim**\n\nWe agree with the reviewer that this sentence should be changed, our meaning is 'to reduce the distortion gap between segments in each group'. \n\n**Q2: About the unfair comparison**\n\nSince our work utilizes the depth ground truth for the supervision of the depth estimation task, thus the comparisons with SOTA methods only RGB may be unfair. Actually, in the table 2, we compare our framework with two kinds of previous work: \n1) Input RGB and output semantic segmentation: SGAT4PASS, Trans4PASS, Trans4PASS+, SFSS-MMSI(RGB)  \n2) Input RGB and output depth estimation & semantic segmentation as ours: FreDSNet \n\nWe also mention the quantitative result of SFSS-MMSI (input RGB+depth and output semantic segmentation), please note that comparing our method with PanoFormer and SFSS-MMSI, which inputs both RGB and ground truth depth, is also unfair. With ground truth depth as input, the PanoFormer or SFSS-MMSI can exploit the geometry more accurately than depth by learning as our method. Please refer the \"Author Response to All Reviewers\" for more detail.\n\n**Q3+Q4: Performance of model without the involvement of depth**\n\nThank the reviewer for the recommendation. Due to time constraints, we just additionally considered ablation studies on the Stanford dataset with two settings. First, we keep the segment partitioning and optimization strategy and remove the depth branch as well as F_pc, F_dist from the input of the Transformer-based Context Module. Second, we still keep the joint learning over-sampled segments with depth estimation, but remove F_pc and F_dist from the Transformer-based Context Module, detail of the quantitative comparison is shown below:\n\n| Methods   |      Avg mIoU (%)      |   F1 mIoU (%) |\n|----------|:-------------:|------:|\n| baseline |  53.7 | 53.6 |\n| w/o depth involvement (both depth estimation and F_pc, F_dist) |    54.3 |   54.7 |\n| with depth estimation but no F_pc and F_dist|    54.6 |   55.0 |\n| with depth involvement (depth estimation and F_pc, F_dist) |  **55.5**  |  **56.8**|\n\nIt can be seen that without depth involvement (both depth estimation and F_pc, F_dist) the performance is still slightly higher than the baseline due to the effectiveness of the proposed segments dividing strategy.  In case of including depth estimation but without F_pc and F_dist, the performance increase is mainly for ceiling-floor, then Avg mIoU just increases slightly. With the full involvement, the model become more robust, indicated by significant improvement with mentioned settings.\nAs suggested by the reviewer, more ablation studies should be conducted and mentioned in the main paper.\n\n**Q5: Performance on Matterport3D dataset**\nPlease refer the \"Author Response to All Reviewers\"\n\n**Q6: Specify DPE**\nIn fact, the specification of DPE was introduced in the Trans4PASS+ paper, and should be mentioned in more detail in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vs14MviCqU", "reviewer": "Reviewer_GGrZ", "summary": "This paper introduces a novel approach to panoramic semantic segmentation. The work views panoramic segmentation from two perspectives including over-sampled segmentation and under-sampled segmentation. The rich geometric depth information is exploited using a transformer-driven context module. The experiments on two public datasets demonstrate the effectiveness of the proposed model.", "questions": "1. How about the current occlusion-aware seamless segmentation, which is relevant for panoramic scene understanding?\n2. Following SFSS-MMSI, would you consider providing more multi-modal ablation results for analysis? For example, when using different representations, how would the performance change?\n3. Would you consider visualizing some feature maps or attention maps to help better understand the effectiveness of the proposed method?\n4. Is it possible to evaluate on outdoor panoramic segmentation benchmarks? For example, some outdoor panorama datasets could be enriched with geometric information by using large models like depth anything. This could be discussed.", "limitations": "This paper has clearly discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of studying over-sampled and under-sampled segments in panoramic segmentation is interesting.\n2. The paper is overall well-written and nicely structured.", "weaknesses": "1. The section 3.5 should be better formalized to help the readers understand the merging process.\n2. The proposed framework contains a lot of steps, which could lead to increased running time. It would be nice if the authors could present the computation complexity of different steps in the framework.\n3.  Matterport3D is also an important benchmark of panoramic indoor segmentation. It would be nice to evaluate the proposed method on the Matterport3D as well.\n4. Regarding the vertical relative distance, would you directly compare the proposed representation against more recent state-of-the-art representations or distance measures to verify the superiority of your proposed method?\n5. The proposed transformer-based context module should be compared against existing state-of-the-art decoders and context aggregation methods like ASPP, PSP, SegFormer, UperNet, FPN, Mask2Former, etc.\n6. Recent panoramic segmentation methods like MultiPanoWise and DATR could be discussed and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer GGrZ, Thank you for appreciating our approach. We will address your comments (both weaknesses and questions) below.\n\n**W1:** About merging process, we agree with the reviewer that the merging processed should be described and visualized clearly for better understanding\n\n**W2:** About computation complexity, we provide the measurement on main components, it can be referred as table below:\n\n| Module    |  Params (G)  |  TFLOPs |\n|----------|:-------------:|------:|\n| Encoder |  0.024 | 0.058 |\n| Over-sampled segments joint depth estimation|   0.003 |0.010 |\n| Transformer-based Context Module related |  0.024  |0.061|\n| Under-sampled segments |  0.002   |  0.006 |\n| Total |  0.053   |  0.135 |\n\n**W3:**  About the performance on Matterport3D dataset, please refer the \"Author Response to All Reviewers\"\n\n**W4:** Comparison with SOTA distance based representation\n\nOur work: dividing segments into sub-groups, tailor optimization with different strategy, and setting constraint components to form the concept of vertical relative distance are novel approaches in the PASS, we did not find any same concept in the previous works to compare, instead of that, we conduct experiment to verify the effectiveness of proposed concept as following below:\n\n1) Model performance with and without vertical relative distance as input of Transformer based Context Module (table 4 of main paper)\n2) Beside vertical relative distance, we also considered another concept, which measures the angle between the normal vector of the ceiling-floor and the normal vector of the 3D points constructed by the predicted depth, but after adding it as an input to the Transformer-based Context Module, the performance did not change significantly. The reason for the ineffectiveness of this concept can be explained as follows: unlike the distance-based representation, where the vertical relative distance of two adjacent points in 3D is quite similar, the point-level normal vector estimated from depth is really sensitive with noise, making it more difficult for the model to achieve coverage.\n\n**W5:** Compare to other context aggregation techniques\n\nWe present the quantitative results on the Stanford dataset with different selections of context aggregation techniques, details of this comparison can be seen:\n| Module    |  mIoU(%)  |\n|----------|:-------------:|\n| Segformer decoder|  53.4 |\n| ASPP |  54.6  |\n| Transformer-based Context Module | **56.8**   |\n\nWe realised that the SegFormer decoder is lightweight but performs poorly in our model, and that ASPP is not efficient for the context aggregation task.\n\n**W6:** we agree that some models such as MultiPanoWise and DATR could be discussed and compared\n\n**Q1: About occlusion-aware seamless segmentation**\n\nWe assume that the mentioned paper is \"Occlusion-aware seamless segmentation (OASS) - ECCV24\". As the paper was published after our research, we did not consider this paper in our study. As we know, OASS has been introduced as a new task in panoramic semantic segmentation, this paper aims to solve the unsupervised domain adaptation between pinhole and panoramic outdoor image by presenting two related techniques, unmasking mechanism to solve the object occlusions and modification of Deformable Patch Embedding to reduce the image distortion. The objective and approach of this paper is different from our work.\n\n**Q2: Analysis of multi-modal ablation results**\n\nThank reviewer for the recommendation, combining our idea with multi representation as input as SFSS-MMSI is interesting and this combination will be conducted\n\n**Q3: Visualization of feature map or attention map**\n\nFor better understanding of each step in our pipeline, we provide the visualization in the attached pdf file at the \"Author Response to All Reviewers\"\n\n**Q4: Perform on outdoor dataset**\n\nUsing this approach for outdoor panoramic segmentation is worth considering, since the road can be considered as a planar object and the superiority of the depth anything model also facilitates this approach, but currently there are some limitations that limit the applicability of our approach in outdoor scenes\n1) Lack of labels for outdoor panoramic depth datasets: As described in the paper, our framework requires dense depth annotation and semantic segmentation for supervised learning. These requirements can be adapted for indoor datasets such as Stanford, Structured3D and Matterport3D. Considering an outdoor dataset for a panoramic semantic segmentation task, Cityscapes and SynPASS provide 5000 and 9080 high quality dense pixel annotations respectively, but lack dense depth annotations. In this case, the use of a large pre-trained model such as depth anything should be a possible approach, but this approach limits the joint learning and increases the runtime of the whole framework due to the execution of a large model such as depth anything. In addition, since the depth anything model is optimized for pinhole images, domain adaptation is also required to be applicable to PASS. In fact, PASS for outdoor is typically solved by unsupervised domain adaptation as presented by Trans4PASS+ or OASS. Meanwhile, the DensePASS dataset provides only 100 images with segmentation annotations, which are intended for unsupervised domain adaptation testing.\n2) For the indoor scene, the depth range is quite small, so the relative depth estimation is quite correct, which facilitates the \"vertical relative distance\" to accurately reflect the spartial properties, for example, the distance from the ceiling to a chair and that to the table are distinguishable . In contrast, for the outdoor scene, with large depth scales, the imperfect depth estimation limits the impact of the distance-based concept. for example, the \"vertical relative distance\" of some segments in the outdoor scene is unclear, e.g. the vertical gap between the road-sidewalk, road-road markings are difficult to distinguish.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f2t06gqayd", "reviewer": "Reviewer_irP4", "summary": "The method divides the indoor panorama semantic segmentation problem into the prediction of over-sampled segmentation (like ceiling, floor, and planar objects) and under-sampled segmentation (like objects in indoor scenery like furniture, windows, door, etc.) subtasks. \n\nThe paper utilizes over-sampled segment prediction with multi-task semantic and depth estimation, to provide spatial relationships of the 3D scene objects concerning the planar constraints in the form of vertical relative distances. The method presents a transformer-based attention mechanism to aggregate the obtained geometric feature representations from the over-sampled segment prediction branch to estimate the under-sampled segments. The prediction from both subtasks is then merged to produce the final panorama semantic segmentation result. Their method produces better results than the current SOTA on the mentioned standard datasets.", "questions": "•\tNetwork details of deformable MLP, segmentation, and depth head module need to be discussed.\n•\tDetailed loss function equation missing.  \n•\tLine 125: HoHoNet [22] constructed a framework ‘for of’ layout structure joint per-pixel dense prediction tasks, e.g., depth estimation, and semantic segmentation based on features of 1D horizontal representation seems mistaken.\n\n•\tThe paper misses mentioning the concept and/or gaps of the current compared methods called tangent [10], SFSS [12] and Panoformer [20] in the related work.", "limitations": "Overall, the idea of dividing the segmentation task into planar and object segmentation subtasks while utilizing easily obtained planar geometric representation to provide spatial relationships to help segment challenging objects seems distinctive but seems to show limited performance improvement using the heavy capacity model. Also, the paper misses the mentioned module, and related work details which need to be addressed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-written and easy to follow. The method provides a geometric representation called vertical relative distance of 3D scene points concerning the ceiling and floor planar context that provides the additional spatial relationship to better estimate the challenging objects in the indoor panorama scenery. The dual branch network provides slightly better performance than the methods listed in the paper", "weaknesses": "•\tThe current method is computationally complex compared to the current methods.\n•\tThe process produces unknown segments due to the merging of the two separate groups of segment predictions.\n•\tThe performance improvement of the proposed heavy capacity model does not seem significant\n•\tThe ablation study of the effect of different utilized geometric representations might require qualitative comparisons highlighting the effect of each module as the performance improvement seems limited \n•\tLack of reproducibility", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer irP4, Thank you for appreciating our approach. We will address your comments below.\n\n**Problem 1: About the heavy computationally complexity**\n\nIn fact, our model is slightly heavier than the baseline (Trans4PASS+) with 53M and 39M parameters respectively. However, in terms of TFLOPS, it is approximately the same as the baseline, which means that the execution time of our model is the same as the baseline. Since the high computational complexity is mainly caused by the Transformer-based Context Module, it can be solved by finding the most appropriate number of transformer layers for each dataset.\n\n**Problem 2: About the limit improvement performance.**\n\nOverall, our method outperforms the baseline (Trans4PASS+) with a large improvement. Besides, our model also provide the better performance compared to the SOTA SFSS-MMSI except for validation on Structured3D dataset, we confirm that this is just test setup problem. SFSS-MMSI choose the best model on val-set to perform on test, while we conducted the opposite way, if we follow the same test setup, our method surpasses SFSS-MMSI on both val and test sets with 72.86% (val) and 71.66% (test), respectively. We also mention the comparison on Matterport3D dataset, please refer the \"Author Response to All Reviewers\".\n\n**Problem 3: For the writing problem**\n\nThank the reviewer for pointing out the writing problem, the lack of mention of the concept, and the incomplete related work, it could be addressed in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOYJkEOw2l", "reviewer": "Reviewer_DACe", "summary": "The goal of the paper is panoramic semantic segmentation.  Given a panoramic image, produce a panoptic segmentation.\n\nMain ideas:\n1) Separate panoramic image into two sets of segments:\n  over-sampled, representing planar objects such as ceiling and floor, and\n  under-sampled, representing for other elements\n2) Tailor optimization to expected properties of two sets:\n  For oversampled:\n   joint semantic segmentation and dense depth estimation\n  For undersampled:\n    add hand-crafted feature representing vertical relative distances.\n  For combining:\n    Transformer-based Context Module\n\nSpecific contributions:\n1) separation of ceiling and floor from rest of scene,\n2) vertical relative distances features\n3) transformer-based context module\n\nThese contributions are different from, but conceptually related to, past work as follows:\n1) The separation of ceiling and floor seems related in concept to previous work that separates the ground in lidar scans\n2) The vertical relative distances features seems related to the HHA geocentric coordinate frame in \"Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images\" (Gupta et al. 2013).", "questions": "I understand the motivation for separating the ceiling and floor from the rest of the scene, partly because they are oversampled and partly because they are planar.   Does the same logic apply for walls?   I understand that they will not be as oversampled as the ceiling and floor in a panoramic image, but still they will be oversampled.   Plus, they are planar.   Did you consider separating them too?   If not, why not?   If so, why didn't it work?", "limitations": "Limitations are discussed in one sentence in the conclusion.   There is no discussion of handling more complex indoor environments, for example like a hotel lobby, or spa, or church, or ... where ceilings and floors are not so well identified.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper describes a practical method for producing a semantic segmentation of panoramic images in indoor environments.  It proposes a few ideas motivated by the specifics of indoor scenes that improve the results.  These ideas are somewhat related to ones proposed in previous papers, and so they are not revolutionary -- however, the experiments show they are helpful in the tested setting.", "weaknesses": "It seems that there is an error in the calculation of the improvement provided by the method in table 1: SGAST4PASS gets 55.5 mIoU, and Trans4PASS++ gets 53.7 -- the difference is 1.8, not 2.8 as reported in the table (bottom row) and text (line 289).   This should be corrected.\n\nIt is not clear why the authors chose Trans4PASS++ as the baseline for this study.   There are newer methods that get better results (e.g., SFSS-MMSI, which appears in table 2).   In particular, it seems weird to have a table where one method performs a little bettern in val mIoU that the proposed one, and then list a +5.18 gap versus a midling method next to the result in the bottom row of the table.   Shouldn't the baseline be the current SOTA method?\n\nIt would have been nice for the authors to show results on Matterport3D, which has a wide variety of buildings with panoramic images and semantic segmentations (see the SFSS-MMSI paper).   S2D3D has only 13 classes, and all scenes are from similar buildings with low ceilings, and so the benchmark is not as varied and difficult.\n\nThe related work in section 2 lists and describes previous papers, but it does not describe their shortcomings or contrast them with the ideas in the submission.  As a result, the authors do not help the reader place the submission in the context of how it addresses an unsolved research problem.   I think it would be much better if the last sentence or two of each paragraph in section 2 lists why the listed methods do not solve the PASS problem, and how those limitations related to the approach taking in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer DACe, Thank you for appreciating our approach. We will address your comments below:\n\n**About the reviewer's summary**\n\nWe would like to clarify that our work is about semantic segmentation of indoor panoramas, not panoptic segmentation as stated in the reviewer's summary. We also provide arguments as to why our approach is novel in indoor PASS:\n\n- Conceptual: To the best of our knowledge, the division of segments into over-sampled and under-sampled segments then optimising them based on geometric properties in our work is the first study in PASS, it directly exploits the unique properties that appear only in panoramic images(over-sampled segments are planar regions). Compare to 360x180 degree view in panoramic image, pinhole camera or normal lidar, images (depth images with lidar) cover only a part of the ceiling or the floor or neither of them, then the idea of division and segmentation with pinhole images is not feasible. So it should not be considered as a similar concept. \n- Furthermore, our motivation is not only a segments dividing method but also provide a tailor optimization strategy for each group. We analyze the unique constraint in the indoor scene that the total distance to the ceiling and floor surfaces tends to be a constant for each 3D point. Based on this observation, we develop 'vertical relative distance', a new representation that reflects the geometric relationship of 3D points to the constraint components of the scene. It can be seen as a new representation for indoor scene understanding.\n\nConsidering the HHA geocentric coordinate frame in \"Perceptual Organization and Recognition of Indoor Scenes from RGB-D Images\". Given RGB-D data, the HHA geocentric frame is designed to estimate the direction of gravity, the algorithm tries to find the direction that is most aligned or most orthogonal to locally estimated surface normal directions at as many points as possible. Meanwhile, our approach segments and considers ceiling-floor as constraint components and then develops a distance-based concept for spatial representation. We did not find a closed similarity between them.\n\nCombine these arguments, we claim that our contribution are novel in PASS\n\n**Weaknesses:**\n\n**W1: Reason for choosing Trans4pass+ as our baseline**\n\nWe verified that Trans4PASS+ already solved the distortion problem in PASS well and have become the baseline for recent methods, our goal is to divide large images into smaller groups of segments based on geometric properties to reduce the distortion gap between segments in each group and to tailor the learning strategy for each group. By combining our strategy with techniques already effectively proposed in TransPASS+, we aim to design a robust model for indoor PASS. \nSFSS-MMSI can be seen as the current SOTA before ours, but it is also based on Trans4PASS+, so is SGAT4PASS. Our goal is to develop a robust method which is also based on Trans4PASS+ to compete with these methods. It is a fact that when SFSS-MMSI was developed, SGAT4PASS was SOTA, but SFSS-MMSI is still based on Trans4PASS+ instead of SGAT4PASS.\n\n**W2: For the performance on Matterport3D dataset**\n\nPlease refer the \"Author Response to All Reviewers\"\n\n**Question: Apply the same logic for the wall**\n\nWe have already considered this approach, but there are some reasons why it should not be the same group as ceiling and floor. Firstly, **the semantic segment of the wall is not a plane**, unlike ceiling-floor, which normally appears as a single ceiling, single floor in each image, a single wall can be considered as a plane, but since it is a semantic segmentation, a class of multiple walls (parallel or orthogonal) cannot be considered as a plane (note that Plane Surface Normal Loss is only applied to planes), if the study is instance segmentation, it could also be separated. Secondly, the unique properties of the indoor scene is that the ceiling-floor sets the 3D upper-lower boundary, it could be considered as the constraint components of the room, then it facilitates the representation of spatial relationships of 3D points to constraints of the scene. It is true that the wall also appears in almost all images, but the appearance of the wall does not follow any rule, so adding the wall to this group does not make theoretical sense. Finally, adding the wall to the first group creates an imbalance between two groups, e.g. considering Figure 6 in the paper, ceiling-floor-wall may make up 75% of the image, while ceiling-floor may make up 40-50% of the image, depending on the dataset.\n\n**Limitation: For handling more complex indoor environments, for example like a hotel lobby, or spa**\n\nWe agree with the reviewer that there are some special cases where ceilings and floors are not so well identified, in which case we define a virtual plane, for example: if no ceiling is identified in over-sampled segments, the ceiling plane will be a plane parallel to the floor, and the distance from the virtual plane to the floor will be the mean ceiling-floor distance analysed over the dataset. If there is no ceiling or floor in over-sampled segments, two virtual planes could be defined, but this is rare in indoor panoramas.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers:

We sincerely appreciate the reviewers for the time and efforts on the review. We first address some common questions, followed by detailed responses to each reviewer separately. We hope our responses clarify existing doubts. We will really appreciate it if reviewer QfCk, GGrZ and DACe can kindly reconsider the decision, because of the novelty and good performance of our approach as well as the main comments are well addressed.

**Update the experiment on the Matterport3D dataset.**

Initially, we dicided to do not conduct the evaluation on the Matterport3D dataset for two reasons: first, the comparison on the Matterport3D dataset did not appear in the recent methodologies (Trans4PASS, Trans4PASS+, SGAT4PASS, FreDSNet), second, the quantitative result of SOTA on the Matterport3D dataset is quite limited, it raises skeptic about the quality of this dataset. However, following the recommendation of almost all reviewers, we update the performance(mIoU%) on this dataset and then update the full comparison table as follows:

| Methods    |Input|  Stanford - fold1  |Stanford - avg|  Structured3D-val |Structured3D-test|  Matterport3D - val|  Matterport3D - test| 
|----------|:------:|------:|------:|------:|------:|------:|------:|
| PanoFormer |RGB|  - | 52.35|55.57 |54.87|30.04|26.87|
| Trans4PASS+ |RGB|  53.60 | 53.70|66.74 |66.90|33.43|29.11|
| SFSS-MMSI|RGB|  53.43 |52.87 |71.94|68.34|35.15|31.30|
| PanoFormer |RGB+Depth|  - | **57.03**|60.98 |59.27|33.99|31.23|
| SFSS-MMSI|RGB+Depth|55.98| 55.49  |**73.78**|  70.17| **39.19**|**35.92**|
| Ours |RGB| 56.80| 55.50 |72.86|**71.66**|36.42|33.06|

There are two notes in this table. 
1) In the main paper, we reported the performance of our in Structured3D val and test are: 71.92 and 71.70, but we just realized that SFSS-MMSI chose the best checkpoint on val set then perform on the test set, while we did the opposite way, after following the existing method, we correct the quantitative results as shown in this table.
2) When the input is RGB images, our method outperforms the baseline as well as the SOTA method in all test datasets.
In addition, we also report the performance of the PanoFormer and SFSS-MMSI with the RGB and depth ground truth as input. We would like to comment that this comparison between our method (input: RGB) with PanoFormer and SFSS-MMSI (input: RGB+depth) is unfair, with corrected depth values as input, PanoFormer or SFSS-MMSI has more advantages to understand the corrected geometry than estimating depth by learning like ours. However, our approach still proves the robustness, gives a better result on the Structured3D test set, and is competitive on other datasets. Once again, these results show the effectiveness of our proposed method.

**Visualisation for a better understanding of our approach**

As requested by reviewer **GGrZ**, we provide the visualisation for better understanding, please check the pdf file for content

**Other concerns**.

We thank all reviewers for helping us to find out some errors related to writing, tables, ... All these errors should be treated carefully.

Meta-review: After the discussion phase all reviewers lean toward acceptance, noting that the manuscript is well written and has novelty. Concerns raised were primarily about the evaluation (e.g., requesting additional ablations, performance on a missing benchmark, study of computational complexity). A detailed rebuttal was submitted that helped to address many of the reviewer concerns, including adding comparisons on Matterport3D and an ablation study related to depth estimation. After the reviewer-author discussion phase, three of the four reviewers raised their rating to borderline accept. As all reviewers are in agreement, and the authors demonstrated commitment to improving the manuscript in the rebuttal phase, the AC reached a decision to accept the paper.  Please take the reviewer feedback into account when preparing the camera ready version.
Decision: Accept (poster)