AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos
Authors: Yuze He
Wang Zhao
Shaohui Liu
Yubin Hu
Yushi Bai
Yu-Hui Wen
Yong-jin Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Planar Reconstruction
3D Plane Representation
Primary area: machine_vision

Abstract: We introduce AlphaTablets, a novel and generic representation of 3D planes that features continuous 3D surface and precise boundary delineation. By representing 3D planes as rectangles with alpha channels, AlphaTablets combine the advantages of current 2D and 3D plane representations, enabling accurate, consistent and flexible modeling of 3D planes. We derive differentiable rasterization on top of AlphaTablets to efficiently render 3D planes into images, and propose a novel bottom-up pipeline for 3D planar reconstruction from monocular videos. Starting with 2D superpixels and geometric cues from pre-trained models, we initialize 3D planes as AlphaTablets and optimize them via differentiable rendering. An effective merging scheme is introduced to facilitate the growth and refinement of AlphaTablets. Through iterative optimization and merging, we reconstruct complete and accurate 3D planes with solid surfaces and clear boundaries. Extensive experiments on the ScanNet dataset demonstrate state-of-the-art performance in 3D planar reconstruction, underscoring the great potential of AlphaTablets as a generic 3D plane representation for various applications.

AlphaTablets: A Generic Plane Representation for 3D Planar Reconstruction from Monocular Videos
Yuze He', Wang Zhao', Shaohui Liu?, Yubin Hu',
Yushi Bai Yu-Hui Wen Yong-Jin Liu
Tsinnhua University 2 ETH Zurich
8 Beijing Jiaotong University
Abstract
We introduce AlphaTablets, novel and generic representation of 3D planes that features continuous 3D surface and precise boundary delineation. By representing 3D planes as rectangles with alpha channels, AlphaTablets combine the advantages of current 2D and 3D plane representations, enabling accurate, consistent and flexible modeling 3D planes. We derive differentiable rasterization on top AlphaTablets to efficiently render 3D planes into images, and propose novel bottom-up pipeline for 3D planar reconstruction from monocular videos. Starting with 2D superpixels and geometric cues from mre-trained models, initialize 3D planes as AlphaTablets and optimize them via differentiable rendering. An effective merging scheme is introduced to facilitate the growth and refinement of AlphaTablets Through iterative optimization and merging, we reconstruct com- plete and accurate 3D planes with solid surfaces and clear boundaries. Extensive experiments on the ScanNet dataset demonstrate state-of- state-of-the-ar performance 3D planar reconstruction, underscoring the great potential of AlphaTaablets as a generic 3D plane representation for various applications. Project page available https::/byzclusttet htw.lautttttggiiiieeeeeeee aechubb.o/aalll wtt.ap/aaaaaabetsss..aea.a.aaaea
Introduction
3D planar reconstruction from monocular videos is crucial aspect of 3D computer vision, dedicated to the precise detection and reconstruction of underlying 3D planes from consecutive 2D imagery. The reconstructed 3D planes serve as a flexible representation of surfaces, facilitating various applications such scene modeling, mixed reality and robotics.
Traditional methods for 3D planar reconstruction rely heavily on explicit geometric inputs [3
hand-craftee features 6 strong assumptions [10] [5] and solvers [37 42 [6] to detect and reconstruct the planes, thereby imposing limitations on scalability and robustness. Learning-base based
methods (27 26
28 48 leverage the power of data-driven training to directly segment the plane instances and regress the plane parameters from single or sparse-view images. Notably, PlanarRecon [50), pioneering monocular video-based learning method, operates within the 3D volume space detect and track planes, and has demonstrated promising results in recovering planar structures. However, it often falls short in detecting complete planar reconstructions and struggles with generalization across diverse scenes. How to build an accurate, complete and generalizable 3d planar reconstruction system is still challenging open problem.
2,
We inspect this problem from the perspective of plane representation. Current methodologies employ various representations, such as view-based 2D masks [42] [6] 3 26 2 48), 3D points [37] 30]. surfels [6) and voxels [50]. While 2D masks can precisely illustrate plane contours, this 2D plane representation faces inconsistencies across different views and necessitates complex matching and fusion processes reconstruct 3D surfaces. contrast, 3D representations directly depict 3D planar surfaces. However, they suffer from discontinous geometry and texture due to discretized sampling, and struggle to accurately model complex plane boundaries.
Coorreponning Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address above limitations, we propose novel plane representation termed AlphaTablets. AlphaT- ablets define 3D planes as rectangles with alpha channels. providing a natural delineation of irregular plane boundaries and enabling continuous solid 3D) surface representation. AlphaTTablets combine the best of 2D and 3D plane presentations: by defining and optimizing in 3D, AlphaTTablet ensure efficiency and consistency across all views; meanwhile, by introducing alpha channels and texture maps in plane's 's canonical coordinates, AlphaTablets can accurately model solid surfaces and complex irregular plane boundaries. Furthermore, we introduce rasterization formulations for AlphaTablets, facilitating differentiable rendering into images.
Based on AlphaTablets, we present a bottom-up pipeline for 3D planar reconstruction from posed monocular videos. Initially, leveraging 2D superpixels (1, we initialize the AlphaTablets using pre-trained depth and surface normal models [2]. This initialization yields dense yet noisy assembly of relatively small and overlapping AlphaTTablets Next, these small planes are optimized via differentiable rendering with hybrid regularizers to adjust both geometry, texture and alpha channels. We further introduce an effective merging scheme that facilitates the fusion of neighboring tablets, thereby promoting the growth of larger, cohesive planes. Through iterative cycles of optimization and merging, our final reconstructions boast solid surfaces, clear boundaries, and interpolatable texture maps, delivering accurate 3D planar structures. Moreover, our approach enables flexible plane-basee scene editing. Extensive experiments on ScanNet [ dataset demonstrate state-of the- art performance of 3D planar reconstruction, showing the great potential of being generic 3D plane representation for subsequent applications.
In summary, our contributions are threefold:
We propose AlphaTTablets, a novel and generic 3D plane representation which features effective and flexible modeling of plane geometry, texture and boundaries. We derive the rasterization formulation for AiphaTablets, enabling differentiable rendering to images.
We build an accurate and generalizable 3D planar reconstruction system from monocular videos upon Alpha'Tablets. The key components are effective initialization from pre-trained monocular cues, differentiable rendering based optimization, and the proposed merging mechanism for AlphaTablets.
The proposed system achieves state-of-ther -frtth--rt performance for 3d planar reconstruction, while also enabling flexible plane-based scene editing for various applications.
2
Related Works
Classical 3D Plane Reconstruction. Traditional 3D plane reconstruction methods typically fit planes from geometric inputs like RGB-D images [36
41
35
21) and 3D point cloud 7 (3), using robust
estimators such as RANSAC [14] and its variants 37 Another line of research approaches utilize detect 3D planes from reconstructed sparse 3D points and lines, and then optimize plane masks as multi-label segmentation through image-based solvers such as Markov Random Fields (MRF) and Graph Cut (23. Argiles et al. [3] propose a plane consistency check to determine plane boundaries using square cells rather than sparse point contours. Manhattan world assumption introduced 10 53 to better reconstruct the dominant planes. While these methods can segment 2D complex planes, the 2D mask representation of planes causes view inconsistency, visibility issues and requires additional efforts to reconstruct 3D planes. contrast, AlphaTablets directly model and optimize planes in 3D with differentiable rendering, eliminating the inconsistency and occlusion problems. Given a monocular video as input, DPPTAM ( reconstructs both the 3D planar and non-planar structure in a SLAM fashion, and adapt superpixel to group homogeneous pixels for textureless regions. Our method also uses superpixels as 2D units to initialize AlphaTablets, but provides greater flexibility in adjusting geometry, texture, and boundaries through optimization.
multi-view images input. Early works 42
16
15
Learning-based 3D Plane Reconstruction. Data-driven methods leverage large-scale training data to learn geometric priors, enabling the reconstruction of 3D planes from single or sparse view images.
PlaneNet (27). PlaneRecover [52] and PlaneRCNN (26] create training datasets for plane detection and reconstruction, and train deep neural networks to directly segment plane instances and regress
plane parameters from single RGB image. Further enhancements along this line include the use of
transformer architectures 447, multi-task collaboration [40), pairwise plane relations [4 and feature
clustering [54], etc. Due to the highly ill-posed nature of single-image plane reconstruction, many

---

Up Vector
RG9 Texture
Transparecyy
Inttal loed from Keyframe images
Posttion
Dstanne
Rablo
Normal
3D Properties
2D Properties
Volumetric Rendering
Figure 1: Illustration of tablet properties and rendering. Normal and up vector determines the rotation of tablet in 3D space, while every tablet maintains distance ratio between the coordinates of the 3D field and 2D-pixel space.
works 22 48 adopt sparse view images as inputs, and explore joint plane detection, association and optimization to help the final reconstruction. However, these methods can only recover local 3D planar structures from sparse views, and challenging to extend them to image sequences. PlanarRecon [50] proposes 3D volume-based planar reconstruction system with plane detection and tracking modules that sequentially process video frames and update 3D planar reconstructions. While effective in producing clean and consistent 3D planes, PlanarRecon often oversimplifies planar structures, resulting in incomplete reconstructions. Furthermore, being trained on indoor datasets with gravity-aligned camera poses, PlanarRecon struggles generalize unseen data. On the contrary, our method optimizes AlphaTablets for each scene, thus can generalize to any video sequence.
3D Planar Surface Representation. While most of current works employ 2D plane representation for detection and reconstruction, 3D representation is more efficient and consistent by aggregating is one of the most widely
2D image information directly in 3D. 3D point cloud (37 38 33 used 3D surface representation, yet they are discrete samples of a surface and cannot fully capture continuous geometry and textures. Extending points into planar primitives, surfels 33 17] and 2D gaussians [88 20 represent surface with 2D disks 3D space, and demonstrate impressive results for both planar and non- non-planar surfaces. Unfortunately, both surfels and 2D gaussians only provide local planar structure within one unit, making difficult to directly represent large geometric planes.
51 32
In terms of continuous surface representation, mesh is the most popular representation with mature graphic pipelines. Many works [50] 42 49, 39
convert other 3D surface representations into mesh for visualization or further optimization. While meshes using 3D triangles or rectangles can represent 3D planes with regular quadrangular shapes, they struggle with irregular complex boundaries and are difficult to build and optimize from scratch. AlphaTablet inherit the advantages of continuous geometry and canonical texture modeling from mesh and further introduce alpha channels to handle the irregular plane boundaries, acting as "rectangle soup with alpha channels". With the popularity of implicit neural representation, several works [25] 8 have explored encoding 3D plane primitives with MLPs. Compared to implicit representations, AlphaTablets offer the advantages of explicit representation, such as easy editing and fast rendering without network inference.
3 Method
Our proposed Alpha a novel and generic 3D plane representation, which enables accurate and generalizable 3D planar reconstruction from monocular video inputs. In Sec 3.1 we first introduce the data format of AlphaTTablets, then discuss the differentiable rasterization of AlphaTTablets in Sec Finally, in Sec 3.3 we introduce a bottom-up pipeline based on Alpha'Table to conduct 3D planar reconstruction from monocular video input.
3.1 AlphaTablets: Representing 3D Planes with Semi-Transparent Rectangular Primitives
As illustrated in Figure I our proposed Tablet is shaped as rectangle with 3D geometry properties and 2D in-tablet properties. The 3D geometry includes the tablet center point p. normal vector n, up vector u and right vector The normal, up and right vectors are orthogonal. The 2D in-tablet information contains a texture map c, an alpha channel a, and a pixel range (ru, rv) that indicates the 2D resolution of the texture and transparency map. The alpha channel a ensures that arbitrary shapes can modeled by our tabler formation. Since the ratio of unit distance in 3D space and 2D in-tablet canonical space varies across different tablets, distance ratios Xu; X, of two directions on the texture map is also maintained for every tablet to acquire the tablet size 3D space.
3

---

3.2 Differentiable Rasterization
As generic 3D planar surface representation, efficient projection from 3D to 2D images is highly demanding for AlphaTablets. Therefore, we introduce the differentiable rasterization of AlphaTablets. The data format of AlphaTablets is the 3D rectangle soup with alpha channels. allowing us to easily adapt and utilize existing mesh-based efficient differentiable rendering frameworks, such as NVDiffrast [24 to composite and render an arbitrary number, shape, and position of tablets in a fully differentiable manner.
Pseudo Mesh Construction. We can leverage differentiable mesh rendering frameworks like NVDiffrast (24) by converting the tablets into pseudo meshes before each rendering pass, Note that this conversion just used for the adaption of NVDiffrast. Given a single tablet 4, we can convert to two mesh triangle faces through the following process:
8/2 = [p= ru; Au,a re//>nn Au,is Pr- rua/ A + rot/ Auçi
P: + ru4 Au, + Au,5 Ps + ruru/uuu ru; Au,a)
f [[, 1, 2], [0,2,3]
(2)
Where U; represents the 3D vertex coordinates, and f; denotes the face indices, consistent with the general mesh definition. The texture and alpha maps of all tablets are tiled onto a global texture map according to their respective resolutions (ru;, rus). The specific tile location serves as the texture coordinates for the four vertices of each tablet.
Multi-layer Rasterization. As transparency information is used in Alphetablets, the rasterization process in our approach requires rasterizing multiple layers through depth peeling extract multiple closest surfaces for each pixel. Given the model-view-ppojectio view the rasterization result of the I-th closest surface for the image pixel (i, can be formed as:
(MVP) matrix Mr of a specific
RM i, = (wa.j Ui.j tris.)
where úrj and U.y are the barycentric coordinates within triangle, and tring is the triangle index. The color and alpha values c(i, j) and aii, j) are then acquired from the texture map using the two coordinates and the triangle index.
Previous anti-aliasing techniques in rasterization predominantly work non-transparent primitives without considering learnable alpha values of each face, yet alpha channel a crucial component for AlphaTaablets An intuitive approach to incorporate alpha channels would be to conduct anti-aliasing for both texture and alpha within each rasterization layer. However, this straight-forwe forward method does not work well on tablets with alpha channels. A simple counterexample is two overlapping planes with constant alpha values of 0 and 1, respectively. The rasterization results in two rasterized layers for the intersection pixel of the two planes. And the anti-aliasing would result in both layers having alpha value below 1, causing incorrect transparency and colors at the intersected boundaries in the final alpha blending process.
Anti-aliasing for AlphaTablets
To address this issue, we propose an anti-aliasing method for the semi-tran transpaaanpaett primitives. Given a pixel through which the boundary lines two planes pass, with the original colors e₁ and c2, and alpha values di and 02, respectively, and anti-aliasing weights w and 1 w, the process to obtain the anti-aliased color can is as follows:
0.1C + a2-2(11111
Con
0i + a2 w)
(4)
And the alpha value not anti-aliased. The intuitive idea is that the blending weights of anti-aliasing should not only be determined by overlapping areas, but also take alpha values into consideration. For each rasterization layer, anti-aliasing can be further expressed using the following formula:
AA(ac) AA(a)*
Cao
Qao =0
(5)
Where a and are the alpha and color values before anti-aliasing, Cao and aza are the alpha and color values after anti- aliasing, and AA is the anti-aliasing function. We show an example figure in appendix to demonstrate the clear improvements after the anti-alias formula correction.
Alpha Composition. Once we have multiple rasterized layers, we can stack them in depth order and blend them using the alpha compositing process widely employed in volume rendcring and neural

---

Tablet Initialization
Texture & Geometry Optimization
Video Keyframes
Normal
Position
Photometric Guidance
Monocular Depths
Projection To 3D Space
Transparency
Iterative Refinement
Camera Pose
Merging Scheme
Monocular Normals
AlphaT
AlphaTablets ablets
SuperPixel
Division
Bottorn Up Morging with arbibrory shopes
Figure 2: Pipeline of our proposed 3D planar reconstruction. Given monocular video as input, we first initialize AlphaTablets using off-the-shelf superpixel, depth, and normal estimation models. The 3D AlphaTablets scheme. This iterative process of optimization and merging refines the 3D AlphaTaaleess, resulting in accurate and complete 3D planar reconstruction.
then optimized through photometric guidance, followed by the merging
radiance fields. The final color of pixel (i, j) can calculated as:
1-1 II (1 (2.j.m) m==1
Ca.j T55.0159995
Tija
(6)
Where Cigjd and ozji are the color and alpha values at pixel (i..) of the 1-th rasterization layer. Tig. represents the accumulated transmittance of the previous | layers for the given pixel.
3.3 A Bottom-up Planar Reconstruction Pipeline with AlphaTablets
AlphaTablets provide flexible 3D plane modeling and efficient differentiable rendering. Building on this, we propose bottom-up 3D planar reconstruction pipeline for monocular video input. As illustrated in Figure 2 AlphaTaaletss are first initialized via off-the-shelf geometric estimations, then optimized using differentiable rendering. Tablets are examined and merged towards larger and more complete 3D planes. The optimization and merging benefit from each other through iterative refinement. By formulating the plane segmentation as bottom-up clustering of 3D AlphaTablets and plane parameter estimation as rendering based optimization, our pipeline can accurately reconstruct detailed planar surfaces.
the texture and geometry parameters of Alpha'Tablet,
Initialization. We initialize the AlphaTablets using off-the-shelf geometric prediction models, including those for depth, surface normal and superpixel. Specifically, for each keyframe of the input video, we first apply SLIC [ method to segment images into 2D superpixels based on local color homogeneity. Next, utilize pretrained geometric models 19 12
estimate depth and surface normal for each image. To obtain initial depth and surface normal values for each superpixel, we perform 2D average pooling within each superpixel's region. These 2D superpixels then back- projected into 3I) space to form the initial tablet representation. Besides 3D geometry, the texture maps and alpha channels are initialized using 2D pixel colors and superpixel masks, respectively. The rectangle bounding box is determined by the minimum and maximum values on each 3D tablet's two orthogonal axes: up and right vector.
Optimization. Initialized from 2D view-base based depth, surface normal and superpixel estimations, the initial 3D AlphaTTablet, may contain errors, overlaps, and inconsistency. We thus perform differentiable rendering based optimization to update the parameters 3D AlphaTablets.
Learnable Purameters. While the 3D Alpha'Tablets offer significant flexibility, directly optimizing them for unrestricted movement in 3D space can cause instability. To mitigate this, we constrain each tablet's s center remain its initial camera ray. In this way, thus optimize the normal vector
5

---

and distance d. rather than 3D center position p, where d represents the distance between the tablet's s center and the camera center of the view from which was initialized. The up vectors of tablets are updated with normal to keep the rigid transformation characteristics of the tablet. Additionally, the texture and alpha channel values of each tablet are treated as learnable parameters, enabling appearance refinement to enhance the fidelity of the reconstructed planar surfaces.
Loss Design. The optimization process is driven by set of carefully designed loss functions that collectively refine the tablet parameters to achieve accurate planar reconstruction. Given input monocular video, we adopt the photometric loss as the mean squared error between the rendered image of the tablets and the observed input images Cgt: Lpho = ||c Cgell 2. By minimizing the photometric loss, the 3D Alpha'Tabless can be optimized to better fit the imput images. However, due to the ambiguity of photometric alignment and the complexity of optimization, updating AlphaTablets only with photometric loss results in fuzzy reconstructions. We thus introduce several important losses to help mitigate the ambiguity and regularize the reconstruction.
Specifically, we use alpha inverse loss to prevent the emergence of semi-transparent regions after alpha composition, which is defined as Laine = I (1 ar). Moreover, we observed that multiple ceni-trrasparency tablets, instead of one solid tablet, may occupy the same surface region to blend the rendering, which harms the geometric surface reconstruction. Inspired by mip-NeRF-360 (5), we utilize the distortion loss for A Alpha' AphaTaablets to penalize the multiple cemi-transpareney surfaces and promote the merging tablets that are in close proximity, The distortion loss is defined as below:
Edist E T.Tiallp: Pit1l12
Where T; is the blending weight of the i-th rasterization layer defined in Sec. 3.2 p: is the 3D intersection point of the i-th rasterization layer. To further regularize the surface geometry and smoothness, we render the tablets to get the depths d, and surface normal maps n,, and supervise them by the input monocular depth and surface normal estimations dm, Dm with mean squared error:
d
Trand,
D2
Tran
(8)
1=1
1=1 n
d,
11 (1 a1)
n ||m,|l2
(9)
Edepth = |ld dm 1|2
Lnormal = ||n nm
(10)
The final objective is defined as:
L = wi Lpho + W2 Cainnn + Wa + wa Leepth + Wg Lnormat
(11)
where W1, W2, W3, W/y Wg are hyperparameters balance the losses.
Merging Scheme. The optimized 3D AlphaTablets are still describing local 3D planar surface, bounded by the 2D superpixels. Therefore, to represent the exact 3D planes, we need to coherently merge the individual tablets into larger tablets. We introduce a hierarchical merging strategy that considers hybrid information including color, distance, and normal, to prevent the wrong merging.
we first construct KD-tree to find the tablet neighborhoods, and initialize union-fini data structure for all tablets. Each union-find set dynamically maintains the average color, center, and normal of all its constituent unit tablets. For each tablet, we search the KD-tree to find the nearest tablets and evaluate whether they satisfy the following constraints for merging: (1) The angle between the normals of the two tablets should be smaller than a threshold f. (2) The angle between the average normals of the union-find sets to which the two tablets belong should be smaller than a threshold 0.. (3) The projected distance between the average centers of the union-fing find sets along their average normal should be smaller than threshold d. (4) The difference between the average colors of the union- find sets should be smaller than a threshold c.
Specifically,
If all these constraints are satisfied, the two tablets are merged into the same union-find set. We repeat this process for all tablets, continually updating the average color, center, and normal of each union-find set as merges occur. This iterative merging procedure continues until no further merges are possible, resulting a set of coherent planar surfaces represented by the final merged tablets. More details about merging are included in the appendix.

---

Melric3D Seq-RANSAC
SuGaR Rey-RANSAC
PlanarRecon
Ours
Ground Truth
High
Low
High
Low
Figure 3: Qualitative results ScanNet. Error maps included. Better viewed when zoomed in.
Table 1: 3D geometry reconstruction results on ScanNet.
Method
Comp
Acc
Recall
Prec
F-Score
NeuralRecon [46 + Seq- RANSAC
0.144
0.128
0.296
0.306
0.296
Atlas [31] + RANSSAC
0.102
0.190
0.316
0.348
0.331
ESTDepth 29
PEAC [3
0.174
0.135
0.289
0.335
0.304
PlanarRecon 50
0.154
0.105
0.355
0.398
0.372
Metric3D
19
Seq- RANSAC
0.074
0.379
0.426
0.161
0.231
SuGaR
18 + Seq-RANSAC RANSAC
0.121
0.324
0.385
0.296
0.327
Ours
0.108
0.161
0.481
0.447
0.456
4 Experiments
4.1 Evaluation on 3D Plane Detection and Reconstruction
Implementation Details. We use Metric3Dv2 [9 for predicting monocular depths and Om-
nidata [12 for surface normals. We leverage the keyframe selection method NeuralRecon [46 to segment the scene into multiple parts. Each part undergoes separate optimization, followed by joint optimizations. The keyframe number of each part set The separate optimization for each part is performed for 32 epochs, while the joint optimization step is executed for 9 epochs. The weights for the loss functions are set as follows: [w), W2, W3, W4, ws] = [1.0, 1.0, 20.0, 4.0, 4.0). We use Adam optimizer, and the learning rates for the tablet's texture, alpha, normal, and distance are set to 0.01, 0.03, Ic-4, and 5e-4, respectively. After the second merge step, the learning rate for the distance is reduced 2c-4. The normal threshold is set 0.93. The entire reconstruction process for a single scene takes around 2 hours on average when executed on a single A100 GPU.
Dataset and Evaluation Metrics. We use ScanNetv2 11 dataset to evaluate the 3D plane detection and reconstruction performance of our proposed method. Following PlanarRecon (50), our method is tested on the validation set of Atlas [1 using generated 3D plane ground truth. For evaluation to use Murez's 3D metries (31 for geometry, and
metrics, we follow previous works [50
rand index (RI), variation of information (VOI), segmentation covering (SC) as plane segmentation metrics. assess the segmentation performance, the reconstructed plane instances are transferred onto the ground truth planes using the nearest neighborhood approach, following common practices.
7

---

Table 3D plane segmentation results on ScanNet.
Method
VOI 4
RI f
SC 1
NeuralRecon [46 + Seq-RANSAC
8.087
0.828
0.066
Atlas [1 + Beq-RANSAS
8.485
0,838
0,057
ESTDepth 29 PEAC
[33
4.470
0.877
0.163
Recnn [50]
3.622
0.897
0.248
Metric3D 19 Seq-RANSAC
4.648
0.862
0.209
SuGaR 18 + Seq- -ANSAA
5,558
0.775
0.082
Ours
3.468
0.928
0.273
TUM Dataset
Replica Dataset
Figure Qualitative results on TUM-RGBD and Replica datasets.
Baselines. We compare our method with different types of representative works. PlanarRecon [50] is the state-of of- off-the-an art method of learning- based 3D planar reconstruction from monocular video. Following it, we compare with strong baselines that first reconstruct the 3D scene and then fit 3D planes using RANSAC, including single or multi-vie view depth-based methods 29 19, 3D volume-
based methods 44 31 and the recent 3D gaussian based method [18]
Quantitative Results. The evaluation results for 3D plane geometry and segmentation are pre- sented in Table and Table [] respectively. Our proposed method achieves clear improvements compared to other state-of-the-an approaches across various evaluation metrics. 3D volume- based reconstruct-then- methods suffer from reconstruction errors and noise threshold-sensittve RANSAC, and exhibit relatively low performance in terms of planes' geometry and 3D coherence. Depth- methods inherently encounter 3D inconsistency, resulting in fragmented and multi- predictions. PlanarRecon [50], which is specially trained on the ScanNet dataset, demonstrates the capability to predict major planes with high geometric accuracy. However, its performance hindered by the limited coverage of the predicted planes and the failure to detect many small plane instances. Ap- proaches based on 2D Gaussian Splatting [8] tend to he heavily influenced by the poor initialization, textureless regions and motion blurs in ScanNet, resulting in degraded reconstruction performance. Compared to other methods, our approach demonstrates much improved overall performance for both geometry and segmentation.
Qualitative Results. To provide qualitative assessment of method's performance, follow PlanarRecon (50) and present the plane reconstruction results in Figure [3] along with the recall егror maps. The S Req-RANSCC method suffers from erroneous geometric reconstructions, and the Metric3D+Seq 100--Se--AANAAA RANSAC is constrained by inconsistent fuzzy points and sub-optima plane segmentations. PlanarRecon, while capable of reconstructing large planar surfaces with high geomet- ric accuracy, struggles to capture and reconstruct smaller plane instances, resulting in incomplete representations of the scene. Our method benefits from the bottom-up planar reconstruction scheme, and can accurately predict the 3D plane instances while excelling in detecting and reconstructing details, particularly for smaller plane instances. This capability significantly outperforms other methods in handling fine- fine-grained planar structures. To demonstrate the generalization ability of our method, we further test it on TUM-RGBD 45) and Replica 44] datasets. Qualitative results are shown in Figure 4
Our method can faithfully reconstruct 3D planar surfaces in various scenarios.
4.2 Ablation Studies
To validate the efficacy of our method's design, we conducted series of ablation experiments exploring the impact of various components, including the loss functions, merge schemes, and tablet anti-aliasing. The results are presented in Table 3 Tablet distortion loss encourages the planar surfaces to converge and merge, leading to improved performance. Furthermore, the normal loss and depth loss significantly contribute the geometric accuracy of the reconstructed planes, particularly

---

Table 3: Ablation studies. Alphahn denotes the alpha inverse loss.
Method
F-score
VoI
RI1
SC 1
Only Photometric and Alphalnv loss
0.240
4.096
0.936
0.191
Tablet Distortion loss
0.271
3.741
0.937
0.253
Normal loss
0.425
3.490
0.944
0,263
Depth loss
0,456
3,466
0.944
0.284
w/o tablet anti-aliasing
0,415
3.545
0.937
0,280
w/o tablet merge
0,188
6.991
0.939
0.098
Full
0.456
3.466
0.944
0.284
eurIPS
Neuri
2024
2024
Editi
My Midbleets
Original Scene
3D Coherent Scene Editings
Figure 5: 3D scene editing examples of our method.
in textureless regions where photometric loss constraints are insufficient. Merging scheme is crucial for producing appropriate 3D planes. Without merging, the 3D AlphaTables remain small plane fragments, and thus can not reconstruct 3D planes, as shown in Table Moreover, tablet antialiasing contributes to smoother results, leading to enhanced overall performance.
4.3 Example Application: 3D Plane-based Scene Editing
One of the significant advantages of Alpha'Tablets representation is its ability to perform consistent 3D scene editing by simply modifying the 2D texture maps associated with the reconstructed planes. As illustrated in Fig- our method achieve impressive results for editing 3D scenes. The accurate plane reconstruction allows for precise texture mapping, enabling the seamless application of textures, text, other visual elements onto the planar regions within the scene. Furthermore, our method offers the flexibility to modify the color or perform style transfer on individual planes, providing powerful tool for creative scene manipulation.
4.4 Limitations and Future Work
AlphaTablets effectively represent 3D planes, but it may struggle in highly non-planar regions where the planar assumption for single superpixel does not hold. Additionally, the current AlphaTaablets representation does not account for view-dependent effects. As a result, the optimization relies on color consistency across views, which can be compromised by non-Lambertian surfaces changes in lighting. the future, we aim to enhance AlphaTTablets with view-dependent modeling, and explore hybrid scene representation such AlphaTTablets with Gaussians.
5 Conclusion
In this work, we introduce AlphaTablets, novel and versatile 3D plane representation. AlphaT- ablets use rectangles with alpha channels represent 3D planes, allowing for flexible and effective arbitrary 3D plane modeling. We derive a differentiable rasterization process for AlphaTableets to enable efficient 3D-to-2D rendering. Building on this, we propose novel bottom-up up 3D planar reconstruction pipeline from monocular video input. Leveraging the AlphaTaalets representation, along with carefully designed optimization and merging schemes, our pipeline can reconstruct highly accurate and complete 3D planar surfaces in generalizable manner. Experiments on the ScanNet dataset demonstrate significant improvements over baseline methods, highlighting the potential of AlphaTablet general representation for 3D planar surfaces.

---

Acknowledgement
This work was supported by Beijing Science and Technology plan project (223 (2231100005923029, the Natural Science Foundation of China (Project Number 62332019) and Beijing Natural Science Foundation (L222008).
References
[I] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to of superpixel methods. IEEE transactions on pattern analysis and machine intelligence, (4(11))2274-221
2012.
[2] Samir Agarwala. Linyi Jin. Chris Rockwell, and David F Fouhey. Planeformers: From sparse view planes to 3d reconstruction. In European Conference Computer Vision. pages 192-209. Springer. 2022.
[3] Alberto Argiles, Javier Civera, and Luis Montesano. Dense multi multi-planar seene estimation from sparse of images. In 201/ I IEEE/RSI International Conference on Intelligent Robots and Systems, pages
4448-4554, IEEE, 2011.
[4] Caroline Baillard and Andrew Zisserman. Automatic reconstruction piecewise planar models from multiple views. In Proceedings. 1999 IEEE Computer Society Conference on Compuder Vision and Puttern Recognition (Cat. No PR00149), volume 2, pages 559-565. IEFF, 1999.
[5] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 5470-5479, 2022.
[6] Adrien Bartoli. A random sampling strategy for piecewise planar scene segmentation. Computer Vision and Image Understanding. 1:42599, 2007.
[7] Dorit Borrmann. Jan Elseberg, Kai Lingemann, and Andreas Nüchter. The 3d hough transform for plane detection point clouds: A review and a new accumulator design. 3D Research. 2(2):1-13, 2011. [8] Zheng Chen, Qingan Yan, Huangying Zhan, Changjiang Cai, Xiangyu Xu, Yuzhong Huang. Weihan Wang. Ziyue Feng. Lantao Liu, and Yi Xu. Planarnerf: Online learning of planar primitives with neural radiance fields. arXiv preprint
2iv:2401.00871 2023.
[9] Alejo Concha and Javier Civera. Dpptam: Dense piecewise planar tracking and mapping from a monocular sequence. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5686-5693. IEFE, 2015.
[10] Alejo Concha, Muhammad Wajahat Hussain, Luis Montano, and Javier Civera. Manhattan and piecewise- planar constraints for dense monocular mapping. In Robotica: Science and systems, 2014.
[11] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pages 5828-5839, 2017.
[12] Ainaz Eftekhar, Alexander Sax. Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10786-1077 (06610 2021.
[13] Chen Feng, Yuichi Taguchi, and Vineet R Kamat. Fast plane extraction in organized point clouds using agglomerative hierarchical clustering. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 6218-6225. IEEE, 2014.
[14] Martin A Fischler and Robert C Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6)3381-395 1981.
[15] Yasutaka Furukawa, Brian Curless, Steven Seitz. and Richard Szeliski. Manhattan- world stereo. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 1422-1429. IEEE, 2009. [16] David Gallup, Jan-Michael Frahm, and Mare Pollefeys. Piecewise planar and non-planar stereo for urban scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1418-1425. IEEE, 2010.
[17] Yiming Gao, Yan-Pei Pei Cao, and Ying Shan. Surfelnerf: Neural surfel radiance fields for online photorealistic reconstruction of indoor scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 108-118, 2023.
[18] Antoine Guédon and Vincent Lepetit. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. arXiv preprint arXiv:2311.12772 2023.
[19] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. arXiv preprint 2404 15506, 2024.
[20] Binbin Huang, Zehao Yu. Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403. 17888, 2024.
[21] Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 3dlite: towards commodity 3d
scanning for content creation. ACM Trans. Graph., 2036))203-1 2017.
[22] Linyi Jin, Shengyi Qian. Andrew Owens, and David F Foubey. Planar surface reconstruction from sparse views. In Proc. of the IEEE/CVF International Conference Computer Vision, pages 12991-13000 13000, 2021.
10

---

[23] Vladimir Kolmogorov and Ramin Zabin. What energy functions can be minimized via graph cuts? IEEE
transactions on pattern analysis and machine intelligence, 2672):147-155. 2004.
[24] Samuli Laine, Janne Hellsten, Tero Karras, Ycongho Scol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (ToG), 39()):-
14, 2020.
[25] Zhi-Hao Lin. Wei-Chiu Ma, Hao- Yu Hsu. Yu-Chiang Frank Wang, and Shenlong Wang. Neurmips: Neural mixture of planar experts for view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15702-15712, 2022.
[26] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. Planerenn: 3d plane detection and reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 4450-4459, 2019.
[27] Chen Liu, Jimei Yang. Duygu Ceylan. Ersin Yumer, and Yasutaka Furukawa. Planenet: Piece-wise planar reconstruction from single rgb image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2579-2588, 2018.
[28] Jiachen Liu, Pan Ji, Nitin Bansal, Changjiang Cai. Qingan Yan, Xiaolei Huang. and Yi Xu. Planemvs: 3d plane reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages (665-8675, 2022.
[29] Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, and Wenping Wang. Multi-view depth estimation using epipolar spatio-temporal networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8258 8267, 2021.
[30] Hannes Möls, Kailai Li, and Uwe D Hanebeck. Highly parallclizable plane extraction for organized point clouds using spherical convex hulls. In 2020 IEEE International Conference on Robotics and Automation
(ICRA), pages 7920-7926. IEEE, 2020.
[31] Zak Murez, Tarrence Van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabi- novich. Atlas: End-to-end 3d scone reconstruction from posed images. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part VII 16, pages 414-431.
Springer, 2020.
[32] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.087535 2022.
[33] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern
recognition, pages 652-660, 2017.
[34] Yiming Qian and Yasutaka Furukawa. Learning pairwise inter inter-plane relations for piecewise planar reconstruction. In Computer
IiroonECCV2220 2020: 16th European Conference, Glasgow, UK, August 23-28, 28.
2020, Proceedings, Part VII 16. pages -330-445.05 Springer. 2020.
[35] Carolina Raposo, Miguel Lourenço, Michel Antunes, and Joao Pedro Barreto. Plane-hased odometry using an rgb-d camera. In BMVC, volume 2. page 6. 2013.
[36] Renato Salas Moreno, Ben Glocken, Paul HJ Kelly, and Andrew J Davison. Dense planar slam. In 2014
IEEE international symposium on mixed and augmented reality (ISMAR), pages 157- 164. IEEE, 2014.
[37] Ruwen Schnabel, Roland Wahl, and Reinhard Klein. Efficient ransac for point-cloud shape detection. In
Computer graphies forum. volume 26, pages 14-2266. Wiley Online Library, 2007.
[38] Johannes L. Schonberger and Jan-Michael Frahm. Structure-from protmmoo-roommmotiion revisited. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 4 4104-4113, 2016
[39] Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yo Yu Liu, and Sanja Fidler. Deep marching tetrahedra:
hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087 6101, 2021.
[40] Jingjia Shi, Shuaifeng 7hi, and Kai Xu. Planerectr: Unified query learning for 3d plane recovery from single view. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9377-9388 2023.
query
[41] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision- ECCV 2012: 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part 12, pages 746 746-760. Springer, 2012.
[42] Sudipta Sinha, Drew Steedly, and Rick Szeliski. Piecewise planar stereo for image-based rendering. In 2009 International Conference on Computer Vision, pages 1881-1888, 2009.
[43] Christiane Sommer, Yumin Sun, Lconidas Guibas, Daniel Cremers, and Tolga Birdal. From planes to corners: Multi-purpose primitive detection in unorganized 3d point clouds. IEEE Robotics and Automation Letters, 5(2):1 764-1771, 2020.
[44] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al. The replica dataset: A digital replica of indoor spaces. arXiv
preprint arXiv:1906 190.:999607777 2019.
[45] J. Sturm, N. Finellhrrd, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgh-d slam systems. In Proc. of the International Conference on Intelligent Robot Systems (IROS), Oct. 2012. [46] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15598-15607, 2021.
[47] Bin Tan, Nan Xue, Song Bai, Tianfu Wu, and Gui-Song Xia. Planetr: Structure-guided transformers for 3d plane recovery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
11

---

4186-4195. 2021.
[48] Bin Tan, Nan Xuc, Tianfu Wu, and Gui-Song Xia. Nope-sac: Neural one one-plane ransac for sparse-view planar 3d reconstruction, IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
[49] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.100 10689, 2021.
[50]
Yiming Xic, Matheus Gadelha, Fengting Yang. Xiaowei Zhou, and Huaizu Jiang. Planarrecon: Real-time 3d plane detection and reconstruction from posed monocular videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6219-6228, 2022.
[51] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. neint-nerf: Point based neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5438 5448, 2022.
[52] Fengting Yang and Zihan Zhou. Recovering 3d planes from single image via convolutional neural networks. In Proceedings the European Conference Computer Vixion (ECCV), pages 85-100, 2018. [53] Shichao Yang and Sebastian Scherer. Monocular object and plane slam in structured environments. IEEE Roboties and Automation Letters, 4(4):3145 3152, 2019.
[54] Zehao Yu, Jia Zheng, Dongze Lian, Zihan Zhou, and Shenghua Gao. Single-image piece-wise planar 3d reconstruction associative embedding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1029- 1037, 2019.
12

---

A Appendix
A.1 More Details of Alpha Optimization
Update of up vector. Note that the tablet's up vector should not be learned during the optimization process. By considering the tablet's motion as rigid transformation, any in-plane rotation can be accounted for by optimizing the texture and alpha values. However, we need to establish an update rule for the up vector to keep the rigid transformation characteristics of the tablet. Our design is to apply the same rotation to the up vector as the one applied to the normal vector during the update. Given the normal vectors n and n' (before and after the update), and the up vector u, we can acquire the new up vector u' by:
0, n- n-n', n',
(12)
n';
(13)
0
-T3
12
K- =
T3
71
(14)
-r2
0,
R cos Or + K sin 0, + +rr"( cos 0,),
(15)
u' Ru
(16)
we first construct a KD-tree to find the tablet neighborhoods, and initialize a union-find data structure for all tablets. Here we actually use the unit tablets, which are defined as the projections of all initial 3D tablets to the current tablets, to build the KD-tree. In other words, maintain the affiliations of initial and current tablets, and use the updated initial tablets to perform the merging. The reason is that using 3D center point distances among tablets with different 3D sizes is ambiguous. For example, small tablet can have smaller center point distance than a non-adjacent small tablet, compared to a spatially adjacent big tablet. Using the unit tablets with similar sizes, the neighboring adjacency be easily determined by checking the center point distances.
Merging Scheme. As explained in Section 3.3
These unit tablets are used only because they have easily defined neighborhoods. Since the unit tablets of one current tablet come from the projection on this tablet, they share the same surface normal and adjacent positions. Thus, merging with unit tablets will definitely produce larger (or the same) tablets than current tablets. After each merging, all the unit tablets are updated by the projection onto the merged new tablets, and the merged new tablets are fed into the next optimization.
Using SLIC superpixel on ScanNet 1296x968 resolution image results in around 10k superpixels for each keyframe, leading to large number of initial tablets. To address the issue, We conduct an initial merge process after AlphaTablets initialization. In practice, we find beneficial to the accuracy and convergence speed. Table 5 the ablations of the initial merge.
Table 4: Ablation studies on initial merge.
Method
F-score f
VOI 4
RI 1
SC 1
w/o in-training merge and init merge
0.188
6.991
0.939
0.098
w/o in- training merge
0.438
5.171
0.941
0.138
w/o init merge
0.454
3.754
0.944
0.273
w/ all merge schemes
0.456
3.466
0.944
0.284
Weight Check Scheme. During the optimization process, there may be cases where some tablets are nearly invisible from all viewpoints, yet they have a relatively large transparency value. In such situations, these tablets should be removed. Additionally, there could be instances where certain regions of a tablet are not visible from any viewpoint. In these cases, those specific regions of the tablet should be excluded.
To address these scenarios, we designed a weight check mechanism: We perform a rasterization step at all viewpoints and extract the points where the alpha blending weight exceeds certain threshold (we choose 0.3 in our implementation). We record the tablet index corresponding to each of these points. Before the merging step, perform the weight check by removing tablets with an excessively
13

---

low number of associated points. Furthermore, for each tablet, we recalculate its boundary based on the texture map coordinate ranges of all the points associated with that tablet.
Tablet-camera assignment. We always maintain affiliations between the initial tablets and the current (merged) tablets (as stated in Sec A. and we keep track of the camera index that initially generated each initial tablet. When tablets are merged, we count the number of each camera index corresponding to all affiliated initial tablets and assign the most frequently occurring camera the newly merged tablet.
A.2 Additional Implementation Details
Baselines. For 3D volume-based methods including Atlas, NeuralRecon, PlanarRecon, and Metric3D with TSDF fusion, we followed PlanarRecon to use their enhanced version of Seq-RANSAC. We refer to PlanarRecon for detailed descriptions. For point-based methods such as SuGaR, since PlanarRecon's RANSAC requires 3D TSDF volume as inputs and cannot be easily adapted to points or meshes, we use the classical vanilla Seq-RANSAC, which iteratively applies RANSAC to fit planes. Here we used Open3D plane RANSAC implementation for each iteration. The hyper- parameters are carefully tuned for optimal performance. For the Metric3D baseline, We used the official Metric3D v2 implementation and pre-trained weights (v2-g) running on each keyframe get depth maps, followed by TSDF fusion to fuse into 3D volume. Finally, PlanarRecon's 's Seq-RANSAC applied to the 3D TSDF volume get the planar results. We adopted the original implementation for the SuGaR baseline, during the COLMAP pre-processing, we feed ground-truth camera poses into the pipeline, which provides better initial sparse points. After optimization, SuGaR outputs the mesh model, and uniformly sampled 100k surface points and applied vanilla Seq-RANSAA on top of sampled points to get the 3D planar results. Quantitative results for other baselines (Atlas, NeuralRecon, PEAC, PlanarRecon) were taken from PlanerRecon.
Details of tablet properties. For pixel range (rus re), each tablet's geometry is located in 3D space, while its texture is stored in 2D. The pixel range represents the resolution at which the texture is stored: it is derived directly from the range in the source image for initial tablets; for merged tablets, the pixel range is calculated as the average of all corresponding initial tablets. The distance ratios (Xu: A,) establish the relationship between the 2D texture resolution and the 3D size of the tablet. For initial tablets, the distance ratio is calculated by dividing the camera's 's focal length by the average initial distance of the tablet. For merged tablets, the distance ratio is the average of all corresponding initial tablets' ratios. The alpha channel of tablets is learnable single-channel map with the same shape as the texture map.
A.3 Additional Discussions
Different initialization for SuGaR baseline. We further experiment on our ablation subset to compare the COLMAP initialization with Metric3D's dense depth-based initialization similar to our method. For Metric3D init, we use the same keyframes as our method and randomly sample a total of 100,000 points as initial points. The results are shown in the table:
Table 5: Ablation studies on different initialization of SuGaR.
Method
F-score
VOI J
RI
SC
SuGaR+COLMAP Initialization
0.300
5.759
0.797
0.090
SuGaR+Metric33 Initialization
0.326
5.670
0.789
0.102
Ours
0.456
3.466
0.944
0.284
The ScanNet dataset presents significant challenges like numerous blurry and textureless regions, which are especially problematic for Gaussian-baas methods like SuGaR when reconstructing clear geometry. Also, SuGaR heavily relies on COLMAP reconstruction to initialize, but the COLMAP reconstruction on ScanNet is sometimes noisy, affecting the final performance. The Metric3D initialization method does indeed enhance the reconstruction quality of SuGaR (as shown in Fig. 6) but the overall reconstruction quality remains constrained, with noticeable jitter and challenges in accurately delineating planar regions, leading to inferior performance to our approach.
14

---

SuGaR with COLMAP Initialization
SuGaR with Metric3D Initializaatin
Figure 6: Qualitative Comparison of Initialization Methods for SuGaR.
Breakdown of Time Budget. Below is breakdown of the time budget for the optimization process of single scene:
Table 6: Breakdown of the time budget of single scene.
Stage
Task
Time (s)
Initialization
texture init
1517.38
geometry init
1672.57
Render
pseudo mesh rasterization
10.39
316.62
alpha composition
2.15
Loss Calculation
photometric loss
1.07
depth loss normal loss distortion loss
28.28
102.44
5.90
Training
backward
3347.83
Merge
kd-tree,union- set
96.41
geometric calculation
23.14
tablet projection weight check
22.26
62.14
The merge and rendering pipeline relatively efficient, while the initialization process (which includes converting every superpixel to an initial tablet, and texture initialization) consumes signifi- cant amount of time. This is due the current naive demonstration implementation, where tens of thousands of Python loops are called, which can be improved to enable parallelized initialization in future work. Furthermore, the NVDifiRast renders more than ten layers to perform alpha composition every forward pass, but most of the scene's structure is single-layered, resulting in a substantial back- ward computation burden during training. We regard this as another potential area for considerable optimization in the future work.
3D reconstruction accuracy. The difference in 3D accuracy (termed as Acc in Tab. of the main paper) between our method and PlanarRecon on the ScanNet dataset can be attributed to several factors. First is the scope of reconstruction: PlanarRecon often only reconstructs large planar regions. This allows for easier localization and high accuracy in these specific areas, but it limits overall coverage and performance. Our method enables more comprehensive reconstruction, including smaller planar regions, which can impact the accuracy metrics but provide a more complete representation of the scene. Another is the ground-truth coverage: worth noting that the 3D ground truth planes in ScanNet only partially cover the scene within the camera's view. Even after
15

---

Ground-Truth Labels
PlanarRccon
Ours (Accuracy)
Original Image
denotes accurncy area, including many regions exist not appear ground-trath. *Rod rectangles show regions reconstructed by our methed but did not included by ground-rruhh
The 3D ground truth labels only partially cover the range within the camera's ' view. Most of the red regions in figure highlight this issue. While these uncovered areas reduce accuracy, they should not be considered a negative outcome.
Figure 7: Demonstration of Insufficient Coverage of 3D Ground-Truth Labels:
Tablets amount after every merge
106
735157
Tablet number (Log Scale)
105
44243
104
103
881
799
496
451
Training Stage
Figure 8: Visualization of Tablet Count Evolution.
excluding areas too distant to be relevant using the camera frustum, significant portions remain uncovered. PlanarRecon learns to exclude distant reconstructions during its training stage, leading to improved accuracy metrics. Our method, however, is capable of identifying planar regions for all visible areas (evident Fig. where most of the red regions highlight this phenomenon). While these uncovered areas affect the evaluation accuracy, they should not necessarily be considered negative outcome. Our method provides more complete reconstruction of the scene, including areas not represented in the ground truth data.
Tablet count evolution. We demonstrate the tablet count evolution of single scene in Fig. The number of tablets decreases rapidly in the early merging stages and gradually converges into several hundred. Notably, the final tablets contain large portion of small tablets representing non-planar regions, while the primary planar scene structure is adequately represented with fewer tablets.
A.4 Additional Qualitative Results
We provide more qualitative results in Fig. 10 and Fig.
11
A.5 Broader Impacts
3D planar reconstruction and editing have the potential to revolutionize numerous fields such as entertainment, media, accessibility, manufacturing, etc, by enhancing visualization, interaction, and
16

---

Naive Anti-aliasion Anti-aliasing
Our Tablet Anti-aliasing
Figure 9: Qualitative comparison of tablet anti-aliasing scheme. Naive anti-aliasing will lead to wrong strip artifacts, while our anti-aliasing scheme effectively mitigates those artifacts.
understanding. However, may raise concerns about privacy and data security, necessitating robust policies and safeguards.
17

---

Mctric3D + Scq-RANSAC
SuGaR SeQ-RANSAC
PlanarRccon
Ours
Ground Truth
High
FO Low
High
High
Low
Figure 10: More qualitative results on ScanNet. Error maps are included. Better viewed when zoomed in.
TUM Dataset
Replica Dataset
Figure 11: More qualitative results on TUM-RGBD and Replica datasets.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 4.4 for limitation discussions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
This paper does not include or claim theoretical contributions or results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Section 4.1 for implementation details about hyperparameters, optimization settings, etc.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [No]
Justification:
Our code will released once the paper is public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpaiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 4.1 for implementation details about hyperparameters, optimization settings, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our evaluation is deterministic and too costly to calculate the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 4.1 for running time and resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research in this paper conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section A.5 for the broader impact discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original papers that produced the code package or dataset are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "QfTNjNwMxt", "reviewer": "Reviewer_VGFx", "summary": "The paper presents a novel scene representation, AlphaTablets, for planar scene reconstruction. AlphaTablets are bounded plains with a texture map and an alpha channel, which can be optimized through a differentiable rendering scheme. By applying conventional photometric losses with regularization, AlphaTablets can be directly fit to posed monocular videos. Experiments show that the proposed method outperforms existing planar reconstruction baselines. Further, due to the texture map representation of AlphaTablets, they can be used to edit planar regions in 3D scenes.", "questions": "Please refer to the weaknesses section above.", "limitations": "Yes the limitations are stated in the main paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Representing scenes with planar structures has many important applications in AR/VR. Despite the importance however, there aren't many papers directly tackling the problem (compared to papers solving related problems such novel view synthesis). The paper proposes a simple yet effective representation to perform 3D planar reconstruction, which I believe can foster further interest in the community for future research.\n\n2. Quantitative evaluations are conducted with a proper set of baselines, and the results suggest that the proposed pipeline effectively outperforms the tested baselines.\n\n3. The downstream application on 3D scene editing is interesting, and seems to have potential adaptations for AR/VR services.\n\n4. The writing is clear to follow. The training scheme presented in Section 3.3 is straightforward and intuitive, quite akin to training procedures used for NeRFs or Gaussian splats. This is a rather good aspect in my opinion: because it indicates that the proposed AlphaTablets representation is effective on its own and does not require any significant modifications in the training setup for successful 3D reconstruction.", "weaknesses": "Overall I find the paper to propose a solid solution to 3D planar reconstruction. I only have a few minor suggestions.\n\n1. Currently, the notion of 'rasterization layers' is not clearly stated in Section 3.2. I suggest improving Figure 1 to show how multiple layers of AlphaTablets are composed to render each pixels in an image.\n\n2. The captions of figures lack information. The paper would be easier to follow if each figure was accompanied with a more detailed caption. For example, Figure 1 could elaborate more on what 'up vectors' are, or what 'distance ratios' in the figure means.\n\n3. What is the size of the resulting AlphaTablets representation? One of the benefits of 3D planar representations is the low storage cost. I wonder if AlphaTablets are sufficiently light-weight in terms of storage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. All responses below will be put into revision.\n\n**[W1: Notion unclear]**\n\nWe will revise Figure 1 to include a more intuitive display with 3D effect that demonstrates how multiple layers of AlphaTablets are composed to render each pixel in the final image.\n\n**[W2: Detailed caption for figures]**\n\nWe will revise the captions of figures with more details and comprehensive explanations.\n\n**[W3: Tablet Storage]**\n\nAs a 3D planar representation, AlphaTablets are designed to be lightweight. For example, the demo scene requires only 20MB storage, including all the texture maps in a lossless PNG format. Similar to mesh, advanced techniques such as texture atlas and compression can also be applied to further improve the storage efficiency.\n\nOnce again, we thank the reviewer for the valuable comments that helps improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pzKkCK2fvM", "reviewer": "Reviewer_ELQv", "summary": "The paper presents a light-weight 3D scene representation, which utilizes oriented 2D rectangles in 3D space with associated 2D texture and alpha maps (AlphaTablets).\n\nFor the task of 3D indoor scene reconstruction and 3D plane decomposition from multi-view posed RGB images (keyframes of a monocular video), the method uses off-the-shelf depth estimation, normal prediction and super-pixel segmentation on each image to initialize a set of the proposed AlphaTablets in 3D space: Each predicted superpixel segment is backprojected to 3D using the predicted, averaged depth of the segment and the known camera information. The orientation, size and texture of the rectangle are initialized based to the predicted, averaged normals, the bounding box of the superpixel and its color, respectively.\n\nThese AlphaTablets are then optimized using differentiable rendering and iteratively merged into larger entities. \nThe differentiable rendering considers the transparency, the merging strategy employs several constraints.\n\nFollowing baseline evaluation protocols, the paper outperforms current baseline methods on 3D plane segmentation metrics and several evaluated 3D geometry reconstruction metrics on the ScanNet validation set.\n\nThe paper shows that the proposed method allows for texture-based scene edits.", "questions": "- The qualitative results of SuGaR look surprisingly bad, can the authors please explain the setup of this baseline in more detail, especially the number of views used? Is it possible to use the proposed AlphaTablet initialization method (backprojected superpixels segments) for the 2D Gaussians in SuGaR?\n\n- Given that the optimization takes 2 hours per scene, (compared to 5sec and 60sec of PlanarRecon and NeuralRecon) it would be interesting to see a break down of the time budget, whether it is spend on the differentiable rendering, merging stage (KNN lookup), or due the evaluation of a large number of constraints on potentially many samples.\n\n- After two tablets are merged, what is the assigned camera ray the newly merged tabled is allowed to optimize its distance to the camera ray?\n\n- A plot of the number of tablets over the course of the optimization/merging would be worth including to get a sense of the magnitude of optimized elements.", "limitations": "The paper addresses some of its limitations. However, a discussion about the method's runtime and suggestions on how to improve it would be interesting.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper presents a light-weight 3D scene representation combining oriented, textured 2D rectangles with an additional alpha mask. The representation and method (initialization and optimization) are described clearly. Figure 2 clearly depicts (most of) the inputs to the system and gives a good overview of the method. The optimization terms seem to be well thought out and ablated.\nThe approach also quantitatively outperforms baselines methods on several 3D metrics.", "weaknesses": "- The descriptions of the newly evaluated baseline methods (Metric3D, SuGaR) (L267-271) is very brief and incomplete, which makes it difficult to understand how they were adapted to the specific scenario, including a description of the Seq-RANSAC setup. Additionally, a reference in the text should be added to indicate that quantitative results of the other baselines were taken from PlaneRecon [49].\n\n- Geometric evaluation: Following the evaluation of Atlas [30], to assess the geometry reconstruction quality, 3D points are sampled both on the ground truth mesh and the prediction. However, given that the proposed method employs alpha masks on the rectangles, it is unclear how points are sampled, i.e., does the point sampling considers the alpha mask? If not, this would lead to an inconsistency between the visible geometry and the evaluated geometry (points are sampled on the rectangles).\n\n- While I appreciate the fact that authors included a video to visualize some of the results, the results look quite blurry in several areas of the reconstructions (the telephone in the desk, the borders of geometry). Could the authors discuss these results in more detail?\n\nMinor:\n- In Fig. 2 it would be helpful to indicate that for the backprojection also the keyframe's camera information is being used.\n- Worth adding related work: Huang et al. 3DLite: Towards Commodity 3D Scanning for Content Creation (TOG, 2017)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. All responses below will be put into revision.\n\n**[W1:Baseline setup]**\n\nThanks and we appreciate the opportunity to clarify:\n\n1. Seq-RANSAC: For 3D volume-based methods including Atlas, NeuralRecon, PlanarRecon, and Metric3D with TSDF fusion, we followed PlanarRecon to use their enhanced version of Seq-RANSAC. We refer to PlanarRecon for detailed descriptions.\n   For point-based methods such as SuGaR, since PlanarRecon’s Seq-RANSAC requires 3D TSDF volume as inputs and cannot be easily adapted to points or meshes, we use the classical vanilla Seq-RANSAC, which iteratively applies RANSAC to fit planes. Here we used Open3D plane RANSAC implementation https://www.open3d.org/docs/latest/tutorial/Basic/pointcloud.html#Plane-segmentation for each iteration. The hyper-parameters are carefully tuned for optimal performance. \n\n2. Metric3D: We used the official Metric3D v2 implementation and pre-trained weights (v2-g) from https://github.com/YvanYin/Metric3D. The Metric3D is run on each keyframe to get depth maps, followed by TSDF fusion (widely adopted implementation from https://github.com/andyzeng/tsdf-fusion-python) to fuse into 3D volume. Finally, PlanarRecon’s Seq-RANSAC is applied to the 3D TSDF volume to get the planar results.\n\n3. SuGaR: We adopted the original implementation from https://github.com/Anttwo/SuGaR. During the COLMAP pre-processing, we feed ground-truth camera poses into the pipeline, which provides better initial sparse points. After optimization, SuGaR outputs the mesh model, and we uniformly sampled 100k surface points and applied vanilla Seq-RANSAC on top of sampled points to get the 3D planar results.\n\nWe will include these details and add references in the revision to indicate that quantitative results for other baselines were taken from PlanerRecon.\n\n**[W2:Geometric evaluation with alpha mask]**\n\nTo clarify, our point sampling process considers the alpha mask during evaluation. Specifically, we first perform a weight check (as described in Appendix Sec.A.1) using alpha channels to mask out transparent or unseen regions from all views. Then we follow the PlanarRecon to convert the remaining visible plane regions into mesh using Delaunay triangulation. Points are then evenly sampled on this generated mesh. We refer to PlanarRecon for more details.\n\n**[W3:blurry results]**\n\nSince our method focuses on reconstructing 3D planar surfaces, it has limitations when dealing with geometrically complex, non-planar regions, as discussed in the limitation section. Also, our current AlphaTablets do not consider view-dependent effects, leading to visual quality affected. As a potential solution, the introduction of a hybrid representation that combines our planar AlphaTablets with other methods, such as 3D Gaussians, may be promising for representing complex, non-planar geometries, which we will discuss in future work in the revision.\n\n**[W4,5:figure&related work]**\n\nThanks and we will revise the text as suggested.\n\n**[Q1:Details and init of SuGaR baseline]**\n\nWe appreciate the opportunity to provide more details on our setup and results:\n\n- Baseline Setup: We utilized all available views when training SuGaR. We adopted the official implementation, and enhanced COLMAP sparse reconstruction with ground-truth camera poses, which provides better initial sparse points.\n- Performance discussion: The ScanNet dataset presents significant challenges like numerous blurry and textureless regions, which are especially problematic for Gaussian-based methods like SuGaR when reconstructing clear geometry. Also, SuGaR heavily relies on COLMAP reconstruction to initialize, but the COLMAP reconstruction on ScanNet is sometimes noisy, affecting the final performance.\n- Initialization Experiment: We experiment on our ablation subset to compare the COLMAP initialization with Metric3D’s dense depth-based initialization similar to our method. For Metric3D init, we use the same keyframes as our method and randomly sample a total of 100,000 points as initial points. The results are shown in the table:\n\n| Method              | FScore    | VOI       | RI        | SC     |\n| ------------------- | --------- | --------- | --------- | --------- |\n| SuGaR+COLMAP Init   | 0.300     | 5.759     | 0.797     | 0.090     |\n| SuGaR+Metric3D Init | 0.326     | 5.670     | 0.789     | 0.102     |\n| Ours                | **0.456** | **3.466** | **0.944** | **0.284** |\n\nThe Metric3D init method does indeed enhance the reconstruction quality of SuGaR (qualitative results in Fig.1 in attached PDF file), but the overall reconstruction quality remains constrained, with noticeable jitter and challenges in accurately delineating planar regions, leading to an inferior performance to our approach. \n\n**[Q2:breakdown of time budget]**\n\nPlease refer to response to **W1 of reviewer Ku7J**.\n\n**[Q3:Tablet-camera assignment]**\n\nTo clarify, (1) we maintain affiliations between the initial tablets and the current (merged) tablets (as stated in Sec A.1); (2) We keep track of the camera index that initially generated each initial tablet; (3) When tablets are merged, we count the number of each camera indices corresponding to all affiliated initial tablets, and assign the most frequently occurring camera to the newly merged tablet. We will include these details in the revision.\n\n**[Q4:tablet number plot]**\n\nThank you for the great suggestion. We plot the numbers in Fig.3 of the attached PDF file. The number of tablets decreases rapidly in the early merging stages and gradually converges into several hundreds. Notably, the final tablets contain a large portion of small tablets representing non-planar regions, while the primary planar scene structure is adequately represented with fewer tablets. \n\n**[Limitations]**\n\nThanks and we will revise the paper with a detailed runtime analysis and potential improvements as discussed above.\n\nOnce again, we thank the reviewer for the valuable comments that helps improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fVDliFUvzd", "reviewer": "Reviewer_Ku7J", "summary": "This paper introduces a novel and generic representation of 3D planes called AlphaTablets. AlphaTablets represent 3D planes as rectangles with alpha and RGB channels, enabling accurate, flexible, and consistent modeling. The paper also proposes a differentiable rasterization method on top of AlphaTablets, as well as a bottom-up planar reconstruction pipeline for the differentiable optimization of target 3D planes. Additionally, an effective merging scheme is introduced during optimization to facilitate the growth and refinement of AlphaTablets. The extensive experiments conducted on the ScanNet dataset demonstrate the state-of-the-art performance of AlphaTablets in 3D planar reconstruction.", "questions": "See weakness.", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe concept of optimizing 3D planar primitives, referred to as AlphaTablets in this paper, through differentiable rendering is interesting. While previous methods such as PlanarRecon adopt a feedforward approach, training a model with 3D plane labels, this process can be costly and result in out-of-distribution issues when applied to unseen datasets. In contrast, reconstructing planar scenes using a differentiable rendering approach and solely relying on 2D supervision offers greater flexibility and potential.\n2.\tThe planar reconstruction results achieved on the ScanNet dataset demonstrate state-of-the-art performance in both 3D geometry and 3D plane segmentation metrics, underscoring the excellence of this paper and the potential of the proposed pipeline.\n3.\tThe coherent scene editing application is good.", "weaknesses": "1.\tI have concerns regarding the speed of the proposed method. As stated in section 4.1, it takes approximately 2 hours to reconstruct a single scene. Could the authors provide details on the time taken for each part of the entire system during optimization? Additionally, where is the primary bottleneck for speed enhancement? A detailed efficiency analysis from the author would be greatly appreciated.\n2.\tSome details regarding the plane parameters are unclear to me. What are the values of the pixel range (ru, rv) in the paper, and are they the same for all tablets? How are the distance ratios $\\lambda_u$ and $\\lambda_v$ calculated? The alpha channel is described as a learnable single-channel map with the same shape as the texture map. Is my understanding correct?\n3.\tThe initialization process is also unclear to me. It appears that each superpixel is initialized as a tablet at the beginning of the optimization. This could potentially introduce a large number of initial tablets across all 9 keyframes in a video segment, which may adversely impact the optimization speed. Why not merge similar superpixels to create larger initial planes?\n4.\tThe 3D accuracy of this method on the ScanNet dataset is notably higher than that of PlanarRecon. What are the main reasons for the lower reconstruction accuracy of the proposed method?\n5.\tWhich model did the author use in practice to predict the monocular depth and normals? Was it Metric3d or Omnidata? Additionally, the scale of the used monocular depth may not align well with the ground-truth depth. Employing mean squared error in eq (10) for depth loss could potentially introduce additional errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the comments. All responses below will be put into revision.\n\n**[W1: break down of time budget and analysis]**\n\nBelow is a breakdown of the time budget for the optimization process of a single scene:\n\n| **Stage**            | **Task**               | **Time (s)** |\n| -------------------- | ---------------------- | ------------ |\n| **Initialization**   | texture init           | **1517.38** |\n|                      | geometry init            | **1672.57** |\n| **Render**           | pseudo mesh            | 10.39 |\n|                      | rasterization          | 316.62 |\n|                      | alpha composition      | 2.15 |\n| **Loss Calculation** | photometric loss       | 1.07 |\n|                      | depth loss             | 28.28 |\n|                      | normal loss            | 102.44 |\n|                      | distortion loss        | 5.90 |\n| **Training**         | backward               | **3347.83** |\n| **Merge**            | kd-tree,union-find set | 96.41 |\n|                      | geometric calculation  | 23.14 |\n|                      | tablet projection      | 22.26 |\n|                      | weight check           | 62.14 |\n\nAs shown above, the merge and rendering pipeline is relatively efficient. The initialization process (which includes converting every superpixel to a initial tablet, and texture initialization) consumes a significant amount of time. This is due to the current naïve implementation for demonstration, where tens of thousands of Python loop is called. We will improve the implementation to enable parallelized initialization in the future work. Furthermore, the NVDiffRast renders more than ten layers to perform alpha composition every forward pass, but most of the scene's structure is single-layered, results in a substantial backward computation burden during training. We regard this as another potential area for considerable optimization in the future work.\n\n**[W2: detail of tablet parameters]**\n\nThanks and we appreciate the opportunity to clarify these details:\n\n1. Pixel Range (ru, rv):\n\n- Each tablet's geometry is located in 3D space, while its texture is stored in 2D.\n- The pixel range represents the resolution at which the texture is stored.\n- For initial tablets, the pixel range is derived directly from the range in the source image.\n- For merged tablets, the pixel range is calculated as the average of all corresponding initial tablets.\n\n2. Distance Ratios (λu and λv):\n\n- These ratios establish the relationship between the 2D texture resolution and the 3D size of the tablet.\n- For initial tablets, the distance ratio is calculated by dividing the camera's focal length by the average initial distance of the tablet.\n- For merged tablets, the distance ratio is the average of all corresponding initial tablets' ratios.\n\n3. Alpha Channel: Yes, the alpha channel is a learnable single-channel map with the same shape as the texture map.\n\n**[W3: Initial tablet merge]**\n\nDuring initialization, each superpixel is initialized as a tablet using the estimated depth and surface normal. Using SLIC superpixel on ScanNet 1296x968 resolution image results in around 10k superpixels for each keyframe, leading to a large number of initial tablets. As detailed in Appendix Sec A.1, we introduce an init merge to create larger initial tablets, which helps to improve the speed. Further efficiency improvements could be achieved through fewer superpixel initial number (by controlling the SLIC hyper-parameters) and more aggressive init merge.\n\n**[W4: reconstruction accuracy]**\n\nThe difference in 3D accuracy (termed as Acc in Table 1 of the main paper) between our method and PlanarRecon on the ScanNet dataset can be attributed to several factors:\n\n1. Scope of Reconstruction:\n\n- PlanarRecon often only reconstructs large planar regions. This allows for easier localization and high accuracy on these specific areas, but it limits overall coverage and performance.\n- Our method enables more comprehensive reconstruction, including smaller planar regions, which can impact the accuracy metrics but provides a more complete representation of the scene.\n\n2. Ground Truth Coverage:\n\n- It is worth noting that the 3D ground truth planes in ScanNet only partially cover the scene within the camera's view. Even after excluding areas too distant to be relevant using the camera frustum, significant portions remain uncovered.\n\n  - PlanarRecon learns to exclude distant reconstructions during its training stage, leading to improved accuracy metrics.\n  - Our method, however, is capable of identifying planar regions for all visible areas. This is evident in Figure 2 of the attached PDF, where most of the red regions highlight this phenomenon.\n\n  - While these uncovered areas affect the evaluation accuracy, they should not necessarily be considered a negative outcome. Our method provides a more complete reconstruction of the scene, including areas that are not represented in the ground truth data.\n\nPlease refer to Figure 2 of the attached PDF for a qualitative illustration of this issue.\n\n**[W5: depth loss and model]**\n\nTo clarify, we use Metric3D v2 for predicting monocular depths and Omnidata for surface normals.\n\nYes, there is a trade-off when using depth loss due to the depth estimation errors. To address this issue, we incorporate multiple loss terms, including photometric, normal and distortion losses, etc., to optimize and regularize the tablets. The depth loss could be viewed as a data term in optimization which constraints the results not too far from the initializations. In practice, we observed that the depth loss helps with the final reconstruction, and it is a common practice in 3D reconstruction to employ it. Further incorporation of uncertainty or better depth estimation methods could be beneficial, which we will put into future work.\n\nOnce again, we thank the reviewer for the valuable comments that helps improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Please see the attached PDF file for figures and captions.

Meta-review: All the reviewers are positive and recommending to accept the submission. The AC agrees with their decision. Please read the weakness  comments and the reviewers' concerns carefully and prepare the camera ready.
Decision: Accept (poster)