PC2WF: 3D Wireframe Reconstruction from Raw Point Clouds

1 INTRODUCTION . Many practical 3D sensing systems , like stereo cameras or laser scanners , produce unstructured 3D point clouds . That choice of output format is really just a `` smallest common denominator '' , the least committal representation that can be reliably generated with low-level signal processing . Most users would prefer a more efficient and more intuitive representation that describes the scanned object ’ s geometry as a compact collection of geometric primitives , together with their topological relations . Specifically , many man-made objects are ( approximately ) polyhedral and can be described by corners , straight edges and/or planar surfaces . Roughly speaking there are two ways to abstract a point cloud into a polyhedral model : either find the planar surfaces and intersect them to find the edges and corners , e.g. , Schnabel et al . ( 2007 ) ; Fang et al . ( 2018 ) ; Coudron et al . ( 2018 ) ; or directly find the salient corner and/or edges , e.g. , Jung et al . ( 2016 ) ; Hackel et al . ( 2016 ) . A wireframe is a graph representation of an object ’ s shape , where vertices correspond to corner points ( with high Gaussian curvature ) that are linked by edges ( with high principal curvature ) . Wireframes are a good match for polyhedral structures like mechanical parts , furniture or building interiors . In particular , since wireframes focus on the edge structure , they are best suited for piece-wise smooth objects with few pronounced crease edges ( whereas they are less suitable for smooth objects without defined edges or for very rough ones with edges everywhere ) . We emphasise that wireframes are not only a `` compression technique '' to save storage . Their biggest advantage in many applications is that they are easy to manipulate and edit , automatically or interactively in CAD software , because they make the salient contours and their connectivity explicit . Reconstructed wireframes can drive and help to create 3D CAD models for manufacturing parts , metrology , quality inspection , as well as visualisation , animation , and rendering . Inferring the wireframe from a noisy point cloud is a challenging task . We can think of the process as a sequence of steps : find the corners , localise them accurately ( as they are not contained in the point cloud ) , and link them with the appropriate edges . However , these steps are intricately correlated . For example , corner detection should `` know '' about the subsequent edge detection : curvature is affected by noise ( as any user of an interest point detector can testify ) , so to qualify as a corner a 3D location should also be the plausible meeting point of multiple , non-collinear edges . Here we introduce PC2WF , an end-to-end trainable architecture for point cloud to wireframe conversion . PC2WF is composed of a sequence of feed-forward blocks , see Fig . 1 for the individual steps of the pipeline . First a `` backbone '' block extracts a latent feature encoding for each point . The next block identifies local neighbourhoods ( `` patches '' ) that contain corners and regresses their 3D locations . The final block links the corners with an overcomplete set of plausible candidate edges , collects edge features by pooling point features along each edge , and uses those to prune the candidate set to a final set of wireframe edges . All stages of the architecture that have trainable parameters support back-propagation , such that the model can be learned end-to-end . We validate PC2WF with models from the publicly available ABC dataset of CAD models , and with a new furniture dataset collected from the web . The experiments highlight the benefit of explicit , integrated wireframe prediction : on the one hand , the detected corners are more complete and more accurate than with a generic corner detector ; on the other hand , the reconstructed edges are more complete and more accurate than those found with existing ( point-wise ) edge detection methods . 2 RELATED WORK . Relatively little work exists on the direct reconstruction of 3D wireframes . We review related work for the subtasks of corner and edge detection , as well as works that reconstruct wireframes in 2D or polyhedral surface models and geometric primitives in 3D . Corner Detection . Corner points play a fundamental role in both 2D and 3D computer vision . A 3D corner is the intersection point of ≥3 ( locally ) non-collinear surfaces . Corners have a well-defined location and a local neighbourhood with non-trivial shape ( e.g. , for matching ) . Collectively the set of corners often is sufficient to tightly constrain the scale and shape of an object . Typical local methods for corner detection are based on the 2nd-moment matrix of the points ( resp . normals ) in a local neighbourhood ( Głomb , 2009 ; Sipiran & Bustos , 2010 ; 2011 ) , a straight-forward generalisation of the 2D Harris operator first proposed for 2D images ( Harris et al. , 1988 ) . Also other 2D corner detectors like Susan ( Smith & Brady , 1997 ) have been generalised to 3D ( Walter et al. , 2009 ) . Point-level Edge Point Detection . A large body of work is dedicated to the detection of 3D points that lie on or near edges . The main edge feature is high local curvature ( Yang & Zang , 2014 ) . Related features , such as symmetry ( Ahmed et al. , 2018 ) , change of normals ( Demarsin et al. , 2007 ) , and eigenvalues ( Bazazian et al. , 2015 ; Xia & Wang , 2017 ) can also be used for edge detection . Hackel et al . ( 2016 ; 2017 ) use a combination of several features as input for a binary edge-point classifier . After linking the points into an adjacency graph , graph filtering can be used to preserve only edge points ( Chen et al. , 2017 ) . EC-Net ( Yu et al. , 2018 ) is the first method we know of that detects edge points in 3D point clouds with deep learning . Learning Structured Models . Many methods have been proposed to abstract point clouds into more structured 3D models , e.g. , triangle meshes ( Lin et al. , 2004 ; Remondino , 2003 ; Lin et al. , 2004 ) , polygonal surfaces ( Fang et al. , 2018 ; Coudron et al. , 2018 ) , simple parametric surfaces ( Schnabel et al. , 2007 ; 2008 ; Li et al. , 2019 ) . From the user perspective , the ultimate goal would be to reverseengineer a CAD model ( Gonzalez-Aguilera et al. , 2012 ; Li et al. , 2017b ; Durupt et al. , 2008 ) . Several works have investigated wireframe parsing in 2D images ( Huang et al. , 2018 ; Huang & Gao , 2019 ) . LCNN ( Zhou et al. , 2019a ) is an end-to-end trainable system that directly outputs a vectorised wireframe . The current state-of-the-art appears to be the improved LCNN of Xue et al . ( 2020 ) , which uses a holistic `` attraction field map '' to characterise line segments . Zou et al . ( 2018 ) estimate the 3D layout of an indoor scene from a single perspective or panoramic image with an encoder-decoder architecture , while Zhou et al . ( 2019b ) obtain a compact 3D wireframe representation from a single image by exploiting global structural regularities . Jung et al . ( 2016 ) reconstruct 3D wireframes of building interiors under a Manhattan-world assumption . Edges are detected heuristically by projecting the points onto 2D occupancy grids , then refined via least-squares fitting under parallelism/orthogonality constraints . Wireframe extraction in 2D images is simplified by the regular pixel grid . For irregular 3D points the task is more challenging and there has been less work . Another related direction of research is 3D shape decomposition into sets of simple , parameterised primitives ( Tulsiani et al. , 2017 ; Li et al. , 2017a ; Zou et al. , 2017 ; Sharma et al. , 2018 ; Du et al. , 2018 ) . Most approaches use a coarse volumetric occupancy grid as input for the 3D data , which works well for shape primitive fitting but only allows for rather inaccurate localisation of edges and vertices . Li et al . ( 2019 ) propose a method for fitting 3D primitives directly to raw point clouds . A limiting factor of their approach is that only a fixed maximum number of primitive instances can be detected . This upper bound is hard-coded in the architecture and prevents the network from fitting more complex structures . Nan & Wonka ( 2017 ) propose Polyfit to extract polygonal surfaces from a point cloud . Polyfit first selects an exhaustive set of candidate surfaces from the raw point cloud using RANSAC , and then refines the surfaces . Similarly to our PC2WF , Polyfit works best with objects made of straight edges and planar surfaces . The concurrent work Wang et al . ( 2020 ) proposed a deep neural network that is trained to identify a collection of parametric edges . In summary , most 3D methods are either hand-crafted for restricted , schematic settings , or only learn to label individual `` edge points '' . Here we aim for an end-to-end learnable model that directly maps a 3D point cloud to a vectorised wireframe . 3 METHOD . Our full architecture is depicted in Fig . 1 . In the following we describe each block and , where necessary , discuss training and inference procedures . We denote a point cloud of N 3D points with P = { pi } Ni=1 where pi ∈ R3 represents the 3D coordinates of point i . The 3D wireframe model of the point cloud P is denoted byW = ( V , E ) and it is defined , analogously to an undirected graph , by a set of vertices V which are connected by a set of edges E ⊆ V × V . As the vertices represent points in the 3D space we can think of them as being described by 3D coordinates vi ∈ V ⊂ R3 . Overall Pipeline . Our feed-forward pipeline consists of four blocks : ( 1 ) a convolutional backbone that extracts per-point features ; ( 2 ) a vertex detector that examines patches of the point cloud and decides which ones contain vertices ; ( 3 ) a vertex localiser that outputs the location of the vertex within a patch ; ( 4 ) an edge detector that instantiates candidate edges and prunes them to a wireframe . Backbone . PC2WF first extracts a feature vector per 3D point that encodes its geometric context . Our pipeline is compatible with any ( hand-coded or learned ) function that outputs a fixed-size descriptor vector per point . We use Fully Convolutional Geometric Features ( FCGF , Choy et al. , 2019b ) , which are compact and capture broad spatial context . The backbone operates on sparse tensors ( Choy et al. , 2019a ) and efficiently computes 32-dimensional features in a single , 3D fully-convolutional pass . Vertex Detector . This block performs binary classification of local patches ( neighbourhoods in the point cloud ) into those ( few ) that contain a corner and those that do not . Only patches detected to contain a corner are passed on to the subsequent block . While one could argue that detecting the presence of a vertex and inferring its 3D coordinates are two facets of the same problem , we found it advantageous to separate them . Detecting first whether a patch is likely to contain a corner , then constructing its 3D coordinates reduces the computational cost , as we discard patches without corners early . Empirically it also reduces false positives and increases localisation accuracy . Our patch generation is similar to EC-Net ( Yu et al. , 2018 ) . A patch contains a fixed number M of points , chosen by picking a patch centroid and finding the closest ( M − 1 ) points in terms of geodesic distance on the underlying surface . The geodesic distance avoids `` shortcuts '' in regions with thin structures or high curvature . To approximate the geodesic distance , the entire point cloud is linked into a k-nearest neighbour graph , with edge weights proportional to Euclidean distance . Point-to-point distances are then found as shortest paths on that graph . During training we randomly draw positive samples that contain a corner and negative ones that do not . To increase the diversity of positive samples , the same corner can be included in multiple patches with different centroids . For the negative samples , we ensure to draw not only patches on flat surfaces , but also several patches near edges , by doing this negative samples with strong curvature are well represented . For inference , patch seeds are drawn using farthest point sampling . The vertex detector takes as input the ( indexed ) features of the M points in the patch , max-pooled per dimension to achieve permutation invariance , and outputs a scalar vertex/no_vertex probability . Its loss function Lpat is binary cross-entropy . Vertex Localiser . The vertex detector passes on a set of P patches predicted to contain a corner . The M points pi of such a patch form the input for the localisation block . The backbone features fi ∈ RC and the 3D coordinates pi ∈ R3 of all M points are concatenated and fed into a multi-layer perceptron with fully connected layers , ReLU activations and batch normalisation . The MLP outputs a vector of M scalar weights wi , where ∑ i wi = 1 , such that the predicted vertex location is the weighted mean of the input points , v = ∑ wipi . The prediction vj is supervised by a regression loss Lvert = 1P ∑P i ‖vi − vGTi ‖22 , summed over all P corner patches . Predicting weights wi rather than directly the coordinates x̃ safeguards against extrapolation errors on unusual patches . We note that the localiser not only improves the corner point accuracy but also improves efficiency , by making it possible to use farthest point sampling rather than testing every input point . Edge Detector . The edge detection block serves to determine which pairs of ( predicted ) vertices are connected by an edge . Candidate edges eij are found by linking two vertices vi and vj , see below . To obtain a descriptor for an edge , we sample theNs equally spaced query locationsQ = { qk } s−1k=0 along the line segment between the two vertices , including the endpoints : q0 = ṽi , qs−1 = ṽj . For each qk we find the nearest 3D input point pNN ( k ) ( by Euclidean distance ) and retrieve its backbone encoding fNN ( k ) . This yields an ( Ns × C ) edge descriptor , which is max-pooled along its first dimension , inspired by the ROI pooling in Faster-RCNN ( Ren et al. , 2015 ) and LOI pooling in LCNN ( Zhou et al. , 2019a ) . The resulting ( 1strideNs × C ) descriptor is flattened and fed through an MLP with two fully connected layers , which returns a scalar edge/no_edge score . During training we select a number of vertex pairs from both the ground truth vertex list { vi } Gi=1 and the predicted vertex list { ṽi } Ki=1 , and draw positive and negative sets of edge samples , E+ , E− . We empirically found that evaluating all vertex pairs contained in E+ , E− during training leads to a large imbalance between positive and negative samples . Similar to Zhou et al . ( 2019a ) , we thus draw input edges for the edge detector from the following sets : ( a ) Positive edges between ground truth vertices : This set comprises all true edges in the ground truth wireframe Egt+ = E ( b ) Negative edges between ground truth vertices : Two situations are relevant : `` spurious edges '' , i.e. , connections between two ground truth vertices that do not share an edge ( but lie on the same surface , such that there are nearby points to collect features ) . And `` inaccurate edges '' where one of the endpoints is not a ground truth vertex , but not far from one ( to cover difficult cases not far from a correct edge ) . Formally : Egt- = { evi , vj = ( vi , vj ) | vi , vj ∈ V , eij /∈ E , ∀p on evi , vj , ∃p′ ∈ P , ‖p− p′‖2 < } ∪ { ei , pk = ( vi , pk ) | ( vi , vj ) ∈ E , pk ∈ P , 1 ≤ ‖vj − pk‖2 ≤ 2 } ( c ) Positive edges between predicted vertices : Connections between two `` correct '' predicted vertices that have both been verified to coincide with a ground truth wireframe vertex up to a small threshold + . Formally : Epred+ = { eṽi , ṽj = ( ṽi , ṽj ) | min ‖ṽi − vk‖2 < + , min ‖ṽj − vl‖2 < + , ( vk , vl ) ∈ Egt+ } ( d ) Negative edges between predicted vertices : These are ( i ) `` wrong links '' as in Egt- , between `` correctly '' predicted vertices ; and ( ii ) `` hard negatives '' where exactly one of the two vertices is close to a ground truth wireframe vertex , to cover `` near misses '' . Epred- = { eṽi , ṽj = ( ṽi , ṽj ) | min ‖ṽi − vk‖2 < − , min ‖ṽj − vl‖2 < − , ( vk , vl ) ∈ Egt− } ∪ { eṽi , ṽj = ( ṽi , ṽj ) | 1 ≤ ‖ṽi − vk‖2 ≤ 2 , ‖ṽj − vl‖2 ≤ 2 , vk , vl ∈ V } Fig . 2 visually depicts the different edge sets . To train the edge detector we randomly sample positive examples from E+ = Egt+ ∪ Epred+ and negative examples from E− = Egt- ∪ Epred- . As loss function Ledge we use the balanced binary cross entropy . During inference , we go through the fully connected graph of edge candidates connecting any two ( predicted ) vertices . Candidates with a too high average distance to the input points ( i.e. , not lying on any object surface ) are discarded . All others are fed into the edge detector for verification . End-to-end training . When training the complete network , we minimise the joint loss function with balancing weights α and β : L = Lpat + αLvert + βLedge . Non-Maximum Suppression ( NMS ) is a standard post-processing step for detection tasks , to suppress redundant , overlapping predictions . We use it ( i ) to prevent duplicate predictions of vertices that are very close to each other , and ( ii ) to prune redundant edges .

Summary: This paper presents a deep architecture to extract a wireframe model from a 3D point cloud. This is a problem of high interest, and the author claim that the approach they present is the first one to address this task, which is true to the best of my knowledge. Since both the approach and the evaluation are sound, this alone seem to warrant publication. There are however several weaknesses in the paper, and my accept recommendation is conditional to clear answer on each of them: