On the Effectiveness of Weight-Encoded Neural Implicit 3D Shapes   

1 INTRODUCTION . While 3D surface representation has been a foundational topic of study in the computer graphics community for over four decades , recent developments in machine learning have highlighted the potential that neural networks can play as effective parameterizations of solid shapes . The success of neural approaches to shape representations has been evidenced both through their ability of representing complex geometries as well as their utility in end-to-end 3D shape learning , reconstruction , and understanding and tasks . These approaches also make use of the growing availability of user generated 3D content and high-fidelity 3D capture devices , e.g. , point cloud scanners . For these 3D tasks , one powerful configuration is to represent a 3D surface S as the set containing any point x ∈ R3 for which an implicit function ( i.e. , a neural network ) evaluates to zero : S : = { x ∈ R3|fθ ( x ; z ) = 0 } , ( 1 ) Implicit Explicit ( mesh ) where θ ∈ Rm are the network weights and z ∈ Rk is an input latent vector encoding a particular shape . In contrast to the de facto standard polygonal mesh representation which explicitly discretizes a surface ’ s geometry , the function f implicitly defines the shape S encoded in z . We refer to the representation in Eq . ( 1 ) as a latent-encoded neural implicit . Park et al . ( 2019 ) propose to optimize the weights θ so each shape Si ∈ D in a dataset or shape distribution D is encoded into a corresponding latent vector zi . If successfully trained , the weights θ of their DEEPSDF implicit function fθ can be said to generalize across the “ shape space ” of D. As always with supervision , reducing the training set from D will affect f ’ s ability to generalize and can lead to overfitting . Doing so may seem , at first , to be an ill-fated and uninteresting idea . Our work considers an extreme case – when the training set is reduced to a single shape Si . We can draw a simple but powerful conclusion : in this setting , one can completely forgo the latent vector ( i.e. , k = 0 ) . From the perspective of learning the shape space of D , we can “ purposefully overfit ” a network to a single shape Si : Si : = { x ∈ R3|fθi ( x ) = 0 } , ( 2 ) where θi now parameterizes a weight-encoded neural implicit for the single shape Si . In the pursuit of learning the “ space of shapes , ” representing a single shape as a weight-encoded neural implicit has been discarded as a basic validation check or stepping stone toward the ultimate goal of generalizing over many shapes ( see , e.g. , ( Chen & Zhang , 2019 ; Park et al. , 2019 ; Atzmon & Lipman , 2020a ; b ) ) . Weight-encoded neural implicits , while not novel , have been overlooked as a valuable shape representation beyond learning and computer vision tasks . For example , the original DEEPSDF work briefly considered – and nearly immediately discards – the idea of independently encoding each shape of a large collection : “ Training a specific neural network for each shape is neither feasible nor very useful. ” – Park et al . ( 2019 ) We propose training a specific neural network for each shape and will show that this approach is both feasible and very useful . I II III IV Point cloud × × • ×/• Mesh • × • × Regular grid • • × • Adaptive grid • • • × Neural implicit • • • • We establish that a weight-encoded neural implicit meets the criteria of a first-class representation for 3D shapes ready for direct use in graphics and geometry processing pipelines ( see inset table ) While common solid shape representations have some important features and miss others , neural implicits provide a new and rich constellation of features . Unstructured point clouds are often raw output from 3D scanners , but do not admit straightforward smooth surface visualization ( I ) . While meshes are the de facto standard representation , conducting signed distance queries and CSG operations remain non-trivial ( II ) . Signed distances or occupancies stored on a regular grid admit fast spatial queries and are vectorizeable just like 2D images , but they wastefully sample space uniformly rather than compactly adapt their storage budget to a particular shape ( III ) . Adaptive or sparse grids are more economical , but , just as meshes will have a different number of vertices and faces , adaptive grids will different storage profiles and access paths precluding consistent data vectorization ( IV ) . While previous methods have explored weight-encoded neural implicits as an intermediary representation for scene reconstruction ( e.g. , ( Mildenhall et al. , 2020 ) ) and noisy point-cloud surfacing tasks ( e.g. , ( Atzmon & Lipman , 2020a ; b ) ) , we consider neural implicits as the primary geometric representation . Beyond this observational contribution , our technical contributions include a proposed architecture and training regime for converting the ( current ) most widely-adopted 3D geometry format – polygonal meshes – into a weight-encoded neural implicit representation . We report on experiments1 with different architectures , sampling techniques , and activation functions – including positional encoding ( Mildenhall et al. , 2020 ) and sinusoidal activation approaches ( Sitzmann et al. , 2020b ) that have proven powerful in the context of neural implicits . Compared to existing training regimes , we benefit from memory improvements ( directly impacting visualization performance ) , stability to perturbed input data , and scalability to large datasets . Weight-encoded neural implicits can be treated as an efficient , lossy compression for 3D shapes . Increasing the size of the network increases the 3D surface accuracy ( see Figure 1 ) and , compared to standard graphics solutions for reducing complexity ( mesh decimation and storing signed distances on a regular grid ) , we achieve higher accuracy for the same memory footprint as well as maintaining a SIMD representation : n shapes can be represented as n weight-vectors for a fixed architecture . The benefits of converting an existing mesh to a neural implicit extends beyond compression : in approximating the signed distance field ( SDF ) of the model , neural implicits are both directly usable for many tasks in graphics and geometry processing , and preferable in many contexts compared to traditional representations . Many downstream uses of 3D shapes already mandate the conversion of meshes to less accurate grid-based SDFs , due to the ease and efficiency of computation for SDFs : here , neural implicits serve as a drop-in replacement . 1Source code , data , and demo at our ( anonymized ) repo : https : //github.com/u2ni/ICLR2021 Encoding : Latent Weight Interpolation : trivial non-trivial Scalability : poor excellent Stability : poor excellent Many works explore latent-encoding methods ( e.g. , ( Park et al. , 2019 ; Atzmon & Lipman , 2020a ; b ) ) , taking advantage of interpolation in latent space as a ( learned ) proxy for exploration in the “ space of shapes ” . We show that this flexibility comes at a direct cost of other desirable proprieties . In particular , we show that latentencoded neural implicits scale poorly as a representation for individual shapes both at training and inference time . Existing latent-encoded neural implicits are sensitive to the distribution of training data : while they may perform well for large datasets of a limited subclass of shapes ( e.g. , “ jet airplanes ” ) , we show that training fails with more general 3D shape datasets . Even within a class , existing methods rely on canonical orientation alignment ( see Figure 2 ) in order to alleviate some of this difficulty – such orientation are notably ( and notoriously ) not present in 3D shapes captured or authored in the wild and , as a result , latent-encoded neural implicits will fail to provide meaningful results for many real-world and practical shape datasets . Fitting latent-encoded neural implicits to each shape independently complicates shape space interpolation , rendering it difficult though not impossible ( Sitzmann et al. , 2020a ) . In contrast , weight-encoded neural implicits leverage the power of the neural network function space without the constraints imposed by the requirement of generalizing across shapes through latent sampling . 2 METHOD . Neural implicits soared in popularity over the last year . While significant attention has been given to perfecting network architectures and loss functions in the context of latent-encoding and pointcloud reconstruction , there is relatively little consideration of the conversion process from 3D surface meshes to weight-encoded neural implicits ( e.g. , both Park et al . ( 2019 ) and Sitzmann et al . ( 2020b ) consider this task briefly ) . We focus on identifying a setup to optimize weight-encoded neural implicits for arbitrary shapes robustly with a small number of parameters while achieving a high surface accuracy . Once successfully converted , we consider how the weight-encoded neural implicit representation compares to standard 3D model reduction techniques and how choosing this representation impacts downstream graphics and geometric modeling operations . 2.1 SIGNED DISTANCE FIELD REGRESSION . In general , the value of an implicit function f away from its zero-isosurface can be arbitrary . In shape learning , many previous methods have considered occupancy where f ( ~x ) outputs the likelihood of ~x being inside of a solid shape ( and extract the surface as the 50 % -isosurface ) Mildenhall et al . ( 2020 ) ; Mescheder et al . ( 2019 ) ; Littwin & Wolf ( 2019 ) ; Chen & Zhang ( 2019 ) ; Maturana & Scherer ( 2015 ) ; Wang et al . ( 2018 ) . We instead advocate that f should approximate the signed distance field ( SDF ) induced by a given solid shape . Learning properties aside ( see , e.g. , ( Park et al. , 2019 ) ) , SDFs are more immediately useful in graphics and geometry processing applications . Given a surface S = ∂V of a volumetric solid V ⊂ R3 , the signed distance field gS : R3 → R induced by S is a continuous function of space that outputs the distance of a query point ~x ∈ R3 modulated by ±1 depending on whether ~x is inside or outside of the solid : gS ( ~x ) = signS ( ~x ) min ~p∈S ‖~x− ~p‖ , where signS ( ~x ) = { −1 if ~x ∈ V , 1 otherwise . ( 3 ) Our goal is to regress a feed-forward network fθ to approximate the SDF of a given surface S : fθ ( ~x ) ≈ gS ( ~x ) . ( 4 ) If successfully trained , the weights θ ∈ Rm encode a neural implicit representing S . 2.1.1 ARCHITECTURE . Our proposed architecture is a feed-forward fully connected network with N layers , of hidden size H . Each hidden layer has ReLU non-linearities , while the output layer is activated by tanh . Increasing the depth and width of this network will generally improve accuracy but at the cost of increasing the memory footprint and , for example , the time required to render the surface . The weight-encoded neural implicit ’ s rendered in Figures 2 , 4 , and 8 all share a common architecture of just 8 fully connected layers with a hidden size of 32 ( resulting in just 7553 weights , or 59 kB in memory ) . Through experimentation on a subset of 1000 meshes from Thingi10k ( Zhou & Jacobson , 2016 ) , we find that this configuration yields a good balance between reconstruction accuracy , rendering speed , and memory impact ( Figure 1 ) . While maintaining acceptable surface quality , our default architecture has a 99 % reduction in number of parameters and 93 % speed up in “ time to render first frame ” compared to the default weight-encoding architecture of ( Park et al. , 2019 ) . Excited by the recent work exploring methods to overcome an MLP ’ s bias to learn low frequency signals faster , we performed experiments using both positional encodings ( Tancik et al. , 2020 ) and SIREN activations ( Sitzmann et al. , 2020b ) . Both perform well when the network architecture is sufficiently wide ( e.g. , H > 64 ) , but introduce surface noise with our more compact architecture . See Appendix A.3 for detailed experimental setup and findings . By increasing N and H , our network could in theory ( Hornik et al. , 1989 ) learn to emulate any arbitrary topology shape with infinite precision . In reality , like any representation , there are tradeoffs . The network complexity can be increased over our base configuration for smaller surface reconstruction error , or decreased for faster rendering speeds depending on the application . A sample of geometries produced at a number of configurations can be seen in Figure 1 .

Summary: The paper proposes a weight-encoded neural implicit representation for 3D shapes. The idea is to encode every shape in the network weights of its own designated small MLP network, instead of trying to learn a latent space of shapes. This leads to a really compact shape representation based on signed distance fields that could be interesting for many applications. The approach uses importance sampling to speed up training and robust losses.