SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization

1 Introduction . Neural implicit functions have recently emerged as a powerful representation paradigm for modeling complex signals . Their continuous nature sets them apart from typical discrete representations ( i.e . pixels , voxels , meshes ) , allowing to capture high resolution details in various domains such as images [ 41 , 43 ] , 3d shapes [ 31 , 6 ] and radiance fields [ 30 , 32 ] , while retaining a reasonably compact representation . In this formulation , a deep neural network is trained with the goal of faithfully mapping input coordinates to a corresponding target domain , effectively learning the representation of signal properties such as magnitude , color , or shape occupancy . Implementing implicit neural representations with common neural structures , e.g. , multilayer pereceptrons with ReLU activations ( ReLU MLPs ) , proves to be challenging in the presence of signals with high frequencies . Consequently , recent works have demonstrated that deep implicit networks benefit from mapping the input coordinates [ 30 , 43 ] , or the intermediate features [ 41 ] to positional encodings . That is , before feeding them into a neural layer , they are first transformed to an overparameterized , high dimensional space , typically by multiple periodic functions . Positional encodings1 have been shown to enable highly detailed mappings of signals by MLP networks . For example , Fourier Feature Networks [ 43 ] suggested to map input coordinates of signals to a high dimensional space using sinusoidal functions . In their work , they show that the frequency 1In this paper , we use the term “ positional encodings ” in lower case letters to denote the family of encoding methods that map coordinates to a higher dimensional space . Not to be confused with the term “ Positional Encoding ” coined by [ 30 , 43 ] , which refers to a particular mapping scheme in this family . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of these sinusoidal encodings is the dominant factor in obtaining high quality signal reconstructions . In particular , they present compelling arguments for randomly sampling frequency values from an isotropic Gaussian distribution with a carefully selected scale , providing a striking improvement over mapping coordinates directly via standard MLPs . Despite the success of positional encodings , there are still some concerns left unaddressed : ( i ) Choosing the right frequency scale requires manual tuning , oftentimes involving a tedious parameter sweep ; ( ii ) The frequency distribution scale may change between different inputs , and accordingly it becomes harder to tune a “ one-fits-all ” model for signals that are composed of a large range of frequencies ( Fig . 1 ) ; ( iii ) Frequencies are selected for the entire input in a global , spatially invariant manner , thus missing an opportunity to better adapt to local high frequencies ( Fig . 2 ) . Our work investigates mitigations to the aforementioned challenges . We study the setting of positional encodings as input to implicit neural networks and present Spatially-Adaptive Progressive Encoding ( SAPE ) . SAPE is a policy for learning implicit functions , relying on two core ideas : ( i ) guiding the neural optimization process by gradually unmasking signal components with increasing frequencies over time , and ( ii ) allowing the mask progression to propagate at different rates among local spatial portions of the signal space . To govern this process , we introduce a feedback loop to control the progression of revealed encoding frequencies as a bi-variate function of time and space . Our work enables MLP networks to adaptively fit a varying spectrum of fine details that previous methods struggle to capture in a single shot , without involved tuning of parameters or domain specific preprocessing . SAPE excels in learning implicit functions with a large Lipschitz constant , without sacrificing quality of details or optimization stability , in problems that require meticulous configuration to achieve convergence . To highlight the latter , in Section 5.1 we present the tasks of 2D silhouettes deformation and 3D mesh transfer – both require stable optimization from the get-go in order to avoid convergence to sub-optimal local minima . SAPE is encoding-agnostic : it is not limited to a specific positional encoding type . It can be easily applied to the learning process of coordinate-based neural implicit functions of various domains including images , 2D shapes , 3D occupancy maps and surfaces , showing improvement in all of them . 2 Related work . Implicit neural representations . Recent works show the capability of deep neural networks in representing different functions ( e.g. , 2D/3D scenes or objects ) as an implicit , memory efficient continuous function [ 8 ] . These networks are used as a signed distance function ( SDF ) [ 1 , 15 , 31 ] or as an occupancy network , either binary [ 6 , 28 , 33 ] or soft volumetric density [ 26 , 30 , 49 ] . Several works train such networks to represent a collection of 3D shapes via 3D data supervision [ 6 , 16 , 29 , 28 , 31 ] , reconstruct 3D shapes [ 5 , 11 , 13 , 14 , 18 , 42 ] or infer them from 2D images [ 22 , 37 , 40 ] . Recent works employ spatial data structures to scale the represented shapes size [ 25 , 42 ] . Positional encodings ( PE ) . have been suggested as higher dimensional network input for various purposes . Radial basis function ( RBF ) networks [ 7 ] use weighted sums of embedded RBF encodings due to their symmetric property . [ 47 ] used random Fourier Features to map time spans . In natural language processing , Transformers [ 19 , 23 , 24 , 46 ] leverage sinusoidal positional encoding to maintain the order of token sequences . Our work differs by focusing on how to encode the inputs to MLPs in order to improve network implicit representations . The closest works to ours are SIREN [ 41 ] and Fourier feature networks ( FFN ) [ 43 ] . SIREN suggests to replace the ReLU activations in the network by periodic ones . FFN [ 43 ] encodes the inputs to the network by projecting them to a higher dimensional space using a family of encoding functions . In the appendix we provide an illustration of this approach with some examplary used encodings . The analysis of [ 43 ] demonstrates that FFN improves the learning of implicit neural representation compared to other alternatives . A follow-up work in [ 44 ] accelerates the training convergence by using a meta-learned initialization of the network weights . The contemporary work of [ 27 ] extends SIREN with a modulation network , to allow representing multiple shapes within a single network . Concurrently to our work , Park et al . [ 32 ] extend NERF to non-rigidly deforming scenes , and BARF [ 21 ] extends NERF for cases of imperfect camera poses . Both these works show the advantage of employing coarse-to-fine annealing linearly over the frequency bandwidth of the positional encoding . The coarse-to-fine approach bears similarity to our progressive frequency approach . Different from us , these works do not use a feedback loop or spatial encoding , which , as we show in the following , further closes the gap between the regressed function and the ground truth . As a final remark , SAPE ’ s concept of feedback loop resonates with the idea of `` predictive coding '' in computational neuroscience [ 35 ] , suggesting another angle to appreciate the rationale of our method . 3 Preliminaries . The setting of implicit neural representation is commonly formulated as learning a mapping function fθ , using a neural network , from an input of low dimensional position coordinates p ∈ Rd to an output value in the target domain . The network training data are coordinates samples as the input and an expected value of the function , or signal , as the output . Examples of such mappings include pixel locations to intensity values for images , or 3D space coordinates to binary surface occupancy labels . Achieving such mappings using conventional neural networks with ReLU is hard . It has been shown that ReLU networks exhibit a behaviour of spectral bias [ 34 , 3 , 36 , 2 , 48 , 4 ] . Networks tend to learn low frequency functions first , as they are characterized by a global behaviour , and are therefore more stable to optimize . Spectral bias , however , also prevents networks from properly learning to fit functions that change at a high rate , e.g. , functions with a large Lipschitz constant . When visual domains are concerned , this is mostly evident in delicate details missing from the network output . To mitigate this deficiency , recent works proposed to replace the ReLU activation layers with periodic sine functions [ 41 ] or map the input to some higher dimensional space in order to learn a mapping with high frequencies [ 30 , 43 ] . In the latter approach , the input p is encoded to a high dimensional embedding by a family of functionals ei : Rd → R , such that : E ( p ) = ( e1 ( p ) , e2 ( p ) , . . . , en ( p ) ) , ( 1 ) where usually n d. Tancik et al . [ 43 ] , for example , suggest the following encoding : EFFN ( p ) = ( cos ( 2πb > 1 p ) , sin ( 2πb > 1 p ) , ... , cos ( 2πb > np ) , sin ( 2πb > np ) ) , ( 2 ) where bi are frequency vectors randomly sampled i.i.d . from a Gaussian distribution with standard deviation σ . Other examples of positional encodings are included in the appendix . These embedding schemes facilitate learning of complex functions , at the price of introducing a second associated phenomenon : the spectral bias characteristic of the network is reduced . The implication in this case is twofold . Positional encodings can cause neural optimizations processes to become unstable , and consequently converge to a bad local minimum ( Fig . 9 ) . In addition , when fitting functions of varying levels of details/frequencies , neurons predicting smooth , low frequency areas are still exposed to encoding dimensions of high frequency . That , in turn , may complicate the learning process , as networks have to learn where to ignore such embedding dimensions . Indeed , Tancik et al . [ 43 ] advocate a careful choice of the standard deviation of frequency distribution , σ in their method , showing that using low values yields missing details in the network output , and using values that are very high results in noisy artifacts . Thereupon , for such static encodings the value of σ requires a parameter sweep per sample . 4 Spatially-Adaptive Progressive Encoding ( SAPE ) . Our proposed approach , SAPE , is a policy for guiding the optimization of implicit neural representations based on input coordinates . SAPE relies on the delicate balance of the two phenomena that govern the learning process : it reconciles the effects of the spectral bias and the expressiveness of high frequency encodings in a manner that benefits from both . It maintains a stable optimization , without sacrificing the ability to fit fine signal details . SAPE is composed of two key components : progressive encoding and spatial adaptivity , which allow it to be less sensitive to the encoding frequency bandwidth , i.e. , the choice of the standard deviation in the case of [ 43 ] ( Fig . 4 ) . Next , we detail each of the mechanisms that compose SAPE . The full algorithm is included in the appendix . Without loss of generality , in our description below we assume the encoding functionals are sorted by their respective Lipschitz constants ( LC ) , i.e . LC ( e1 ) ≤ LC ( e2 ) ≤ · · · ≤ LC ( en ) . For example , we sort the Fourier features encoding by the frequency value ‖bi‖ . We set the first d dimensions to the identity encodings : ei ( p ) = pi , ∀i ≤ d , thereby exposing the network to the original input position coordinates p ∈ Rd as well . Progressive encoding . A core part of SAPE is progressively fitting the frequencies . The first layer of the network encodes an input position p with a set of functionals ei . In order to use only part of the functionals at different stages of training , it multiplies the encoding dimensions with a soft mask , which progresses as a function of the optimization iteration t : Eprog ( p , t ) = ( ê0 ( p , t ) , ê1 ( p , t ) , ê2 ( p , t ) , . . . , ên ( p , t ) ) = α ( t ) > E ( p ) , ( 3 ) where êi ( p , t ) = αi ( t ) ei ( p ) is the progression control for the encoding functional ei , regulated by the elements of masking vector α ( t ) ∈ [ 0 , 1 ] n. During the optimization , we progressively reveal encoding features in a manner that allows the encoding neurons to be tuned to different states according to the value of αi ( t ) , where αi ( t ) = 1 corresponds to the state on , αi ( t ) = 0 means off and 0 < αi ( t ) < 1 denotes partially on ( Fig . 3 ) . We define a policy Φ that controls the progression of the mask vector α ( t ) by α ( t + 1 ) = Φ [ α ( t ) |α ( 0 ) ] . In this work , we use the following progression rule for the mask of the ith encoding dimension at time step t : ΦP [ α ( t ) ] : αi ( t+ 1 ) = clamp ( t− τ · ( i− d ) τ , 0 , 1 ) , if i > d 1 , otherwise , ( 4 ) where τ = T/2n represents the number of iterations each encoding dimension takes to progress from 0 to 1 and T is the maximal number of iterations in the optimization . Essentially , ΦP performs a linear progression sequentially for each encoding i until it is fully exposed , s.t . αi ( t ) = 1 . When a certain encoding dimension mask achieves saturation , we continue to the following one . We allow a technical exception to this rule , for proper progression of correlated masks of encoding dimensions sharing the same Lipschitz constant , e.g. , pairs of cos , sin in the encoding in Eq . ( 2 ) . For initialization , the first d positional encoding masks are fully exposed right from the beginning , and in our setting , correspond with the identity encoding functionals . Notice that previous methods , which introduce all encoding dimensions to the network at once , can now be formulated as a specific case in our framework , where α ( t+ 1 ) = 1 . Spatial adaptivity . When t > T/2 , the progression policy ΦP achieves global saturation in terms of revealing encoding dimensions , so that the masking vector becomes α ( t ) = 1 . For signals of global-like characteristics , e.g. , have a low LC , the network is compelled to learn how to cope with undesirable encoding dimensions of high frequency to regress a smooth , slowly changing signal . This can lead to sub-optimal outputs , often visible in the form of visual artifacts ( Fig . 1 , mid-bottom ) . Given loss function L and convergence threshold ε , a simple improvement to policy ΦP may apply early stopping when L < ε . However , for signals with varying levels of detail as a function of spatial position p , this improvement is not sufficient : early stopping does not occur due to areas characterized by high frequencies . Relying on a global threshold ε is inherently a sub-optimal decision , as when learning an implicit neural function with positional encoding , different spatial segments of the signal in question may converge at different rates ( Fig . 2 ) . To solve this problem , we set the mask vector to be spatially sensitive : α ( t , p ) . By design , the new policy ΦSA progresses the encoding of each spatial location p separately , per encoding dimension i . To simplify our explanation , in the current setting we assume the signal can be discretized to distinct spatial segments on a regular grid , each tracked independently ( e.g. , pixels , voxels ) . The progressive , spatially adaptive policy is controlled by a feedback loop , where the independent regression loss of each signal segment is used to control the progression : ΦSA [ α ( t , p ) ] : α ( t+ 1 , p ) = { ΦP ( α ( t ) ) , if L ( t , p ) ≥ ε α ( t , p ) , otherwise ( 5 ) For implicit functions , at inference time we assume signals can be regressed with “ pseudo-continuous ” coordinates p. Therefore , to support coordinates p that did not appear during training and have no mask recordings , we extend the estimated parameters of α ( T , p ) continuously over the entire input domain by a linear interpolation . One common example that benefits from spatial adaptivity is natural 2D image containing blurry , out-of-focus areas in the background together with sharp , detailed foreground objects . To shed more light on the behaviour of our algorithm , Figs . 6 and 8 demonstrate examples of spatial heat maps , highlighting the maximal encoding frequency achieved per spatial location , upon convergence . a . Forward mask b . Loss feedback Sparse grid sampling . We now extend our method to cases where model fθ is optimized by coordinates p that do not lie on a regular grid , or when it is simply infeasible to store parameters α ( t , p ) per sampled coordinate p during training . In this setting , we discretize the input domain and store parameters αu in a sparse grid G , where for a grid of resolution rd ⊆ Rd , we denote u as the multidimensional grid coordinates . In the forward pass , we obtain the original encoding mask α ( t , p ) per sampled point p by encoding a linear interpolation of the masking parameters over the nearest grid nodes ( see the inset ( a ) on the left ) : α ( t , p ) = ∑ u∈NG ( p ) wp , uα ( t , u ) , ( 6 ) where NG ( p ) are the neighboring grid coordinates in the vicinity of p , and wp , u are the interpolation weights for a sample p over the multidimensional grid G such that ∑ u∈NG wp , u = 1 . During training , the loss L ( t , u ) at time t for a given grid point u is accumulated over all sampled points p affected by u ( inset ( b ) above ) : L ( t , u ) = 1∑ p wp , u ∑ p wp , uL ( t , p ) . ( 7 ) Here , L ( t , p ) is the loss for the original coordinate p. To obtain the overall training loss , we sum over all the used points for training at time t. We note that the linear interpolation used during the forward pass has an added benefit : since only local neighbors participate per update of the original coordinate p , sparse weights allow for efficient accumulations during forward and backward pass updates .

Summary: Implicit neural representations are a promising approach to learning compressed representations of data --- for example they can learn to represent an image where the network inputs are the (x,y) coordinates and the outputs are the RGB pixel values. This idea has all sorts of applications in data science and physics.   This work provides a method for improving the optimization of these implicit neural representations. It addresses the challenge that standard MLPs are biased towards learning low-spatial frequencies, and that a dataset (e.g. an image to compress) can have different frequency distributions in different regions.