SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization

1 Introduction . Neural implicit functions have recently emerged as a powerful representation paradigm for modeling complex signals . Their continuous nature sets them apart from typical discrete representations ( i.e . pixels , voxels , meshes ) , allowing to capture high resolution details in various domains such as images [ 41 , 43 ] , 3d shapes [ 31 , 6 ] and radiance fields [ 30 , 32 ] , while retaining a reasonably compact representation . In this formulation , a deep neural network is trained with the goal of faithfully mapping input coordinates to a corresponding target domain , effectively learning the representation of signal properties such as magnitude , color , or shape occupancy . Implementing implicit neural representations with common neural structures , e.g. , multilayer pereceptrons with ReLU activations ( ReLU MLPs ) , proves to be challenging in the presence of signals with high frequencies . Consequently , recent works have demonstrated that deep implicit networks benefit from mapping the input coordinates [ 30 , 43 ] , or the intermediate features [ 41 ] to positional encodings . That is , before feeding them into a neural layer , they are first transformed to an overparameterized , high dimensional space , typically by multiple periodic functions . Positional encodings1 have been shown to enable highly detailed mappings of signals by MLP networks . For example , Fourier Feature Networks [ 43 ] suggested to map input coordinates of signals to a high dimensional space using sinusoidal functions . In their work , they show that the frequency 1In this paper , we use the term “ positional encodings ” in lower case letters to denote the family of encoding methods that map coordinates to a higher dimensional space . Not to be confused with the term “ Positional Encoding ” coined by [ 30 , 43 ] , which refers to a particular mapping scheme in this family . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of these sinusoidal encodings is the dominant factor in obtaining high quality signal reconstructions . In particular , they present compelling arguments for randomly sampling frequency values from an isotropic Gaussian distribution with a carefully selected scale , providing a striking improvement over mapping coordinates directly via standard MLPs . Despite the success of positional encodings , there are still some concerns left unaddressed : ( i ) Choosing the right frequency scale requires manual tuning , oftentimes involving a tedious parameter sweep ; ( ii ) The frequency distribution scale may change between different inputs , and accordingly it becomes harder to tune a “ one-fits-all ” model for signals that are composed of a large range of frequencies ( Fig . 1 ) ; ( iii ) Frequencies are selected for the entire input in a global , spatially invariant manner , thus missing an opportunity to better adapt to local high frequencies ( Fig . 2 ) . Our work investigates mitigations to the aforementioned challenges . We study the setting of positional encodings as input to implicit neural networks and present Spatially-Adaptive Progressive Encoding ( SAPE ) . SAPE is a policy for learning implicit functions , relying on two core ideas : ( i ) guiding the neural optimization process by gradually unmasking signal components with increasing frequencies over time , and ( ii ) allowing the mask progression to propagate at different rates among local spatial portions of the signal space . To govern this process , we introduce a feedback loop to control the progression of revealed encoding frequencies as a bi-variate function of time and space . Our work enables MLP networks to adaptively fit a varying spectrum of fine details that previous methods struggle to capture in a single shot , without involved tuning of parameters or domain specific preprocessing . SAPE excels in learning implicit functions with a large Lipschitz constant , without sacrificing quality of details or optimization stability , in problems that require meticulous configuration to achieve convergence . To highlight the latter , in Section 5.1 we present the tasks of 2D silhouettes deformation and 3D mesh transfer – both require stable optimization from the get-go in order to avoid convergence to sub-optimal local minima . SAPE is encoding-agnostic : it is not limited to a specific positional encoding type . It can be easily applied to the learning process of coordinate-based neural implicit functions of various domains including images , 2D shapes , 3D occupancy maps and surfaces , showing improvement in all of them . 2 Related work . Implicit neural representations . Recent works show the capability of deep neural networks in representing different functions ( e.g. , 2D/3D scenes or objects ) as an implicit , memory efficient continuous function [ 8 ] . These networks are used as a signed distance function ( SDF ) [ 1 , 15 , 31 ] or as an occupancy network , either binary [ 6 , 28 , 33 ] or soft volumetric density [ 26 , 30 , 49 ] . Several works train such networks to represent a collection of 3D shapes via 3D data supervision [ 6 , 16 , 29 , 28 , 31 ] , reconstruct 3D shapes [ 5 , 11 , 13 , 14 , 18 , 42 ] or infer them from 2D images [ 22 , 37 , 40 ] . Recent works employ spatial data structures to scale the represented shapes size [ 25 , 42 ] . Positional encodings ( PE ) . have been suggested as higher dimensional network input for various purposes . Radial basis function ( RBF ) networks [ 7 ] use weighted sums of embedded RBF encodings due to their symmetric property . [ 47 ] used random Fourier Features to map time spans . In natural language processing , Transformers [ 19 , 23 , 24 , 46 ] leverage sinusoidal positional encoding to maintain the order of token sequences . Our work differs by focusing on how to encode the inputs to MLPs in order to improve network implicit representations . The closest works to ours are SIREN [ 41 ] and Fourier feature networks ( FFN ) [ 43 ] . SIREN suggests to replace the ReLU activations in the network by periodic ones . FFN [ 43 ] encodes the inputs to the network by projecting them to a higher dimensional space using a family of encoding functions . In the appendix we provide an illustration of this approach with some examplary used encodings . The analysis of [ 43 ] demonstrates that FFN improves the learning of implicit neural representation compared to other alternatives . A follow-up work in [ 44 ] accelerates the training convergence by using a meta-learned initialization of the network weights . The contemporary work of [ 27 ] extends SIREN with a modulation network , to allow representing multiple shapes within a single network . Concurrently to our work , Park et al . [ 32 ] extend NERF to non-rigidly deforming scenes , and BARF [ 21 ] extends NERF for cases of imperfect camera poses . Both these works show the advantage of employing coarse-to-fine annealing linearly over the frequency bandwidth of the positional encoding . The coarse-to-fine approach bears similarity to our progressive frequency approach . Different from us , these works do not use a feedback loop or spatial encoding , which , as we show in the following , further closes the gap between the regressed function and the ground truth . As a final remark , SAPE ’ s concept of feedback loop resonates with the idea of `` predictive coding '' in computational neuroscience [ 35 ] , suggesting another angle to appreciate the rationale of our method . 3 Preliminaries . The setting of implicit neural representation is commonly formulated as learning a mapping function fθ , using a neural network , from an input of low dimensional position coordinates p ∈ Rd to an output value in the target domain . The network training data are coordinates samples as the input and an expected value of the function , or signal , as the output . Examples of such mappings include pixel locations to intensity values for images , or 3D space coordinates to binary surface occupancy labels . Achieving such mappings using conventional neural networks with ReLU is hard . It has been shown that ReLU networks exhibit a behaviour of spectral bias [ 34 , 3 , 36 , 2 , 48 , 4 ] . Networks tend to learn low frequency functions first , as they are characterized by a global behaviour , and are therefore more stable to optimize . Spectral bias , however , also prevents networks from properly learning to fit functions that change at a high rate , e.g. , functions with a large Lipschitz constant . When visual domains are concerned , this is mostly evident in delicate details missing from the network output . To mitigate this deficiency , recent works proposed to replace the ReLU activation layers with periodic sine functions [ 41 ] or map the input to some higher dimensional space in order to learn a mapping with high frequencies [ 30 , 43 ] . In the latter approach , the input p is encoded to a high dimensional embedding by a family of functionals ei : Rd → R , such that : E ( p ) = ( e1 ( p ) , e2 ( p ) , . . . , en ( p ) ) , ( 1 ) where usually n d. Tancik et al . [ 43 ] , for example , suggest the following encoding : EFFN ( p ) = ( cos ( 2πb > 1 p ) , sin ( 2πb > 1 p ) , ... , cos ( 2πb > np ) , sin ( 2πb > np ) ) , ( 2 ) where bi are frequency vectors randomly sampled i.i.d . from a Gaussian distribution with standard deviation σ . Other examples of positional encodings are included in the appendix . These embedding schemes facilitate learning of complex functions , at the price of introducing a second associated phenomenon : the spectral bias characteristic of the network is reduced . The implication in this case is twofold . Positional encodings can cause neural optimizations processes to become unstable , and consequently converge to a bad local minimum ( Fig . 9 ) . In addition , when fitting functions of varying levels of details/frequencies , neurons predicting smooth , low frequency areas are still exposed to encoding dimensions of high frequency . That , in turn , may complicate the learning process , as networks have to learn where to ignore such embedding dimensions . Indeed , Tancik et al . [ 43 ] advocate a careful choice of the standard deviation of frequency distribution , σ in their method , showing that using low values yields missing details in the network output , and using values that are very high results in noisy artifacts . Thereupon , for such static encodings the value of σ requires a parameter sweep per sample . 4 Spatially-Adaptive Progressive Encoding ( SAPE ) . Our proposed approach , SAPE , is a policy for guiding the optimization of implicit neural representations based on input coordinates . SAPE relies on the delicate balance of the two phenomena that govern the learning process : it reconciles the effects of the spectral bias and the expressiveness of high frequency encodings in a manner that benefits from both . It maintains a stable optimization , without sacrificing the ability to fit fine signal details . SAPE is composed of two key components : progressive encoding and spatial adaptivity , which allow it to be less sensitive to the encoding frequency bandwidth , i.e. , the choice of the standard deviation in the case of [ 43 ] ( Fig . 4 ) . Next , we detail each of the mechanisms that compose SAPE . The full algorithm is included in the appendix . Without loss of generality , in our description below we assume the encoding functionals are sorted by their respective Lipschitz constants ( LC ) , i.e . LC ( e1 ) ≤ LC ( e2 ) ≤ · · · ≤ LC ( en ) . For example , we sort the Fourier features encoding by the frequency value ‖bi‖ . We set the first d dimensions to the identity encodings : ei ( p ) = pi , ∀i ≤ d , thereby exposing the network to the original input position coordinates p ∈ Rd as well . Progressive encoding . A core part of SAPE is progressively fitting the frequencies . The first layer of the network encodes an input position p with a set of functionals ei . In order to use only part of the functionals at different stages of training , it multiplies the encoding dimensions with a soft mask , which progresses as a function of the optimization iteration t : Eprog ( p , t ) = ( ê0 ( p , t ) , ê1 ( p , t ) , ê2 ( p , t ) , . . . , ên ( p , t ) ) = α ( t ) > E ( p ) , ( 3 ) where êi ( p , t ) = αi ( t ) ei ( p ) is the progression control for the encoding functional ei , regulated by the elements of masking vector α ( t ) ∈ [ 0 , 1 ] n. During the optimization , we progressively reveal encoding features in a manner that allows the encoding neurons to be tuned to different states according to the value of αi ( t ) , where αi ( t ) = 1 corresponds to the state on , αi ( t ) = 0 means off and 0 < αi ( t ) < 1 denotes partially on ( Fig . 3 ) . We define a policy Φ that controls the progression of the mask vector α ( t ) by α ( t + 1 ) = Φ [ α ( t ) |α ( 0 ) ] . In this work , we use the following progression rule for the mask of the ith encoding dimension at time step t : ΦP [ α ( t ) ] : αi ( t+ 1 ) = clamp ( t− τ · ( i− d ) τ , 0 , 1 ) , if i > d 1 , otherwise , ( 4 ) where τ = T/2n represents the number of iterations each encoding dimension takes to progress from 0 to 1 and T is the maximal number of iterations in the optimization . Essentially , ΦP performs a linear progression sequentially for each encoding i until it is fully exposed , s.t . αi ( t ) = 1 . When a certain encoding dimension mask achieves saturation , we continue to the following one . We allow a technical exception to this rule , for proper progression of correlated masks of encoding dimensions sharing the same Lipschitz constant , e.g. , pairs of cos , sin in the encoding in Eq . ( 2 ) . For initialization , the first d positional encoding masks are fully exposed right from the beginning , and in our setting , correspond with the identity encoding functionals . Notice that previous methods , which introduce all encoding dimensions to the network at once , can now be formulated as a specific case in our framework , where α ( t+ 1 ) = 1 . Spatial adaptivity . When t > T/2 , the progression policy ΦP achieves global saturation in terms of revealing encoding dimensions , so that the masking vector becomes α ( t ) = 1 . For signals of global-like characteristics , e.g. , have a low LC , the network is compelled to learn how to cope with undesirable encoding dimensions of high frequency to regress a smooth , slowly changing signal . This can lead to sub-optimal outputs , often visible in the form of visual artifacts ( Fig . 1 , mid-bottom ) . Given loss function L and convergence threshold ε , a simple improvement to policy ΦP may apply early stopping when L < ε . However , for signals with varying levels of detail as a function of spatial position p , this improvement is not sufficient : early stopping does not occur due to areas characterized by high frequencies . Relying on a global threshold ε is inherently a sub-optimal decision , as when learning an implicit neural function with positional encoding , different spatial segments of the signal in question may converge at different rates ( Fig . 2 ) . To solve this problem , we set the mask vector to be spatially sensitive : α ( t , p ) . By design , the new policy ΦSA progresses the encoding of each spatial location p separately , per encoding dimension i . To simplify our explanation , in the current setting we assume the signal can be discretized to distinct spatial segments on a regular grid , each tracked independently ( e.g. , pixels , voxels ) . The progressive , spatially adaptive policy is controlled by a feedback loop , where the independent regression loss of each signal segment is used to control the progression : ΦSA [ α ( t , p ) ] : α ( t+ 1 , p ) = { ΦP ( α ( t ) ) , if L ( t , p ) ≥ ε α ( t , p ) , otherwise ( 5 ) For implicit functions , at inference time we assume signals can be regressed with “ pseudo-continuous ” coordinates p. Therefore , to support coordinates p that did not appear during training and have no mask recordings , we extend the estimated parameters of α ( T , p ) continuously over the entire input domain by a linear interpolation . One common example that benefits from spatial adaptivity is natural 2D image containing blurry , out-of-focus areas in the background together with sharp , detailed foreground objects . To shed more light on the behaviour of our algorithm , Figs . 6 and 8 demonstrate examples of spatial heat maps , highlighting the maximal encoding frequency achieved per spatial location , upon convergence . a . Forward mask b . Loss feedback Sparse grid sampling . We now extend our method to cases where model fθ is optimized by coordinates p that do not lie on a regular grid , or when it is simply infeasible to store parameters α ( t , p ) per sampled coordinate p during training . In this setting , we discretize the input domain and store parameters αu in a sparse grid G , where for a grid of resolution rd ⊆ Rd , we denote u as the multidimensional grid coordinates . In the forward pass , we obtain the original encoding mask α ( t , p ) per sampled point p by encoding a linear interpolation of the masking parameters over the nearest grid nodes ( see the inset ( a ) on the left ) : α ( t , p ) = ∑ u∈NG ( p ) wp , uα ( t , u ) , ( 6 ) where NG ( p ) are the neighboring grid coordinates in the vicinity of p , and wp , u are the interpolation weights for a sample p over the multidimensional grid G such that ∑ u∈NG wp , u = 1 . During training , the loss L ( t , u ) at time t for a given grid point u is accumulated over all sampled points p affected by u ( inset ( b ) above ) : L ( t , u ) = 1∑ p wp , u ∑ p wp , uL ( t , p ) . ( 7 ) Here , L ( t , p ) is the loss for the original coordinate p. To obtain the overall training loss , we sum over all the used points for training at time t. We note that the linear interpolation used during the forward pass has an added benefit : since only local neighbors participate per update of the original coordinate p , sparse weights allow for efficient accumulations during forward and backward pass updates .

Summary: This paper highlights a problem with Position Encoding models that use Fourier features: they're very sensitive to a bandwidth parameter that trades-off between reproduction of low- and high-spatial frequency features. Their solution is an iterative feedback algorithm that reconstructs images in a coarse-to-fine manner. They describe incremental improvements across-the-board, and large and impressive improvements in their 2D silhouette reconstruction task.