Signal Processing for Implicit Neural Representations
Authors: Dejia Xu
Peihao Wang
Yifan Jiang
Zhiwen Fan
Zhangyang Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Implicit Neural Representation
Signal Processing
Differential Systems

Abstract: Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks. Despite many successful applications, editing and processing an INR remains intractable as signals are represented by latent parameters of a neural network. Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR. In this work, we present a pilot study on the question: how to directly modify an INR without explicit decoding? We answer this question by proposing an implicit neural signal processing network, dubbed INSP-Net, via differential operators on INR. Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant to translation, while mathematically we show that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. With these two knobs, INSP-Net instantiates the signal processing operator as a weighted composition of computational graphs corresponding to the high-order derivatives of INRs, where the weighting parameters can be data-driven learned. Based on our proposed INSP-Net, we further build the first Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet. Our experiments validate the expressiveness of INSP-Net and INSP-ConvNet in fitting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting, and smoothening) as well as for high-level tasks on implicit fields such as image classification.

The idea that our visual world can be represented continuously has attracted increasing popularity in the field of implicit neural representations (INR). Also known coordinate- based neural repre- sentations, INRs learn to encode a coordinate-to-valu mapping for continuous multi-media data. Instead of storing the discrete signal values in grid of pixels voxels, INRs represent discrete data as samples of a continuous manifold. Using multi-layer perceptrons, INRs bring practical benefits to various computer vision applications, such as image and video compression and generative Signal Processing for Implicit Neural Representations
Dejia Xu"
Peihao Wang"
dejiachtexas.eet
peihasvang@uueeas.eedd
Yifan Jiang yifanjiang97eutexa. edu
Zhiwen Fan zhiwenfan@utexas edu
Zhangyang Wang
atlaswang@utexas.c edu
The University of Texas at Austin
https://vita group. github.ioo INSPP/
Abstract
Implicit Neural Representations (INRs) encoding continuous multi-media data via multi-layer perceptrons has shown undebatable promise in various computer vision tasks. Despite many successful applications, editing and processing INR remains intractable as signals are represented by latent parameters neural network. Existing works manipulate such continuous representations via processing on their discretized instance, which breaks down the compactness and continuous nature of INR. In this work, we present a pilot study on the question: how directly modify an INR without explicit decoding? ? We answer this question by proposing an implicit neural signal processing network. dubbed INSP-Net, via differential operators on INR. Our key insight is that spatial gradients of neural networks can be computed analytically and are invariant translation, while mathematically we show that any continuous convolution filter can be uniformly approximated by linear combination of high-order differential operators. With these two knobs, INSP-Net instantiates the signal processing operator as weighted composition of computational graphs corresponding to the high-order derivatives INRs, where the weighting parameters can be data-driven learned. Based on our proposed INSP- Net, we further build the first Convolutional Neural Network (CNN) that implicitly runs on INRs, named INSP-ConvNet. Our experiments validate the expressiveness INSP-Net and INSP-ConvNet in fitting low-level image and geometry processing kernels (e.g. blurring, deblurring, denoising, inpainting. and smoothening) as well as for high-level tasks on implicit fields such as image classification.
Introduction
3D shape representation
5
6 7
9 [00] ITI inverse problems
models [1 16
Despite their recent success, INRs are not yet amenable to flexible editing and processing as the standard images could do. The encoded coordinate-to-vall mapping is too complex comprehend and the parameters stored in multi-layer perceptrons (MLPs) remains less explored. One direction of existing approaches enables editing INRs by training them with conditional input. For example,
Equaa Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Sigral Processing
Multicaking Expicit Encoding
Precooing
Input
INSP-Naa
Ourpu
3
Net
Clokiicttion
Debler
Dosolse
Grometric
Preceeing
Illamin
Inpaining
Figure 1: An illustration of implicit neural signal processing. Given an INR representing digital signals, our INSP-Net is capable of direct signal processing without needing to explicitly decode it. Our model first constructs derivative computation graphs of the original INR and then generates linear combination of them into new INR. It can be later decoded into discretized forms such as image pixels. The framework is capable of fitting low-level image processing kernels as well as performing high-level processing such as image classification.
utilize conditional codes to indicate different characteristics of the scene including shape and color. Another main direction benefits from existing image editing techniques and operates on discretized instances of continuous INRs such as pixels or voxels. However, such solutions break down the continuous characteristic of INR due to the prerequisite of decoding and discretizing before editing and processing.
23 24 25
20
21
26
In this paper, we conduct the first pilot study on the question: how to generally modify an INR without explicit decoding? The major challenge is that one cannot directly interpret what the parameters in an INR stand for, not to mention editing them correctly. Our key motivation is that spatial gradients can be served as favorable tool to tackle this problem as they can be computed analytically, and possess desirable invariant properties. Theoretically, we prove that any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. Based on above two rationales, we propose an Implicit Neural Signal Processing Network, dubbed INSP-Net, which processes INR utilizing high-order differential operators. The proposed INSP-Net is composed of inception fusion block connecting computational graphs corresponding to derivatives of INRs. The weights in the branchy part are loaded from the INR being processed, while the weights in the fusion block parameters of the operator, which can be either hand-crafted or learned by the data-driven algorithm. Even though we are not able to perform surgery on neural network parameters, can implicitly process them by retrofitting their architecture and reorganizing the spatial gradients.
We further extend our framework to build the first Convolutional Neural Network (CNN) operating directly on INRs, dubbed INSP-ConNNet. Each layer of INSP-ConvNet is constructed by linearly combining the derivative computational graphs of the former layers. Nonlinear activation and normalization are naturally supported they are clement-wiss functions. Data augmentation can be also implemented by augmenting the input coordinates of INRs. Under this pipeline (shown in Fig. M. we demonstrate the expressiveness of our INSP-Net framework in fitting low-level image processing kernels including edge detection, blurring, deblurring, denoising, and image inpainting. We also successfully apply our INSP-Convile to high-level tasks implicit fields such classification.
Our main contributions can be summarized as follows:
We propose novel signal processing framework, dubbed INSP-Net, that operates on INRs analytically and continuously by closed-form high-order differential operator Repeatedly cascading the computational paradigm of INSP-Net, we also build a convolutional network, called INSP-ConvNet, which directly runs on implicit fields for high-level tasks.
We illustrate the advantage of adopting differential operators by revealing their inherent group invariance. Furthermore, we rigorously prove that the convolution operator in the continuous regime can be uniformly approximated by a linear combination of the gradients.
Extensive experiments demonstrate the effectiveness of our approach in both low-level processing (e.g. edge detection, blurring, deblurring, denoising, image inpainting, and smoothening) and high-level processing such image classification.
By saying mean the computation follows from analytical mathematical expression.

---

Networds
Network
Pinsteedder Denvative Network
Concol
MLP
Impul INR
k
Figure 2: The left image provides overview of our INSP-Net framework. Each layer combines the high-order derivative computational graphs of the original INR network. The right image illustrates the weight sharing scheme in calculating the derivative sub-networks.
Preliminaries: Implicit Neural Representation
Implicit Neural Representation (INR) parameterizes continuous multi-media signals or vector fields with neural networks. Formally, consider INR continuous function d Rm - E that maps low-dimension spatial/temporal coordinates to the value space For example, to represent 2D image signals, the domain of & is (Ir, y) spatial coordinates, and the range of t the pixel intensities. The typical use of INR to solve a feasibility problem where is sought to satisfy a set of constraints {C(#, aj} [1,)))} where C is functional that relates function & some observable quantities aş evaluating over a measurable domain Q2 C I This problem can be cast into an optimization problem that minimizes deviations from each of the constraints:
= arg min Σ
C 1219)1
1=1
For instance, we can let (ax,) a with Q2 (a: then our objective boils down
point-to-point supervision which memorizes signal into + 17 When functional C is combination of differential operators taking values in a point set, i.e.. C(a (()),
V'(x), ...), Vx E S2y. Eq. 24 Note that in this paper, without particular specification, the gradients are all computed with respect to the input coordinate æ. C can [12]. In practice of computer vision,
i objective to solving bunch of differential equations 28
also form integral equation system over some intervals 22
reconstruct signal by capturing sparse observations D {92,, a 1- from unknown function 2, and dynamically sampling a mini-batch from D to minimize Eq. ! to obtain a feasible Φ.
A handy parameterization of function + is fully-connected neural network, which enables solving Eq. M via gradient descent through a differentiable C. Common INR networks consist of pure Multi-Layer Perceptrons (MLP) with periodic activation functions. Fourier Feature Mapping (FFM) 2r places a sinusoidal transformation before the MLP, while Sinusoidal Representation Network (SIREN) 2X replaces every piece-wise linear activation with a sinusoidal function. Below we give unified formulation of INR networks:
① (x)) W n. ( o 2 (x),
(x = o, (W æ + b;),
where W; E R xdd, b; E RR4 the weight matrix and bias of the i-th layer, respectively, n is the number layers, and (-(-) an element-wise nonlinear activation function. For FFM architecture, o, sin(-) when i denotes the positional encoding layer 212 30 and otherwise o, ReLU(-).
For SIREN, a; = sin(.) for every layer 1,
3 Implicit Representation Processing via Differential Operators
Digital Signal Processing (DSP) techniques have been widely applied in computer vision tasks, such as image restoration [II] signal enhancement 32 and geometric processing 33 Even modern deep
Without loss of generality. here simplify & to be scalar field, i.e., the range of & is one-dimensional.

---

learning models are consisting of the most basic signal processing operators. Suppose we already ac- quire an Implicit Neural Representation (INR) $ R + R. now we are interested in whether we can run signal processing program on the implicitly represented signals. One straightforward solution to rasterize the implicit field with 2D/3D lattice and run a typical kernel on the pixel/voxel grids. However, this decoding strategy produces a finite resolution and discretizes signals, which is mem- ory inefficient and unfriendly to modeling fine details. In this section, we introduce a computation paradigm that can process an INR analytically with spatial/temporal derivatives. We show that our pro- posed method serves as a universal operator that can represent any continuous convolutional kernels.
3.1 Computational Paradigm
It has not escaped our notice that spatial/temporal gradients on INRs & can be computed analyt- ically due the differentiable characteristies of neural networks. Inspired by this, we propose an Implicit Neural Signal Processing (INSP) framework that composes a class of closed-form operators for INRs using functional combinations of high-order derivatives.
We denote our proposed signal processing operator by A built upon high-order derivatives. Given an acquired INR Φ we denote the resultant INR processed by operator A as Vi = A4 E + R. To evaluate point æ E II' of processed INR, we propose the following computational paradigm:
(x) == Aa = (#(z), v+(x)), V²Φ(x), V=
(3)
where II : M + IR can be arbitrary continuous functions, which can be either handcrafted or learned from data. To learn an operator A from data, represent II by Multi-Layer Perceptrons (MLP) with parameters 0. Here we can slightly abuse the notation of Ç to be flattened vector high-order derivatives without multiplicity since differential operators defined over continuous functions form a commutative ring. The input dimension of II depends on the highest order of used derivatives.
Suppose we compute derivatives up to K-th order, then M = >> (k++m-) (K [K+1] (+T))////)))))))
where (k+ the number of distinctive k-th order differential operators Intuitively, directional derivatives encode (local) neighboring information, which can have similar effects of convolution.
As we will show in Sec. 3.2 II can construct both shift-invariant and rotation-invarian operators, which introduces favorable inductive bias to images and 3D geometry processing. More importantly, we rigorously prove that Eq. B is also a universal approximator of arbitrary convolutional operators.
We note that (xx) as a whole can also be regarded as a neural network. Recall the architecture of & (xx) in Eq. 2 its k-th order derivative is another computational graph parameterized by W, and b, that maps to <*T((r). For example, the first-order gradient will have the following form:
(+) òn-1 o (2n-20 .
Φ1) (xx) o 22 Φ1 (xx C W1,
where di(v)= = W (W + 1 and (-) is the first-order derivative of o.()). Since d shares the weights with Φi. VΦ is represented by a closed-form computational network re-using the weights from Φ. which we refer to as the first-order derivative network. The higher-order derivatives should induce the derivative network of similar forms. Therefore, the processed INR V will have an Inception like architecture, namely, a multi-branch structure connecting the original INR network and weight-sharing derivative subnetworks followed by fusion layer II. We call the entire model (v/- A or Eq. 3 an Implicit Neural Signal Processing Network or an INSP-Net. Note that the only param- eters of IN INSP-Net 0 are located at the last fusion layer, and can be trained in an end-to-end manner.
We illustrate an INSP-Net in Fig. where the color indicates the weight-sharing scheme. A similar weight-sharing scheme is also adopted in AutoInt [e In practice, we employ auto dittediiferentiatioon in PyTorch (35) to automatically create such derivatives networks and reassemble them parallelly constitute the architecture of an INSP-Net. When inputting an INR, we load the weights of the INR to our model following the weight-sharing scheme, and then we obtain an INSP-Net, which implicitly and continuously represents the processed INR (zz). To effectively express high-order derivatives, we choose SIREN as the base model [28].
3.2 Theoretical Analysis
In this section, we provide theoretical justification for the design of our INSP-Net. We will focus on discussing the latent invariance property and the expressive power of INeP-Net.
"This is equal to the number of monic monomials over Th with degree k.

---

Translation and Rotation Invariance. Group invariance has been shown to be a favorable inductive that
bias for image [6 video [7 and geometry processing B8 It has also been
group invariance is an intrinsic property of Partial Differential Equations (PDEs) 39 40 Since
our INSP-Net built using differential operators, we are motivated to reveal its hidden invariance property to demonstrate its advantage in processing visual signals.
In this section, we only consider two transformation groups: translation group T(m) and the special orthogonal group S0 (a.k.a. rotation group). Elements T. E T(m) in translation group shift the function & by some offset v E IR.. The shifted function can be denoted as Ty (x) = o(x + v). Similarly, elements in rotation group perform a coordinate transformation on function & by a rotation matrix R (0m). The transformed function can be written as R(x) = $ ((Rz). Group invariance means deforming the input space of function first and then processing it via an operator equivalent to directly applying the transformation to the processed function. For a more rigorous argument, A is said to be translation-invariant if 4., E T(m),, V(n + v) = A( Tul (ar). Likewise, @
if Y E (m) we have V/ (Rz) A R (ax). Below we provide Theorem | to characterize the invariance property for our model.
A is
Theorem 1. Given function II: TR-M I. the composed operator (Eg. 3 can satisfy:
1. shift invariance for every II.
2. rotation invariance if II has the form: IIyy) = f(lllll) for some f ER- - R.
We prove Theorem in Appendix A Our Theorem I implies that operator A is inherently shift- invariant. This is due to the shift-invariant intrinsics differential operators as we show in the proof. Rotation invariance is not guaranteed in general. However, if one carefully designs II, it can also be achieved via our framework. Moreover, we also suggest feasible solution to constructing a omation-invariaal operator A in Theorem I In our construction, IT first isotropically pools over the squares of all directional derivatives, and then maps the summarized information through another scalar function f. We refer interested readers to 39 for more group invariance in differential forms.
Universal Approximation. Convolution, formally known as the linear shift-invariant operator, has served as one of the most prevalent signal processing tools in the vision domain. Given two (real-valued) signals and g, we denote their convolution as g * f = f * g. In this section, we examine the expressiveness of our INSP-Net (Eq, 3) by showing it can represent any convolutional filter. To draw this conclusion, we first present an informal version of our main results follows:
Theorem 2. (Informal statement) For every real-valued function g : R - R. there exists polynomial p(=1,.... .mm) with real coefficients such that (V) f can uniformly approximate g * by arbitrary precision for all real-valued signals f.
The formal statement and proof can be found Appendix B Theorem involves the notion of polynomials in partial differential operators (see more details in Appendix p(V) (V) in turn can be written as a linear combination of high-order derivatives of f (a special case of Eq. 3 when 11 is linear). The key step prove Theorem 2 is applying the Fourier domain. However, we note that functions obtained by the Fourier transform are generally complex functions. The prominence of our proof is that can constrain the range of the polynomial coefficients into the real domain, which makes implementable via a common deep learning infrastructure. The implication of Theorem i is that the mapping between convolution and derivative is as simple as a linear transformation. Recent works A1l 4 show the converse argument that derivatives establishes the equivalence between differential operator and convolution in the continuous regime. In our proof, k-th order derivatives correspond to k-th order monomial in the spectral domain. Fitting convolution using derivatives amounts to approximating spectrum via polynomials. This implies higher degree of polynomial induces closer approximation. Since v(V) is not difficult to be approximated by a neural network IIg, we can easily derive the next result Corollary K
approximation theorem on
be approximated via a linear combination of discrete convolution. Theorem
Corollary 3. For every real-valued function g, there exists neural network To such that V = AA
(Eg. 5 can uniformly approximate g x t by arbitrary precision for every real-valued signals t.
As we discussed in Theorem [ A are constantly shift-invariant. This means when approximating a convolutional kernel, the trajectory of A is restricted into the shift-invariant space. Moreover, we emphasize that INSP-Net is far more expressive than convolutional kernels since Ho can also fit any
nonlinear continuous functions due to the universal approximation theorem 44 55 46).
5

---

3.3 Building CNNs for Implicit Neural Representations
Convolutional Neural Networks (CNN) are capable of extracting informative semantics by only piling up basic signal processing operators. This motivates to build CNNs based on INSP-Net that can directly run on INRs for high-level downstream tasks. In fact, to simulate exact convolution, our
Theorem suggests simplify TTg linear mapping. Then our former computational paradigm Eq.
B is changed to:
v(xe == (VV)(()) (x) + 0 (x + 02 (2 + ... 0g (<)
(5)
where O2 E R (1+ are parameters of the operator p( V). We name this special case of Eq. 3 as INSP-Conv. One plausible implementation of INSP-Conv is to employ a one-layer MLP to represent 1lg. When A (V), INSP-Conv preserves both linearity and shift-invariance when evolving during the training. We propose to repeatedly apply INSP-Conv with non non-linearity to INRs that mimics CNN-like architecture. We name this class of CNNs composed by multi-layer INSP-Conv (Eq. 5) as INSP-ConvNet. Previous works 47 4
extracting semantic features from INR either lack local information by point-wisely mapping INR's ' intermediate representation to semantic space or explicitly rasterize INR into regular grids. To the best of our knowledge, it is the first time that one can run CNN directly on an implicit representation thanks to closed-formness of INSP-Net. The overall architecture of INSP-ConvNet can be formulated as:
ConvNet (x)) (4) a A (1-1) .o.............................. A 22
A ((x),
(6)
where a is an element-wise non- non-linear activation, L is the number of INSP-Net layers, and O is an input INR. We use the symbol to denote operator functioning, and to denote function composition. Due to page limit, we defer detailed introduction to INSP-ConVNet to Appendix
Related Work
4.1 Implicit Neural Representation
Implicit Neural Representation (INR) represents signals by continuous functions parameterized by multi-layer perceptrons (MLPs) [2X 27 which is different from traditional discrete representations (c.g., pixel, mesh). Compared with other representations, the continuous implicit representations capable of representing signals at infinite resolution and have become prevailing to be applied In addition, INR has
upon image fitting (28) image compression [
49 and video compressing [3]
been applied more efficient and effective shape representation 4
5
K
7
10
11], texture
mapping [50 [5]], inverse problems 112 2
!2! and generative models 15
19 20 21
22]. There are also efforts speeding up the fitting of INRs 52 and improving the representation
efficiency (53] Nowadays, editing and manipulating multi-media objects gains increasing interest
and demand (4 Thus, signal processing implicit neural representation essentially an important
task worth investigating.
4.2 Editable Implicit Fields
Editing implicit fields has recently attracted much research interest. Several methods have been proposed to allow editing the reconstructed 3D scenes by rearranging the objects or manipulating the shape and appearance. One line of work alters the structure and color of objects by conditioning Another direction involves
latent codes for different characteristics of the scene discretizing the continuous fields. By converting the implicit fields into pixels or voxels, traditional image and voxel editing techniques (55 [56] can be applied effortlessly. These approaches, however, are not capable directly performing signal processing on continuous INRs. Functa 5 can use latent code to control implicit funtions. NID (7 represents neural fields linear combination of implicit functional basis, which enables editing by change of sparse coefficients. However, such editing scheme suffers from limited flexibility. Recently, NFGP [58] proposes to neural fields for geometry processing by exploring various geometric regularization. INS 559 distills stylized features into INRs via the neural style transfer framework (60). Our INSP-Net that makes smart use of closed-form differential operators does not require neither additional per-scene fine-tuning nor discretization to grids.
2
20
21
26

---

Input Image
Filter
Canny
Prewitt
INSP-Net
Figure 3: Edge detection. We fit the natural images with SIREN and use our INSP-Net to process implicitly into new INR that can be decoded into edge maps.
INR Fitted Noicy
Mean Filter
MPRNet
INt
Target Image
20.14/0.60
20.09/0.61
20.51/0.66
24.0220.76
PSNR/SSIM
Figure 4: Image denoising. We fit the noisy images with SIREN and train our INSP-Net to process implicitly into new INR that can be decoded into natural clear images.
4.3 PDE based Image Processing
Partial differential equations (PDEs) have been successfully applied to many tasks image process- ing and computer vision, such as image enhancement [61] 62 63], segmentation 644 40 image
registration [65], saliency detection [666] and optical flow computation 667. Early traditional PDEs
are written directly based on mathematical and physical understanding of the PDEs (e.g.. anisotropic Variational design 70 start from an energy function describing the desired properties of output im- ages and compute the Euler-Lagrange equation derive the evolution equations. Learning- based attempts 440 66 build PDEs from image pairs based on the assumption (without proof) that PDEs could be written as linear combinations of fundamental differential invariants. Although might be feasible to let INRs solve this bunch of signal processing PDEs, one needs to per-case re-fit INR with an additional temporal axis, which is presumably sampling inefficient. The multi-layer structure appearing in INSP-Net can be viewed as unfolding network 72 73 of the Euler method to solve time-variant PDEs !4
diffusion 61. shock filter 32 and curve evolution based equations 688
69 70
methods 663 7
We elaborate this connection in Appendix D
5 Experiments
In this section, we evaluate the proposed INSP framework on several challenging tasks, using different combinations of II. First, we build low-level image processing filters using either band-crafted or learnable IT. Then, we construct convolutional neural networks with our INSP-ConvNer framework and validate its performance on image classification. More results and implementation details are provided in the Appendix
5.1 Low-Level Vision for Implicit Neural Images
For low-level image processing, we operate natural images from Set5 dataset 75 Set14 dataset 776 and DIV-2k dataset [77]. Originally designed for super-resolution, the images are diverse in style and content. Note that the unprocessed images presented in figures are the images decoded from unprocessed INRs.
Since our method operates directly on INRs, we firstly fit the images with INRs and then feed the INRs into our framework. The final output another INR which can be decoded into desired images. The training set of our method consists of 90 examples of INRs, where each INR is built on SIREN 28 architectures.
7

---

Blur Image
INSP-Net
Target Image
21.88/0.77
21.72/0,52
26.73/0.83
27.67/0.79
PSNR/SSIM
Figure 5: Image deblurring. We fit the blurred images with SIREN and train our INSP-Net to process implicitly into new INR that can be decoded into clear natural images.
Original Image
Box Filter
Gaussian Filter
INSP-Net
Figure 6: Image blurring. We fit the natural images with SIREN and train our INSP-Net to process implicitly into new INR that can be decoded into blurred images.
Edge Detection Since the edges correspond to gradients in the images, using gradients of INRs to obtain edges straightforward. 0, set to I while other coefficients are set to O. We provide visual
comparisons against Sobel filter (78) Canny detector 77 and Prewitt operator 80 in Fig. 3
Image Denoising For classical image denoising filters, we compare against the median filter and mean filter. We use MPRNet [8] as a learning-based baseline. The input noisy images synthesized using additive gaussian noise. Visual results are provided in Fig.
Image Blurring Image blurring is low-pass filtering operation. We provide visual comparison against classical filters including 3 x box filter and 3 x 3 gaussian filter. The target images used for training our INSP-Net are the results of the Gaussian filter. Visual results are provided Fig. C
Image Deblurring We compare the proposed method with both traditional algorithms (e.g., wiener filter 82) ) and learning-based algorithms(e..) MPRNet [B.). We synthesize blurry images using Gaussian filters. As shown in Fig. [ Wiener Filter produce severe artifacts and MPRNet successfully reconstructs clear textures. INSP-Net is capable of generating competitive results against MPRNet and outperforms the Wiener Filter.
Image Inpainting We conduct two kinds of experiments in image inpainting, to inpaint 30% random masked pixels or to remove the texts ("INSP-Net"). Comparison methods include mean filter, median filter, and LaMa [ LaMa is learning-based method using Fourier convolution for inpainting. As shown in Fig. mean filter and median filter partially restore the masked pixels, but severely hurt the visual quality of the rest parts. Also, they can not handle the text region. LaMa successfully removes the text and inpaint the masked pixels. Our proposed method largely outperforms the filter-based algorithms and performs as well the LaMa.
5.2 Geometry Processing on Signed Distance Function
We demonstrate that the proposed INSP framework is not only capable of processing images, but also capable of processing geometry. Signed Distance Function (SDF) 25 is adopted represent geometries in this section. We first fit an SDF from point cloud following the training loss proposed
28 Then we train our INSP-Net to simulate a Gaussian-like filter similar to image blurring. After Afterwards, we apply the trained INSP-Net to process the specified INR. When visualization, we use marching cube algorithm extract meshes from SDF We choose Thai Statue, Armadillo, and to demonstrate our results. Fig.
Dragon from Stanford 3D Scanning Repository
84
Xo 77
8

---

Inpat Image
Median
INSP-Net
Target Image
12.60N0.3
17.02/0.53
18.9919.63
20.4088888
23.07/0.76
PSNR/SSIM
26.9870.6
26.80N0.90
26.4180.88
23,29/0.73
33.4400.95
PSNR/SSIM
Figure 7: Image inpainting. We fit the input images with SIREN and train INSP-Net process implicitly into a new INR that can decoded into natural images. Note that LaMa requires explicit masks select the regions for inpainting and the masks are roughly provided.
INR Fitted SDF
SDF Smoothened by INSP-Net
Figure Left: unprocessed geometry decoded from an unprocessed INR. Right: smoothened geometry decoded from the output INR of our INSP-Net. Best view in a zoomable electronic copy.
exhibits our results on Thai Statue. Our method is able to smoothen the surface of the geometry and erase high- frequeeyyy details acting if low-pass filter. We defer more results Fig.
5.3 Classification Implicit Neural Representations
We demonstrate that the proposed INSP framework only capable to express low-level image processing filters, but also supports high-level tasks such as image classification. To achieve this goal, we construct 2-layer INSP-ConvNe them decomposes the INR via the differential operator and combines them with learnable II. We build another 2-layer depthwise ConvNets running on pixels the baseline for fair comparison, since it has comparable expressiveness to our NNSP-ConNNet in theory. We also build PCA + SVM method and an MLP classifier that directly classify INRs according to (vectorized) weight matrices.
The INSP-ConvNet consists of INSP-Net layers. Each of
We evaluate the proposed INSP-ConvNet on MNIST (28 x 28 resolution) and CIFAR- 10 (32 x 32 resolution) datasets, respectively. For each dataset, will firstly fit each image into implicit representation using SIREN [28]. Both experiments take 1000 epochs to optimize with AdamW optimizer
X and learning rate of 10-4 Results shown in Tab.
Accuracy MNIST
Depthwise CNN PCA + SVM
MLP classifier
INSP-ConvA
87.6%
11.3%
9.8%
88. 88.1%
CIFAR-10
59.5%
9.4%
10.1%
62.5%
Table 1: Quantitative Results of Image Classification. All methods except "Depthwise CNN" operate on the parameter of INR directly, while "Depthwise CNN" operates on images decoded from INR.

---

We categorize Depthwise CNN explicit method, which requires to extract the image grids from INRs before classification. PCA + SVM and MLP classifier working on the network parameter space can be regarded as two straightforward implicit baselines. We find that traditional classifiers can hardly classify INR on weight space due its nigh-dimensional unstructured data distribution. Our method, however, can effectively leverage the information implicitly encoded in INRs by exploiting their derivatives. As a consequence, INSP-ConVNet can achieve classification accuracy on-par with CNN-based explicit method, which validates the learning representation power of INsP-ConyNet.
6 Conclusion
Contribution. We present INSP-Net framework, an implicit neural signal processing network that is capable of directly modifying INR without explicit decoding. By incorporating differential operators on INR, we can instantiate the INR signal operator as a composition of computational graphs approximating any continuous convolution filter. Furthermore, we make the first effort to build a convolutional neural network that implicitly runs on INRs. While all other methods run on discrete grids, our experiment demonstrates INSP-Net can achieve competitive results with entirely implicit operations.
Limitations. (Theory) Our theory only guarantees the expressiveness of convolution by allowing infinite sequence approximation. Construction of more expressive operators and more effective parameterization for convolution remain widely open questions. (Practice) INSP-Net requires the computation of high-order derivatives which is neither memory efficient nor numerically stable. This hinders the scalability of our INSP-ConvNe that requires recursive computation of derivatives. Addressing how to reconstruct INRs in a scalable manner is beyond the scope of this paper. All INRs used in experiments are fitted by per-scene optimization.
Acknowledgement
Z. Wang is in part supported by an NSF Scale-MoDL grant (award number: 2133861).
References
[1] Emilien Dupont, Adam Goliński, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2003.03123, 2021.
[2] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34, 2021.
[3] Yunfan Zhang, Ties van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. arXiv preprint arXiv:2 2021.
[4] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7154-7164, 2019.
[5] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, Haggai Maron, and Yaron Lipman. Controlling neural level sets. Advances in Neural Information Processing Systems, 32, 2019.
[6] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5948, 2019.
[7] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
[8] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEF/CVÉ Conference Computer Vision and Pattern Recognition, pages 4460 2019.
[9] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEF/CVF international conference computer vision, pages 5379-5389, 2019.
10

---

[10] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 165-174, 2019.
[11] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In European Conference Computer Vision, pages 523-540. Springer, 2020.
[12] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer. 2020.
[13] Vincent Sitzmann, Semon Rezchikov, William Freeman, Joshua B Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. arXiv preprint arXiv:2106.026:: 02634, 2021.
[14] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. arXiv preprint arXiv: (tV:2//11.13699 13779, 2021.
[15] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5799-5809, 2021.
[16] Terrance VeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind. Unconstrained scene generation with locally conditioned radiance fields. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 14304-14313, 2021.
[17] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. Stylenerf: style-based 3d-aware generator for high-resolution image synthesis. arXiv preprint arXiv:2110.08985, 2021.
[18] Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Yu Liu. Gancraft: Unsupervised 3d neural rendering of minecraft worlds. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14072-14082, 2021.
[19] Quan Meng, Anpei Chen, Haimin Luo, Minye Wu, Hao Su, Lan Xu, Xuming He, and Jingyi Yu. Gnerf: Gan-based neural radiance field without posed camera. In Proceedings of the IEEL/CVF International Conference Computer Vision, pages 6351-6361, 2021.
[20] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 11453-11 11664, 2021.
[21] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33:20154-20166, 2020.
[22] Peng Zhou, Lingxi Xie, Bingbing Ni, and Qi Tian. Cips-3d: A 3d-aware generator of gans based on conditionally-indeeeedeett pixel synthesis. arXiv preprint arXiv:21 2 10.09788, 2021.
[23] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun- Yan Zhu, and Bryan Russell. Editing conditional radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5773-5783, 2021.
[24] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text- and-image driven manipulation of neural radiance fields. arXiv preprint arXiv:2112.05139, 2021.
[25] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[26] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. 2019 IFEF/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 5932-5941, 2019.
11

---

[27] Matthew Tancik, Pratul P Srinivasan, Ben Mildenball, Sara Keil., Nithin Ragha- van, Utkarsh Singhal. Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006. 10739, 2020.
[28] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[29] Jiequn Han, Arnulf Jentzen, and Weinan. Solving high-dimensional partial differential equations using deep learning, Proceedings of the National Academy of Sciences, 15(34)):500-- 8510, 2018.
[30] Ellen D Zhong, Tristan Bepler, Bonnie Berger, and Joseph Davis, Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks. Nature Methods, 18(2):17-1885, 2021.
[31] Dejia Xu, Yihao Chu, and Qingyan Sun. Moiré pattern removal via attentive fractal network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 472-473, 2020.
[32] Yaqian Xu, Wenqing Zheng. Jingchen Qi, and Qi Li. Blind image blur assessment based on markov-consttraing fem and blur entropy. In 2019 IEEE international conference on image processing (icip), pages 4519-4523. IEFE, 2019.
[33] Peng-Shuai Wang, Xiao-Ming Fu, Yang Liu, Xin Tong, Shi-L.in Liu, and Baining Guo. Rolling guidance normal filter for geometric processing. ACM Transactions on Graphics (TOG), 34(6))1-9,2011
[34] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14556-14565, 2021.
[35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[36] Jan J Koenderink and Andrea J van Doorn. Image processing done right. In European Conference on Computer Vision, pages 158-172. Springer, 2002.
[37] Hae-Kwang Kwang Kim and Jong-Deuk Kim. Region- Region-based shape descriptor invariant to rotation, scale and translation. Signal Processing: Image Communication, 16(1-2):87-93, 2000.
[38] Niloy Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. Symmetry in 3d geometry: Extraction and applications. In Computer Graphics Forum, volume 32, pages 1-23. Wiley Online Library, 2013.
[39] Peter J Olver. Applications Lie groups to differential equations, volume 107. Springer Science & Business Media, 2000.
[40] Risheng Liu, Zhouchen Lin, Wei Zhang, and Zhixun Su. Learning pdes for image restoration via optimal control. In European Conference on Computer Vision, pages 115-128. Springer, 2010.
[41] Bin Dong, Qingtang Jiang, and Zuowei Shen. Image restoration: Wavelet frame shrinkage, nonlinear evolution pdes, and beyond. Multiscale Modeling & Simulation, 15(1):606-66) 660, 2017.
[42] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pages 3208-3216. PMLR, 2018
[43] Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. Journal of Computational Physics, 399: 108925, 2019.
[44] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals and Systems, 1989.
[45] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks,
1991.
12

---

[46] Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[47] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[48] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes. arXiv preprini arXiv:21 2021.
[49] Brandon Yushan Feng and Amitabh Varshney. Signet: Efficient neural representation for light fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14224-14233, 2021.
[50] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4531-4540, 2019.
[51] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao
Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314, 2019.
[52] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-basse neural repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846-2855, 2021.
[53] Jaeho Lee, Jihoon Tack, Namhoon Lec, and Jinwoo Shin. Meta-lcarning sparse implicit neural representations. Advances in Neural Information Processing Systems, 34:1 1769-11780, 2021.
[54] Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you should treat it like one. arXiv preprins arXiv:2201. 22222.22224,,... 2022.
[55] James W. Hennessey, Wilmot Li, Bryan Russell, Eli Shechtman, and Niloy J. Mitra. Transferring image-based edits for multi-channel compositing. ACM Transactions on Graphics, 36(6), 2017.
[56] Jerry Liu, Fisher Yu, and Thomas Funkhouser. Interactive 3d modeling with a generative adversarial network. In 20/7 International Conference on 3D Vision (3DV), pages 126-134. IEEE, 2017.
[57] Peihao Wang, Zhiwen Fan, Tianlong Chen, and Zhangyang Wang. Neural implicit dictionary learning via mixture-of-exper training. International Conference on Machine Learning. pages 2261-222624. PMLR, 2022.
[58]
Guandao Yang. Serge Belongie, Bharath Hariharan, and Vladlen Koltun. Geometry processing with neural fields. Advances in Neural Information Processing Systems, 34, 2021.
[59] Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. arXiv preprint vxXiv:2244.09943 2022.
[60] Leon A Gatys, Alexander S Eeker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[61] Pietro Perona and Jitendra Malik. Scale-space and edge detection using anisotropic diffusion.
IEEE Transactions on pattern analysis and machine intelligence, 12(7):629-63 1990.
[62] Stanley Osher and Leonid I Rudin. Feature-oriented image enhancement using shock filters.
SIAM Journal on numerical analysis, 27(4)9919-94) 1990.
[63] Xue-Cheng Tai, Stanley Osher, and Randi Holm. Image inpainting using a tv-stokes equation. In Image Processing based on partial differential equations, pages 3-22. Springer, 2007.
[64] Zhouchen Lin, Wei Zhang, and Xiaoou Tang. Designing partial differential equations for image processing by combining differental invariants, 2009.
[65] Lars Hömke, Claudia Frohn-Schauf, Stefan Henn, and Kristian Witsch. Total variation based image registration. Image processing based on partial differential equations, pages 343-361. Springer, 2007.
13

---

[66] Risheng Liu, Junjie Cao. Zhouchen Lin, and Shiguang Shan. Adaptive partial differential equation learning for visual saliency detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3866-3873, 2014.
[67]
Adam Rabcewicz. Clg method for optical flow estimation based on gradient constancy assump- tion. In Image Processing Based on Partial Differential Equations, pages 57-66. Springer, 2007.
[68] Guillermo Sapiro. Geometric partial differential equations and image analysis. Cambridge university press, 2006.
[69] Frédéric Cao. Geometric curve evolution and image processing. Springer Science & Business Media, 2003.
[70] Bart M Haar Romeny. Geometry-driven diffusion in computer vision, volume 1. Springer Science & Business Media, 2013.
[71] Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259 268, 1992.
[72] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pages 396, 2010.
[73] Jialin Liu and Xiaohan Chen. Alista: Analytic weights are as good as learned weights in lista. In International Conference on Learning Representations (ICLR), 2019.
[74] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
[75] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In Proceedings of the British Machine Vision Conference, pages 135.1-135.10. BMVA Press, 2012.
[76] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pages 71 711-730. Springer, 2010.
[77] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[78] I Sobel. An isotropic 3x 3 image gradient operator, presentation at stanford ai project (1968), 2014.
[79] John Canny. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-698, 1986.
[80] Judith MS Prewitt et al. Object enhancement and extraction. Picture processing and Psychopic- torics, 10(1):15-19, 1970.
[81] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In CVPR, 2021.
[82] Saced V Vaseghi. Advanced digital signal processing and noise reduction. John Wiley & Sons, 2008.
[83] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lem- pitsky. desolution-robus large mask inpainting with fourier convolutions. arXiv preprint arxivv2/09.071661 2021.
[84] Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 303-312, 1996.
[85] Andrew Gardner, Chris Tchou, Tim Hawkins, and Paul Debevec. Linear light source reflectom- etry, ACM Transactions on Graphics (TOG), 22(3):749-758, 2003.
[86] Venkat Krishnamurthy and Marc Levoy. Fitting smooth surfaces to dense polygon meshes. In Proceedings of the 23rd annual conference on Computer graphics and inteructive techniques, pages 313-324, 1996.
14

---

[87] Greg Turk and Mare Levoy. Zippered polygon meshes from range images. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pages 311 311-318. 1994.
[88] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05001, 2017.
[89] Kathrin Schacke. On the kronecker product. Master's 's thesis, University of Waterloo, 2004.
[90] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[91] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang. Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mip for image processing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5769-5780, 2022.
15


Reviews:
[{"review_id": "_z7F07wK6zw", "reviewer": "Reviewer_1Ys6", "summary": "The paper proposes a method to perform signal processing of a signal represented via a neural field directly, without the need to first sample the underlying MLP. To this end, the authors propose to express operators on top of the neural field as functions of the first- and higher-order derivatives of the MLP. Though the output is *not* a new, monolithic MLP parameterizing the output of the operator, but instead a linear combination of derivative MLPs, the output nevertheless retains the \"resolution-invariant\" property of the underlying neural field representation. \n", "questions": "Could the authors comment on the effective (not theoretical) receptive field of the operators that can be parameterized by the proposed approach? In particular, SIREN is known to be a highly local INR: the directions in weight space to change the colors of pixels even a small distance apart are basically orthogonal (increasingly so with higher frequencies). It would seem that this \"local support\" also somehow impacts the receptive field of a convolutional operator expressed via the gradients of the SIREN?\n\nDo the authors have a way in mind in which they might lift the limitation that their approach requires a-priori fitting of all the neural fields? Note that I do not expect this to be the case, any insight is simply appreciated!\n\nMinor comments:\n- line 49: \"are loaded from the INR being processing...\" - this sentence does not make sense to me. Maybe being -> during?\n- line 51: \"even though we are not able to (perform) surgery\"\n", "limitations": "The processing remains expensive, in that the output of the proposes INSP-Net is not a single neural field, but rather, a linear combination of derivative operators, each of which is expensive to query. \n\nThe present effort still requires recovering all neural fields of all the signals upfront. \n\nThe authors certainly discuss these limitations, but I would have expected a \"limitations\" section where these issues are made explicit.\n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Originality\nThe paper is original and the proposed approach novel. To the best of my knowledge, no prior work on directly computing convolutional operators on top of neural fields without sampling exists. The treatment of how to approximate any shift-equivariant and certain group-invariant kernels as the linear combination of first- and higher-order gradients of a neural field is similarly novel. The authors propose and demonstrate a set of applications, ranging from image processing to a simple convolutional neural network, that have also not been tackled in this way before. All in all, this paper is highly original and was joyful to read.\n\n# Quality\nThe paper is of high quality. Figures are informative to a degree that a single pass through the paper already provides high-level understanding of the method. A detailed read-through yields cool insights, such as the derivation of universal approximation of any convolutional filter, though that is, as the authors remark, not a new result. The quality of the experiments is similarly high, with relevant experiments that show-case the use-case of the proposed method - in particular, I enjoyed the low-level image processing tasks and image inpainting.\n\n# Clarity\nThe paper is very clear, especially the figures serve to lay out the key contributions of the paper.\n\n# Significance\nThe practical significance of the proposed approach seems to be limited for now: First, it requires one to already have access to a set of neural fields fit to the signal at hand. Second, the cost of evaluating the resulting network can be rather prohibitive, as it requires evaluating the stack of all derivative networks to decode a single pixel.\n\nHowever, I am not overly concerned with the practical significance of the approach. In my opinion, the key significance of this paper lies in educating the neural field community about the fact that operators on neural fields can be expressed as functions of the derivative networks. Further, I am convinced that this paper will inspire work that will push this idea further, maybe eventually to the point where we will be able to compute functions of neural fields directly in the weight space.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M6iD6n_FP-M", "reviewer": "Reviewer_edqT", "summary": "This paper studies implicit (and continuous) neural representations (INRs) of data defined on discrete lattices. In particular, the authors present a way of performing different operations on these INRs directly without the need for \"decoding\" them by parametrizing these operators with learnable operators on these representations. The authors demonstrate these ideas on Image denoising, deblurring (and blurring), inpainting and classification.", "questions": "- In Eq. (1), did the authors mean \\arg\\min instead of \\arg\\max?\n- Notation is confusing, as the authors use m and M to denote several things: m is the dimension of the discrete input lattice; e.g. m=2 for images, and M denotes the number of discrete measurements of the continuous function. However, line 117 denotes $x\\in R^m$ - shouldn't it be $x\\in R^M$? At the same time, Eq. (3) then makes use of $M$ to define the dimension of the domain of the operator $\\mathcal A$, which is independent of $m$ and $M$ and depends on the order of the computed derivatives.\n- One of the main theses of this paper is that common signal processing operations can be instead defined on INRs, which the authors attempt to demonstrate. However, all of these operators (with the exception of edge detection\") are defined only as parametric functions learned from data in a supervised way - as opposed to the original way of defining these operators, \"from scratch\". This should be clarified, because while gradients can be obtained \"in closed form\", these operators cannot.\n- This reviewer believes that while the idea proposed in this work is nice, the results are far from satisfying: \n\ni) the denoising performance seems quite poor, and the 2 db difference with DnCNN looks very strange given the (inverse) difference in SSIM. Indeed, the INSP-Net image looks considerably more noisy. Moreover, while it's fine comparing with DnCNN, this method is by now 5 years old, and much better algorithm exist (see the methods used for comparison in [1] for a good subset of them). Generally speaking, the denoising performance does not look good and appears far below normal current methods (even below previous methods based on non-local means, dictionary learning, and more).\n\nii) Likewise, the deblurring performance is also not very satisfying: note the considerable noise introduced in the sky area shown on Fig. 5. The same holds for inpainting (very high levels of noise are introduced in the sky).\n\niii) Importantly, image processing results are just demonstrations and numbers are presented for only those individual examples - no numbers are reported over the entire dataset. \n\niv) I also appreciate the comparison on MNIST and CIFAR-10, but such low numbers are concerning. Recall that simple linear models (e.g. PCA +SVM) can better than 87%. \n\n- In line 230, the authors mention that \"nonlinear activations and normalizations are naturally element-wise functions\", but the authors never state what these are. What are they?\n- Throughout the paper, a supervised learning setting is never presented, and so labels are never defined. Thus, it's unclear what the authors mean by \"semantic labels y_i\" in line 235. Only later in the experimental section one understands what labels the authors are referring to.\n- I appreciate the result in Theorem 1, although it's novelty is very limited: indeed, differential operators are shift invariant, and any function defined on norms of vectors is rotation invariant. \n- It is unclear why the authors mention that the differential operators form a *wavelet* basis - do these operators satisfy the properties that wavelet must satisfy?\n- In line 204, the authors comment that their operators can simulate *exact* convolutions. This seems incorrect to me, as the exact equivalence holds for an infinite sequence, and thus only a finite approximation is possible in this framework.\n- Lastly, there are many errors that make reading quite difficult. A non exhaustive list (just the first page) includes:\n  - \"Implicit Neural Representations (INR) encoding continuous multi-media data...\" -> encodes?\n  - what are \"agnostic\" parameters? (line 4)\n  - \".. can be computed analytically and invariant to translation\" -> invariant to translation, line 11.\n  - \"presented\" -> represented (line 23)\n  - \"study to encoder\" -> study how to encode?\n  - what do the authors mean by \"parameter stored\" in line 33?\n\n\n\n[1] Tu, Zhengzhong, et al. \"Maxim: Multi-axis mlp for image processing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "limitations": "The authors did not address negative societal impacts nor the limitations of their contribution.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ This reviewer likes the general idea of this paper. While INR are becoming increasingly popular, it's not obvious how one can perform operations on these continuous elements that are similar to the discrete operations defined over lattices. This paper proposes the first solution to do this (to the best of my knowledge).\n+ The method is intuitive and demonstrated on a series of image processing tasks, from low to higher level.\n\n- While the general idea is intuitive, the text is not very clear. There are several typos and general English usage issues that are cumbersome while reading (see below).\n- While this demonstrates that these operations can be *approximated* on INRs, the quality of the solutions is not very satisfying. As a result, the advantages of working on the INR domain are not very appealing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B3bzwgmnmI", "reviewer": "Reviewer_qGk1", "summary": "The paper introduces a novel signal processing framework named INSP-Net that directly operates on INRs without explicit decoding. The INSP-Net can process INRs by combining high-order differential operators. The authors prove any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators and empirically validate the effectiveness of the proposed method across both low- and high-level vision tasks.", "questions": "See weakness.", "limitations": "The authors points out some future works without adequately discussing limitations and negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. The idea is novel. The proposed method attempts to directly process INRs by utilizing high-order differential operators. Theoretically, the authors prove continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. Besides, extensive experiments are conducted to validate the effectiveness across various vision tasks.\n2. The paper is well organized and easy to be understood.\n\nWeakness:\n\n1. From experiments, it seems that INSP-Net operating on INRs cannot exhibit superiority against convolution operating on pixels. For example, edge detection results of other filters seems contain more detailed textures than that of INSP-Net. In Figures 4 and 5, although the PSNR score is higher, the processed images still contain obvious noise or other artifacts. As for image classification, why is Depthwise CNN used as baseline rather typical CNN? \n\n2. There are some typos in the paper, e.g. Line 77 “sought to to”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A3BPjTcVr7J", "reviewer": "Reviewer_Uh51", "summary": "Motivated by recent advances in implicit neural representation (INR) for images, this paper seeks to develop an approach to modify learned INR *\"without decoding”*. Specifically, the paper proposes training an additional network which operates on the high-order spatial derivatives of the original INR previously fitted for some image. This extra neural network takes in the original INR prediction, as well as its high-order derivatives w.r.t. the input coordinates (hence the spatial derivative); the objective of this network depends on the specific task, and the paper discusses tasks such as denoising, deblurring, inpainting, and classification.", "questions": "* Why is the title about general signal processing but the paper only deals with images? \n* What happened to the geometric processing and the 3D shape shown in Fig. 1?\n* For Fig. 4 to 7, why not show the initial decoded images after fitting the INR on the corrupted images? How much does the extra network actually help?\n* Isn’t the INSP-Net with the “A” kernel just a fully connected MLP (Line 119)? Why would stacking multiple INSP-Net be called INSP-ConvNet? Is it because MLP is equivalent to 1D Convolution? By this logic, isn’t the original INR also a “ConvNet”?\n* What does “modify an INR without explicit decoding” mean? Doesn’t the proposed method need to decode the INR (Eq. 3) by evaluating the “acquired INR” on every single pixel “x”? Isn’t that decoding already?\n* If we need to evaluate every pixel “x” with the INR, before feeding it to the extra network, what benefit does it bring over simply using the decoded image?\n* Why not show the inference speed / network run-time? How much slower would it be due to computing the high-order derivatives?\n\nIn particular, it would be critical to elaborate the \"without decoding\" claim, clarify why the proposed method is not also decoding from the INR, and how the proposed INSP-ConvNet really is **convolutional** instead of just **multiple MLPs**. These would determine the validity of the main claims and conceptual contributions of this paper, no matter how good or bad the experimental results are.", "limitations": "The authors have not really addressed them. Discussion about at least the limitation of a potentially slower run-time should be expected in the revision.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n**Clear and reasonable motivations.**\nINR has achieved great success in accurately fitting images, and it is therefore well-motivated to explore possibly modifying the learned INR for further downstream tasks. It is also admirable that the paper points out the potential benefits of improving INR based on ideas from the field of digital signal processing\n\n**Variety of tasks.**\nThis paper presents a wide range of applications, such as edge detection, blurring, deblurring, denoising, inpainting, and classification. It is helpful for the community to be aware of the huge potential of INR of images.\n\nWeaknesses:\n\n**Experimental design.**\n\n* The title suggests a general framework of signal processing, and the teaser figure (Fig. 1) shows a 3D mesh and explicitly states “geometric processing”, but the paper is actually only about image processing. \n\n* The paper has no experiment on geometric processing on signed distance fields, despite the fact that DeepSDF [10] is one of the most prominent examples of successful INR applications. \n\n* The paper has no experiment on signal processing of time series signals (e.g. audio or speech), which is arguably the most representative data domain for the entire field of digital signal processing. The SIREN paper [27] that this paper is based on has covered audio signals, so it is not a foreign type of data for INR.\n\n* It is completely fine if a paper only focuses on images, but the title and its first figure should at least appropriately reflect the content and scope.\n\n**Validation rigor.**\n* The paper does indeed cover a lot of image processing applications, but it lacks sufficient rigor in validating the proposed approach.\nThe paper presents quantitative metrics (PSNR/SSIM) for five selected images only (Figures 4 to 7). Those provided in the supplement do not contain any quantitative metric, and even if they do, the paper should still provide average results across a large dataset to avoid cherry picking.\n* In the denoising/deblurring/inpainting experiments, the INR is first fitted against the noisy/blurry/masked image before then processed by the task-specific network. The results seem to suggest that this task-specific network can successfully denoise/deblur/inpaint from the corrupted input image. However, the paper does not show the decoded image from the original INR. It is well-known that INR itself can at least do denoising [30], so it is not clear how much the proposed training actually improves from the INR simply fitted against the corrupted input image.\n* In the classification experiment, it is not clear how only training a 2-layer depthwise CNN is a “fair comparison”. The paper does not provide the number of network parameters, nor does it explain why it does not consider training a small MLP.\n* Across all experiments, it is not clear what number k, or how many high-order derivatives, should one use. The paper presents no analysis on the impact of having more high-order derivatives. It is not even clear if one really needs high-order derivatives.\n\n**Literature review.** \\\n*Some related papers are not correctly interpreted.*\n* While discussing “existing approaches\" for “editing on INRs”, the paper mentions (Line 36 to 37) that “another main direction” and only references three works, [27], [28], and [29]. However, [28] and [29] are both from 2017, well before the recent introduction of INR, whose main characteristic is the “coordinate-to-value mapping” (Line 25). In fact, [28] makes no use of neural networks, and does not even mention the phrase “neural network” at all; [29] is about using GAN to generate 3D voxels, hardly related to INR either.\n* While introducing FFM (Line 89 to 98) and Eq. (2), the notation seems to suggest that “positional encoding” is merely a sine function (Line 97 to 98), and ignores the fact that FFM [30] introduces the Gaussian mapping and uses it by default.\n* Section 4.2 discusses the supposedly related works on “editing the reconstructed 3D scenes”. This paper only focuses on image processing, so the relevance of this subsection is really weak.\n\n*Some highly related papers are not included.*\n* Using the derivatives of INR has been explored by [a] (with similar computational graph discussion) in the context of faster INR-based rendering. \n* Using INR for downstream applications (e.g. classification) has been explored by [b].\n* Using INR for image and video compression has been also explored by [c] (with a PDE-based positional encoding) in the context of light fields.\n* [d] introduces a technique that significantly accelerates the fitting of INR on 2D images.\n* [e] introduces a technique for improving the representation efficiency of INR of images via reducing the parameter count.\n\n* These works contain far higher relevance to this paper, compared to the 7 referenced papers on 3D shapes (there’s no 3D shape in this paper) and the other 7 referenced papers about 3D GANs\n\n[a] Autoint: Automatic integration for fast neural volume rendering. Lindell et al. CVPR 2021. \\\n[b] From data to functa. Dupoint et al. ICML 2022. \\\n[c] Signet: Efficient Neural Representation for Light Fields. Feng et al. ICCV 2021. \\\n[d] Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tancik et al. CVPR 2021. \\\n[e] Meta-learning Sparse Implicit Neural Representations\" Lee et al. NeurIPS 2021. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept