Discretization Invariant Learning on Neural Fields
Authors: Clinton Wang
Polina Golland
Venue: Submitted to ICLR 2023 (2023)
Keywords: discretization invariance
neural fields
universal approximation
numerical integration
quasi-Monte Carlo

Abstract: While neural fields have emerged as powerful representations of continuous data, there is a need for neural networks that can perform inference on such data without being sensitive to how the field is sampled, a property called discretization invariance. We develop DI-Net, a framework for learning discretization invariant operators on neural fields of any type. Whereas current theoretical analyses of discretization invariant networks are restricted to the limit of infinite samples, our analysis does not require infinite samples and establishes upper bounds on the variation in DI-Net outputs given different finite discretizations. Our framework leads to a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. DI-Nets enjoy several desirable theoretical properties such as universal approximation of a large class of maps between $L^2$ functions with gradients that are also discretization invariant. DI-Nets can also be seen as generalizations of many existing network families as they bridge discrete and continuous network classes, such as convolutional neural networks (CNNs) and neural operators respectively. Experimentally, DI-Nets derived from CNNs are demonstrated to classify and segment visual data represented by neural fields under various discretizations, and sometimes even generalize to new types of discretizations at test time.

Under review as a conference paper at ICLR 2023
DISCRETIZATION INVARIANT LEARNING ON NEURAL FIELDS
Anonymous authors Paper under double-blind review
ABSTRACT
While neural fields have emerged powerful representations of continuous data, there is a need for neural networks that can perform inference on such data with- out being sensitive to how the field is sampled, property called discretization invariance. We develop DI-Net, framework for learning discretization invariant operators on neural fields of any type. Whereas current theoretical analyses of discretization invariant networks are restricted to the limit of infinite samples, our analysis does not require infinite samples and establishes upper bounds on the vari- ation in DI-Net outputs given different finite discretizations. Our framework leads to a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. DI-Nets manifest desirable theoretical properties such as universal approximation of large class maps between L functions, and gradients that are also discretization invariant. DI-Nets can also be seen as generalizations of many existing network families as they bridge discrete and continuous network classes, such convolutional neural networks (CNNs) and neural operators respectively. Experimentally, DI-Nets derived from CNNs are demonstrated to classify and segment visual data represented by neural fields under various discretizations, and sometimes even generalize to new types of discretizations test time. Code: supplementary materials (URL to be released).
INTRODUCTION
Neural fields (NFs), which encode signals as the parameters of a neural network, have many useful properties. NFs can efficiently store and stream continuous data (Sitzmann et al. 2020b Dupont al.
2022, Gao et al. 2021 Takikawa et al. 2022 Cho et al. 2022p. represent and render detailed 3D scenes at lightning speeds (Müller et al. 2022). and integrate data from a wide range of modalities
(Gao et al. 2022). NFs are thus an appealing data representation for many applications.
dataset of NFs have major limitations. The sampling-based approach converts such data to pixels or voxels as input to discrete networks (Vora) et al. 2021). but it incurs interpolation errors and does not leverage the ability to evaluate the NF anywhere on its domain. The hypernetwork approach trains a model to predict NF parameters (or a lower dimensional "modulation" of such parameters) which can be tailored for downstream tasks (Tancik et al., 2020aa Dupont et af., 2022 Mehta et al.] (021). but hypernetworks based on the parameter space one type of NF are incompatible with other types. Moreover, hypernetworks are unsuitable for important classes of NFs whose parameters extend beyond a neural network, such as those with voxel (Sun et al.] 202 Alex Yu and Sara Fridovich-Keil et al. 202T), octree (Yu et al. 202I) or hash table (Müller et al. 2022 Takikawa et al. 2022) components. We seek to strengthen the sampling-bast based upproach with the notion discretization invariance: the output of an operator that processes a continuous signal by sampling it at set of discrete points should be largely independent of how the sample points are chosen, particularly as the number of points becomes large.
However, current approaches for training networks
In this paper we propose the DI-Net, discretization invariant neural network for learning and inference on neural fields (Fig. [b. By parameterizing layers as integrals over parametric functions of the input field, DI-Nets have access to powerful numerical integration techniques that yield strong convergence properties, including a universal approximation theorem for a wide class of maps between function spaces. DI-Nets can be applied to any type of NF, in fact any data that can be represented as integrable functions on a bounded measurable set. Thus DI-Nets are a broad class of neural networks that encompass other continuous networks such neural operators, and also extend

---

Under review as a conference paper at ICLR 2023
DI-Net
(x.
Dataset
DI-Net
NF
label Classification
Numerical Integration
NE
Dense Prediction
NE
NF
NF
NF
non-linearities
TOOiE
NF
Generation
Figure 1: The DI-Net processes neural field by evaluating it on point set (discretization) which used to perform numerical integration throughout the network. DI-Nets are interoperable between all types of NFs and can be trained on a broad range of tasks.
discrete networks that act on pixels, point clouds, and meshes. They can be applied to classification, segmentation, and many other tasks.
Our contributions are as follows:
We show that discretization invariance gives rise to family of neural networks based numerical integration, which we call DI-Nets.
Backpropagation through DI-Nets discretization invariant, and they universally approximate large class of maps between function spaces.
DI-Nets generalize wide class of discrete models to the continuous domain, and we derive continuous analogues of convolutional neural networks for inference on neural fields that encode visual data.
We demonstrate convolutional DI-Nets NF classification and dense prediction tasks, and show it can perform well under a range of discretization schemes.
We probe the limits of discretization invariance in practice, finding that DI-Net has some ability to generalize to discretizations at test time, modulated by the task and the type of discretizations it was trained on.
RELATED WORK
Neural fields Multilayer perceptrons (MLPs) can be trained capture wide range of continuous data with high fidelity. The most prominent domains include shapes (Park et al. 2019 Mescheder 2018), objects (Niemeyer al. 2020: Müller et al. 2022), , and 3D scenes (Mildenhall et al.
et al.
2020
Sitzmann et 20211) but previous works also apply NF's to gigapixel images (Martel et al.
2021 9, volumetric medical images CorrnaaFFigeeron
al. 2022). acoustic data (Sitzmann al. Gao et al.) 2021). tactile data (Gao et al. 2022). depth and segmentation maps (Kundu et al.
20206
2022). and 3D motion [Niemeyer et al. 2019) Hypernetworks and modulation networks were developed for learning directly with NFs, and have been demonstrated on tasks including generative modeling, data imputation, novel view synthesis and classification (Sitzmann al. 2020b 2021
Tancik et al. 2020a Sitzmann et al. 2019 2020a Mehta et al. 2021 Chan et al. 2021] Dupont et al. 2021]2022), Hypemetworks use meta- meta-learnigg to learn to produce the MLP weights desired output NFs, while modulation networks predict modulations that can be used to transform the parameters of an existing NF or generate a new NE, Another approach for learning NF-NF maps evaluates an input NF grid points, produces features the same points via U-Net, and passes interpolated features through an MLP to produce output values at arbitrary query points (Vora et al. (2021).
Discretization invariant networks Networks that are agnostic the discretization of the data domain has been explored in several contexts. Hilbert space PCA, DeepONets and neural operators learn discretization invariant maps between function spaces (Bhattacharya et al. 2020 Lu et al.,
2021 Li al. 2020: Kovachki et al. 202lb) and are tailored to solve partial differential equations efficiently. On surface meshes, DiffusionNet (Sharp et al.. 2022) uses the diffusion operator to achieve convergent behavior under mesh refinement. These previous works define discretization invariance as convergent behavior in the limit of infinite sample points, but do not characterize how different discretizations yield different behaviors in the finite case. In this work, choose stronger
2

---

Under review as a conference paper at ICLR 2023
definition that bounds the difference between any two finite discretizations, which we show implies the convergence condition. We formulate discretization invariant networks on general metric spaces, which generalizes DeepONets and neural operators, then we focus on the continuous convolution as a core layer for vision applications.
Continuous convolutions At the core of many discretization invariant approaches is the continuous convolution, which also provides permutation invariance, translation invariance and locality. Its applications include modeling point clouds (Wang et al., 2021 Boulch, 2019), graphs (Fey et al. 2017), fluids (Ummenhofer et al. 2019), and sequential data (Romero et al. 2021b a). where there typically no choice of how the data should be discretized. This work focuses on the effect of different discretizations, proposes quasi-Monte Carlo canonical method of generating discretizations, and can produce neural fields as output.
Approximation capabilities of neural networks A fundamental result in approximation theory is that the set of single-layer neural networks is dense in large space of functionals including (R") (Hornik) 1991). Subsequent works designed constructive examples using various non-linear activations (Chen al. 1995 Chen & Chen [993). While this result is readily extended to multi- dimensional outputs, existing approximation results for the case of infinite dimensional outputs (e.g., (R)") +
["(R")) do not explicitly characterize the contribution of data discretization the
approximation error (Bhattacharya et al. 2020 Lanthaler et al. 2022 Kovachki et al. 2021b a).
PRINCIPLES OF DI-NETS
In this section we formalize discretization invariance, define DI-Nets, and derive properties that enable DI-Nets to serve as a general deep learning framework on continuous data.
3.1 NEURAL FIELDS AS FUNCTIONS
We treat NFs as integrable maps from domain Ω to R In particular, let Ω be a bounded measurable subset of d-dimensional compact metric space. The most common case Ω C Rd, and in the Appendix we consider the case of a d-dimensional manifold. Denote the space of NFs as F. [fe Ω - R Sa fel du < oo and ((fe) < oo), where the variation V ((fa) measures how much an NF fluctuates over its domain The variation of ID function f E (aa b)) is given by:
v(n - J. (x))
We call 0 the NF's parameters, d its dimensionality, and e its number of channels. For example, an occupancy network (Mescheder et al. 2018) is 3-dimensional and has I channel. NeRF (Mildenhall al. 2020 5-dimensional (3 world coordinates and 2 view angles) and has 4 channels.
and more general definitions are given in Appendix A.
The discretization of NF is point set X Ω which the field is evaluated. We say that map H: J. + 1" discretizable if it induces a map H. Fe + R. that depends only on the input's values at X
3.2 DISCRETIZATION INVARIANCE VIA NUMERICAL INTEGRATION
We consider notions of discretization invariance: (1) upper bounding the deviation of the map H.X from HY for any two discretizations X, Y. and (2) establishing convergence of H to H under particular sequences xXx NNE of discretizations. We use the first notion to define discretization invariant maps, then characterize sequences of discretizations under which such maps converge.
Definition 1. discretizable map H: F. + RR discretization invariant if for every discretization X and neural field fe. [[[fa] H. [fa]] ||, is bounded by the product of V (fe) and the discrepancy
'The parameterization of multi- perceptrons guarantees that NFs square integrable and of bounded variation over compact domain.
We assume that neural field evaluation yields pointwise values, i.e. the point spread function of the underlying signal is delta function. It is possible to accommodate non-trivial point spread functions, but this is beyond the scope of this work.
3

---

Under review as a conference paper at ICLR 2023
of X. A map H. Fe Fu discretization invariant if (a)) is discretization invariant for all I E Q
This definition establishes an upper bound on the deviation between any two discretizations by simple application of the triangle inequality. The discrepancy of a discretization is lower for dense, evenly distributed points. For a ID point set it given by:
d c b
D((
sup aSeSdeh
N
See Appendix A.1 for general definitions of discrepancy. The product of variation and discrepancy is precisely the upper bound in the celebrated Koksma-Hlawka inequality, which bounds the difference between the integral of function L2( and its sample mean on any point set X C Q2:
Σ Z'EX
h(x')-
| h(x) de (Vh)DD(X
X
This naturally leads to family of discretization invariant (DI) layers which specify a parametric map on neural fields and output its sample mean. Specifically, the action of DI layer Hs on a neural field la under discretization is:
2 So +
1
Σ Hafoo(z). FE
(4)
X
We propose two forms of Holf]:
Vector-valued DI layers (H FF RR") with Ha]fe](x) = (x(x)))()) E IR. Such layers include global pooling and learned inner products, and could be used as one of the final layers in classification network.
fo(x), NF-valued DI layers (H FF with Ho]fol(=): x' fo(c,=",fooss) for all x' E 2 Such layers include continuous convolutions and deconvolutions, self-attention, and pooling.
In each case, hg must be bounded and continuous in all its variables (so that outputs remain of bounded variation), differentiable w.r.t. á (to enable backpropagation), and Gateaux differentiable w.r.t. fe (which will make backpropagation discretization invariant, consider more general discretization invariant maps in Appendix A.2
discuss in Section 3.4). We
Importantly, the functional form of H3 does not depend on the NF parameters 9 except through fe(z).
Thus DI layers are invariant to the parameterization of the NFs that takes as input. This property allows these layers to be applied or modulation-base based learning approaches.
mixture of NF types, which is not possible with hypernetwork
Lastly note that many common loss functions and regularizers generalize naturally to the continu- ous domain as bounded continuous maps F. + R (e.g., L2 regularization) or F. x F. + IR (e.g.. mean squared error), the properties of DI layers extend to losses on NFs.
3.3 DISCRETIZATION INVARIANT NETWORKS
A DI-Net is a directed acyclic graph of the following types of layers:
DI layers: numerical integrators mapping NF to new NF or vector, defined above
Pointwise layers: a bounded continuous scalar function applied to each point in Ω; includes nonlinear activations, batch normalization, as well addition concatenation of NFs
Vector decoder: map T F. specified by elements of basis on F. or interpolation of n/c points (Appendix Ch
NNoe that the action of the layer produces neural field with parameters (0, σ). In Appendix how reparameterize NFs formed by sequence of DI layers control parameter size.
discuss
4

---

Under review as a conference paper at ICLR 2023
Since pointwise layers preserve an NF's property of bounded variation, DI-Net is discretization invariant. A prototypical DI-Net for classification might consist NF-vulued DI layers separated by normalization and activation layers, and end with vector-valued DI layer followed by softmax.
Grounding our network architecture in DI layers opens up a rich toolkit of numerical integration methods. We can generate low discrepancy discretizations using quasi-Monte Carlo (QMC), numerical integration method with favorable convergence rates compared to standard Monte Carlo (Caflisch) (998). The QMC discretization only requires a single pass through the network, can be deterministic or pseudorandom, and can accelerate computation when the same discretization is used for multiple network layers or all fields in a minibatch. To sample from non- non-uniform measures, the standard Monte Carlo method with rejection sampling can be used instead. To integrate over fixed discretization of high discrepancy, we can use a quadrature method that replaces | X in the sample mean with quadrature weights. Adaptive quadrature techniques can be used to attain specific error bounds at inference time, which can be valuable applications requiring robustness or verification.
3.4 CONVERGENCE UNDER EQUIDISTRIBUTED DISCRETIZATIONS
We call sequence of discretizations { NEN whose discrepancy tends to 0 as N + an equidistributed discretization sequence By Equation (3). DI layers converge under such sequences, i.e. 1im xx = Ha. and hence forward passes through DI-Nets are also convergent. But demonstrating convergence of DI-Net's discretized gradients less straightforward.
Consider a scalar-valued DI layer on a single-channel NF, Hyp F1 + RR. The derivatives of its output w.r.t. each of its weights Φe can be shown to converge as:
i HxN =
lim No OΦk
OΦb Hap
under any equidistributed discretization sequence {Xx] NeN.. Describing the derivative of the layer's output w.r.t. the input NF is more nuanced, since pointwise derivatives are not sufficient to represent backpropagation the continuous case. We must instead use the Gateaux derivative, which describes the linear change in map between functions given an infinitesimal change in the input function. We prove the following in Appendix B.I
Proposition 1. For every E Fi and fixed i Ωt {vE NNN which is I in a small neighborhood around ā and vanishes at each X MZ), such that:
design sequence of bump functions
lim N N
xxx
lim dHo v N
(6)
(f)
where dfHolfo; N is the Gateaux derivative when fo is perturbed in the direction of J
Using the chain rule for Gateaux derivatives, we can then show that backpropagation through the entire DI-Net is convergent. These properties can be summarized in the following theorem:
Theorem 1. DI-Net permits backpropagation of its outputs with respect to its input as well as all its learnable parameters. The gradients converge under any equidistributed discretization sequence.
3.5 UNIVERSAL APPROXIMATION THEOREM
Functions of bounded variation are piccewise smooth, hence they can be represented as the integral of some function. Since numerical integration yield arbitrarily small invariance error, DI-Nets are universal approximators in the following sense:
Theorem For every Lipschitz continuous map R F. C. E N, there exists a DI-Net that approximates it to arbitrary accuracy w.E.t. a finite measure on F.. As a corollary, every Lipschitz continuous map F. - Thd or R. - Fe can also approximated by some DI-Net.
A high-level sketch of the Fi + Fi case is as follows. Appendix B.2 provides full proof, including the extension multi-channel NFs.
AAy equidistributed sequence of points generates an equidistributed discretization sequence by truncating to the first N terms, although the class of all equidistrihuted discretization sequences is much larger than this. Quasi-Mom Carlo sampling can efficiently generate cquidistributed sequences wide range of domains.
5

---

Under review as a conference paper at ICLR 2023
1. Fix discretization X of sufficiently low diserepancy to approximate any function in Fi desired accuracy. Let = |
2. Let π be the projection that maps any function f E Fi IR by selecting its N values along the discretization. Through the measure on Fi induces measure ja on R
3. We can approximate any function in L2 (RR*) by covering the volume under the graph of the function with almost disjoint rectangles, and then inference time summing the heights of the rectangles at the given I. imput.
4. Note that a multilayer perceptron (MLP) can approximate this rectangle cover to arbitrary accuracy with sufficiently steep slopes at their boundaries (Lu al. 2017).
5. R Tf + (x) in L (IR for each z E X. We want to build a DI-Net that specifies f2 using element-wise products with cutoff' functions and linear
the desired connections combinations of channels. The cutoff' functions extract the input values along X into separate channels, and the weights of the channels match the weights of the hypothetical MLP from step 4.
6. We repeat this construction N times to specify values at each of the N output points in X, and map all other output points to the value of the closest specified point. Then we have fully specified the desired behavior of + R[f] to desired accuracy w.rt. the measure
This construction points the similarity between specifying a map between Fi - Fi and specifying maps between two grids of points. Thus many of the strategies for imposing structure on how different points influence each other can inform DI-Net design. For example, if the influence should be local and innllatioonivaiintt, we can design convolutional DI-Nets. If the influence should be sparse, we can design atlention-bian based DI-Nets. If the influence should be distributed globally among small patches, we can design transformer-lig like DI-Nets with tokenization. If the influence should be modulated by lower frequency patterns, we can use Fourier neural operators (Li et al., 2020).
4 DESIGN AND IMPLEMENTATION OF DI-NETS
DI-Nets encompass very large family of neural networks: we have only specified the architecture as a directed acyclic graph, and DI layers include a broad variety of network layers. DI-Nets include DeepONets and neural operators (Kovachki al. 2021b Lu al. (02T), which learn general maps between function spaces but in practice are designed to solve partial differential equations. DI-Nets also encompass continuous adaptations of networks designed on discrete domains such as convolutional neural networks (CNNs). In the same way that neural fields extend signals on point clouds, meshes, grids, and graphs compact metric space, DI-Nets extend networks that operate on discrete signals by converting every layer to an equivalent discretizable map. We make this connection concrete in the case of CNNs below.
4.1 CONVOLUTIONAL DI-NETS
We describe how to extend convolutional layers and multi-scale architectures to DI-Nets here (also see Fig. 2), and discuss other layers in Appendix C including normalization, max pooling, tokenization and attention. The resulting convolutional DI-Net can be initialized directly with the weights of pre-trained CNN as we investigate in Appendix E.I
Convolution For measurable C Ω and polynomial basis (p, 32o that spans I.2(S), S is the support of a polynomial convolutional kernel K3 Ω x Q + R defined by:
EE= (x - x')' i S 0
K (5 x')=
otherwise.
for some chosen n E N. A convolution the linear map Hos Fi + Fi given by:
Holf] = 1 (o(:) (((')'d'
(8)
An MLP convolution is defined similarly except the kernel becomes K2 2') MLP z': in the non-zera case. While MLP kernels are favored over polynomial kernels in many applications due to their expressive power (Wang et al. (02T), polynomial bases can be used to construct filters

---

Under review as a conference paper at ICLR 2023
Continuous Convolution
Multi-Scale Connections
Pre-trained
Downsampling
discrete weights
Interpolate
Upsampling
3x3 kernel
B-spline surface
Numerical Integration
Figure 2: Convolutional DI-Nets generalize convolutional neural networks arbitrary discretizations of the domain. Low discrepancy point sets used in quasi-Monte Carlo integration are amenable to the multi-scale structures often found in discrete networks. Convolutional DI-Nets may be initialized directly from pre-trained CNNs.
satisfying desired properties such as group equivariance (Cohen & Welling 201 continuity, or boundary conditions.
b). k-Liischitz
The input and output discretizations of the layer can be chosen independently. allowing for padding or striding (see Appendix C). The input discretization fully determines which points on S are evaluated.
Multi-scale architectures Many discretizations permit multi-scale structures by subsampling the discretization, and QMC is particularly conducive such design. Under QMC, downsampling is efficiently implemented by truncating the list of coordinates in the low discrepancy sequence to the desired number of terms, the truncated sequence is itself low discrepancy. Similarly, upsampling can be implemented by extending the low discrepancy sequence to the desired number of terms, then performing interpolation by copying the nearest neighbor(s) or applying some (fixed or learned) kernel. Residual or skip connections can also be implemented efficiently since downsampling and upsampling are both specified with respect the same discretization (Fig. right..
4.2 TRAINING DI-NETS
The pipeline for training DI-Nets is similar to that for training discrete networks, except that input and/or output discretizations should be specified. When training a DI-Net classifier, the input discretization may be specified manually sampled from a low discrepancy sequence to perform QMC integration. When training DI-Nets for dense prediction, the output discretization should be chosen match the coordinates of the ground truth labels. Any input discretization in most experiments we set it equal to the output discretization. At inference time, the network can be evaluated with any output discretization (Fig. D.2), making the output in effect an NE. We outline steps for training a classifier and dense prediction DI-Net in Algorithms and
be chosen
EXPERIMENTS
We apply convolutional DI-Nets toy classification (NF->scalar) and dense prediction (NF-NFF tasks, and analyze its behavior under different discretizations. Our aim is not to compete with discrete networks these tasks, but rather demonstrate that simply deriving DI-Nets from CNNs yields a feasible class of models for discretization invariant learning on NFs, without introducing any new techniques or types of layers. Appendix provides further experimental details. We discuss techniques that could be leveraged to design more competitive DI-Nets in Appendix F
5.1 NEURAL FIELD CLASSIFICATION
We perform classification on a dataset of 8,400 NFs fit to a subset of ImageNet (Deng et al. 2009). with 700 samples from each of 12 superclasses (Engstrom et al. 2019). For each class train
500 SIRENs (Sitzmann et al. 2020b) and evaluate on 200 Gaussian Fourier feature networks (Tancik al. 2020b).
We train DI-Nets with 2 and 4 MLP convolutional layers, as well as CNNs with similar architectures. We also train MLP that predicts class labels from SIREN parameters, and "non-uniform
7

---

Under review as a conference paper at ICLR 2023
convolution" (Jiag et al., 2019) that applies non-uniform Fourier transform to input points (sampled with QMC) to map them to gri values, then applies a 2-layer CNN.
Each network is trained for 8K iterations with learning rate 10-3, In training, the CNNs sample neural fields along the 32 32 grid. DI-Nets and the non-uniform network sample 1024 points generated from a scrambled Sobol sequence (QMC discretization). We evaluate mod- els with top-1 accuracy at the same resolution as well as several other resolutions.
43%
40%
DD-- Accuracy
35%
30%
z-layer
25%
4-layer DI-Net
layer CNN
The MLP and the non-uninn uniform method significantly un- derperform DI-Net, with 13.9% and 28.9% accuracy re- spectively compared to 32.9% for our 2-layer network. At 32 32 resolution, DI-Nets somewhat underperform their CNN counterparts, and this performance gap is larger deeper models. However, our discretization invariant model better generalizes to images of different resolutions than CNNs (Fig. 3 particularly at lower resolutions.
20%
4-yayer CNN
16a
Resalution
Figure 3: Classifier performance with
different resolutions at test time.
We next examine whether DI-Net can adapt ferent type of discretization at test time. We use grid, QMC and shrunk discretizations of 1024 (32 32) points. The Shrunk dis- cretization shrinks a Sobol (QMC) sequence towards the center of the image (each point [-1, 1| is mapped to =²))n(x)). In image classification, the object of interest is usually centered, hence the discretizations despite its higher discrepancy.
entirely dif-
Table Net under various discretizations.
Accuracy of 2-layer DI-
Train->Tes Disc.
Accuracy
QMC-QME Grid->Grid
32.9%
30.5%
setting performs on par with other
Srunk->Shrunk
30.3%
QMC++Grd
27.1%
Grid+
27.8%
Interestingly, changing discretization type at inference time has varying impact. Usually it only slightly degrades DI-Net's accu- racy (T. but performance falls dramatically when shifting from high to low discrepancy discretizations (shrunk-<QMC). Thus discretization invariance only provides weak guarantee on the stability of model's behavior, and points to the importance of training on the right discretizations to attain network that generalizes well to other discretizations for the given task.
QMC Chr-SShunkk
25.4%
Shrunk-QMM
13.4%
5.2 NEURAL FIELD SEGMENTATION
We perform semantic segmentation SIRENs fit to street view images from Cityscapes (Cordts et al. 2016), grouping segmentation labels into 7 categories. We train 2975 NFs with coarsely annotated segmentations only, and test on 500 NFs with both coarse and fine annotations (Fig. 4 We 48 x 96 grid discretization since segmentation labels only given at pixel coordinates. We compare the performance of 3 and 5 layer DI-Nets and fully convolutional networks (FCNs), as well
non-uniform CNN (Jiang al. 2019). We also train hypernetwork that learns map each SIREN to the parameters of new SIREN representing its segmentation.
Networks are trained for 10K iterations with a learning rate of 10-3 We evaluate each model with mean intersection over union (mloU) and pixel-wise accuracy (PixAcc).
Table 2:
Segmentation performance on NFs fit to Cityscapes images (trained coarse segs).
Coarse Segs
Fine Segs
Model Type
mIoU
PixAce
mloU
PixAce
The hypernetoork and non-umine non-uniform CNN perform poorly compared to both FCNs and DI-Nets (Table 2). DI-Net-3 outperforms the equivalent FCN, and less often confuses features such as shadows and road markings (Fig. 4.. However, the performance deteri- orates when downsampling and upsampling layers added (DI-Net-5), echoing the dif-
3-layer FCN DI-Net-3
0.409
69.6%
0.374
63.6%
0.471
78.5%
0.417
69.4%
5 5-layer FCN
0.488
79.4%
0.436
72.5%
DI-Net-5
0.443
77.7%
0.394
68.4%
Hypernetwork
0.038
7.9%
0.042
8.3%
Non-uniform
0.109
26.5%
0.106
22.7%
ficulty in scaling DI-Nets observed classification. We suggest potential methods for remedying this in Appendix
8

---

Under review as a conference paper at ICLR 2023
Input NF
DI-Net-3 (ours)
3-layer FCN
Ground Truth
Coarse
Fine
Figure 4: Cityscapes NF segmentations for models trained coarse segmentations only. NF-Net produces NF segmentations, which can be evaluated at the subpixel level.
5.3 SIGNED DISTANCE FUNCTION PREDICTION
We train a convolutional DI-Net to map field of RGBA values (SDF). We create a synthetic dataset of 3D scenes with randomly colored balls embedded in 3D space, and train on RGBA-SDD SDF pairs using a mean squared error (MSE) loss on the predicted SDF. We train with grid (16 x 16 x 16), QMC, shrunk, Monte Carlo (i..d. points drawn uniformly from the domain), and mixed discretizations of 4096 points. In the mixed setting, each minibatch uses one of the other four discretizations at random.
its signed distance function
Under any fixed discretization, the convolutional DI-Net significantly outperforms the equivalent discrete network (MSE of 0.022 0.067 respectively). In Figure D.2 illustrate our model's ability to also produce outputs that are discretized differently than the input, making DI-Net's output in effect neural field. By changing the output discretization of the last convolutional DI-Net layer, we can evaluate the output SDF anywhere on the domain without changing the input discretization. Whereas the discrete network is forced to output predictions at the resolution was trained on, convolutional DI-Net can produce outputs along a high-quality grid discretization given coarse QMC discretization, even when it is only trained under QMC output discretizations.
DI-Net performs almost equally well under grid, QMC and shrunk discretizations in the in- distribution setting, but on this task it is more sen- sitive to out-of-distribution discretizations than the classifier in Section 5.1 While shrunk QMCC is the worst performing combination for the classifier, here is one of the better performing combinations.
Table 3: Mean squared error (x10- 3) of pre- dicted SDFs under different discretizations. Top 3 settings bolded. MC=Monte Carlo.
Train
Test
Grid
QMC
Shrunk
MC
Grid
2.18
2.54
3.77
3.81
DI-Net likely struggles with Monte Carlo sampling due its high discrepancy discretizations, result- ing in cases where smaller balls are entirely missed. Interestingly, the model fares worse when trained on multiple discretizations simultancously. suggest-
QMC
3.60
2.01
2.94
3.92
Shrunk
3.72
2.88
2.00
4.30
MC
6.45
5.97
4.89
5.92
Mixed
4.65
4.41
3.26
4.09
ing that the network may be guided in opposing directions by difffernnt discretizations resulting in unstable training. These observations illustrate the complex task- lask-dependent interplay between the type of discretizations observed training time and the ability of the model to generalize to new discretizations.
CONCLUSION
DI-Net constitutes the first discretization invariant sampling approach for performing inference directly on neural fields. Motivated by discretization invariance, designed parameterization based on numerical integration that gives rise to strong convergence properties and universal approximation theorem for a wide class of functions. Not only is our framework agnostic to NF parameterization, but it also extends to functions of bounded variation over a wide class of domains, and thus can be applied to many systems that process continuous signal by querying on a subset of its domain. We outline several directions for enhancing such models in Appendix [F] which could enable them to scale to deeper architectures and tackle more challenging tasks. With the increasing popularity and diversity of neural fields well as the emergence of tools to efficiently create large datasets of NFs, DI-Nets may become an attractive option when interoperability and discretization invariance are desired.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Christoph Aistleitner and Josef Dick. Low-diserepancy point sets for non-uniform measures, 2013.
URL httpss/////axx wttp:/iw.orr//b//133333 abs/1308 5049
org/
Alex Yu and Sara Fridovich-Keil Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks, 2021.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. arXiv preprint arXiv:2005.03188) 2020.
Alexandre Boulch. Generalizing discrete convolutions for unstructured point clouds. CoRR, abs/1904.02375, 2019. htt.pp://aaxxl..orr//////1/1111/ 02375
I Luca Brandolini, Leonardo Colzani, Giacomo Gigante, and Giancarlo Travaglini. On the koksma-hlawka imequality. Journal of Complexity, 29(2):158-77,, 2013. ISSN 0885-064X. doi: ottp://o...rrr.00000...........0
URL
www http://wwwwcceeeeereee..ooo com/
Russel E Caflisch. Monte carlo and quasi-mont carlo methods. Acta numerica, 7:1-49, 1998.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein, pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 5799-5809, 2021.
Tianping Chen and Hong Chen. Approximations of continuous functionals by neural networks with application to dynamic systems. IEEE Transactions on Neural networks, 4(6)9900-918, 1993.
Tianping Chen, Hong Chen, and Ruey wen Liu. Approximation capability in c(r /sup n/) by multilayer feedforward networks and related problems. IEEE Transactions on Neural Networks, 6(1):25-30, 1995. doi: 10. 109/77227.323555
Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, and Funbyung Park. Streamable neural fields. arXiv preprint arXiv:2207.09665. 2022.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International confereeee on machine learning, pp. 2990-2999, PMLR, 2016a.
Taco Cohen and Max Welling. Steerable cnns. arXiv preprint arXiv:1612.084900 2016b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Abril Forona-Figueroa, Jonathan Frawley, Sam Bond-Taylor, Sarath Bethapudi, Hubert P. H. Shum, and Chris G. Willcocks. Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray, 2022. URL /ttp:::/axx.v..o/////////// wttp:://g//////.....///
Deng. Dong, R. Socher, L.-J. J. Li. Li, and L. Fei-Fei. ImageNet: Large- Scale Hierarchical Image Database. In CVPR09, 2009.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of functions. CoRR, abs/2102.04776, 2021. URL htL.tttps//axxvw.ocz//////////// 04776
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you should treat it like one. arXiv preprint arXiv:2201.12204, 2022.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL Atthttp:: /ttp:::///hhhb/ com. http//Mddrryya// haw.aabbrobuttteseee
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels. CoRR, abs/1711.08920, 2017. URL http:
10

---

Under review as a conference paper at ICLR 2023
Ruohan Gao, Yen- Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021.
Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jeannette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer, 2022. URL
attpss://ariiie org/ htwp/////////oooo//t//////aaa/e/ 2204. 02389
Kurt Hornik, Approximation capabilities of multilayer feedforward networks. Neural networks, 4(2): 251-257, 1991.
Chiyu "Max" Jiang, Dequan Wang, Jingwei Huang, Philip Marcus, and Matthias Nießner. Convolu- tional neural networks non-uniform geometrical signals using euclidean spectral transformation, 2019. URL /tttps://arxi.. org abs taaba/19011 1901. 00000000000000000 02070
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art-No, 202 la.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.088488 2021b.
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object- aware neural scene representation, 2022. URL
org/ abs 2205. 04334
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for DeepONets: deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1). 03 2022. ISSN 2398-4945. doi: 10. 10.1099//maam///ac000.1 1093/ mat ww..attmmmtttmmm Inac001
org/
tnac001.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint ar arXiv:2010008895, 2020.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. wwwpxivvorg//b... org/abs/
A convnet for the 2020s. CoRR, abs/2201.033 00354,, 2022. URL
2201.03544
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, 1.00010 2017. URL http///axvvvv.///// wwtp://wwwrg/ab//////////////// C5101
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):118 229, mar 2021. doi: 10011///2225-001-0000000 httpps:/dda http:///d...or......13323/2///// -21-22555-21--1111 2200:442222250000000-222---22--0 00302-5
URL
Zhou Lu, Hongming Pu, Feicheng Wang. Zhigiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
Julien NP Martel, David Lindell, Connor Z Lin, Eric Chan, Marco Monteiro, and Gordon Wetzstein. Acor:: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:210S.02288, 2021.
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14214-1422 2021.
Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Leariing 3d reconstruction function space. CoRR, abs/18 11s/1812.,388... 2018. URL http:://axiiiii http:/awwwwvv.rraa///a/aaaa///// 03828
11

---

Under review as a conference paper at ICLR 2023
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. CoRR, 0bss2203.08893,, 2020. URL / httec//ab//200333 28933
M.J. Muckley, R. Stern, T. Murrell, and F. Knoll. TorchKbNufft: A high-level, hardware-agnoste non- uniform fast Fourier transform. In ISMRM Workshop on Data Sampling & Image Reconstruction, 2020. Source code available at http://hww....oooooonn///oooohna
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. arXiv:2201 NDN::2201.05999, January 2022.
Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder: Autorf: Learning 3d object radiance fields from single view observations, 2022.
URL
wwwp////rrg/o/oooooooooo..o.s att///ss///22 2204. 03593
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. Proceedings of the IEEE/CVE international conference on computer vision, pp. 5379-5389, 2019.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. CoRR, abs/1901 hts/1901..5103. .05103, 2019. URL |Rt/htt:: htttt:::/axxii.org//eaa abs/ 05103
Charles R. Qi. Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet+:: Deep hierarchical feature learning on point sets in a metric space, 2017. https:///aa/ wttp//////////.//////////// 02413
David W. Romero, Robert-Jan Bruintjes, Jakub M. Tomczak, Erik J. Bekkers, Mark Hoogendoorn, and Jan van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. CoRR, abs/2110.0800 10.08059, 2021a. URL
org/ abs 2 2110..1.... 08059
David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kemel convolution for sequential data. CoRR, abs/2102.02611, 2021b. URL
http::://axxxv.orr//a///////
Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic learning on surfaces. ACM Transactions on Graphics (TOG), 41(3):1-16, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d -dwstrucctre--wware neural scene representations. CoRR, abs/1906.01618, 2019. URL
http:://rxiv..rx
990600011 016618
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-learning signed distance functions. CoRR, abs/2006. 096622 2020a. URL (RRLhhtps:
org/abs/ 2006. 09662
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. CoRR, abs/2006.09661. 2020b. URL htt.httsss//ccxiv.oro/// abs 2206 09661
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with single-evaluation rendering. CoRR, abs/2106.02634, 2021.
Cheng Sun, Min Sun, and Hwann- -zong Chen. Direct voxel grid optimization: fuper- con- vergence for radiance fields reconstruction, 2021. URL httpttttts/stii...oeaaa abs/211 11215
Towaki Takikawa, Alex Evans. Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate neural fields. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022.
12

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoc V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905. 1946, 2019. Sttp::///A....ooo.///////1...1.. 11946
org/ http://b//11000
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. CoRR, abs/2012.02189) 2020a. URL https::/aaxxl:l:// www.v..rg//as///122.... 2011
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. CoRR, abs/2006. 10739, 2020b, URL
https:/ parpps//arxiinn org/ abs// 2006. 10739
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference Learning Representations, 2019.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova. Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, 2021.
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. CoRR, abs/2101.06742, 2021. URL https://
0ttp://abb/2001007700
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
13

---

Under review as a conference paper at ICLR 2023
Appendix
Appendix A provides additional background on the variation of a function and discrepancy of point set. as well as general forms of DI layers. Appendix B provides proofs of the Univer- sal Approximation and Convergent Empirical Gradients theorems, as well as extensions of these properties from the single-channel case stated in the main text to multi-channel maps. Appendix C provides a detailed specification of DI-Net layers that enable DI-Nets to replicate the behavior of grid-based networks. Appendix D provides additional details of the data and architectures used in our classification experiment. Appendix D provides several analyses including properties of DI-Nets under different discretizations. Appendix describes limitations and directions for future work.
A MORE DETAILS ON DISCRETIZATION INVARIANCE
A. KOKSMA-HLAWK. INEQUALITY AND LOW DISCREPANCY SEQUENCES
Recall that a function f E L2(1)) satisfies a Koksma -Hlawaa inequality if for any point set X Ω
Σ [(x) / f(x) dx x'EX
X
for normalized measure dx, some notion of variation V of the function and some notion of discrepancy D of the point set. The classical inequality gives tight error bound for functions of bounded variation the sense of Hardy- Krause (BHH a generalization of bounded variation to multivariate functions on [0, 1] which has bounded variation in each variable. Specifically, the variation is defined as:
ga dra
VHK(f)=
Σ aEt0.11a 10.1]10|
f(xa)( dx,
(10)
(0.1j//
with {0, 1]d the multi-indices and Ta E (0. 1] such that Pajj I. if E and sa.j = otherwise. The classical inequality also uses the star discrepancy of the point set X, given by:
D' = sup IeJ
1,(x,) - x'
Σ
(11)
X
where J is the set of d-dimensional intervals that include the origin, and À the Lebesgue measure.
point set called low discrepancy its discrepancy on the order O(In N /))/N). Quasi-Monte Carlo calculates the sample mean using low discrepancy sequence (see Fig. A.l for examples in 2D), as opposed to the i.idd point set generated by standard Monte Carlo, which will generally be high discrepancy. Because the Koksma -lawka inequality sharp, when estimating the integral of BVHK function on [0, 1]d the error of the QMC approximation decays as O(In NY²/N), in contrast to the error of the standard Monte Carlo approximation that decays as O(N-1/2) (Caflisch 1998
Low discrepancy sequences
High discrepancy sequences
Sobol
(2.3)-Halton
I.I.D. samples
Uniform grid
Figure A.1: Examples of low and high discrepancy sequences in 2D.
However, BVHK is rather restrictive class of functions defined on [0, 1]d that excludes all functions with discontinuities, [Banndoliii et al. (2013) extended the MakksmaHHlwwe inequality to two classes of functions defined below:
14

---

Under review as a conference paper at ICLR 2023
Piecewise smooth functions Let f be smooth function on [0, 1]d and f Borel subset of [0, 1 Then fl is a piecewise smooth function with the Koksma- Hlawka inequality given by variation
00
v(f-
Σ aE(0,1}
20-1al
No.ja
f(x)
dz,
(12)
O
and discrepancy:
((X) = 2d
sup IC|0.1]
Σ zEX
(nnr(ay)) (
(13)
X
W functions on manifolds Let M be smooth compact d-dimensional manifold with nor- malized measure dz. Given local charts {ōr 1 Φk : [0, - M, the variation of a function f E Wd, (.11 ).l)iz is characterized as:
K
K v(f==)> = E
8a (VLO(()))))))) dæ,
(14)
k= (ai Sn [0.1]
with {1/x 1K smooth partition of unity subordinate to the charts, and constant c > 0 that depends
on the charts but not on f. Defining the set of intervals in Al as J {U U 6x (f) for some
and I C [0, ]] with measure u(U) X(1), the discrepancy of a point set Y {v, } Ml is:
D(Y) = sup UEJ
1 X
Σ ZEX
( -
(15)
Neural fields We define the variation of neural field as the sum of the variations each channel.
The notion of discrepancy is not limited to the Lebesgue measure. The existence of low discrepancy point sets has been proven for non-negative, normalized Borel measures on [0, 1] due to Aistleitner & Dick (2013). An extension of our framework to non-uniform measures is promising direction for future work (see Appendix
Note:
A.2 MORE GENERAL FORMS OF DI LAYERS
Recall that defined DI layers having the form Holf] = Jq H=(f(x))d for neural fields f (we drop 0 here for readability). In the case where Hols] Ω - R", i.e. Hg is layer that maps neural fields to vectors, we permit layers of the following more general form:
Holl = -10 f()
f(x), (*
(16)
gialy for multi-index The dependence of h on weak derivatives up to order k: |ox| requires that the weak derivatives are integrable, i.c., f is in the Sobolev space W and that ha is Gateaux differen- tiable w.r.t. these weak derivatives. Note that a non-uniform measure changes the discrepancy of sampled sequences.
for weak derivatives up order |ai| taken with respect each channel. D Zx ".
In the NF-valued case, we can similarly have:
Holf((z') / Holf((z,z')d((2)
(17)
= [ (()) D (x)), ,(d'), D (()(()))du(())
(18)
where we require HoLf](.,x)) F. Both of our key theoretical results (convergence of discretized gradients and universal approximation) apply to this general form. Gateaux differentiability of h& allows to apply the same proof in Appendix to the derivatives. Since allowing layers to depend on weak derivatives results in an even more expressive class of DI-Nets, the universal approximation theorem still holds.
15

---

Under review as a conference paper at ICLR 2023
B PROOFS
B. PROOF OF THEOREM (CONVERGENCE OF DISCRETIZED GRADIENTS)
A DI-Net permits backpropagation of its outputs with respect to imput as well as all learnable parameters. Under equidistributed discretization sequence, the gradients of each layer converge to the appropriate derivative under the measure on Q.
We note that this property automatically holds if the layer does not perform numerical integration. This includes layers which take IR" as input, as well as point-wise transformations. Then the (subaderrvatives with respect to inputs and parameters need only be well-defined at each point of the output in order to enable backpropagation.
Choose equidistributed discretization sequence {Xv Ω. We consider DI layer H.s which takes an NF f (we drop dependence on f for readability) as input and may output vector or NF.
Halll J. Holf(a))dr.
(19)
Recall we write the estimate under XN as:
P [
1
Σ Holf(()) EEXN
(20)
XN
and call its derivatives the discretized derivatives of Holf)- We are interested in proving the conver- gence of the discretized gradients of Hip [f] with respect to its input well the parameters Φ of the layer.
Definition
For given discretization X, the projection ग : + a quotient map (22 - L (02)~~ under the equivalence relation f ~ 9 iff f()) g(x) for all x E X.
Thus write π (fx when the discretization is clear from the context.
Lemma 1. For any DI-Net layer Hos which takes an NF f E Fm as imput, the discretized gradient of Holf If) w.t. its parameters convergent in N:
lim
H |i| <0.
(21)
Additionally, the discretized gradient of (o w.r.t. its discretized imput of convergent N:
lim V V H [f]l
<0.
(22)
Proof. NF Vector: gradients w.r.t. parameters
Consider the case of a layer Ho : Fm + R". I = (o.....,á)). then denote & + TEk =
(Ø1,.... ΦK 1, the T, T, (K))
a
lim
H [f] = lim
Σ Holf((o))
(23)
N +00
N Jør
[NN]
xEXx
N
= lim lim N T-1
H Hotrer [f(xx) Hollf(z))
(24)
TN
j=1
lim J Hottellf]( (f((x) - Holf((a) dx
(25)
7-1
Hotrey [ Holf
lim --0
(26)
T
agk Hollf,,
(27)
16

---

Under review as a conference paper at ICLR 2023
where (2) follows by (16) and the Moore-Ose -aooood theorem. Thus the discretized gradient converges to the Jacobian of Hy w.r.t each parameter, which is finite by differentiability and boundedness of hg.
NF to NF: gradients w.r.t parameters
From 17 stated that DI-Net layer Ho (St) (92) can be expressed as:
(1(a)) = Ja ((t)...... D, x'
D s(x'); (')))
(28)
We can in fact follow the same steps the NF to Vector above, arrive at:
lim N 0o'
Σ
Hol
Harlf).
(29)
[NN]] xEXx
00
with equality at each point x' E Ω and channel of the output NE.
NF input: gradients w.r.t inputs
Here we combine the NF to vector and NF to NF cases for brevity. For fixed 2 E Ω the discretized derivative of Hy w.r.t f(Ø) can be written:
0
(
Σ Holf(())
(30)
Of($)
0f(i) [NNl
zEXN
lim
1
Σ
Holf + + Holf(()),
(31)
5-0 T X
NE
where viA is any function in Wlll (52 that is I at H and 0 on XN' X {ē), and whose derivatives are 0 on XN- As an example, take the bump function which vanishes outside a small neighborhood of and smoothly ramps to
smaller neighborhood of ž, making its weak derivative 0 at ă.
By 3 we know that the sequences
[ - Holf and | +f T
Holf + TUA con-
verge uniformly in N to 0 for any > 0, where we can use the & norm for vector outputs or the L' morm for NF outputs. So for any > 0 and any > O. we can choose No large enough such that for
any N > No:
H2 [s + + - Hols + TUS /
(32)
and
ANDI - HaLlll <
(33)
Then,
N + + H6. + +W + | H660l <c.
(34)
by the triangle inequality,
(PE [f + Tv- Hol + + - (12 - Ho101 <c
(35)
|| (H + + ANN
(Hol5 + T H46l1)
<E,
(36)
hence (ANT 1 - AN
(fLL4 + r Hol
converges uniformly to 0. Since
the distance between two vectors is iff they are the same, we can write:
lim H If + - H. [f] = lim Holl + + - Holf
(37)
N-
N
lim lim T- N-500
H / -8
lim lim (Ho + T Hollf) T-+ N
(38)
17

---

Under review as a conference paper at ICLR 2023
By the Moore-Osgood theorem,
lim lim N-000 7-00
( + TWE 1 1 (f)) [f)
lim lim (Hoss + -vall Holl)
(39)
N r-0
lim N-+00
HNN [f] = lim θf(z)
[H
(40)
N-000
Since he is Gateaux differentiable and bounded, H& is also Gateaux differentiable for f of bounded variation, hence the limit on the right hand side is finite.
For each discretization Xv, choose a sequence of bump functions around each x E XN, { X An example of such family is the (appropriately designed) partitions of unity with [Xx] elements.
Then the discretized gradient w.r.t. Tf converges to the limit of the Gateaux derivatives of Hg w.r.t the bump function sequence as N - 00.
Lemma 2. Chained discretized derivatives converge to the chained Gateaux derivatives.
Proof. Consider a two-layer DI-Net with NF input f (Ha H.))ff. For the case of derivatives w.r.t. the input, we would like show the analogue of (40
lim Of(T)
HC)
[f]
lim d (Ho Ho) ( N
(41)
where the bump function di is defined similarly at ã and 0 each r # क).
HW o H
[f]
0f(r) He H [f] (z)
(42)
Of
1
Ho H If + TV (x) Ho [HNN [1] (x)
lim
(43)
7-0 T
as in 31
By 3 we know
Ho H [f
+
r
HN
H [f + T r converges to 0 in N for all
0 (where can use the & norm for vector outputs the L norm for NF outputs), as does
He H L/]
H N
Reasoning as in 32 (39). we have:
lim
lim
HNN
HA [f + T
Ab [2 LFl)
(44)
N T-2
lim
lim
(No AN If - +
Hos F LF)
(45)
No0
lim
lim
(Ho
( [s + -W
Ho A2 unl
(46)
No r-00
Note that
HHols:NN = (Holf + T 7 Holl o(T))
(47)
Holf + + = 7d7
(48)
18

---

Under review as a conference paper at ICLR 2023
Then complete the equality in 41
follows:
LHS
lim N-oo
HH o Hg [5]
(49)
lim lim
1
He H [ + v
Hy [#E 111)
(50)
N r-0 T
1
lim lim
(Ho [Hol/] TdHolls 11 Ho (Ho
(51)
N-000 7
lim dHla [Hol[]: dHLp[::
(52)
N-200
lim d (Ho Hop) [f: +1 N-
(53)
RHS,
(54)
by the chain rule for Gateaux derivatives.
The case of derivatives w.r.t. parameters is straightforward. the same way we used 32
39 to
obtain (46). we have:
lim -HD H No000 dΦr
= lim lim No000 T-0
(H HN [f(ttrell)] [1] - H.. [ 1/1]
(55)
lim r-0
(Ho (HotHoo [f]l Ho (H.llll
(56)
(Ho Ho)]f).
(57)
Jōk
By induction, the chained derivatives converge for an arbitrary number of layers.
Since the properties of DI-Net layers extend to loss functions on DI-Nets, we can treat loss function similarly to layer, and write:
Lgr
(58)
[g]
Σ
Llg.5((x)
(59)
[
zEXN
lim +0
θ 0f(*)
( H. Hg
[-]=
lim d (Lo Ho H4p [ v
(60)
N
where g' can be some other input to the loss function such ground truth labels. Thus, state the following result:
Corollary 1. The gradients of a DI-Net's loss function w.rl its inputs and all its parameters are convergent under an equidistributed discretization sequence.
B.2 PROOF OF THEOREM 2
(UNIVERSAL APPROXIMATION THEOREM)
Note: By our definition of F. (Section 3.1) there exists V such that every f E Fi satisfies Koksma-Hlawka inequality 3 with ( V*
Fi is bounded in 1 norm since all their functions are compactly supported and bounded.
Consider Lipschitz continuous map R Fi Fi such that d(R[s), R(g))) s Mod(f.g) for some constant Mo and all f.g E Fi. Let M max{{Mo, 1}.
Fix discretization X C Ω with discrepancy D(X)
1 By 3 this yields:
Σ f(x') / f(xx(( f(z)dz dx 12(M + 2)'
(61)
X
x'eX
for all f E Fi. Let N be the number of points in X.
19

---

Under review as a conference paper at ICLR 2023
Define the equivalence relation and projection as in Definition 2 R!Xi, and thus can be given the normalized e! norm:
L2())~~ is isomorphic to
|Tffle
Σ If(z')1
(62)
[X| xEX
Definition 3. Denote the preimage of π as π -1 f' + =F: = Invoking the axiom of choice, define the inverse projection x TFI + Fi by choice function over the sets π (TFFi.
Note that this inverse projection corresponds some way of interpolating the N sample points such that the output is in Fi. Although our definition implies the existence of such an interpolator, leave its specification as an open problem. Since Q only permits discontinuities along fixed Borel subset of [0. 1] these boundaries can be specified priori in the interpolator. Since all functions in Fi are bounded and continuous outside this set, the interpolator can be represented by bounded continuous map, hence it is expressible by DI-Net layer.
Definition 4. π generates a o-algebra on F1 given by o = (T S L), with L the a- algebra of Lebesgue measurable sets on RNN Because this o-algebra depends on and the Lipschitz constant R
R. via the point set's 's discrepancy, we may wrile il as A.R.
In this formulation, we let the tolerance and the Lipschitz constant of R dictate what subsets of Fi are measurable, and thus which measures on Fi are permitted. However, if the desired measure v is more fine- gra-gra than what is permitted by dR, then it is that should determine the number of sample points N, rather than e or R.
We now state the following lemmas which will be used prove our universal approximation theorem. Lemma 3. There is map R nFi FFF such that
([ Ro Ro (f) dr 6'
(63)
9
Proof.
Let g(x) = lf(x)) for f E Fi. Because 61
applies to g(z), we have:
Σ g(x') - 1 g(x) dr S 12(M + 2)
(64)
X
x'eX
|| - 11511 S
(65)
12(M 2)
Eqn. 65
also implies that for any f E TFI,, we have:
f le - 1| +flle 5
(66)
12(M + +2)
Combining (65) and (66), we obtain
| - ||= | x
< e
(67)
6(M 2)
By the triangle inequality and applying R:
J [R[[(=) - - oooo dr S 6(M + 2)
(68)
For any f.g E Fi such that / - Tg.
65 tells us that d(f,g) L is at most c/6(M + 2). Recall M Md(f.g)r: for any R.
was defined such that d(R[f), R(gl)r
Mc
d( R[f], To R(gl) < 6(M + +2)
(69)
6(M + 2)
(M 1)
(70)
(M 2) 6
20

---

Under review as a conference paper at ICLR 2023
So defining:
R arg min d( aTf], R(f))e.
(71)
*
we have
(M +11 c (M +2 6
R RoTif To Ro
1
(72)
Then by (68).
/ R
R[f((z) - -- Ro R dx 5
+
(M + 1)
(73)
6(M + 2)
(M + 2)
(74)
Lemma 4. Consider the extension of R to RN - R in which each component of the output has the form:
R[ (()x if aFFi R2(1) (00
10
(75)
otherwise.
Then any finite measure v on the measurable space (F1, of) induces a finite measure μ on (IR ,L)), and Sgy (((dd)) < xo for each j.
Proof. Since the r-algebra a on Fi is generated by æ. the measure μe: ((TT) v(S) for all S E a is finite and defined w.r.t. the Lebesgue measurable sets on T.FI. Since RF can be identified with measurable subset of R. l can be naturally extended to R.N. Doing so makes absolutely continuous w.r.t. the Lebesgue measure R.
To show R is integrable, it is sufficient to show it is bounded and compactly supported.
Fi is bounded in the L norm. Thus by (65) Fi is bounded in the normalized l norm. The & norm in R. is strongly equivalent the uniform norm, so there some compact set [-c, c) for which the extension of FJ to R vanishes, so supp(R,(F)) cl
Similarly, JFi bounded the norm, hence there exists c' such that R, c' for all
Lemma 5. For any finite measure p. absolutely continuous w.r. the Lebesgue measure on IR", E (p) and there is a network I. such that:
f ((()) ((f))-(()) ((())
(76)
Proof. The following construction is adapted from Lu et (2017). Since integrable, there is cube E [-c, ci" such that:
[ RRAE
[f())u(dd)
6 8
(77)
-V-LE-Il <
(78)
Case 1: J non-negative all of I.
Define the set under the graph of J g:
GE. A ((f.)) E,y [f))
(79)
GF,J is compact in 1=+1 hence there is finite cover of open rectangles (R. satisfying p(U, ((GE,)) on RR, Take their closures, and extend the sides of all rectangles indefinitely. This
21

---

Under review as a conference paper at ICLR 2023
results in a set of pairwise almost disjoint rectangles {R,). Taking only the rectangles R {R,
((R, n GE,J) > 0} results in a finite cover satisfying:
R
((R.) p(GE,)) <
(80)
This implies:
(RR,) < ||
(81)
and also,
LR ((d)) + ||-1|
(82)
JR"
[
2 / (f())
Σ 1=1
1 J(f))
(83)
by the triangle inequality. For each R; = [a bu] x Jean binl x ISe, Si + vel, let X; be its first components (i.c., the projection of R, onto RR). Then we have
R
J 1
Σ x
(84)
Let (f) & > Yi 1x, (f). By the triangle inequality,
/ (J(F) - K(FF) ((df) $ $|J-- -1g + 11g 111 + - Y
(85)
c < + -1-111 4
(86)
So remains to construct K. such that || - YI11 < Because 1 x, is discontinuous the boundary of the rectangle X,, it cannot be produced directly from DI-Net (recall that all layers are continuous maps). However, we can approximate it arbitrarily well with piece-wise linear function that rapidly ramps from 0 to the boundary.
by 78 and 84
For fixed rectangle X; and å E (0, 0.5), consider the inner rectangle X8 C X
X, = (a, 8(b, 0,). b, 8(b, a,)) x (an + 8(b, an), 5(bb an)), (87)
where we omit subscript for clarity. Letting Dy == b; 5(b; az), define the function:
T(f) II ReLu(8 - ReLU(f; - b)) - Re ReLU(f, - a.))],
(88)
where ReLU(x) = max 0). T(f) a pise linear function that ramps from at the boundary of X, to within Xs, and vanishes outside X,. Note that
||x - Th < 1 (X) - (xX,)
(89)
(XX))
(90)
if y is the Lebesgue measure. ò may need to be smaller under other measures, but this adjustment is independent of the input
be specified a priori.
Recall that the function we want to approximate Y(f) = Σ |z| 1x1x(ff) We can build NF-Net R layers K. f K(f)) E.a y: Ti(),, since this only involves linear combinations and ReLUs.
22

---

Under review as a conference paper at ICLR 2023
Then,
|/C - 1I1 =
Vi (Ti(f) 1 xx, (f)) df
(91)
[/2]
yell 3x|1ux - Till2
(92)
1=1
< <(1- (1 12-28))))
(xp
(93)
|R]
(1 (1 28)
>
4e (R.))
(94)
< (1- (1 26)")) (1J11, + 3)
(95)
by 81. And so by choosing:
c [1]] + 8) 7- 1/n
1
8 8=
1 - 1
(96)
2
4
we have our desired bound ||K- < and thereby ||J - KG1l < 2.
Case 2: J is negative some region of g
Letting (f) = max(0, (f)) and J (f) = max(0, max(00,_a(m)))) (f)), define:
G _ {f.. y) f E.y. e [0. (f)])
(97)
G A (ff. f E. (0. (f))
(98)
As in 80 construct covers of rectangles R+ over G and R over G each with bound and R" projections X+, X= Let:
R
Y+(l)=
Σ
V/
(f)
(99)
i=1
|
Y-(I)
Σ i=1
1/1 1x; (f)
(100)
Y -= --
(101)
We can derive an equivalent expression to
i+
((f)
Σ
u.
x+
(f) +
Σ
3/ 1 x,- (f) df
(102)
8 E
i=1
1=1
|| YII--
(103)
Similarly to earlier, we use
78 and (03 get:
(J()) K(()) di |KC Yll--
(104)
Choosing (P) and T, (f) the piece-wise linear functions associated with X,+ and X, and:
R T
x((f) = Σ y.
Σ TT (f),
(105)
1=1
23

---

Under review as a conference paper at ICLR 2023
have:
|n_
|K- 111
Σ
(T)
- -xx+()))
- Σ oi (T) xx-(1) dr.
(106)
==1
applying the triangle inequality,
R
R
yi
4 x
2
+ Σ yı 1x - -T.
(107)
1R+
(1 (1 20)
Σ
v (xxx
(1 (1 (1-28-)) Σ
yi (xxT))
(108)
(1 (1 - 20) (1]tt|12
16 + +(1 (1- (1- 28 y0) (115
16
(109)
By choosing:
1- 11(1- 1- (IJ- 1| 16) 1)) 1/n)
8+
(110)
2
1/n
8 - 2 1 1 8
1
(1 + 16
(111)
and proceeding as before, we arrive at the same bounds ||C YIl < and || K3I| < 5-
Putting all together, Algorithm implements the network logic for producing the function K.
Algorithm 1: DI-Net approximation of + J(f)
Setup;
Input:
target function J, L1 tolerance e/2
Choose rectangles R |a b x .... (a betl x 1st St + v1"1 satisfying (80 and R
similarly;
8+
1/2 (1 - $ (|]+ + 1tbo -))
2 (1 8 (IJJ-)) 1| + 16) - )1/11)
1
+
Inference;
Input: discretized input f = (fr )
1,
for rectangle R R+ do
for dimension k E 1::m do
a fn - a 23, Lt, 25): x +- (fa - bū + 8(b
ReLU(x);
z + (ō- (6-11.) 21,8 8 - 22 F3. 24. 25);
x ReLU(x);
+ 0, 23(0, x2) 22) 18,
end
23 + 25)3
end
for rectangle RT R do
for dimension E 1 n do
(fx by + 8(b
a fr
a 23, Lr, 23);
end
x <- (0, 0, 0, VE x3 + xs};
end
Output: 24 TT
24

---

Under review as a conference paper at ICLR 2023
We provide with access to f either through skip connections by appending channels with the values fc + fl Mk=1 (which will be preserved under ReLU).
Theorem 3 (Maps between Lingl-Chamme NEs). For any Lipschitz continuous map R. Fi + Fi, any >> O, and any finite measure V/ w.r.t. the measurable space (F1, H,,.) there exists a DI-Net T that satisfies:
J ((R()) - - <c.
(112)
Proof. If is not normalized, the discrepancy of our point set needs to be further divided by max (ax)) We assume for the remainder of this section that v is normalized. Perform the construction of Lemma times, each with tolerance of o eT/2NN K, where K is the Lipschitz constant of R. Choose a partition of unity { for which (2) = [at = arg min, d(x, x')], and By summing these channels we obtain network K that fully specifies the desired behavior of R EN EN, with combined error:
output N channels with the values (f),s
[1 -K(()/((())))))))) ((df) < 2
(113)
Thus,
J
Ro Ro R Ro v(df) s 2
(114)
X
x'Ex
By 666 we have:
Fi |/1 Tff((z) - n oo x(fl((()) do v(df) s
(115)
6(M + 2)
By Lemma we have:
/ / [I(((() -' -
oo <oo do v(df) S 2 +
+
(116)
6(M +
And thus the network = - o K
gives the desired bound:
f F
[R( - -(f))l v(ddf) <
(117)
Corollary 2 (Maps from NFs to vectors). For any Lipschitz continuous map R Fi - R" any e > O, and any finite measure v w.r.t. the measurable space (F1, A,RR), there exists a DI-Net T that satisfies:
f
- -Tfillln <e.
(118)
Proof. Let My be the Lipschitz constant of RR in the sense that R(g))e Mod(J,9) (1 Let
M max 1}. There exists R TF + R such that R o [ -R[[]]] S 1/12. As in
Lemma
consider the extension of R to NN + 1 in which each component of the output has the
form:
( [RT--1]] -f, if
R
(119)
0
otherwise.
Then for similar reasoning, / F, induces measure μ on RN that is finite and absolutely continuous w.r.t. the Lebesgue measure, and fgx (P)()) for each j.
25

---

Under review as a conference paper at ICLR 2023
We construct our R N
I approximation times with tolerance of e/ c/2m, such that:
JA.l1 ( p(df) c
K(())
(120)
Applying 665 we find that the network T K. gives the desired bound:
JA [R(()) (e) <c.
(121)
Corollary 3 (Maps from vectors to NFs). For any Lipschitz continuous map R. :R - Fi and any 0, there exists DI-Net T that satisfies:
[[((x) 2( <.
(122)
Proof. Define the map R. R² + TF C CRN by R = R. Since R. is bounded and compactly supported, JgN [R, (x) do < oo for each
We construct IR" + R approximation N times each with a tolerance of e/2NK with K Lipschitz constant, such that:
f [R()) k(() (-()) dr 2
(123)
Applying (66). we find that the network
T x o K gives us the desired bound:
f [R(xx) T T(x))l (x)))122000 <e.
(124)
Denote the space of multi-channel NFs F 15:00 R Salllll 0o, Si Fi for each i}. Denote the norm on this space
- i=1
(125)
Definition 5 (Concatenation). Concatenation is map from two NF channels fi, E Fi to [Si, [y] E Fz. The concatenation of NF's can be defined inductively yield Fn Fm 7 Fntm for any n. m E N.
All maps F, Fm -F can be expressed as a concatenation followed by map F Fn+m + Fe. map RD + F is also equivalent to maps R - Fi followed by concatenation. Thus, we need only characterize the maps that take one multi-channel NF as input.
Considering the maps F, -> F.. we choose lower discrepancy point set X on Ω such that the Klakaa inequality yields a bound of e/12mn(M + 2). Let ा project each component of the input to nFi, and inverts this projection under some choice function. We take af to be the product oallgerra generated from this n: o {Er x ... E. E,,..., , E. of where of is the o-algebra F, from Definition 4
Corollary 4 (Maps between multi-channel NFs). For any Lipschitz continuous map R. : Fr + Fm. any and any finite measure w.r.t. the measurable space (Fn, L.x)), there exists DI-Net T that satisfies:
[
v(d)) <c.
(126)
The proof is very similar to that of Theorem 3 Our network now requires n.N maps from RmN - each with error c/2mnN. Summing the errors across all input and output channels yields desired bound.
26

---

Under review as a conference paper at ICLR 2023
The multi-channel analogue of Corollary 2 is clear, and state it here for completeness: Corollary 5 (Maps from multi-channel NFs to vectors). For any Lipschitz continuous map R Fn - IRm, any e > O, and any finite measure V w.E. the measurable space (Fn, Rxx) there exists
DI-Net T that satisfies:
f F.
[[((f) -
(Rm <c
(127)
C PIXEL-BASED DI-NET LAYERS
Here we present variety of layers that show how to generalize pixel-based networks (convolutional neural networks and vision transformers) to DI-Net equivalents. Many the following layers were not directly used in experiments, and we leave investigation of their properties for future work. We use Cn to denote the number of channels of an input NF and Cout. to denote the number of channels of output NF.
Convolution layer The convolution layer aggregates information locally and across channels. has Cin Cout learned filters Kúj. which are defined on some support S which may be ball or orthotope. The layer also learns scalar biases by for each output channel:
Cin
9j
Kig fa + bb.
(128)
with * the continuous convolution as in
To transfer weights from discrete convolutional layer, K can be parameterized as a rectangular B-spline surface that interpolates the weights (Fig. left). To replicate the behavior of discrete convolution layer with odd kernel size, S is zero-centered. For even kernel size, we shift by half the dimensions of pixel. We use 2nd order B-spline for 3 x 3 filters and 3rd order for larger filters. We use deBoor's 's algorithm evaluate the spline at intermediate points.
Strided convolution is implemented by simply truncating the output discretization to the desired factor as described Section Different padding behaviors from the discrete case are treated differently. Zero-padding is replicated by scaling {z)((x) by [(S+z)n0) where + x is the kernel support $ S+r
translated by z. For reflection padding. the value of the NF at points outside its domain are calculated by reflection. For padding, the NF's domain is reduced accordingly, dropping all sample points that are no longer on the new domain.
Linear combinations of channels Linear combinations of channels mimic the function of x 1 convolutional layers in discrete networks. For learned scalar weights Wij and biases by:
Gim
g (I)
M W Wiff(()) + bū,
(129)
for all x E Ω. These weights and biases can be straightforwardly copied from 1 x 1 convolutional layer obtain the same behavior. One can also adopt normalized version, sometimes used in attention-base based networks:
Wiy
W
(130)
Wkj
Normalization All forms of layer normalization readily generalize to the continuous setting by estimating the statistics of each channel with numerical integration, then applying point-wise operations. These layers typically rescale each channel have some mean m, and standard deviation 8.
P4
(x)dæ
(131)
of
(1x)) dx - 11,
(132)
f((x) Hi
(x(x)
x s; + mi.
(133)
σi +
27

---

Under review as a conference paper at ICLR 2023
where we assume dr is normalized and > 0 is small constant. Just as in the discrete case, a4 and of can be a moving average of the means and variances observed over the course of training different NFs, and they can also be averaged over minibatch of NFs (batch normalization) or calculated per datapoint (instance normalization). Mean my and standard deviation 8; can be learned directly (batch normalization), conditioned on other data (adaptive instance normalization), or fixed at O and I respectively (instance normalization). These layers are not discretization invariant in the sense of Definition [] since the output can be poorly behaved for small values of o but the convergence condition still holds, i.e. normalization is convergent under an equidistributed sequence of discretizations.
Max pooling There are two natural generalizations of the max pooling layer to a collection of points: 1) assigning each point to the maximum of its nearest neighbors, and 2) taking the maximum value within a fixed-size window around each point. However, both of these specifications change the output's behavior as the density of points increases. In the first case, nearest neighbors become closer together pooling occurs over smaller regions where there is less total variation in the NF. In the second case, the empirical maximum increases monotonically as the NF is sampled more finely within each window. Because we may want to change the number of sampling points on the fly, both of these behaviors are detrimental.
If we consider the role of max pooling activation, then it is sufficient to use a fixed-size window with some scaling factor that mitigates the impact of changing the number of sampling points. Consider the following simplistic model: assume each point in a given patch of an NF channel is an i.i.d. sample from 4([1b, 6]). Then the maximum of N samples (f)()))) is on average N-1b. So we can achieve an "unbiased" max pooling layer N+1 by taking the maximum value observed in each window and scaling it by N-I (if N or our empirical maximum is negative then we simply return the maximum), then (optionally) multiplying constant to match the discrete layer.
layer that shuttles gradients through strong local
To replicate the behavior of discrete max pooling layer with even kernel size, shift the window by half the dimensions of a pixel, just as in the case of convolution.
Tokenization A tokenization layer chooses finite set of non-overlapping regions wy C Z of equal measure such that Ujwy = Ωt We apply the indicator function of each set to each channel ft. An embedding of each filw, into I can be obtained by taking its inner product with a polynomial function whose basis spans each L (wz,). To replicate a pre-trained embedding matrix, we interpolate the weights with B- B-spline surfaces.
Average pooling An average pooling layer performs a continuous convolution with box filter, followed by downsampling. To reproduce a discrete average pooling with even kemels,, the box filter shifted, similarly to max pooling.
An adaptive average pooling layer can be replicated by tokenizing the NF and taking the mean of each token to produce a vector of the desired size.
Attention layer There are various ways replicate the functionality of an attention layer. Here present an approach that preserves the domain. For some dy E N consider a self-attention layer with d parametric functions ęj E (1)), ean dg parametric functions kij E L (S2), and convolution with dy output channels, produce the output NF as:
Q (gij, fi)
(134)
(kijs Ji)
(135)
Vi
Σ
Mi fr bj
(136)
QKT
g softmax
Vlf((z)
(137)
Vdr.
A cross-attention layer generates queries from a second input NE. A aalliheaa-atteeniine layer generates several sets of (Q, K, V) triplets and takes the softmax of each set separately.
28

---

Under review as a conference paper at ICLR 2023
Data augmentation Most data augmentation techniques, including spatial transformations, point- wise functions and normalizations, translate naturally to NFs. Furthermore, spatial transformations efficient and do not incur the usual cost of interpolating back to the grid. Thus DI-Nets might be suitable for a new set of data augmentation methods such as adding Gaussian noise to the discretization coordinates.
Positional encoding Given their central role in neural fields, positional encodings (adding sinusoidal functions of the coordinates to each channel) would likely play an important role helping pixel- based DI-Nets capture high-frequency information under range of discretizations.
Vector decoders (R - F.) and parametric functions in F. A vector can be expanded into an NF in several ways. We can create an NF that simply places input values at fixed coordinates and produces values at all other coordinates by interpolation. Alternatively, we can define a parametric function that spans Fe using the input vector as the parameters, for example by taking as input numbers and treating them as coefficients of the first n elements an orthonormal polynomial basis on Ω. If Ω is subset of [a, bjd, one can use separable basis defined by the product of rescaled 1D Legendre polynomials along each dimension. If is d-ball, we can use the Zernike polynomial basis. For general coordinate system, small MLP could be used where IR" can represent its parameters a learned nower-dimensions modulation (Dupont et al. 2022) of its parameters. Beyond using such parametric functions vector decoder layers, they also give rise to n-parameter layers that compute an inner product ("learned global pooling layer") or elementwise product ("dense modulation layer") of input NF with the learned functions.
Warp layer Layers that apply a soaff-ommooorpiim on Ω (a bicontinuous map from Ω + Q2) preserve discretization invariance since it simply modifies the upper bound of the invariance error in subsequent layers to use a discrepancy of g(X) rather than discrepancy of X.
D EXPERIMENTAL DETAILS
D.1 ALGORITTMS
Algorithm 2: Classifier Training
Input: network Tp. NF label dataset D, classifier loss L, input discretization X for step $ 1: Nateps do NFs fs. labels < minibatch(? (D)
Label estimates Di Tolfi; X
Update 0 based on VoL W1)
end
Output: trained network To
Algorithm 3: Dense Prediction Training
Input: network To, dataset with dense coordinate- pairs, task-specific loss
for step s 1 Nsteps do
NFs fi, point labels (æij) Vis) + minibatch(D)
Output NF' y: [/]
Point label estimates 8ig t (x
Update / based on VoL Biy)
end
Output: trained network Te
D.2 DETAILS ON IMAGENET CLASSIFICATION
We split ImageNet into 12 superclasses (dog, structure/constructiion, bird, clothing, wheeled vehicle, reptile, carnivore. insect, musical instrument, food, furniture, primate) based on the big 12 dataset (Fingstrom et al. 2019), which is in turn derived from the WordNet hierarchy.
29

---

Under review as a conference paper at ICLR 2023
We fit SIREN (Sitzmann al. 2020b to each image in ImageNet using fully connected layers with 256 channels and sine non non-lineann linearities, trained for 2000 steps with an Adam optimizer at a learning rate of 10 4. It takes coordinates on [-1, 1] and produces RGB values in [-1, We fit Gaussian Fourier feature (Tancik et al.) 2020b) networks using 4 fully connected layers with 256 channels with ReLU activations. It takes coordinates on [0, 1]2 and produces RGB values in [0, 1]s.
The average pixel-wise error SIREN is 3 10 + 2 10 compared to 1.6 10-2 + 8 10-3 for Gaussian Fourier feature networks. The difference in quality visible high resolution, but indistinguishable at low resolution (Fig. D.I. DI-Net-2 uses strided MLP convolutions, global average pooling layer. then two fully connected layers. DI-Net-4 adds residual block with two MLP convolutions after the strided convolutions. We train all models with an Adam W optimizer
(Lohhchilov & Hutter 2017
Figure D.1: Comparison of SIREN (left) and Gaussian Fourier feature network (right) representations of an image, rendered at 256 x 256 resolution. The Fourier feature network's representation is slightly blurrier (compare the tree bark), but this effect is not noticeable at lower resolutions.
The architecture of the MLP is fully connected layers with 128 hidden units each and ReLU activation separated by batch normalization. It learns map the SIREN parameters to the class label. found that the model's loss curve becomes unstable after 3000 iterations so we reduce the number of iterations to 2000.
The non-uniform CNN applies the non-uniform Fourier transform (Muckley et al. 2020) followed by inverse Fast Fourier Transform to resample the input signal to the grid. It then feeds the result to 2-layer CNN to perform classification.
During training, we augment with noise, horizontal flips, and coordinate perturbations.
D.3 DETAILS ON CITYSCAPES SEGMENTATION
SIREN is trained on Cityscapes images for 2500 steps, using the same architecture and settings as ImageNet. Seven segmentation classes are used for training and evaluation, labeled in dataset as 'flat" (e.g. road), "construction" (e.g. building), 'object' (e.g. pole), 'nature', 'sky", 'human', and 'vehicle'.
DI-Net-3 uses two MLP convolutional layers at the same resolution followed by channel mixing (pointwise convolution). There are 16. 32 and 32 channels in the intermediate features. The support of the kernels in the MLP convolutional layers is .025 .05 and .075 .15 respectively, account for the wide images in Cityscapes being remapped to 1|
DI-Net-5 uses strided MLP convolution to perform downsampling and nearest neighbor interpolation for upsampling. There are channels in all intermediate features. There a residual connection between the higher resolution layers.
30

---

Under review as a conference paper at ICLR 2023
RGBA
True SDF
DI-Net (ours)
Discrete CNN
Figure D.2: 2D slices of toy 3D scenes with signed distance functions predicted by DI-Net and fully convolutional network.
The CNN baselines use 3x3 convolutions with the same number of layers and channels DI-Net. All networks use ReI.U activation and batch normalization.
The hypernetwork learns map from the SIREN RGB SIREN with the same architecture that represents the segmentation. It predicts changes to the weights of all layers before the final fully connected layer, and predicts raw values for the weights of the final layer since it has 7 output channels for segmentation instead of 3 for RGB.
The non uniform CNN applies the non-uniform Fourier transform followed by inverse Fast Fourier Transform, and feeds the result to the 3-layer FCN to perform segmentation.
D.4 SIGNED DISTANCE FUNCTIONS
In SDF prediction experiment, we construct toy scenes of 2-4 balls of random radii (range 0.2-0.5), centers, and colors scattered in 3D space (?2 = []1, For simplicity, we train each network directly on the closed form expressions for the RGBA fields and signed distance functions. rather than fitting neural fields first.
The FCN contains convolutional layers of kernel lengths and respectively, Accordingly, the convolutional DI-Net contained 2 convolutional layers followed by a linear combination layer. There 8 channels all intermediate features. We train each network for 1000 iterations with AdamW optimizer with a batch size of 64 and learning rate of 0.1 with an MSE loss on the SDF.
E
ADDITIONAL ANALYSIS
E. INITIALIZATION WITH DISCRETE NETWORKS
When DI-Net initialized with large pre-trained convolutional neural network, its outputs are identical by construction. However, the behavior of the pre-trained CNN is not preserved when the DI-Net switches to other discretizations even tiny perturbations from the regular grid are sufficient to change classifier's predictions. Although the effect on the output of single layer is much lower than the signal, small differences each layer accumulate to exert a large influence on the final output. Figure B.2 illustrates this phenomenon for DI-Net initialized with truncated EfficientNet. In addition, we find that once grid discretization abandoned, large DI-Nets cannot easily be fine-tuned to restore the behavior of the discrete network used to initialize it. This suggests that not only does the discretization at training time not necessarily permit new discretizations at inference time, but also that the optimization landscape of maps on T,2 //x R can vary significantly with X.
31

---

Under review as a conference paper at ICLR 2023
Table E.I: Pre-trained models fine-tuned ImageNet NF classification.
Model Type EfficientNet (Tan & Le 2019 DI-Net-EN
Accuracy
66.4% 48.1 1
Table E.2: Pre-tree ce-ttraned models fin-tuued on Cityscapes segmentation.
Model Type
Mean IoU
Pixel Accuracy
ConvNexT (Liu al. 2022)
0.429
68.1% 1%
DI-Net-CN
0.376
68.7 %
In Tables E.1 and E.2
illustrate that DI-Net initialized with large pre-trainn pre-trained discrete network does not match the performance the original model when fine-tuned with QMC sampling. We use truncated version of EfficientNet (Tan & Le 2019) for classification, and fine-tune for 200 samples per class. For segmentation we use a truncated version of ConvNexT-UPerNee (Liu et al. 2022). fine-tuning with 1000 samples.
We also find that the output of an NF-Net is less stable under changing sampling resolution with a grid pattern (Fig. E..). While the output of network with QMC sampling converges high resolution, the grid sampling scheme has unstable outputs until very high resolution. Only the grids that overlap each other (resolutions in powers of two) produce similar activations.
QMC
Grid
Output
Deviation
Output Deviation
322
642
1282
Sampling Resolution
Grid
Q QMC
Figure E.1: Distance of the output of a DI-Net from its grid output 32 32 resolution, when sampling at various resolutions. Its outputs de- viate rapidly its discretization shifts from regular grid to low discrepancy sequence
Figure E.2: An DI-Net's output diverges sam- ple points are gradually shifted from grid layout to a low discrepancy sequence.
Our preliminary experience with DI-Nets highlights the need for improved sampling schemes and parameterizations that will allow large continuous-domail neural networks to learn effectively. Stable, scalable methods needed realize DI-Nets' full potential for continuous data analysis.
E.2 COMPUTATIONAL COMPLEXITY
The DI-Net's complexity is similar to that of a discrete model with an equivalent architecture. In general time and memory both scale linearly with the number of sample points (regardless of the dimensionality of Q),, well as with network depth and width.
Implemented naively, the computational cost of the continuous convolution is quadratic in the number of sample points, it must calculate weights separately for each neighboring pair of points. We can reduce this to a linear cost by specifying a Nbin Voronoi partition of the kernel support B, then using the value of the kernel each seed point for all points in its cell. Thus the kernel need only be evaluated Npin times regardless of the number of sample points. Additionally Nain can modified during training and inference.
32

---

Under review as a conference paper at ICLR 2023
DI-Net-4 (our ImageNet classifier) performs forward pass on a batch of 48 images in 96 + 4ms on a single NVIDIA RTX 2080 Ti GPU.
F FUTURE DIRECTIONS
Scaling convolutional DI-Nets Our initial experiments suggest that convolutional DI-Nets do not scale well in depth. We suspect that within CNN-like architecture, the gradients of discrete convolu- tional layers with respect to kernel parameters have much smoother optimization landscapes over large networks relative to continuous convolutional layers parameterized with MLPs coefficients of polynomial basis. It is then no surprise that implementations of neural networks with continuous convolutions do not simply substitute the convolutional layers in a standard CNN architecture, but also make use of a variety of additional techniques (Qi et al. [0117] Wang et al. which would likely be helpful for scaling convolutional De-Nets.
2021
Boulch 2019)
Parameterization of output NFs In this work assume that DI-Net that produces NF specifies the output discretization a priori, but some applications may need the output to sampled several times at different discretizations. It is inefficient re-evaluate the entire network in such cases, and we propose two solutions for future work. One method can store the last few layers of the network alongside the input activation, and adapt the discretizations as needed in these last few layers only. A second approach can treat the discretized outputs DI-Net as parameters of the output NF in the manner of Vora al. (202T), which would maintain interoperability of the entire framework.
Extending DI-Net to high discrepancy sequences In many applications, there are large regions of domain that are less informative for the task interest. For example, most of the information in 3D scenes is concentrated at object surfaces, so DI-Nets should not need to process a NeRF by densely sampling all 5 dimensions. Moreover, ground truth labels for dense prediction tasks may only be available along high discrepancy discretization. Such discretization can be handled by quadrature, but more work is required to design efficient quadrature methods within a neural network. Additional techniques such as learned coordinate transformations learned discretizations may also be helpful for extending our model to extreme discretizations or highly non-uniform measures.
Error propagation When an NF does faithfully represent the underlying data, it is important to characterize the influence on DI-Net's output. In the worst case, these deviations are adversarial examples, and robustness techniques for discrete networks can also be applied DI-Net. But what can we say about typical deviations of NFs? Future work should analyze patterns in the mistakes that different types of NF's make, and how to make DI-Nets robust to these.
33


Reviews:
[{"review_id": "gqfMSlxYw0f", "reviewer": "Reviewer_pEzC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a general framework for building neural networks that take neural fields (NFs) - vector fields over compact spaces - as input and/or output. Processing NFs requires choosing a discretization on the compact space. The paper studies the sensitivity of maps between NFs on the choice of discretization. They define \"Discretization Invariant\" (DI) maps between NFs to be those for which the discretization error on a finite sample X is bounded by the variation of the input NF and the discrepancy of X.\n\nThe paper then proposes a family of DI invariant maps to be continuously defined convolution-like maps between fields on a particular discretization. The authors prove that this construction is universal and that the gradients can be estimated consistently on finite samples.\n\nVarious train/test discretizations are evaluated on classification on a NF version of imagenet, segmentation and signed distance function prediction.", "summary_of_the_review": "The paper does a novel in-depth analysis of maps between NFs, but it is difficult to follow and lacks comparisons to baslines.", "strength_and_weaknesses": "Strength:\n- Appears a thorough and novel theoretical analysis of maps between NFs, proving desirable properties\n- Gives an interesting empirical insight into how to choose discretizations.\n- The method is evaluated on three different interesting problem domains.\n\nWeaknesses:\n- I'm a quite confused about the weak derivatives in eqns (3) and (6) and the assumption that the NF is in the Sobolev space. Is the existence of the weak derivatives assumed throughout? If so, that's not made very explicit. Step 2 in the sketch of the proof of Theorem 2 also only mentions function evaluation, not taking the weak derivatives. Does it matter for the universality if the DI-Net takes gradients as input? It'd be great if the authors could clarify the role of these weak gradients.\n- The paper cites other methods to process NFs, such as via hypernetworks, but only compares grid discretizations to other discretizations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "agVGQToPq8", "reviewer": "Reviewer_XJni", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces DI-Nets, a formulation of neural networks that is invariant to discretization of the input space. More specifically, the authors consider building models that take neural fields as input, and unlike other recent work, do not depend on the specific parameterization of that neural field, which is desirable. The authors prove various theoretical properties of their model, including discretization invariance for the resulting networks and their gradients as well as a universal approximation theorem. The authors test an instantiation of their model in various toy experiments. \n", "summary_of_the_review": "Overall, I think this is a fairly good paper, which will be of interest to the ICLR community. The paper is well written, the method is sound and the experiments are fairly interesting. However, there are some issues around discretization invariance as well as empirical performance in practice. I therefore recommend a weak accept.\n", "strength_and_weaknesses": "**Strengths**\n\n- As neural fields are becoming increasingly popular, the paper tackles an interesting and timely problem, namely that of performing various machine learning tasks on neural fields.\n- The proposed model is invariant to the parameterization of the neural field, which is a desirable property. The experiments training on SIRENs and testing on Fourier feature networks highlight this nicely.\n- The paper is well written and polished and the figures are nice.\n- The authors provide a thorough theoretical analysis of the proposed model.\n- The appendix is thorough and well done, containing lots of details and exciting ideas for future work.\n\n**Weaknesses**\n\n- While the authors claim their model is discretization invariant, this statement is contradicted by experimental results. Of course, the model is discretization invariant in the sense of Definition 2, but the both the variation of f and discrepancy of X can be large, which means that the model is not discretization invariant in the practical sense of the term. This is demonstrated in numerous experiments throughout the paper, e.g. Table 2 and Table 4. I think it would therefore be good to temper the claims about discretization invariance or at least highlight early on that the theoretical definition of discretization invariance does not necessarily imply discretization invariance in practice.\n- In general, it feels like the theory is so \"general\" that it doesn't really provide any particularly useful guidance on how to actually instantiate a model. Indeed, the resulting models are very similar to various continuous convolution models.\n- The experimental results are quite weak (this is also acknowledged by the authors). The authors only perform small scale experiments and compare their results to fairly weak baselines (e.g. a 2 layer CNN). While it is okay to underperform CNNs designed for image classification, it would be interesting to include comparisons to other neural field based approaches (or explain why they are not relevant).\n- One of the advantages of using neural fields is that they typically provide a very compact representation of data (which was the motivation for the original 3D works on neural fields). However, using DI-Nets, we are back to sampling the original \"data space\" which, for 3D experiments for example, still would require a very large number of samples and so defeats the purpose of using the compact neural field representation in the first place. In contrast, other methods that act directly on some compact parameterization of a neural field are likely to scale better. The scaling of DI-Nets is then similar to just acting directly on the discrete signal (as mentioned in the appendix, \"DI-Net's complexity is similar to that of a discrete model with an equivalent architecture\"). I believe it would be worth discussing this when comparing to related work.\n- This might be a style preference, but the language/formulation sometimes seems unnecessarily formal, and a more intuitive presentation might have worked better.\n\n**Typos**\n\n- The figure reference on page 26 in the appendix is wrong (fig:layers left)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Uvbj6HkwZ3", "reviewer": "Reviewer_6WWK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel approach for learning over neural fields that is discretization invariant. The authors propose a mathematical framework and design a network following this definition. They show experimental results on several vision benchmarks.", "summary_of_the_review": "The paper proposes an interesting and novel approach to an important problem, but I do not think it is clear enough and I have very low confidence in my ability to reproduce the results from the papers description.", "strength_and_weaknesses": "Strengths:\n- The paper addresses an important problem, the dependency of standard learning approaches on discretization.\n- The work in novel and very interesting\n\nWeaknesses:\n- The paper is not well written. \n(1) Lots of space wasted on trivial points, like prop.1 that simply says that the function is well defined. Or the list of operations later that doesn't really helps understand the work proposed. Also Prop. 2 is trivial from definition (eq. 1).\n(2) The main technical part of the paper is glossed over, 3 lines in bullet point 3 in page 6. Everything up to that point was simply replacing an integral with an average.\n(3) The whole mathematical machinery feels redundant or under-utilized. How is it different then convolutions on a non-uniform grid?\n- The experimental part should compare to previous works such as Jiang et al \"CONVOLUTIONAL NEURAL NETWORKS ON NONUNIFORM GEOMETRICAL SIGNALS USING EUCLIDEAN SPECTRAL TRANSFORMATION\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "GQ88hAZc7gG", "reviewer": "Reviewer_NWbG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a general framework of discretization invariant network, DI-Net. The\nauthors give a theoretical upper bound w.r.t. variation and discrepancy in the finite samples case.\nThey also reveal that DI-Net is a generalization of existing neural fields networks such as CNN.\nThey demonstrate the performance of DI-Net against CNN and FCN with different discretization\nstrategies.", "summary_of_the_review": "I think this paper offers a general idea of discretization invariant learning on neural fields and\nsupplies plenty of theoretical results. However, I think the detail of the DI-Net is not abundant\nenough for readers. Also, personally the numerical results are not good enough to convince me the\nsuperiority of DI-Net.", "strength_and_weaknesses": "Strength\nThis paper gives a general idea of discretization invariant learning. It connects its key idea\nDI-Nets with existing popular architectures, such as CNN and neural operators. It offers a general\nperspective of these models.\n\nWeakness\nThe pseudo code of algorithm 1 and 2 in page 7 is negligible as they are the standard procedure\nof general learning process.\nNeural operators, as the authors mentioned in the paper, is an important category of discretiza-\ntion invariant network. But the numerical experiments focus mainly on the computer vision, e.g.\nclassification and segmentation. I think it would be better if there were an experiment on neural\noperators. Personally, I think such experiments are an important aspect of discretization invariant\nlearning.\nIn the numerical experiments part, the authors mainly compare different methods of dis-\ncretizaion, such as MC, QMC, grid and shrunk. But there were few attention on the discretization\nlevel. There was only a partial comparison in Table 1 on 32*32 and 64*64. I think it would be\nbetter if there were comparison between more sampling levels (of the same sampling method).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject