Discretization Invariant Networks for Learning Maps between Neural Fields
Authors: Clinton Wang
Polina Golland
Venue: Accepted by TMLR ()

Abstract: With the emergence of powerful representations of continuous data in the form of neural fields, there is a need for discretization invariant learning: an approach for learning maps between functions on continuous domains without being sensitive to how the function is sampled. We present a new framework for understanding and designing discretization invariant neural networks (DI-Nets), which generalizes many discrete networks such as convolutional neural networks as well as continuous networks such as neural operators. Our analysis establishes upper bounds on the deviation in model outputs under different finite discretizations, and highlights the central role of point set discrepancy in characterizing such bounds. This insight leads to the design of a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. We prove by construction that DI-Nets universally approximate a large class of maps between integrable function spaces, and show that discretization invariance also describes backpropagation through such models. Applied to neural fields, convolutional DI-Nets can learn to classify and segment visual data under various discretizations, and sometimes generalize to new types of discretizations at test time.

Published in Transactions on Machine Learning Research (10/2023)
Discretization Invariant Networks for Learning Maps between Neural Fields
Clinton Wang MIT CSAIL
clintonw@csoil. mit edu
Polina Golland MIT CSAIL
polina Cecaail mát. edu
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id- Op YBA BAg
Abstract
With the emergence of powerful representations contimuous data in the form neural fields, there need for discretization invariant learning an approach for learning maps between functions on continuous domains without being sensitive to how the function sampled. We present framework for understanding and designing discretization invariant neural networks (DI-Nets), which generalizes many discrete networks such convolutional neural networks well as continuous networks such as neural operators. Our analysis establishes upper bounds on the deviation in model oulputs under different finile discrelizations, and highlights the central role point set discrepancy in characterizing such bounds. This insight leads to the design of a family of neural networks driven by numerical integration via quasi- Monte Carlo sampling with discretizations of low discrepancy. We prove by construction that DI-Nets universally approximate large class of maps between integrable function spaces, and show that discretization invariance also describes backpropagation through such models. Applied to neural fields, convolutional DI-Nets can learn to classify and segment visual data under various discrelizations and sometimes generalize to new types of discretizations at test time. Code: https: //////tttt//////////////// com/ /itm://ioioooaaaDD--eeeee
1
Introduction
Neural fields (NFs), which encode signals as the parameters of a neural network, have many useful properties NEs can efficiently store and stream continuous data (Sitzmann et al. 2020b Takikawa al. 2022 Cho
et al. 2022). represent and render detailed 3D scenes at lightning speeds (Müller et al.) 2022 Barron et al.
Kerbl et al. 2023 and integrate data from wide range of modalities (Gao et al. 2021 2022). NFs are thus an appealing data representation for many applications, but learning downstream tasks such as NF classification, segmentation, or generation remains a challenging problem. Their continuous domain makes NEs unsuitable as inputs to traditional neural network architectures that are designed for discrete pixel or voxel grids.
2023
Current approaches for training networks on dataset of NF's predominantly focus on learning in NF parameter space (Tancik et al.) 2020a Dupont et al. 2022. Mehta et al.) 2021 but such approaches have two major disadvantages: 1) once trained on the parameter space of one type of NF, they become incompatible with other types; 2) they are unsuitable for important classes of NFss whose parameters extend beyond neural network, such as those with voxel (Sun et al. 2021 Alex Yu and Sara Fridovich-Keil et al. 2021), octree Yu et al.] )021), hash table (Miller et al. 2022 Takikawa et al. 2022 Barron et al.] 2023) or other (Kerbl et al. 2023) components. We instcad view NFs as black box vector-valued functions, and hence performing
inference on NFs can be understood learning operators on function space. Given that an algorithm can only evaluate NF at finite number of points, ask how learning algorithm should sample each NF

---

Published in Transactions on Machine Learning Research (10/2023)
its contimuous domain, and how to design a model whose output is largely independent of how the sample points are chosen, a property called discrelization. invariance.
Current analyses of discretization invariance are limited to showing the existence of operators that converge in the limit of discretizations with infinite points (Kovachki et al. (021b), or they demand that the function spaces be coistr to those that can be discretized losslessly (Bartolucci et al. (023). They do not examine the effect of the discretization itself on the approximation error in the general case. Characterizing this effect is particularly relevant in the context of neural fields, which permit many different types of discretizations and are often queried repeatedly under slightly different discretizations in applications such novel view synthesis. In this paper we describe discretization invariant neural networks (DI-Nets), a broad class of networks for learning maps between integrable function spaces such as those represented by neural fields. We explore how different discretizations yield different behaviors in the finite case, and establish the central role of point set discrepancy - points limiting discretization must be evenly distributed to achieve convergence. By spocifying layers as integrals over parametric functions of the imput field, DI-Nets have access to powerful numerical integration techniques such as quasi-Monte Carlo sampling, which yields fast convergence by choosing low discrepancy discretizations. DI-Nets encompass continuous networks such neural operators (Kovcchii et al. 2021b), while also extending discrete networks that act on pixcls, point clouds, and meshos.
Summary of contributions: we analyze discretization invariance in the finite sample case, where discrepancy and equidistributed sequences play a central role. Our analysis gives rise to a large family of discretization invariant networks (DI-Nets) that universally approximate a large class of maps between function spaces. We show that backpropagation through DI-Nets also yields discretization invariant gradients. We probe the limits of discretization invariant networks in practice, demonstrating that convolutional DI-Nets can learn classification and dense prediction tasks neural fields under range of different discretizations. We show that DI-Net has some ability to generalize to new discretizations at test time, whereas maps learned by pre-trained discrete networks collapse under slight perturbations of the discretization.
2 Related Work
Neural fields Neural fields (also called implicit neural representations) are neural networks that can be trained to capture a wide range of continuous data with high fidelity. They are usually parameterized as MLPs, sometimes with additional components such as features stored in voxel (Sun et al. 2021). octree
Yuu et al.] 2021) or hash table Mullerr et al.] 2022 Takikawa et al.
2022 Barron et al. 2023) structures.
Alternatively, neural fields can be represented directly as a set of parameters optimized directly with gradient
descent (Alex Yu and Sara Fridovich- Keil et al.) 2021 Kerbl ct al.]
2023
The most prominent domains
include shapes (Park et al.) [2019] Mescheder et al.; 2018), objects (Neemeyer et al., 2020 Müller et al.)
2022]. and 3D scenes (Mildenhall et al.] 2020) /Sizzmann et al., 2021 but previous works also apply NFs to gigapixel images (Martel al. 2021), volumetric medical images (Corona-Figueroa et al., 2022), acoustic
data (Sitzmann et al.) 2020b Gao et al. 2021), tactile data (Gao et al. 2022), depth and segmentation maps
Kundu et al.) 2022 and 3D motion [Niemeyer et al. 2019
Learning on neural fields Hypernetworks and modulation networks were developed for learning on neural fields, and have been demonstrated on tasks including generative modeling, data imputation, novel view synthesis and classification (Sitzmann et al.] 2020b] 2021: Tancik et al., 2020a. Sitzmann et al. 2019 2020a Mehta et al. 2021 Chan et al.] |021 Dupont et al. 2021 2022). Hypernetworks use meta-learning to learn to produce the MLP weights of desired output NFs, while modulation networks predict modulations that can be used to transform the parameters of an existing NF' or generate new NF. An alternative approach uses the derivative networks of the NF (Xu et al., 2022). training an MLP that learns mapping between first through kth order derivatives of the NF and the desired output signal. Both of these approaches require that the imput NEs are entirely MLPs, and do not generalize to new architectures once trained. An alternative approach for learning 3D NF-NF maps evaluates an input NF at fixed grid points, produces features at the same points via a U-Net, raytraces interpolated features, then uses an MLP decoder to produce output values from arbitrary camera views (Vora et al.) 2021
2

---

Published in Transactions on Machine Learning Research (10/2023)
Group invariant neural networks There is a rich literature exploring the design and training of neural networks for learning maps between function spaces subject to symmetries such as permutation invariance (Zweig& & Bruna 2021 Zaheer et al.) 2017], rotational invariance (Cheng et al. 2018), or more general group invariances (Yarotsky 2022 Lyle et al. 2020). Discretization invariance must be treated differently from group invariance as discretizations lack all the properties of groups (associalivity, identity element, and inverse element). Moreover, producing identical outputs under arbitrary discretizations is impossible on all but the most trivial function spaces, hence discretization invariance must be described in an approximate or limiting sense. Since group actions map between different discretizations, discretization invariance can be seen as weaker yet more general form of group invariance. Discretization invariance may be more useful in settings where the task of interest should be able to be solved under a wide range of discretizations which are not related by a single group.
A Approximation capabilitics of neural networks A fundamental result in approximation theory is that the set of single-layer neural networks is dense a large space of fuucct ionals including (P"))) (Hornik) 1991]. J.
Subsequent works designed constructive examples using various non-linear activations (Chen al. 1995 Chen
& Chen 1993). While this result is readily extended to multi-diiennsonal outputs, existing approximation results for the case of infinite dimensional outpuls (e.g., LP(R)) + LP(())) do not explicitly characterize the contribution of data discretization to the approximation error (Bhattacharya et al. 2020 Lanthaler et al.,
2022 Kovachki et al. 2021baa a). Universal approximation results for operator learning frameworks typically
quantify the approximation error in terms of the class of functions being approximated or the type of layers used in the network (Kovachki et al. b021b; Bartolucci et al. 2023 K Kissas et al. 2022 Prasthofer el al.,
2022 rather than the choice of discrctization, a gap that we seek to fill this work.
Discretization invariant networks Networks that are agnostic to the discretization of the data domain has been explored primarily in the context of learning operators between function spaces. Hilbert space PCA, DeepONets and neural operators are tailored to solve partial differential equations elficiently in a manner that converges as more sensors are added and the discretization of the input space is refined (Bhattacharya et al.) 2020: Lu et al. 2021a: Li et al., 2020a Kovachki et al., 2021b). In the context of learning on surface meshes, DiffusionNet (Sharp et al., 2022 also defines discretization invariance as convergent behavior in the limit of infinite sample points (mesh refinement). The recent. operator learning framework ReNO (Bartolucci et al.j 2023) starts with the assumption that there exists a lossless discretization of the input and output function spaces which is known priori (c.g., they are bandlimited functions), then establishes necessary conditions for learning (lossless) operators between such spaces. Other works are concerned with more practical aspects of operator learning: LOCA (Kissas et al., 2022) leverages attention to more efficiently learn correlations between related points in output space; NOMAD (Seidman et al. 2022) aims to increase expressivity given finite basis elements by equipping neural operators with nonlinear decoders; VIDON (Prasthofer et al. 2022 builds on DeepONet to accommodate arbitrary locations and numbers of input and output query points (as does our framework); other extensions of DeepONet and neural operators refine the original works to improve
learning efficiency or generalizability on certain domains (Li et al. 2020b
Lu et al. 2021b Lee al. 2022
Continuous convolutions At the core of many discretization invariant approaches is the continuous
convolution, which also provides permutation invariance, translation invariance and locality. Its applications include modeling point clouds (Wang et al. [021 Boulch 2019), graphs (Fey et al. 2017), fluids (Ummenhofer ctt al. 2019), and sequential data (Romero et al. 2021b a). where there is typically no choice of how the data should be discretized. This work focuses on the elfect of different discretizations, proposes quasi-Monte Carlo as canonical method of generating discretivations, and can produce neural fields as output.
3 Discretization Invariant Learning
3.1 Discretization Invariance
Let Ω be a bounded measurable subset of a d-dimensional compact metric space, for example a compact subset of IRd or a d-dimensional manifold. Consider the space of vector-valued functions of bounded variation
F. = ( : Q + T : Iq and V(J) < oo), where the variation V(J) measures how much the
3

---

Published in Transactions on Machine Learning Research (10/2023)
function fluctuates over its domain. The variation of 1D function C' ([a, bl) is given by:
v() - Lo (2)),,,
(1)
and more general definitions are given in Appendix A.1
The discretization of a function finite point set X C on which is queriod. We say that map H F. + R is discretizable if it induces discrete operator H Fe I2, which seeks replicate the behavior of the original map but depends only the input's values at X. Ideally design maps that are truly invariant to the choice of discretization, meaning that all its discrete operators are identical. But such idcalized discretization invariance is only possible on the most trivial function spaces, and thus a more practical definition of discrelization invariance is necessary:
would be able to
Definition 3.1. A discretizable map H F. IR is discretization invariant if there exists constant k > 0 such that for every discretization and function f. | - 7 [f]],, & (V()) where D(X) is the discrepancy of X. A map H. Fe + Fn is discretization invariant if Hz+](2) is uniformly discretization invariant for all Ω
This definition establishes upper bound the deviation between any two discretizations by simple application of the triangle inequality. The discrepancy of discretization is lower for dense, evenly distributed points. For 1D point set on domain Q = [a, b]. it is given by:
IN
IN [c
d-c b-a
((( = sup aSeddeb
(2)
N
See Appendix A.1 for general definitions of discrepancy. The product of variation and discrepancy is precisely the upper bound in the celebrated Koksma- -lawka inequality, which bounds the difference between the integral of a function E (22) and its sample mean on any point set X C
h(x') - h(x) dx S VV(h) D(X).
(3)
X
3.2 Discretization Invariant Layers
This naturally leads a family of discretization invariant (DI) layers specified as the integral of a parametric map input function:
Ho f + / Holf((x)dz,
(4)
whose discrete operator is simply its sample mean:
A
Σ Holf(()).
(5)
X
xE
The parametric map HolS] can have two forms:
In vector-valued DI layers, HXX AX: F. - R and Hz)f](z) = ha, f(x)) E IR. Such layers include global pooling and learned inner products, and could be used as one of the final layers classification network.
In veleiion-valudd DI layers, H. Fe + F, and Ho Hoff(a) I' + ha(a, x', f(), f(x)) e R" for all x' Q.
Such layers include continuous convolutions, deconvolutions, and self-attention layers.
4

---

Published in Transactions on Machine Learning Research (10/2023)
In each case, he must be bounded and continuous in all its variables so that outputs remain of bounded variation (hence satisfying our definition of discretization invariance). hg must also be differentiable w.r.t. & to enable backpropagation, and Gateaux differentiable w.r.t. f to make its gradients discretization invariant,
as we discuss in Section 3.3 We consider even more general forms of DI layers in Appendix A.2
The upper bound in Eq. points to two levers for reducing the approximation error of a given layer. We can design Ho to have low variation for most f, for example by imposing Lipschitz regularization, but there is tradcoff between reducing variation and maintaining the layer's ability to capture information relevant to the downstream task. We can also choose X to have low discrepancy by using fast methods for generating low discrepancy sequences. Obtaining the sample mean in this way is called quasi-Monte Carlo (QMC), a numerical inlegration method with favorable convergence rates compared to standard Monte Carlo (Caflisch) 1998p. In this work, we assume that the discretization is chosen once without a priorí knowledge of the function, making QMC optimal. But we note that tighter bounds on the approximation error can be attained by choosing a different discrete operator basod on quadrature, and/or by refining the discretization after initial evaluations of the operator. Replacing |/|X| in the sample mean with quadrature weights will achieve better convergence when the discrepancy of X is high, and adaptive quadrature updates the discretization to attain specific error bounds at inference time, which can be valuable in applications requiring robustnoss or verification. Additionally rejection sampling can be used for the Monte Carlo method under non-uniform measures.
A discretization invariant network (DI-Net) is a directed acyelic graph of DI layers as well as pointwise layers M A pointwise layer is a bounded continuous scalar function applied to each point in Q2 and includes nonlinear activations, batch normalization, as well addition concatenation of NFs. Since pointwise layers preserve an NF's property of bounded variation, DI-Net is discretization invariant. A prototypical DI-Net for classification might consist of NF-valued DI layers separated by normalization and activation layers, and end with a vector-valued D1 layer followed by softmax.
3.3 Convergence under Equidistributed Discretizations
From our definition of discretization invariance, it is clear that the approximation error of D1 layers converges to 0 under sequences of discrctizations whose discrepancy tends to O. We call such a sequence of discrctizations an equidistribuled discretization sequence. A simple way to generate an equidistributed discretization sequence is to truncate any equidistributed sequence of points to the first N terms for each N e N. Quasi-Monte Carlo wide range of domains.
(QMC) sampling efficiently generates equidistributed sequences
We can also ask whether the DI-Net's learning algorithm is also discretization invariant: specifically whether its discretized gradients are convergent, and to what value they converge. Consider a vector-valued DI layer Hg. The derivatives of its output w.r.t. its weights @ = (ōI...,,OR) can be shown to converge under any equidistributed discretization sequence (XN]NeN. Denote o + Tek = (o.....; .h......oe ........kk-..
d 1. db + T, 7.
(
a N abx
lim 00 Odbr
lim A
Σ Holf((t)
(6)
|NNl #EXN
N
= lim lim N T-10
Hotro [M(()) - Holl(x))
(7)
rN
j=1
=lim-
H Hotter [1(x) - Holf)(x) dr
(8)
--0 T J2
Hotre [f] -Holfl
lim T-10
(9)
T
Ô Dok Hols),
(10)
As its first. layer a DI-Net that maps vectors functions may require vector decoder: a map R + specified by n elements of basis Fe or interpolation of n/c points (see Appendix C for examples).
5

---

Published in Transactions on Machine Learning Research (10/2023)
where S uses the Moore-Osgood theorem. The case of function-valued DI layer proceeds identically, showing this condition holds for the derivatives at each point on the output domain. Thus the discretized gradient converges to the Jacobian of Hi w.r.t. each parameter, which is finite by differentia billiy and boundedness of Hg.
Describing the derivative of the layer's output w.r.t. the input function is more nuanced, since pointwise derivatives are not sufficient to represent backpropagation in the continuous case. We must instead use the Gateaux derivative, which describes the linear change in map between functions given infinitesimal change in the input function. We prove the following in Appendix B.1
Lemma 3.2. For every I E Fi and fixed ā Ω can design sequence of bump functions ( NeN which is 1 in a small neighborhood around it and vanishes at each X XN\[ such that:
lim N Qf(3)
lim dHs ff: o N-000
(11)
where dHo [/: 1 V.A.
lim Hols+nng] - is the Galeaur derivative when f is perturbed in the divection of
Having established convergence of the discretized gradients w.r.t. both parameters and inputs, we then note that many common loss functions and regularizers generalize naturally to the continuous domain as bounded continuous maps Je + (e.g., L2 regularization) or Fe x Fe + E (e.g., mean squared error), so discretization invariance readily extends to gradients with respect to loss terms. Finally, using the chain rule for Gateaux then show that all discretized gradients obtained during backpropagation through DI-Net are convergent. These results lead to the following statement (detailed proof in Appendix B.2):
derivatives,
Theorem 3.3. A DI-Net permits backpropagation with respect to its input and all its learnable parameters. The discretized gradients converge under any equidistriuuted discretization sequence.
4 Universality of DI-Nets
We observed that the approximation error can be made arbitrarily small by choosing a discretization with sufficiently small discrepancy. Functions of bounded variation are piecewise smooth, hence they can be represented as the integral of some function. Our parameterization of DI layers as integrals DI-Nets are universal approximators in the following sense:
Fn; n, E,, there erists DI-Net that approrimates it to arbitrory accuracy w.r.t. finite measure on Fo. As corollary, every Lipschitz continuous map F. + RD or R. + F. can also be approzimated by some DI-Net.
Theorem 4.1. For every Lipschitz continuous map R:
Appendix B.3 provides a full proof. Here we provide a high-level sketch of the Fi + Fi case, where there exists a DI-Net that satisfies:
f - <e.
(12)
Fix discretization X of sufficiently low discrepancy to approximate any function in F, to desired accuracy. Let N ||,,, the number of points in the discretization.
Let æ be a projection operator that maps every function f E Fi to R. by selecting its N values along the discretizatoon. Through π,, the measure on F, induces a measure μ on N
3. Any function in (2RRRN) can be approximated by covering the volume under the graph of the function with almost disjoint rectangles, and then at inference time summing the heights of the rectangles at the given R input. A multilayer perceptron (MLP) can then approximate this rectangle cover to arbitrary accuracy with sufficiently steep slopes at their boundaries (L et al. 2017
4. The map : ff + (z) is in (R for each E X. We can mimic the MLP from step 3 with a DI-Net that specifies the desired connections Ω using wlement-wise products with cutoff functions and

---

Published in Transactions on Machine Learning Research (10/2023)
linear combinations of channels. The cutoff functions extract the imput values along X into separate channels, and the weights of the channels match the weights of the hypothetical ML.P, projecting the result to the appropriate æ. We present the design of such a network in Algorithm | with additional details in the Appondix.
5. We repeat this construction N times to specify values at each of the N output points in X, and map all other output points to the value of the closest specified point. Then we have fully specified the desired behavior of f R[f] to desired accuracy w.r.t. the measure v.
Algorithm 1: DI-Net approximation of J J
Input:
target function J, discretized input = ( in Li tolerance e/2
Choose rectangles R [o 6/]] x ... |a 6t,] ls S + y covering the graph of
(f) A max(0, J(f)) and rectangles R- covering the graph of J-(() & maxx00,-((f (precise
conditions stated in (22 );
8+ 1 1 (1 (1- (1]J+1 11 Te) )2/0)
-1 32/n 8 + 1 (1- (I|J 11 + Te)
I + (0,0,1,0,0):
for rectangle R R do for dimension k E 1 do
+ (f.- b + 5(b - a fs a.t. F3. Ts)
I < ReLU(*);
I - (6 21, x2 I3, I4, Tz);
+ ReLU(z):
- (0,0, 2z(21 22)/6, 21) 25);
end
x + (0, y, 33 Ts);
end
for rectangle RT E R do
for dimension k E 1 do
+ (f&- b + 5 8(b
), fb
-Osk 23, 375;
end
+ (0,0, 1, 33 x5);
end
Output: Ø1 35
This construction suggests that the complexity of representing a given operator is linked to the minimum number of rectangles necessary to cover its pointwise graph (J in step 4) with desired tolerance, and the degree to which this graph is similar for neighboring points in the output domain. Intuitively, this means that it is easier to represent operators in which each point in the output is influenced by relatively few points in the input domain, and where there is shared structure in this dependency. Thus DI-Net design can be guided by strategies for imposing structure on how different points influence each other: for example, convolutions produce translation invariant outputs that depend on the input function locally; attention layers produce outpuls using sparse disconnected regions of the inpul function; Fourier neural operators (Li et al. 2020a produce outputs driven by low frequency patterns in the input function.
5 Design and Implementation of DI-Nets
DI-Nets encompass very large family of neural networks: we only specify the architecture as directed acyclic graph, and DI layers include wide variety of network layers. We first discuss their application to
7

---

Published in Transactions on Machine Learning Research (10/2023)
DI-Net
(xr
Dataset
DI-Net
NF
label Classification
Numerical Integration
NE
Dense
NE
NF
NF
Prediction
NF
non- non- linearities
noise
NF
Generation
Figure 1: A discretization invariant network (DI-Net) treats neural fields as vector-valued functions. It. evaluates an input field on a point set (discretization) which used to perform numerical integration throughout the network. DI-Nets are interoperable between all types of NFs and can be trained on wide range of tasks.
neural fields, which can be treated function spaces to achieve parrameterizztiio--annoot learning. We then discuss the connection between DI-Nets and neural operators (Kovachki et al. 2021b Lu et al. (021a), which can learn general maps between function spaces but in practice are designed to solve partial differential equations. Next we show that DI-Nets also encompass continuous adaptations of networks designed on discrete domains such as convolutional neural networks (CNNs). In the same way that neural fields extend signals on point clouds, meshes, grids, and graphs to a compact metric space, DI-Nets extend networks that operate on discrete signals by converting every layer to an equivalent discrctizable map. Lastly we describe training and inference pipelines for DI-Nets.
5.1 Learning on Neural Fields
The parameterization of neural fields in terms of multi-layer perceptrons guarantees that they produce integrable functions of bounded variation throughout. Section 3 Specifically, a d-dimensional neural field with channels represents a vector-valued function from a d-dimensional domain Q to R. An occupancy network (Mescheder et al. 2018) is 3- dimensional and has channel. NeRF (Mildenhall et al. 2020) is 3-dimensional (3 world coordinates and 2 view angles) and has 4 channels. We assume that neural field evaluation yields pointwise values, ie. the point spread function of the underlying signal is a delta function. It possible to accommodate non-trivial point. spread functions, but this is beyond the scope of this work.
compact domain, which is the function space considered
By treating neural fields functions, this deep learning framework is parameterization-aggotse of DI layers does not depend on the parameterization of the NFs that it takes as input. This property allows DI-Nets to be applied to mixture of neural field types, which is not possible with most approaches for learning on neural fields such as hypernetworks or modulation-based networks.
the behavior
The action of a function-valued DI layer produces neural field with paramcters (), and retaining all the DI-Net parameters can lead an output NF with greatly inflated parameters, making it inefficient evaluate repeatedly. This is unsuitable for applications requiring the output to be sampled several times at different discretizations, where the output NF needs to be stored in its entirety. To solve this problem we can reparameterize the output NF by storing only the last few layers of the DI-Net alongside the discretized input activations, and adapt the discretizations as needed in these last few layers only. This approach is reminiscent of strategies that use discretized outputs a (non-DI) neural network as parameters of an output. neural field Vora al. 2021)
5.2 Connection to Neural Operators
Neural Operators KKvachki al. 2021b) are defined terms of pointwise lifting operator P R. + Rd a pointwise projection operator Q d - Bour, and kernel operators (Ki)= K, (d-1)) + L (S); Rd)
associated with non-linear activations at IRde-1 Rdd, weight matrices W, E Rd Rd. and biases

---

Published in Transactions on Machine Learning Research (10/2023)
b, : - Rd. Then the network becomes G Ri Reoue:
G = ==Qoor(TT Kr + br) o oW Ki + b1)
(13)
Because these layers are all DI layers or pointwise layers, neural operators represent an instantiation of DI-Ncts, and are an effective choice the task involves solving partial differential equations. Alternative designs become necessary when the imput and output domains do not match or when the task can benefit from multi-scale architectures.
5.3 Convolutional DI-Nets
We describe how to design DI-Nets that generalize the behavior convolutional neural networks to the continuous domain. The resulting convolutional DI-Net can be initialized directly with the weights of a pre-trained CNN, as we investigate in Section 6.4. We note that there is a rich literature of methods for designing improved continuous convolutions (Qi et al. 2017 Wang al. 2021 Boulch 2019) which would be helpful for scaling convolutional DI-Nets, although we do not use these methods that the connection between DI-Nets and discrete CNNs is more direct.
Continuous Convolution
Multi-Scale Connections
Pre-trained
Downsampling
discrete weights
Interpolate
Upsampling
S
3x3 kernel
B-spline surface
Numerical Integration
Figure 2: Convolutional DI-Nets generalize convolutional neural networks to arbitrary discretizations of the domain. Low discrepancy point sets used in quasi-Monte Carlo integration are amenable to the multi-scale structures often found in discrete networks. Convolutional DI-Nets may be initialized directly from pre-trained CNNs.
Convolution layer For a measurable S C and a polynomial basis {pyb20 that spans L²(S), S is the support of polynomial convolutional kernel K3 Ω R defined by:
Ei= 65 x'y
if x x S
Kg(2,
(14)
otherwise.
for some chosen n E N. A convolution is the lincar map Ho Fi + Fi given by:
Holfl - Ko x'), (=("x'))(
(15)
If Cin is the number of input channels and Cout the number of output channels, then the convolution layer aggregate information across channels using CinCout different filters.
An MLP convolution is defined similarly except the kernel becomes K2( = MLP(x Φ) in the non-zero case. While MLP kernels are favored over polynomial kernels in many applications due to their expressive power (Wang et al. (2021), polynomial bascs can be used to construct filters satisfying desired properties such group equivariance (Cohen & Welling 2016a b), k-Lipschitz continuity, or boundary conditions.

---

Published in Transactions on Machine Learning Research (10/2023)
The input and output discretizations of the layer can be chosen independently, allowing for padding striding. The input discretization fully determines which points on S are evaluated. To transfer weights from discrete convolutional layer, K can be parameterized as a rectangular B-spline surface that interpolates the weights (Fig. 2 left). We use a 2nd order B-spline for x 3 filters and 3rd order for larger filters. We use deBoor's algorithun to evaluate the spline at intermediale points.
Pointwise channel mixing Linear combinations of channels can be used similarly to 1 x convolutional layers in discrete networks. For learned scalar weights Wi and biases by and all x E Ω2
g5(x) Σ Wijf(z)+
(16)
Normalization All forms of layer normalization readily generalize to the continuous setting by estimating the statistics of each channel with numerical integration, then applying point-wise operations. These layers typically rescale each channel to have some learned or fixed mean m, and standard deviation 8;.
/ f((x)dz
P4
(17)
J
0
[xx))dxx - 11g
(18)
(x) f((z) p4
31 + mi:
(19)
where > 0 is a small constant. Just in the discrete case, μ4 and o, can be moving average of the mcans and standard deviations observed over the course of training different NEs, and they also be averaged over minibatch of NFs (batch normalization) or calculated per datapoint (instance normalization). Mean m. and standard deviation 9, can be learned directly (batch normalization), conditioned on other data (adaptive instance normalization). or fixed at and respectively (instance normalization).
Multi-scale architectures Many discretizations permit multi-scale structures by subsampling the dis- cretization, and QMC is particularly conducive to such design. Under QMC, downsampling is efficiently implemented by truncating the list of coordinates in the low discrepancy sequence to the desired number of terms, as the truncated sequence is itself low discrepancy. Similarly, upsampling can be implemented by extending the low discrepancy sequence to the desired number of terms, then performing interpolation by copying the nearest neighbor(s) or applying some (fixed or learned) kernel. Residual or skip connections can also be implemented efficiently since downsampling and upsampling are both specified with respect to the same discretizzaiinn (Fig. 2 right).
5.4 Training DI-Nets
The pipeline for training DI-Nets is similar to that for training discrete networks, except that input and/or output discretizations should be specified. When training a DI-Net classifier, the input discretization may be specified manually or sampled from a low discrepancy sequence to perform QMC integration. The QMC discretization can be either deterministic or pseudorandom, and can accelerate computation when the same discretization is used for multiple network layers or all functions in minibatch. When training DI-Nets for dense prediction (e.g, segmentation), the output discretization should be chosen to match the coordinates of the ground truth labels. Any input discretization can be chosen. Unless otherwise stated set it equal to the output discretization. At inference time, the network can be evaluated with any output discretization, making the output in effect a neural field. We outline steps for training a classifier and dense prediction DI-Net in Algorithms 2 and 3 to illustrate their similarity to the discrete case, besides the specification the discrotization. Similar pipclines could be used to train DI-Net on other tasks including generative modeling, inverse problems, or representation learning. In most cases where discrete networks are sufficient, the gains for using DI-Nets may be limited.
10

---

Published in Transactions on Machine Learning Research (10/2023)
5.5 Computational complexity
In general DI-Net's time and memory complexity both scale linearly with the number of sample points (regardless of the dimensionality of Q), as well with network depth and width. Implemented naively, the computational cost of the continuous convolution is quadratic in the number of sample points, as it must calculate weights separately for each neighboring pair of points. But we can reduce this to a linear cost. by specifying a Nain Voronoi partition of the kernel support B, then using the value of the kernel at each seed point for all points in its cell. Thus the kernel need only be evaluated Nbin times regardless of the number of sample points. Additionally Nbin can be modified during training and inference.
Thus DI-Nets are similar in computational complexity to discrete neural networks, with the added flexibility of being able to sample in non-grid patterns that can converge more efficiently. Although networks that operate directly on the parameters of the neural field can access the entire NF in constant time, the NF itself needs more parameters to capture finer resolutions, and in practice many downstream tasks can be solved at a much coarser resolution than would be captured by the NF. Thus such networks do not necessarily scale better than DI-Nets, and would likely sulfer from higher computational costs when there is a mismatch between the resolution of the NF and the resolution needed for the task.
6 Experiments
We apply convolutional DI-Nets to toy classification (NF->scalar) and dense prediction (NF-NF) tasks, and analyze its behavior under different discretizations. Our aim is not to compete with discrete networks on these tasks, but rather to illustrate the learning behavior of CNNs compared to DI-Nets with the equivalent architectures, without additional techniques or lypes of layers. We demonstrate that convolutional DI-Nets learn maps that often generalize to unseen discretizations, whereas maps learned by pre-trained CNNs are highly sensitive to perturbations of the discretization.
6.1 Neural Field Classification
Data We perform classification on a dataset of 8,400 NTs fit. to a subset of ImageNetlk (Deng et al. 2009) with 700 samples from each of the 12 superclasses in the big_ 12 dataset (Engstrom et al. 2019), which is derived from the WordNet hierarchy. We fit SIREN (Sizznann et al. 2020b) to each image in ImageNet using 5 fully connected layers with 256 channels and sine non-lincarities, trained for 2000 steps with an Adam optimizer at a learning rate of 10-4 It takes coordinates on [-1,1] and produces RGB values in []1, We fit Gaussian Fourier feature (Tancik et al.) 2020b) networks using 4 fully connected layers with 256 channels and ReLU activations. It takes coordinates on [0, 1J2 and produces RGB values in [0, 1]s. The average pixel-wise error of SIREN is 3- 100 4 10 4, compared to 1.6 10 2 + 100-3 for Gaussian Fourier feature networks. The difference in quality is visible at high resolution, but indistinguishable at low resolution.
Architecture and Baselines We train DI-Nets with 2 and 4 MLP convolutional layers, as well as CNNs with similar architoctures. We also train an MLP that predicts class labels from SIREN parameters, and a "non-uniform convolution" (Jiang et al. 2019) that applies a non-uniform Fourier transform input points (sampled with QMC) to map them to grid values, then applies 2-layer CNN. DI-Net-2 uses strided MLP convolutions, a global average pooling layer, then two fully connected layers. DI-Net-4 adds a residual block with two MLP convolutions after the strided convolutions. We train all models with an AdamW optimizer [Loshchilov & Hutter 2017). The architecture of the MLP is 3 fully connected layers with 128 hidden units each and ReLU activation separated by batch normalization. learns to map the SIREN parameters to the class label. We found that the model's loss curve becomes unstable after 3000 iterations so we reduce the number of iterations to 2000. The non-uniform CNN applies the non-uniform Fourier transform (Muckley et al.) 2020) followed by inverse Fast Fourier Transform to resample the input signal to the grid. It then feeds the result to a 2-layer CNN to perform classification.
Training and Evaluation For each class, we train DI-Net on 500 SIRENs Sitzzann et al. 2020bb and evaluate on 200 Gaussian Fourier feature networks (Tancik et al. 2020b). Training and testing different.
11

---

Published in Transactions on Machine Learning Research (10/2023)
45%
40%
Top-1 Accuracy
35%
30%
2-layer DI-Net
4-layer DI-Net
2-layer CNN
20%
4-layer CNN
16a
32
48
642
80
962
Resolution
Figure 3: Classifier performance with different resolutions at test time.
NF types is not possible for the MLP approach, so it is evaluated on SIREN images instead. During training, we augment with noise, horizontal flips, and coordinate perturbations.
Each network is trained for SK iterations with learning rale 10-3, In training, the CNNs sample neural fields along the 32 32 grid. DI-Nets and the non-uniform network sample 1024 points generated from scrambled Sobol sequence (QMC discrotization). We evaluate models with top-1 accuracy at the same resolution as well at several other resolutions. A 4-layer DI-Net performs forward pass on batch of 48 images in 96 + 4ms single NVIDIA RTX 2080 Ti GPU.
The MLP and the non-uniform method significantly underperform DI-Net, with 13.9% and 28.9% accuracy respectively compared to 32.9% for our 2-layer network. At 32 x 32 resolution, DI-Nets somewhat underperform their CNN counterparts, and this performance gap is larger for deeper models. However, our discretization invariant model better generalizes images of different resolutions than CNNs (Fig. 3,, particularly at lower resolutions.
Performance
Table 1: Accuracy of 2-layer DI-Net under various discretizations.
Train -Tet Disc.
Accuracy
QMC-QMC
32.9%
Grid->Grid
30.5%
Shrunk-SShrun
30.3%
QMC+GGrid
27.1%
Grid+QMC
27.8%
QMC-+Shruk
25.4%
Shrunk-QMC
13.4%
We next examine whether DI-Net can adapt to an entirely different type of discretization at test time, We use grid, QMC and shrunk discretizations of 1024 (32 32) points. The Shrunk discretization shrinks Sobol (QMC) sequence towards the center of the image (each point E [-1, 1J2 is mapped to =zsgnæz)). In image classification, the object of interest is usually centered, hence the shrunk -shnunk setting performs on par with other discretizations despite its higher discrepancy.
Interestingly, changing discretization type at inference time has varying impact. Usually it only slightly degrades DI-Net's accuracy (Table 1 but performance falls dramatically when shifting from high low discrepancy discretizations (shrunk+QMe) This observation points the importance of training on the right discretizations to attain network that generalizes well to other discretizations for the given task.
6.2 Neural Field Segmentation
Data
We perform semantic segmentation of SIRENs fit street view images from Cityscapes (Cordts 2016), grouping segmentation labels into categories. We train 2975 NFs with coarsely annotated
et al.
12

---

Published in Transactions on Machine Learning Research (10/2023)
segmentations only, and test 500 NFs with both coarse and fine annotations (Fig. 4 We use 48 96 grid discretization since segmentation labels are only given at pixel coordinates. STREN is trained on Cityscapes images for 2500 steps, using the same architecture and settings as ImageNet. Seven segmentation classes are used for training and evaluation, labeled as 'flat' (c.g. road), 'construction' (e.g. building), 'object" (e.g. pole), 'nature', "sky', "human', and 'vehicle..
Architecture and Basclines We compare the performance of and 5 layer DI-Nets and fully convolutional networks (FCNs), as well as a non-uniform CNN (Jiang et al., 2019). , We also train hypernetwork that learns to map each SIREN to the parameters of SIREN representing its segmentation.
DI-Net-3 uses two MLP convolutional layers at the same resolution followed by channel mixing (pointwise convolution). There are 16, 32 and 32 channels in the intermediate features. The support of the kernels in the MLP convolutional layers is .025 x .05 and .075 x .15 respectively, to account for the wide images in Cityscapes being remapped to [-1, DI-Net-5 uses a strided MLP convolution to perform dowusampling and nearest neighbor interpolation for upsampling. There are 16 channels all intermediate features. There is a residual connection between the higher resolution layers.
The CNN baselines use 3x3 convolutions with the same number of layers and channels as DI-Net. All networks use ReLU activation and batch normalization. The hypernetwork learns a map from the SIREN RGB to a SIREN with the same architecture that represents the segmentation. It predicts changes to the weights of all layers before the final fully connected layer, and predicts raw values for the weights of the final layer since it has output channels for segmentation instead of 3 for RGB. The non-uniform CNN applies the non-uniform Fourier transform followed by inverse Fast Fourier Transform, and feeds the result to the 3-layer FCN perform segmentation.
Training and Evaluation Networks trained for 10K iterations with a learning rate of 10-3. We evaluate each model with mean intersection over union (mloU) and pixel-wise accuracy (PixAcc)..
Performance The hypernetwork and non-uniform CNN perform poorly compared to both FCNs and DI-Nets (Table 2). DI-Net-3 outperforms the equivalent FCN, and less often confuses feal such as shadows and road markings (Fig. 4). However, the performance deteriorates when downsampling and upsampling layers are added (DI-Net-5), echoing the difficulty in scaling DI-Nets observed in classification. We suggest potential methods for remedying this Section 7
Table 2: Segmentation performance on NFs fit to Cityscapes images (trained on coarse segs).
Coarse Segs
Fine Segs
Model Type
mIoU
PixAcc
mIoU
PixAce
3-layer FCN
0.409
69.6%
0.374
63.6%
DI-Net-3
0.471
78.5%
0.417
69.4%
5-layer FON
0.488
79.4%
0.436
72.5%
DI-Net-5
0.443
77.7%
0.394
68.4%
Hypernetwork
0.038
7.9%
0.042
8.3%
non-uniform
0.109
26.5%
0.106
22.7%
6.3 Signed Distance Function Prediction
Data We train convolutional DI-Net to map field of RGBA values in 3D to its signed distance function (SDF). We create synthetic dataset of 3D scenes with randomly colored balls embedded in 3D space. Each toy scene contains 2-4 balls of random radii (range 0.2-0.5), centers, and colors scattered in 3D space (Ω [-1, For simplicity, we train each network directly on the closed form expressions for the RGBA fields and signed distance functions, rather than fitting neural fields first. We train on RGBA-SDE pairs using a mcan squared error (MSE) loss on the predicted SDF. We use grid (16 x 16 16), QMC, shrunk, Monte Carlo (i.idd. points drawn uniformly from the domain), and mixed discretizations of 4096 points. the mixed setting, each minibatch uses one of the other four discretizations at random.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Input NF
DI-Net-3 (ours)
3-layer FCN
Ground Truth
Coarse
Fine
Figure 4: Cityscapes NF segmentations for models trained on coarse segmentations only. NF-Net produces NF segmentations, which can be evaluated at the subpixel level.
Architecture and Basclines The FCN contains 3 convolutional layers kernel lengths 3, 5 and respectively. Accordingly, the convolutional DI-Net contained 2 convolutional layers followed by a linear combination layer. There channels in all intermediate features.
Training and Evaluation We train each network for 1000 iterations with an AdamW optimizer with a batch size of 64 and learning rate of 0.1 with an MSE loss on the SDF.
Performance Under any fixed discretization, the convolutional DI-Net significantly outperforms the equivalent discrete network (MSE of 0.022 vs. 0.067 respectively). In Figure 5 we illustrate our model's ability to also produce outputs that discretized differently than the input, making DI-Net's output in effect neural field. By changing the output discretization of the last convolutional DI-Net layer, evaluate the output SDF anywhere on the domain without changing the inpul discretization. Whereas the discrete network is forced to output predictions at the resolution it was trained on, convolutional DI-Net can produce outputs along a high-quality grid discretization given a coarse QMC discrctization, even when is only trained under QMC output discretizations.
Table 3:
Mean squared error (x10-2) of predicted SDFs under different discretizations. The three best settings are bolded. MC=Monte Carlo.
Train
Test
Grid
QMC
Shrunk
MC
Grid
2.18
2.54
3.77
3.81
QMC
3.60
2.01
2.94
3.92
Shrunk
3.72
2.88
2.00
4.30
MC
6.45
5.97
4.89
5.92
Mixed
4.65
4.41
3.26
4.09
DI-Net performs almost equally well under grid, QMC and shrunk discretizations in the in-distribution setting, but this task it is more sensitive to out-of-distribution discretizations than the classifier in Section
While ShC is the worst performing combination for the classifter, here it is one of the better performing combinations. DI-Net likely struggles with Monte Carlo sampling duc to its high discrepancy discretizations, resulting in cases where smaller balls are entirely missed. Interestingly, the model fares worse when trained on multiple discretizations simultaneously, suggesting that the network may guided in opposing directions by different discretizations resulting unstable training. These obscrvations illustrate the complex task desendepnnd interplay between the type of discretizations observed training time and the ability of the model to generalize to new iiscretizations.
6.1
14

---

Published in Transactions on Machine Learning Research (10/2023)
RGBA
True SDF
DI-Net (ours)
Discrete CNN
Figure 5: 2D slices of two toy 3D scenes (black is transparent) with signed distance functions predicted by DI-Net and fully convolutional network.
6.4 Initialization with discrete networks
A convolutional DI-Net can be initialized with the weights of a pre-trained convolutional neural network. fact, every CNN induces convolutional DI-Net that behaves identically when every layer is restricted particular grid discretization. However, the behavior of the pre-trained CNN is not preserved when the DI-Net switches to other discretizations. Tiny perturbations from the regular grid can change a classifier's predictions, small differences in each layer can accumulate to exert large influence the final output. Figure 6 illustrates this phenomenon for DI-Net initialized with a truncated EfficientNet.
Morcover, we find that once grid discretization is abandoned, large DI-Nets cannot casily be fine-tuned to restore the behavior the discrete network used to initialize it. In Tables | and 5 we illustrate that DI-Net initialized with a large pre-trained discrete network does not match the performance of the original model when fine-tuned with QMC sampling. We a truncated version of EfficientNet (Tan & Le. 2019) for classification, and fine-tune for 200 samples per class. For segmentation ConvNexT-UPPrNel! (Liu el al. 2022), fine-tuning with 1000 samples.
truncated version of
Table 4: Pre-trained models fine-tuned on ImageNet NF classification.
Model Type
Accuracy
ElficientNet (Tan & Le 2019
66.4%
DI-Net-EN
48 48.1%
Table 5: Pre-trained models fine-tuned on Cityscapes segmentation.
Model Type
Mcan IoU
Pixel Accuracy
CouvNexT (Liu et. al. 2022
0.429
68.1%
DI-Net-CN
0.376
68.7%
These observations suggest that the optimization landscape of discretized maps vary significantly with small changes in X, and that the behavior of pre-trained CNNs should not be expected to generalize well to other discretizations. We suspect that this instability may be linked to the innate sensitivity of the grid discrelization due (at. least. in part) to its higher discrepancy. We observe that the output of a randomly initialized DI-Net less stable under changing sampling resolution with a grid pattern relative to a low discrepancy discrctization (Fig. 7b While the output of DI-Net with QMC sampling converges at high resolution, the grid sampling scheme has unstable outputs until very high resolution. Only the grids that overlap each other (resolutions in powers of two) produce similar activations.
15

---

Published in Transactions on Machine Learning Research (10/2023)
QMC
Grid
Output Deviation
Output Deviation
322
642
128²
Sampling Resolution
Grid
QMC
Figure 7: Distance of the output of a DI-Net from its grid output at 32 x 32 resolution, when sampling at various resolutions. Its outputs deviate rapidly as its discrctization shifts from a regular grid to low discrepancy sequence
Figure 6: An DI-Net's output diverges as sample points are gradually shifted from a grid layout to a low discrepancy sequence.
7 Future Directions
Discretization invariance learning signal Our experiments show that gap remains in trained DI- Net's ability to generalize to unseen discretizations at test time. One approach to bridging this generalization gap may be to explicitly encourage discretization invariance during training by adding regularization term that minimizes the distance between intermediate features obtained with different discretizations Such a schemee could also be used for contrastive pre-training, where different discretizations of the same neural field should be projected onto closer points in latent space than discrotizations of different NFs.
Extending DI-Net to adaptive, high discrepancy discretizations Our approach of choosing the discretization of the domain priori breaks down in applications where large regions of the domain are less informative for the task of interest. For example, most of the information in 3D scenes is concentrated at object surfaces, so densely sampling all five dimensions of neural radiance field is poor choice of discretization for most downstream tasks. Morcover, ground truth labels for dense prediction tasks may only be available along a high discrepancy discrelization. However, low discrepancy sampling can still guide an initial discretization (e.g. selecting an initial set of camera poses or rays), and adaptive sampling techniques such as adaptive quadrature or sphere tracing can be used to refine the discrctization. Future work should explore the design of DI-Nets that can achieve low approximation errors under task-specific discretizations. It may also be possible reduce the number of samples needed to achieve low discretization error the course of training or at inference time, for example by learning which discretizations produce more reliable estimates, or by designing layers that encourage the integral to converge to predefined set of quantized values and propagating these quantized values downstream.
Error propagation Neural fields do not always faithfully represent the underlying data, whether due to insufficient coverage by sensors or due to a suboptimally trained field. Our analysis can be extended to account for this influence on the error in model's output. In the worst case, these deviations are adversarial examples, and robustness techniques for discrete networks can also be applied DI-Net. But what can we say about typical deviations of NFs? Future work should analyze patterns in the mistakes that different types of NEs make, and investigate how to make downstream models robust to these.
8 Conclusion
We present a general framework for understanding and constructing discretization invariant neural networks, highlighting the importance of discrepancy in bounding the deviation of the network outputs under differ-
16

---

Published in Transactions on Machine Learning Research (10/2023)
ent discretizations, well establishing convergence in both its learning and inference behavior under equidistributed discretization sequences. DI-Nets can learn arbitrary maps between integrable functions of bounded variation, making them a useful tool for performing inference on neural fields in a parameterization- agnostic manner, or for applications to systems that process a continuous signal by querying it on a point set. Discretization invariance may be a particularly useful concept harness the context. of 3D scene understanding, as the information a scene that relevant for most tasks of interest invariant under a much wider range of discretizations (e.g., points, rays and light fields; 360 degree or forward-facing) than can be described purely by group symmetries. With the increasing popularity and diversity of neural fields well as the emergence of tools to efficiently create large datasets of NFs, understanding discretization invariant. learning may be key to developing interoperable approaches for learning on such data.
17

---

Published in Transactions on Machine Learning Research (10/2023)
References
Christoph Aistlcitner and Josef Dick. Low-discrepancy point sets for non-uniform measures, 2013. URL http://arxivv..r/bbbbaa //////b//1308.54// 5049
Alex Yu and Sara Pridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks, 2021.
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. 1CCV, 2023.
Francesca Bartolucci, Emmanuel de Bézenac, Bogdan Raonić, Roberto Molinaro, Siddhartha Mishra, and Rima Alaifari. Are neural operators really neural operators? frame theory meets operator learning, 2023.
Kaushik Bhattacharya, Bamdad Hosscini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. ar preprint orXic22005003880 2020.
Alexandre Boulch. Generalizing discrete convolutions for unstructured point clouds. CoRB, abs/ 19044.09777 2019. URL http://arxxvv.rr//bb////e 02375
Luca Brandolini, Leonardo Colzani, Giacomo Gigante, and Giancarlo Travaglini. On the koksma-hlawka inequality. Journal of Complexily, 29(2):158 172, 2013. ISSN 0885-064X. doi: jco.2012.10.000 URL
Russel E Caflisch. Monte carlo and quasi-monte carlo methods. Acta numerica, 7:1 7:1-49, 1998.
Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pottern recognition, pp. 5799-5809, 2021.
Tianping Chen and Hong Chen. Approximations continuous functionals by neural networks with application to dynamic systems. IEEE Transactions on Newral networks, 4(6):910 0-11,,18111111 1993.
Tianping Chen, Hong Chen, and Ruey wen Liu. Approximation capability in c(r /sup n/) by multilayer feedforward networks and related problems. IEEE Transactions on Neural Networks, 6(1):25-30, 1995. doi: 10. 10.1109/72.363453
Gong Cheng, Junwei Han, Peicheng Zhou, and Dong Xu. Learning rotation-invariant and fisher discriminative convolutional neural networks for object detection. JEEE Transactions on Image Processing, 28(1):265- 2018.
J Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Streamable neural fields. arXiv preprinl (xXx::2277.09665 2022.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference machine learning, pp. 2990 2999. PMLR, 2016a.
Taco Cohen and Max Welling. Steerable cmns. arXiv preprint arXiv:1612.08498, 2016b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Abril Corona- Figuero,, Jonathan Frawley, Sam Bond-r Sarath Bethapudi, Hubert P. Н. Shum, and Chris G. Willcocks. Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray, 2022. URL bistps://arxin wwtp:oogg//./...////////
J. Deng, W. Dong, R. Socher, L.-J. Li, Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
18

---

Published in Transactions on Machine Learning Research (10/2023)
Emilien Dupont, Yee Whye Tch, and Arnaud Doucet. Generative models as distributions of functions. CoRR, abs/2102.04477,, 2021. URL https:///axxii htttp:://axxi..orr/bbb////////
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point function and you should treat like one. arXiv preprint arXiv:2201.1200 2022.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL https://gi http:///ggthho..c.oooooccca//rrr con////atrryllb/rrouusesses
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels. CoRR, abs/1711.0892 089200, 2017. URL htttt://axxxiii/ http:///axxvvv../abbeee/ wwt.ooogg///aa/oooeeeeee 17811089111000
Ruohan Gao, Yun-Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A datasel of objects with implicit visual, auditory, and tactile representations. In CoRL, 2021.
Ruohan Gao, Zilin Si, Yun-Yu Chang, Samuel Clarke, Jeannette Bohg,, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Objectfolder 2.0: A multisensory object dataset for sim2real transfer, 2022. URL https 02389
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Newral networks, 4(2):251 257, 1991.
Chiyu 'Max' Jiang, Dequan Wang, Jingwei Huang, Philip Marcus, and Matthias Nießner. Convolutional neural networks non-uriform geometrical signals using euclidean spectral transformation, 2019. URL http::///axxiv.og/bbbbb////
02070
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https:
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M. Preciado, George J. Pappas, and Paris Perdikaris. Learning operators with coupled attention, 2022.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art No, 2021a.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. ar preprinl 0rXiv:210S.08/58 2021b.
Abbijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Taglissacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: A semantic object-aware neural scene representation, 2022. URL
04334
Samuel Lanthaler, Siddhartha Mishra, and George E Karniadakis. Error estimates for DeepONets: a deep learning framework in infinite dimensions. Tronsactions of Mathematics and Its Applications, 6(1), 03 2022. ISSN 2398-4945. doi: 10.1093/imman /imattmmtnacc01....... URL https:://oi..rrg// 1et110999/mmaa 10993//mattm/tnac0001 tnac001.
Jac Yong Lee, CHO Goong and Hyung Ju Hwang. Hyperdceponct: learning operator with complex target function space using the limited resources via hypernelwork. In The Eleventh International Conference on Learning Representations, 2022.
Zongyi Li, Nikola Kovachki, Kamyar Aziwadenesheli,, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprinl ar Xix::22010. 088955 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigode Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, 2020b.
19

---

Published in Transactions on Machine Learning Research (10/2023)
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. CoRR, abs/2201.0354// 2022. URT htttp:://arxww.or///abaaa .3545
Ily Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/ 0bs/1771.05101, 2017. httbs//711.000.
URL http://arxiv
Lu Lu, Pengzhan Jin, Guolei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via. DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, mar 2021a. doi: 10.003//11356-01-00222. URL htttps//ddi.oorg110 1038/2Fs42256 0010:/22242256-01--0000000 021-00302-5
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-22) 229, 2021b.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural information processing systems, 30, 2017.
Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-RRdd On the benefits of invariance in neural networks. CoRR, abs/ (bs//200500117 2020. URL
00178
Julien NP Martel, David B Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02780 2021.
Ishit Mehta, Michaël Gharbi, Comnelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chan- draker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14214 14223, 2021.
Lars M. Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. CoRR, abs/11812.03828, 2018. URL http:
03828
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. CoRR, abs/2000 0003308934, 2020.
URL https://arxivv www./r//gggg//oooooee//e////////
M. Muckley, R. Stern, T. Murrell, and F. Knoll. TorchKbNuftt: high-level, hardware-agnostic non- non-uniform fast Fourier transform. In ISMRM Workshop on Data Sampling & Image Reconstruction, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. (xXi:22211.0958, January 2022.
Norman Müller, Andrea Simonelli, Lorenzo Porzi, Samuel Rota Bulò, Matthias Nießner, and Peter Kontschieder. Autorf: Learning 3d object radiance fields from single view observations, 2022. URL
attps://arxiiy
http:://aaaiivvoorra//////////// wtwg/rrr/////////o/ 03593
Michael Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5379-5388 2019.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. IEEE Conf. on Computer Vision and Pallern Recognition (CVPR), 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard A. Newcombe, and Steven Lovegrove. Decpsdf: Learning continuous signed distance functions for shape representation. CoRR, Ahs/1190005103, 2019. URL
/ttt:://arxiv.oo wwarxiv..or/abbb. www./ogg........ htg//9bb/1901..... 05103
Michael Prasthofer, Tim De Ryck, and Siddhartha Mishra. Variable-input deep operator networks, 2022.
20

---

Published in Transactions on Machine Learning Research (10/2023)
Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnct++:: Deep hierarchical feature learning point sets in metric space, 2017. URL
httg///b/1170.... 02413
David W. Romero, Robert-Jan Bruintjes, Jakub M. Tomczak, Frik J. Bekkers, Mark Hoogendoorn, and Jan C. van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. CoRR, abs/20 20100000599 2021a. URL http://aaxxii..ogabb//////a 08059
David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub M. Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. CoRR, abs/2102.02611 2021b. URL https://arxiv org/ 2abs/2202.026111 02611
Jacob H. Seidman, Georgios Kissas, Paris Perdikaris, and George J. Pappas. Nomad: Nonlinear manifold decoders for operator learning, 2022.
Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov. Diffusionnet: Discretization agnostic
learning on surfaces. ACM Transactions on Graphics (TOG), 41(3):1-16, 2022.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-strucctur--wware neural scene representations. CoRR, abs/1906.01618, 2019. URL http://arxiv.org/ 0bbs1190.00161
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein, Metasdf: Meta- learning signed distance functions. CoRR, abs/2006.09600 09000, 2020a. URI, https: pottps://arxxe //arxiv.. wwwpexiii.ooo/aboaee org/abs/2006 09662
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. CoRR, abs/2006.0966) 2020b. URL
httr:://aaxiv..orr/abb//226
wrgg/bs//////// 2ttg/aab//00000966111
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Frédo Durand. Light field networks: Neural scene representations with single-evaluation rendering. CoRR, abs/2106.0200 026644 2021.
URL httpss//arra...og//a///10......
Cheng Sun, Min Sun, and Twann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction, 2021. URL
Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, and Sanja Fidler. Variable bitrate noural fields. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-9, 2022.
Mingxing Tan and Quoe V. Le. Efficientnel: Rethinking model scaling for convolutional neural networks. CoRR, abs/1905. 11946, 2019. URL http::/arax. www.://owwwbbbb..1....m
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, and Ren Ng. Learned initializations for optimizing coordinate--ased neural representations. CoRR, 0ttp/2012.221100 2020a. URL //tpttteeee//aaa..oorr//////////
wtgp/bs////a.//aaaa 02189
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. CoRR, Abs//2006.1733) 2020b. URL https:
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continmous convolutions. In International Conference on Learning Representations, 2019.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mchdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, 2021.
21

---

Published in Transactions on Machine Learning Research (10/2023)
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. CoRR, ////2111.007742 2021. URL httpp::/arxiv.. org/abs/ 2101.06742
Dejia Xu, Peihao Wang, Yifan Jiang, Zhiwen Fan, and Zhangyang Wang. Signal processing for implicit neural representations, 2022. URL httpsss//arr...or///a/////.....0 2270008772
Dmitry Yarotsky. Universal approximations invariant maps by neural networks. Constructive Approzimation, 55(1):407-474, 2022.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexan- der J. Smola. Deep sets. CoRR, abs/1703.06114, 14, 2017. CRL httppp///aaaiv...rraa////O/.... 06114
Aaron Zweig and Joan Bruna. functional perspective on learning symmetric functions with neural networks. In International Conference on Machine Learning, pp. 13023 13032. PMLR, 2021.
22

---

Published in Transactions on Machine Learning Research (10/2023)
Appendix
Appendix A provides additional background on the variation of a function and discrepancy of a point set, well as more general forms of DI layers. Appendix B provides detailed proofs of the Universal Approximation and Convergent Empirical Gradients theorems, as well as some corollaries. Appendix C provides additional examples of DI layers that extend convolutional neural networks and vision transformers.
A More Details on Discretization Invariance
A.1 Koksma-Hlawke inequality and low discrepancy sequences
Recall that a function f E (2) satisfies a Koksma Hlawka inequality if for any point set X S
Σ f(x') - 1(z))- x'EX
f(x) do SV()
(20)
for normalized measure dr, some notion of variation V of the function and some notion of discrepancy D of the point set. The classical inequality gives a tight error bound for functions of bounded variation in the sense of Hardy-Krause (BVHK), a generalization of bounded variation multivariate functions on [0, 1) which has bounded variation in each variable. Specifically, the variation is defined
ga Ora ((aa))))))))))))))())) dx,
Vnk(J)=
Σ aEf0,11a 10,1
(1
with {0, 1jdd the multi-indices and za E [0, 1jdt such that Tois = Ij if E a and Fo,f = 1 otherwise. The classical inequality also uses the star discrepancy of the point X, given by:
Σ 1,(2,) 1,(x)) x(D) x'EX
(*
sup TEJ
(22)
X
where J is the set of d-dimensional intervals that include the origin, and À the Lebesgue measure.
A point set is called low discrepancy if its discrepancy is the order of O(In NY Quasi-Monte Carlo calculates the sample mean using low discrepancy sequence (see Fig. A.1 for examples in 2D), as opposed to the ii.d. point set generated by standard Monte Carlo, which will generally be high discrepancy. Because the Koksma- Koksma-Hlawka inequality is sharp, when estimating the integral of a BVHK function on [0, 1] the error of the QMC approximation decays as () in contrast to the error of the standard Monte Carlo approximation that decays as ONN-1/2) Caflisch 1998
Low discrepancy sequences
High discrepancy sequences
Sobol
(2,3)-Halton
I.I.D. samples
Uniform grid
Figure A.1: Examples of low and high discropancy sequences in 2D.
However, BVHK is a rather restrictive class of functions defined on [0, 1]d that excludes all functions with discontinuttess. Brandolini al. (2013) extended the Koksma-Hlawe inequality to two classes of functions defined below:
23

---

Published in Transactions on Machine Learning Research (10/2023)
Piecewise smooth functions Let be smooth function on [0. 1]" and Ω Borel subset of [0, 1] Then Il is piecewise smooth function with the Koksma Hllakaa inequality given by variation
v()=
Σ 2d11al
L 10.11
O 1aam//rad00 S(x) de,
00
(23)
aE(0,1}
and discrepancy:
D(X) 2d sup IC
(anu(ay)) (())))))(((nn)) x'E
Σ
(24)
[
Wd,1 functions on manifolds Let M be a smooth compact d-dimensional manifold with normalized
measure dr. Given local charts {Ør]Æ=1 ds : [0, + M, the variation of a function f w..1 is characterized as:
v(f) Σ
ΣE Jon (a Sn 10.11
K
80 dra (vL(0L(()))(()(())) dz,
(25)
with {W*JA= smooth partition of unity subordinate to the charts, and constant > 0 that depends on the charts but not on f. Defining the set of intervals in M as J = {U U = Φx (I) for some and I I [0,1]*) with measure p(U)= = A(1), the discrepancy of a point set Y {8, د on Ml is:
D(Y) = sup UEJ
Σ ZEX
Iv(w,) m(()
(26)
X
Neural fields We define the variation of neural field as the sum of the variations of cach channel.
The notion of discrepancy is not limited to the Lebesgue measure. The existence of low discrepancy point sets has been proven for non-negative, normalized Borel measures on [0, 1] due to Aistleitner & Dick (2013). An extension of our framework to non-uniform measures is a promising direction for future work (sec Appendix
Note:
A.2 More General forms of DI layers
Recall that we defined DI layers as having the form Holf] = Sn Hz(f) for neural fields f (we drop 0 here for readability). In the case where H&lf]: Ω + RR, i.e. Hi layer that maps neural fields to vectors, we permit layers of the following more general form:
Hols] = - hola, ho) (x))..., ((:)dd)((()),
(27)
for weak derivatives up to order |a| taken with respect to cach channel. D "..""... for multi-index The dependence of on weak derivatives up to order h == |o| requires that the weak derivatives are integrable, i.e,, f is in the Sobolev space (*))( and that hs is Gateaux differentiable w.r.t. these weak derivatives. Note that a non-uniform measure y changes the discrepancy of sampled sequences.
8 g
In the funution-valued case, we can similarly have:
Holf((z') ( Holf((z,
(28)
= [ I(( o' DO ((d)))d()))),
(29)
where we require Holf](.,x') E Fu. Both of our key theoretical results (convergence of discretized gradients and universal approximation) apply to this general form. Gatcaux differentiability of ha allows us to apply the same proof in Appendix B.2 the derivatives. Since allowing layers depend on weak derivatives results in more expressive class of DI-Nets, the universal approximation theorem still holds.
24

---

Published in Transactions on Machine Learning Research (10/2023)
B Proofs
B.1 Proof of Lemma 3.2 (convergence of discretized gradients w.r.t. inputs)
Here we combine the function to vector and function to function cases for brevity. For fixed ă E 9 the discretized derivative of Ho w.r.b. (() can be writlen:
Proof.
0
H
E
Holf(()))
(30)
af(+)
(f(})
[NN]
zEXN
lim
1
Σ TEXN
Holf +T Holf((x),
(31)
XN
where di is any function in Wlo. that is 1 at and 0 on Xv\(a,, and whose derivatives are on Xx. As an example, take the bump function which vanishes outside a small neighborhood of and smoothly ramps to on smaller neighborhood of ž, making its weak derivative at
By (3) we know that the sequences ||
| [s] Holl and ANNI + + Holf + T converge uniformly
in N to 0 for any r > 0, where we can use the & norm for vector outputs or the L norm for function outputs.
So for any e > 0 and any T > O, we can choose No large enough such that for any N > No:
|| A + + Holf + Tut || < 2
c
(32)
and
AN - Hollsl <
(33)
Then,
H + T Hol + + + MNN Holl <E:
(34)
by the triangle inequality,
(AN [f+ + r Hol + + - (F2 Holt1) /| <c
(35)
|| (AN + + - (1 (Hols + + - Ho(l))
<,
(36)
hence || (A + + H [11) (Hols 15 + + -
Ho(li
converges uniformly to O. Since the distance
between two vectors is 0 iff they are the same, we can write:
lim AN + 1 AN = lim HolF + - -Ho[s] N
(37)
N-x
lim lim AN + T - R2 = -lim 7-
lim (Holf + 1 - HollT)
(38)
N
7-0 N
By the Moore- Osgrod thcorem,
lim N-000
lim
- ( + T - (11)
lim lim = (Holf + Tu -Holf)
1
(39)
N 7-0 7
lim
MN [f]
lim dHasfi W1 N
(40)
N -00
Since hy is Gateaux differentiable and bounded, Hs is also Gateaux differentiable for f of bounded variation, hence the limit on the right hand side is finite.
For each discretization XN, choose sequence of bump functions around each x E XN, (v- >ZE An example of such a family is the (appropriately designed) partitions of unity with [Xx] elements.
Then the discretized gradient converges to the limit of the Gateaux derivatives of Hs w.r.t. the bump function sequence N x.
25

---

Published in Transactions on Machine Learning Research (10/2023)
B.2 Proof of Theorem 3.3 (convergence of discretized gradients)
A DI-Nel permils buckprepagalion with respect to ils inpul and all ils learnable paromelers. The discrelized gradients converye under any equidistributed discretization sequence.
We note that this property automatically holds if the layer does not perform numerical integration. This includes layers which take Rn as imput, as well as point-wise transformations. Then the (sub)derivatives with respect inputs and parameters need only be well-defined at each point of the output in order enable backpropagation.
Choose equidistributed discretization sequence (NN Ω. We consider DI layer Ho which takes function f as input and may output vector function.
Holf = J Holfl())dr.
(41)
Recall its discrete operator under Xx:
AN XN EEXx
Σ Holl()))
(42)
Lemma B.1. Chained discretized derivatives converge to the chained Gateaur derivatives.
Proof. Consider a two-layer DI-Net with function input f + (Ha o Ho)l[]- For the case of derivatives w.r.t. the input, we would like to show the analogue of (40)
lim N N
HN
H
[f] = lim d (Ho Ho) [f: d
(43)
Of(T)
N
where the bump function wir is defined similarly (1 at ä and 0 at each x + i).
HH o AN [ 0f(x) He 1 (z)
(44)
1 (Hl H + TV (x) Ho [ANL1]] (x)
lim 7
(45)
as in 31
By 3
know 1 iI + T
AN iINis + ToNN converges to 0 in N for all the l for vector outputs the L' norm for function outputs), does Reasoning as in -29 have:
0
(where
Hy A [1]
AW A [f]
lim lim
1 HN
A [f + T
Ag AC [il
(46)
N
lim
1
lim
Ho R2W [s + r
Ho FRE [rl))
(47)
N
= lim lim
(Hg [A [s + -Wa
Ho [ Ln])
(48)
No00 7-50
Note that
aH4lf:031 = (Holf + T [6 + o(T))
(49)
Holf + T = =Hoof TdH. +o(T).
(50)
26

---

Published in Transactions on Machine Learning Research (10/2023)
Then we complete the equality in (43 as follows:
LHS = lim
(H o H [f]
(51)
N Of(ā)
1 (No [ [ + rw3il| Ho (28111)
lim lim
(52)
No 7-+0 T
1
lim lim (Ho [Hols] rdHooll:011 Ho (H6s (f1)
(53)
N-+00 7 T
lim dHi [Holf): dHolf:
(54)
N
lim d (Hg Hop) [f N
(55)
= RHS,
(56)
by the chain rule for Gateaux derivatives.
The case of derivatives w.r.t. parameters is straightforward. In the same way we used we have:
32
39 to obtain
2 (H
1 (RL [ [n] .PN [AL (11111))
lim -0000
H
lim lim No000 r-0
(57)
dΦk
lim (Ho [[sTTex(f]] Ho(1
(58)
T-0 r
(Ho Ho)lf),
(59)
By induction, the chained derivatives converge for an arbitrary number of layers.
Since the properties of DI-Net layers extend to loss functions DI-Nets, treat loss function similarly to layer. For a loss on model output and optional ground truth label g':
Log = 1 L(g, g' (x))dx,
(60)
with discrete operator:
2 [ol
1
Σ
Llg. Llg.g'(xx),
(61)
[Nvl
ZENN
we can write:
à +00 0s(T)
lim
(Ny H. H
[f]
lim d (C. Ho Ho) [f:ve].
(62)
N
So by Lemma 3.2 and B.1 have shown that backpropagation is discretization invariant.
B.3 Proof of Theorem 4.1 (Universal Approximation Theorem)
Note: By our definition of F. (Section 5.1 there exists V such that every f E Fi satisfies Loksma-Hlawike inequality 3 with V(I[l) < V*.
Fi is bounded in L norm since all their functions are compactly supported and bounded.
Consider a Lipschitz continuous map R Fi + Fi such that ((R])) S Mod(ff g)L for some constant Mo and all f.g E Fi. Let M max(Mo, 1}.
Fix discretization X Ω with discrepancy D(X) 12(M+2/V By 3 this yields:
|X
f(x)) / [(x)) dx
(63)
12(M + +2)
Z'X
for all f E Fi. Let N be the number of points in X.
27

---

Published in Transactions on Machine Learning Research (10/2023)
Definition B.2. For given discretization X, the projection x : + f is quotient map L2 - (2(11//~ under the equivalence relation / ~~g iff f(x) g(x) for all a E X.
I. (9t//~ is isomorphic to RIxi, and thus can be given the normalized e norm:
Σ If(z)) x'E
(64)
X
Definition B.3. Denote the preimage of π as π [f f'}. define the inverse projection x- mfii + Fi by a choice function over the sets π (
Invoking the axiom of choice,
Note that this inverse projection corresponds to some way of interpolating the sample points such that the output is in F1. Although our definition implies the existence of such an interpolator, we leave its specification as an open problem. Since only permits discontinuities along a fixed Borel subset of [o, 1]4, these boundaries can be specified a priori in the interpolator. Since all functions in Fi are bounded and continuous outside this set, the interpolator can be represented by bounded continuous map, hence it is expressible by DI-Net layer.
Definition B.4. generates g-algebra on Fi given by s/ = (x S L). with L the o-algebra of Lebesgue measurable sets on R.V. Because this o-algebra depends and the Lipschitz constant of R via the point set's discrepancy, we may write it as d.R.
In this formulation, we let the tolerance e and the Lipschitz constant of R. dictate what subsets of Fi are measurable, and thus which measures on F1 are permitted. However, if the desired measure v is more fine-grained than what is permitted by d.R, then is v that should determine the number of sample points N, rather than or R.
We now state the following lemmas which will be used to prove our universal approximation theorem. Lemma B.5. There a map R TF + nFi such that
(R(((=)) o R [[](()) dr =
(65)
n
6'
Proof.
Let g(x)= = |s(x)) for / E Fi. Because
63
applies to g(x), we have:
Σ g(xa) - g(x) dr 1 12(M + 2)
(66)
X
x'EX
|| - -11fl12
5 12(M 2)
(67)
Eqn. 67
also implics that for any f E TF,, we have:
|ielle - -|| (frl)lll)) 5
(68)
12(M + 2)
Combining
67 and
688
obtain
| -
o 5
(69)
6(M 2)
By the triangle inequality and applying R:
/ R(f(x)) - -1 om ([f((() dr S 6(M + 2)
(70)
28

---

Published in Transactions on Machine Learning Research (10/2023)
For any f.g F1 such that xf Tg, 67 tells us that d(f.g) is at most c/6(M 2). Recall M was defined such that d(R[[]. R(ol)ra & Md(J: g. for any R.
ME
d(ι o R[f), R(g)) -
(71)
6(M + +2)
6(M + 2)
(M + 1)
(72)
(M 2)
So defining:
R = arg min d( o n fIf], a R[f))e.
(73)
x
we have
R T TO R[A] (M 1) c (M + 2) 6'
(74)
Then by 70
J RI)[(x) - -t R [To((f(<=)] dr S
(M + 1)
+
(75)
6(M + +2)
(M 2
(76)
6
Lemma B.6. Consider the extension of R to RN + RN in which each component of the output has the form:
R(m (z) IfETT
Rf(
(77)
0
otherwise.
Then any finite measure v on the measurable space (F1, induces a finite measure on (&N,2) and JRy (f)l(((d)) < oc for each j.
Proof. Since the o-algebra. a on Fi is generated by म, the measure μ () v(S) for all S E a is finite and defined w.r.t. the Lebesgue measurable sets on nFi. Since TFi can identified with a measurable subset of R.W, i can be naturally extended to R.N. Doing so makes it absolutely continuous w.r.t. the Lebesgue measure on N.N
To show (f) is integrable, it is sufficient to show it is bounded and compactly supported.
Fi is bounded in the L norm. Thus by 667 TF is bounded in the normalized & norm. The & norm in R is strongly equivalent to the uniform norm, so there is some compact set [-c,c]* > O for which the extension TF1 to R vanishes, so supp(R, (f)) S [-c,c]]
Similarly, TF is bounded in the ll norm, hence there exists d' such that R, < for all
Lemma B.7. For any finite measure a absolutely continuous w.r.t. the Lebesgue measure on R", L (μ) and 0, there is network K. such that:
[ (((F) (CF))K((f))ll))) <
(78)
Proof. The following construction is adapted from
Lu et al. 2017 Since J is integrable, there is a cube
E = c) such that:
R [[f))l(dd))
(79)
||J - 1 gg.
(80)
29

---

Published in Transactions on Machine Learning Research (10/2023)
Case 1: J is non-negotive all of R Define the set under the graph of J|æ:
( :fE U [0,
(81)
Gæ.J is compact in R=+1, hence there is finite cover of open rectangles {Rt satistying -(U; on R" Take their closures, and extend the sides of all reclangles indefinitely. This results in a set pairwise almost disjoint rectangles {Ri]. Taking only the rectangles {R: p(R, n Gg,J) 0} results in a finite cover satisfying:
n Σ
((R.) p(Gr. <
(82)
This implies:
m8
Σ (R,)) < [|]
(83)
and also,
RR
1 nn, (f,, J(f) ((df) + ||/
(84)
2
(f(
Ln. (f, (f)))
(85)
by the triangle incquality. For each R₂ = [as,, bal x (i.e., the projection of R₁ onto IR*). Then we have
Jain. Dinl (Sis Si + yil. let X, be its first n components
R Σ n (f)
f
J E
(86)
Let (f) A Ei xxIx((f) By the triangle inequality,
f (5(() (1(0) K(F)) =11J -1 + |
Y11 + + 1---I-- Y
(87)
c $ + ||AC ++ Yll
(88)
by 80 and 86 So it remains to construct AC such that ||A- i. Because 1x, is discontinuous at the boundary of the rectangle X,, it cannot be produced directly from DI-Net (recall that all layers are continuous maps). However, we can approximate it arbitrarily well with a piece-wise linear function that rapidly ramps from 0 to at the boundary.
For fixed rectangle X; and 8 E (0, 0.5), consider the inner rectangle X8 C X;:
X5 (a1 + 6(b, az), by 8(b) a1)) ... x (an + 5(b, an), 5(b, an)),
(89)
where we omit subscript j for clarity. Letting U/ b; 5(b: a;), define the function:
((f) ReLU(S ReLU(f; - b)) ReLU(6 ReLU(f, - -a:)]]
(90)
30

---

Published in Transactions on Machine Learning Research (10/2023)
where ReLU(2) = max(x, 0). T(f) is pieco-wise linear function that ramps from 0 at the boundary of X; to within Xs, and vanishes outside X,. Note that
1 - T|, x()) MM(xs) (1 (1-20)) ((X),
(91)
(92)
if fa is the Lebesgue measure. 5 may need to be smaller under other mcasures, but this adjustment is independent of the input so it can be specified a priori.
Recall that the function we want to approximate is (f) = E (f). We can build DI-Net layers
K:: f + K() = E!= (f), since this only involves linear combinations and ReLUs. Then,
n
|/C - 1111 =
Σ
Us (T.(f) - 1x, (f)) df
(93)
R
[||
Σ
1 xx, Tilll
(94)
1=1
|
< <(1 (1 28)
Σ Vin(X.)
(95)
[R]
(1 (1 2))
Σ
1u (R:))
(96)
< -1- (1 21- (IJ)1 + )
(97)
by 83 And so by choosing:
1 8 == 2
1 - 1 E ||5|| + 5 ))
1/n
(98)
we have our desired bound ||KC- Y < and thereby ||J KCIl, < 3.
Case 2: J is negative some region of TR
Letting J+( max(0, and J- (f) = ((()),, define:
G _ {(f, v) if (0,
(99)
GE A [(f, v) f E, (-))
(100)
As in 82 construct covers of rectangles Rt over G and R- over GE,J each with bound 16 and F"
projections X+, X- Let:
R
Y+( = Σ
vr 4 1x+ (f)
(101)
t=1
IR
(f)= = Σ
vr 1x (f)
(102)
==1
Y= -==+-Y
(103)
We can derive an equivalent expression
S6
iR
/
J(f)
Σ
v 1 x
(f) + Σ
31 1 x,
(f) df
(104)
8
E
i=1
L
-YlI.-
(105)
31

---

Published in Transactions on Machine Learning Research (10/2023)
Similarly to earlier, we use 80 and 105
to get:
(J()) - ((f)
4 + |IC YIl--
(106)
Choosing 77 (f) and T,7 (f) the piece-wise linear functions associated with X; and X,, and:
R
<(())
Σ
(f)
Σ D7
(T)
(107)
we have:
|
[ - 111
/
Σ V1
1x)
Σ y1
(T) - 1x- xx-(1) df,
(108)
R
2=1
t=
applying the triangle inequality,
|R*1
R
Vi
1x;
T
Vi
1x; T;
(109)
1=1
t=1
|R+
R
< (1 - (1 -8+)²*)
Σ
2 (x ) + (1 - (1- 28 )m)
Σ
Vi m(x,)
(110)
1=1
< (1 (1- (11228t77) (|
+ 16
+ 111-1111 (1- -1-100111 )0 (1 1
(111)
By choosing:
11, + 16) 1)=
1/23
8+ 1
(IJ
(112)
2
8
1/n
8- 2
é 8
[| + 16
(113)
and proceeding before, arrive at the same bounds |K Ylh < and ||- KCI| < 5.
implements the network logic for producing the function K. We can provide x with access to either through skip connections or by appending channels with the values {e + f- (which will be preserved under ReI.U).
Putting it all together, Algorithm I
Theorem B.8 (Maps between functions). For any Lipschitz continuous map R Fi + Fi, any e > 0, and any finite measure w.r.t. the measurable space (r,, there crists DI-Net T that satisfies:
f [[(()- - -(00- <e.
(114)
Proof. If v is not normalized, the discrepancy of our point set needs be further divided by max (M(F), 1}. We assume for the remainder of this section that v is normalized. Perform the construction of Lemma B.7 N times, each with tolerance of e/2N K, where K is the Lipschitz constant of R. Choose partition of unity { for which (x) = 1 [x+ arg miny'ex d(x, x')], and output N channels with the values (C, (.) By summing these channels we obtain a ner &C that fully specifies the desired behavior of R. R + N with combined error:
f [R((F) K((F) (e <
(115)
32

---

Published in Transactions on Machine Learning Research (10/2023)
Thus,
Σ - v(df) - 2
(116)
X
By 68 have:
/
Rox((((() 1
|
x-' - [(ooT((f((() dr r(df) 2
(117)
6(M + 2)
By Lemma 3.5 we have:
S/ R[f](x) -- Ko dx v(df)
< 2 6(M +2)
+ (118)
And thus the network T = = Ro gives us the desired bound:
/ [R(() - (0) ((d) <e.
(119)
Corollary B.9 (Maps from functions vectors). For any Lipschitz continuous map R. Fi + R", any 0, and any finile measure w.r.l. the measurable space (F1, S,R), there exisls DI-Nel T that salisfies:
JL - (( <e.
(120)
Proof. Let Mo be the Lipschitz constant of R. in the sense that [[R[J], Rig])e= S Mod(S.g) Let M max[Mo, 1}. There exists R TF1 + R" such that || x R[flll da S e/12. As in Lemma B.6 consider
the extension of R to RN R= in which each component of the output has the form:
R|
if otherwise.
R2(1)
(121)
0
Then for similar reasoning, v on Fi induces a measure a on R that is finite and absolutely continuous w.r.t. the Lebesgue measure, and JgN (f)(u((( oo for each j.
We construct N R approximation times with tolerance of e/2n, such that:
J (itelem
(122)
Applying
67 we find that the network = o gives us the desired bound:
f [I(()- -( <c.
(123)
Corollary B.10 (Maps from vectors to functions). For any Lipschillz contirvoous map R: R. F and any O, there erists a DI-Net T that satisfies:
[[R(x) - T(x) (2) <c
(124)
Rm
33

---

Published in Transactions on Machine Learning Research (10/2023)
Proof. Define the map R: R" - TFi RN by R T R. Since R is bounded and compactly supported, Jex (x) < oo for each i.
We construct a _ + IR approximation N times each with a tolerance of e/2N K with K the Lipschitz constant, such that:
1 R2 [R(x) - K(()) (o))
(125)
Applying
68) we find that the network T= T=T 11 o K gives the desired bound:
[[((x) - (<<))llooo <e.
(126)
Rm
Consider the space of vector-valued functions Fe = (f + -R da < <0, fi E Fi for each i}. Denote the norm on this space as:
] [f(x))d
||
(127)
Definition B.11 (Concatenation). Concatenation is map from two scalar functions fi, E Fi to the vector-valued function [fa, fyl E F2. The concatenation of vector-valued functions can be defined inductively to yield Fn x Fm - Fntm for any n, m N.
All maps Fr x F + Fe can be expressed as concatenation followed by map Futm + Fe. A map R" + Fm is also equivalent to m maps _ + Fi followed by concalenation. Thus, we need only characterize the maps that take one vector-valued function as input.
Considering the maps Fn Fm, we choose lower discrepancy point set on f such that the Koksma-Hlawkke inequality yields a bound of e/12mn [M 2). Let project each component of the input to =F1, and m- inverts this projection under some choice function. We take a' to be the product o-algebra generated from this _ {Ei .... x E.: E E. E ) where a is the o-algebra on Fi from Definition B.4
Corollary B.12 (Maps between vector-valued functions). For any Lipschilz continuous map R Fn Fm, any O, and any finite measure v w.r.t. the measurable space (Fn: AiRR). there crists DI-Net T that sotisfies:
/ [[R(S) - -()),)- <E
(128)
Our network now requires nN maps from RMN + E each with crror c/2mnN. Summing the errors across all input and output channels yields our desired bound.
Proof. The proof is very similar to that Theorem B.8
The vector-valle valued analogue of Corollary B.9 is clear, and we state it here for completeness:
Corollary B.13 (Maps from vector- valued functions to vectors). For any Lipschitz continuous map R F. + Rmh any O, and any finite measure u w.r.t. the measurable space (Fn, F',x) there cxists DI-Net T that satisfies:
f F. [[R(f) - ( <e
(129)
C Pixel-Based DI-Net Layers
Here we present a number additional DI layers that show how to generalize pixcl-based networks (convolu- tional neural networks and vision transformers) to DI-Net equivalents. Many of the following layers were not directly used in our experiments, and leave investigation of their properties for future work.
34

---

Published in Transactions on Machine Learning Research (10/2023)
Max pooling There are two natural generalizations of the max pooling layer to collection of points: 1) assigning each point to the maximum of its k nearest neighbors, and 2) taking the maximum value within a fixed-size window around each point. However, both of these specifications change the output's behavior as the density of points increases. In the first case, nearest ncighbors become closer together so pooling occurs over smaller regions where there less total variation in the NF. In the second case, the empirical maximum increases monotonically as the NF is sampled more finely within each window. Because we may want change the number of sampling points on the fly, both of these behaviors are detrimental.
If we consider the role of max pooling then it is sufficient to use a fixed-size window with some scaling factor that mitigates the impact of changing the number of sampling points. Consider the following simplistic model: assume each point in a given patch of an NF channel is an i..d. sample from u[[-6, bl). Then the maximum of N samples {f((zz)]í= is on average 6. So we can achieve "unbiased" max pooling layer by taking the maximum value observed in N-1 (if N = 1 or our empirical maximum is negative then we simply return each window and scaling it by N+1
layer that shuttles gradients through strong local activation,
the maximum), then (optionally) multiplying constant to match the discrete layer.
To replicate the behavior of discrete max pooling layer with even kernel size, we shift the window by half the dimensions of pixel, just as in the case of convolution.
Tokenization A tokenization layer chooses finite set of non non-overlapping regions wy C Q of equal measure such that Ujay = Ω. We apply the indicator function of each set to each channel fi. An embedding of each fils, into R" can be obtained by taking its inner product with a polynomial function whose basis spans each (w)))) To replicate a pre-trained embedding matrix, we interpolate the weights with B-spline surfaces.
Average pooling An average pooling layer performs continuous convolution with box filter, followed by downsampling. To reproduce discrete average pooling with even kernels, the box filter is shifted, similarly to max pooling.
An adaptive average pooling layer can be replicated by tokenizing the NF and taking the mean of each token to produce a vector of the desired size.
There are various ways to replicate the functionality of an attention layer. Here we present an approach that preserves the domain. For some dę e consider a self attentionn layer with dn parametric functions qú L22),,, ands parametric functions kij E L22(),, and a convolution with da output channels, produce the output NF g as:
Attention layer
Qj (gijs fi)
(130)
K₁ = (kis, Ji)
(131)
vinl = _ Uij fi + +b₁
(132)
QKT
(xx) = tmax
v(f((x)
(133)
Vilk
A cross-attention layer generates queries from a second input NF. A multihead-attennion layer generates several sets of (Q, VVV triplets and takes the softmax of each set separately.
Data augmentation Most data augmentation techniques, including spatial transformations, point-wise functions and normalizations, translate naturally to NFs. Furthermore, spatial transformations are efficient. and do not incur the usual cost of interpolating back to the grid. Thus DI-Nets might be suitable for a new set of data augmentation methods such as adding Gaussian noise to the discretization coordinates.
Positional encoding Given their central role in neural fields, positional encodings (adding sinusoidal functions of the coordinates to each channel) can help bxsel-based DI-Nets learn high-frequency patterns under range of discretizations.
35

---

Published in Transactions on Machine Learning Research (10/2023)
Vector decoders (R" Fe) and parametric functions in Fe A vector can be expanded into an NF in several ways. We can create an NF that simply places input values at fixed coordinates and produces values at all other coordinates by interpolation. Alternatively, we can define a parametric function that spans Fo using the input vector as the parameters, for example by taking as input n numbers and treating them as coefficients of the first. n elements of an orthonormal polynomial basis on Ω. If is a subset of bj, one can use a separable basis defined by the product of rescaled 1D Legendre polynomials along each dimension. Ω is d-ball, we can use the Zernike polynomial basis. For general coordinate system, & small MLP could be learned
used where R" can represent its parameters of its parameters. Beyond using such parametric functions vector decoder layers, they also give rise to n-parameter layers that compute an inner product ("learned global pooling layer") or elementwise product ("dense modulation layer") of an input NF with the learned functions.
modulation (Dupont et al. 2022
Warp layer Layers that apply a self- homemoophhim q on Ω (a bicontinuous map from + Q2) preserve discretization invariance since it simply modifies the upper bound of the invariance error subsequent layers use a discrepancy of q(X) rather than discrepancy of X.
Algorithm 2: Classificr Training
Input: network To, dataset D, classifier loss C, imput discretization X for step s E 1: Nateps do
Neural fields fi. labels y/ <- minibatch(D)
Label estimates fi <- To X] Update & based on Vg yi) end
Output: trained network To
Algorithm 3: Dense Prediction Training
Input: network To, dataset D with coordinate-label pairs, task-specific loss L
for step s 1: Nsteps do
Neural fields fi, point labels (æij. Myy) minibatch(D) (
Point label estimates Hi Telfi: {aty N3
Update 9 based on VgL(drg vy)
end
Output: trained network To
36


Reviews:
[{"review_id": "2Y3kTfRtuh", "reviewer": "Reviewer_Pe3A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The direction is important and timely, particularly with the prominent rise of neural field models in many applications.\n- The paper includes a thorough and precise mathematical treatment.\n\n**Weaknesses**\n\n- While these are not supposed to be criteria for TMLR, I'll mention this anyway: I am not very positive about this paper in terms of either novelty or impact.\n    * In terms of novelty, there are similar treatments in the (recent) literature on understanding and trying to mitigate out the effects of discretization, particularly in the context of operator learning. Why is a new framework needed at all? \n      The authors state that (p1) _\" a general theory of discretization invariant deep learning has not yet been explored ...\"_ and point to Kovachki et al. (2021) as an exception, but this paints a woefully incomplete picture since there is a wide body of literature on this topic. See, for example, Bartolucci et al., \"Are Neural Operators Really Neural Operators?\" (and also the numerous other papers referred to in this work.) The representation-equivalent neural operator (ReNO) framework should serve as a starting point.\n    * In terms of impact, the main contributions are nice but somewhat uninteresting. (This is perhaps a byproduct of being too general a framework.)  \n          - Universal approximation: fine, but not surprising; the proof of Theorem 4.1 is AFAIK the usual rectangle-covering idea employed in UAT-style proofs. If the authors truly believe FNOs or attention layers can guide further design of DI-Nets they should investigate this further in detail.\n          - Implementation: again, fine but didn't see too many new insights. Algorithms 2 and 3 are utterly uninformative and I would either remove them or restate in a more interesting way.\n           - Experiments: all shown on super-shallow networks. Image classification and segmentation are anyway such mature problems that the techniques here are probably never going to be used. I'd recommend focusing on applications where discretization effects play a bigger role, and invariance to discretization is a key criterion (perhaps turbulent fluid flows? I am not sure).\n\nTherefore, I give a \"no\" answer to the \"Audience\" question below, but am willing to debate this with the authors. \n- I would have also liked to see a more honest discussion on the framework's limitations. It is impossible to achieve true, universal discretization-invariance in practice, since at some point effects with quantization and spatio-temporal resolution kick in. Where does the framework break down, and what is a likely path forward in such cases?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents DI-Nets, a framework for building in \"discretization-invariance\" into neural fields. Such models are particularly relevant in a bunch of different application settings, including high-fidelity computer graphics. The key technical contributions are: \n- a universal approximation-style result for a large family of discretization-invariant networks\n- analysis of backprop for such networks\n- experimental demonstration of their capabilities in classification and segmentation\n- experimental demonstration of \"invariance\" .", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "**(Critical)**\n\n- Please engage with the (theoretical) literature on continuous operator learning better (see Bartolucci et al above and many references therein). Motivate why your new framework is needed; e.g. how the point set discrepancy idea solves some of the issues in earlier work, practical aspects of implementing this idea and how you resolved them, etc.\n- Consider including at least one motivating application where discretization invariance plays a key role; I don't believe segmentation on Cityscapes fits this setting tbh.\n\n**(Other)**\n\n- Consider highlighting limitations and barriers to \"true\" discretization equivalence, and possible ways forward.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xz8RbtvTv1", "reviewer": "Reviewer_13K2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper was well-motivated and well-written with good structures.\n2. The proposal is a new framework for designing and understanding discretization invariant neural networks (DI-Nets) that can learn maps between functions on continuous domains without being sensitive to how the function is sampled. The authors establish upper bounds on the deviation in model outputs under different finite discretizations and highlight the central role of point set discrepancy in characterizing such bounds. \n3. In contrast of theortical understandings, authors design a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy, which universally approximate a large class of maps between integrable function spaces. The use of low discrepancy sampling can guide an initial discretization, and adaptive sampling techniques such as adaptive quadrature or sphere tracing can be used to refine the discretization\n4. They also show that DI-Nets can be applied to neural fields to learn to classify and segment visual data under various discretizations and sometimes generalize to new types of discretizations at test time. The proposed soluton outperform baseline algorithms on neural field classification, neural field segmentation, signed distance function prediction tasks with certain margin.\n\nWeaknesses\n1. The proposed solution combines neural networks with numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. I am wondering if it is possible to use neural ODE-based solution or other NN-based solver to infer the numerical integration directly, rather than performing MCMC-based integration. In this, authors can form the solution as an end-to-end NN-based solution.\n2. MCMC-based solution is considered to be inefficient, in terms of lattency or time consumption, especially when the number of dimensions is high. I am wondering if it is possible to report and analyze the time consumption for the field generation versus resolutions and spaces.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the problem of discretization invariant learning for neural fields ( a kind of continuous data). Authors proposed a new framework for designing and understanding discretization invariant neural networks (DI-Nets) for learning maps between functions on continuous domains. The analysis establishes upper bounds on the deviation in model outputs under different finite discretizations and highlights the central role of point set discrepancy in characterizing such bounds. The authors design a family of neural networks driven by numerical integration via quasi-Monte Carlo sampling with discretizations of low discrepancy. The DI-Nets can universally approximate a large class of maps between integrable function spaces, and show discretization invariance in backpropagation through such models. The article demonstrates the effectiveness of DI-Nets in learning to classify and segment visual data under various discretizations, and sometimes generalize to new types of discretizations at test time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address my concerns in weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9ubDxuJIDx", "reviewer": "Reviewer_agB5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. As the authors point out, the key advantage of learning a DI network over neural fields as opposed to using a rendered image or using the neural field model parameter as inputs is that the framework becomes parameterization agnostic. This allows the model to work across different neural field parameterizations and discretizations. This is a useful property.\n\n2. Experiments investigating several aspects of the proposed methodology and problem space.\n\n3. Detailed analysis covering different aspects of the prediction problem when dealing with NFs.\n\n\nWeakness/Concerns:\n\n1. Since number of point sampled should grow quadratically with the image resolution, does this imply the complexity is quadratic w.r.t. the image resolution?\n\n2. It would be helpful to discuss the complexity of the proposed method with methods that do not operate on the NF samples, but rather work either on the rendered image or the NF model parameters. This discussion is currently missing.\n\n3. One comparison that I think is important but missing in the paper is to compare the performance of DI-Net with existing applicable methods when training and test time discretizations are different.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a simple methodology for learning a mapping function that takes a neural field as input and maps it to either another neural field (regression) of a discrete target (classification). The main benefit of the proposed methodology compared with existing techniques is that existing techniques either directly use the model, that encodes the neural field, as the input to the classification/regression model, or they use some discrete sampling of the neural field as the input. The former technique has the limitation that it cannot work across different encodings, while the latter does not generalize beyond the specific discretization used for discretizing the input. \n\nAlong with the proposed methodology, the authors also studies various aspects of discretization invariance of the proposed mapping function in the context of neural fields-- \n\n1. they define a formal notion of discretization invariance in terms of the variation of the underlying function and the discrepency of the discretization (which intuitively measures how far the discrete samples are from their continuous counterpart). Specificaly, absolute difference between the integral estimate and the sample estimate is upper bounded by the product of the variation function and the discrepancy function. This bound establishes the condition in which the discretization map is convergent, i.e., the absolute difference approaches zero, leading to discretiztion invariance. This condition requires the discretization to be equidistributed, i.e., the discrepancy function asymptotically tends to zero.\n\n2. they also show that under such conditions, the gradients of the mapping function w.r.t. its parameters are also convergent, i.e., the gradients computed using the integral form and the discretized form are asymptotically identical.\n\n3. they show that the proposed Discretization Invariant (DI) networks are universal approximators when the discretization method has a small enough discrepancy.\n\n4. they discuss existing commonly used neural network layers that can be used as part of the proposed DI network.\n\n5. they discuss the computational complexity of the proposed framework and claim that the complexity scales linearly with the number of points sampled in the discretization process. \n\n\nFinally, the authors conduct experiments covering various aspects of the proposed methodology:\n1. Classification performance of DI-Net across different resolutions.\n2. Classification performance of DI-Net comparision when training and test time discretizations are different.\n3. Segmentation performance comparision with other existing methods.\n4. Regression performance of DI-Net comparision when training and test time discretizations are different.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness/concern above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines discretization-invariant networks for learning maps between neural fields. Among other results, it proves upper bounds on how much model outputs can deviate under different finite discretizations and establishes universal approximation for a large class of maps between integrable function spaces. Empirical results are presented which demonstrate enhanced discretization-invariance relative to standard CNN models.

The reviewers generally agree that the topic is salient and that the mathematical results are sound. One weakness highlighted in the discussion concerned the novelty/impact. These concerns were effectively addressed by the authors in their response, and, moreover, as the reviewer noted, are not primary criteria for evaluation in any case. The claims of this paper are well supported by mathematical and empirical evidence, and on those grounds I support acceptance.
Decision: Accept (poster)