Neural Fields with Hard Constraints of Arbitrary Differential Order
Authors: Fangcheng Zhong
Kyle Thomas Fogarty
Param Hanji
Tianhao Walter Wu
Alejandro Sztrajman
Andrew Everett Spielberg
Andrea Tagliasacchi
Petra Bosilj
Cengiz Oztireli
Venue: NeurIPS 2023 poster (2023)
Keywords: neural fields
constrained optimization
TL;DR: A series of approaches for enforcing hard constraints on neural fields at any differential order

Abstract: While deep learning techniques have become extremely popular for solving a broad range of optimization problems, methods to enforce hard constraints during optimization, particularly on deep neural networks, remain underdeveloped. Inspired by the rich literature on meshless interpolation and its extension to spectral collocation methods in scientific computing, we develop a series of approaches for enforcing hard constraints on neural fields, which we refer to as Constrained Neural Fields (CNF). The constraints can be specified as a linear operator applied to the neural field and its derivatives. We also design specific model representations and training strategies for problems where standard models may encounter difficulties, such as conditioning of the system, memory consumption, and capacity of the network when being constrained. Our approaches are demonstrated in a wide range of real-world applications. Additionally, we develop a framework that enables highly efficient model and constraint specification, which can be readily applied to any downstream task where hard constraints need to be explicitly satisfied during optimization.

Neural Fields with Hard Constraints of Arbitrary Differential Order
Fangcheng Zhong University of Cambridge
Kyle Fogarty University of Cambridge
Param Hanji University of Cambridge
Tianhao Wu University of Cambridge
Alejandro Sztrajman University of Cambridge
Andrew Spielberg
Harvard University
Andrea Tagliasacchi Simon Fraser University
Petra Bosilj University of Lincoln
Cengiz Oztireli University of Cambridge
Abstract
While deep learning techniques have become extremely popular for solving a broad range optimization problems, methods to enforce hard constraints during optimization, particularly on deep neural networks, remain underdeveloped. In- spired by the rich literature on meshless interpolation and its extension to spectral collocation methods in scientific computing, we develop series of approaches for enforcing hard constraints on neural fields, which we refer to Constrained Neural Fields (CNF). The constraints can be specified plied to the neural field and its derivatives. We also design specific model rep- resentations and training strategies for problems where standard models may encounter difficulties, such as conditioning of the system, memory consumption, and capacity of the network when being constrained. Our approaches are demon- strated in a wide range of real-world applications. Additionally, we develop a framework that enables highly efficient model and constraint specification, which can be readily applied to any downstream task where hard constraints need be explicitly satisfied during optimization. Source code is publicly available btttps://2fc90 weteee996...iititt github. io/ github.
linear operator ap-
Introduction
Deep neural networks serve as universal function approximators 110 and have achieved tremendous success across a wide variety of tasks. Gradient- based learning algorithms enable optimizing the parameters of networks approximate any desired model. However, despite the existence of numerous advanced optimization algorithms, the issue of enforcing strict equality constraints during training has not been sufficiently addressed. Constrained optimization has a broad spectrum of real-world applications. For example, in trajectory optimization ([38 [Chapter 10), and, egg., 14 2,22))).
the agent poses at the start, end, and potentially intermediate steps are be explicitly enforced. In signal representation, it is desirable for the continuous model to interpolate the discrete samples [22
In physics, the models interest must comply with fundamental physical principles [14 27).
Applying traditional solvers for general constrained optimization, such as SQP 266 to neural networks can be nontrivial. Since traditional methods express constraints as a function of learnable parameters, this formulation becomes extremely high sigh--imensional, nonlinear, and non-convex in the context of neural networks. As the number of constraints increases, the computational complexity
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

grows substantially. Other solutions for deep neural networks may encounter issues related to system conditioning, memory consumption, and network capacity when subject constraints (20].
In this work, we consider the following constrained optimization problem:
arg min L (fo; s.t. F (x) g (x) Vx E S,
arg
(I)
where fo is parametric model with learnable parameters 0: and FF is linear operato that transforms f into another function F[]] which has the same imput and output dimensions as f. Common examples of valid operators F include the identity map, integration, partial differentiation of arbitrary order, and many composite operators that involve differentiation, such as Laplacian, divergence, and advection operators. F can represent broad class of constraints on the function ff in real-world applications. Enforcement of multiple constraints, expressed as different operators on f. is also possible. In practice, one typically does not have access to (or require) a closed-fon expression g, but rather the evaluation of on a set S of discrete samples of æ. In this work, we focus on equality constraints, as inequality constraints can be more easily satisfied through heavy regularization or a judicious choice of the activation function.
Our solution draws inspiration from meshless interpolation [6 and its extension to spectral collocation methods (7) in the scientific computing literature. These methods model a parametric function as a linear sum basis functions. By computing the weights of the bases, hard constraints on the local behavior of the function at the collocation points, where the constraints need to be satisfied, are enforced accordingly. However, the selection of the basis functions also determines the inductive bias of the model, i.e., the behavior the function outside the collocation points. Our perspective is to formulate the problem of constrained optimization bias. Instead of using a set of fixed basis functions, we allow each basis function to have additional learnable parameters. While the weights of the basis functions used to enforce constraints, other learnable parameters are optimized via a training loss to match the optimal inductive bias.
collocation problem with learnable inductive
With this formulation of the problem, demonstrate series of approaches for enforcing hard constraints in the form of Eq. Toto a neural field, i.e., a parametric function modeled as a neural network that takes continuous coordinates as input. In theory, we can formulate basis functions from existing neural fields of any architecture and enforce constraints. Nevertheless, certain network architectures may encounter difficulties the number of collocation points or the order of differ- entiation increases. To tackle these challenges, we propose specific network designs and training strategies, and demonstrate their effectiveness in several real-world applications spanning physies, sig- nal representation, and geometric processing. Furthermore, we establish a theoretical foundation that elucidates the superiority of approach over existing solutions, and extend prior works with shared underlying designs to broader context. Finally, we develop a Py Torch framework for the highly efficient adoption of Constrained Neural Fields (CNF), which can be applied to any downstream task requiring explicit satisfaction of hard constraints during optimization.
In summary, work makes the following key contributions:
We introduce a novel approach for enforcing linear operator constraints neural fields; We propose a series of model representations and training strategies to address various challenges that may arise within the problem context, while offering a theoretical basis for CNF's superior performance and its generalization of prior works;
Our framework is effective across wide range of real-world problems; in particular, CNF:
achieves state-of-the-an performance in learning material appearance representations. - is the first that can enforce an exact normal constraint on a neural implicit field. improves the performance of the Kansa method (18), a meshless partial differential equation (PDE) solver, when applied to irregular input grids.
2 Related Work
Applying constraints to a deep learning model has been a long-standing problem. Numerous studies have aimed to enforce specific properties on neural networks. For instance, sigmoid and softmax
While our approach can be theoretically extended to nonlinear operators, we focus on linear ones due to the efficiency of linear solvers and GPU vectorization.

---

activation functions are used to ensure that the network output represents a valid probability mass function. Convolutional neural networks (CNNs) were introduced to address translation invariance. Pointnet 30 was proposed to achieve permutation invariance. POLICE 3 was introduced to enforce affine properties on the network. However, these works were not specifically designed to handle strict and general-form equality constraints. There are also studies that train a multilayer perceptron (MLP) [34 37] or neural basis functions [8] 43 with a regression loss to approximate the equality constraints through overfitting, which we refer to soft constraint approaches. However, in those works, strict adherence to hard constraints is rarely guaranteed. DC3 [12] is the first work that can enforce general equality constraints onto deep learning models by selecting a subset of network parameters for constraint enforcement and utilizing the remaining parameters for optimization. However, this approach cannot offer theory or guidance regarding the existence of a solution when selecting a random subset of the network. In contrast, our method rigorously analyzes and guarantees the existence of solution. Linearly constrained neural networks [7] can also explicitly satisfy linear operator constraints, but their method only applies to the degenerate case where g(x) (Eq. is zero everywhere. Neural kernel fields (NKF) [4] employs kernel basis functions similar to ours solve constrained optimization problems in geometry, but their method cannot accommodate strict higher- order constraints. Their use of dense matrix also cannot scale up to accommodate large number of constraints, due to limitations in processor memory. A concurrent work, PDE-CL (24] utilizes a constrained layer to solve partial differential equations as a constrained optimization problem. We demonstrate that a constrained layer may not be the ideal choice for designing basis functions and propose alternative solutions. In fact, our approach can be seen a generalization of NKF and PDE-CI. While seemingly unrelated, both works approach constrained optimization problems by leveraging the weights of basis functions to enforce constraints. They differ only in the design of the basis and the optimization function. Finally, there has been substantial development in implicit layers [13] that can enforce equality constraints between the layer's input and output while efficiently tracking gradients. However, this approach is not specifically designed for constraints over the entire network, although can be potentially incorporated into our method to facilitate computation.
3 Method
At the core of our approach is the representation of a neural field f ((): RR.M. -> RRN as a linear sum of basis functions, as with the collocation methods:
(x) Σ (z)
where each V, (.) R - N represents basis function that can be expressed as any parametric function with learnable parameters, such as neural networks; B3 E TR is the weight of the i-th basis; and @ indicates the Hadamard product.
3.1 Constrained Optimization
Consider S {x,)]= set of I constraint points such that Fif (z,) (I.). Vi where the ground truth value of g (xx) is accessible. Using the neural basis representation ( and assuming that F is linear and F [V.]] well-defined for all the following needs to hold for the constraints:
F(A(2)==F Σ 3. O
(x) EB.
P.B.OR O F.F. = (x), V E S.
(3)
Eq. 3 is collocation equation in the context of spectral collocation methods for solving differential and integral equations []. These constraints can be explicitly satisfied by defining / basis functions expand Eq. into batched matrix form:
and solving Eq. 3 for their weights. For a linear J,
(mm)
F (11)
Ta17
(m)
( (m2)
(22))
Ba
(122)
F (()))()))) FMa)(
(w²)
There are in total N such matrix equations to solve, each corresponding to an output dimension of f if N We denote the matrix as A for reference. As long as we apply a differentiable linear
In Einstein summation notation, the batched matrix equation in Eq, can be expressed as IIN, IN - IN.
3

---

result, even Algorithm Training
Algorithm 2 Inference
1: repeat
Input: æ
2:
Compute 3 as the solution of Eq.
Output: (x)
3: Compute gradient 8c 0c 0f
1: if [3 is None then
30 8f 80
4: Update 0
2:
Compute ,3 from Eq. 4 w/o gradients
5:
until converged
3: end
4: -,
solver that provides valid gradients to compute the weights, the neural basis representation (Eg, can be integrated with any optimization loss C (fe: 8) to approximate the desired behavior, while ensuring that the hard constraints are strictly enforced. Algorithm describes the training procedure in detail, where Step 2 solves for the weights 3 every iteration, ensuring consistent satisfaction of hard constraints throughout the entire training process, while Steps 3 and 4 update the training parameters to optimize for the training loss L under the constraints. We compute ,3 using an LU decomposition with partial pivoting and row interchanges, which is fully differentiable when A, is full rank. The computation of B constructs computational graph that tracks 03 thereby ensuring a valid af Consequently, any loss function with a valid gradient ac af can be smoothly integrated into this training procedure. The inference procedure is detailed in Algorithm 2 Here, B only needs to be pre-computed once since it is independent of the evaluation point æ. As result, performing inference with CNF is highly efficient and boils down to computing linear combination of I basis functions without solving any matrix equations, task that can be vectorized for efficiency.
ō0-
This approach can also be extended to multiple linear operator constraints, indexed by j:
(5) where S, indicates the set of points z where the values of Fj [f) (z) constrained. Note that Sj corresponding to different operator constraints can have repeated æ elements. For example, to constrain V f(())) where æ E S S T a partial differentiation operator needs to be specified for each of the . partial derivatives at the same point æ. These constraints be explicitly satisfied by defining E; [Sy1 basis functions and computing their weights by solving the batched matrix equation:
F3 [f] (a) 9E (#) V x E Sj
Tail
Fi [
B
(1
W1
92( (2
where 2/ indicates the n-th point in Sj. The training and inference procedures are the same as previously described, except that 3 is now solved using Eq. iinstead.
3.2 Neural Basis Functions
To construct a set of basis functions from an existing neural field, we consider a few approaches.
Independent basis The most straightforward method is to specify separate neural field for each basis. This method is the most expressive but can only be used with a small number of constraints. This is because the number of networks and learnable parameters increases linearly with the number of constraints. bigger concern is the quadratic growth the size of the matrix A high-end processors guaranteed to out of memory working with a large number of constraints.
Constraint basis We can directly use the last hidden layer of multi multi-layer perceptron (MLP) as the basis and compute the weight of each node to enforce the constraints. This strategy has been referred to as a construint layer concurrent work 24). The size of the network, excluding hidden layer, does not grow the number of constraints increases. However, this approach may lead to -eepeddet basis functions, resulting the matrix A being singular or having a large condition number. This can lead to significant error bound in the satisfaction of hard constraints.

---

Hypernetwork basis The third approach is to generate the parameters of each neural basis function using a hypernetwork [T6]. The imput to the hypernetwork is one-hot encoding of length equal to the number of bases. The hypernetwork ensures that the network size does not increase with the number of basis functions. Nonetheless, the quadratic growth in the size of A remains an issue for both the constraint and the hypernetwork bases.
In several empirical studies, we found that the two aforementioned approaches hinder the network's capacities when subjected to hard constraints. They also encountered difficulties reducing the condition number; see Appendix.
Kernel basis As a result, we propose several variants of a neural of basis functions:
as a special design
V (z) = N (æ.). (x)),
(7)
where ob)) is a neural encoder that maps æ into a nigh-dimensional feature space; k(:.)) is a kernel
function that measures the similarity between two points in the feature space; is the i-th constraint point, which we refer to as an anchor point. With this design, the bolsters the learning capacity of the basis function. The size of parameters in the network also does not grow with the number of constraints. If x((.-) is a dot- dot-product kernel, Eq becomes the same basis function used in NKF [4]] However, we empirically found that Gaussian kernel better promotes the linear independence of basis functions due to its local compact support, while still preserving sufficient learning capacity; see Appendix.
feature space
Hypernetwork kernel basis One drawback of the kernel basis function that only one constraint can be applied to each anchor point. For example, one cannot enforce constraints on both the value of f(x)) and its gradient (xx) at the same anchor point, as would result in repeated basis functions V; = Vi = (á (manchor) & (a=) and thus ill-conditioned system; see Appendix for math details. For multiple constraints at repeated anchor points, we propose a hypernet kernel function:
(8) where the weights of Φb are controlled by hypernetwork conditioned the anchor point æ;. This hypernet construction of the encoder ensures that the network would not have repeated basis functions and the number of network parameters does not grow with the number of basis functions.
V (x) = k (Ø: (z,) Φ; (x))
Hybrid kernel basis Central our approach the construction and inversion of linear system, the size of which grows quadratically with the number of constraints. To mitigate severe memory issues in large-scale constraints, we design a basis that promotes sparsity in matrix A y for large-scale problems. While using Gaussian kernel can promote sparsity, it is difficult to select a proper bandwidth to explicitly control the number of nonzero elements since the kernel operates in the feature space rather than the original domain. To overcome this, we propose hybrid kernel function:
V, (x) = k (ói (zx) di (æ)) Ki x),
where the first part i (d) (x:), o. (x)) can be either a Gaussian kernel basis or a hypernet kernel basis, depending on the task, and Ki is some compactly supported kernel function, allowing for explicit adjustment of the zero behavior of V,, A candidate KG can be a truncated Gaussian kernel such as:
x x|1
KG (X,, x)
exp
2
if X x| 30
(10)
if | x| 2 3o. Reducing the magnitude of yields a matrix A with more zero entries, breaking the quadratic dependency on the number constraint points.
Properties of various basis functions are summarized in Tab. T We recommend using the regular Gaussian kernel basis for standard-scale problems without multiple constraints at repeated anchor points, and employing hypernet kernel and hybrid kernel for corresponding advanced scenarios. In Sec. we demonstrate the application of each type of basis function in concrete real-world examples.
3.3 Training Strategies
Regularization In many applications, we recommend incorporating an additional loss term to regularize the condition number of the matrix A, y during optimization. A low condition number corresponds to a reduced level of matrix singularity and smaller error bound in the solution vector, which is crucial for satisfying hard constraints.
5

---

Table 1: Basis functions summary.
Indepesdeat basis
Constraint basis
Hypemetwork
basis
(Dut-prodec))
basis
(Gossian)
busis
Hypemetwork
kernel bosis
(Gasaiam)
Hybrid kenel hasis
(Gascian)
Learning espacity
Pior
Poor
Fair
Good
Good
Good
Linear independence
Pour
Fair
Pour
Good
Good
Good
Matris sparsity
Dense
Dense
Dinse
Dense
Sparse
Spume
Sparse
Parms independent
constraints #
No
Yes
Yes
Yes
Yes
Yes
Yes
Conrollable sparsity
No
No
No
No
No
No
Yes
Multiple constraints repeated points
Yes
Yes Yes No No Yes
(if hypernet)
Yes
Transfer learning Since CNF is formulated as collocation method with a learnable inductive bias, can pre-train the inductive bias of basis functions with custom configurations, such as well-conditioning. smoothness, or more advanced variations. This allows us to apply the pre-trained basis functions to unseen data, directly computing the weights with little as the problem reduces to near-collocation scenario. We demonstrate this by solving a partial differential equation
additional training,
irregular grid inference time and without further training; see Appendix.
Sparse solver
we introduce patch-based approach that leverages the restricted support induced by the hybrid kernel function (Eq. 9 to solve the sparse linear system. For each evaluation point œ, we construct a subset of basis create
Drawing inspiration from classical reconstruction techniques 23
40
functions such that V (a)) F O using the support criterion induced by KG- This enables lower-dimensional dense submatrix (Ar)mo to solve for the weights of only the nonzero items so that the size of the system sufficiently small fit within the processor's memory limitations.
4 Applications
For efficient adoption of CNF in real-world applications, we have developed a PyTorch framework that abstracts the definition of the basis function, enabling users to conveniently define and experiment with custom architectures. The framework also enables vectorized basis function operations for highly efficient training and inference. Additionally, linear differential operator constraint by providing regular expression in LРTEX. This can be processed to compute the exact post-operation function evaluation via automatic differentiation. thereby eliminating the need for manual execution. Further clarification of the features and usage of our framework, along with illustrative examples, is provided alongside the code release.
have the flexibility specify an arbitrary
With this framework, we leverage CNF in four real-world applications, utilizing distinct basis functions tailored to the specific context of the problem. We elucidate and empirically demonstrate the merits of the chosen basis function in each scenario. Our intention is to use these examples as guideline for selecting appropriate basis functions, which has been briefly summarized the end of Sec. 3.2 Apart from Sec. 4.1 which serves as a toy example, we demonstrate unique advantages or state-of-the-an results achieved in each application when employing CNF.
4.1 Fermat's Principle
Fermat's s principle in optics states that the path taken by a ray between two given points is the path that can be traveled in the least time. This can be formulated as constrained optimization problem:
{ ds
arg min
ac (Po. P1),
(11)
where ds is the differential are length that the ray travels along contour C and v is the light speed that varies along the contour due to the refractive index of the medium. The contour's entry and exit points OC are hard constraints and the contour is optimized to minimize the travel time Sc We model the contour C as collection of points from parametric equation, i.e. C == (p(z) (x)) + (2)2(2)) x E [0, 1]}. there are in total two constraint points, we use two independent quadratic polynomials (f. fe as basis functions, We chose polynomials over neural networks due

---

Table 2: Quantitative evaluation of learning material appear- ance representations. We evaluate each method by render- ing the learned BRDFs with environment map illumination and computing the mean and standard deviation of image metrics across all materials the MERL dataset 21
spoch O: trovel time 42.51 epoch 36.70
RMSEx 21
PSNRT
SSIM
NBRDF 35
1.57 + 2.86
44.96 + 11.55
.96 + .07
Figure 1: The optical paths and travel time before and after the optimization. The constraints are strictly enforeed throughout the optimization.
FFN 37
1.16 1.36
44.72
1 10.46
.98 1 .03
SIREN 34
3.32
3.05
33.43
8.84
.93 1 .08
Kerne FFN
0,94 + 1.16
46.32 1 9.92
.98 1 .03
Kerne SIREN
0.60
+ 0.54
47.95 +
7.99
.99 - .01
to the simplicity of the problem. The weights [31, (2] are solved differentially to fit hard constraints, while the coefficients the polynomials are learnable parameters that can be updated with respect to the loss function. We show the results in Fig. with an increasing refractive index along the Y-axis.
4.2 Learning Material Appearance
In visual computing, the synthesis of realistic renderings fundamentally requires an accurate descrip- tion of the reflection of light on surfaces [1]. This is commonly expressed by the Bidirectional Reflectance Distribution Function (BRDF), described by Nicodemus et al. (25). which quantifies the ratio of incident and outgoing light intensities for single material: fr (w, Wo). where w,, W. E S are the incident and outgoing light directions.
Our aim is to learn a neural field ig Pg(w;, with parameters that closely matches the ground truth BRDF f, (w,,wo) wo), V w;, w. E S We constrain by by selecting 100 sample pairs of (w,, as), where half of the pairs are uniformly sampled, and the other half of w, are concentrated around the direction of the perfect reflection of wr This means that the latter constraint points situated around the specular highlight, which contains high-frequency details that regular neural networks struggle compare CNF, formulated with a Gaussian kernel basis (Eq. 7 with SIREN 34 or FFN (37 as the encoder network, against
to learn 32 31.19 19 35
We illustrate these issues in Fig. 2 where
multiple state-of- the-art neural-based fitting approaches [34 37 (35] for BRDF reconstruction. We perform this evaluation on highly specular materials from the MERL dataset [21]. with size-matching networks for fair assessment, and subsequently rendering standard fixed scene; see Appendix for the complete MERL rendering results. Following previous work (35], we train &g on 640k (w) w,) samples, minimizing LI loss in the logarithmic domain account for the large variation of BRDF values due to the specular highlight, while enforcing the aforementioned hard constraints on (w), a.)):
arg min Ew,, (Wbo wo) log (fr Wo) 1)
(12)
s.t. ( 2.) log (fr i.) + +)
We summarize the results computed over all MERL materials in Tab. where CNF demonstrates superior performance in learning material appearances with state-of-the accuracy; see Appendix for additional evaluations and ablation studies.
4.3
Interpolatory Shape Reconstruction
The representation of shapes via neural implicit functions 15] has recently garnered much attention 28 2 34
Compared to traditional forms of representation such as point clouds, meshes, or voxel grids, neural implicit functions offer a flexible and continuous representation capable of handling unrestricted topology. In this section, we demonstrate how CNF can be applied to enforce exact interpolation constraints on implicit shape representations, while retaining the ability to train the field with geocetry-specifie inductive bias. Given oriented point cloud P := {( her. our aim is to construct a continuous implicit function (xx) on the domain Ω whose O-level-set S:= {p Ω ((p) = 0} represents the surface of the geometry.
Exact normal constraints We first demonstrate how our framework can be utilized to learn an implicit shape representation, while exactly interpolating both surface points and their associated
7

---

chrome steel
chrome
GT
NSRDF
FFN
SIREN
Kernel FFN (Ours)
Kernel SIREN (Ours)
Figure 2: Learned material appearance by enforcing hard constraints around the specular highlights.
Exuet Nocmal Constcsiats
Reconsteuction
Toioet Tritalontion
Figure 3: (a): Learning shapes from 2D point clouds (yellow points) with oriented normals (black arrows) using CNF; black lines depict the zero level sets the learned implicit field. The method interpolates both exact point and normal constraints during initialization and throughout training, yielding a plausible surface upon minimizing the Eikonal constraint across the field. (b): Surface reconstruction of 3D point clouds containing 10,000 points.
normals. We conduct experiments in 2D, and formulate CNF with a hypernetwork kernel basis (Eq. S) as the problem involves both zero-order and higher-order constraints repeated constraint points. Note that this upproach is the first to enforce exact normal constraints on neural implicit field, as prior works [1] can only approximate the exact normal with a norumalnormal constraint (39). In contrast, we constrain the implicit field such that 4 ((x,) ) 0, Væ, P and ( = n2 Væ,, E P. Inspired by the geometrically motivated initialization 22. we use weights that are pre-trained to represent the signed distance function (SDF) of circle with the desired number of constraint points. We found this placed the network a more favorable starting position produce plausible surfaces under new constraint points. We then train the network to solve the Eikonal equation, following previous work [15,, via minimizing the quadratic loss function:
L = Eg (1 V (xx)] - 1)2
(13)
where the expectation taken with respect a uniform distribution over the domain Q. As shown in Fig. B()), CNF produces plausible surfaces to explain the point cloud, with the advantage of not depending on manufactured offset points 39 to ensure existence of a non-trivial implicit surface.
Large-scale constraints To demonstrate our hybrid kernel approach (Eq, 9 with the sparse solver for handling large-scale constraints, we reconstruct 3D point clouds comprising 10,000 points uniformly sampled over the surface. We set the support radius of the kernel to four times the average nearest neighbor distance, and assign constant value 10 to regions of space with no support. The reconstruction is obtained by adhering to the points and pseudo-norme constraints [39 41 during initialization, with no additional training required for the dense reconstruction to produce smooth surfaces, as shown in Fig. 3b; Appendix for additional results and evaluations.

---

Ground truth with irregular grid
Simple RBF error
Table 3: Solving the 1D advection PDE (3 0.1) with simple (left) and skewed (right) RBF kernel. Both versions were optimized by our framework. We can see the benefit of using the skewed RBF at various perturbation levels.
Space (+)
Skowed RBF enror
RBF
Skewed RBF
Pertubation
RMSE
nRMSE
RMSE
nRMSE
0,01
0,0086
0.0149
0.0061
0.0116
0.05
0.0252
0.0442
0.0024
0.0043
Time (1)
0.10
0.1484
0.2534
0.0344
0.0651
0.50
0.0170
0.0297
0.0167
0.0282
Figure 4: Left: Ground truth advection function and skewed input grid with a = 0.1 perturbation. Right: Error maps for simple and skewed RBF.
1.00
0,2830
0.4838
0.0231
0.0365
4.4 Self-tuning PDE Solver
Our framework can also be applied to solve partial differential equations (PDEs). If using fixed radial basis functions (RBFs) without optimization, CNF simplifies to the Kansa method (18), type of spectral collocation method where partial derivatives are enforced via hard constraints. Compared to the classical mesh-based numerical PDE solvers such as the finite difference method (FDM) and finite element method (FEM), Kansa has the advantage of offering an analytical and closed-form solution function across the entire domain, and solving PDE on irregular grid without requiring interpolation or meshing, i.e. grouping the points into triangles or quadrilaterals. Meshing on an irregular grid proves to extremely difficult and severely harms the accuracy of these methods [9]. However, one disadvantage of the Kansa method is the difficulty in choosing the appropriate RBF kernel (5). When using a Gaussian kernel, large bandwidth may lead an ill-conditioned system, while a small bandwidth may break the global smoothness of the solution function. This is especially challenging when the number or dimensionality of collocation points is large, or when the points are closely or irregularly spaced, making it difficult to manually fine-tune the hyperparameters of the RBF Our method addresses this limitation by supporting the automatic fine-tuning of hyperparameters. optimizing both the conditioning and global smoothness of the solution:
arg min wi cond(A,s) - w2 fo(x dæ
s.t. F, = 91 (x) Vj,
(14)
total variation
where cond(A) (A is the condition number of the linear system and F, corresponds to the PDE constraints along with the initial and boundary conditions. We demonstrate this approach by solving the following ID advection equation (36) on an irregular grid:
Ōu(I, t) Øt
+1x(x) Ōu(x, t) dr O, u(x, 0) uz(
I E (0, LE (0, 1),
(15)
I E (0,
where (1)) represents the initial state of the system. We sample 32 points on the boundary t= 0 and 31 x 32 points on the space-time domain (0. 1) x (0, 1), which amount to a total of 1024 points, as hard constraints. The non-boundary samples, shown as red crosses in Fig. 3 are generated by uniform grid perturbed by Gaussian noise. this example, we cannot apply a single unified RBF bandwidth to fit such an irregular grid. Rather, we design skewed radial basis function:
Ri (Z;, x.) = exp (-1 (xx -
(16)
where = (x. 1) and Σ; diag( (a,, b:]) is per-basis learnable covariance matrix. Our results Tab. 3 show that using the skewed RBF kernel and optimizing per-basis parameters consistently improves performance on the irregular grid; see Appendix for additional results and evaluations.

---

The skewed RBF is essentially another variant of the Gaussian kernel (Eq. 7 but without the neural encoder. We recommend using neural networks only when the behavior of the basis functions to be optimized away from the constraint points is highly complex, as seen in the BRDF and shape- reconstruction examples. For problems where the desirable priors are as simple as linear independence and smoothness, such as solving PDEs, our proposed skewed RBF demonstrates sufficient capacity.
5 Discussion
Training and inference efficiency CNF is highly efficient in training (minutes) and inference (seconds); see Appendix for detailed reports, learning curves, and additional evaluations.
Theoretical analysis We offer a theoretical basis for CNF's superior performance:
Comparison with general solvery Most numerical methods for constrained optimization in scientific computing, including the popular SQP (26) and the more recent DC3 [12 adopt the following formulation for the constrained optimization problem:
arg min & (0) s.t. hn (0) = 0, h2 (0) = 0,
(17)
where 0 denotes the learnable parameters. For problems where 0 represents the weights of deep neural network, these constraints become extremely high-dimensional and nonlinear, especially as the number of constraints increases. In this formulation, constraint is only linear when (0) is linear function 0. Therefore, linearity no longer holds as h()) involves constraints of a neural function. In contrast, our formulation (Eq. о and 5) only requires the operator F to be linear, while the neural function f can still be highly nonlinear with respect to 8. Hence, CNF significantly reduces the problem's complexity, enabling the explicit determination and promotion of solution's 's existence.
Comparison with soft constraint approaches While there have been attempts model constraints by overfitting a neural function trained with regression
34 37,, CNF offers several clear advantages over such soft approaches. Most notably, CNF satisfies hard constraints without the need for training, whereas soft approaches may require extensive training to approximate constraints. Furthermore, despite extensive training, soft approaches may still fail to satisfy hard constraints due to inherent limitations in their learning capacity. In contrast, CNF provides robust guarantee of hard constraint satisfaction within machine precision error, provided the condition number is small. Another drawback of employing soft approaches becomes evident when effectively imposing priors. The incorporation of priors often involves introducing additional terms to the loss. Consequently, a tradcoff arises between the constraints and the prior, which is controlled by hyperparameter. In contrast, CNF offers a clean solution without such a tradcoff. With CNF, the inclusion of priors does not compromise hard constraints, thereby maintaining harmonious balance among various aspects of the model.
Generalization of NKF and PDE-CL CNF generalizes prior works NKF 41 and PDE-CL 24, NKF employed dot-product kernel basis for 3D reconstruction, while PDE-CL used a constraint basis to solve PDEs. Yet, these bases exhibit suboptimal performance in terms of linear independence and learning capacity. Their use of dense matrices also renders them impractical for large-scale problems. Moreover, the dot dot-product kernel used by NKF cannot address strict higher-order constraints; see Appendix. We propose several novel variants of basis functions enhance linear independence, learning capacity, and matrix sparsity, with strategies for analyzing and facilitating convergence. Our methodology unifies NKF and PDE-CL, extending their formulations to broader theoretical context.
Extension to nonlinear operator constraints Although we restrict the operator to be linear in this work, CNF can be extended to accommodate nonlinear operators by solving nonlinear version of Eq. 3 provided that the solver differentiable. This could be accomplished through the use of implicit layers (13), which we leave future work.
6 Summary
We introduce Constrained Neural Fields (CNF), a method that imposes linear operator constraints on deep neural networks, and demonstrate its effectiveness through real-world examples. As constrained optimization a fundamental problem across various domains, we anticipate that CNF will unlock a host of intriguing applications in fields such physics, engineering, finance, and visual computing.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the UKRI Future Leaders Fellowship [G104084] and the Engineering and Physical Sciences Research Council [EP/S023917/1].
References
[1] K Atkinson, Ivan Graham, and Ian Sloan. Piecewise continuous collocation for integral equations. SIAM Journal on Numerical Analysis, 20(1):172-186, 1983.
[2] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2574,, 2020.
[3] Randall Balestriero and Yann LeCun. Police: Provably optimal linear constraint enforcement for deep neural networks. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[4] Jernej Barbič, Marco Silva, and Jovan Popović. Deformable object animation using reduced optimal control. In ACM SIGGRAPH 2009 papers, pages 1-9. 2009.
[5] Martin D Buhmann. Radial basis functions: theory and implementations, volume 12. Cambridge university press, 2003.
[6] Richard L. Burden and J Douglas Faires. Numerical analysis (nineth edition). Thomson Brooks/Cole, pages 57-58, 2010.
[7] Claudio Canuto, M Yousuff Hussaini, Alfio Quarteroni, and Thomas A Zang. Spectral methods: evolution to complex geometries and upplications to fluid dynamics. Springer Science Business Media, 2007.
[8] Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: unified framework for neural fields and beyond. arXiv preprint arXiv:230..01226, 2023.
[9] Ward Cheney and David R Kincaid. Numerical mathematics and computing. Cengage Learning, 2012.
[10] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314, 1989.
[11] Yue Dong. Deep appearance modelling: survey. Visual Informatics, 3(2):59-66 68, 2019.
[12] Priya Donti, David Rolnick, and Zico Kolter. Dc3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
[13] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, Armin Askari, and Alicia Tsai. Implicit deep learning. SIAM Journal on Mathematics of Data Science, 3(3):330-958, 2021.
[14] Herbert Goldstein. Classical mechanics. Addison- Weslly World Student Series, 1950.
[15] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. arXiv preprint arXiv:2002. 10099, 2020.
[16] David Ha, Andrew Dai, and Quoe V Le. Hypernttorks.. arXiv preprint arXiv: 1609.09/000
2016.
[17] Johannes Hendriks, Carl Jidling, Adrian Wills, and Thomas Schön. Linearly constrained neural networks. arXiv preprint 200000000..06000, 2020.
[18] Edward J Kansa. Multiquadrics a scattered data approximation scheme with applications to computational Buid-dynamics-1 ii solutions parabolic, hyperbolic and elliptic partial differential equations. Computers & mathematics with applications, 19(8-9):1471661. 1990.
[19] Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš Hašan, and Ravi Ramamoorthi. Neu- mip: Multi-resolution neural materials. Transactions on Gruphics (Proceedings of SIGGRAPH), 40(4), July 2021.
[20] Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal Scientific Computing, B105): 2021.

---

[21] Wojciech Matusik, Hanspeter Pfister, Matt Brand, and Leonard McMillan. A data-driven reflectance model. ACM Transactions on Graphics, 22(3):759-769, July 2003.
[22] Sylvain Meignen and Valérie Perrier. new formulation for empirical mode decomposition based on constrained optimization. JEEE Signal Processing Letters, 14(12):932-935, 2007.
[23] Bryan S Morse, Terry S Yoo, Penny Rheingans, David T Chen, and Kalpathi R Subramanian. Interpolating implicit surfaces from scattered surface data using compactly supported radial basis functions. In ACM SIGGRAPH 2005 Courses, pages 78-es. 2005.
[24] Geoffrey Négiar, Michael W Mahoney, and Aditi S Krishnapriyan. Learning differentiable solvers for systems with hard constraints. arXiv preprint wrXiv:2207.0867 2022.
[25] Fillemon Nicodemus, C. Richmond, J J. Hsia, I W. Ginsberg, and LL. Limperis. Geometrical considerations and nomenclature for reflection. 160, 10 1977.
[26] Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer, 1999.
[27] John G Papastavridis. Analytical mechanics: A comprehensive treatise the dynamics of constrained systems (reprint edition). 2014.
[28] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEF/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[29] Michael Posa, Cecilia Cantu, and Russ Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 33(1):69-81, 2014.
[30] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[31] Gilles Rainer, Ghosh, Wenzel Jakob, and Weyrich. Unified neural encoding of btfs. Comput. Graph. Forum, 39:167-178, 2020.
[32] Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim Weyrich. Neural BTF compression and interpolation. Computer Graphics Forum (Proc. Eurographics). 38(2);1-10, May 2019.
[33] Szymon Rusinkiewicz. new change of variables for efficient BRDF representation. In Rendering Techniques (Proceedings of Eurographics Workshop on Rendering), Vienna, Austria, June 1998.
[34] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In arXiv, 2020.
[35] Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. Neural brdf representa- tion and importance sampling. Computer Graphics Forum, 2021.
[36] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 35:1 596-1611, 2022.
[37] Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Tridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurlPS, 2020.
[38] Russ Tedrake. Underactuated Robotics. 2023.
[39] Greg Turk and James F O'brien. Shape transformation using variational implicit functions, In ACM SIGGRAPH 2005 Courses, pages 13- es. 2005.
[40] Christian Walder, Bernhard Schölkopf, and Olivier Chapelle. Implicit surface modelling with a globally regularised basis of compact support. In Computer Graphics Forum, volume 25, pages 335-644. Amsterdam: North Holland, 1982-, 2006.
[41] Francis Williams, Zan Gojeic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEF/VVF Conference on Computer Vision and Pattern Recognition, pages 18500-18510, 2022.
12

---

[42] Francis Williams, Zan Gojeic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields learnable kernels for 3d reconstruction. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 18500-18510, 2022.
[43] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwa- janakorn Nex: Real-time view synthesis with neural basis expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8534-8543, 2021.
13

---

Neural Fields with Hard Constraints of Arbitrary Differential Order
Appendix
In this appendix, we provide additional evaluations various designs of neural basis functions (Sec. Ap. along with additional details and results regarding their applications in material appearance learning (Sec.[B), interpolatory shape reconstruction (Sec. C and the self-tuning PDE solver (Sec. D).
which also includes transfer learning example (Sec. D.3).
A Neural Basis Functions
A.1 Issues with the Neural Kernel Basis Function
We illustrate how, when using the neural kernel basis function (Eq. 7). enforcing multiple linear operator constraints the same anchor point would result in repeated basis functions V. V, (o (aanchor) ) & (2:)), consequently leading to an ill-conditioned linear system. Recall Eq.
FL 1 (mn F 10.l
(m))
(m))
(1
(18)
Fo Wi (w)
2 02(=5)
where 2y indicates the n-th point in Sn such that F, [[] (z) - 9, (x), V E Sy. When applying
the neural kernel basis function v (x) = k (o (ax) , (z)) KN (Z2, becomes:
(2,, x) the above matrix equation
TiRy(a) m/) Finy mb)
91
Finy(as.
(19)
(an m F2RN(0).
Tany of) (ann
92
Without loss of generality, assuming f IR RR Fi is identity map: F2 is differentiation; and S = -2 = {mr) (same anchor point for both constraints), the matrix equation reduces to:
(NN 2o)
KN xo)
31
[9 (xo) 92 (xx)
(20)
ny (iy Ho) ( xo)
82
singular
Therefore, the system is ill-conditioned the matrix has repeated columns. NKF 4 uses this basis function with a doo-pproduct kernel and thus cannot enforce strict constraints simultancously on both the points and their normals in shape reconstruction, which further evaluate in Sec. C.I.I We also discovered significant disadvantages of using dot-product kernel as opposed to a Gaussian kernel. We elaborate these findings in Sec. A.2 Sec. B.2
A.2 Basis Function Ablation
Choosing the appropriate basis function for an application requires consideration of two fundamental properties of basis functions: their linear independence and learning capacity. Linear independence is crucial for satisfying hard constraints, whereas learning capacity is key for fitting the inductive bias. In this ablation study, we train various neural basis functions discussed in Sec. .3.2 regularize the condition number of the matrix A,, which is used to constrain a 2D function over 4096 points. A large condition number often corresponds to a substantial error satisfying hard constraints. This study offers insights into how each design is prone to the ill-conditioning of the linear system. For a fair comparison, we use the same architecture for all neural encoders- 2-layer MLP with Tanh activations and total of 1.3M trainable parameters.
14

---

As shown in Fig. [] the Gaussian kernel basis is well-conditioned throughout training, while all other methods struggle reduce the condition number.
In Sec. B.2 we also discovered that the Gaussian kernel basis, when constrained, exhibits the superior learning capacity for fitting the BRDF In contrast, the performance of other methods was found be extremely poor. Note that the dot-product kernel and the constrained layer are the basis functions used in [41 and [24], respectively. Based on our findings, we strongly recommend employing the Gaussian kernel basis function when performing constrained optimization for neural fields.
340
Figure 5: Condition number of different neural basis functions in the following order (left to right): constraint layer (pink), hypernet basis (yellow), dot-product kernel (blue), and Gaussian kernel (black) throughout training.
B Learning Material Appearance
B.1 Experiment Details
In the experiments on material appearance fitting, all neural networks have single hidden layer and we controlled the network width so that methods have roughly 200k parameters for a fair comparison. Both baseline FFN and kernel FFN use frequency of 16, meaning that the inputs are mapped to encodings of size 38 before feeding to MI.P. The kernel FFN model uses FFN as an encoder which has width of 248 and outputs a latent vector of dim 512. The baseline FFN MLP has a width of 78.. The kernel SIREN model has width of 256 and also outputs latent vector of size 512. The baseline SIREN and NBRDF have width of 442. We use Adam optimizer with learning rate of x 10-4 for methods except for SIREN baseline and kernel SIREN, which use learning rate of x 10 for more stable performance.
For constraint points, we sample the angles 0,, 02 and Φ4 as in Rusinkiewiez's parameterization [33 A figure of those angles is shown in Fig. [ We sample 0g and Φa uniformly at random within range [0, r/2) and [0, 2n] respectively. Half of the 0, are also sampled uniformly at random within [0, t whereas the other half are sampled from Gaussian distribution with mean of 0 and standard deviation 0.1. Those angles are then converted to 6D in-going and out-going directions as inputs to the networks.
B.2 Ablation Studies
In Tab. 4 Fig. 7 we report the ablation results of fitting the BRDF with constraints, including the dot-product kernel using SIREN as the encoder, the hypernet basis with SIREN the main network, and the constrained layer integrated into SIREN. The hypernet basis uses a SIREN that has width of 442 the main network, and ReLU MLP with hidden layer with 100 nodes as the hypernetwork. The constraint layer has width of 319. The hypernet basis and constraint layer produce extremely poor results and completely unable perform this task. While the dot-product kernel performs better than both the hypernet basis and the constraint layer, it still significantly underperforms when compared the Gaussian kernel.
B.3 Learning Curves
In Fig. 8 and Tab. 5 we report the learning curves, training times, and inference times for various approaches in the BRDF reconstruction experiment. CNF with Gaussian kernel basis outperforms other approaches in terms of convergence, while maintaining similar training and inference efficiency.
15

---

Table 4: Ablations material appearance learning task. All methods here are based on SIREN 34
RMSEx10
PSNR
SSIM
Hypernet Basis
39,02 16.92
9.74 6.41
0.70
Dot-product Kernel
9.49 + 11.32
28.33 + 13.22
0.86 + 0.13
Constraint Layer
41.85 + 14.14
8.37 + 4.37
0.70 + 0.06
Figure 6: Angles for sampling constraint points for material
Gaussian Kernel
0.60 = 0.54
47.95 + 7.99
0.99 - 0.01
learning. Figure from 33
chrome-sted
chrome
GT
Hypernet Basis
Dot-product Kernel
Constraint Layer
Gaussian Kernel
Figure 7: Ablation on the material appearance fitting.
104
NBRDF
Kernel
Hypemet Basis
Table 5: Training and inference time (in seconds)
Train Loss
Constraint Layer
Hypernet Kerne
of various approaches for BRBF reconstruction.
Method
Training
Inference
10a
SIREN [34]
578.89
0.3348
NBRDF [35]
19,702
0.2966
Constraint layer [26]
577.49
0.2539
2
Gaussian kernel
1101.3
0.7284
Hypemet basis
2997.5
0.6933
0
1000
2000
3000
4000
5000
6000
Hypernet kernel
1784.1
1.1692
Train Iter
Figure 8: Learning curves of various approaches for BRDF reconstruction.
B.4 Full Results
The complete rendering results for all materials included in the MERL dataset 22 are shown Fig.
in Fig. 9 Fig.
TO
1I
Fig. 12 Fig. 13 and Fig. [4] and are available on the project page
https: / c24
:ttps::/zfffo0
www.eee................ gitgub....... io/ CNF github............. io/ We also show the SSIM errors along with the
renderings.
16

---

C Interpolatory Shape Reconstruction
C.1
Experiment Details
C.1.1 Exact Normal Interpolation
In the experiments on exact normal interpolation, the shapes are represented by the level sets of our hypernet kernel. The encoder of our hypernet constructed of MI.,, which has single hidden layers containing 1200 hidden units. Between each layer we use softplus activation, with 9 = 10 and the encoder maps to feature space of size 800. We with a standard deviation of 50. For all experiments shown, we use a maximum of 50 epochs, learning rate of 10-5 and the Adam optimizer. To extract the zero-order level set of the implicit representation we use the Marching Cubes algorithm, on a uniform grid sampled at a step size of 1/60. For all shapes except the circle, we pretrain the weights of the hypernet encoder to first represent circle with uniformly sampled constraints equal to the number of constraints of the target shape. The Eikonal loss function defined in Eq. 13 is defined on the domain f2 [-2.2] [-2, 2], and each iteration we use 1000 points randomly sampled from Ω
Gaussian kernel which is initialized
C.1.2 Large Scale Constraints
In the demonstration of our patched-based approach, reconstruct point clouds with 10,000
uniformly sampled points. Our current implementation of the patch-based solver doesn't I support the hypernet kernel, so we regularize the reconstruction problem using a finite difference scheme 339 42
and impose the point constraints of ( + En;) = and & (x; -nn, -6, where we set E = 0.01
for all reconstructions. As well as the tunable support size, our implementation also places a hard limit on the total number of constraints that are considered for each query point; for all experiments we limit the nearest neighborhood size 60. Furthermore, our current implementation does not support tensor batching, the run-time for the patch-based approach, with marching cubes grid resolution of [160 x 160 x 160). approximately hours.
In the context of sparse solvers, we acknowledge that our current implementation is not completely vectorized for fast GPU execution with batched training points, although the operation of the solver itself is highly efficient (execution within seconds). This limitation primarily arises due to the limited support of sparse matrix operations in popular ML frameworks. However, is crucial emphasize that this constraint does not reflect shortcoming in CNF's design. We hope to attract more attention from the ML community to enhance support for sparse matrix operations within the frameworks to resolve this implementation constraint.
C.2 Evaluation of Normal Interpolation
To evaluate the merit of our normal interpolation approach for shape representation, we compare shape representation using our exact gradient constraints and pseudo- normmll constraints as used in NKF [4]] and SIREN 34 We use the same hypernet kernel approach for all experiments, with the same training setup as Sec. C.I.I except that we do not pretrain the hypernet encoder to represent a circle. For the pseudo-normal constraints, we the value of 0.1.
In Fig. [5 we show a visual comparison between the reconstructions achieved when using pseudo- normal constraints and exact gradient constraints. To quantitatively assess each approach also report the mean normal error:
N
En
[n(,,) n(xx)).
(21)
N
where nTxx;) V. (xx)) and fi(:;) is the ground truth normal value at æ;, The results are tabulated in Table
As is evident both from our qualitative visualization and quantitative evaluation, CNF is able to impose exact normal constraints on neural implicit surfaces during initialization and throughout training (to within floating point accuracy).
17

---

SIREN
Pseudo-normal (f)
Pseudo-normal (+)
Exact normal (t)
Exact normal (+)
Circle
1.71 x 10-2
0.776
0.499
4.516 x 10-6
1.083 x 10
Line
1.02 X 10-3
3.817
1.589 10-2
2.737 10-6
6.963 10-6
Triangle
3.21 X 10-1
0,466
0.238
7.371 x 10
6.292 x 10 10-6
Diamond
1.05 x 10-1
1.588
0.277
1.450 x 10
3.368 x 10-6
Table 6: Evaluation of error in the normal vector of the implicit field, as defined in Eq. 21 The symbols aadd refer measurements taken initialization and after training the field, respectively.
Psoudo-normal constraints
Exact normal constraints
Initalisation
Traned
Initalisation
Trined
Chrolo
MAAAAAAA
MAAAAAMA
Line
Triunglo
Diamood
Ground
Figure 15: We assess the effectiveness of our precise normal interpolation technique for shape representation. We present the normal vector of the encoded implicit surface (red arrow) before and after training to minimize an Eikonal constraint, for fields bound by pseudo-normal constraints and our exact constraint method. We compare with the ground-trutt -ruth shape normal vectors (black arrow); our approach produces surface that faithfully represents both the constraint points and the constraint normal vector.
18

---

C.3 Learning Curves
In Fig. 16 and Tab. we report the learning curves, training times, and inference times for various approaches in the experiment on interpolatory surface reconstruction with exact normal constraints. CNF outperforms prior work in terms of convergence, while maintaining similar training and inference efficiency.
0.35
Cled (STREN)
0.30
(Rypers krrzel)
Domood (SIREN)
[lypetnet leczd)
0.25
Trinngle (SHLEN)
Table 7: Average training and inference
Edkondl Less
(lyperse Line (SIREN)
time (in seconds) for surface reconstruc- tion with exact normal interpolation.
(Hlpernet
0.15
Method
Training
Inference
Hypernet kernel
242
0.0663
0.05
SIREN [34]
148.4
0.1437
Train Itee
Figure 16: Learning curves of various approaches for surface reconstruction with exact normal interpolation.
D Self-tuning PDE Solver
D.1 Experiment Details
We selected ID advection (described in Eq, 15 Sec. 4.4 as a demonstration of the effectiveness of CNF in solving partial differential equations (PDEs). This choice was made due to its simple, linear form. Additionally, one of the advantages is that the ground truth function has convenient analytical expression, eliminating the need for iterative solvers for validation. Specifically, our objective is to recover the following function:
u(x,1)= = sin 2m( - B(), E (0.1), E (0, 1).
(22)
D.2 Learning Curves
In Fig. and Tab. we report the learning curves, training times, and inference times for various approaches in the PDE experiment. We include SIREN 34 an additional baseline for evaluation. CNF outperforms SIREN in terms of convergence, while maintaining similar training and inference efficiency.
19

---

cards
Advection loss
SIREN
Skowed RBF
0.01
1e4
Table 8: Training and inference time (in sec- onds) of various approaches for self-tuning PDE solver.
100
Method
Training
Inference
Regression loss
STREN [34]
36.7
0.0087
RBF
160.7
0.0056
0.09
Skewed RBF
265.6
0.0061
1e4
Train Her
Figure 17: Learning curves of various approaches for self-tuning PDE solver.
D.3 Transfer Learning
In Sec. we detail how CNF can automatically fine-tune the hyperparameters of the skewed RBF on an irregular grid. Here, we illustrate that, given optimized hyperparameters of the skewed RBF for particular grid, we can employ the resulting skewed RBF directly solve a different advection equation (such as different initial condition) at inference time, without additional training, if the grid remains the same.
For demonstration, we incrementally add shift, to the initial condition of the advection equation, starting from small to large shifts:
uo (x) sin (2TT) + +,, I E (0,
(23)
The hyperparameters of the skewed RBFs were trained solely when = O. When the initial condition, ur(r), shifted, the new advection equation can be solved with the same kernel inference time. No further training was conducted; see Tab. 9 for the results.
The efficacy of this approach optimization problem as a collocation problem with learnable inductive bias. Once the preferred inductive bias away from the collocation points is obtained, solving PDE the same collocation points is reduced to a pure collocation problem, which requires minimal to no further training.
be explained by perspective of formulating the constrained
Table Demonstration of transfer learning with the skewed RBF kernels. The kernels were initially trained exclusively for ID advection function with no shift (first row). Subsequently, their transfer learning capabilities were tested at varying shifts. No training was conducted other than for the first
row.
Shift J
Random init
Pre-trained
Skewed RBF
Skewed RBF
RMSE
nRMSE
RMSE
nRMSE
0.0127
0.0227
0.0070
0.0129
0.0460
0.0736
0.0049
0.0074
10
0.5788
0.0594
0.0177
0.0018
100
5.8613
0.0587
0.2221
0.0022
1000
60.143
0.0602
2.3113
0.0023
10000
635.78
0.0636
23.844
0.0024
20

---

FTN
Kewel FRR
Kewl
Figure 9: Learned material appearance.
21

---

Coler
215
MERDD
FTN
Sits
Kewd FR
Kerl 94R6 (bae)
Figure 10: Learned material appearance.
22

---

wobeea 200
MERDD
FTN
Kewd FRI
Kewl 94R6 (bae)
Figure 11: Learned material appearance.
23

---

MERDO
FTN
Kewd FRI
Kewl 94R6 (bae)
Figure 12: Learned material appearance.
24

---

MERDD
FTN
Kewd FRI
Kewl
Figure 13: Learned material appearance.
25

---

S15
Figure 14: Learned material appearance.
26


Reviews:
[{"review_id": "6ovdGXEkEA", "reviewer": "Reviewer_t12b", "summary": "The authors look at enforcing hard constraints on neural fields. Here, the problem formulation is to take continuous coordinates as input and predict the solution on these points as output. The neural field is represented as a linear sum of basis functions, and specifically, variants of a neural kernel function. The constraints must be linear operators which are then satisfied via using a linear solver. ", "questions": "- Is the constraint enforced at both training and test time?\n\n- What linear solvers are being used here? The authors mention “note that when employing general solvers such as SQP and DC3…” but don’t give details about the solvers themselves.\n\n- What is the training time and inference time, and how does it compare to other NN fitting approaches? In particular, these linear solvers could be very expensive.\n\n- Can you show an example of using the method when the constraint is a non-linear operator?\n\n- In example 4.3, is eqn. 12 (Eikonal equation) being solved in a “soft constraint” way? Where are the hard constraints being enforced?\n\n- The format of some of the references needs to be corrected. For example, references 7, 16, 25, 39 do not specify a publication venue.", "limitations": "- Limitations are not discussed (besides that this approach does not work for non-linear operators), but it seems like this approach would also be difficult to scale up for larger systems because of the expensive linear solves on larger systems.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Enforcing constraints more precisely on neural fields could improve prediction performance, and trying to do this via harder constraints seems promising.\n\n- The method utilizes flexible representations of neural fields (i.e., neural kernel fields; as well as other representations of the basis functions) to enforce the relevant constraints. ", "weaknesses": "- The evaluation metrics are limited, and it is hard to contextualize these results with respect to other neural field methods. For example 1, the authors compare to other neural representation methods, but what about comparing to “soft constraint” approaches as well? It would be helpful to know how speed vs accuracy compares when enforcing the constraint is a softer way. \n\n- There is no discussion of the speed and training time to implement this hard constraint approach. It seems like it would be expensive to do a linear solve on these matrix systems, greatly hindering efficiency. More details about the linear solver would be helpful (as well as how the authors treat the system as fully differentiable).\n\n- One major limitation of this approach is that only linear operators are used as constraints (thereby being able to utilize eon 3). The practicality of this method seems especially useful when the operator is non-linear.\n\n- It would be helpful to describe the problem formulation for each problem (inputs/outputs, form of constraint, what is being minimized, etc.), as this is not always clear from the paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback.\n\n## Comparison to soft constraint approaches\n\nIn our experiment on learning material appearance described in Sec. 4.2, we compared CNF with FFN [35] and SIREN [32], two representative soft constraint methods. Our approach surpasses them qualitatively and quantitatively. Complete results are available in the supplementary (Section B3). \n\nFurthermore, we conducted extra evaluations by comparing CNF with SIREN in shape-reconstruction and PDE-solver experiments. CNF excels in both cases; please refer to Q2 in the general response for details. For a theoretical argument on CNF's superiority over soft approaches, please see Q3 in the general response.\n\n## Training and inference efficiency, large-scale system\n\nDetailed insights into training duration, inference speed, and learning curves are available in the general response (Q2). CNF is efficient in both training (minutes) and inference (seconds).\n\nThe design of CNF also effectively handles large-scale problems. We developed a hybrid kernel to ensure matrix sparsity (line 163-170) and a patch-based sparse solver (introduced in lines 181-185 and evaluated in line 252-255) tailored for this purpose.\n\nImportantly, as CNF weights, $\\beta$, can be precomputed, inference does not require solving matrix equations. This further enhances its applicability to large-scale problems. For additional inference procedure details, please see general response (Q1).\n\nIn the context of sparse solvers, we acknowledge that our current implementation is not completely vectorized for fast GPU execution with batched training points, although the operation of the solver itself is highly efficient (execution within seconds). This limitation primarily arises due to the limited support of sparse matrix operations in popular ML frameworks. However, it is crucial to emphasize that this constraint does not reflect a shortcoming in CNF's design. We hope to attract more attention from the ML community to enhance support for sparse matrix operations within the frameworks to resolve this implementation constraint. We will include this discussion in the revised paper.\n\n## Details about the linear solver\n\nFor typical cases, we solve the matrix equation through LU decomposition with partial pivoting and row interchanges, which is fully differentiable if the matrix is full rank. Please see general response Q1 for details on how the linear solver is integrated in training.\n\nFor large systems, we use the hybrid kernel basis to ensure matrix sparsity (line 163-170) and a patch-based sparse solver (line 181-185) for solving large sparse matrix systems.\n\n## Practicality of solvers to linear operator constraints vs nonlinear operator constraints\n\nThe linear operator constraints cover a wide range of real-world problems, such as interpolating fixed points, fitting exact differentials, and solving linear ODEs and PDEs. Our performance is the best compared to all the prior approaches on the same linear problems. \n\nMost importantly, a general solution to nonlinear problems is typically not the ideal solver for linear problems, which typically require special treatment to improve efficiency. For example, the simplex method as a solver for linear programming typically takes less iterations and memory compared to a general solver. Therefore, linear and nonlinear problems are two separate problems requiring different solutions. A nonlinear solver cannot be a universal solution for both scenarios.\n\nTherefore, we focus on linear problems, and leave nonlinear problems as separate future work.\n\nCNF can be extended to non-linear operator constraints if step 1 of the training process (please refer to the general response Q1) is replaced with a nonlinear solver to solve a nonlinear version of Eq 3. However, such a nonlinear solver would not be an ideal solution for linear operator constraints, as its convergence and efficiency are both difficult to analyse and promote.\n\n## Problem formulation of each experiment\n\nWe are happy to provide detailed math formulation of each experiment in the author-reviewer discussion period. Unfortunately, the rebuttal does not allow for a sufficient word limit to include these details.\n\n## Is the constraint enforced at both training and test time?\n\nYes, please refer to the training and inference details in Q1 of the general response for how the constraint is enforced at both training and test time.\n\n## Discussion of SQP and DC3\n\nPlease refer to general response Q3 for details.\n\n## Show an example of using the method when the constraint is a non-linear operator?\n\nCNF can be extended to non-linear operator constraints if step 1 of the training process (please refer to the general response) is replaced with a differentiable nonlinear solver to solve a nonlinear version of Eq 3. Any iterative method that converges, such as Gauss-Newton, can be a candidate solver. However, the challenge lies in the convergence and the costly computational graph of such iterative solutions. The latter can be potentially addressed through the use of implicit layers [12], which we leave as future work.\n\n## Is the Eikonal equation being solved in a “soft constraint” way? Where are the hard constraints being enforced?\n\nThat is correct, as the Eikonal equation is a non-linear PDE we choose to introduce its geometric bias into the optimization as a soft constraint via training. In this case, the hard constraints are on the points and the point normals themselves - that is, for all $x$ and $n(x)$ in our point set $P$, $F(x) = 0$ and $\\nabla F(x) = n(x)$.\n"}, {"review_id": "oKzS2g4Hsf", "reviewer": "Reviewer_RAef", "summary": "The paper presents a method for integrating hard constraints, represented by a linear operator, into neural field basis functions. This is achieved by learning kernel functions as basis functions at specific constraint points. Through experimentation, the paper provides evidence to show the effectiveness of the proposed method in comparison to unconstrained neural fields across diverse practical tasks.", "questions": "- To enhance clarity, it would be better for the authors to dedicate a separate section to explaining the model training process in detail. This section needs to explicitly highlight which parameters or weights are optimized and specify the optimizer/solver employed for this purpose. ", "limitations": "The authors have addressed the limitations of the work in the Summary section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The paper introduces a novel approach by directly applying linear operator constraints to the basis functions constructed with neural field functions and learning a linear representation. Also, the weights are found by applying a solver to a linear system, which is nice and removes some optimization problems when the weights should also be learned.\nQuality: The results obtained in the paper demonstrate the effectiveness of the proposed method across various tasks. I appreciate the efforts in comparing different common implementations of neural field bases.\nClarity: The paper effectively explains the reasoning behind critical implementation choices, such as the selection of basis functions and the choice of kernel.\nSignificance: The proposed approach addresses the significant challenge of the application of explicit hard constraints for neural fields.", "weaknesses": "- As a reader, I found it challenging to comprehend the training procedure without delving into the code. Therefore, providing a comprehensive explanation would facilitate understanding. This would make it easier to grasp the methodology.\n- Furthermore, sharing more details about how regularization is applied would provide valuable insights into the approach. Explaining the specific methods used for regularization and their impact on the model's performance would enhance the clarity and comprehensiveness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n## Details regarding the training procedure\n\nWe appreciate your suggestion to dedicate a separate section to explain the training process in detail. Please refer to the general response Q1 for training details.\n\n## Regularization\n\nThe only regularization we recommend is a term containing the condition number of the matrix in Eq. 4. The condition number is a standard metric in numerical linear algebra that measures the sensitivity of the solution to errors in the input data, such as matrix coefficients or the right-hand side vector. A singular, noninvertible matrix has an infinite condition number.\nA smaller condition number corresponds to a smaller error in the solution satisfying the hard constraints.\n\nThe condition number can be added as a regularization term to the main loss. It can be computed by the ratio of the maximal and minimal singular values of the matrix from SVD decomposition.\n\nSmoothness is also a commonly preferred quality of neural fields, depending on the task. Smoothness can be promoted by adding a total variation term.\n"}, {"review_id": "Zm6Gk3jNdy", "reviewer": "Reviewer_rctK", "summary": "A broad range of problems can be formulated as linearly constrained problems, e.g., learning material appearance, interpolatory reconstruction, solving linear PDE, etc. In order to solve linearly constrained optimization problems, this paper developed a novel hard constraint method that builds upon neural fields and differentiable linear solver, naming constrained neural fields (CNF). \n\n**Contribution**:\n1. A novel methodology CNF is proposed, specifically, linear equality constraints are transformed into a linear system, i.e., eq. (3)-(4);\n2. Then both weights of neural fields, $\\beta_i$, and learnable parameters, $\\theta$, of neural fields (see eq. (2)) can be learned from gradient descent of the objective function in eq. (1) given a differentiable linear solver is applied to eq. (3)-(4);\n3. Hyper kernel basis is proposed, which is benchmarked with various basis functions (see section 3.2) and demonstrates advantages such as stable conditional number throughout training (Fig. 5);\n4. In experiments, 4 examples from very different background are solved by CNF with superior performance, showing the potential of CNF as a general learning framework for linearly constrained problem.", "questions": "N.A.", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality**: CNF is a novel method of solving linearly constrained problem by implementing neural fields based on differentiable linear solver.\n\n**Quality**: The paper shows that CNF can solve various problems with high performance.\n\n**Clarity**: The methodology is clearly presented. Worthy of mentioning, the analysis of conditioning of the matrix due to different kernel methods (see Appendix A) is quite convincing about why the authors believe Gaussian kernel with hybrid kernel basis (see eq. (9)) is the best choice.\n\n**Significance**: This work can be applied to a broad range of linearly constrained problems. ", "weaknesses": "The work relies on differentiable linear solvers, and therefore the most suitable problem for CNF is a linearly constrained problem. In section 5, the authors discussed that CNF can be applied to nonlinear problems given a differentiable solver. However, such a differentiable solver to nonlinear problems is in general not easy to get. Hence, CNF is currently limited to linear problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the positive comments and feedback. \n\nWe acknowledge the challenges posed by nonlinear problems, particularly in terms of convergence and the expensive computational graph of nonlinear solvers. Potentially, the latter could be addressed through the use of implicit layers. We plan to further study it in future work."}, {"review_id": "xm5TsSE0jA", "reviewer": "Reviewer_o12o", "summary": "The authors propose a method to enforce hard constraint points on neural fields. Instead of a single black-box coordinate network predicting the field values, this work uses neural networks to learn basis functions which are then combined in a linear transformation. Given enough basis functions, the weights of this linear transformation can be found by solving a system of linear equations.", "questions": "In Eq. 11, why do you minimize the log L1 loss, not the log L2 loss?", "limitations": "While some advantages and disadvantages of various basis functions are mentioned, the authors do not provide a limitations section and only lightly touch on general limitations of the method.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand. The authors have included their source code which seems reasonably well organized.\n\nThe method itself is simple and guarantees that constraint points are not violated. It outperforms existing methods on the MERL BRDF dataset.\n\nThe paper includes a varied selection of experiments to validate the method, and some ablation studies have been performed and reported in the appendix.\n", "weaknesses": "The paper contributes little in terms of theory. While the derivation is simple enough, it would have been nice to see a theoretical argument for why this method converges faster than unconstrained training, at least for special cases.\n\nThe constraints seem to be applicable only to single points, which is not clear from the abstract. Consequently, initial states and boundary conditions cannot properly be handled using this method.\n\nExperiments 4.1 and 4.4 are synthetic and very simple. Only experiments 4.2 and 4.3 measure real-world performance.\n\nThe paper introduces six basis functions in chapter 3.2 but they are never compared against each other in the experiments. The authors simply employ different basis functions for different experiments. This makes it hard to judge which one to use for any given problem. Comparisons against state-of-the-art is also sparse. Only the BRDF fitting experiment compares to related work.\n\nThe paper does not include learning curves for the various experiments. Figure 5 is somewhat related to learning curves, but the quantitative evaluation of the training process is severely lacking. Training time is not discussed in the paper either. Learning curves against wall-clock time would be appreciated.\n\nMinor:\n* L11: The claim in “Our approaches are demonstrated in a wide range of real-world applications.” seems a bit of a stretch. It’s more like one to two.\n* L41: Missing citation for the statement about inequality constraints.\n* L72-78: The review of previous work is missing stream functions for divergence-free fields (e.g. Deep fluids, 2019) and other conservation properties (e.g. Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics, 2022). Hamiltonian and Lagrangian networks have also been used with conserved properties in mind.\n* The citations in the main text do not have hyperlinks to the references at the end.\n* The hybrid kernel basis is poorly explained in 3.2.\n* Footnote 2 is not explained. It is unclear what kind of LaTeX expressions can be specified.\n* Chapter 5 is more outlook than summary\n* A.2 and Figure 5: Which experiment does this belong to?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback.\n\n## Theoretical analysis\n\nWe greatly appreciate this suggestion. Please refer to the general response Q3.\n\n## Summary of basis functions\n\nHere, we summarize the properties of various basis functions.\n\n|                                                                  | Independent basis | Constraint basis | Hypernetwork basis | Dot-product Kernel basis | Gassian Kernel basis | Hypernetwork kernel basis |  Hybrid kernel basis  |\n|:----------------------------------------------------------------:|:-----------------:|:----------------:|:------------------:|:---------------------------------:|:-----------------------------:|:-------------------------:|:---------------------:|\n|                         Learning capacity                        |                   |       Poor       |        Poor        |                Fair               |              Good             |            Good           |          Good         |\n|                        Linear independence                       |                   |       Poor       |        Fair        |                Poor               |              Good             |   Good (if using Gaussian)  |          Good         |\n|                          Matrix sparsity                         |       Dense       |       Dense      |        Dense       |               Dense               |             Sparse            |  Sparse (if using Gaussian) |         Sparse        |\n|  Number of model parameters independent of number of constraints |         No        |        Yes       |         Yes        |                Yes                |              Yes              |            Yes            |          Yes          |\n|                       Controllable sparsity                      |         No        |        No        |         No         |                 No                |               No              |             No            |          Yes          |\n|                     Higher-order constraints                     |        Yes        |        Yes       |         Yes        |                 No                |               No              |            Yes            | Yes (if using hypernet) |\n\n\nFor problems lacking high-order constraints, like the BRDF experiment, we recommend the Gaussian kernel basis. For problems involving high-order constraints, such as sparse shape reconstruction (line 239-251), we suggest using the hypernetwork kernel basis. For large-scale tasks, such as dense shape reconstruction (line 252-255), we recommend the hybrid kernel.\n\nPlease refer to Sec. 3.2 for relevant explanations, and supplementary A2 and B2 for empirical comparison.\n\n## Single point constraints\n\nWe agree that our method applies to discrete points rather than a continuous set. However, CNF is perfectly suitable for initial and boundary value problems, as demonstrated in Sec 4.4. In fact, all the major PDE solvers such as FDM, FEM, and spectral methods solve the initial and boundary problems at discrete grid points. CNF can improve the performance of the Kansa method, a type of spectral method for solving general PDEs.\n\nCompared to mesh-based solvers such as FDM and FEM, the Kansa method has the advantage of solving a PDE on an irregular grid without meshing, i.e. grouping the points into triangles or quadrilaterals. CNF addresses a major limitation of the Kansa method (please refer to line 260-266). We plan to perform a more extensive evaluation of CNF in comparison to other PDE solvers in future work.\n\n## Hybrid kernel basis\n\nThe hybrid kernel was designed to promote the sparsity of the matrix in Eq. 4.\n\n$\\Psi_i\\left(\\mathbf{x}\\right) = \\kappa\\left(\\phi_i\\left(\\mathbf{x}_i\\right), \\phi_i\\left(\\mathbf{x}\\right)\\right) \\kappa_G\\left(\\mathbf{x}_i ,\\mathbf{x}\\right)$\n\nThe first part $ \\kappa\\left(\\phi_i\\left(\\mathbf{x}_i\\right), \\phi_i\\left(\\mathbf{x}\\right)\\right)$ could be either the Gaussian kernel basis or the hypernet kernel as defined in Sec. 3.2 (depending on the task). We multiply it with a compactly supported function $\\kappa_G$ so that the matrix sparsity can be explicitly adjusted. A candidate $\\kappa_G$ can be a truncated Gaussian kernel such as:\n\n$\n    \\kappa_G\\left(\\mathbf{x}_i, \\mathbf{x}\\right)  =  \\begin{cases}\n       \\exp\\left( -\\frac{\\|\\mathbf{x}_i- \\mathbf{x}\\|^2}{2\\sigma^2}\\right) & \\text{if } \\|\\mathbf{x}_i- \\mathbf{x}\\| < 3\\sigma\\\\\n        \\text{ or } 0 & \\text{if } \\|\\mathbf{x}_i- \\mathbf{x}\\| \\ge 3\\sigma.\n    \\end{cases}\n$\n\nReducing $\\sigma$ yields a matrix with more zero entries.\n\n## Comparisons against SOTA\n\nBesides the BRDF fitting experiment, our shape reconstruction experiment also compares with the state-of-the-art work, NKF. It is important to note that, unlike our work, the design NKF cannot model normals with hard constraints. Please refer to supplementary A1 for theoretical explanation and C2 for empirical evaluation.\n\n## Footnote 2\n\nThis highlights our user-friendly interface for defining complex, higher-order linear operators (source code in `diff_utils.py` in supplementary):\n```\ndef compute_op(latex_str, y, x)\n```\nIn the context of the advection operation $\\frac{\\partial f(x, t)}{\\partial t} + \\beta(x) \\frac{\\partial f(x, t)}{\\partial x}$, our interface can parse `latex_str` if provided as `f_{x_1} + {beta}f_{x_0}` and compute this operation automatically. The usage of this interface will be further clarified through illustrative examples, which will be provided alongside the code release.\n\n## A.2 and Figure 5\n\nThis is a standalone experiment where we minimize the condition number of the matrix $A_f$ in Eq. 4 given various basis functions. This experiment measures the inherent linear independence of basis functions of different designs.\n\n## log L1 loss\n\nWe follow the prior work NBRDF, which also uses log L1, for a fair comparison. BRDF values have a large variation in scale (0.1-700). To correctly fit both ends, logL1 provides a good balance as reported by the NBRDF paper."}]

Author rebuttal:
We thank the reviewers for the detailed and constructive feedback. Below are our responses to the common questions:

# Q1. Training and inference details

We offer a thoroughly tested codebase that assists users in modeling challenging constraints using CNF. To ensure comprehensiveness, we will also incorporate the following descriptions and pseudocode blocks in the supplementary.

Recall the training objectives

$
\underset{\theta}{\arg\min} $ $\mathcal{L}\left(f_\theta;\theta\right) \quad \text{s.t. } 
\mathcal{F} \left[f_\theta\right] \left(\mathbf{x}\right)
= g\left(\mathbf{x}\right) \;\forall \mathbf{x} \in \mathcal{S}:= \left\lbrace x_i\right\rbrace_{i=1}^I,
$

where
$
f_\theta\left(\mathbf{x}\right) = \sum_i \beta_i \odot \Psi_i\left(\mathbf{x}\right),
$

and $\theta$ indicates the learnable parameters of each basis function $\Psi_i$.

The following procedure describes the training process:

```algorithm
Repeat:
    1. Compute 𝜷 by solving Eq. 4
    2. Compute gradient ∂𝐿/∂𝜃 = ∂𝐿/∂𝑓 ∂𝑓/∂𝜃
    3. Update 𝜃 via gradient descent
Until converged
```
Step 1 computes the weights $\mathbf{\beta}$ to ensure that the constraints are always satisfied throughout training. Steps 2 and 3 update the training parameters $\theta$ to optimize the training loss $\mathcal{L}$ under the constraints.

$\mathbf{\beta}$ is computed using an LU decomposition with partial pivoting and row interchanges, which is fully differentiable if the matrix in Eq.4 is full rank.
The computation of $\mathbf{\beta}$ constructs a computational graph that tracks $\frac{\partial \mathbf{\beta}}{\partial \theta}$. 

Any loss function with a valid gradient $\frac{\partial \mathcal{L}}{\partial f}$ can be smoothly integrated into our training process.

Next, we have the inference algorithm:

```algorithm
Input: 𝑥
Output: 𝑓_𝜃(𝑥)

If 𝜷 is None:
    Compute 𝜷 from Eq. 4
else:
    𝑓_𝜃(𝑥) ← Σᵢ 𝜷ᵢ ⊙ Ψᵢ(𝑥)
```
Here, $\mathbf{\beta}$ only needs to be pre-computed once since it does not depend on the evaluation point $\mathbf{x}$. As a result, performing inference with CNF is very efficient and boils down to computing a linear combination of $I$ basis functions, a task that can be vectorized for efficiency.

# Q2. Training/inference time, learning curve, and additional evaluation

Please refer to the attached PDF for details. CNF is efficient in training (minutes) and inference (seconds). We also compare with SIREN [32] for shape reconstruction and solving PDEs. CNF demonstrates superior performance in all experiments compared to SIREN.

# Q3. Theoretical analysis

Here, we provide a summary of the theoretical argument that elucidates CNF's superior performance over existing solutions:

### Comparison to general approaches

Many general algorithms in scientific computing, including the popular SQP and more recent DC3 [11], adopt a formulation for the constraint optimization problem as:

$
\underset{\theta}{\arg\min} $ $ \mathcal{L}\left(\theta\right)
\quad \text{s.t. } \quad 
h_1 \left(\theta \right)
= 0,  h_2 \left(\theta \right)
= 0, ...
$

where $\theta$ denotes the learnable parameters. When it comes to the case where $\theta$ represents the weights of a deep neural network, the constraints become extremely high-dimensional and nonlinear, especially when the number of constraints grows. Under this formulation, a constraint is only linear when $h(\theta)$ is a linear function of $\theta$. Therefore, the linearity does not hold when $\theta$ represents the weights of a neural network.

In our formulation, we only require the operator $\mathcal{F}$ to be linear, while the neural basis functions $\Psi$ can still be highly nonlinear with respect to $\theta$. Therefore, our formulation reduces the problem's complexity and allows us to explicitly determine and promote the existence of the solution.

### Comparison to soft constraint approaches

While there have been attempts to model constraints by overfitting an NN trained with regression, CNF has several clear advantages over such soft approaches:
- CNF satisfies hard constraints without training, while soft approaches may require extensive training.
- Despite extensive training, soft constraint approaches may fail to satisfy hard constraints due to inherent limitations in their learning capacity. In contrast, CNF provides a robust guarantee of hard constraint satisfaction within machine precision error, provided the condition number is small.
- Another drawback of employing soft approaches becomes evident when effectively imposing priors. The incorporation of priors often involves introducing an additional term to the loss. Consequently, a tradeoff arises between the constraints and the prior, which is controlled by a hyperparameter. In contrast, CNF offers a clean solution without such a tradeoff. With CNF, the inclusion of priors does not compromise hard constraints, thereby maintaining a harmonious balance among various aspects of the model.

### Generalization of NKF and PDE-CL

CNF generalizes the prior works NKF [38] and PDE-CL [25]. NKF employed dot-product kernel bases for 3D reconstruction, while PDE-CL used constraint bases to solve PDEs. However, their bases exhibit poor performance in terms of linear independence and learning capacity (refer to supplementary A2, B2). Additionally, their dense matrices make them unsuitable for large-scale problems. The simple dot-product kernel in NKF also cannot handle higher-order constraints (refer to supplementary A1 for explanations and C2 for evaluation).
We introduce several novel variations of basis functions to enhance linear independence, learning capacity, and matrix sparsity, along with strategies to analyze and promote solution existence. Our work unifies NKF and PDE-CL and demonstrates that CNF can be applied to general constrained optimization problems.


Meta-review: The paper proposes a method for neural solvers for optimization problems with hard (linear) constraints. At a high level one can view this as a collocation method where the basis functions are learnable and parametrized by shallow neural networks. The net benefit is that the constraints are automatically enforced, and no tuning or regularization is necessary --- a property that is particularly helpful for enforcing higher order differential constraints while solving PDE systems.

The core idea is somewhat simple but well executed. The paper writing and evaluations across a range of applications are clearly communicated.  During the discussion phase, there were concerns raised about the methods' limitations (currently its scope is restricted to problems with linear constraints) but I feel that the authors responded well to these concerns. As such I am happy to recommend acceptance.
Decision: Accept (poster)