SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Transfer learning Since CNF is formulated as a collocation method with a learnable inductive bias, we can pre-train the inductive bias of basis functions with custom configurations, such as well-conditioning, smoothness, or more advanced variations. This allows us to apply the pre-trained basis functions to unseen data, directly computing the weights with little or no additional training, as the problem reduces to a near-collocation scenario. We demonstrate this by solving a partial differential equation on an irregular grid in inference time and without further training; see Appendix.
Sparse solver Drawing inspiration from classical reconstruction techniques [23, 40], we introduce a patch-based approach that leverages the restricted support induced by the hybrid kernel function (Eq. 9) to solve the sparse linear system. For each evaluation point  $\pmb{x}$ , we construct a subset of basis functions such that  $\Psi_{i}(\pmb{x})\neq \mathbf{0}$  using the support criterion induced by  $\kappa_{G}$ . This enables us to create a lower-dimensional dense submatrix  $(\mathbf{A}_f)_{\kappa_G}$  to solve for the weights of only the nonzero items so that the size of the system is sufficiently small to fit within the processor's memory limitations.
For efficient adoption of CNF in real-world applications, we have developed a PyTorch framework that abstracts the definition of the basis function, enabling users to conveniently define and experiment with custom architectures. The framework also enables vectorized basis function operations for highly efficient training and inference. Additionally, users have the flexibility to specify an arbitrary linear differential operator constraint by providing a regular expression in  $\mathrm{LATEX}$ . This can be processed to compute the exact post-operation function evaluation via automatic differentiation, thereby eliminating the need for manual execution. Further clarification of the features and usage of our framework, along with illustrative examples, is provided alongside the code release.
With this framework, we leverage CNF in four real-world applications, utilizing distinct basis functions tailored to the specific context of the problem. We elucidate and empirically demonstrate the merits of the chosen basis function in each scenario. Our intention is to use these examples as a guideline for selecting appropriate basis functions, which has been briefly summarized at the end of Sec. 3.2. Apart from Sec. 4.1, which serves as a toy example, we demonstrate unique advantages or state-of-the-art results achieved in each application when employing CNF.
4.1 Fermat's Principle
Fermat's principle in optics states that the path taken by a ray between two given points is the path that can be traveled in the least time. This can be formulated as a constrained optimization problem:
$$
\underset {\mathcal {C}} {\arg \min } \int_ {\mathcal {C}} \frac {d \mathbf {s}}{v} \quad \text {s . t .} \quad \partial \mathcal {C} = \left\{\mathbf {p} _ {0}, \mathbf {p} _ {1} \right\}, \tag {11}
$$
where  $ds$  is the differential arc length that the ray travels along contour  $\mathcal{C}$  and  $v$  is the light speed that varies along the contour due to the refractive index of the medium. The contour's entry and exit points  $\partial \mathcal{C}$  are hard constraints and the contour is optimized to minimize the travel time  $\int_{\mathcal{C}} \frac{ds}{v}$ . We model the contour  $\mathcal{C}$  as a collection of points from a parametric equation, i.e.  $\mathcal{C} := \{\mathbf{p}(x) := \beta_1 f_1(x) + \beta_2 f_2(x) \mid x \in [0, 1]\}$ . As there are in total two constraint points, we use two independent quadratic polynomials  $\{f_1, f_2\}$  as basis functions. We chose polynomials over neural networks due
Figure 1: The optical paths and travel time before and after the optimization. The constraints are strictly enforced throughout the optimization.
Table 2: Quantitative evaluation of learning material appearance representations. We evaluate each method by rendering the learned BRDFs with environment map illumination and computing the mean and standard deviation of image metrics across all materials in the MERL dataset [21].
RMSE×10-2↓	PSNR↑	SSIM↑
NBRDF [35]	1.57 ± 2.86	44.96 ± 11.55	.96 ± .07
FFN [37]	1.16 ± 1.36	44.72 ± 10.46	.98 ± .03
SIREN [34]	3.32 ± 3.05	33.43 ± 8.84	.93 ± .08
Kernel FFN	0.94 ± 1.16	46.32 ± 9.92	.98 ± .03
Kernel SIREN	0.60 ± 0.54	47.95 ± 7.99	.99 ± .01
to the simplicity of the problem. The weights  $\{\beta_1, \beta_2\}$  are solved differentially to fit hard constraints, while the coefficients of the polynomials are learnable parameters that can be updated with respect to the loss function. We show the results in Fig. 1 with an increasing refractive index along the Y-axis.
4.2 Learning Material Appearance
In visual computing, the synthesis of realistic renderings fundamentally requires an accurate description of the reflection of light on surfaces [11]. This is commonly expressed by the Bidirectional Reflectance Distribution Function (BRDF), described by Nicodemus et al. [25], which quantifies the ratio of incident and outgoing light intensities for a single material:  $f_{r}(\omega_{i},\omega_{o})$ , where  $\omega_{i},\omega_{o}\in S^{2}$  are the incident and outgoing light directions.
Our aim is to learn a neural field  $\Phi_{\theta}(\omega_i,\omega_o)$  with parameters  $\theta$  that closely matches the ground truth BRDF  $f_{r}(\omega_{i},\omega_{o}),\forall \omega_{i},\omega_{o}\in S^{2}$ . We constrain  $\Phi_{\theta}$  by selecting 100 sample pairs of  $(\tilde{\omega}_i,\hat{\omega}_o)$ , where half of the pairs are uniformly sampled, and the other half of  $\tilde{\omega}_{o}$  are concentrated around the direction of the perfect reflection of  $\tilde{\omega}_{i}$ . This means that the latter constraint points are situated around the specular highlight, which contains high-frequency details that regular neural networks struggle to learn [32, 31, 19, 35]. We illustrate these issues in Fig. 2, where we compare CNF, formulated with a Gaussian kernel basis (Eq. 7), with SIREN [34] or FFN [37] as the encoder network, against multiple state-of-the-art neural-based fitting approaches [34, 37, 35] for BRDF reconstruction. We perform this evaluation on highly specular materials from the MERL dataset [21], with size-matching networks for a fair assessment, and subsequently rendering a standard fixed scene; see Appendix for the complete MERL rendering results. Following previous work [35], we train  $\Phi_{\theta}$  on  $640\mathrm{k}(\omega_i,\omega_o)$  samples, minimizing L1 loss in the logarithmic domain to account for the large variation of BRDF values due to the specular highlight, while enforcing the aforementioned hard constraints on  $(\tilde{\omega}_i,\hat{\omega}_o)$ :
$$
\arg \min  _ {\boldsymbol {\omega} _ {i}, \boldsymbol {\omega} _ {o}} | \Phi_ {\theta} (\boldsymbol {\omega} _ {i}, \boldsymbol {\omega} _ {o}) - \log (f _ {r} (\boldsymbol {\omega} _ {i}, \boldsymbol {\omega} _ {o}) + 1) |
$$
$\theta$  s.t.  $\Phi_{\theta}(\hat{\omega}_i,\hat{\omega}_o) = \log (f_r(\hat{\omega}_i,\hat{\omega}_o) + 1)$  (12)
We summarize the results computed over all MERL materials in Tab. 2, where CNF demonstrates superior performance in learning material appearances with state-of-the-art accuracy; see Appendix for additional evaluations and ablation studies.
4.3 Interpolatory Shape Reconstruction
The representation of shapes via neural implicit functions [15] has recently garnered much attention [28, 2, 34]. Compared to traditional forms of representation such as point clouds, meshes, or voxel grids, neural implicit functions offer a flexible and continuous representation capable of handling unrestricted topology. In this section, we demonstrate how CNF can be applied to enforce exact interpolation constraints on implicit shape representations, while retaining the ability to train the field with geometry-specific inductive bias. Given an oriented point cloud  $\mathcal{P} := \{(x_i, n_i)\}_{i \in I}$ , our aim is to construct a continuous implicit function  $\Phi(\pmb{x})$  on the domain  $\Omega$  whose 0-level-set  $S := \{\pmb{p} \in \Omega \mid \Phi(\pmb{p}) = 0\}$  represents the surface of the geometry.
Exact normal constraints We first demonstrate how our framework can be utilized to learn an implicit shape representation, while exactly interpolating both surface points and their associated
Figure 2: Learned material appearance by enforcing hard constraints around the specular highlights.
Figure 3: (a): Learning shapes from 2D point clouds (yellow points) with oriented normals (black arrows) using CNF; black lines depict the zero level sets of the learned implicit field. The method interpolates both exact point and normal constraints during initialization and throughout training, yielding a plausible surface upon minimizing the Eikonal constraint across the field. (b): Surface reconstruction of 3D point clouds containing 10,000 points.
normals. We conduct experiments in 2D, and formulate CNF with a hypernetwork kernel basis (Eq. 8), as the problem involves both zero-order and higher-order constraints at repeated constraint points. Note that this approach is the first to enforce exact normal constraints on a neural implicit field, as prior works [34, 41] can only approximate the exact normal with a pseudo-normal constraint [39]. In contrast, we constrain the implicit field such that  $\Phi (\pmb {x}_i) = 0,\forall \pmb {x}_i\in \mathcal{P}$  and  $\nabla \Phi (\pmb {x}_i) = \pmb {n}_i$ $\forall \pmb {x}_i$ $n_i\in \mathcal{P}$ . Inspired by the geometrically motivated initialization [2], we use weights that are pre-trained to represent the signed distance function (SDF) of a circle with the desired number of constraint points. We found this placed the network in a more favorable starting position to produce plausible surfaces under new constraint points. We then train the network to solve the Eikonal equation, following previous work [15], via minimizing the quadratic loss function:
$$
\mathcal {L} = \mathbb {E} _ {\boldsymbol {x}} \left(\left\| \nabla_ {\boldsymbol {x}} \Phi (\boldsymbol {x}) \right\| - 1\right) ^ {2}, \tag {13}
$$
where the expectation is taken with respect to a uniform distribution over the domain  $\Omega$ . As shown in Fig. 3(a), CNF produces plausible surfaces to explain the point cloud, with the advantage of not depending on manufactured offset points [39] to ensure the existence of a non-trivial implicit surface.
Large-scale constraints To demonstrate our hybrid kernel approach (Eq. 9) with the sparse solver for handling large-scale constraints, we reconstruct 3D point clouds comprising 10,000 points uniformly sampled over the surface. We set the support radius of the kernel to four times the average nearest neighbor distance, and assign a constant value of  $10^{5}$  to regions of space with no support. The reconstruction is obtained by adhering to the points and pseudo-normal constraints [39, 41] during initialization, with no additional training required for the dense reconstruction to produce smooth surfaces, as shown in Fig. 3(b); see Appendix for additional results and evaluations.
Figure 4: Left: Ground truth advection function and skewed input grid with  $\sigma = 0.1$  perturbation. Right: Error maps for simple and skewed RBF.
Table 3: Solving the 1D advection PDE  $(\beta = 0.1)$  with a simple (left) and skewed (right) RBF kernel. Both versions were optimized by our framework. We can see the benefit of using the skewed RBF at various perturbation levels.
Perturbation	RBF	Skewed RBF
RMSE	nRMSE	RMSE	nRMSE
0.01	0.0086	0.0149	0.0061	0.0116
0.05	0.0252	0.0442	0.0024	0.0043
0.10	0.1484	0.2534	0.0344	0.0651
0.50	0.0170	0.0297	0.0167	0.0282
1.00	0.2830	0.4838	0.0231	0.0365
4.4 Self-tuning PDE Solver
Our framework can also be applied to solve partial differential equations (PDEs). If using fixed radial basis functions (RBFs) without optimization, CNF simplifies to the Kansas method [18], a type of spectral collocation method where partial derivatives are enforced via hard constraints. Compared to the classical mesh-based numerical PDE solvers such as the finite difference method (FDM) and finite element method (FEM), Kansas has the advantage of offering an analytical and closed-form solution function across the entire domain, and solving a PDE on an irregular grid without requiring interpolation or meshing, i.e. grouping the points into triangles or quadrilaterals. Meshing on an irregular grid proves to be extremely difficult and severely harms the accuracy of these methods [9]. However, one disadvantage of the Kansas method is the difficulty in choosing the appropriate RBF kernel [5]. When using a Gaussian kernel, a large bandwidth may lead to an ill-conditioned system, while a small bandwidth may break the global smoothness of the solution function. This is especially challenging when the number or dimensionality of collocation points is large, or when the points are closely or irregularly spaced, making it difficult to manually fine-tune the hyperparameters of the RBF. Our method addresses this limitation by supporting the automatic fine-tuning of hyperparameters, optimizing both the conditioning and global smoothness of the solution:
$$
\underset {\theta} {\arg \min } \omega_ {1} \operatorname {c o n d} \left(\mathbf {A} _ {f _ {\theta}}\right) + \omega_ {2} \underbrace {\int | \nabla f _ {\theta} (\boldsymbol {x}) | d \boldsymbol {x}} _ {\text {t o t a l v a r i a t i o n}} \quad \text {s . t .} \quad \mathcal {F} _ {j} [ f ] (\boldsymbol {x}) = g _ {j} (\boldsymbol {x}) \forall j, \tag {14}
$$
where  $\operatorname{cond}(\mathbf{A}_f)$  is the condition number of the linear system and  $\mathcal{F}_j$  corresponds to the PDE constraints along with the initial and boundary conditions. We demonstrate this approach by solving the following 1D advection equation [36] on an irregular grid:
$$
\frac {\partial u (x , t)}{\partial t} + \beta (x) \frac {\partial u (x , t)}{\partial x} = 0, \quad x \in (0, 1), t \in (0, 1), \tag {15}
$$
$$
u (x, t = 0) = u _ {0} (x), \qquad x \in (0, 1),
$$
where  $u_{0}(x)$  represents the initial state of the system. We sample 32 points on the boundary  $t = 0$  and  $31 \times 32$  points on the space-time domain  $(0,1) \times (0,1)$ , which amount to a total of 1024 points, as hard constraints. The non-boundary samples, shown as red crosses in Fig. 4 are generated by a uniform grid perturbed by Gaussian noise. In this example, we cannot apply a single unified RBF bandwidth to fit such an irregular grid. Rather, we design a skewed radial basis function:
$$
\kappa_ {i} ^ {\prime} \left(\boldsymbol {x} _ {i}, \boldsymbol {x}\right) = \exp \left(- \frac {1}{2} \left(\boldsymbol {x} - \boldsymbol {x} _ {i}\right) ^ {T} \boldsymbol {\Sigma} _ {i} ^ {- 1} \left(\boldsymbol {x} - \boldsymbol {x} _ {i}\right)\right), \tag {16}
$$
where  $\boldsymbol{x} = (x,t)$  and  $\Sigma_{i} = \mathrm{diag}([a_{i},b_{i}])$  is a per-basis learnable covariance matrix. Our results in Tab. 3 show that using the skewed RBF kernel and optimizing per-basis parameters consistently improves performance on the irregular grid; see Appendix for additional results and evaluations.
The skewed RBF is essentially another variant of the Gaussian kernel (Eq. 7) but without the neural encoder. We recommend using neural networks only when the behavior of the basis functions to be optimized away from the constraint points is highly complex, as seen in the BRDF and shape-reconstruction examples. For problems where the desirable priors are as simple as linear independence and smoothness, such as solving PDEs, our proposed skewed RBF demonstrates sufficient capacity.
Training and inference efficiency CNF is highly efficient in training (minutes) and inference (seconds); see Appendix for detailed reports, learning curves, and additional evaluations.
Theoretical analysis We offer a theoretical basis for CNF's superior performance:
Comparison with general solvers Most numerical methods for constrained optimization in scientific computing, including the popular SQP [26] and the more recent DC3 [12], adopt the following formulation for the constrained optimization problem:
$$
\underset {\theta} {\arg \min } \mathcal {L} (\theta) \quad \text {s . t .} \quad h _ {1} (\theta) = 0, h _ {2} (\theta) = 0, \dots \tag {17}
$$
where  $\theta$  denotes the learnable parameters. For problems where  $\theta$  represents the weights of a deep neural network, these constraints become extremely high-dimensional and nonlinear, especially as the number of constraints increases. In this formulation, a constraint is only linear when  $h(\theta)$  is a linear function of  $\theta$ . Therefore, linearity no longer holds as  $h(\theta)$  involves constraints of a neural function. In contrast, our formulation (Eq. 1, 3, and 5) only requires the operator  $\mathcal{F}$  to be linear, while the neural function  $f$  can still be highly nonlinear with respect to  $\theta$ . Hence, CNF significantly reduces the problem's complexity, enabling the explicit determination and promotion of a solution's existence.
Comparison with soft constraint approaches While there have been attempts to model constraints by overfitting a neural function trained with regression [34, 37], CNF offers several clear advantages over such soft approaches. Most notably, CNF satisfies hard constraints without the need for training, whereas soft approaches may require extensive training to approximate constraints. Furthermore, despite extensive training, soft approaches may still fail to satisfy hard constraints due to inherent limitations in their learning capacity. In contrast, CNF provides a robust guarantee of hard constraint satisfaction within machine precision error, provided the condition number is small. Another drawback of employing soft approaches becomes evident when effectively imposing priors. The incorporation of priors often involves introducing additional terms to the loss. Consequently, a tradeoff arises between the constraints and the prior, which is controlled by a hyperparameter. In contrast, CNF offers a clean solution without such a tradeoff. With CNF, the inclusion of priors does not compromise hard constraints, thereby maintaining a harmonious balance among various aspects of the model.
Generalization of NKF and PDE-CL CNF generalizes prior works NKF [41] and PDE-CL [24]. NKF employed a dot-product kernel basis for 3D reconstruction, while PDE-CL used a constraint basis to solve PDEs. Yet, these bases exhibit suboptimal performance in terms of linear independence and learning capacity. Their use of dense matrices also renders them impractical for large-scale problems. Moreover, the dot-product kernel used by NKF cannot address strict higher-order constraints; see Appendix. We propose several novel variants of basis functions to enhance linear independence, learning capacity, and matrix sparsity, with strategies for analyzing and facilitating convergence. Our methodology unifies NKF and PDE-CL, extending their formulations to a broader theoretical context.
Extension to nonlinear operator constraints Although we restrict the operator to be linear in this work, CNF can be extended to accommodate nonlinear operators by solving a nonlinear version of Eq. 3, provided that the solver is differentiable. This could be accomplished through the use of implicit layers [13], which we leave as future work.
6 Summary
We introduce Constrained Neural Fields (CNF), a method that imposes linear operator constraints on deep neural networks, and demonstrate its effectiveness through real-world examples. As constrained optimization is a fundamental problem across various domains, we anticipate that CNF will unlock a host of intriguing applications in fields such as physics, engineering, finance, and visual computing.
This work was supported by the UKRI Future Leaders Fellowship [G104084] and the Engineering and Physical Sciences Research Council [EP/S023917/1].
[42] Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, and Or Litany. Neural fields as learnable kernels for 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18500-18510, 2022.
[43] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. Nex: Real-time view synthesis with neural basis expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8534-8543, 2021.
In this appendix, we provide additional evaluations of various designs of neural basis functions (Sec. A), along with additional details and results regarding their applications in material appearance learning (Sec. B), interpolatory shape reconstruction (Sec. C), and the self-tuning PDE solver (Sec. D), which also includes a transfer learning example (Sec. D.3).
A Neural Basis Functions
A.1 Issues with the Neural Kernel Basis Function
We illustrate how, when using the neural kernel basis function (Eq. 7), enforcing multiple linear operator constraints at the same anchor point would result in repeated basis functions  $\Psi_{i} \equiv \Psi_{j} \equiv \kappa (\phi(\boldsymbol{x}_{\mathrm{anchor}}), \phi(\boldsymbol{x}))$ , consequently leading to an ill-conditioned linear system. Recall Eq. 6:
$$
\left[ \begin{array}{c c c} \mathcal {F} _ {1} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {1} ^ {1}\right) & \mathcal {F} _ {1} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {1} ^ {1}\right) & \dots \\ \mathcal {F} _ {1} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {2} ^ {1}\right) & \mathcal {F} _ {1} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {2} ^ {1}\right) & \dots \\ \vdots & \vdots & \ddots \\ \mathcal {F} _ {2} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {1} ^ {2}\right) & \mathcal {F} _ {2} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {1} ^ {2}\right) & \dots \\ \mathcal {F} _ {2} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {2} ^ {2}\right) & \mathcal {F} _ {2} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {2} ^ {2}\right) & \dots \\ \vdots & \vdots & \ddots \\ \end{array} \right] \left[ \begin{array}{l} \beta_ {1} ^ {1} \\ \beta_ {2} ^ {1} \\ \vdots \\ \beta_ {1} ^ {2} \\ \beta_ {2} ^ {2} \\ \vdots \\ \end{array} \right] = \left[ \begin{array}{l} g _ {1} (\boldsymbol {x} _ {1} ^ {1}) \\ g _ {1} (\boldsymbol {x} _ {2} ^ {1}) \\ \vdots \\ g _ {2} (\boldsymbol {x} _ {1} ^ {2}) \\ g _ {2} (\boldsymbol {x} _ {2} ^ {2}) \\ \vdots \\ \end{array} \right], \tag {18}
$$
where  $\pmb{x}_n^j$  indicates the  $n$ -th point in  $\mathcal{S}_j$  such that  $\mathcal{F}_j[f](\pmb{x}) = g_j(\pmb{x}), \forall \pmb{x} \in \mathcal{S}_j$ . When applying the neural kernel basis function  $\Psi_i(\pmb{x}) = \kappa(\phi(\pmb{x}_i), \phi(\pmb{x})) = \kappa_N(\pmb{x}_i, \pmb{x})$ , the above matrix equation becomes:
Without loss of generality, assuming  $f: \mathbb{R} \mapsto \mathbb{R}$ ;  $\mathcal{F}_1$  is identity map;  $\mathcal{F}_2$  is differentiation; and  $S_1 = S_2 = \{x_0\}$  (same anchor point for both constraints), the matrix equation reduces to:
$$
\underbrace {\left[ \begin{array}{l l} \kappa_ {N} \left(x _ {0} , x _ {0}\right) & \kappa_ {N} \left(x _ {0} , x _ {0}\right) \\ \kappa_ {N} ^ {\prime} \left(x _ {0} , x _ {0}\right) & \kappa_ {N} ^ {\prime} \left(x _ {0} , x _ {0}\right) \end{array} \right]} _ {\text {s i n g u l a r}} \left[ \begin{array}{l} \beta_ {1} \\ \beta_ {2} \end{array} \right] = \left[ \begin{array}{l} g _ {1} \left(x _ {0}\right) \\ g _ {2} \left(x _ {0}\right) \end{array} \right]. \tag {20}
$$
Therefore, the system is ill-conditioned as the matrix has repeated columns. NKF [41] uses this basis function with a dot-product kernel and thus cannot enforce strict constraints simultaneously on both the points and their normals in shape reconstruction, which we further evaluate in Sec. C.1.1. We also discovered significant disadvantages of using a dot-product kernel as opposed to a Gaussian kernel. We elaborate on these findings in Sec. A.2 and Sec. B.2.
A.2 Basis Function Ablation