SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fangcheng Zhong
Kyle Fogarty
Param Hanji
Tianhao Wu
Alejandro Sztrajman
AndrewSpielberg
Andrea Tagliasacchi
Petra Bosilj
University of Lincoln
Cengiz Oztireli
While deep learning techniques have become extremely popular for solving a broad range of optimization problems, methods to enforce hard constraints during optimization, particularly on deep neural networks, remain underdeveloped. Inspired by the rich literature on meshless interpolation and its extension to spectral collocation methods in scientific computing, we develop a series of approaches for enforcing hard constraints on neural fields, which we refer to as Constrained Neural Fields (CNF). The constraints can be specified as a linear operator applied to the neural field and its derivatives. We also design specific model representations and training strategies for problems where standard models may encounter difficulties, such as conditioning of the system, memory consumption, and capacity of the network when being constrained. Our approaches are demonstrated in a wide range of real-world applications. Additionally, we develop a framework that enables highly efficient model and constraint specification, which can be readily applied to any downstream task where hard constraints need to be explicitly satisfied during optimization. Source code is publicly available at https://zfc946.github.io/CNF.github.io/.
Deep neural networks serve as universal function approximators [10] and have achieved tremendous success across a wide variety of tasks. Gradient-based learning algorithms enable optimizing the parameters of networks to approximate any desired model. However, despite the existence of numerous advanced optimization algorithms, the issue of enforcing strict equality constraints during training has not been sufficiently addressed. Constrained optimization has a broad spectrum of real-world applications. For example, in trajectory optimization ([38][Chapter 10], and, e.g., [4, 29]), the agent poses at the start, end, and potentially intermediate steps are to be explicitly enforced. In signal representation, it is desirable for the continuous model to interpolate the discrete samples [22]. In physics, the models of interest must comply with fundamental physical principles [14, 27].
Applying traditional solvers for general constrained optimization, such as SQP [26], to neural networks can be nontrivial. Since traditional methods express constraints as a function of learnable parameters, this formulation becomes extremely high-dimensional, nonlinear, and non-convex in the context of neural networks. As the number of constraints increases, the computational complexity
grows substantially. Other solutions for deep neural networks may encounter issues related to system conditioning, memory consumption, and network capacity when subject to constraints [20].
In this work, we consider the following constrained optimization problem:
$$
\underset {\theta} {\arg \min } \mathcal {L} \left(f _ {\theta}; \theta\right) \quad \text {s . t .} \quad \mathcal {F} \left[ f _ {\theta} \right] (\boldsymbol {x}) = g (\boldsymbol {x}) \forall \boldsymbol {x} \in \mathcal {S}, \tag {1}
$$
where  $f_{\theta}$  is a parametric model with learnable parameters  $\theta$ ; and  $\mathcal{F}$  is a linear operator<sup>1</sup> that transforms  $f$  into another function  $\mathcal{F}[f]$  which has the same input and output dimensions as  $f$ . Common examples of valid operators  $\mathcal{F}$  include the identity map, integration, partial differentiation of arbitrary order, and many composite operators that involve differentiation, such as Laplacian, divergence, and advection operators.  $\mathcal{F}$  can represent a broad class of constraints on the function  $f$  in real-world applications. Enforcement of multiple constraints, expressed as different operators on  $f$ , is also possible. In practice, one typically does not have access to (or require) a closed-form expression of  $g$ , but rather the evaluation of  $g$  on a set  $S$  of discrete samples of  $x$ . In this work, we focus on equality constraints, as inequality constraints can be more easily satisfied through heavy regularization or a judicious choice of the activation function.
Our solution draws inspiration from meshless interpolation [6] and its extension to spectral collocation methods [7] in the scientific computing literature. These methods model a parametric function as a linear sum of basis functions. By computing the weights of the bases, hard constraints on the local behavior of the function at the collocation points, where the constraints need to be satisfied, are enforced accordingly. However, the selection of the basis functions also determines the inductive bias of the model, i.e., the behavior of the function outside the collocation points. Our perspective is to formulate the problem of constrained optimization as a collocation problem with learnable inductive bias. Instead of using a set of fixed basis functions, we allow each basis function to have additional learnable parameters. While the weights of the basis functions are used to enforce constraints, other learnable parameters are optimized via a training loss to match the optimal inductive bias.
With this formulation of the problem, we demonstrate a series of approaches for enforcing hard constraints in the form of Eq. 1 onto a neural field, i.e., a parametric function modeled as a neural network that takes continuous coordinates as input. In theory, we can formulate basis functions from existing neural fields of any architecture and enforce constraints. Nevertheless, certain network architectures may encounter difficulties as the number of collocation points or the order of differentiation increases. To tackle these challenges, we propose specific network designs and training strategies, and demonstrate their effectiveness in several real-world applications spanning physics, signal representation, and geometric processing. Furthermore, we establish a theoretical foundation that elucidates the superiority of our approach over existing solutions, and extend prior works with shared underlying designs to a broader context. Finally, we develop a PyTorch framework for the highly efficient adoption of Constrained Neural Fields (CNF), which can be applied to any downstream task requiring explicit satisfaction of hard constraints during optimization.
In summary, our work makes the following key contributions:
- We introduce a novel approach for enforcing linear operator constraints on neural fields;
- We propose a series of model representations and training strategies to address various challenges that may arise within the problem context, while offering a theoretical basis for CNF's superior performance and its generalization of prior works;
- Our framework is effective across a wide range of real-world problems; in particular, CNF:
- achieves state-of-the-art performance in learning material appearance representations.
- is the first that can enforce an exact normal constraint on a neural implicit field.
- improves the performance of the Kansa method [18], a meshless partial differential equation (PDE) solver, when applied to irregular input grids.
Applying constraints to a deep learning model has been a long-standing problem. Numerous studies have aimed to enforce specific properties on neural networks. For instance, sigmoid and softmax
<sup>1</sup>While our approach can be theoretically extended to nonlinear operators, we focus on linear ones due to the efficiency of linear solvers and GPU vectorization.
activation functions are used to ensure that the network output represents a valid probability mass function. Convolutional neural networks (CNNs) were introduced to address translation invariance. Pointnet [30] was proposed to achieve permutation invariance. POLICE [3] was introduced to enforce affine properties on the network. However, these works were not specifically designed to handle strict and general-form equality constraints. There are also studies that train a multilayer perceptron (MLP) [34, 37] or neural basis functions [8, 43] with a regression loss to approximate the equality constraints through overfitting, which we refer to as soft constraint approaches. However, in those works, strict adherence to hard constraints is rarely guaranteed. DC3 [12] is the first work that can enforce general equality constraints onto deep learning models by selecting a subset of network parameters for constraint enforcement and utilizing the remaining parameters for optimization. However, this approach cannot offer a theory or guidance regarding the existence of a solution when selecting a random subset of the network. In contrast, our method rigorously analyzes and guarantees the existence of a solution. Linearly constrained neural networks [17] can also explicitly satisfy linear operator constraints, but their method only applies to the degenerate case where  $g(x)$  (Eq. 1) is zero everywhere. Neural kernel fields (NKF) [41] employs kernel basis functions similar to ours to solve constrained optimization problems in geometry, but their method cannot accommodate strict higher-order constraints. Their use of a dense matrix also cannot scale up to accommodate a large number of constraints, due to limitations in processor memory. A concurrent work, PDE-CL [24], utilizes a constrained layer to solve partial differential equations as a constrained optimization problem. We demonstrate that a constrained layer may not be the ideal choice for designing basis functions and propose alternative solutions. In fact, our approach can be seen as a generalization of NKF and PDE-CL. While seemingly unrelated, both works approach constrained optimization problems by leveraging the weights of basis functions to enforce constraints. They differ only in the design of the basis and the optimization function. Finally, there has been substantial development in implicit layers [13] that can enforce equality constraints between the layer's input and output while efficiently tracking gradients. However, this approach is not specifically designed for constraints over the entire network, although it can be potentially incorporated into our method to facilitate computation.
At the core of our approach is the representation of a neural field  $f(\cdot): \mathbb{R}^M \mapsto \mathbb{R}^N$  as a linear sum of basis functions, as with the collocation methods:
$$
f (\boldsymbol {x}) = \sum_ {i} \beta_ {i} \odot \Psi_ {i} (\boldsymbol {x}), \tag {2}
$$
where each  $\Psi_i(\cdot):\mathbb{R}^M\mapsto \mathbb{R}^N$  represents a basis function that can be expressed as any parametric function with learnable parameters, such as neural networks;  $\beta_{i}\in \mathbb{R}^{N}$  is the weight of the  $i$ -th basis; and  $\odot$  indicates the Hadamard product.
3.1 Constrained Optimization
Consider  $\mathcal{S} \coloneqq \{\pmb{x}_i\}_{i=1}^I$ , a set of  $I$  constraint points such that  $\mathcal{F}[f](\pmb{x}_i) = g(\pmb{x}_i)$ ,  $\forall i$  where the ground truth value of  $g(\pmb{x}_i)$  is accessible. Using the neural basis representation (2), and assuming that  $\mathcal{F}$  is linear and  $\mathcal{F}[\Psi_i]$  is well-defined for all  $i$ , the following needs to hold for the constraints:
$$
\mathcal {F} [ f ] (\boldsymbol {x}) = \mathcal {F} \left[ \sum_ {i} \beta_ {i} \odot \Psi_ {i} \right] (\mathbf {x}) = \sum_ {i} \beta_ {i} \odot \mathcal {F} \left[ \Psi_ {i} \right] (\boldsymbol {x}) = g (\boldsymbol {x}), \forall \boldsymbol {x} \in \mathcal {S}. \tag {3}
$$
Eq. 3 is a collocation equation in the context of spectral collocation methods for solving differential and integral equations [9, 1]. These constraints can be explicitly satisfied by defining  $I$  basis functions and solving Eq. 3 for their weights. For a linear  $\mathcal{F}$ , we can expand Eq. 3 into batched matrix form:
$$
\underbrace {\left[ \begin{array}{c c c c} \mathcal {F} [ \Psi_ {1} ] (\boldsymbol {x} _ {1}) & \mathcal {F} [ \Psi_ {2} ] (\boldsymbol {x} _ {1}) & \cdots & \mathcal {F} [ \Psi_ {I} ] (\boldsymbol {x} _ {1}) \\ \mathcal {F} [ \Psi_ {1} ] (\boldsymbol {x} _ {2}) & \mathcal {F} [ \Psi_ {2} ] (\boldsymbol {x} _ {2}) & \cdots & \mathcal {F} [ \Psi_ {I} ] (\boldsymbol {x} _ {2}) \\ \vdots & \vdots & \ddots & \vdots \\ \mathcal {F} [ \Psi_ {1} ] (\boldsymbol {x} _ {I}) & \mathcal {F} [ \Psi_ {2} ] (\boldsymbol {x} _ {I}) & \cdots & \mathcal {F} [ \Psi_ {I} ] (\boldsymbol {x} _ {I}) \end{array} \right]} _ {\mathbf {A} _ {f}} \left[ \begin{array}{c} \beta_ {1} \\ \beta_ {2} \\ \vdots \\ \beta_ {I} \end{array} \right] = \left[ \begin{array}{c} g (\boldsymbol {x} _ {1}) \\ g (\boldsymbol {x} _ {2}) \\ \vdots \\ g (\boldsymbol {x} _ {I}) \end{array} \right]. \tag {4}
$$
There are in total  $N$  such matrix equations to solve, each corresponding to an output dimension of  $f$  if  $N > 1^2$ . We denote the matrix as  $\mathbf{A}_f$  for reference. As long as we apply a differentiable linear
2In Einstein summation notation, the batched matrix equation in Eq. 4 can be expressed as  $IIN, IN \rightarrow IN$ .
1: repeat
2: Compute  $\beta$  as the solution of Eq. 4
3: Compute gradient  $\frac{\partial\mathcal{L}}{\partial\theta} = \frac{\partial\mathcal{L}}{\partial f}\frac{\partial f}{\partial\theta}$
4: Update  $\theta$
5: until converged
Algorithm 2 Inference
Input:  $x$
Output:  $f_{\theta}(\pmb{x})$
1: if  $\beta$  is None then
2: Compute  $\beta$  from Eq. 4 w/o gradients
3: end if
4:  $f_{\theta}(\pmb {x})\gets \sum_{i}\beta_{i}\odot \Psi_{i}(\pmb {x})$
solver that provides valid gradients to compute the weights, the neural basis representation (Eq. 2) can be integrated with any optimization loss  $\mathcal{L}(f_{\theta};\theta)$  to approximate the desired behavior, while ensuring that the hard constraints are strictly enforced. Algorithm 1 describes the training procedure in detail, where Step 2 solves for the weights  $\beta$  in every iteration, ensuring consistent satisfaction of hard constraints throughout the entire training process, while Steps 3 and 4 update the training parameters  $\theta$  to optimize for the training loss  $\mathcal{L}$  under the constraints. We compute  $\beta$  using an LU decomposition with partial pivoting and row interchanges, which is fully differentiable when  $\mathbf{A}_f$  is full rank. The computation of  $\beta$  constructs a computational graph that tracks  $\frac{\partial\beta}{\partial\theta}$ , thereby ensuring a valid  $\frac{\partial f}{\partial\theta}$ . Consequently, any loss function with a valid gradient  $\frac{\partial\mathcal{L}}{\partial f}$  can be smoothly integrated into this training procedure. The inference procedure is detailed in Algorithm 2. Here,  $\beta$  only needs to be pre-computed once since it is independent of the evaluation point  $\pmb{x}$ . As a result, performing inference with CNF is highly efficient and boils down to computing a linear combination of  $I$  basis functions without solving any matrix equations, a task that can be vectorized for efficiency.
This approach can also be extended to multiple linear operator constraints, indexed by  $j$ :
$$
\mathcal {F} _ {j} [ f ] (\boldsymbol {x}) = g _ {j} (\boldsymbol {x}), \forall \boldsymbol {x} \in \mathcal {S} _ {j}, \tag {5}
$$
where  $S_{j}$  indicates the set of points  $\pmb{x}$  where the values of  $\mathcal{F}_j[f](\pmb{x})$  are constrained. Note that  $S_{j}$  corresponding to different operator constraints can have repeated  $\pmb{x}$  elements. For example, to constrain  $\nabla f(\pmb{x})$  where  $\pmb{x} \in S \subseteq \mathbb{R}^{M}$ , a partial differentiation operator needs to be specified for each of the  $M$  partial derivatives at the same point  $\pmb{x}$ . These constraints can be explicitly satisfied by defining  $\sum_{j} |S_{j}|$  basis functions and computing their weights by solving the batched matrix equation:
$$
\left[ \begin{array}{c c c} \mathcal {F} _ {1} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {1} ^ {1}\right) & \mathcal {F} _ {1} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {1} ^ {1}\right) & \dots \\ \mathcal {F} _ {1} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {2} ^ {1}\right) & \mathcal {F} _ {1} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {2} ^ {1}\right) & \dots \\ \vdots & \vdots & \ddots \\ \mathcal {F} _ {2} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {1} ^ {2}\right) & \mathcal {F} _ {2} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {1} ^ {2}\right) & \dots \\ \mathcal {F} _ {2} \left[ \Psi_ {1} \right] \left(\boldsymbol {x} _ {2} ^ {2}\right) & \mathcal {F} _ {2} \left[ \Psi_ {2} \right] \left(\boldsymbol {x} _ {2} ^ {2}\right) & \dots \\ \vdots & \vdots & \ddots \\ \end{array} \right] \left[ \begin{array}{l} \beta_ {1} ^ {1} \\ \beta_ {2} ^ {1} \\ \vdots \\ \beta_ {1} ^ {2} \\ \beta_ {2} ^ {2} \\ \vdots \\ \end{array} \right] = \left[ \begin{array}{l} g _ {1} (\boldsymbol {x} _ {1} ^ {1}) \\ g _ {1} (\boldsymbol {x} _ {2} ^ {1}) \\ \vdots \\ g _ {2} (\boldsymbol {x} _ {1} ^ {2}) \\ g _ {2} (\boldsymbol {x} _ {2} ^ {2}) \\ \vdots \\ \end{array} \right], \tag {6}
$$
where  $\pmb{x}_n^j$  indicates the  $n$ -th point in  $S_j$ . The training and inference procedures are the same as previously described, except that  $\beta$  is now solved using Eq. 6 instead.
3.2 Neural Basis Functions
To construct a set of basis functions from an existing neural field, we consider a few approaches.
Independent basis The most straightforward method is to specify a separate neural field for each basis. This method is the most expressive but can only be used with a small number of constraints. This is because the number of networks and learnable parameters increases linearly with the number of constraints. A bigger concern is the quadratic growth in the size of the matrix  $\mathbf{A}_f$ . As a result, even high-end processors are guaranteed to run out of memory working with a large number of constraints.
Constraint basis We can directly use the last hidden layer of a multi-layer perceptron (MLP) as the basis and compute the weight of each node to enforce the constraints. This strategy has been referred to as a constraint layer in concurrent work [24]. The size of the network, excluding the hidden layer, does not grow as the number of constraints increases. However, this approach may lead to linearly-dependent basis functions, resulting in the matrix  $\mathbf{A}_f$  being singular or having a large condition number. This can lead to a significant error bound in the satisfaction of hard constraints.
Hypernetwork basis The third approach is to generate the parameters of each neural basis function using a hypernetwork [16]. The input to the hypernetwork is a one-hot encoding of length equal to the number of bases. The hypernetwork ensures that the network size does not increase with the number of basis functions. Nonetheless, the quadratic growth in the size of  $\mathbf{A}_f$  remains an issue for both the constraint and the hypernetwork bases.
In several empirical studies, we found that the two aforementioned approaches hinder the network's capacities when subjected to hard constraints. They also encountered difficulties reducing the condition number; see Appendix.
Kernel basis As a result, we propose several variants of a neural kernel function as a special design of basis functions:
$$
\Psi_ {i} (\boldsymbol {x}) = \kappa (\phi (\boldsymbol {x} _ {i}), \phi (\boldsymbol {x})), \tag {7}
$$
where  $\phi (\cdot)$  is a neural encoder that maps  $\pmb{x}$  into a high-dimensional feature space;  $\kappa (\cdot ,\cdot)$  is a kernel function that measures the similarity between two points in the feature space;  $\pmb {x}_i$  is the  $i$ -th constraint point, which we refer to as an anchor point. With this design, the higher-dimensional feature space bolsters the learning capacity of the basis function. The size of parameters in the network also does not grow with the number of constraints. If  $\kappa (\cdot ,\cdot)$  is a dot-product kernel, Eq. 7 becomes the same basis function used in NKF [41]. However, we empirically found that a Gaussian kernel better promotes the linear independence of basis functions due to its local compact support, while still preserving sufficient learning capacity; see Appendix.
Hypernetwork kernel basis One drawback of the kernel basis function is that only one constraint can be applied to each anchor point. For example, one cannot enforce constraints on both the value of  $f(\pmb{x})$  and its gradient  $\nabla f(\pmb{x})$  at the same anchor point, as it would result in repeated basis functions  $\Psi_{i} \equiv \Psi_{j} \equiv \kappa (\phi (\pmb{x}_{\mathrm{anchor}}), \phi (\pmb{x}))$  and thus an ill-conditioned system; see Appendix for math details. For multiple constraints at repeated anchor points, we propose a hypernet kernel function:
$$
\Psi_ {i} (\boldsymbol {x}) = \kappa \left(\phi_ {i} \left(\boldsymbol {x} _ {i}\right), \phi_ {i} (\boldsymbol {x})\right), \tag {8}
$$
where the weights of  $\phi_{i}$  are controlled by a hypernetwork conditioned on the anchor point  $\pmb{x}_i$ . This hypernet construction of the encoder ensures that the network would not have repeated basis functions and the number of network parameters does not grow with the number of basis functions.
Hybrid kernel basis Central to our approach is the construction and inversion of a linear system, the size of which grows quadratically with the number of constraints. To mitigate severe memory issues in large-scale constraints, we design a basis that promotes sparsity in matrix  $\mathbf{A}_f$  for large-scale problems. While using a Gaussian kernel can promote sparsity, it is difficult to select a proper bandwidth to explicitly control the number of nonzero elements since the kernel operates in the feature space rather than the original domain. To overcome this, we propose a hybrid kernel function:
$$
\Psi_ {i} (\boldsymbol {x}) = \kappa \left(\phi_ {i} \left(\boldsymbol {x} _ {i}\right), \phi_ {i} (\boldsymbol {x})\right) \kappa_ {G} \left(\boldsymbol {x} _ {i}, \boldsymbol {x}\right), \tag {9}
$$
where the first part  $\kappa (\phi_i(\mathbf{x}_i),\phi_i(\mathbf{x}))$  can be either a Gaussian kernel basis or a hypernet kernel basis, depending on the task, and  $\kappa_{G}$  is some compactly supported kernel function, allowing for explicit adjustment of the zero behavior of  $\Psi_{i}$ . A candidate  $\kappa_{G}$  can be a truncated Gaussian kernel such as:
$$
\kappa_ {G} \left(\mathbf {x} _ {i}, \mathbf {x}\right) = \left\{ \begin{array}{l l} \exp \left(- \frac {\left\| \mathbf {x} _ {i} - \mathbf {x} \right\| ^ {2}}{2 \sigma^ {2}}\right) & \text {i f} \| \mathbf {x} _ {i} - \mathbf {x} \| <   3 \sigma \\ 0 & \text {i f} \| \mathbf {x} _ {i} - \mathbf {x} \| \geq 3 \sigma . \end{array} \right. \tag {10}
$$
Reducing the magnitude of  $\sigma$  yields a matrix  $\mathbf{A}_f$  with more zero entries, breaking the quadratic dependency on the number of constraint points.
Properties of various basis functions are summarized in Tab. 1. We recommend using the regular Gaussian kernel basis for standard-scale problems without multiple constraints at repeated anchor points, and employing hypernet kernel and hybrid kernel for corresponding advanced scenarios. In Sec. 4, we demonstrate the application of each type of basis function in concrete real-world examples.
3.3 Training Strategies
Regularization In many applications, we recommend incorporating an additional loss term to regularize the condition number of the matrix  $\mathbf{A}_f$  during optimization. A low condition number corresponds to a reduced level of matrix singularity and a smaller error bound in the solution vector, which is crucial for satisfying hard constraints.
Table 1: Basis functions summary.
Independent basis	Constraint basis	Hypernetwork basis	Kernel basis (Dot-product)	Kernel basis (Gaussian)	Hypernetwork kernel basis (Gaussian)	Hybrid kernel basis (Gaussian)
Learning capacity	Poor	Poor	Fair	Good	Good	Good
Linear independence	Poor	Fair	Poor	Good	Good	Good
Matrix sparsity	Dense	Dense	Dense	Dense	Sparse	Sparse	Sparse
Params # independent of constraints #	No	Yes	Yes	Yes	Yes	Yes	Yes
Controllable sparsity	No	No	No	No	No	No	Yes
Multiple constraints at repeated anchor points	Yes	Yes	Yes	No	No	Yes	Yes (if using hypernet)