SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tackgeun You<sup>3</sup>
tackgeun.you@postech.ac.kr
Mijeong Kim
mijeong.kim@snu.ac.kr
Jungtaek  $\mathbf{K}\mathbf{m}^{4}$
Bohyung Han1,2
bhhan@snu.ac.kr
$^{1}$ ECE &  $^{2}$ IPAI, Seoul National University, South Korea
$^{3}$ CSE, POSTECH, South Korea
<sup>4</sup>University of Pittsburgh, USA
We propose a novel approach to learning the generative neural fields represented by linear combinations of implicit basis networks. Our algorithm learns basis networks in the form of implicit neural representations and their coefficients in a latent space by either conducting meta-learning or adopting auto-decoding paradigms. The proposed method easily enlarges the capacity of generative neural fields by increasing the number of basis networks while maintaining the size of a network for inference to be small through their weighted model averaging. Consequently, sampling instances using the model is efficient in terms of latency and memory footprint. Moreover, we customize denoising diffusion probabilistic model for a target task to sample latent mixture coefficients, which allows our final model to generate unseen data effectively. Experiments show that our approach achieves competitive generation performance on diverse benchmarks for images, voxel data, and NeRF scenes without sophisticated designs for specific modalities and domains.
Implicit neural representation (INR) is a powerful and versatile tool for modeling complex and diverse data signals in various modalities and domains, including audio [12], images [35], videos [40], 3D objects [28, 6], and natural scenes. INR expresses a data instance as a function mapping from a continuous coordinate space to a signal magnitude space rather than using a conventional representation on a discrete structured space. In particular, a representation with a continuous coordinate allows us to query arbitrary points and retrieve their values, which is desirable for many applications that only have accessibility to a subset of the target data in a limited resolution. INRs replace the representations based on high-dimensional regular grids, such as videos [9] and 3D scenes [25], with multi-layer perceptrons (MLPs) with a relatively small number of parameters, which effectively memorize scenes.
Generative neural fields aim to learn distributions of functions that represent data instances as a form of neural field. They typically rely on INRs to sample their instances in various data modalities and domains. The crux of generative neural fields lies in how to effectively identify and model shared and instance-specific information. To this end, feature-wise linear modulation (FiLM) [29] and hyper-network (HyperNet) [16] are common methods for modulating representations. FiLM
Figure 1: Overview of the generation procedure using the proposed generative neural field based on mixtures of neural implicit function (mNIF). Our model is applicable to various types of data such as images, voxels, and radiance fields. To generate an instance, we first sample a context vector  $\phi_j$  from a prior distribution  $(P_{\mathrm{latent}})$  estimated by a denoising diffusion probabilistic model. We then perform a weighted model averaging on implicit bases  $\{\mathbf{W}_m^{(i)}\mathbf{b}_m^{(i)}\}$  using mixture coefficients  $(\alpha_m^{(i)})$  derived from the context vector.
adjusts hidden features using an affine transform, which consists of element-wise multiplication and bias addition. Due to the ease of optimization, the effectiveness of FiLM has already been validated in several tasks including 3D-aware generative modeling [6, 7]. However, the operation of FiLM is too restrictive while introducing additional parameters for modulation. On the other hand, HyperNet is more flexible because it directly predicts network parameters. Unfortunately, the direct prediction of model parameters is prone to unstable training and limited coverage of the data distribution generated by the predicted network. Therefore, to reduce solution spaces, HyperNet often employs dimensionality reduction techniques such as low-rank decompositions of weight matrices [35, 12] and combinations of multiple network blocks [16]. However, all these works give limited consideration to the inference efficiency.
Inspired by the flexibility of HyperNet and the stability of FiLM while considering efficiency for inference, we propose a mixture of neural implicit functions (mNIF) to represent generative neural fields. Our approach employs mNIFs to construct generative networks via model averaging of INRs. The mixture components in an mNIF serve as shared implicit basis networks while their mixands define relative weights of the bases to construct a single instance. Our modulation step corresponds to computing a weighted average of the neural implicit functions. The mixture coefficients are optimized by either meta-learning or auto-decoding procedures with signal reconstruction loss. Such a design is effective for maximizing the expressibility of INRs while maintaining the compactness of inference networks. The proposed approach is versatile for diverse modalities and domains and is easy to implement.
Figure 1 demonstrates the overview of the proposed approach, and we summarize the contributions of our work as follows:
- We introduce a generative neural field based on mNIF, a modulation technique by a linear combination of NIFs. The proposed approach effectively extracts instance-specific information through conducting meta-learning or applying auto-decoding procedures.
- Our modulation allows us to easily extend the model capacity by increasing the number of basis networks without affecting the structure and size of the final model for inference. Such a property greatly improves the efficiency of the sampling network in terms of inference time and memory requirements.
- Our model achieves competitive performance on several data generation benchmarks without sophisticated domain-specific designs of algorithm and architecture configuration.
The rest of this paper is organized as follows. We first discuss related works about generative neural fields in Section 2. The main idea of the proposed approach and the training and inference procedures are discussed in Sections 3 and 4, respectively. Section 5 presents empirical quantitative and qualitative results with discussions, and Section 6 concludes this paper.
Implicit neural representations (INR) are often optimized for representing a single instance in a dataset, and have been extended in several ways to changing activation functions [33, 30], utilizing input coordinate embeddings [36, 26] or taking advantage of a mixture of experts [37, 23, 24]. Since the INR framework is difficult to generalize a whole dataset, generative neural fields have been proposed to learn a distribution of functions based on INRs, each of which corresponds to an example in the dataset.
Generative manifold learning (GEM) [12] and generative adversarial stochastic process (GASP) [14] are early approaches to realizing generative neural fields. GEM implements the generative neural field using the concept of hyper-network (HyperNet) [16], which predicts a neural network representing an instance. Training GEM is given by the auto-decoding paradigm with manifold learning. However, GEM involves extra computational cost because the manifold constraint requires accessing training examples. GASP also employs a HyperNet for sampling instances, and its training utilizes a discriminator taking an input as a set of coordinate and feature pairs. Since the output of the discriminator should be permutation invariant, it follows the design of the network for the classification of point cloud data. However, training GASP is unstable and the discriminator design is sometimes tricky, especially in NeRF. Functa [13] proposes the generative neural fields with SIREN [33] modulated by FiLM [29] and introduces a training strategy based on meta-learning. However, meta-learning is computationally expensive due to the Hessian computation. On the other hand, diffusion probabilistic fields (DPF) [41] proposes a single-stage diffusion process with the explicit field parametrization [19]. However, its high computational cost hampers the applicability to complex neural fields such as NeRF. HyperDiffusion [15], which is concurrent to our work, presents a framework directly sampling the entire INR weight from the learned diffusion process instead of exploiting latent vectors for modulating INR weights. However, this work demonstrates instance generation capability only on voxel domains, not on images.
Model averaging [38] is often used for enhancing performance in discriminative tasks without increasing an inference cost. Our approach applies model averaging to generative models and verifies its effectiveness in generative neural fields based on INRs. Note that we are interested in efficient prediction by learning a compact representation through a weighted mixture; our method jointly trains basis models and their mixture coefficients.
3 Generative Neural Fields with a Mixture of Neural Implicit Functions
This section describes how to define generative neural fields. The formulation with a mixture of neural implicit functions (mNIF) enforces implicit basis networks to represent shared information across examples and allows latent mixture coefficients to encode instance-specific information through a linear combination of basis networks. We show that generative neural field via mNIF is effectively optimized to predict the latent vector for the construction of models generating high-quality samples.
3.1 Implicit Neural Representations
Implicit neural representation (INR) expresses a data instance using a function from an input query,  $\mathbf{x} \in \mathbb{N}^d$ , to its corresponding target value,  $\mathbf{y} \in \mathbb{R}^k$ . Since each input coordinate is independent of the others, a mapping function parametrized by  $\theta$ ,  $\mathbf{f}_{\theta}(\cdot)$ , is typically represented by a multi-layer perceptron (MLP) with  $L$  fully connected (FC) layers, which is given by
$$
\mathbf {y} = \mathbf {f} _ {\boldsymbol {\theta}} (\mathbf {x}) = \mathbf {f} ^ {(L + 1)} \circ \dots \circ \mathbf {f} ^ {(1)} \circ \mathbf {f} ^ {(0)} (\mathbf {x}), \tag {1}
$$
where  $\mathbf{f}^{(i)}$  for  $\forall i\in \{1,2,\dots ,L\}$  denote FC layers while  $\mathbf{f}^{(0)}$  and  $\mathbf{f}^{(L + 1)}$  indicate the input and output layers, respectively. Each layer  $\mathbf{f}^{(i)}$  performs a linear transform on its input hidden state  $\mathbf{h}^{(i)}$  and then applies a non-linear activation function to yield the output state  $\mathbf{h}^{(i + 1)}$ , expressed as
$$
\mathbf {h} ^ {(i + 1)} = \mathbf {f} ^ {(i)} (\mathbf {h} ^ {(i)}) = \sigma \left(\mathbf {W} ^ {(i)} \mathbf {h} ^ {(i)} + \mathbf {b} ^ {(i)}\right), i \in \{1, 2, \dots , L \}, \tag {2}
$$
where  $\sigma (\cdot)$  is an activation function and  $\mathbf{W}^{(i)}\in \mathbb{R}^{W\times W}$  and  $\mathbf{b}^{(i)}\in \mathbb{R}^{W}$  are learnable parameters. The operations of the input and output layers are defined as  $\mathbf{h}_1 = \mathbf{f}^{(0)}(\mathbf{x})$  and  $\hat{\mathbf{y}} = \mathbf{f}^{(L + 1)}(\mathbf{h}^{(L)})$
respectively. Consequently, a collection of learnable parameters  $\theta$  in all layers is given by
$$
\boldsymbol {\theta} \triangleq \left\{\mathbf {W} ^ {(0)}, \mathbf {b} ^ {(0)}, \mathbf {W} ^ {(1)}, \mathbf {b} ^ {(1)}, \dots , \mathbf {W} ^ {(L + 1)}, \mathbf {b} ^ {(L + 1)} \right\}, \tag {3}
$$
where  $\mathbf{W}^{(0)}\in \mathbb{R}^{W\times d}$ ,  $\mathbf{b}^{(0)}\in \mathbb{R}^W$ ,  $\mathbf{W}^{(L + 1)}\in \mathbb{R}^{k\times W}$ , and  $\mathbf{b}^{(L + 1)}\in \mathbb{R}^k$ . For INR, the mean squared error between a prediction  $\hat{\mathbf{y}}$  and a target  $\mathbf{y}$  is typically adopted as a loss function,  $\mathcal{L}(\cdot ,\cdot)$ , which is given by
$$
\mathcal {L} (\hat {\mathbf {y}}, \mathbf {y}) = \| \hat {\mathbf {y}} - \mathbf {y} \| ^ {2}. \tag {4}
$$
Among the variations of INRs, we employ SIREN [33], which adopts the sinusoidal function as an activation function and introduces a sophisticated weight initialization scheme for MLPs. Supposing that SIREN is defined with an MLP with  $L$  hidden layers, the  $i^{\mathrm{th}}$  layer of SIREN is given by
$$
\mathbf {h} ^ {(i + 1)} = \sin \left(w _ {0} \left(\mathbf {W} ^ {(i)} \mathbf {h} ^ {(i)} + \mathbf {b} ^ {(i)}\right)\right), i \in \{1, \dots , L \}, \tag {5}
$$
where  $w_{0}$  is a scale hyperparameter to control the frequency characteristics of the network. The initialization of SIREN encourages the distribution of hidden sine activations to follow a normal distribution with a standard deviation of 1.
3.2 Mixtures of Neural Implicit Functions
We propose generative neural fields based on mNIF, which is an extension of the standard INR for modulating its model weight. We define a set of NIFs, which is used as basis networks, and construct a generative neural field using a mixture of the NIFs. This is motivated by our observation that a generative neural field is successfully interpolated using multiple basis networks and model averaging works well for model generalization [38].
The operation in each layer of our mixture model is given by
$$
\mathbf {h} ^ {(i + 1)} = \sin \left(w _ {0} \left(\sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {g} _ {m} \left(\mathbf {h} ^ {(i)}\right)\right)\right), \tag {6}
$$
where  $\mathbf{g}_m^{(i)}(\mathbf{h}^{(i)}) = \mathbf{W}_m^{(i)}\mathbf{h}^{(i)} + \mathbf{b}_m^{(i)}$  is the  $i^{\mathrm{th}}$ -layer operation of the  $m^{\mathrm{th}}$  neural implicit function, and  $\alpha_{m}^{(i)}$  is a mixture coefficient of the same layer of the same neural implicit function. Note that modulating the network is achieved by setting the mix and values,  $\{\alpha_m^{(i)}\}$ . Similar to the definition of  $\theta$  described in Eq. (3), the parameter of each mixture is given by
$$
\boldsymbol {\theta} _ {m} \triangleq \left\{\mathbf {W} _ {m} ^ {(0)}, \mathbf {b} _ {m} ^ {(0)}, \dots , \mathbf {W} _ {m} ^ {(L + 1)}, \mathbf {b} _ {m} ^ {(L + 1)} \right\}. \tag {7}
$$
The operation in the resulting INR obtained from model averaging is given by
$$
\sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {g} _ {m} \left(\mathbf {h} ^ {(i)}\right) = \left(\sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {W} _ {m} ^ {(i)}\right) \mathbf {h} ^ {(i)} + \sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {b} _ {m} ^ {(i)} = \overline {{\mathbf {W}}} ^ {(i)} \mathbf {h} ^ {(i)} + \overline {{\mathbf {b}}} ^ {(i)}, \tag {8}
$$
$$
\overline {{\mathbf {W}}} ^ {(i)} \triangleq \sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {W} _ {m} ^ {(i)} \quad \text {a n d} \quad \overline {{\mathbf {b}}} ^ {(i)} \triangleq \sum_ {m = 1} ^ {M} \alpha_ {m} ^ {(i)} \mathbf {b} _ {m} ^ {(i)}. \tag {9}
$$
All the learnable parameters in the mixture of NIFs is defined by  $\{\pmb{\theta}_1, \dots, \pmb{\theta}_M\}$ . The remaining parameters are mixture coefficients, and we have the following two options to define them: (i) sharing mixands for all layers  $\alpha_m^{(i)} = \alpha_m$  and (ii) setting mixands to different values across layers. The first option is too restrictive for the construction of INRs because all layers share mixture coefficients while the second is more flexible but less stable because it involves more parameters and fails to consider the dependency between the coefficients in different layers. Hence, we choose the second option but estimate the mixture coefficients in a latent space, where our method sets the dimensionality of the latent space to a constant and enjoys the high degree-of-freedom of layerwise coefficient setting. To this end, we introduce a projection matrix  $\mathbf{T}$  to determine the mixture coefficients efficiently and effectively, which is given by
$$
\left[ \alpha_ {1} ^ {(0)}, \dots , \alpha_ {M} ^ {(0)}, \alpha_ {1} ^ {(1)}, \dots , \alpha_ {M} ^ {(1)}, \dots , \alpha_ {1} ^ {(L + 1)}, \dots , \alpha_ {M} ^ {(L + 1)} \right] = \mathbf {T} \phi , \tag {10}
$$
where  $\phi \in \mathbb{R}^H$  denotes a latent mixture coefficient vector. This choice compromises the pros and cons of the two methods. Eventually, we need to optimize a set of parameters,  $\pmb{\theta}_{\mathrm{all}} \triangleq \{\pmb{\theta}_1, \dots, \pmb{\theta}_M, \mathbf{T}\}$  for a generative neural field, and a latent vector  $\phi$  to obtain an instance-specific INR.
3.3 Model Efficiency
The proposed modulation technique is based on model averaging and has great advantages in the efficiency during training and inference. First, the computational complexity for each input coordinate is small despite a large model capacity given by potentially many basis networks. Note that, even though the number of neural implicit basis function in a mixture increases, the computational cost per query is invariant by Eq. (8), which allows us to handle more coordinates queries than the models relying on FiLM [29]. Second, the model size of our generative neural fields remains small compared to other methods due to our model averaging strategy, which is effective for saving memory to store the neural fields.
When analyzing the efficiency of generative neural fields, we consider two aspects; one is the model size in the context adaptation stage, in other words, the number of all the learnable parameters, and the other is the model size of the modulated network used for inference. By taking advantage of the modulation via model averaging, our approach reduces memory requirement significantly both in training and inference.
4 Learning Generative Neural Fields
The training procedure of the proposed approach is composed of two stages: context adaptation followed by task-specific generalization. In the context adaptation stage, we optimize all the learnable parameters for implicit basis functions and their coefficients that minimize reconstruction error on examples in each dataset. Then, we further learn to enhance the generalization performance of the task-specific generative model, where a more advanced sampling method is employed to estimate a context vector, i.e., a latent mixture coefficient vector. This two-stage approach enables us to acquire the compact representation for neural fields and saves computational costs during training by decoupling the learning neural fields and the training generative model. We describe the details of the two-stage model below.
4.1 Stage 1: Training for Context Adaptation
The context adaptation stage aims to learn the basis networks and the latent mixture coefficient vector, where the mixture model minimizes the reconstruction loss. This optimization is achieved by one of two approaches: meta-learning or auto-decoding paradigm.
With meta-learning, we train models in a similar way as the fast context adaptation via meta-learning (CAVIA) [42]; after a random initialization of a latent vector  $\phi_j$  for the  $j^{\mathrm{th}}$  instance in a batch, we update  $\phi_j$  in the inner-loop and then adapt the shared parameters  $\theta$  based on the updated  $\phi_j$ . Since this procedure requires estimating the second-order information for the gradient computation of  $\theta$ , the computational cost of this algorithm is high. Algorithm 1 presents this meta-learning procedure.
We observe that random initialization of a latent vector during meta-learning is more stable than the constant initialization originally proposed in CAVIA when training mNIF with a large number of mixture components or the high dimensionality of latent vectors. We intend to diversify INR weights at the start phase of meta-training, ensuring that not all samples are represented with identical INR weights.
Auto-decoding paradigm [12, 28] is a variant of multi-task learning, where each instance is considered a task. Contrary to the meta-learning, we do not reset and initialize the latent vector  $\phi_j$  for each instance but continue to update over multiple instances. The optimization procedure via auto-decoding is presented in Algorithm 2.
4.2 Stage 2: Optimizing for Task-Specific Generalization
The aforementioned context adaptation procedure focuses only on the minimization of reconstruction error for training samples and may not be sufficient for the generation of unseen examples. Hence,
Algorithm 1 Meta-learning with mNIF
1: Randomly initialize the shared parameter  $\theta$  of mNIF.  
2: while training do  
3: Sample a mini-batch  $\mathcal{B} = \{(\mathbf{c}_j,\mathbf{y}_j)\}_{j = 1:|\mathcal{B}|}$   
4: for  $j = 1$  to  $|\mathcal{B}|$  do  
5: Initialize a latent vector  $\phi_j^{(0)}\sim \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})\in \mathbb{R}^H$   
6: for  $n = 0$  to  $N_{inner} - 1$  do  
7: Update the latent vector:  $\phi_j^{(n + 1)}\gets \phi_j^{(n)} - \epsilon_{latent}\nabla_\phi \mathcal{L}(f_\theta ,\phi (\mathbf{c}_j),\mathbf{y}_j)|_{\phi = \phi_j^{(n)}}$   
8: end for  
9: Update the shared parameters:  $\theta \leftarrow \theta -\epsilon_{shared}\nabla_{\theta}\mathcal{L}(f_{\theta ,\phi}(\mathbf{c}_j),\mathbf{y}_j)|_{\phi = \phi_j^{(N_{inner})}}$   
10: end for  
11: end while
Algorithm 2 Auto-decoding with mNIF
1: Randomly initialize the shared parameter  $\theta$  of mNIF.  
2: Initialize a latent vector  $\phi_j \sim \mathcal{N}(\mathbf{0}, \sigma^2\mathbf{I}) \in \mathbb{R}^H$  for all samples.  
3: while training do  
4: Sample a mini-batch  $\mathcal{B} = \{(\mathbf{c}_j, \mathbf{y}_j)\}_{j=1:|\mathcal{B}|}$ .  
5: Define a joint parameter:  $\phi_{\mathcal{B}} = \{\phi_j\}_{j=1:|\mathcal{B}|}$   
6: Update the parameters:  $\{\theta, \phi_{\mathcal{B}}\} \gets \{\theta, \phi_{\mathcal{B}}\} - \epsilon \nabla_{\theta, \phi_{\mathcal{B}}} \sum_{j} \mathcal{L}(f_{\theta, \phi}(\mathbf{c}_j), \mathbf{y}_j)|_{\phi = \phi_j}$   
7: end while
we introduce the sampling strategy for the context vectors  $\phi$  and customize it to a specific target task. To this end, we adopt the denoising diffusion probabilistic model (DDPM) [18], which employs the residual MLP architecture introduced in Functa [13].
This section demonstrates the effectiveness of the proposed approach, referred to as mINF, and discusses the characteristics of our algorithm based on the results. We run all experiments on the Vessl environment [2], and describe the detailed experiment setup for each benchmark in the appendix.
5.1 Datasets and Evaluation Protocols
We adopt CelebA-HQ  $64^{2}$  [21], ShapeNet  $64^{3}$  [8] and SRN Cars [34] dataset for image, voxel and neural radiance field (NeRF) generation, respectively, where  $64^{2}$  and  $64^{3}$  denotes the resolution of samples in the dataset. We follow the protocol from Functa [13] for image and NeRF scene and generative manifold learning (GEM) [12] for voxel.
We adopt the following metrics for performance evaluation. To measure the reconstruction quality, we use mean-squared error (MSE), peak signal-to-noise ratio (PSNR), reconstruction Fréchet inception distance (rFID), reconstruction precision (rPrecision) and reconstruction recall (rRecall). In image generation, we use Fréchet inception distance (FID) score [17], precision, recall [32, 27] and F1 score between sampled images and images in a train split. Voxel generation performance is evaluated by coverage and maximum mean discrepancy (MMD) metrics [1] on a test split. In NeRF scene generation, we use FID score between rendered images and images in a test split for all predefined views for evaluation. To compare model size of algorithms, we count the number of parameters for training and inference separately; the parameters for training contain all the weights required in the training procedure, e.g., the parameters in the mixture components and project matrix for our algorithm, while the model size for inference is determined by the parameters used for sampling instances. For the evaluation of efficiency, we measure the number of floating point operations per second (FLOPS), latency in terms of frames per second (fps), and the amount of memory consumption for a single sample.