SlowGuess/NeurIPS_2023_OCR huggingface dataset:
High-Resolution Image Reconstruction We further evaluate our framework on the reconstruction of FFHQ images with  $256 \times 256$ ,  $512 \times 512$ ,  $1024 \times 1024$  resolutions to demonstrate our effectiveness to capture fine-grained data details in Table 2. Although the performance increases as the MLP dimensionality  $d$  and the number of latents  $R$  increases, we use the same experimental setting with  $178 \times 178$  im
Table 2: PSNRs on the reconstructed FFHQ with  ${256} \times  {256},{512} \times  {512}$  ,and  ${1024} \times  {1024}$  resolutions.
256×256	512×512	1024×1024
TransINR	30.96	29.35	-
IPC [19]	34.68	31.58	28.68
Ours	39.88	35.43	31.94
age reconstruction to validate the efficacy of our framework. Our framework consistently achieves higher PSNRs than TransINR and IPC for all resolutions. Figure 3 also shows that TransINR and IPC cannot reconstruct the fine-grained details of a  $512 \times 512$  image, but our framework provides a
Figure 4: (a) PSNRs on novel view synthesis of ShapeNet Chairs, Cars, and Lamps according to the number of support views (1-5 views). (b) Examples of novel view synthesis with 4 support views.
high-quality result of reconstructed images. The results demonstrate that leveraging the locality of data is crucial for generalizable INR to model complex and high-resolution data.
4.2 Few-Shot Novel View Synthesis
We evaluate our framework on novel view synthesis with the ShapeNet Chairs, Cars, and Lamps datasets to predict a rendered image of a 3D object under an unseen view. Given few views of an object with known camera poses, we employ a light field [32] for novel view synthesis. A light field does not use computationally intensive volume rendering [24] but directly predicts RGB colors for the input coordinate for rays with  $d_{\mathrm{in}} = 6$  using the Plücker coordinate system. Our INR decoder uses  $d = 256$  and two levels of feature modulations with  $\sigma_{\mathrm{q}} = 2$  and  $(\sigma_1,\sigma_2) = (8,4)$ .
Figure 4(a) shows that our framework outperforms IPC for novel view synthesis. Our framework shows competitive performance with IPC when only one support view is provided. However, the performance of our framework is consistently improved as the number of support views increases, while outperforming the results of IPC. Note that defining a local relationship between rays is not straightforward due to its non-grid property of the Plücker coordinate. Our Transformer encoder can learn the local relationship between rays to extract locality-aware latent tokens during training and achieve high performance. We analyze the learned locality of rays encoded in the extracted latents in Section 4.3. Figure 4(b) shows that our framework correctly predicts the colors and shapes of a novel view corresponding to the support views, although the predicted views are blurry due to the lack of training objectives with generative modeling. We expect that combining our framework with generative models [5, 38] to synthesize a photorealistic novel view is an interesting future work.
4.3 In-Depth Analysis
Learning Curves on ImageNette  $178 \times 178$  Figure 1 juxtaposes the learning curves of our framework and previous approaches on ImageNette  $178 \times 178$ . Note that TransINR, IPC, and our framework use the same Transformer encoder to extract data latents, while adopting different modulation methods. While the training speed of our framework is about  $80\%$  of the speed of IPC, we remark our framework achieves the test PSNR of 38.72 after 400 epochs of training, outperforming the PSNR of 38.46 achieved by IPC trained for 4000 epochs, hence resulting in  $8 \times$  speed-up of training time. That is, our locality-aware latents enables generalizable INR to be both efficient and effective.
Selective token aggregation and multi-band
feature modulations We conduct an ablation study on image reconstruction of with ImageNette  $178\times 178$  and FFHQ  $256\times 256$ , novel view synthesis with Lamp-3 views to validate the effectiveness of the selective token aggregation and the multiband feature modulation. We replace the multiband feature modulations with a simple stack of MLPs (ours w/o multiFM), and the selective token aggregation with the weight modulation of IPC
(ours w/o STA). If both two modules are replaced together, the INR decoder becomes the same
Table 3: Ablation study on ImageNette  ${178} \times  {178}$  ,FFHQ  ${256} \times  {256}$  ,and Lamp-3 views.
ImageNette	FFHQ	Lamp
Ours	37.46	38.01	26.00
w/o STA	34.54	34.52	25.31
w/o multiFM	33.90	33.65	25.78
IPC [19]	34.11	34.68	25.09
Figure 5: Visualization of differences between model predictions after replacing a latent token with the zero vector, for IPC [19] and our framework.
architecture as IPC. We use single-head cross-attention for the selective token aggregation to focus on the effect of two modules. Table 3 demonstrates that both the selective token aggregation and the multi-band feature modulation are required for the performance improvement, as there is no significant improvement when only one of the modules is used.
Choices of frequency bandwidths Table 4 shows that the ordering of frequency bandwidths in Eq. (4) and Eq. (6) can affect the performance. We train our framework with two-level feature modulations on ImageNette  $178\times 178$  during 400 epochs with different settings of the bandwidths  $\sigma_{1},\sigma_{2},\sigma_{\mathrm{q}}$  Although our framework outperforms IPC regardless of the bandwidth settings, the best PSNR is achieved with  $\sigma_{1}\geq \sigma_{2}\geq$ $\sigma_{\mathrm{q}}$  . The results imply that selective token aggregation does not require high-frequency features, but the high-frequency features need to be processed by more nonlinear operations than lower-frequency features as discussed in Section 3.3.2.
Table 4: PSNRs of reconstructed ImageNette  ${178} \times  {178}$  with various frequency bandwidths.
(σ1, σ2)	σq	ImageNette
(128, 32)	16	37.46
(32, 128)	16	35.00
(128, 128)	16	35.30
(128, 32)	128	35.58
IPC (σ = 128)	34.11
The role of extracted latent tokens Figure 5 shows that our framework encodes the local information of data into each latent token, while IPC cannot learn the locality in data coordinates. To visualize the information in each latent token, we randomly select a latent token to be replaced with the zero vector. Then, we visualize the difference between the model predictions with or without the replacement. Each latent token of our framework encapsulates the local information in different regions of images and light fields. However, the latent tokens of IPC cannot exploit the local information of data, while encoding the global information over whole coordinates. Note that our framework learns the structure of locality in light fields during training, although the structure of the Plücker coordinate system is not regular as the grid coordinates of images. Thus, our framework can learn the locality-aware latents of data for generalizable INR regardless of the types of coordinate systems.
4.4 Generating INRs for Conditional Image Synthesis
We examine the potentials of the extracted latent tokens to be utilized for a downstream task such as class-conditional image generation of ImageNet [9]. Note that we cannot use the architecture of U-Net in conventional image diffusion models [4, 30], since our framework is not tailored to the 2D grid coordinate. Thus, we adopt a Transformer-based diffusion model [15, 27] to predict a set of latent tokens after corrupting the latents by Gaussian noises.
Figure 6: The examples of generated  $256 \times 256$  images by generating latents of IPC (left) and ours (right), trained on ImageNet.
We train 458M parameters of Transformers during 400 epochs to generate our locality-aware latent tokens. We attach the detailed setting in Appendix A.3. When we train a diffusion model to generate latent tokens of IPC in Figure 6, the generated images suffer from severe artifacts, because the prediction error of each latent token for IPC leads to the artifacts over all coordinates. Contrastively, the diffusion model for our locality-aware latents generates realistic images.
In addition, although we do not conduct exhaustive hyperparameter search, the FID score of generated images achieves 9.3 with classifier-free guidance scale [16] in Table 5. Thus, the results validate the potential applications of the local latents for INRs. Meanwhile, a few generated images may exhibit checkerboard artifacts, particularly in simple backgrounds, but we leave the elaboration of a diffusion process and sampling techniques for generating INR latents as future work.
Table 5: Reconstructed PSNRs and FID of generated images on ImageNet  $256\times 256$
Latent Shape	rPSNR	FID
Ours	256×256	37.7	9.3
Spatial	16×16×256	37.2	11.7
Functa [4]	32×32×64	37.7	8.8
LDM [30]	64×64×3	27.4	3.6
4.5 Comparison with Overfitted INRs
Figure 7 shows that our generalizable INR efficiently provides meaningful INRs compared with individual training of INRs per sample. To evaluate the efficiency of our framework, we select ten images of FFHQ  $256 \times 256$  and train randomly initialized FFNet [36] per sample using one NVIDIA V100 GPU. The individual training of FFNets requires over 10 seconds of optimization to achieve the same PSNRs of our framework, where our inference time is negligible. Moreover, when we apply the test-time optimization (TTO) only for the extracted latents, it consistently outperforms per-sample FFNets for 30 seconds while maintaining the structure of latents. When we consider the predicted INR as initialization and finetune all parameters of the INR decoder per each sample, our framework consistently outperforms the per-sampling training
Figure 7: Comparison with individually trained FFNets [36] per sample.
of INRs from random initialization. Thus, the results imply that leveraging generalizable INR is computationally efficient to model unseen data as INRs regardless of a TTO.
We have proposed an effective framework for generalizable INR with the Transformer encoder and locality-aware INR decoder. The Transformer encoder captures the locality of data entities and learns to encode the local information into different latent tokens. Our INR decoder selectively aggregates the locality-aware latent tokens to extract a modulation vector for a coordinate input and exploits the multiple bandwidths of frequency features to effectively predict the fine-grained data details. Experimental results demonstrate that our framework significantly outperforms previous generalizable
INRs on image reconstruction and few-shot novel view synthesis. In addition, we have conducted the in-depth analysis to validate the effectiveness of our framework and shown that our locality-aware latent tokens for INRs can be utilized for downstream tasks such as image generation to provide realistic images. Considering that our framework can learn the locality in non-grid coordinates, such as the Plücker coordinate for rays, leveraging our generalizable INR to generate 3D objects or scenes is a worth exploration. In addition, extending our framework to support arbitrary resolution will be an interesting future work. Furthermore, since our framework has still room for performance improvement of high-resolution image reconstruction, such as  $1024 \times 1024$ , we expect that elaborating on the architecture and techniques for diffusion models to effectively generate INRs is an interesting future work.
This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No.2018-0-01398: Development of a Conversational, Self-tuning DBMS, 35%; No.2022-0-00113: Sustainable Collaborative Multi-modal Lifelong Learning, 30%) and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. NRF-2021R1A2B5B03001551, 35%)
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
[2] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning, pages 685-694. PMLR, 2020.
[3] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
[4] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130, 2023.
[5] Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In arXiv, 2023.
[6] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34:21557-21568, 2021.
[7] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628-8638, 2021.
[8] Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision, pages 170–187. Springer, 2022.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[10] Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: CCompression with implicit neural representations. In Neural Compression: From Information Theory to Applications - Workshop @ ICLR 2021, 2021.
[11] Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning, pages 5694-5725. PMLR, 2022.
[31] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[32] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19325, 2021.
[33] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[34] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248–8258, 2022.
[35] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846-2855, 2021.
[36] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[38] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In The Eleventh International Conference on Learning Representations, 2023.
[39] Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
[40] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pages 7693-7702. PMLR, 2019.
We describe the implementation details of our locality-aware generalizable INR with the Transformer encoder and locality-aware INR decoder. We implement our framework based on the official open-sourced implementation of  $\mathrm{IPC}^3$  for a fair comparison. Our Transformer encoder comprises six blocks of self-attention with 12 attention heads, where each head uses 64 dimensions of hidden features, and  $R = 256$  latent tokens for all experiments. We use the Adam [20] optimizer with  $(\beta_{1},\beta_{2}) = (0.9,0.999)$  and constant learning rate of 0.0001. The batch size is 16 and 32 for image reconstruction and novel view synthesis, respectively.
A.1 Image Reconstruction
$178 \times 178$  image reconstruction For the image reconstruction of CelebA, FFHQ, and ImageNette with  $178 \times 178$  resolution, we use  $L = 2$  level of modulation features for multi-band feature modulation of locality-aware INR decoder. The dimensionality of frequency features and hidden layers in the INR decoder is 256, where  $(\sigma_{1}, \sigma_{2}, \sigma_{\mathrm{q}}) = (128, 32, 16)$ . We represent a  $178 \times 178$  resolution of the image as 400 tokens, where each token corresponds to a  $9 \times 9$  size of the image patch with zero padding. We use a multi-head attention block with two attention heads for our selective token selection via cross-attention. Following the experimental setting of previous studies [8, 19], we train our framework on CelebA, FFHQ, and ImageNette during 300, 1000, and 4000 epochs, respectively. When we use four NVIDIA V100 GPUs, the training takes 5.5, 6.7, and 4.3 days, respectively.
ImageNet  $256 \times 256$  We use  $L = 2$  level of feature modulation for the image reconstruction of ImageNet with  $256 \times 256$  resolution. We use eight heads of selective token aggregation, 256 dimensionality of frequency features and hidden layers of the INR decoder, and  $(\sigma_{1}, \sigma_{2}, \sigma_{\mathrm{q}}) = (128, 32, 16)$ . An image is represented as 256 tokens, where each token corresponds to a  $16 \times 16$  patch in the image. We use eight NVIDIA A100 GPUs to train our framework on ImageNet during 20 epochs, where the training takes about 2.5 days.
FFHQ  $256 \times 256$ ,  $512 \times 512$ , and  $1024 \times 1024$  Our framework for FFHQ  $256 \times 256$  and  $512 \times 512$  uses  $L = 2$  level of feature modulation with  $(\sigma_{1}, \sigma_{2}, \sigma_{\mathrm{q}}) = (128, 32, 16)$ . The size of each patch is 16 and 32 for  $256 \times 256$  and  $512 \times 512$  resolutions, respectively, the number of latent tokens is  $R = 256$ , and the dimensionality of the INR decoder is  $d_{\mathrm{F}} = d = 256$ . Our selective token aggregation uses two and four heads of cross-attention for FFHQ  $256 \times 256$  and  $512 \times 512$ , respectively. We randomly sample the  $10\%$  of coordinates to be decoded at each training step to increase the efficiency of training. We train our framework during 400 epochs, while the training takes about 1.5 days using four NVIDIA V100 GPUs for FFHQ with  $256 \times 256$  and about 1.4 days using eight V100 GPUs for FFHQ with  $512 \times 512$ . For FFHQ  $1024 \times 1024$ , we use 48 patch size to represent an image as 484 data tokens and  $L = 2$  level of feature modulation with  $(\sigma_{1}, \sigma_{2}, \sigma_{\mathrm{q}}) = (256, 64, 32)$ . The training of 400 epochs takes about 3.4 days using eight NVIDIA V100 GPUs.
A.2 Novel View Synthesis
We train our framework for the task of novel view synthesis on ShapeNet Cars, Chairs, and Lamps. Given a few known camera views as support views of a 3D object, our framework predicts a light field of the 3D object to predict unseen camera views. For a fair comparison, we use the same splits of train-valid samples with previous studies of generalizable INR [8, 19, 35]. Given rendering images of support views with  $128 \times 128$  resolution, we first patchify each rendered image into 256 tokens with  $8 \times 8$  size of patches. Then, we concatenate the patches of all support views with learnable tokens for the input of our Transformer. We use the Plücker coordinate to represent a ray for a pixel as an embedding with six dimensions and concatenate the ray embedding into each pixel along the channel dimension. Since our INR decoder estimates a light field of a 3D object, the INR decoder has six input channels  $d_{\mathrm{in}} = 6$  for a ray coordinate and three output channels  $d_{\mathrm{out}} = 3$  for a RGB pixel. Our INR decoder uses  $L = 2$  level of feature modulation with  $(\sigma_1, \sigma_2, \sigma_{\mathrm{q}}) = (8, 4, 2)$ . We use  $d_{\mathrm{F}} = d = 256$  dimensionality of the frequency features and hidden features of the INR decoder. We use 1000 training epochs for ShapeNet Cars and Chairs, while using 500 epochs for ShapeNet Lamps.
<sup>3</sup>https://github.com/kakaobrain/ginr-ipc
A.3 Diffusion Model for INR generation
We implement a diffusion model to generate the latent tokens for INRs of ImageNet  $256 \times 256$ . Different from the conventional approaches, which use a U-Net architecture to generate an image, we use a vanilla Transformer with a simple stack of self-attention, since the latent tokens do not predefine 2D grid structure but are permutation-equivariant. The Transformer for the diffusion model has 458M parameters having 24 self-attention blocks with 1024 dimensions of embeddings and 16 heads. We remark that the locality-aware generalizable INR is not updated during the training of diffusion models. For the training of the diffusion model, we follow the formulation of DDPM [15]. The linear schedule with  $T = 1000$  is used to randomly corrupt the latent tokens for INRs using isotropic Gaussian noises, and then we train our Transformer to denoise the latent tokens. Instead of the  $\epsilon$ -parameterization that predicts the noises used for the corruption, our Transformer  $\mathbf{x}_0$ -parameterization to predict the original latent tokens. We drop  $10\%$  of class conditions for our model to support classifier-free guidance following the conventional setting [16]. For the stability of training, we standardize the features of latent tokens, after computing the mean and standard deviation of feature channels of each latent token based on the training data. We use eight NVIDIA A100 GPUs to train the model with 256 batch size during 400 epochs, where the training takes about 7 days. The Adam [20] optimizer with constant learning rate 0.0001 and  $(\beta_1, \beta_2) = (0.9, 0.999)$  is used without learning rate warm-up and any weight decaying. During training, we further compute the exponential moving average (EMA) of model parameters with a decaying rate of 0.9999. During the evaluation, we use the EMA model with 250 DDIM steps [33] and 2.5 scales of classifier-free guidance [16].
B.1 Ablation Study on the Number of Levels