Title: Locality-Aware Generalizable Implicit Neural Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit neural representations

representation learning

neural fields
Abstract: Generalizable implicit neural representation (INR) enables a single continuous function, i.e., a coordinate-based neural network, to represent multiple data instances by modulating its weights or intermediate features using latent codes. However, the expressive power of the state-of-the-art modulation is limited due to its inability to localize and capture fine-grained details of data entities such as specific pixels and rays. To address this issue, we propose a novel framework for generalizable INR that combines a transformer encoder with a locality-aware INR decoder. The transformer encoder predicts a set of latent tokens from a data instance to encode local information into each latent token. The locality-aware INR decoder extracts a modulation vector by selectively aggregating the latent tokens via cross-attention for a coordinate input and then predicts the output by progressively decoding with coarse-to-fine modulation through multiple frequency bandwidths. The selective token aggregation and the multi-band feature modulation enable us to learn locality-aware representation in spatial and spectral aspects, respectively. Our framework significantly outperforms previous generalizable INRs and validates the usefulness of the locality-aware latents for downstream tasks such as image generation.

Locality-Aware Generalizable Implicit Neural Representation

Abstract

Generalizable implicit neural representation (INR) enables a single continuous function, i.e., a coordinate-based neural network, to represent multiple data instances by modulating its weights or intermediate features using latent codes. However, the expressive power of the state-of-the-art modulation is limited due to its inability to localize and capture fine-grained details of data entities such as specific pixels and rays. To address this issue, we propose a novel framework for generalizable INR that combines a transformer encoder with a locality-aware INR decoder. The transformer encoder predicts a set of latent tokens from a data instance to encode local information into each latent token. The locality-aware INR decoder extracts a modulation vector by selectively aggregating the latent tokens via cross-attention for a coordinate input and then predicts the output by progressively decoding with coarse-to-fine modulation through multiple frequency bandwidths. The selective token aggregation and the multi-band feature modulation enable us to learn localityaware representation in spatial and spectral aspects, respectively. Our framework significantly outperforms previous generalizable INRs and validates the usefulness of the locality-aware latents for downstream tasks such as image generation.

1 Introduction

Recent advances in generalizable implicit neural representation (INR) enable a single coordinate-based multi-layer perceptron (MLP) to represent multiple data instances as a continuous function. Instead of per-sample training of individual coordinate-based MLPs, generalizable INR extracts latent codes of data instances [13, 14, 40] to modulate the weights or intermediate features of the shared MLP model [8, 11, 19, 35]. However, despite the advances in previous approaches, their performance is still insufficient compared with individual training of INRs per sample.

We postulate that the expressive power of generalizable INRs is limited by the ability of locality-awareness to localize relevant entities from a data instance and control

0

1000

2000

3000

4000

Epochs

20

25

30

35

40

45

Test PSNR (dB)

TransINR

IPC

Ours

their structure in a fine-grained manner. Primitive entities of a data instance, such as pixels in an image, tend to have a higher correlation with each other if they are closer in space and time. Thus, this locality of data entities has been used as an important inductive bias for learning the representations of complex data [3]. However, previous approaches to generalizable INRs are not properly designed to leverage the locality of data entities. For example, when latent codes modulate intermediate features [11, 12] or weight matrices [8, 19, 35] of an INR decoder, the modulation methods do not exploit a specified coordinates for decoding, which restricts the latent codes to encoding global information over all pixels without capturing local relationships between specific pixels.

To address this issue, we propose a novel encoder-decoder framework for locality-aware generalizable INR to effectively localize and control the fine-grained details of data. In our framework, a Transformer encoder [37] first extracts locally relevant information from a data instance and predicts a set of latent tokens to encode different local information. Then, our locality-aware INR decoder effectively leverages the latent tokens to predict fine-grained details. Specifically, given an input coordinate, our INR decoder uses a cross-attention to selectively aggregate the local information in the latent tokens and extract a modulation vector for the coordinate. In addition, our INR decoder effectively captures the high-frequency details in the modulation vector by decomposing it into multiple bandwidths of frequency features and then progressively composing the intermediate features. We conduct extensive experiments to demonstrate the high performance and efficacy of our locality-aware generalizable INR on benchmarks as shown in Figure 1. In addition, we show the potential of our locality-aware INR latents to be utilized for downstream tasks such as image synthesis.

Our main contributions can be summarized as follows: 1) We propose an effective framework for generalizable INR with a Transformer encoder and locality-aware INR decoder. 2) The proposed INR decoder with selective token aggregation and multi-band feature modulation can effectively capture the local information to predict the fine-grained data details. 3) The extensive experiments validate the efficacy of our framework and show its applications to a downstream image generation task.

2 Related Work

Implicit neural representations (INRs). INRs use neural networks to represent complex data such as audio, images, and 3D scenes, as continuous functions. Especially, incorporating Fourier features [24, 36], periodic activations [31], or multi-grid features [25] significantly improves the performance of INRs. Despite its broad applications [1, 6, 10, 32, 34], INRs commonly require separate training of MLPs to represent each data instance. Thus, individual training of INRs per sample does not learn common representations in multiple data instances.

Generalizable INRs. Previous approaches focus on two major components for generalizable INRs; latent feature extraction and modulation methods. Auto-decoding [23, 26] computes a latent vector per data instance and concatenates it with the input of a coordinate-based MLP. Given input data, gradient-based meta-learning [4, 11, 12] adapts a shared latent vector using a few update steps to scale and shift the intermediate activations of the MLP. Learned Init [35] also uses gradient-based meta-learning but adapts whole weights of the shared MLP. Although auto-decoding and gradientbased meta-learning are agnostic to the types of data, their training is unstable on complex and large-scale datasets. TransINR [8] employs the Transformer [37] as a hypernetwork to predict latent vectors to modulate the weights of the shared MLP. In addition, Instance Pattern Composers [19] have demonstrated that modulating the weights of the second MLP layer is enough to achieve high performance of generalizable INRs. Our framework also employs the Transformer encoder, but focuses on extracting locality-aware latent features for the high performance of generalizable INR.

Leveraging Locality of Data for INRs Local information in data has been utilized for efficient modeling of INRs, since local relationships between data entities are widely used for effective process of complex data [3]. Given an input coordinate, the coordinate-based MLP only uses latent vectors nearby the coordinate, after a CNN encoder extracts a 2D grid feature map of an image for super-resolution [7] and reconstruction [22]. Spatial Functa [4] demonstrates that leveraging the locality of data enables INRs to be utilized for downstream tasks such as image recognition and generation. Local information in 3D coordinates has also been effective for scene modeling as a hybrid approach using 3D feature grids [18] or the part segmentation [17] of a 3D object. However, previous approaches assume explicit grid structures of latents tailored to a specific data type. Since

Locality-Aware INR Decoder

Transformer Encoder

with Self-Attentions

Learnable Tokens

Localized Latents

Localized Latents

Selective

Token Aggregation

(Cross-Attention)

modulation

vector

Linear

Query

Key

Value

Linear-ReLU

output feature

R

G

B

coordinate input

Fourier-Linear

coordinate input

multi-band feature modulation

m

v

image/views patches

Linear-ReLU

Linear

Fourier-Linear

Fourier-Linear

: element-wise addition

: ReLU activation

coordinate input

we do not predefine a relationship between latents, our framework is flexible to learn and encode the local information of both grid coordinates in images and non-grid coordinates in light fields.

3 Methods

We propose a novel framework for locality-aware generalizable INR which consists of a Transformer encoder to localize the information in data into latent tokens and a locality-aware INR decoder to exploit the localized latents and predict outputs. First, we formulate how generalizable INR enables a single coordinate-based neural network to represent multiple data instances as a continuous function by modulating its weights or features. Then, after we introduce the Transformer encoder to extract a set of latent tokens from input data instances, we explain the details of the locality-aware INR decoder, where selective token selection aggregates the spatially local information for an input coordinate via cross-attention; multi-band feature modulation leverages a different range of frequency bandwidths to progressively decode the local information using coarse-to-fine modulation in the spectral domain.

3.1 Generalizable Implicit Neural Representation

Given a set of data instances X = { x ( n ) } N n =1 , each data instance x ( n ) = { ( v ( n ) i , y ( n ) i ) } M n i =1 comprises M n pairs of an input coordinate v ( n ) i ∈ R d in and the corresponding output feature y ( n ) i ∈ R d out . Conventional approaches [24, 31, 36] adopt individual coordinate-based MLPs to train and memorize each data instance x ( n ) . Thus, the coordinate-based MLP cannot be reused and generalized to represent other data instances, requiring per-sample optimization of MLPs for unseen data instances.

A generalizable INR uses a single coordinate-based MLP as a shared INR decoder F θ : R d in → R d out to represent multiple data instances as a continuous function. Generalizable INR [8, 11, 12, 19, 26] extracts the R number of latent codes Z ( n ) = { z ( n ) k ∈ R d } R k =1 from a data instance x ( n ) . Then, the latents are used for the INR decoder to represent a data instance x ( n ) as y ( n ) i = F θ ( v ( n ) i ; Z ( n ) ) , while updating the parameters θ and latents Z ( n ) to minimize the errors over X :

We remark that each previous approach employs a different number of latent codes to modulate a coordinate-based MLP. For example, a single latent vector ( R = 1 ) is commonly extracted to modulate intermediate features of the MLP [11, 12, 26], while a multitude of latents ( R > 1 ) are used to modulate its weights [8, 19, 35]. While we modulate the features of MLP, we extract a set of latent codes to localize the information of data to leverage the locality-awareness for latent features.

<latexi sh

1_b

64="

o

EK

yu

0

GI

p

LkFz

H

>A

B/n

c

V

NS8

X7

+R

WT

U

CqJ

P

Y9O

g

2

d

M

3

ZQ

D

v

r

j

m

w

5

f

3.2 Transformer Encoder

Our framework employs a Transformer encoder [37] to extract a set of latents Z ( n ) for each data instance x ( n ) as shown in Figure 2. After a data instance, such as an image or multi-view images, is patchified into a sequence of data tokens, we concatenate the patchified tokens into a sequence of R learnable tokens as the encoder input. Then, the Transformer encoder extracts a set of latent tokens, where each latent token corresponds to an input learnable token. Note that the permutationequivariance of self-attention in the Transformer encoder enables us not to predefine the local structure of data and the ordering of latent tokens. During training, each latent token learns to capture the local information of data, while covering whole regions to represent a data instance. Thus, whether a data instance is represented on a grid or non-grid coordinate, our framework is flexible to encode various types of data into latent tokens, while learning the local relationships of latent tokens during training.

3.3 Locality-Aware Decoder for Implicit Neural Representations

We propose the locality-aware INR decoder in Figure 2 to leverage the local information of data for effective generalizable INR. Our INR decoder comprises two primary components: i) Selective token aggregation via cross attention extracts a modulation vector for an input coordinate to aggregate spatially local information from latent tokens. ii) Multi-band feature modulation decomposes the modulation vector into multiple bandwidths of frequency features to amplify the high-frequency features and effectively predict the details of outputs.

3.3.1 Selective Token Aggregation via Cross-Attention

We remark that encoding locality-aware latent tokens is not straightforward since the self-attentions in Transformer do not guarantee a specific relationship between tokens. Thus, the properties of the latent tokens are determined by a modulation method for generalizable INR to exploit the extracted latents. For example, given an input coordinate v and latent tokens { z 1 , ..., z R } , a straightforward method can use Instance Pattern Composers [19] to construct a modulation weight W m = [ z 1 , ..., z R ] ⊤ ∈ R R × d in and extract a modulation vector m v = W m v = [ z ⊤ 1 v , ..., z ⊤ R v ] ⊤ ∈ R R . However, the latent tokens cannot encode the local information of data, since each latent token equally influences each channel of the modulation vector regardless of the coordinate locations (see Section 4.3).

Our selective token aggregation employs cross-attention to aggregate the spatially local latents nearby the input coordinate, while guiding the latents to be locality-aware. Given a set of latent tokens Z ( n ) = { z ( n ) k } R k =1 and a coordinate v ( n ) i , a modulation feature vector m ( n ) v i ∈ R d shifts the intermediate features of an INR decoder to predict the output, where d is the dimensionality of hidden layers in the INR decoder. For the brevity of notation, we omit the superscript n and subscript i .

Frequency features We first transform an input coordinate v = ( v 1 , · · · , v d in ) ∈ R d in into frequency features using sinusoidal positional encoding [31, 36]. We define the Fourier features γ σ ( v ) ∈ R d F with bandwidth σ > 1 and feature dimensionality d F as

where n = d F 2 d in . A frequency ω j = σ j/ ( n -1) is evenly distributed between 1 and σ on a log-scale. Based on the Fourier features, we define the frequency feature extraction h F ( · ) as

where W ∈ R d × d F and b ∈ R d are trainable parameters for frequency features, d denotes the dimensionality of hidden layers in the INR decoder.

Selective token selection via cross-attention To predict corresponding output y to the coordinate v , we adopt a cross-attention to extract a modulation feature vector m v ∈ R d based on the latent tokens Z = { z k } R k =1 . We first extract the frequency features of the coordinate v in Eq (3) as the query of the cross-attention as

where W q ∈ R d × d F and b q ∈ R d are trainable parameters, and σ q is the bandwidth for query frequency features. The cross-attention in Figure 2 enables the query to select latent tokens, aggregate

its local information, and extract the modulation feature vector m v for the input coordinate:

An intuitive implementation for selective token aggregation can employ hard attention to select only one latent token for each coordinate. However, in our primitive experiment, using hard attention leads to unstable training and a latent collapse problem that selects only few latent tokens. Meanwhile, multi-head attentions encourage each latent token to easily learn the locality in data instances.

3.3.2 Multi-Band Feature Modulation in the Spectral Domain

After the selective token aggregation extracts a modulation vector m v , we use multi-band feature modulation to effectively predict the details of outputs. Although Fourier features [24, 36] reduce the spectral bias [2, 28] of neural networks, adopting a simple stack of MLPs to INRs still suffers from capturing the high-frequency data details. To address this issue, we use a different range of frequency bandwidths to decompose the modulation vector into multiple frequency features in the spectral domain. Then, our multi-band feature modulation uses the multiple frequency features to progressively decode the intermediate features, while encouraging a deeper MLP path to learn higher frequency features. Note that the coarse-to-fine approach in the spectral domain is analogous to the locally hierarchical approach in the spatial domain [21, 29, 39] to capture the data details.

Extracting multiple modulation features with different frequency bandwidths We extract L level of modulation features m (1) v , · · · , m ( L ) v from m v using different bandwidths of frequency features. Given L frequency bandwidths as σ 1 ≥ σ 2 ≥ · · · ≥ σ L ≥ σ q, we use Eq (3) to extract the ℓ -th level of frequency features of an input coordinate v as

where W ( ℓ ) F and b ( ℓ ) F are trainable parameters and shared across data instances. Then, the ℓ -th modulation vector m ( ℓ ) v is extracted from the modulation vector m v as

with a trainable weight W ( ℓ ) m and bias b ( ℓ ) m . Considering that ReLU cutoffs the values below zero, we assume that m ( ℓ ) v filters out the information of m v based on the ℓ -th frequency patterns of ( h F ) ( ℓ ) v .

Multi-band feature modulation After decomposing a modulation vector into multiple features with different frequency bandwidths, we progressively compose the L modulation features by applying a stack of nonlinear operations with a linear layer and ReLU activation. Starting with h (1) v = m (1) v , we compute the ℓ -th hidden features h ( ℓ ) v for ℓ = 2 , · · · , L as

where W ( ℓ ) ∈ R d × d and b ( ℓ ) ∈ R d are trainable weights and biases of the INR decoder. ˜ h ( ℓ ) v denotes the ℓ -th pre-activation of INR decoder for coordinate v . Note that the modulation features with high-frequency bandwidth can be processed by more nonlinear operations than the features with lower frequency bandwidths, considering that high-frequency features contain more complex signals.

Finally, the output ˆ y is predicted using all intermediate hidden features of the INR decoder as

where f ( ℓ ) out : R d → R d out are a linear projection into the output space. Although utilizing only h ( L ) v is also an option to predict outputs, skip connections of all intermediate features into the output layer enhances the robustness of training to the hyperparameter choices.

D

4 Experiments

Weconduct extensive experiments to demonstrate the effectiveness of our locality-aware generalizable INR on image reconstruction and novel view synthesis. In addition, we conduct in-depth analysis to validate the efficacy of our selective token aggregation and multi-band feature modulation to localize the information of data to capture fined-grained details. We also show that our locality-aware latents can be utilized for image generation by training a generative model on the extracted latents. Our implementation and experimental settings are based on the official codes of Instance Pattern Composers [19] for a fair comparison. We attach the implementation details to Appendix A.

4.1 Image Reconstruction

We follow the protocols in previous studies [8, 19, 35] to evaluate our framework on image reconstruction of CelebA, FFHQ, and ImageNette with 178 × 178 resolution. Our framework also outperforms previous approaches on high-resolution images with 256 × 256, 512 × 512, and 1024 × 1024 resolutions of FFHQ. We compare our framework with Learned Init [35], TransINR [8], and IPC [19]. The Transformer encoder predicts R = 256 latent tokens, while the INR decoder uses d in = 2 , d out = 3 , d = 256 dimensionality of hidden features, L = 2 , σ q = 16 and ( σ 1 , σ 2 ) = (128 , 32) bandwidths.

178 × 178 Image Reconstruction Table 1 shows that our generalizable INR significantly outperforms previous methods by a large margin. We remark that TransINR, IPC, and our framework use the same capacity of the Transformer encoder, latent tokens, and INR decoder except for the modulation methods. Thus, the results imply that our locality-aware INR decoder with selective token aggregation and multi-band feature modulation is effective to

capture local information of data and fine-grained details for high-quality image reconstruction.

High-Resolution Image Reconstruction We further evaluate our framework on the reconstruction of FFHQ images with 256 × 256, 512 × 512, 1024 × 1024 resolutions to demonstrate our effectiveness to capture fine-grained data details in Table 2. Although the performance increases as the MLP dimensionality d and the number of latents R increases, we use the same experimental setting with 178 × 178 im-

age reconstruction to validate the efficacy of our framework. Our framework consistently achieves higher PSNRs than TransINR and IPC for all resolutions. Figure 3 also shows that TransINR and IPC cannot reconstruct the fine-grained details of a 512 × 512 image, but our framework provides a

1 view

2 views

3 views

4 views

5 views

18

19

20

21

22

23

24

25

26

27

28

29

PSNR (dB)

IPC (Chairs)

IPC (Cars)

Ours (Chairs)

Ours (Cars)

Support views

Predicted

view

Query

view

high-quality result of reconstructed images. The results demonstrate that leveraging the locality of data is crucial for generalizable INR to model complex and high-resolution data.

4.2 Few-Shot Novel View Synthesis

We evaluate our framework on novel view synthesis with the ShapeNet Chairs, Cars, and Lamps datasets to predict a rendered image of a 3D object under an unseen view. Given few views of an object with known camera poses, we employ a light field [32] for novel view synthesis. A light field does not use computationally intensive volume rendering [24] but directly predicts RGB colors for the input coordinate for rays with d in = 6 using the Plücker coordinate system. Our INR decoder uses d = 256 and two levels of feature modulations with σ q = 2 and ( σ 1 , σ 2 ) = (8 , 4) .

Figure 4(a) shows that our framework outperforms IPC for novel view synthesis. Our framework shows competitive performance with IPC when only one support view is provided. However, the performance of our framework is consistently improved as the number of support views increases, while outperforming the results of IPC. Note that defining a local relationship between rays is not straightforward due to its non-grid property of the Plücker coordinate. Our Transformer encoder can learn the local relationship between rays to extract locality-aware latent tokens during training and achieve high performance. We analyze the learned locality of rays encoded in the extracted latents in Section 4.3. Figure 4(b) shows that our framework correctly predicts the colors and shapes of a novel view corresponding to the support views, although the predicted views are blurry due to the lack of training objectives with generative modeling. We expect that combining our framework with generative models [5, 38] to synthesize a photorealistic novel view is an interesting future work.

4.3 In-Depth Analysis

Learning Curves on ImageNette 178 × 178 Figure 1 juxtaposes the learning curves of our framework and previous approaches on ImageNette 178 × 178. Note that TransINR, IPC, and our framework use the same Transformer encoder to extract data latents, while adopting different modulation methods. While the training speed of our framework is about 80% of the speed of IPC, we remark our framework achieves the test PSNR of 38.72 after 400 epochs of training, outperforming the PSNR of 38.46 achieved by IPC trained for 4000 epochs, hence resulting in 8 × speed-up of training time. That is, our locality-aware latents enables generalizable INR to be both efficient and effective.

Selective token aggregation and multi-band feature modulations We conduct an ablation study on image reconstruction of with ImageNette 178 × 178 and FFHQ 256 × 256, novel view synthesis with Lamp-3 views to validate the effectiveness of the selective token aggregation and the multiband feature modulation. We replace the multiband feature modulations with a simple stack of MLPs (ours w/o multiFM), and the selective token aggregation with the weight modulation of IPC

(ours w/o STA). If both two modules are replaced together, the INR decoder becomes the same

IPC

Ours

IPC

Ours

The difference between model predictions

after replacing a latent token with the zero vector

Reconstructed/

rendered image

architectrure as IPC. We use single-head cross-attention for the selective token aggregation to focus on the effect of two modules. Table 3 demonstrates that both the selective token aggregation and the multi-band feature modulation are required for the performance improvement, as there is no significant improvement when only one of the modules is used.

Choices of frequency bandwidths Table 4 shows that the ordering of frequency bandwidths in Eq. (4) and Eq. (6) can affect the performance. We train our framework with twolevel feature modulations on ImageNette 178 × 178 during 400 epochs with different settings of the bandwidths σ 1 , σ 2 , σ q. Although our framework outperforms IPC regardless of the bandwidth settings, the best PSNR is achieved with σ 1 ≥ σ 2 ≥ σ q. The results imply that selective token aggregation does not require high-frequency features, but the high-frequency features need to be processed by more nonlinear operations than lower-frequency features as discussed in Section 3.3.2.

The role of extracted latent tokens Figure 5 shows that our framework encodes the local information of data into each latent token, while IPC cannot learn the locality in data coordinates. To visualize the information in each latent token, we randomly select a latent token to be replaced with the zero vector. Then, we visualize the difference between the model predictions with or without the replacement. Each latent token of our framework encapsulates the local information in different regions of images and light fields. However, the latent tokens of IPC cannot exploit the local information of data, while encoding the global information over whole coordinates. Note that our framework learns the structure of locality in light fields during training, although the structure of the Plücker coordinate system is not regular as the grid coordinates of images. Thus, our framework can learn the locality-aware latents of data for generalizable INR regardless of the types of coordinate systems.

4.4 Generating INRs for Conditional Image Synthesis

We examine the potentials of the extracted latent tokens to be utilized for a downstream task such as class-conditional image generation of ImageNet [9]. Note that we cannot use the architecture of U-Net in conventional image diffusion models [4, 30], since our framework is not tailored to the 2D grid coordinate. Thus, we adopt a Transformer-based diffusion model [15, 27] to predict a set of latent tokens after corrupting the latents by Gaussian noises.

CRaAM

We train 458M parameters of Transformers during 400 epochs to generate our locality-aware latent tokens. We attach the detailed setting in Appendix A.3. When we train a diffusion model to generate latent tokens of IPC in Figure 6, the generated images suffer from severe artifacts, because the prediction error of each latent token for IPC leads to the artifacts over all coordinates. Contrastively, the diffusion model for our locality-aware latents generates realistic images.

In addition, although we do not conduct exhaustive hyperparamter search, the FID score of generated images achieves 9.3 with classifier-free guidance scale [16] in Table 5. Thus, the results validate the potential applications of the local latents for INRs. Meanwhile, a few generated images may exhibit checkerboard artifacts, particularly in simple backgrounds, but we leave the elaboration of a diffusion process and sampling techniques for generating INR latents as future work.

4.5 Comparison with Overfitted INRs

Figure 7 shows that our generalizable INR efficiently provides meaningful INRs compared with individual training of INRs per sample. To evaluate the efficiency of our framework, we select ten images of FFHQ 256 × 256 and train randomly initialized FFNet [36] per sample using one NVIDIA V100 GPU. The individual training of FFNets requires over 10 seconds of optimization to achieve the same PSNRs of our framework, where our inference time is negligible. Moreover, when we apply the test-time optimization (TTO) only for the extracted latents, it consistently outperforms per-sample FFNets for 30 seconds while maintaining the structure of latents. When we consider the predicted INR as initialization and finetune all parameters of the INR decoder per each sample, our framework consistently outperforms the per-sampling training

0

5

10

15

20

25

30

Avg. runtime (sec / sample)

25

30

35

40

45

50

PSNR (dB)

Ours (no TTO)

Ours (TTO of latents)

Ours (TTO of all parameters)

FFNet (per-sample optim.)

of INRs from random initialization. Thus, the results imply that leveraging generalizable INR is computationally efficient to model unseen data as INRs regardless of a TTO.

5 Conclusion

We have proposed an effective framework for generalizable INR with the Transformer encoder and locality-aware INR decoder. The Transformer encoder capture the locality of data entities and learn to encode the local information into different latent tokens. Our INR decoder selectively aggregates the locality-aware latent tokens to extract a modulation vector for a coordinate input and exploits the multiple bandwidths of frequency features to effectively predict the fine-grained data details. Experimental results demonstrate that our framework significantly outperforms previous generalizable

INRs on image reconstruction and few-shot novel view synthesis. In addition, we have conducted the in-depth analysis to validate the effectiveness of our framework and shown that our locality-aware latent tokens for INRs can be utilized for downstream tasks such as image generation to provide realistic images. Considering that our framework can learn the locality in non-grid coordinates, such as the Plücker coordinate for rays, leveraging our generalizable INR to generate 3D objects or scenes is a worth exploration. In addition, extending our framework to support arbitrary resolution will be an interesting future work. Furthermore, since our framework has still room for performance improvement of high-resolution image reconstruction, such as 1024 × 1024, we expect that elaborating on the architecture and techniques for diffusion models to effectively generate INRs is an interesting future work.

References

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. In International Conference on Machine Learning , pages 685-694. PMLR, 2020.

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 , 2018.

Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03130 , 2023.

Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In arXiv , 2023.

Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems , 34:21557-21568, 2021.

Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8628-8638, 2021.

Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision , pages 170-187. Springer, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: COmpression with implicit neural representations. In Neural Compression: From Information Theory to Applications - Workshop @ ICLR 2021 , 2021.

Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning , pages 5694-5725. PMLR, 2022.

Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goli´ nski, Yee Whye Teh, and Arnaud Doucet. Coin++: Data agnostic neural compression. arXiv preprint arXiv:2201.12904 , 2022.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239 , 2020.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6001-6010, 2020.

Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. Relu fields: The little non-linearity that could. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1-9, 2022.

Chiheon Kim, Doyup Lee, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11523-11532, 2022.

Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14214-14223, 2021.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4460-4470, 2019.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , 41(4):102:1-102:15, July 2022.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 165-174, 2019.

William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748 , 2022.

Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning , pages 5301-5310. PMLR, 2019.

Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems , pages 14866-14876, 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems , 33:7462-7473, 2020.

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems , 34:19313-19325, 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2846-2855, 2021.

Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems , 33:7537-7547, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In The Eleventh International Conference on Learning Representations , 2023.

Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation. In Proceedings of the International Conference on Neural Information Processing Systems , 2022.

Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning , pages 7693-7702. PMLR, 2019.

A Implementation Details

We describe the implementation details of our locality-aware generalizable INR with the Transformer encoder and locality-aware INR decoder. We implement our framework based on the official opensourced implementation of IPC 3 for a fair comparison. Our Transformer encoder comprises six blocks of self-attentions with 12 attention heads, where each head uses 64 dimensions of hidden features, and R = 256 latent tokens for all experiments. We use the Adam [20] optimizer with ( β 1 , β 2 ) = (0 . 9 , 0 . 999) and constant learning rate of 0.0001. The batch size is 16 and 32 for image reconstruction and novel view synthesis, respectively.

A.1 Image Reconstruction

178 × 178 image reconstruction For the image reconstruction of CelebA, FFHQ, and ImageNette with 178 × 178 resolution, we use L = 2 level of modulation features for multi-band feature modulation of locality-aware INR decoder. The dimensionality of frequency features and hidden layers in the INR decoder is 256, where ( σ 1 , σ 2 , σ q ) = (128 , 32 , 16) . We represent a 178 × 178 resolution of the image as 400 tokens, where each token corresponds to a 9 × 9 size of the image patch with zero padding. We use a multi-head attention block with two attention heads for our selective token selection via cross-attention. Following the experimental setting of previous studies [8, 19], we train our framework on CelebA, FFHQ, and ImageNette during 300, 1000, and 4000 epochs, respectively. When we use four NVIDIA V100 GPUs, the training takes 5.5, 6.7, and 4.3 days, respectively.

ImageNet 256 × 256 We use L = 2 level of feature modulation for the image reconstruction of ImageNet with 256 × 256 resolution. We use eight heads of selective token aggregation, 256 dimensionality of frequency features and hidden layers of the INR decoder, and ( σ 1 , σ 2 , σ q ) = (128 , 32 , 16) . An image is represented as 256 tokens, where each token corresponds to a 16 × 16 patch in the image. We use eight NVIDIA A100 GPUs to train our framework on ImageNet during 20 epochs, where the training takes about 2.5 days.

FFHQ 256 × 256, 512 × 512, and 1024 × 1024 Our framework for FFHQ 256 × 256 and 512 × 512 uses L = 2 level of feature modulation with ( σ 1 , σ 2 , σ q ) = (128 , 32 , 16) . The size of each patch is 16 and 32 for 256 × 256 and 512 × 512 resolutions, respectively, the number of latent tokens is R = 256 , and the dimensionality of the INR decoder is d F = d = 256 . Our selective token aggregation uses two and four heads of cross-attention for FFHQ 256 × 256 and 512 × 512, respectively. We randomly sample the 10% of coordinates to be decoded at each training step to increase the efficiency of training. We train our framework during 400 epochs, while the training takes about 1.5 days using four NVIDIA V100 GPUs for FFHQ with 256 × 256 and about 1.4 days using eight V100 GPUs for FFHQ with 512 × 512. For FFHQ 1024 × 1024, we use 48 patch size to represent an image as 484 data tokens and L = 2 level of feature modulation with ( σ 1 , σ 2 , σ q ) = (256 , 64 , 32) . The training of 400 epochs takes about 3.4 days using eight NVIDIA V100 GPUs.

A.2 Novel View Synthesis

We train our framework for the task of novel view synthesis on ShapeNet Cars, Chairs, and Lamps. Given a few known camera views as support views of a 3D object, our framework predicts a light field of the 3D object to predict unseen camera views. For a fair comparison, we use the same splits of train-valid samples with previous studies of generalizable INR [8, 19, 35]. Given rendering images of support views with 128 × 128 resolution, we first patchify each rendered image into 256 tokens with 8 × 8 size of patches. Then, we concatenate the patches of all support views with learnable tokens for the input of our Transformer. We use the Plücker coordinate to represent a ray for a pixel as an embedding with six dimensions and concatenate the ray embedding into each pixel along the channel dimension. Since our INR decoder estimates a light field of a 3D object, the INR decoder has six input channels d in = 6 for a ray coordinate and three output channels d out = 3 for a RGB pixel. Our INR decoder uses L = 2 level of feature modulation with ( σ 1 , σ 2 , σ q ) = (8 , 4 , 2) . We use d F = d = 256 dimensionality of the frequency features and hidden features of the INR decoder. We use 1000 training epochs for ShapeNet Cars and Chairs, while using 500 epochs for ShapeNet Lamps.

A.3 Diffusion Model for INR generation

We implement a diffusion model to generate the latent tokens for INRs of ImageNet 256 × 256. Different from the conventional approaches, which use a U-Net architecture to generate an image, we use a vanilla Transformer with a simple stack of self-attentions, since the latent tokens do not predefine 2D grid structure but are permutation-equivariant. The Transformer for the diffusion model has 458M parameters having 24 self-attention blocks with 1024 dimensions of embeddings and 16 heads. We remark that the locality-aware generalizable INR is not updated during the training of diffusion models. For the training of the diffusion model, we follow the formulation of DDPM [15]. The linear schedule with T = 1000 is used to randomly corrupt the latent tokens for INRs using isotropic Gaussian noises, and then we train our Transformer to denoise the latent tokens. Instead of the ϵ -parameterization that predicts the noises used for the corruption, our Transformer x 0 -parameterization to predict the original latent tokens. We drop 10% of class conditions for our model to support classifier-free guidance following the conventional setting [16]. For the stability of training, we standardize the features of latent tokens, after computing the mean and standard deviation of feature channels of each latent token based on the training data. We use eight NVIDIA A100 GPUs to train the model with 256 batch size during 400 epochs, where the training takes about 7 days. The Adam [20] optimizer with constant learning rate 0.0001 and ( β 1 , β 2 ) = (0 . 9 , 0 . 999) is used without learning rate warm-up and any weight decaying. During training, we further compute the exponential moving average (EMA) of model parameters with a decaying rate of 0.9999. During the evaluation, we use the EMA model with 250 DDIM steps [33] and 2.5 scales of classifier-free guidance [16].

B Additional Experiments

B.1 Ablation Study on the Number of Levels

Table 6 demonstrates the effect of the number of levels L on image reconstruction benchmarks of FFHQ images with 256 × 256, 512 × 512, and 1024 × 1024 resolutions. Our INR decoder uses bandwidths σ q = 16 and ( σ ℓ ) L ℓ =1 equal to (128) , (128 , 32) , (128 , 64 , 32) and (128 , 90 , 64 , 32) for L = 1 , 2 , 3 , 4 respectively in case of 256 × 256 and 512 × 512 resolution, and all bandwidths are doubled for 1024 × 1024 to leverage high-frequency details.

Note that our framework outperforms previous studies [8, 19] even with L = 1 . Moreover, the results demonstrate that increasing L improves

the performance, while the performance saturates beyond L ≥ 3 . We postulate that higher resolution requires a larger number of levels, as the performance gap between L = 3 and L = 4 decreases as the resolution increases.

B.2 Additional Examples of Novel View Synthesis

In Figure 8, we show additional examples of novel view synthesis of ShapeNet Chairs, Cars, and Lamps with one to five support views.

B.3 Additional Examples of High-resolution Image Reconstruction

Figure 9 and 10 shows image reconstruction examples of FFHQ with 256 × 256, 512 × 512, and 1024 × 1024 resolution by previous studies [8, 19] and our locality-aware generalizable INR. Unlike previous studies, our framework can successfully reconstruct fine-grained details in high resolutions.

Support

view

Predicted

view

Query

view

Support views

Predicted

view

Query

view

Support views

Predicted

view

Query

view

Support views

Predicted

view

Query

view

口

1

T

森

IPC

Ours

IPC

Ours

The difference between model predictions

after replacing a latent token with the zero vector

Reconstructed/

rendered image

B.4 Additional Examples of Conditional Image Synthesis

Figure 11 shows additional examples of generated images with 256 × 256 resolution by generating locality-aware latents of our framework.

B.5 Additional Visualization for Locality Analysis

Figure 12 visualizes which local information of data is encoded in each latent token of IPC [19] and our locality-aware generalizable INR in addition to Figure 5. We randomly select a latent token and replace it with the zero vector, then visualize the difference between the model predictions with or without the replacement as described in Section 4.3. The differences are rescaled to have the maximum value of 1 for clear visualization. Furthermore, we fix the set of replaced latent tokens for different samples in Figure 12 to emphasize the role of each latent token. Note that each latent token of our framework encodes the local information in a particular region of images or light fields, while latent tokens of IPC encode global information over whole coordinates.

B.6 Ablation Study on Linear Layers in Selective Token Aggregation

Our framework adds a linear layer in Eq (6) and Eq (7) to exploit complex frequency patterns, improving the performance. While the Fourier features consist of periodic patterns along an axis, the frequency patterns in Eq (6) can also include non-periodic patterns. Note that IPC [19] also uses a similar design, while modulating the second MLP layer to exploit complex frequency patterns. The linear layer in Eq (7) is used to process the modulation vector according to each frequency bandwidth, motivated by the design of separate projections for (query, key, value) in

self-attention. The results below also show that removing the linear layers in Eq (6) and Eq (7) significantly deteriorates the image reconstruction performance on ImageNette 178 × 178.

Peer reviews:
comments: SUMMARY:
This paper studies the problem of generalizable implicit neural representation. The proposed method combines a transformer encoder with a locality-aware decoder to predict the output with the feature modulation through multiple frequency bandwidths. Experiments are performed on image reconstruction, few-shot novel view synthesis, and image synthesis to show the effectiveness of the proposed method for generalizable implicit neural representation.

STRENGTHS:
- This paper studies an important problem in implicit neural representation and the motivation is clear.

- This paper is generally well structured and the proposed method is easy to follow.

- Experiments are sufficient to show the effectiveness of the proposed method.

WEAKNESSES:
- This paper mainly emphasis on locality-aware, but fails to provide a formal definition of the notation "locality". It would be better to provide a formal formulation and some intuitive examples.

- Many recent works tend to explore a hybrid neural representations (e.g., feature maps with small mlps), while this paper only considers the pure MLPs. It would be interesting to have a further discussion about this.

QUESTIONS:
See weakness.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGeneralizable implicit neural representation (INR) can represent multiple data instances with a coordinate-based neural network, of which weights or intermediate features are modulated using instance-wise latent codes. A significant constraint of current generalizable INR is their struggle to localize and capture fine-grained details of data entities. This limitation results in a diminished expressive power of modulation.\n\nThis research tackles this problem by presenting an innovative framework for a generalizable Implicit Neural Representation (INR). This framework merges a transformer encoder with a locality-aware INR decoder. Two key components are designed to enable to capture of the local information of data, including selective token aggregation and multi-band feature modulation.\n\nThe authors demonstrate the effectiveness of their method with state-of-the-art performances on both image reconstruction and novel view synthesis. Furthermore, they show the potential of the proposed generalizable INR on the conditional image synthesis task.\n\nSTRENGTHS:\n- **Scope and relevance**: This paper studies an important topic that enables INR with generalization ability. \n\n- **Technical contribution**: The authors present a novel locality-aware INR decoder to improve the expressive power of modulation by learning locality-aware representation from data.\n\n- **Experiments**: The authors conduct extensive experiments and demonstrate significantly improved performances on multiple benchmarks.\n\n- **Clarity**:  The paper is well-written.\n\nWEAKNESSES:\nSome technical details might need further examination.\n \n- **Selective Token Aggregation**: In Section 3.3.1, the cross-attention is applied in extracting a modulation vector $\\mathbf{m}_\\mathbf{v}$  with the coordinate $\\mathbf{v}$ and the latent tokens. Can the cross-attention be replaced with standard transformer attention with input from the concatenation of latent tokens and the frequency features of the coordinate? Can other operations that fusion these two inputs also work well?\n\n- **Multi-Band Feature Modulation**: The authors use a range of frequency bandwidths to predict the details of outputs with a deeper MLP path to learning higher-frequency features. If I am not mistaken, more learnable parameters are introduced into the coordinate-based neural network. Is it possible the improved performance stem from the increased model size?\n\nQUESTIONS:\nIn addition to the above weaknesses, here are two more questions about conditional image synthesis with the extracted latent tokens (Section 4.4 and Appendix A.3):\n\n- To realize conditional image synthesis, do the authors mean by training a diffusion model to generate latent tokens with pre-trained generalizable INR models?\n\n- In lines 522-523, \"We drop 10% of class conditions for our model to support classifier-free guidance.\" Why 10%? Does this number percentage matter?\n\nOverall, this paper is a good effect. I will raise my rating if the authors can address my concerns.\n\nLIMITATIONS:\nThe authors haven't adequately addressed limitations and the potential negative societal impact of their work.\n\nI would suggest showing some failure examples, in which the proposed model can not work well, so as to let the community know the boundary of this work.\n\nMoreover, considering the critical importance of AI safety, it could be beneficial to discuss any potential adverse societal impacts that may arise from this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper enhances the performance of generalizable INR via improving its locality awareness. The transformer encoder feed on patchs of an image and produce latent tokens with local information, and later extracted as modulation vectors. The proposed INR decoder with selective token aggregation and the multi-band feature modulation learn locality-aware representation in spatial and spectral domains respectively. Experiments show that the proposed method outperform previous methods both on reconstruction and downstream image generative tasks.\n\nSTRENGTHS:\n1. The proposed make a lot of sense to enhance the locality of INRs and via patch processing and attentions in transformers etc., and it reaches very good results compared to previous SOTAs. The method is clearly illustrated, and the results are well displayed both quantitatively and qualitatively. There are also many ablations and discussions/analyses into the details and depth. The supplementary material also provides more details, discussions and results.\n\n2. Besides the main reconstruction task, the proposed method is also able to perform a few generative tasks well, and also better than previous SOTAs.\n\n3. Figure 5 (and Figure 12 in Supp) shows interesting results on replacing latent tokens with null ones, unvealing some underlying rationales of the learned space.\n\n4. Source code is provided in the supplementary material.\n\nWEAKNESSES:\n1. [Core] An INR usually refers to a neural network that store information in its model weights instead of feature maps. The proposed modulation vector m_v seems to be more like a latent code (in an encoder-decoder framework). I hope to see clear clarification on how the parameters of the proposed INR is composed of, i.e. how much portion is feature maps and how much portion is model weights. In addition, if most (or almost all) parameters of the \"INR\" is just feature maps, then it might not be very difficult to perform those downstream generative tasks, as it is essentially more an improved VAE than an independent INR. I'm wondering how this could help the displayed performance.\n\n2. [Core] The INR (decoder or modulation vectors, i.e. the non-shared parts to independently form a single INR representing a single image) model size / compression ratio seems not to be mentioned in the paper. How big the produced INR is to representing one image of 256x256 for example? The paper has conducted a few ablations on the model structure details, including Table 4 and Table 6, while I'm wondering a more overall metric, such as number of paramters in an INR, its size in MB, or BPP (bits per pixel). How does this compared to other methods such as TransINR ordan IPC?\n\n(The above two points are my main concerns and the game-changer for my assessment.)\n\n3. Figure 2 shows the core mechanism clearly, but in addition I think a specific figure to show the produced INR itself (e.g. many layers or blocks duplicated) would make the full pipeline clearer to understand.\n\n4. Table 5 seems not to be mentioned in the main body text nor illustrated anyhow (while L271 should be referring to Figure 4 as a typo).\n\nQUESTIONS:\n1. The provided ablations are on reconstruction performance only. While this is the main task and objective, I'm also wondering how these ablations perform on other downstream applications in the paper, including novel view synthesis and image generation. Does any options or hyperparameters make a difference for these applications? Or do you find you don't have to carefully pay attention to them in these applications in your experiments?\n\nLIMITATIONS:\n1. More discussions and analyses on the learned INR model weight space would be welcomed and inspiring. How do you think the locality awareness may improve the INR space distance smoothness? For example, in LDM they applied KL loss or VQ regularization on VAE (although not big) to help the distance in the latent space (feature map) be consistent with that in the pixel space. Do you think this might be one of the point that the proposed method reaches better results in generative tasks including novel view synthesis and image generation?\n\n2. IPC also conduct experiments on downstream classification tasks. It would be great to see similar results on your proposed method and comparisons with them. (Or do you think there might be any difference in design making that your method is better at and more fitful for generative tasks than classifications? Any discussion is welcomed.)", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper tries to improve the expressive power of neural implicit function modulation by enhancing its ability to localize and capture fine-grained details of data samples. A novel framework for generalizable INR is proposed that combines a transformer encoder with a locality-aware INR decoder. It further utilizes selective token aggregation and the multi-band feature modulation to learn locality aware representation in both spatial and spectral aspects.

STRENGTHS:
+ combine locality aware transformer encoder with global neural implicit function to improve the representation quality of local details
+ use selective token aggregation for spatial locality
+ use multi-band feature modulation for spectral locality

WEAKNESSES:
- All the quantitative results are reported with PSNR metric. More evaluation metrics emphasizing local details should be used.
- Limited results reported on novel view synthesis and conditional image generation.
- In conditional generation, the latents are corrupted by Gaussian noise and denoised with a trained diffusion model. If the Gaussian noise is small, the task becomes trivial. An comparison experiment should be done for directly reconstructing with corrupted latents using the implicit neural function.

QUESTIONS:
Can we interpolate between the latents of two data samples to generate a new sample?

LIMITATIONS:
Experiment results are limited in evaluation scale and in-depth analysis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

comments: SUMMARY:
This paper aims to enhance the performance of generalizable implicit neural representation locality-aware model designs. A transformer encoder is applied to convert image patches into latent tokens, which the proposed locality-aware decoder composed of the selective token aggregation and the multi-band feature modulation is based on to predict outputs. Experiments on tasks including image reconstruction and novel view synthesis are performed to demonstrate the proposed method's state-of-the-art performance.

STRENGTHS:
+ The proposed locality-aware decoder containing selective token aggregation and multi-band feature modulation is novel and effective.
+ Extensive experiments show that the proposed method achieves good performance.
+ The paper is clear and easy to follow.

WEAKNESSES:
- It is not very straightforward to get the idea of  "Locality" in this work. Any empirical evidence like visualization in Selective Token Aggregation to reveal how the locality is enhanced?
- Analysis and comparison of model/runtime efficiency are missing.
- Some important implementation details are missing, e.g. what's the value of L and what's the impact of this hyperparameter?

QUESTIONS:
see weakness.

LIMITATIONS:
- No analysis for failure cases.
- Qualitative results in the paper are limited, and should be more diverse.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the task of training a single coordinate-based neural network to represent multiple scenes or instances. There are two main technical contributions that improve the quality of these generalized representations: (1) a Transformer-based encoder that extracts localized features of each target instance (e.g. image or scene), and adaptively weights these localized features during inference at different coordinate locations, (2) the coordinate-based neural network operates coarse-to-fine in the frequency domain, taking in high-frequency instance-specific features at earlier layers and lower-frequency instance-specific features at later layers, so that higher-frequency features are processed by a deeper network. The paper includes compelling results on medium and high resolution image datasets, and promising preliminary results on few-shot novel view synthesis. Ablation studies show that of the two technical contributions, neither alone yields improvement but both together do, compared to the main baseline Instance Pattern Composers.\n\nSTRENGTHS:\nThese two technical ideas make sense, particularly the idea of having localized features, and the results on image datasets are very compelling—though some important omitted details make these difficult to fully interpret, as does my own lack of familiarity with the baseline methods. Ablation studies and more in-depth analysis of learned features (figure 5) are also interesting.\n\nWEAKNESSES:\nThe paper writing/presentation could be substantially improved. The first roughly 2 pages of the paper are laden with jargon (and some typos/grammatical issues), making the actual new ideas in the paper difficult to tease out. As a reader who has worked in implicit neural representations but not generalizable ones, many parts of the paper required some assumptions or terms that might be common to those who work in generalizable INRs but unfamiliar to researchers in even a very adjacent area. Some examples: what exactly is a latent (ie what is the input used to produce a latent)? What is modulation (in signal processing this would be element-wise multiplication)? The methods section does largely (though not fully) clarify the method and the new ideas, but leaves me wondering about the motivation behind some of the design decisions that are stated without much explanation. I list these remaining questions in the “questions” section of the review, along with some questions about the experimental setup of the results.\n\nOne separate comment/suggestion is about the clarity of the figures. Figure 1 shows that the proposed method trains a lot faster/better than two baselines, but doesn’t explain what the task is (just the dataset). Figure 2 gives a helpful overview, but doesn’t actually explain the two core new ideas of the paper. For example, from the figure and caption I can’t tell what is different about the two yellow blocks (one is for high-frequency features and one for low-frequency features, but I didn’t find this out until I read 2 pages past the figure). Some terms in the figure are also not defined; for example I assume that “FC” means fully-connected, but I’m not sure how this differs from “Linear”. Figure 3 shows a compelling comparison to prior work, but I would encourage the authors to include the ground truth image for full comparison.\n\nQUESTIONS:\n- I am still not fully sure what exactly is the input to the model during inference. Is it basically an autoencoder, taking an image as input and compressing it into latents and then decoding them into the image again? Likewise for the generative experiment (figure 6), how are these latents generated?\n- Equation 2 describes the Fourier featurization; it appears to use the same featurization as the original NeRF paper (with axis-aligned frequencies) rather than e.g. a random Gaussian (not axis-aligned) set of frequency vectors as in the “Fourier Features Let Networks Learn…” paper, though both of these papers are cited. I wonder why the authors chose to use the original Fourier featurization rather than the newer version?\n- Equation 5 describes multi-head attention as the mechanism for aggregating/weighting the localized latent features when performing inference on a given coordinate vector. I wonder if the authors considered a more straightforward kernel function (e.g. Gaussian kernel based on Euclidean distance, or even a learned kernel) or what the motivation was for using multi-head attention.\n- Equations 6 and 7 describe a process for separating the modulation vector into different frequency bands, by using a shallow MLP with inputs based on different Fourier features (in the desired frequency band). I wonder why the MLP is necessary here, compared to just taking the subset of Fourier features that are in each frequency band?\n- Tables 1 and 2 show compelling numerical comparisons vs Learned Init, TransINR, and IPC. The accompanying text notes that the capacity of the encoder, latent tokens, and decoder are matched among all methods, except for the modulation methods. So I wonder how the capacity of the modulation methods compares?\n- I also wonder what the total training time is on each dataset, compared to prior work.\n- Figure 7 shows that test-time optimization of all parameters outperforms per-sample optimization. How is this possible? I wonder if training longer would allow the per-sample FFNet to “catch up”, or if the FFNet here has lower capacity, or if there is some other explanation for how a generalized model could outperform a more specialized model. Maybe the generalized model is starting from a better initialization and thus reaches a better optimum?\n\nLIMITATIONS:\nLimitations are not really discussed, beyond mentioning some directions for future work--but for this topic I believe this is appropriate. I do not foresee direct negative societal impact from this work, though like all image processing research it has the potential to be misused e.g. towards surveillance or other harmful ends.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the important problem of bridging the gap between generalizable implicit neural representations (INR) and per-sampled trained ones. The core hypothesis is that previous generalizable INRs failed to capture local details in the global latent code due to their inductive bias. The idea of the proposed method is to equip learning such INRs with a Transformer encoder, which selects and encodes local information into multiple tokens. The paper elaborates on all parts of the pipeline and presents the empirical study on image reconstruction and novel view synthesis with shapenet. Overall, the proposed method shows a great advantage over the prior art. Ablation studies are thorough.\n\nSTRENGTHS:\nThe idea behind the method is novel and interesting. The paper borrows the best ideas from other domains, including transformers and nerf. Experimental studies on FFHQ and Shapenet are convincing. The paper is well-written and rather polished.\n\nWEAKNESSES:\nThe introduction is a bit too broad and needs more specificities about the proposed method. The statements from the title, abstract, and intro, set expectations for something more generic than what can be processed by a transformer encoder.\n\nThe frequency decomposition mechanism is not very well explained. Particularly, the choice of two blocks for frequency decomposition is never explained nor ablated. Would the performance keep growing if more of those blocks had a different frequency coverage scheme? \n\nThe locality property of the produced tokens is also not very clear. On the one hand, the authors claim that the tokens are locality aware; on the other that thanks to permutation equivariance of the attention mechanism in the transformer encoder, the tokens form an unordered set. It would be better to explain these aspects more carefully (including in the rebuttal).\n\nList of typos\n- L49: outperformance\n\nQUESTIONS:\nIn Fig. 2, how do the transformer inputs form in each considered setting (image / NVS)? \nDoes the framework require retraining everything from scratch for every new resolution? \nCan any of the pretrained backbones be used?\nCan the authors think of a way to visualize the effect they discuss regarding ordering frequencies based on the depth of the layer in the decoder?\nWhat do the results look like for the best possible setting for 1Kx1K resolution in Table 2?\n\nLIMITATIONS:\nDiscussed adequately in the conclusion. However, it would be interesting to learn about dealing with arbitrary resolutions, or other forms of inputs not directly amenable to transformer encoders.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}