Locality-Aware Generalizable Implicit Neural Representation
Authors: Doyup Lee
Chiheon Kim
Minsu Cho
Wook-Shin Han
Venue: NeurIPS 2023 poster (2023)
Keywords: implicit neural representations
representation learning
neural fields

Abstract: Generalizable implicit neural representation (INR) enables a single continuous function, i.e., a coordinate-based neural network, to represent multiple data instances by modulating its weights or intermediate features using latent codes. However, the expressive power of the state-of-the-art modulation is limited due to its inability to localize and capture fine-grained details of data entities such as specific pixels and rays. To address this issue, we propose a novel framework for generalizable INR that combines a transformer encoder with a locality-aware INR decoder. The transformer encoder predicts a set of latent tokens from a data instance to encode local information into each latent token. The locality-aware INR decoder extracts a modulation vector by selectively aggregating the latent tokens via cross-attention for a coordinate input and then predicts the output by progressively decoding with coarse-to-fine modulation through multiple frequency bandwidths. The selective token aggregation and the multi-band feature modulation enable us to learn locality-aware representation in spatial and spectral aspects, respectively. Our framework significantly outperforms previous generalizable INRs and validates the usefulness of the locality-aware latents for downstream tasks such as image generation.

Locality-Aware
Generalizable Implicit Neural Representation
Doyup Lee" Kakao Brain
Chiheon Kim
Kakao Brain
doyup wee.kakabbaain.cmmmm com
chiheon www.kakaobbaain.ccooo
Minsu Cho POSTECH mscho@postech..o ac.kr
Wook-Shin Han POSTECH
wshan@dblab. www.mndddbab..psttchh.ooo ac .c kr
Abstract
Generalizable implicit neural representation (INR) enables a single continuous func- tion, i.c., coordinate- based neural network, to represent multiple data instances by modulating its weights or intermediate features using latent codes. However, the expressive power of the state-of-the-arr modulation is limited due to its inability to localize and capture fine-gu mne-grained details data entities such as specific pixels and rays. To address this issue, propose a novel framework for generalizable INR that combines a transformer encoder with a localit-aaware INR decoder. The transformer encoder predicts set of latent tokens from data instance encode local information into each latent token. The locality-aware INR decoder extracts a modulation vector by selectively aggregating the latent tokens via cross-attention for coordinate input and then predicts the output by progressively decoding with coarse-to to-se-tt-inne modulation through multiple frequency bandwidths. The selective token aggregation and the multi-band feature modulation enable to learn locality- aware representation in spatial and spectral aspects, respectively. Our framework significantly outperforms previous generalizable INRs and validates the usefulness locality-aware latents for downstream tasks such as image generation.
Introduction
Recent advances in generalizable implicit neural represen- tation (INR) enable a single coordinate-based multi-layer perceptron (MLP) to represent multiple data instances continuous function. Instead of per-sample training of indi- vidual coordinate-bases MLPs, generalizable INR extracts
45
Test PSNR (dB)
35
latent codes of data instances 13
14
40 to modulate
the weights intermediate features of the shared MLP
TransiNR
model 8 previous approaches, their performance is still insufficient compared with individual training of INRs per sample.
335. However, despite the advances
25
IPC
20
Ours
1000
2000
3000
4000
We postulate that the expressive power of generalizable INRs limited by the ability of locality-awarenes, to localize relevant entities from data instance and control
Epochs
Figure 1: Learning curves of PSNRs dur-
ing training on ImageNette 178x 178.
"Equal contribution 'Corresponing authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

their structure in a fine- grained manner. Primitive entities of a data instance, such as pixels in an image, tend to have a higher correlation with each other if they are closer in space and time. Thus, this locality of data entities has been used as an important inductive bias for learning the representations of complex data [3] However, previous approaches to generalizable INRs are not properly designed to leverage the locality of data entities. For example, when latent codes modulate intermediate features [1 [1] or weight matrices [8 35] of an INR decoder, the modulation methods do not exploit specified coordinates for decoding, which restricts the latent codes to encoding global information over all pixels without capturing local relationships between specific pixels.
To address this issue, we propose novel encoder-decoder framework for locality generalizable INR to effectively localize and control the fine-grain fine-grained details of data. In our framework, a Trans- former encoder (37] first extracts locally relevant information from data instance and predicts set of latent tokens to encode different local information. Then, our locality-aware INR decoder effectively leverages the latent tokens to predict fine-grained details. Specifically, given input coordinate, our INR decoder uses a cross-attention to selectively aggregate the local information in the latent tokens and extract a modulation vector for the coordinate. In addition, our INR decoder effectively captures the high-frequency details in the modulation vector by decomposing it into multiple bandwidths of frequency features and then progressively composing the intermediate features. We conduct extensive experiments to demonstrate the high performance and efficacy of our locality-aware generalizable INR benchmarks as shown in Figure I In addition, we show the potential of our locality- INR latents to be utilized for downstream tasks such as image synthesis.
Our main contributions can be summarized as follows: 1) We propose an effective framework for generalizable INR with a Transformer encoder and locality-aware INR decoder. 2) The proposed INR decoder with selective token aggregation and multi-band feature modulation can effectively capture the local information to predict the fine-grained data details. 3) The extensive experiments validate the efficacy of our framework and show its applications to downstream image generation task.
2 Related Work
Implicit neural representations (INRs). INRs use neural networks to represent complex data such as audio, images, and 3D scenes, as continuous functions. Especially, incorporating Fourier features [24. 36), periodic activations [31], or multi-grid features (25) significantly improves the
performance of INRs. Despite its broad applications (1 6 10 32 34), INRs commonly require separate training of MLPs to represent each data instance. Thus, individual training of INRs per sample does not learn common representations in multiple data instances.
Generalizable INRs. Previous approaches focus on two major components for generalizable INRs; latent feature extraction and modulation methods. Auto-dccoding 23 26 computes a latent vector MLP. Given input data, gradient-based meta-learning [ [2] adapts a shared latent vector using a few update steps to scale and shift the intermediate activations of the MLP. Learned Init [351 also uses gradient- based meta-learning but adapts whole weights of the shared MLP. Although auto-decoding and gradient- based meta-learning are agnostic to the types of data, their training is unstable on complex and large-scale datasets. TransINR [ employs the Transformer (37] as a hypernetwork to predict latent vectors to modulate the weights of the shared MLP. In addition, Instance Pattern Composers [19] have demonstrated that modulating the weights of the second MLP layer is enough to achieve high performance of generalizable INRs. Our framework also employs the Transformer encoder, but focuses on extracting locality-aware latent features for the high performance of generalizable INR.
per data instance and concatenates it with the input of a coordinate-basse
Leveraging Locality of Data for INRs Local information in data has been utilized for efficient modeling INRs, since local relationships between data entities are widely used for effective process of complex data 33. Given an input coordinate, the coordinate-basse
MLP only uses latent vectors nearby the coordinate, after a CNN encoder extracts a 2D grid feature map of image for super-resolution 7 and reconstruction 22). Spatial Functa 4 demonstrates that leveraging the locality of data enables INRs to be utilized for downstream tasks such as image recognition and generation. Local information in 3D coordinates has also been effective for scene modeling hybrid approach using 3D feature grids [18] or the part segmentation [17 of 3D object. However, previous approaches assume explicit grid structures of latents tailored to specific data type. Since

---

Locality-Aware INR Decoder
coordinate input
multi-band modulation
Localized Latents
coordinate input
Fourter-Lnnaar
Fourler-Linear
Query
modulation vector
Linear
Transformer Encoder with Self-Attentions
Selective
Aggregation
ma
coordinate input
LinearRRoLU
(Croes-Atteniion)
Fourier-Lnnaar
Key
Volue
Linear
Linear-ReLU
Learnable Tokens
element-wii addition
image/vises patches
Localized Latents
ReLU activation
output
Figure 2: Overview of our framework for locality-aware generalizable INR. Given data instance, Transformer encoder extracts its localized latents. Then, locality-aware INR decoder uses selective token aggregation and multi-band feature modulation to predict the output for the input coordinate.
we do not predefine a relationship between latents, our framework flexible to learn and encode the local information of both grid coordinates in images and non-grid coordinates in light fields.
3 Methods
We propose a novel framework for locality-aware generalizable INR which consists Transformer encoder to localize the information in data into latent tokens and a locality-aware aware INR decoder to exploit the localized latents and predict outputs. First, we formulate how generalizable INR enables continuous function
single coordinate-baat -baee neural network to represent multiple data instances by modulating its weights or features. Then, after we introduce the Transformer encoder extract set of latent tokens from input data instances, explain the details of the locality-awar INR decoder, where selective token selection aggregates the spatially local information for an input coordinate via cross-attention; multi-bante feature modulation leverages a different range of frequency bandwidths to progressively decode the local information using coarse-to- caase-to-ffie modulation in the spectral domain.
3.1 Generalizable Implicit Neural Representation
Given a set of data instances = {x(x)) J- each data instance x(") = ((v " ) comprises M, pairs of an input coordinate (n) Rd and the corresponding output feature y IR.... Conventional approaches [24] 31, 36) adopt individual foordinate-basse MLPs to train and memorize each data instance x("). Thus, the coordinate-base MLP cannot be reused and generalized to represent other data instances, requiring per- per-sample optimization of MLPs for unseen data instances.
generalizable INR uses single coordinate- based MLP as a shared INR decoder Fg Rd dmn
to represent multiple data instances as a continuous function. Generalizable INR [8]
19 26
extracts the R number latent codes Z (n)
{z}
)= from data instance ("). Then,
the latents used for the INR decoder to represent data instance (") y, (n)) Foly; (n),
Zz (n)).
while updating the parameters 9 and latents (n) to minimize the errors over X::
N
Ma
(m) y
(v
Z (()))
NM,
20
We remark that each previous approach employs a different number of latent codes to modulate a coordinate-based MLP. For example, a single latent vector (R 1) commonly extracted to modulate intermediate features of the MLP [1] 12 26 while multitude of latents (R > 1) are used modulate weights [8] 19 35 While we modulate the features of MIP, we extract a set of latent codes to localize the information of data to leverage the locality-awarenee for latent features.

---

3.2 Transformer Encoder
Our framework employs Transformer encoder 37 to extract a set of latents ( for each data instance x(n) as shown in Figure 2 After data instance, such as image multi-view images, is patchified into a sequence of data tokens, we concatenate the patchified tokens into a sequence of R learnable tokens as the encoder input. Then, the Transformer encoder extracts set of latent tokens, where each latent token corresponds to an input learnable token. Note that the permutation- equivariance of self-attention in the Transformer encoder enables us not to predefine the local structure of data and the ordering of latent tokens. During training, each latent token learns to capture the local information of data, while covering whole regions represent a data instance. Thus, whether data instance is represented on a grid or non- -in-grid coordinate, our framework is flexible to encode various types of data into latent tokens, while learning the local relationships of latent tokens during training.
3.3 Locality-Aware Decoder for Implicit Neural Representations
We propose the locality-aware INR decoder in Figure to leverage the local information data for effective generalizable INR. Our INR decoder comprises two primary components: i) Selective token aggregation via cross attention extracts a modulation vector for an input coordinate to aggregate spatially local information from latent tokens. ii) Multi-band feature modulation decomposes the modulation vector into multiple bandwidths of frequency features to amplify the high-frequency features and effectively predict the details of outputs.
3.3.1 Selective Token Aggregation via Cross-Attention
We remark that encoding locality-awar latent tokens is not straightforward since the self-attentions in Transformer do not guarantee a specific relationship between tokens. Thus, the properties of the latent tokens are determined by a modulation method for generalizable INR to exploit the extracted latents. For example, given an input coordinate v and latent tokens {zı, ZR), }. straightforward method can use Instance Pattern Composers and extract a modulation vector my W m [z]] v, ..., 2 v] E RR However, the latent tokens cannot encode the local information of data, since each latent token equally influences each channel of the modulation vector regardless of the coordinate locations (see Section 4.3).
[99 to construct modulation weight W... [æ1, .... z R Rxd,
Our selective token aggregation employs cross-attention to aggregate the spatially local latents nearby the input coordinate, while guiding the latents to be locality-aware. Given a set of latent tokens Z (n) = {z
(x) (x) R =k=1 and a coordinate v a modulation feature vector mv, E Rd shifts the intermediate features of an INR decoder to predict the output, where d is the dimensionality of hidden layers in the INR decoder. For the brevity of notation, we omit the superscript and subscript i.
Frequeney features We first transform an input coordinate v = (V), Ud,)) E Rd into fre- quency features using sinusoidal positional encoding 3I 36 We define the Fourier features To (v) E dede with bandwidth a > 1 and feature dimensionality dF as
Ta (cos(mwy (x),, sin (mw))))) i === I, ddn 0, n 1]
where n
A frequency wj (5((n1) is evenly distributed between and a on log-scale. Based on the Fourier features, we define the frequency feature extraction by (-) as
2dh
(v; b) = ReLU (V) + bb),
where W E ddxde and b E Idd are trainable parameters for frequency features, d denotes the dimensionality of hidden layers in the INR decoder.
Selective token selection via cross-attention To predict corresponding output y to the coordinate v, we adopt a cross-attention to extract modulation feature vector m, E Rd based on the latent tokens Z {z,} R We first extract the frequency features of the coordinate v in Eq B as the
query of the cross- attenttttnniinn as
qv == hp Wq.
where Wg E Radde and by E Rd are trainable parameters, and ag is the bandwidth for query frequency features. The cross-attention in Figure 2 enables the query to select latent tokens, aggregate

---

its local information, and extract the modulation feature vector my for the input coordinate:
my MuliiHeaaAAtteetionnnnn
qv, Key
Z, Value Z).
(5)
An intuitive implementation for selective token aggregation can employ hard attention select only one latent token for each coordinate. However. in our primitive experiment, using hard attention leads unstable training and latent collapse problem that selects only few latent tokens. Meanwhile, multi-head attentions encourage each latent token to easily learn the locality in data instances.
3.3.2 Multi-Band Feature Modulation in the Spectral Domain
After the selective token aggregation extracts modulation vector my, we use multi-band feature modulation to effectively predict the details of outputs, Although Fourier features 24 36 reduce the spectral bias 2 28 neural networks, adopting simple stack of MLPs to INRs still suffers from capturing the frehffreyencc data details. To address this issue, we use different range of frequency bandwidths decompose the modulation vector into multiple frequency features the spectral domain. Then, our multi-band feature modulation uses the multiple frequency features to progressively decode the intermediate features, while encouraging a deeper MI.P path to learn higher frequency features. Note that the coarse-to-fine approach in the spectral domain is analogous to the locally hierarchical approach in the spatial domain 21 29 39 to capture the data details.
Extracting multiple modulation features with different frequency bandwidths We extract level of modulation features m .... my (L) from my using different bandwidths of frequency
features. Given frequency bandwidths as σ 2 02 2 ... 2 as. 2 og. we use Eq (3 to extract the
(-th level of frequency features of an input coordinate
(hr) = he(v; de. W! b(() ReLU (w Yo, (v) +
(6)
where w and ( are trainable parameters and shared across data instances. Then, the l-th modulation vector m is extracted from the modulation vector my as
( ReLU ((h=) + w m. + b(())
with a trainable weight ( and bias bm assume that m
Considering that ReLU cutoffs the values below zero, we (2) filters out the information of m, based on the ε-th frequency patterns of (hp) (e)
Multi-band feature modulation After decomposing modulation vector into multiple features with different frequency bandwidths, we progressively compose the L modulation features by applying a stack of nonlinear operations with a linear layer and ReI.U activation. Starting with h =m/) we compute the L-th hidden features
h( for l 2, 2...... as
m(m) + h h(e-1) and h(C == ReLU(W(OH() + bb()).
(8)
where ((O) E dxd and (() E R are trainable weights and biases of the INR decoder. h denotes the ε-th pre-activation of INR decoder for coordinate Note that the modulation features with high-frequency bandwidth can be processed by more nonlinear operations than the features with lower frequency bandwidths, considering that high-frequency features contain more complex signals.
Finally, the output ý predicted using all intermediate hidden features of the INR decoder as
(b(?),
where (_(() Rdd -> B are linear projection into the output space. Although utilizing only h(() is also an option to predict outputs, skip connections of intermediate features into the output layer enhances the robustness of training to the hyperparameter choices.
5

---

Figure 3: Reconstructed images of FFHQ with 512x512 resolution by TransINR (left), IPC (middle), and our locality-aware generalizable INR (right).
19
Experiments
We conduct extensive experiments to demonstrate the effectiveness of our locality- generalizable INR on image reconstruction and novel view synthesis. addition, we conduct in-depth analysis to validate the efficacy of our selective token aggregation and multi-band feature modulation to localize the information of data to capture fined-grained details. We also show that our locality-aware latents can be utilized for image generation by training generative model the extracted latents. Our implementation and experimental settings are based on the official codes of Instance Pattern Composers [19] for fair comparison. We attach the implementation details Appendix
4.1 Image Reconstruction
We follow the protocols in previous studies 19 35 to evaluate our framework on image reconstruc-
tion of CelebA, FFHQ, and ImageNette with 178 x 178 resolution. Our framework also outperforms previous approaches on high-resolution images with 256 256, 512x and 1024 1024 resolutions of FFHQ. We compare our framework with Learned Init [35], TransINR [8). and IPC [19]. The Transformer encoder predicts R = 256 latent tokens, while the INR decoder uses dn = 2, dout = 3, d 256 dimensionality of hidden features, L = 2, 0g 16 and (01, (128, 32) bandwidths.
178x178 Image Reconstruction Table shows that generalizable INR significantly outperforms previous methods by large mar- gin. We remark that TransINR, IPC, and our framework use the same capacity of the Trans- former encoder, latent tokens, and INR de- coder except for the modulation methods. Thus, the results imply that locality-aware INR decoder with selective token aggregation and multi-band feature modulation effective to capture local information of data and fine-grained details for high-quality image reconstruction.
Table I: PSNRs of reconstructed images of 178x 178 CelebA, FFHQ and ImageNette.
CelebA
FFHQ
ImageNette
Leamed Init 35
30.37
27.07
TransINR IPC
33.33
33.66
29.77
35.93
37.18
38.46
Ours
50.74
43.32
46.10
Rigo-Resn Image Reconstruction We further evaluate our framework on the re- construction of FFHQ images with 256 256. 512x512, 1024x 1024 resolutions to demon- strate our effectiveness capture fine-grained data details in Table 2 Although the perfor- mance increases as MLP dimensionality and the number of latents R increases, we use same experimental setting with 178x 178 im- age reconstruction to validate the efficacy of our framework. Our framework consistently achieves higher PSNRs than TransINR and IPC for all resolutions. Figure Jllso shows that Trans and IPC cannot reconstruct the fine-grain grained details of x12x512 image, but our framework provides
Table PSNRs on the reconstructed FFHQ with
256 256, 512x and 1024x 1024 resolutions.
256x256
512x512
1024x1024
TransINR
30.96
29.35
IPC [99
34,68
31.58
28.68
Ours
39.88
35.43
31,94

---

Support
Quary
Predicted view
view
25
24
NR
23
22
21
20
(Chains
(Chains)
-A. IPC (Cars)
Ours [Cars]
IPC (Lamps)
Ours [Lamps)
view
views
3 views
views
5 views
Figure 4: (a) PSNRs on novel view synthesis of ShapeNet Chairs, Cars, and Lamps according to the number of support views (1-5 views). (b) Examples of novel view synthesis with 4 support views.
high-quality result of reconstructed images. The results demonstrate that leveraging the locality of data is crucial for generalizable INR to model complex and high-resolution data.
4.2 Few-Shot Novel View Synthesis
We evaluate our framework on novel view synthesis with the ShapeNet Chairs, Cars, and Lamps datasets to predict a rendered image of a 3D object under an unseen view. Given few views of an object with known camera poses, we employ light field [32 for novel view synthesis. A light field does not use computationally intensive volume rendering 24 but directly predicts RGB colors for the input coordinate for rays with dim 6 using the Plücker coordinate system. Our INR decoder uses d 256 and two levels of feature modulations with og and (01, 02) = (8, 4).
Figure a shows that our framework outperforms IPC for novel view synthesis. Our framework shows competitive performance with IPC when only one support view is provided. However, the performance of our framework is consistently improved as the number of support views increases, while outperforming the results of IPC. Note that defining a local relationship between rays is not straightforward due to its non- grid property of the Plücker coordinate. Our Transformer encoder can learn the local relationship between rays to extract locality-aware latent tokens during training and achieve high performance. We analyze the learned locality of rays encoded in the extracted latents in Section 4.3 Figure (b)) shows that our framework correctly predicts the colors and shapes of a novel view corresponding to the support views, although the predicted views are blurry due to the lack of training objectives with generative modeling. We expect that combining our framework with generative models 55 38 to synthesize photorealistic novel view is an interesting future work.
4.3 In-Depth Analysis
Learning Curves ImageNette 178x 178 Figure juxtaposes the learning curves of frame- work and previous approaches on ImageNette 178x 178. Note that TransINR. IPC, and our framework use the same Transformer encoder extract data latents, while adopting different modulation meth- ods. While the training speed of our framework is about 80% of the speed of IPC, we remark our framework achieves the PSNR of 72 after 400 epochs of training, outperforming the PSNR of 38.46 achieved by IPC trained for 4000 epochs, hence resulting in 8x speed-up of training time. That is, our locality-aware latents enables generalizable INR be both efficient and effective.
Selective token aggregation and multi-band feature modulations We conduct an ablation study on image reconstruction of with ImageNette 178x 178 and FFHQ 256 x56,, novel view synthe- sis with Lamp-3 views to validate the effectiveness of the selective token aggregation and the multi- band feature modulation. We replace the multi- band feature modulations with a simple stack of MIPs (ours w/o multiFM), and the selective to- ken aggregation with the weight modulation of IPC (ours w/o STA). If both two modules are replaced together, the INR decoder becomes the same
Table 3: Ablation study on ImageNette
178 178, FFHQ 256x256, and Lamp-3 views.
ImageNette
FFHQ
Lamp
Ours w/o STA w/o multiFM
37.46
38.01
26.00
34,54
34.52
25.31
33.90
33.65
25.78
IPC 19
34.11
34.68
25.09
7

---

Reconstructed' rendered image
The difference between model predictions after replacing a latent token with zero vector
IPC
Ours
IPC
Figure 5: Visualization of differences between model predictions after replacing latent token with the zero vector, for IPC 19 and our framework.
architectrure as IPC. We use single-head cross-attention for the selective token aggregation to focus on the effect of two modules. Table ( demonstrates that both the selective token aggregation and the multi-band feature modulation are required for the performance improvement, as there is no significant improvement when only one of the modules is used.
Choices of frequency bandwidths Table 4 shows that the ordering of frequency bandwidths in Eq. 4 and Eq. 6 can
affect the performance. We train our framework with two- level feature modulations on ImageNette 178x 178 during 400 epochs with different settings of the bandwidths 01, 22 Q... Although our framework outperforms IPC regardless of the bandwidth settings, the best PSNR is achieved with 0 2 02 2 04- The results imply that selective token aggregation does not require high-frequency features, but the nigh-frequency features need to be processed by more nonlinear operations than power-frequency features as discussed in Section 3.3.2
Table 4: PSNRs of reconstructed Im- ageNette 178 178 with various fre- quency bandwidths.
(a1, 02)
Fg
ImageNette
(128, 32)
16
37.46
(32, 128)
35.00
(128, 128)
16
35,30
(128, 32)
128
35.58
IPC (a 128)
34.11
The role of extracted latent tokens Figure 5 shows that framework encodes the local infor- mation of data into each latent token, while IPC cannot learn the locality in data coordinates. To visualize the information in each latent token, we randomly select latent token to be replaced with the zero vector. Then, we visualize the difference between the model predictions with or without the replacement. Each latent token of framework encapsulates the local information in different regions of images and light fields. However, the latent tokens of IPC cannot exploit the local informa- tion of data, while encoding the global information over whole coordinates. Note that our framework learns the structure of locality in light fields during training, although the structure of the Plücker coordinate system is not regular as the grid coordinates of images. Thus, our framework can learn the locality-awar latents of data for generalizable INR regardless of the types of coordinate systems.
4.4 Generating INRs for Conditional Image Synthesis
We examine the potentials of the extracted latent tokens to be utilized for downstream task such as class-conditional image generation of ImageNet [9]. Note that we cannot use the architecture of U-Net in conventional image diffusion models [ [30,, since framework not tailored to the 2D grid coordinate. Thus, we adopt Transformer-basee
diffusion model [15] 27 to predict set of latent tokens after corrupting the latents by Gaussian noises.

---

Figure 6: The examples of generated 256 256 images by generating latents of IPC (left) and ours (right), trained on ImageNet.
We train 458M parameters of Transformers during 400 epochs to generate our locality-aware latent to- kens. We attach the detailed setting Appendix [A.3 When we train diffusion model to generate latent tokens of IPC in F the generated images suffer from severe artifacts, because the prediction error of each latent token for IPC leads to the artifacts over all coordinates. Contrastively, the diffusion model for our locality-aware latents generates realistic images.
Table 5: Reconstructed PSNRs and FID of
generated images ImageNet 256x
Latent Shape
rPSNR
FID
Ours
256x256
37.7
9.3
Spatial
16x 16x266
37.2
11.7
Functa
32x32x64
37.7
8.8
T.DM 33
64x64x3
27.4
3.6
In addition, although do not conduct exhaustive hyperparamter search, the FID score of generated images achieves 9.3 with classifier- free guidance scale [16] in Table[5] Thus, the results validate the potential applications of the local latents for INRs. Mcanwhile, a few generated images may exhibit checkerboard artifacts, particularly in simple backgrounds, but we leave the elaboration of diffusion process and sampling techniques for generating INR latents future work.
4.5 Comparison with Overfitted INRs
Figure shows that our generalizable INR efficiently pro- vides meaningful INRs compared with individual training of INRs per sample. To evaluate the efficiency of our framework, we select ten images of FFHQ 256x251 and train randomly initialized FFNet (36) per sample using one NVIDIA V 100 GPU. The individual training of FFNets requires over 10 seconds of optimization to achieve the same PSNRs of our framework, where inference time negligible. Moreover, when we apply the test-time optimization (TTO) only for the extracted latents, it con- sistently outperforms per-sample FFNets for 30 seconds while maintaining the structure of latents. When sider the predicted INR as initialization and finetune all parameters of the INR decoder per each sample, our frame- work consistently outperforms the per-sampling training of INRs from random initialization. Thus, the results imply that leveraging generalizable INR is computationally efficient to model unseen data as INRs regardless of TTO.
50
45
40
PSNR
35
Ours (TTO of latents)
parameters)
Net (per-sample optim.)
10 15
20
25
Avg. (sec/ sample)
Figure 7: Comparison with individually trained FFNets (36) per sample.
Conclusion
We have proposed an effective framework for generalizable INR with the Transformer encoder and locality- INR decoder. The Transformer encoder capture the locality of data entities and learn to encode the local information into different latent tokens. Our INR decoder selectively aggregates the locality-aware latent tokens to extract modulation vector for coordinate input and exploits the multiple bandwidths of frequency features to effectively predict the fine- -reigea data details. Experimental results demonstrate that framework significantly outperforms previous generalizable

---

INRs on image reconstruction and few-shot novel view synthesis. In addition, we have conducted the in-depth analysis to validate the effectiveness of our framework and shown that our locality-awarc latent tokens for INRs can be utilized for downstream tasks such as image generation to provide realistic images. Considering that our framework can learn the locality non-grid coordinates, such as the Plücker coordinate for rays, leveraging our generalizable INR to generate 3D objects or scenes is a worth exploration. In addition, extending our framework to support arbitrary resolution will be an interesting future work. Furthermore, since our framework has still room for performance improvement of high-resolution image reconstruction, such as 1024 x 1024, we expect that elaborating on the architecture and techniques for diffusion models to effectively generate INRs is interesting future work.
Acknowledgements
This work supported by Institute of Information & communications Technology Planning & Evaluation grant funded by the Korea (MVermment(MMST)) (No.2018-0-0132))) Development of Conversational, Self-tuning DBMS, 35%; No. modal Lifelong Learning, 30%) and the National Research Foundation of Korea(NRF) grant funded by the Korea (MV)) (No. NRF-202 220-2021R1A228 14B-22221R1A2255800015511111 35%)
Sustainable Collaborative Multi-
References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Bartin-ruualll,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[2] Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non- non-uniform density. International Conference on Machine Learning, pages 685-694 PMLR, 2020.
[3] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez- Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:180.00/261, 2018.
[4] Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.00 03 130, 2023.
[5] Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In arXiv, 2023.
[6] Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Sys- tems, 34:21557-21566 2021.
[7] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1628-8633 2021.
[8] Yinbo Chen and Xiaolong Wang. Transformers meta-learners for implicit neural representa- tions. In European Conference on Computer Vision, pages 170-187. Springer, 2022.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[10] Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: COmpression with implicit neural representations. In Neural Compression: From Information Theory Applications- Workshop @ ICLR 2021, 2021.
[11] Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point a function and you can treat like one, International Conference Machine Learning, pages 5694-5725. PMLR, 2022.
10

---

[12] Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliński, Yee Whye Teh, and Armaud Doucet. Coin++: Data agnostic neural compression. arXiv preprint arXiv:2201. 2es222201.2904, 2022.
[13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-111 1335. PMLR, 2017.
[14] David Ha, Andrew M. Dai. and Quoc V. Le. Hypernetorrks. In International Conference Learning Representations, 2017.
[15] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 12.30, 2020.
[16] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint 2XXv:2207.12598. 2022.
[17] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, Thomas Funkhouser, et al. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6001-6010, 2020.
[18] Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra. Relu fields: The little non-linearity that could. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
[19] Chiheon Kim, Doyup Lee, Sachoon Kim, Minsu Cho, and Wook- Han. Generalizable implicit neural representations via instance pattern composers. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, 2023.
[20] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[21] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15523-11532. 2022.
[22] Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Man- mohan Chandraker. Modulated periodic activations for generalizable local functional represen- tations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14214-14223, 2021.
[23] Lars Mescheder, Michael Oechsle, Michael Niemeyer. Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4460 4470, 2019.
[24] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[25] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 4111):10:11102:155, July 2022.
[26] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[27] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 1224922 2022.
[28] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pages 5301-5310. PMLR, 2019.
[29] Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelit images with vaevae-2. In Advances in neural information processing systems, pages 14866-14876. 2019.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
11

---

[31]
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions, Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[32]
Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19325 2021.
[33] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
[34] Matthw Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-8258, 2022.
[35] Matthew Tancik, Ben Mildenhall, Terrance Wang. Divi Schmidt, Pratul Srinivasan, Jonathan Barron, and Ren Ng. Learned initializations for optimizing coordinate-basse neural repre- sentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2846-2855, 2021.
[36] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998 6008, 2017.
[38] Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho. Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. The Eleventh International
Conference Leurning Representations, 2023.
[39] Tackgeun You, Sachoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation. In Proceedings of the International Conference on Neural Information Processing Systems, 2022.
[40] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In International Conference Machine Learning, pages 7693-7702. PMLR, 2019.
12

---

A Implementation Details
We describe the implementation details of our locality-aware generalizable INR with the Transformer encoder and locality-awart INR decoder. We implement our framework based on the official open- sourced implementation of IPC5 for a fair comparison. Our Transformer encoder comprises six blocks of self-attentions with 12 attention heads, where each head uses 64 dimensions of hidden features, and R = 256 latent tokens for all experiments. We use the Adam (20) optimizer with ($1, B2) (0.9, 0.999) and constant learning rate of 0.0001. The batch size is 16 and 32 for image reconstruction and novel view synthesis, respectively.
A.1 Image Reconstruction
178 x 178 image reconstruction For the image reconstruction of CelebA, FFHQ, and ImageNette with 178x 178 resolution, we use L= 2 level of modulation features for multi-band feature modu- lation of locality aware INR decoder. The dimensionality of frequency features and hidden layers in the INR decoder is 256, where (0 7.) = (128, 32, 16). We represent a 178 x 178 resolution 02
of the image 400 tokens, where each token corresponds to 9x9 size of the image patch with zero padding. We use a multi-head attention block with two attention heads for our selective token selection via cross-attentiin our framework on CelebA, FFHQ, and ImageNette during 300, 1000, and 4000 epochs, respectively, When we use four NVIDIA V100 GPUs, the training takes 5.5, 6.7, and 4.3 days, respectively.
Following the experimental setting of previous studies [8] 19], we train
ImageNet 256x256 We use L = 2 level of feature modulation for the image reconstruction of ImageNet with 256x256 resolution. We use eight heads of selective token aggregation, 256 dimensionality of frequency features and hidden layers of the INR decoder, and (01 02 og) = (128, 16). An image is represented as 256 tokens, where each token corresponds to 16x 16 patch in the image. We use eight NVIDIA A100 GPUs to train our framework on ImageNet during 20 epochs, where the training takes about 2.5 days.
FFHQ 256 512x512, and x024x 1024 Our framework for FFHQ 256x256 and 512x512
uses L = 2 level of feature modulation with (01,02: Fyy = (128, 32, 16). The size of each patch 16 and 32 for 256x256 and 512x512 resolutions, respectively, the number of latent tokens is R = 256, and the dimensionality of the INR decoder is d: = = 256. Our selective token aggregation uses two and four heads of cross-attention for FFHQ 256 and 512x512, respectively. We randomly sample the 10% of coordinates to be decoded at each training step to increase the efficiency of training. We train our framework during 400 epochs, while the training takes about 1.5 days using four NVIDIA V100 GPUs for FFHQ with 256 256 and about 1.4 days using eight V100 GPUs for FFHQ with 512,512, For FFHQ 1024x x 1024, we use 48 patch size to represent an image as 484 data tokens and L = 2 level of feature modulation with (01, F2 σg) = (256, 64, 32). The training of 400 epochs takes about 3.4 days using eight NVIDIA V100 GPUs.
A.2 Novel View Synthesis
We train our framework for the task of novel view synthesis on ShapeNet Cars, Chairs, and Lamps. Given a few known camera views as support views of 3D object, our framework predicts a light field of the 3D object to predict unseen camera views. For fair comparison, we use the same splits of train-valid samples with previous studies of generalizable INR E
Given rendering images of support views with 128x x 128 resolution, we first patchify each rendered image into 256 tokens with 8x8 size of patches. Then, we concatenate the patches of all support views with learnable tokens for the input of our Transformer. We use the Plücker coordinate to represent a ray for pixel as an embedding with six dimensions and concatenate the ray embedding into each pixel along the channel dimension. Since our INR decoder estimates light field of a 3D object, the INR decoder has six input channels dn = for a ray coordinate and three output channels dout = 3 for a RGB pixel. Our INR decoder uses L = 2 level of feature modulation with (01, 02, oa) (8, 4,2). We use dy = d = 256 dimensionality of the frequency features and hidden features of the INR decoder. We use 1000 training epochs for ShapeNet Cars and Chairs, while using 500 epochs for ShapeNet Lamps.
19 35
hhans//githbb.com/kkkkbaaannnnnn
13

---

A.3 Diffusion Model for INR generation
We implement a diffusion model to generate the latent tokens for INRs ImageNet 256x256. Different from the conventional approaches, which use U-Net architecture to generate image, we use a vanilla Transformer with a simple stack of self-attentions, since the latent tokens do not predefine 2D grid structure but permutation-equuivariae model has 458M parameters having 24 self-attention blocks with 1024 dimensions embeddings and 16 heads. We remark that the locality-aware generalizable INR is not updated during the training of diffusion models. For the training of the diffusion model, we follow the formulation of DDPM 15]. The linear schedule with 7 1000 is used randomly corrupt the latent tokens for INRs using isotropic Gaussian noises, and then we train our Transformer to denoise the latent tokens. Instead of the e-parameterization that predicts the noises used for the corruption, our Transformer oo-parameterization to predict the original latent tokens. We drop 10% of class conditions for our model to support classifier-free guidance following the conventional setting [6]]. For the stability of training, we standardize the features of latent tokens, after computing the mean and standard deviation of feature channels of each latent token based on the training data. We use eight NVIDIA A100 GPUs to train the model with 256 batch size during 400 epochs, where the training takes about days. The Adam (20 optimizer with constant learning rate 0.0001 and (31, (2) = (0.9, 0.999) used without learning rate warm-up and any weight decaying. During training, we further compute the exponential moving average (EMA) of model parameters with a decaying rate of 0.9999. During the evaluation, use the EMA model with 250 DDIM steps [33] and 2.5 scales of classifier-free free guidance []6.
The Transformer for the diffusion
B Additional Experiments
B.1 Ablation Study on the Number of Levels
Table d demonstrates the effect of the number of levels I. on image reconstruction benchmarks of FFHQ images with 256x2566 512x512, and 1024x 1024 resolutions. Our INR de- coder uses bandwidths 0g = 16 and (a equal (128), (128, 32), (128, 14,8,64.77. 32) and (128, 90, 64, 32) for L = 1, 3,4 respectively in case of 256 and 512x512 resolution, and all bandwidths are doubled for 1024x 1024 leverage greuuereyencc details.
Table 6: PSNRs on the reconstructed FFHQ with 256x 512x512, and 1024 1024 resolutions for different number of levels.
256x256
512x512
1024x 1024
TransINR IPC [9
30.96
29.35
34.68
31.58
28.68
Ours (I. = 1)
37.09
34.84
31.56
Ours (L 2)
39,88
35,43
31.94
Ours (L =3)
40.13
35,58
32.40
Note that our framework outperforms previous studies 88 19 even with 7. Moreover, the results demonstrate that increasing improves the performance, while the performance saturates beyond L. 3. We postulate that higher resolution requires larger number of levels, as the performance gap between L = 3 and L = 4 decreases as the resolution increases.
Ours (L 4)
39.79
35.40
32.32
B.2 Additional Examples of Novel View Synthesis
In Figure 8 we show additional examples of novel view synthesis of ShapeNet Chairs, Cars, and Lamps with one five support views.
B.3 Additional Examples of High-resolution Image Reconstruction
Figure and 10 shows image reconstruction examples of FFHQ with 256x256, 512x512, and x024x 1024 resolution by previous studies [8] [9] and our locality-aware generalizable INR. Unlike previous studies, our framework successfully reconstruct fine-grained details in high resolutions.
14

---

Support view
Query view
Predicted view
Support views
Query view
Predicted view
Query view
Predicted view
Support views
Query view
Predicted view
Support views
Figure 8: Examples of novel view synthesis of ShapeNet Chairs, Cars and Lamps with one, two, three, and five support views.
15

---

Figure Examples of reconstructed images of FFHQ with 256x256 resolution (top row) and 512x512 resolution (bottom row) by TransINR g (left), IPC [19] (middle), and our locality-aware generalizable INR (right).
Figure 10: Examples of reconstructed images of FFHQ with 1024x 1024 resolution IPC (left) and our locality-aware generalizable INR (right).
16

---

73 + ho
Figure 11: Additional examples of class-conditional image synthesis by generating the locality-aware latents of our framework via a transformer-basse diffusion model with 458M parameters. All images are generated with classifier-fe freeeffer-freeeeee guidance at scale 2.5.
17

---

Revonstructed/ rendered image
difference between model predictions
after replacing latent token with the zero vector
IPC
Ours
Ours
Figure 12: Additional visualization differences between model predictions after replacing a latent token with the zero vector for IPC [19] and our framework.
18

---

B.4
Additional Examples of Conditional Image Synthesis
Figure I1 shows additional examples of generated images with 256x256 resolution by generating locality-awee aware latents of our framework.
B.5 Additional Visualization for Locality Analysis
Figure 12 visualizes which local information of data is encoded in each latent token of IPC [9 and our locality-aware generalizable INR in addition to 5igure We randomly select a latent token and replace it with the zero vector, then visualize the difference between the model predictions with or without the replacement as described in Section 4.3 'he differences are rescaled to have the maximum value of for clear visualization. Furthermore, we fix the set of replaced latent tokens for different samples in Figure t2 emphasize the role of each latent token. Note that each latent token of our framework encodes the local information in a particular region of images or light fields, while latent tokens of IPC encode global information over whole coordinates.
B.6 Ablation Study on Linear Layers in Selective Token Aggregation
Our framework adds linear layer in Eq (6) and Eq 7 to exploit complex frequency patterns, improving the per- formance. While the Fourier features consist of periodic
Table 7: PSNRs on the reconstructed ImageNette with 178x 178 resolution.
patterns along an axis, the frequency patterns in Eg (6) can also include non-periodic patterns. Note that IPC [99] also uses similar design, while modulating the second MLP layer to exploit complex frequency patterns. The linear layer in Eq 7 is used to process the modulation vector according to each frequency bandwidth, motivated by the design of separate projections for (query, key, value) in self-attention. The results below also show that removing the linear layers Eq (6) and Eq significantly deteriorates the image reconstruction performance on ImageNette 178x 178.
PSNR
Ours
37.46
w/o Linear Eq 6
31.95
w/o Linear Eq
32.07
w/o Linear Eq
6 and Eq
31.57
(
19


Reviews:
[{"review_id": "WpDAxDEIz2", "reviewer": "Reviewer_QDhX", "summary": "The paper tackles the important problem of bridging the gap between generalizable implicit neural representations (INR) and per-sampled trained ones. The core hypothesis is that previous generalizable INRs failed to capture local details in the global latent code due to their inductive bias. The idea of the proposed method is to equip learning such INRs with a Transformer encoder, which selects and encodes local information into multiple tokens. The paper elaborates on all parts of the pipeline and presents the empirical study on image reconstruction and novel view synthesis with shapenet. Overall, the proposed method shows a great advantage over the prior art. Ablation studies are thorough.", "questions": "In Fig. 2, how do the transformer inputs form in each considered setting (image / NVS)? \nDoes the framework require retraining everything from scratch for every new resolution? \nCan any of the pretrained backbones be used?\nCan the authors think of a way to visualize the effect they discuss regarding ordering frequencies based on the depth of the layer in the decoder?\nWhat do the results look like for the best possible setting for 1Kx1K resolution in Table 2?", "limitations": "Discussed adequately in the conclusion. However, it would be interesting to learn about dealing with arbitrary resolutions, or other forms of inputs not directly amenable to transformer encoders.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea behind the method is novel and interesting. The paper borrows the best ideas from other domains, including transformers and nerf. Experimental studies on FFHQ and Shapenet are convincing. The paper is well-written and rather polished.", "weaknesses": "The introduction is a bit too broad and needs more specificities about the proposed method. The statements from the title, abstract, and intro, set expectations for something more generic than what can be processed by a transformer encoder.\n\nThe frequency decomposition mechanism is not very well explained. Particularly, the choice of two blocks for frequency decomposition is never explained nor ablated. Would the performance keep growing if more of those blocks had a different frequency coverage scheme? \n\nThe locality property of the produced tokens is also not very clear. On the one hand, the authors claim that the tokens are locality aware; on the other that thanks to permutation equivariance of the attention mechanism in the transformer encoder, the tokens form an unordered set. It would be better to explain these aspects more carefully (including in the rebuttal).\n\nList of typos\n- L49: outperformance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Broad Introduction]**  \nWe will revise the Introduction by adding the following specific description of our transformer encoder. Specifically, we will add the detailed explanations about the use of cross-attention in selective token aggregation to Lines 43-44. The cross-attention is used for each coordinate to extract the information of the latents, which are the output of the Transformer Encoder. We will also describe how multi-band frequency decomposition is designed in Lines 45-46.\n\n\n**[Choice of the Number of Blocks for Frequency Decomposition]**  \nWe will revise the explanations for better understanding of the frequency decomposition mechanism. In addition, Appendix B.1 includes the ablation study, which the reviewer has mentioned, on the number of blocks $L$ for frequency decomposition. For image reconstruction of FFHQ 256$\\times$256, 512$\\times$512, 1024$\\times$1024, the performance improves with respect to $L$ increasing from 1 to 3, but the performance saturates at $L \\geq 3$. In our main experiments, we use $L=2$, considering the trade-off between computational costs and performance.\n\n\n**[Clarifying Locality Property of the Produced Tokens]**  \nFigure 5 demonstrates the locality property of the produced tokens. Figure 5 visualizes that each latent token captures the local information of a data instance and affects the pixels/rays in a certain local area. \n\nWe will clarify how the permutation-equivariance of self-attention affects the design of our framework in terms of locality property as follows. The permutation-equivariant of self-attention in the Transformer encoder makes our framework not assume the local structures of data and latent tokens. That is, we do not assume the permutation of the location of latent tokens (ordering of latent tokens), but consider the latent tokens as a set of local information. During training, each latent token learns to capture the local information of data, while covering whole regions to represent a data instance. This property enables our framework to be readily applied to diverse data with non-grid coordinates. For example, determining the order and size of local regions is not straightforward for Plücker coordinates of rays. However, self-attention enables each local latent to capture the local information of data, while the set of local latents represents the information of whole rays. \n\n**[Typos]**  \nThanks for the detailed comment. We will fix all typos in our manuscript.\n\n**[The Inputs of Transformer Encoder]**  \nA transformer input is the concatenation of image patches and learnable tokens, as described in Appendix A.1 and A.2. For image reconstruction, an image is represented as a set of patches, where each patch has $P \\times P$ size. We use P=9, 16, 32, 48 for 178$\\times$178, 256$\\times$256, 512$\\times$512, and 1024$\\times$1024 resolution, respectively. For 178$\\times$178 and 1024$\\times$1024 resolution, images are zero-padded to make the size evenly divisible by the patch size. We use zero-padding of 1 and 16 pixels on every side, respectively.\n\nFor novel view synthesis, we use the Plücker coordinate to represent the information of the rays in the rendering view of a 3D object. Given rendering images of support views, we concatenate the ray coordinates with pixels along the channel, and then patchify the support views using $P=8$ patch size. Then, we concatenate the patches of all support views with learnable tokens for the input of our Transformer.\n\n**[Retraining for new resolution]**  \nYes, we train our framework for each resolution of the dataset.\n\n**[The Use of Pretrained Backbones]**  \nSince conventional pretrained models are not trained for the latents of INRs, we believe that such pretrained backbones cannot be utilized within our framework.\n\n**[Visualization of the Effect regarding Frequency Ordering]**  \nIn the additional pdf file for author responses, we visualize the effect of frequency ordering on the reconstruction of high-frequency details. We visualize the pixel-wise reconstruction error for models trained on ImageNette 178x178 with $(\\sigma_1, \\sigma_2)=(128, 32)$ and $(\\sigma_1, \\sigma_2)=(32, 128)$. Our design choice $(\\sigma_1, \\sigma_2)=(128, 32)$ shows superior performance in reconstructing high-frequency details of data.\n\n**[Qualitative result in 1024x1024 resolution]**  \nAppendix B.3 includes the qualitative results for 1024x1024 image reconstruction. \n\n**[Limitations]**  \nThanks for the constructive comments. We agree that extending our framework to support arbitrary resolution will be an interesting future work. We will also describe the limitation of our framework that requires an amenable form of input for transformer encoders.\n"}, {"review_id": "rOxNiL8DYe", "reviewer": "Reviewer_AhKU", "summary": "The paper focuses on the task of training a single coordinate-based neural network to represent multiple scenes or instances. There are two main technical contributions that improve the quality of these generalized representations: (1) a Transformer-based encoder that extracts localized features of each target instance (e.g. image or scene), and adaptively weights these localized features during inference at different coordinate locations, (2) the coordinate-based neural network operates coarse-to-fine in the frequency domain, taking in high-frequency instance-specific features at earlier layers and lower-frequency instance-specific features at later layers, so that higher-frequency features are processed by a deeper network. The paper includes compelling results on medium and high resolution image datasets, and promising preliminary results on few-shot novel view synthesis. Ablation studies show that of the two technical contributions, neither alone yields improvement but both together do, compared to the main baseline Instance Pattern Composers.", "questions": "- I am still not fully sure what exactly is the input to the model during inference. Is it basically an autoencoder, taking an image as input and compressing it into latents and then decoding them into the image again? Likewise for the generative experiment (figure 6), how are these latents generated?\n- Equation 2 describes the Fourier featurization; it appears to use the same featurization as the original NeRF paper (with axis-aligned frequencies) rather than e.g. a random Gaussian (not axis-aligned) set of frequency vectors as in the “Fourier Features Let Networks Learn…” paper, though both of these papers are cited. I wonder why the authors chose to use the original Fourier featurization rather than the newer version?\n- Equation 5 describes multi-head attention as the mechanism for aggregating/weighting the localized latent features when performing inference on a given coordinate vector. I wonder if the authors considered a more straightforward kernel function (e.g. Gaussian kernel based on Euclidean distance, or even a learned kernel) or what the motivation was for using multi-head attention.\n- Equations 6 and 7 describe a process for separating the modulation vector into different frequency bands, by using a shallow MLP with inputs based on different Fourier features (in the desired frequency band). I wonder why the MLP is necessary here, compared to just taking the subset of Fourier features that are in each frequency band?\n- Tables 1 and 2 show compelling numerical comparisons vs Learned Init, TransINR, and IPC. The accompanying text notes that the capacity of the encoder, latent tokens, and decoder are matched among all methods, except for the modulation methods. So I wonder how the capacity of the modulation methods compares?\n- I also wonder what the total training time is on each dataset, compared to prior work.\n- Figure 7 shows that test-time optimization of all parameters outperforms per-sample optimization. How is this possible? I wonder if training longer would allow the per-sample FFNet to “catch up”, or if the FFNet here has lower capacity, or if there is some other explanation for how a generalized model could outperform a more specialized model. Maybe the generalized model is starting from a better initialization and thus reaches a better optimum?", "limitations": "Limitations are not really discussed, beyond mentioning some directions for future work--but for this topic I believe this is appropriate. I do not foresee direct negative societal impact from this work, though like all image processing research it has the potential to be misused e.g. towards surveillance or other harmful ends.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "These two technical ideas make sense, particularly the idea of having localized features, and the results on image datasets are very compelling—though some important omitted details make these difficult to fully interpret, as does my own lack of familiarity with the baseline methods. Ablation studies and more in-depth analysis of learned features (figure 5) are also interesting.", "weaknesses": "The paper writing/presentation could be substantially improved. The first roughly 2 pages of the paper are laden with jargon (and some typos/grammatical issues), making the actual new ideas in the paper difficult to tease out. As a reader who has worked in implicit neural representations but not generalizable ones, many parts of the paper required some assumptions or terms that might be common to those who work in generalizable INRs but unfamiliar to researchers in even a very adjacent area. Some examples: what exactly is a latent (ie what is the input used to produce a latent)? What is modulation (in signal processing this would be element-wise multiplication)? The methods section does largely (though not fully) clarify the method and the new ideas, but leaves me wondering about the motivation behind some of the design decisions that are stated without much explanation. I list these remaining questions in the “questions” section of the review, along with some questions about the experimental setup of the results.\n\nOne separate comment/suggestion is about the clarity of the figures. Figure 1 shows that the proposed method trains a lot faster/better than two baselines, but doesn’t explain what the task is (just the dataset). Figure 2 gives a helpful overview, but doesn’t actually explain the two core new ideas of the paper. For example, from the figure and caption I can’t tell what is different about the two yellow blocks (one is for high-frequency features and one for low-frequency features, but I didn’t find this out until I read 2 pages past the figure). Some terms in the figure are also not defined; for example I assume that “FC” means fully-connected, but I’m not sure how this differs from “Linear”. Figure 3 shows a compelling comparison to prior work, but I would encourage the authors to include the ground truth image for full comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Improvement of Introduction]**  \nAs per suggestion, we will revise our manuscript to clarify the terminologies in the Introduction for better understanding as follows. The latents refer to the outputs of our Transformer Encoder, corresponding to the positions of learnable input tokens. The input of the Transformer Encoder is the concatenation of data tokens and learnable tokens, where the number of learnable tokens equals the number of latent tokens.   \nWeight/feature modulation refers to the modification of a weight/feature by element-wise multiplication or addition to adapt a shared weight/feature for a data instance. Thus, the Transformer encoder extracts a set of latent tokens to modulate the shared parts of generalizable INRs to represent a data instance.  \nOur study is primarily motivated by the limitation of SOTA methods, TransINR and IPC, which do not consider the locality in the data. We have elaborated on why the previous approach, IPC, cannot consider the local information of data in Lines 134-138 in Section 3.3.1. We will include the details of previous studies to clarify the motivation of our study and the limitations of previous studies. \n\n\n**[Typos and Grammatical Issues]**  \nWe will fix all typos and grammatical issues.  \n\n\n\n**[Suggestions for the Clarity of Figures]**  \nReflecting the reviewer’s suggestions, we will add and clarify our figures as follows:\n- Figure 1: The task is image reconstruction.\n- Figure 2: The two frequency features have different bandwidths. We will replace “FC” with “Linear.”\n- Figure 3: We will add the ground truth image for detailed comparison.\n\n\n**[Inputs to the model]**  \nRegarding explaining the input of Transformer Encoder, please refer to **[The Inputs of Transformer Encoder]** in our responses to Reviewer QDhX. Our framework for image reconstruction can be viewed as an autoencoder. However, our framework for synthesizing novel views should not be seen as an autoencoder, given that it involves the synthesis of previously unseen perspectives.\n\nWe adopt the two-stage framework to generate an image for conditional image generation. We first train our framework on ImageNet 256x256 to represent an image as a set of localized latents for INRs. After representing each image as localized latents, a diffusion model is trained following the experimental setting in Appendix A.3. After the training, the diffusion model gradually denoises the corrupted latents, which start from isotropic Gaussian noises, to generate new images.\n\n\n\n**[Fourier Featurization]**   \nWe adopt the Fourier featurization with axis-aligned frequencies in the original NeRF paper to ensure the stable and high performance of generalizable INRs. Since the random initialization of Fourier features [36] requires a careful selection of the variance for each sample, adopting random Fourier features deteriorates the performance of generalizable INRs, as shown below. We also emphasize that recent NeRF models [NewRef: RefNerf] and generalizable INRs [8, 10] have still adopted the Fourier featurization with axis-aligned frequencies. \n\n| ImageNette 178x178 | PSNR |\n|:-----|:-----:|\n| IPC | 34.11 |\n| Ours | 37.46 |\n| IPC w/ random FF | 29.27 |\n| Ours w/ random FF | 30.94 |\n\n[NewRef: Ref-Nerf] Verbin, Dor, et al. \"Ref-nerf: Structured view-dependent appearance for neural radiance fields.\" 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.\n\n**[Multi-Head Attention in Selective Token Aggregation]**  \nWe exploit multi-head cross attentions for selective token aggregation, since using cross-attention is an intuitive choice within the context of modern deep learning architectures. As learned kernel functions, cross-attentions can consider various similarity patterns between projected queries and keys, instead of a tailored similarity pattern. As shown below, adopting multi-head cross attentions improve the reconstruction performance on FFHQ 178x178, compared with single-head cross-attention, which is close to a learned kernel function.\n\n| | ImageNette 178x178 | FFHQ 256x256 |\n|:------|:-----:|:-----:|\n| Ours (2 heads) | 38.72 | 39.88 |\n| w/ single-head | 37.46 | 38.01 |\n\n**[A Linear Layer in Multi-Band Feature Modulation]**  \nWe add a Linear layer in Equations (6) and (7) to exploit complex frequency patterns, improving the performance. While the Fourier features consist of periodic patterns along an axis, the frequency patterns in Equation (6) can also include non-periodic patterns. Note that IPC [19] also uses a similar design, while modulating the second MLP layer to exploit complex frequency patterns. The linear layer in Equation (7) is used to process the modulation vector according to each frequency bandwidth, motivated by the design of separate projections for (query, key, value) in self-attention. The results below also show that removing the linear layers in Equations (6) and (7) significantly deteriorates the image reconstruction performance on ImageNette 178x178.\n\n| | ImageNette 178x178 |\n|:----|:----:|\n| Ours | 37.46 |\n| w/o Linear in Eq. (6) | 31.95 |\n| w/o Liner in Eq. (7) | 32.07 |\n| w/o Liner in Eq. (6) and (7) | 31.57 |\n\n\n**[The Capacity of the Modulation Methods]**  \nThe modulation capacity is determined by the size of latents to represent instance-specific information, except for the shared weights. TransINR, IPC, and our framework commonly use latents of size 256$\\times$256 as instance-specific information to modulate a shared architecture. \n\n**[Total Training Time compared to Prior Work]**  \nPlease refer to our responses to Reviewer wXRn.\n\n\n**[Performance of Test-Time Optimization]**  \nWhile per-sample optimization of FFNet starts from a random initialization, our framework provides a good initialization of an INR having high performance, as shown in Figure 7. Since our test-time optimization also updates whole parameters of the INR, it is reasonable to expect that improved initialization could lead to savings in training costs."}, {"review_id": "3nizBusrtc", "reviewer": "Reviewer_wXRn", "summary": "This paper aims to enhance the performance of generalizable implicit neural representation locality-aware model designs. A transformer encoder is applied to convert image patches into latent tokens, which the proposed locality-aware decoder composed of the selective token aggregation and the multi-band feature modulation is based on to predict outputs. Experiments on tasks including image reconstruction and novel view synthesis are performed to demonstrate the proposed method's state-of-the-art performance.", "questions": "see weakness.", "limitations": "- No analysis for failure cases.\n- Qualitative results in the paper are limited, and should be more diverse.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed locality-aware decoder containing selective token aggregation and multi-band feature modulation is novel and effective.\n+ Extensive experiments show that the proposed method achieves good performance.\n+ The paper is clear and easy to follow.", "weaknesses": "- It is not very straightforward to get the idea of  \"Locality\" in this work. Any empirical evidence like visualization in Selective Token Aggregation to reveal how the locality is enhanced?\n- Analysis and comparison of model/runtime efficiency are missing.\n- Some important implementation details are missing, e.g. what's the value of L and what's the impact of this hyperparameter?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[The Idea of Locality]**    \nIn our study, “locality” describes a concept that the features in a data instance have potentially a high correlation with each other, where the distance between their corresponding coordinates is close. For example, in a 2D image, pixels located close to each other tend to have similar RGB colors when the two pixels are located in a nearby position. Thus, the notion of “locality-aware\" implies that our latent tokens learn to contain the information of a local region for efficient and effective representation of a data instance. \n\nFigure 5 provides the empirical evidence of enhanced locality-awareness, since the previous study, IPC, has overlooked modeling the local information of a data instance. Figure 5 shows that each latent of IPC affects most coordinates' features, failing to model local information. However, our framework shows that each latent covers a certain region of the local area, enhancing the locality-awareness of our framework. \n\n\n**[Analysis and Comparison of Model Efficiency]**    \nFor the experiments on ImageNette 178x178 in Figure 1, our framework has 0.9% more parameters, 44.14M, than IPC having 43.75M of trainable parameters. The runtime of IPC is 78 seconds per  training epoch, but our framework takes 90 seconds. However, Figure 1 shows that our framework is significantly more efficient and effective than IPC and TransINR, despite a 15% longer runtime per training epoch.\n\n\n**[The Value of $L$]**    \nOur experiments use $L=2$ as described in Line 206 and Line 238, since $L$ is the number of frequency bandwidths. We will explicitly specify the total number of $L$ in our revised version. The experiments to study the impact of the hyperparameter $L$ are attached to Appendix B.1 in our supplementary material.\n\n\n**[Analysis for Failure Cases]**    \nAlthough our framework significantly improves the performance of a generalizable INR, the reconstruction performance on 1024x1024 image resolution is still incapable of perfectly reconstructing all high-frequency details in original images. In addition, the qualitative results on novel view synthesis show blurry examples due to the lack of the training objective of generative modeling to synthesize unseen views. We will attach the explanation of failure cases above to inform the research community about the boundary of this study.\n\n**[Diverse Qualitative Results]**    \nPlease refer to the attached supplementary material for more diverse qualitative results. Our supplementary material includes various examples of novel view synthesis with different numbers of support views, image reconstruction with 256x256, 512x512, and 1024x1024 resolutions, class-conditional image generation on ImageNet, and additional visualization for locality analysis. \n"}, {"review_id": "uktcpc2SjR", "reviewer": "Reviewer_TnVm", "summary": "This paper tries to improve the expressive power of neural implicit function modulation by enhancing its ability to localize and capture fine-grained details of data samples. A novel framework for generalizable INR is proposed that combines a transformer encoder with a locality-aware INR decoder. It further utilizes selective token aggregation and the multi-band feature modulation to learn locality aware representation in both spatial and spectral aspects.", "questions": "Can we interpolate between the latents of two data samples to generate a new sample?", "limitations": "Experiment results are limited in evaluation scale and in-depth analysis.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ combine locality aware transformer encoder with global neural implicit function to improve the representation quality of local details\n+ use selective token aggregation for spatial locality\n+ use multi-band feature modulation for spectral locality\n", "weaknesses": "- All the quantitative results are reported with PSNR metric. More evaluation metrics emphasizing local details should be used.\n- Limited results reported on novel view synthesis and conditional image generation.\n- In conditional generation, the latents are corrupted by Gaussian noise and denoised with a trained diffusion model. If the Gaussian noise is small, the task becomes trivial. An comparison experiment should be done for directly reconstructing with corrupted latents using the implicit neural function.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[More Evaluation Metrics to Emphasize Local Details]**  \nWe compare our framework with IPC [19] to emphasize local details using HF-PNSR-r, which calculates PSNR only using the high frequency components of an image. We have also considered other evaluation metrics such as rFID, SSIM, LPIPS during the rebuttal period. However, for the reviewer’s request to emphasize local details, PSNR is the best metric to measure per-pixel errors, since the others measure the structural or perceptual quality rather than local details. The metrics shown in the table below are PSNRs between the ground truth image and the reconstruction of ImageNette 178x178, after the bottom 5%, 10%, 20%, 40%, and 80% of frequencies are filtered out. The results show that our framework outperforms a previous study, IPC, to reconstruct various ranges of high-frequency details.\n\n|     | remove 5% | remove 10% | remove 20% | remove 40% | remove 80% |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| IPC | 38.81 | 39.18 | 40.35 | 43.37 | 50.20 |\n| Ours | 46.26 | 46.38 | 46.88 | 48.77 | 54.48 |\n\n\n**[Limited Results on Novel View Synthesis and Conditional Image Generation]**  \nWe emphasize that our experiments on novel view synthesis are not insufficient compared with previous studies. We follow the experimental setting with previous studies such as TransINR [8], and IPC [19], while using the same experimental setting with IPC [19]. Compared with TransINR [8], which has trained their framework only with 1-2 support views, we provide the experimental results with 1-5 support views. While our framework and earlier research have concentrated on category-specific and synthetic datasets, we anticipate that the enhancements in our performance regarding novel view synthesis offer the potential for our framework to extend its learning capabilities to open-domain and large-scale 3D object datasets, such as Shap-E [NewRef: Shap-E], in future endeavors.\n\nFor conditional image generation, we have trained a diffusion model on ImageNet 256x256, since ImageNet 256x256 is the renowned and standard benchmark for evaluating class-conditional image generation. Extending the range of conditional generation, such as layout-to-image or text-to-image, would be an interesting future work. However, we consider the in-depth analysis for training diffusion models to generate INRs is beyond the scope of this study.\n\nIf the reviewer suggests detailed experiments to add, we will attach the experimental results in the revised version if possible. \n\n[NewRef: Shap-E] Jun, Heewoo, and Alex Nichol. \"Shap-e: Generating conditional 3d implicit functions.\" arXiv preprint arXiv:2305.02463 (2023).\n\n\n**[The Magnitude of Gaussian Noise for Diffusion Model]**  \nThe task of denoising Gaussian noise is not trivial, since we follow the conventional setting to train diffusion models. Given a diffusion time step $t$, a Gaussian noise $\\epsilon$ is added to the localized latents $\\mathbf{Z}^{(n)}$ as $\\sqrt{\\alpha_t} \\mathbf{Z}^{(n)} + (1-\\alpha_t) \\epsilon$, where $\\alpha_0=1$ and $\\alpha_T=0$. When a diffusion time step is close to $t=0$, the added noise is small. However, when the diffusion time step is close to $t=T$, the latents $\\mathbf{Z}^{(n)}$ become a Gaussian noise after noise addition, making the denoising task nontrivial. We note that the generation process of diffusion models starts from pure Gaussian noise to generate INR latents. \n\n\n**[Interpolation of the Latents of Two Data]**   \nAlthough we can interpolate the latents of two samples, the outcomes do not yield semantically meaningful images akin to a straightforward linear interpolation between two images.\n"}, {"review_id": "U1ml4zeowk", "reviewer": "Reviewer_KNfA", "summary": "This paper enhances the performance of generalizable INR via improving its locality awareness. The transformer encoder feed on patchs of an image and produce latent tokens with local information, and later extracted as modulation vectors. The proposed INR decoder with selective token aggregation and the multi-band feature modulation learn locality-aware representation in spatial and spectral domains respectively. Experiments show that the proposed method outperform previous methods both on reconstruction and downstream image generative tasks.", "questions": "1. The provided ablations are on reconstruction performance only. While this is the main task and objective, I'm also wondering how these ablations perform on other downstream applications in the paper, including novel view synthesis and image generation. Does any options or hyperparameters make a difference for these applications? Or do you find you don't have to carefully pay attention to them in these applications in your experiments?", "limitations": "1. More discussions and analyses on the learned INR model weight space would be welcomed and inspiring. How do you think the locality awareness may improve the INR space distance smoothness? For example, in LDM they applied KL loss or VQ regularization on VAE (although not big) to help the distance in the latent space (feature map) be consistent with that in the pixel space. Do you think this might be one of the point that the proposed method reaches better results in generative tasks including novel view synthesis and image generation?\n\n2. IPC also conduct experiments on downstream classification tasks. It would be great to see similar results on your proposed method and comparisons with them. (Or do you think there might be any difference in design making that your method is better at and more fitful for generative tasks than classifications? Any discussion is welcomed.)", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The proposed make a lot of sense to enhance the locality of INRs and via patch processing and attentions in transformers etc., and it reaches very good results compared to previous SOTAs. The method is clearly illustrated, and the results are well displayed both quantitatively and qualitatively. There are also many ablations and discussions/analyses into the details and depth. The supplementary material also provides more details, discussions and results.\n\n2. Besides the main reconstruction task, the proposed method is also able to perform a few generative tasks well, and also better than previous SOTAs.\n\n3. Figure 5 (and Figure 12 in Supp) shows interesting results on replacing latent tokens with null ones, unvealing some underlying rationales of the learned space.\n\n4. Source code is provided in the supplementary material.", "weaknesses": "1. [Core] An INR usually refers to a neural network that store information in its model weights instead of feature maps. The proposed modulation vector m_v seems to be more like a latent code (in an encoder-decoder framework). I hope to see clear clarification on how the parameters of the proposed INR is composed of, i.e. how much portion is feature maps and how much portion is model weights. In addition, if most (or almost all) parameters of the \"INR\" is just feature maps, then it might not be very difficult to perform those downstream generative tasks, as it is essentially more an improved VAE than an independent INR. I'm wondering how this could help the displayed performance.\n\n2. [Core] The INR (decoder or modulation vectors, i.e. the non-shared parts to independently form a single INR representing a single image) model size / compression ratio seems not to be mentioned in the paper. How big the produced INR is to representing one image of 256x256 for example? The paper has conducted a few ablations on the model structure details, including Table 4 and Table 6, while I'm wondering a more overall metric, such as number of paramters in an INR, its size in MB, or BPP (bits per pixel). How does this compared to other methods such as TransINR ordan IPC?\n\n(The above two points are my main concerns and the game-changer for my assessment.)\n\n3. Figure 2 shows the core mechanism clearly, but in addition I think a specific figure to show the produced INR itself (e.g. many layers or blocks duplicated) would make the full pipeline clearer to understand.\n\n4. Table 5 seems not to be mentioned in the main body text nor illustrated anyhow (while L271 should be referring to Figure 4 as a typo).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[The Portion of Localized Latents in an INR]**  \nThe localized latents account for 9% of INR parameters (=65,536/725,446), which is a small portion of INR. Since the shape of extracted localized latents is 256x256, we emphasize that the size of localized latents is equivalent to **one weight matrix of a single MLP layer** with 256 hidden dimensions. \nMeanwhile, as the reviewer comments, our framework can also be interpreted as an improved version of encoder-decoder (INR) architecture, since we also use a hypernetwork to predict the latents for modulating INRs. A conventional VAE has a decoder of substantial size, having tens of millions of parameters (>10M), which is used to generate or reconstruct an image from feature maps. However, our INR decoder has a much smaller number of parameters, (0.73M) with its role being to simply represent a data instance itself as INR.\n\n\n**[The Size of Non-Shared Part]**  \nThe size of non-shared latents is fixed to 256x256 for all experiments for TransINR, IPC, and our framework, including experiments on high resolution images such as FFHQ 1024x1024. Thus, TransINR, IPC, and our framework require 0.25MB (=32-bit x 256 x 256) to specify a data instance as an INR. The compression ratio is 130%, 33%, and 8% for 256x256, 512x512, and 1024x1024 images, respectively. Compared with a previous study (e.g., COIN [10]), which exploits an INR for data compression and combines quantization techniques to improve compression ratio, generalizable INRs show worse data compression performance, since they have focused on improving the reconstruction performance. Considering that our framework significantly improves the reconstruction performance of a generalizable INR, combining quantization techniques with localized latents will be an interesting future work for data compression. We will attach this discussion to our revised manuscript.\n\n\n**[Explanation about Figure 2]**  \nFigure 2 shows the produced INR in our experiments, illustrating the number of duplicated layers or blocks. We will add a detailed explanation to ensure understanding. \n\n**[Referring Table 5]**  \nThanks for the detailed comment. We will change “Figure 5” in line 271 to “Figure 6” and explicitly mention Table 5 in Section 4.4.\n\n\n**[Ablation Studies on Novel View Synthesis and Image Generation]**  \nSince training a diffusion model for image generation is expensive, we also attach the ablation study on the novel view synthesis of ShapeNet-Lamp with 3 support views. The results below show that both selective token aggregation (STA) and multi-band frequency modulation (multiFM) improve the performance of our framework.\n\n| | ImageNette | FFHQ | Lamps-3 views |\n|:------|:------:|:------:|:------:|\n|Ours | 37.46 | 38.01 | 26.00 |\n|w/o STA | 34.54 | 34.52 | 25.35 |\n|w/o multiFM | 33.90 | 33.65 | 25.78 |\n|IPC [19] | 34.11 | 34.68 | 25.09 |\n\n\n\n\n**[More Discussions and Analyses on the Learned INR]**  \nAs the reviewer’s comment, we believe that the locality awareness can also enhance the smoothness of the INR space. Figure 6 in our submission shows that the generated latents of IPC cannot provide realistic local details, since an artifact affects all coordinates, as described in Lines 303-305 and Figure 5. The results imply that the INR space of IPC is sensitive to slight variations or corruptions, while our framework has a more smooth INR space than IPC.\n\nMeanwhile, the regularization techniques (e.g., KL loss or VQ regularization), which the reviewer has mentioned, can also be applied to our framework, making the INR space further smoother. However, when employing the KL regularization with 1E-6 loss weight, the PSNR on ImageNet 256x256 decreases from 37.7 to 30.7. We have also changed the KL regularization in LDM to the deterministic version and fixed the predicted covariance as Isotropic Gaussian, then the reconstruction PSNR becomes 33.9. We believe that further exploration of hyperparameters can enhance our framework, expecting that a smoother INR space can help a diffusion model improve the performance of downstream tasks. \n\n\n\n**[Experiments on Downstream Classification Tasks]**  \nWe have checked that IPC does not conduct an experiment on any downstream task, but Spatial Funta [4] has conducted an experiment on a downstream classification task. We have yet been able to conduct the classification task due to the limited computational resources during the rebuttal period. However, we expect similar results with Spatial Functa, since the performance of generative tasks also shows similar results with Spatial Functa. We do not assume that our framework is more fitful for generative tasks than classification. Instead, we perceive a generative task to be more difficult than classification. Note that a generative task requires learning all local data details, while a classification task requires understanding the global semantics. We will do our best to reserve additional computational resources for conducting experiments on classification tasks to attach the results in the final version.\n"}, {"review_id": "QxggfUm2go", "reviewer": "Reviewer_binY", "summary": "Generalizable implicit neural representation (INR) can represent multiple data instances with a coordinate-based neural network, of which weights or intermediate features are modulated using instance-wise latent codes. A significant constraint of current generalizable INR is their struggle to localize and capture fine-grained details of data entities. This limitation results in a diminished expressive power of modulation.\n\nThis research tackles this problem by presenting an innovative framework for a generalizable Implicit Neural Representation (INR). This framework merges a transformer encoder with a locality-aware INR decoder. Two key components are designed to enable to capture of the local information of data, including selective token aggregation and multi-band feature modulation.\n\nThe authors demonstrate the effectiveness of their method with state-of-the-art performances on both image reconstruction and novel view synthesis. Furthermore, they show the potential of the proposed generalizable INR on the conditional image synthesis task.\n", "questions": "In addition to the above weaknesses, here are two more questions about conditional image synthesis with the extracted latent tokens (Section 4.4 and Appendix A.3):\n\n- To realize conditional image synthesis, do the authors mean by training a diffusion model to generate latent tokens with pre-trained generalizable INR models?\n\n- In lines 522-523, \"We drop 10% of class conditions for our model to support classifier-free guidance.\" Why 10%? Does this number percentage matter?\n\nOverall, this paper is a good effect. I will raise my rating if the authors can address my concerns.", "limitations": "The authors haven't adequately addressed limitations and the potential negative societal impact of their work.\n\nI would suggest showing some failure examples, in which the proposed model can not work well, so as to let the community know the boundary of this work.\n\nMoreover, considering the critical importance of AI safety, it could be beneficial to discuss any potential adverse societal impacts that may arise from this work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- **Scope and relevance**: This paper studies an important topic that enables INR with generalization ability. \n\n- **Technical contribution**: The authors present a novel locality-aware INR decoder to improve the expressive power of modulation by learning locality-aware representation from data.\n\n- **Experiments**: The authors conduct extensive experiments and demonstrate significantly improved performances on multiple benchmarks.\n\n- **Clarity**:  The paper is well-written.", "weaknesses": "Some technical details might need further examination.\n \n- **Selective Token Aggregation**: In Section 3.3.1, the cross-attention is applied in extracting a modulation vector $\\mathbf{m}_\\mathbf{v}$  with the coordinate $\\mathbf{v}$ and the latent tokens. Can the cross-attention be replaced with standard transformer attention with input from the concatenation of latent tokens and the frequency features of the coordinate? Can other operations that fusion these two inputs also work well?\n\n- **Multi-Band Feature Modulation**: The authors use a range of frequency bandwidths to predict the details of outputs with a deeper MLP path to learning higher-frequency features. If I am not mistaken, more learnable parameters are introduced into the coordinate-based neural network. Is it possible the improved performance stem from the increased model size?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Standard Transformer Attention for Selective Token Aggregation]**   \nAlthough the standard transformer’s self-attention can be used to predict the modulation vector for a coordinate input, we adopt cross-attention due to the computational efficiency. Using self-attention with the concatenation of latent tokens with the frequency features of the coordinate results in the computational costs with $O( (R+1)^2)$, where $R$ denotes the number of localized latents. Contrastively, using cross attention requires $O(R \\times 1)$. While other operations, such as k-NN, can also be used for selective token aggregation, using cross-attention is a natural and intuitive design choice within a modern deep learning architecture. \n\n\n**[Multi-Band Feature Modulation]**  \nIn Section 4.1, the number of trainable parameters for our framework is 44.14M, while IPC has 43.75M parameters. That is, the increased number of parameters is 0.4M, which is 0.9% parameters of the IPC framework. Although our framework uses 0.9% more total parameters than IPC, the improved performance is significant and not dependent on the increased parameters. To be precise, we increase the hidden dimension of MLP, $d$, for IPC and train the framework of IPC on FFHQ 256x256, as shown in the table below. Increasing the number of MLP parameters also improves the performance of IPC. However, the results show that our performance improvement does not come from a simple increase of trainable parameters but from effective architecture design. \n\n| FFHQ 256x256  | # parameters | d |  R | PSNR | \n|:------|:------:|:------:|:------:|:------:|\n|Ours | 44.14 M | 256 | 256 | **39.88** | \n|IPC [19] | 43.75 M | 256 | 256 | 34.68 | \n|IPC [19] | **65.31 M** | **1024** | 256 | 38.43 | \n\n\n\n**[Training a Diffusion Model]**  \nYes, we utilize a pre-trained generalizable INR model to train a diffusion model for image generation. Specifically, after our generalizable INR is trained on ImageNet 256x256 to extract the localized latents of the images, the extracted latents of images are randomly corrupted by Gaussian noises to train a diffusion model.\n\n\n**[10% Drop of Class Conditions]**  \nWe follow the original paper on classifier-free guidance and conventional settings to train class-conditional diffusion models. While we have not conducted an ablation study on the probability of unconditional training (class condition drop), the original paper on classifier-free guidance describes the following findings: when $p_\\text{uncond} \\in \\{ 0.1, 0.2, 0.5 \\}$ is ablated, $p_\\text{uncond}=0.5$ consistently performs worse than $p_\\text{uncond} \\in \\{ 0.1, 0.2 \\}$. Thus, it has been concluded that only a small portion of the model capacity is needed for the unconditional generation task for classifier-free guidance. \n\n**[Limitations]**  \nWe will add the following discussion about the limitations of our study. Although our framework significantly improves the image reconstruction performance for INRs, there is still room for improvement, especially in high-resolution image reconstruction, such as 1024x1024. Our experiments on novel view synthesis have been conducted only on a category-specific and synthetic dataset such as ShapeNet. Since our framework can be applied to generation tasks, our study is not exempt from generative models' potential negative societal effects.\n"}, {"review_id": "EMeQpeowv9", "reviewer": "Reviewer_9zhe", "summary": "This paper studies the problem of generalizable implicit neural representation. The proposed method combines a transformer encoder with a locality-aware decoder to predict the output with the feature modulation through multiple frequency bandwidths. Experiments are performed on image reconstruction, few-shot novel view synthesis, and image synthesis to show the effectiveness of the proposed method for generalizable implicit neural representation.", "questions": "See weakness.", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n- This paper studies an important problem in implicit neural representation and the motivation is clear.\n\n- This paper is generally well structured and the proposed method is easy to follow.\n\n- Experiments are sufficient to show the effectiveness of the proposed method.\n", "weaknesses": "\n- This paper mainly emphasis on locality-aware, but fails to provide a formal definition of the notation \"locality\". It would be better to provide a formal formulation and some intuitive examples.\n\n- Many recent works tend to explore a hybrid neural representations (e.g., feature maps with small mlps), while this paper only considers the pure MLPs. It would be interesting to have a further discussion about this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Locality: Definition and Intuitive Examples]**  \nWe will include the following additional explanation of the concept of locality for a better understanding of readers. Although it is challenging to provide a formal definition of ‘locality’, we can provide its conceptual description and intuitive examples for clear understanding. Note that the term locality in the operating system field is also informally described. In our study, ’locality’ is a concept that the features/attributes of data points are highly correlated when these points in the input space are located nearby, having a small distance between data points. For example, two nearby pixels in a 2D image have potentially similar colors. Thus, our framework learns to extract a latent corresponding to specific local regions for effective data representation, leveraging the inherent high correlation of features within local regions.   \n\n\n**[Discussion about a Hybrid Neural Representation]**  \nOur framework and recent approaches of a hybrid neural representation commonly extract latents from local regions to predict the features on continuous coordinates. However, our framework does not assume the explicit grid structure of local latents, unlike the recent hybrid neural representations that assume an explicit grid structure of data (e.g., 2D grid for an image or 3D grid for a 3D object) to extract the latents of each region [NewRef: InstantNGP]. Thus, our framework can be viewed as a broader version of hybrid neural representations, capable of learning the local structure across diverse data types.\n\n[NewRef: InstantNGP] Müller, Thomas, et al. \"Instant neural graphics primitives with a multiresolution hash encoding.\" ACM Transactions on Graphics (ToG) 41.4 (2022): 1-15."}]

Author rebuttal:
We appreciate all reviewers's constructive comments to improve our paper.   
We have tried our best to sincerely respond to all concerns and questions.   

Meta-review: This paper received a mixing of ratings with strengths on the novelty of idea, good performance on multiple benchmarks, and weakness on the unclear descriptions (Reviewer AhKU) and definition of generalizable INRs (Reviewer KNfA).  AC has read the reviewers’ feedbacks, the authors’ rebuttals, and the paper. For the concern from Reviewer KNfA on the definition of generalizable INR,  AC agrees with Reviewer QDhX that representing the signal instance with model weights and using latent features can both be considered as INR and thus discounts this factor in making decision. 

As suggested by Reviewer AhKU, the authors need carefully revise the paper to properly addressing writing problem. 

The frequency decomposition mechanism is not very well explained (Reivewer QDhX). The authors have visualized the effect of frequency ordering on the reconstruction of high-frequency details. However, there is still a lack of deep analysis. For example, why the design choice (128,32) shows superior performance? whether a deeper MLP path corresponding the learning of higher frequency features and why (how to demonstrate that)? In addition, the cross-attention for selective token aggregation shares some similar idea with the work “attentive neural processes, ICLR19”. The authors could add the discussion.

Given the strength of the performance and interestingness of the idea, AC reached a decision to acceptance. 
AC strongly encourages the authors to revise the paper to make sure that the writing/presentation in the final version is clear to readers and add deep analysis on whether features align the expectation and whether the content features align with the frequencies of the position encoding.
Decision: Accept (poster)