ASMR: Activation-Sharing Multi-Resolution Coordinate Networks for Efficient Inference
Authors: Jason Chun Lok Li
Steven Tin Sui Luo
Le Xu
Ngai Wong
Venue: ICLR 2024 poster (2024)
Keywords: Implicit Neural Representation
Coordinate Network
Multi-resolution
Efficient Inference
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose the Activation-Sharing Multi-Resolution (ASMR) coordinate network to enhance the inference efficiency of INRs. ASMR achieves high-quality reconstruction results with a low MAC count in various signal-fitting tasks.

Abstract: Coordinate network or implicit neural representation (INR) is a fast-emerging method for encoding natural signals (such as images and videos) with the benefits of a compact neural representation. While numerous methods have been proposed to increase the encoding capabilities of an INR, an often overlooked aspect is the inference efficiency, usually measured in multiply-accumulate (MAC) count. This is particularly critical in use cases where inference bandwidth is greatly limited by hardware constraints. To this end, we propose the Activation-Sharing Multi-Resolution (ASMR) coordinate network that combines multi-resolution coordinate decomposition with hierarchical modulations. Specifically, an ASMR model enables the sharing of activations across grids of the data. This largely decouples its inference cost from its depth which is directly correlated to its reconstruction capability, and renders a near $O(1)$ inference complexity irrespective of the number of layers. Experiments show that ASMR can reduce the MAC of a vanilla SIREN model by up to 500$\times$ while achieving an even higher reconstruction quality than its SIREN baseline.

Published as a conference paper at ICLR 2024
ASMR: ACTIVATION-SHARRE MULTI-RESOLUTION COORDINATE NETWORKS FOR EFFICIENT INFERENCE
Jason Chun Lok Li Steven Tin Sui Luo²; Le Xu' Ngai Wong'
Department of Electrical and Electronic Engineering, University of Hong Kong
2 Department of Computer Science, University of Toronto jasonic1@connede hku.hk stevents. werrents..uu@ee luoêmail www.untooooo...oo. (xule, nwong)&eee beong)@eee.hku.tt hku.hk
ABSTRACT
Coordinate network or implicit neural representation (INR) is a fast-emerging method for encoding natural signals (such as images and videos) with the benefits of a compact neural representation. While numerous methods have been proposed to increase the encoding capabilities of an INR, an often overlooked aspect is the inference efficiency, usually measured in multiply-accumulkke (MAC) count. This is particularly critical in use cases where inference throughput is greatly limited by hardware constraints. To this end, we propose the Activation-Sharing Multi- Resolution (ASMR) coordinate network that combines multi-resolution coordi- nate decomposition with hierarchical modulations. Specifically, an ASMR model enables the sharing of activations across grids of the data. This largely decouples its inference cost from its depth which is directly correlated to its reconstruction capability, and renders near O(1) inference complexity irrespective of the num- ber of layers. Experiments show that ASMR can reduce the MAC of vanilla SIREN model by up to 500x while achieving an even higher reconstruction qual- ity than its SIREN baseline.
INTRODUCTION
Neural networks have been proven to be very effective at learning representations of various modali- ties of data such as images, videos, 3D shapes, neural fields, and many more. particular, Sitzmann et al. (2020); Mildenhall et al. (202 Park et (2019): Li et al. (2024) have demonstrated that sim- ple coordinate networks, taking in a coordinate system and outputting the modality -specifcc data, ex- hibit state-of-the-arr (SOTA) expressivity as implicit neural representation (INR). However, while numerous methods have been proposed to improve the encoding capabilities of an INR, an aspect that is often overlooked is the network's cost of inference] A low cost of inference is of particular importance when the inferencing throughput at test time is restricted by hardware constraints.
Currently, hybrid INRs that make use of explicit representations such Plenoxels (Fridovich-Keil et al. 2022) and Instant-NGP (Müller et al. 2022) are the best at low-cost inference as they remove the need to learn complex neural network. However, we will show in Section 4.4 hybrid methods lose the ability to learn a global implicit representation which is required for tasks such as dataset learning. On the other hand, KiloNeRF (Reiser et al. 202T) and MINER (Saragadam et al. 2022) are purely implicit methods that indirectly reduce cost inference through the use of tiny MLPs. Hao et al. (2022) proposed Level-of-Experts (LoE), which uses shared weights instead of completely independent MLPs to further reduce the cost of inference. However, we analyze and show that existing methods that use an ensemble of weights produce undesirable trade-offs between parameter count and inference cost.
It is remarked that the cost of inference of virtually all existing INR architectures is dependent on both the depth and width of the neural network, which are the direct indicators of the network's
"qqual contribution
In this paper, measure the cost of inference by the number of multilly-acuumultt (MAC) operations required by the network's 's weights. We will use MAC and "the cost of inference" interchangeably.

---

Published as a conference paper at ICLR 2024
expressivity. While methods that distribute individual MLPs or weights across grid of the data can effectively reduce the size of the inference network, the significant increase in parameters goes against the original intention of reducing the overall memory footprint. To this end, we argue that a viable way to genuinely reduce the cost of inference is by decoupling the inference cost from the network depth, which could be achieved by amortizing the per-sample inference cost across the entire data instance. To do so, we combine three ideas: (1) shared activations; (2) multi-resolution coordinate decomposition; and (3) pesttoon-eependent modulations to formulate "hierarchical modulation", resulting in an Activation aetivation-Sharing Multi-RRssoltion (ASMR) network.
As shown in Section 3.3 the immediate benefit of activation sharing is the ability to achieve a near ((1) inference cost with respect to network depth and hence reconstruction quality. This permits an ultra-low MAC model with inference cost going even below 2K MAC. This is roughly the cost of inference of single hidden layer MLP with 32 hidden units, which has an expressivity barely sufficient in representing low-resolution image. We validate the robustness of our method by fitting variety of complex signals, and show that the decoupling effect of ASMR does not come at the cost of affecting the model's original expressivity. In particular, ASMR even improves upon the original SIREN model on fitting megapixel images (Section 4.3) and an entire dataset (Section 4.2) Lastly, we highlight the benefit of using ASMR over hybrid representations (Section 4.4). We demonstrate that ASMR permits the learning of global latent structure of signals, enabling it to encode entire dataset with single INR, which oftentimes is infeasible for methods employing explicit features.
To summarize, this paper makes four main contributions: (i) We propose novel hierarchical mod- ulation scheme that efficiently incorporates multi-resolution coordinates with minimal parameter increase. (ii) We develop the activation-sharing ASMR, the first INR decouple MAC from its depth. This leads to a near ((1) complexity regardless of the number of layers. (iii) The proposed ASMR achieves better quality reconstruction with 500x fewer MAC than a vanilla SIREN in high- resolution image fitting tasks. (iv) ASMR remains purely implicit, allowing it to handle tasks that require global latent structure, such as meta-learning.
RELATED WORK
Multi-resolution INR Multi-resolution representation of signals has always been efficient way to learn INRs. Training INRs to reconstruct a progressive resolution of the original signal has been shown to greatly reduce training speed and the number of parameters. For instance, [Saragadam
iet al. (2022) utilizes Gaussian/Laplaciann pyramids to allow INR learning of the much "simpler" residual signals, while while|Lindell bands learned by each INR layer in a coarse-to-fine manner for better interpretabilityy Instead of training the INR in a multi-resolution manner, some methods have found benefits by simply using multi-resolution coordinate inputs to an INR. Along this line. Müller et al. (2022); Takikawa et al. (202T): Dou et al. (2023) belong to a family of hybrid explicit representation INRs that learn a grid of embeddings at multiple levels of resolution, and concatenate them when feeding into INR. Lastly, multi-resolution coordinates can also be used to partition the input space into grids, where each grid has its specific tiny MLP (Reiser et al. 2021) weight layer (Hao et al. 2022).
al. (2022) and [Shekarforoush et al. (202)) constrain the frequency
Modulated INR Perez et al. (2018) proposes to modulate a network's activation with a simple affine transformation. Although it is designed enhance visual reasoning abilities, this technique has proven valuable in the context of INRs. For instance, [Chan et al. (202T) injects modulations to a generative adversarial network (GAN) generate samples. [Skorokhodov et al. (202T) also utilizes modulation in generative tasks. They employ low-rank modulation which uses parameters predicted by the hypernetwork as inputs. For minimal overhead, our approach considers bias-only modulations instead. Our method of hierarchical modulations takes inspiration from Dupont et al.) 2022a b): Bauer et al.) (2023), which proposes to store data in the form of a data-specific bias vector that serves as the modulator of a SIREN model meta-learned on the entire dataset. In our case, instead of storing an explicit modulation for the data instance, we generate, at inference time, hierarchy of prrdiinate-dppnneenn modulations for injecting into each network layer. Later, in Section 4.4] we demonstrate that the data-specific modulation can be used in conjunction with our proposed hierarchical modulation.
2

---

Published as a conference paper at ICLR 2024
CLU
17A4
RUD
10:0
(22)
OLT
(72)
X00
1000
FLCK
11.5
(22
1751
XL1
(V
(22)
1721
Levelt Coondratee
(+)
Upiamp
Da,, Sps
IW's
IN's
W
Uppampte (R)
fyy Sy
Mie
X
(c)
Overall framework for ASMR. (a) Multi-resolution Coordinates: The original coordi- nates are decomposed into multiple hierarchical levels, each with its own set of axes. To illustrate repetitive patterns, the coordinates are folded into a higher-dimensional space. (b) Hierarchical Modulation: The number of layers in the model is equal to the number of hierarchical levels. At each level (except level-O), the coordinates are first projected to the hidden dimension via modula- tors, then added elementwise to the activations of the corresponding layer. (c) Activation-Sharing Inference: The MAC-saving activation-sharing inference performed by utilizing upsampling op- erations on both modulations and hidden features. Here, Bz, represents the base level-i along x-axis, while Cz, denotes the cumulative product of bases along r-axis from hierarchical level-O to level-i (i.e. Bz, Br--1 .... Bzz). uniform base B is used in this example.
Figure 1:
Low-MAC INR
To our best knowledge, no existing INR architecture can decouple the cost of inference from the depth of the network, where the depth is often the main determining factor of the
INR expressivity (Yüce et al. 2022) and hence signal reconstruction quality. general, existing
SOTA INRs that achieve low inference cost either do so through learning explicit representa-
tion (uüller et al., 2022 Takikawa et al.) 202T) or learning multiple tiny MLPs (Reiser et al.) 202T) reduce the required size of the INR, or through weight sharing (Hao et al., 2022). Oftentimes, these methods achieve a lower cost of inference at the expense of largely increased parameter count or decreased reconstruction quality (by reducing hidden dimension). Furthermore, as stated in Reiser et al.] (2021), methods that involve training multiple sets of weights often necessitate the use of carefully crafted CUDA kernels to handle non-uniform sampling of the training data. We highlight that ASMR does not suffer from such parallelization constraints due to its nature simple modularized addition to a vanilla backbone model. While (Skorokhodov et al. (202I) also octivation-sharing like to INR inference costs, they fail to decouple the inference cost from model depth, achieving only minor savings. They utilize multi-scale coordinates, which have progressively increasing resolutions along the network's depth. This significantly increases the mod- ulation cost in the subsequent layers. Contrarily, our ASMR decomposes coordinates considering hierarchical and periodic data patterns, resulting in a complexity reliant solely on the partition basis.
3 METHOD
3.1
MULTI-RESOLUTION COORDINATES
The idea of multi-resolutiin coordinates has been introduced in previous literature (Hao al. 2022 Bauer et al.) 2023). but in different contexts. Here, we revisit this idea and provide more intuition underlying it. To begin with, Bauer et al. (2023) interprets coordinate decomposition change of
3

---

Published as a conference paper at ICLR 2024
base for the original global coordinates. It discovered that changing the coordinates into binary representation gives better results terms of PSNR on image fitting tasks. However, the work does not offer further explanation for this observation. On the other hand, Hao et al. (2022) generalizes this coordinate decomposition motivated by the hierarchical and periodic structure of data. Both explanations refer the same concept that refer to coordinate decomposition.
Expanding on this, we offer an additional perspective that encompasses both explanations. As de- picted in Figure a decomposing global coordinates can be interpreted as an axis partition op- eration, wherein the axes of the original coordinate system (z, y) are partitioned into shorter axes multiple levels ([z0. 21. 22], [yo, y1. 32). with 2; and y/ denoting the base of partition. This can be seen as mapping the original coordinates into higher-dimensional space, enabling subsequent coordinate networks to analyze the repetitive pattern of data. Specifically, given a coordinate
(0. 1,...... N 1}. where N is the input data size (e.g. sequence length of audio input), can be decomposed in I hierarchical levels:
decomp
x
{Io: T1, ..,...,,I......................, IL-1) ; I (O. B, -1]
where B; is base i-th level and N I Bg. We call [Bu,... B1-1] the bases of partition.
Let C. = IT - j be the i-th cumulative base and G, = [N/C,] be the grid size of the partitioned axis at the i-th level. The decomposed coordinates given by z; [x/G,] mod B,. Such coordinate decomposition creates a hicrarchical data representation, starting from a coarse level and progressively refining finer level. This computation is only performed once at the beginning of training, making an efficient way to encode multi-resolution information into the network.
3.2 HIERARCHICAL MODULATION
Our method encodes information in each resolution level (except level-0) using independent mod- ulators as illustrated in Fig. [b), where the number of modulators is equal to T. 1. The output activations from each modulator are then treated as the bias of the corresponding resolution level. Specifically, an L-layer ASMR with hidden dimension d; in the i-th layer is defined as follows:
TO
2 = (W b; + M,(I:))
i i=1...,,--
2L WL2L-1
where 2, E RR,, W, E x4: 1, b, E R4,, M.(.) R + Rd denote the activations, weight matrix, bias and modulator the i-th layer, respectively. (Ix) represents the nonlinear activation. In this paper, we employ SIREN (Sitzmann et al. 2020) as our backbone where oxx) = sin (wp+r) and wo IR+ a predefined positive scalar control the frequency of the model.
Prior work (Chan et al. 2021 Dupont et al. 2022a by applies modulations to INRs in a global manner, where low-dimensional latent vector mapped set of global modulations. These modulations are then applied to hidden activations through scales and shifts. Herein we adopt similar approach to Dupont et al. (20226), where modulation applied as bias vector. However, instead of storing set of global modulations per INR, we generate "local" modulations by utilizing a position-dependent modulator whereby independent modulators are employed for each resolution level. In particular, the modulator is taken in its simplest form as a projection matrix given by M.(-) == wo W., E xd; where wo is set to be the same as the SIREN backbone and WM, ~ U(- (1/V1//d- /1/dn). The modulator can be interpreted as storing the sigee-modulations for each hierarchical level. This hierarchical modulation technique serves a simple but effective way to utilize multi-resolution decomposed coordinates. It improves reconstruction fidelity with only a negligible increase in parameter count.
3.3 ACTIVATION-SHANE INFERENCE
The fusion of multi-resolution coordinates and hierarchical modulation permits the sharing of ac- tivations across data points. In particular, instead of inferring all N data points on every layer,
For the ease of illustration, assume ID data. However, the idea is easily generalized to multi- dimensional data.
4

---

Published as a conference paper at ICLR 2024
*ie KIloNeRF
KiloNeRf
++-LEE
++-LoE
SESIREN
ASMR
ASMR
Hidden Layer
MAC
200
13
400
200
400
#Param (K)
Figure 2: MAC-#Params curves SIREN, ASMR, KiloNeRF and LoE. We highlight that ASMR reduces the MAC of SIREN models of width 256 by 50 ~ 200x to near the theoretical limit of single-layer MI.P with 32 hidden units (red dotted line).
each hidden layer and modulator only has to infer on C, data points and B; grids, respectively, as illustrated in Fig. Tc). By adopting activation -hariinn inference, Equation could be rewritten as:
= (o( b,, B.) R(M;(x.))) C.-))
(2)
here, 2xx r) and R(x,7 r) represent the nearest neighbor and tile replication upsampling of the input v, performed times along the corresponding axis, respectively. Such upsampling operations be easily implemented using einops (Rogozhnikov/ (022). This amortizes the inference cost across the entire data, making the per-sample inference of ASMR dependent only the width of the model (i.e. number of hidden units) and independent of the depth, which is contrary and superior other common INR architecture families such as vanilla SIREN. LoE, and KiloNeRF. This is particularly important because the number of layers directly correlates to the reconstruction quality.
Proposition 1. ASMR decouples its inference cost (in terms of MAC) from its depth L. and conse- quently its corresponding reconstruction quality.
Proof. Suppose the ASMR model has L layers and each layer-i inferences with MAC of M,. For notational convenience, we assume that the bases of partition at each level of the ASMR model to be the samel (i.e., uniform bases), namely, B. Since the cumulative product of all bases must be equal to N, i.e. the partitioned grid of the highest resolution must be equal to the resolution of the data, gets L. log N.
For any vanilla MI.,, one can approximate its inference cost with the equation MAC - N Fixing the hidden dimension of each layer gives us approximation of M, with M max (M,, ,,,...,,My Then, we can simplify the expression for MAC into MAC S NLM and the per-sample inference cost is MACsample & LMM
For ASMR, layer-i only has to infer B times. Hence, the inference cost of ASMR is given by:
L MB
BL
< M Σ B MAC B'M
(geometric sum)
B -
MB
MB
B MN 2MN
B²
Blogn
B B-1
B
B-1
since B > This gives per-sample inference cost of ASMR of MACsample Ro. 2M which is solely dependent on the hidden dimension (width) of the model.
The non-uniform hase case will lead to much more tedious form of expression, but we note that the idea of amortizing per-sample inference could he easily generalized.
5

---

Published as a conference paper at ICLR 2024
Table 1 and Figure 2 summarize the MAC-to-parament count relationships among low-inference cost architectures analytically and empirically, respectively. For KiloNeRF, its inference cost increases with network depth and hence its overall signal-fitring ability. On the other hand for LoF, its multi-resolution weight-sharing approach induces a nega- tive correlation between its parameter count and depth. This is because an increase in network depth must be accompa- nied by decrease in the number of weight tiles a partic- ular layer by a multiplicative factor that depends on the way of partition. LoE asserts that the cumulative product of its weight tile dimensions equal to the data size, and the smallest weight tile size is 2. Hence, the maximum value of T, is log2 N. Note that both LoE and KiloNeRF reduce their inference cost by increasing their grid resolutions which leads to significant increase in their parameter counts.
Model
MACs
#Params
SIREN KiloNeRF
O(L)
O(L)
O(L)
O(L)
LoE
O(L)
O(log, N L)
ASMR
0(1)
O(L)
Table 1: Relating parameter count and MACs to network depth when network width is fixed.
4 EXPERIMENTS
All codes are implemented using the PyTorch (Paszke et al. 2019) framework. The baselines are adopted from the official codes provided by the authors, except LoF (Hao al., 2022) which has no publicly available code, and KiloNeRF (Reiser et al. (021) and Instant-NGP (Müller et al., 2022) which the original implementation could not conveniently accommodate for our use cases.
4.1 ULTRA-LOW MAC TRAINING
To test the performance of ASMR under ultra-low inference cost scenario, train the models to fit a 512x512 gray-scale Cameraman (Van der Walt et al.) 2014) image. We train 2 sets of SIREN models and apply ASMR show that our method could achieve more than 50x reduction in inference cost while also increasing the model's reconstruction quality, For the 3-layer SIREN, our ASMR uses partition bases of [8, 8, 8), while the 4-layer SIREN uses partition bases of [4, 4, 8). As the only additional weights are contributed by the single-layer linear modulators, the increase in parameter count nearly negligible, especially when considering the significant increase in PSNR for a 4-layer SIREN mode from 32.4dB to 37.8dB. It should also be noted that the reduction in inference cost results in a decrease in training time by 46% from 389s to 210s for the 3-layer SIREN and by 45% from 509s to 280s for the 4-layer SIREN.
ASMR
Model (Rlayers)
#Params (K)
MACs (K)
(d)
Giohek oF
KiloNeRF (3)
250
0.98
34.35
LoF (4)
126
2.07
33,27
SIREN (3)
66.8
66.3
28.62
SIREN (4)
133
132
32.37
28
ASMR (3)
67.8
1.29
31.44
500
1500
2500
3500
ASMR (4)
134
1.35
37.84
Table 2: Ultra-low MAC fitting results on Cam-
Figure 3:
Comparison of MAC-PSNR curves.
eraman image.
Circle's 's area is proportional to #Params.
We also compare ASMR with SOTA low inference cost methods, namely KiloNeRF and LoE, under the ultra-low MAC regime. To simulate a real-life situation where hardware resources are highly restrictive, we limit the parameter counts to less than 150K except for KiloNeRI It should be noted that even though KiloNeRF and LoE could further reduce MACs by increasing their grid resolutions, our experiments show that they fail to reach > 30dB PSNR at a low parameter count. Bigurndextends the results of KiloNeRF. LoE, and ASMR from Table to include their full MAC- PSNR curves at 3-7 hidden layers. We highlight that the ASMR's near-constant trade-off between inference footprint and reconstruction quality is superior to KiloNeRF's positive linear and LoE's negative linear trade-off.
*Our experiments in Appendix A show that a KiloNeRF configuration with less than 150K parameters does achieve 30dB PSNR.

---

Published as a conference paper at ICLR 2024
Latency (ms) +
Method
#Params (K)
MACs (K)
PSNR (dB)t
SSIM
Server
Desktop
Raspberry
FFN
132.4
131.8
31.037=3.233
0.84140.005
369.5
2859.2
8975,4
SIREN
133.1
132.4
32.398=3.333
0.890::00037
441.4
4400.0
13476.6
MFN
134.7
133.4
27.565-3.132
0.782++0.078
459.9
10291.4
(Failed)
WIRE
136.8
135.9
31.266=2.055
0.85410.066
2296.5
(Failed)
(Failed)
ASMR
134.7
1.85
33.087-2.704
0.892++0.021
143.9
957.7
3202.0
Table 3: Image fitting results (mean = std.) on the Kodak dataset. The mean + std. across all images is reported. ASMR achieves the best results in terms of both PSNR and SSIM while having the lowest MACs and latency across various hardware platforms. The reported latency is an average across 10 runs. Processes that are too computationally intensive and result in hanging or being automatically killed due to excessive RAM usage are indicated by (Failed).
4.2 NATURAL IMAGE FITTING
To test the robustness of our method on a wide variety of natural images we conduct image fitting on the entire Kodak (Fastman Kodak Company 1999) dataset. Each image is of either 512x or 768 x68x512 resolution with RGB channels. The mean and standard deviation across all images are reported. We benchmark ASMR against various basclines, including Fourier Feature Net- work (FFN) (Tancik et al.) 2020 SIREN (Sitzmann et al. 2020), MFN (Fathony et al. 2020), WIRE (Saragadam et al., (023). Table Jable demonstratess that ASMR surpasses all other baselines in both PSNR and SSIM metries with matching number of parameters (approximately 130K), while operating in the ultra-low MAC regime, highlighting the efficiency of our approach. For full imple- mentation details and qualitative results, please refer to Appendix C and Appendix L respectively.
To demonstrate the improved inference speed of our ASMR method on resource- resource-limited devices, we also measure the rendering latency various hardware platforms. These platforms range from high-end to low-end CPUs. As shown in Table ASMR consistently speeds up SIREN by > 3x in terms of latency, and the speed-up becomes more significant when tested on devices with reduced computation power (viz. > 4x on Raspberry Pi).
4.3
OTHER MODALITIES
Method
#Params (K)
MACs (K)
PSNR (dB))
Method
(M)
(M)
PSNR(dB)*
Instant-NGP
32,9
1.0
07.30:33.74
Instant-NGPP
1.229
0.006
33.61
0.892
KiloNeRF
33.4
8.16
42.20:1:2.39
KiloNeRF
1.459
0.004
27.91
0.804
SIREN
33.4
33.0
46.21
SIREN
1.054
1.054
31.66
0.825
SIREN-BIG
4.205
4.205
35.66
0.895
ASMR
33.8
1.0
61.66:11.61
ASMR
1.059
0.006
33.10
0.857
ASMR-BIG
4.215
0.008
38.73
0.938
Table Audio fitting results on the Lib- riSpeech dataset. The mean + std. across all samples is reported.
Table 5: Megapixel image
fitting on the
8192x8192 Pluto image.
To showcase the versatility of ASMR across different modalities, we present additional experimental results on audio, megapixel images, video, and 3D shapes.
Audio. We evaluate ASMR's 's performance on an audio-fitting fitting task, using the LibriSpeech (Panay- otov et al. (2013) dataset as the benchmark. Our findings suggest that ASMR is particularly effective with low-dimensional data like audio. As shown in Table ASMR, with matching parameter count of around 33K, significantly outperforms other baselines, including Instant-NGP, KiloNeRF and its SIREN counterpart, with an absolute gain of over 14dB in PSNR, while also exhibiting the lowest MAC of around IK. Moreover, we also find that ASMR converges much more quickly as compared to the standard SIREN model. For detailed settings, please refer to Appendix D
Megapixel Image. Learning to fit a high- high-resolution megapixel image an example where de- coupling network parameter count from inference cost is particularly important since oftentimes a significantly larger model is required. Here, we trained our models to fit the 8192 8192 RGB Pluto (NASA 2018) image. To achieve >30dB PSNR, SIREN model must have at least 1M pa-
7

---

Published as a conference paper at ICLR 2024
SIREN (1.3M MACS) KiloNeRF (6.2K MACS)
ASMR (45K MACs)
ASMR (5 MACs)
PSNR (dB)
32
29.56 + 1.21
31.60 0.75
31.51 1.40
Figure 4: Visual comparison of video fitting results. The mean together with the standard deviation of PSNR across all frames of each model is labeled the top-left corner of each image. Qualita- tively, it is clearly seen that SIREN reconstructs the video with serious smoothing effects while both ASMR models are able to learn the finer details (e.g. whiskers).
rameters (6 layers with 512 hidden dimensions), bringing the cost of inference to more than IM MACs. However, by applying ASMR to the same SIREN model, we not only could reduce the cost of inference by ~ x x to only 6K MACs, but also increase the reconstruction quality from 3 31.66dB to 33.10dB PSNR. The reduction in MAC is more significant (> 500x) when applying ASMR to a larger SIREN with 1024 hidden units (SIREN-BIG), improving the PSNR by more than 3dB. We also benchmark ASMR against the KiloNeRF and Instant-NGP models at similar inference costs and parameter counts. We found that KiloNeRF does not achieve reconstruction quality compa- rable to the other models, while Instant-NGP reaches similar levels in both PSNR and SSIM. For detailed settings, please refer to Appendix B
Video. Compared to megapixel images, video data is a complex signal with even more pixels. The additional time dimension also differentiates the nature of video data from that of an image. Here, we follow [Sitzmann et al. (2020) and train our models on an RGB cat video (Ehlers) (219) with 300 frames of 512x512 resolution. The two different configurations of ASMR share the same set of partition bases but have different permutations. This results in varying inference costs, as discussed Appendix K frame where the cat relatively static is visualized in Figure Despite the original SIREN having higher PSNR score, we can clearly see that ASMR reconstructs the video a much higher visual fidelity with fine details encoded. While SIREN has shown significant hypothesize that
improvement from the original ReLU MLP with the use of sine activations, SIREN still suffers from a certain degree of spectral bias due to a fixed scaling factor of wiy = 30. On the other hand, KiloNeRF, fitting independent MIPs, exhibits tiling effects where the borders can be seen in the enlarged section. For additional results on the UVG dataset and detailed model configurations, please refer Appendix
G and F respectivvely
3D Shapes. A direct extension of ASMR to 3D data is demonstrated by training it to fit occupancy grids. One random sample from each category of ShapeNet is se- lected, amounting total of 55 objects, and evaluated with pntersection-over-mme
Method
(M)
(M)
ToU
SIREN
2.100
2.10
89 93.27=6.33
ASMR
0.039
2.12
89.57=6.72
(IoU). A 10-layer SIREN with 512 hidden units, along with its ASMR counterpart with basis of partition of 2, are trained over 200 epochs. All hyperparameters remain constant across both models (see Appendix H for details). The results in Table 6 indicate that ASMR retains the reconstruction quality of SIREN, while significantly reducing inference cost (> 50x MAC reduction). However, it is worth noting that ASMR performs significantly worse when trained on continuous 3D signals such as signed distance fields (SDFs) (refer to Appendix M for qualitative results). This suggests that the coordinate decomposition in ASMR has a strong bias towards rasterized data.
Table 6: 3D shape reconstruction results (mean + std.) on the ShapeNet dataset.
8

---

Published as a conference paper at ICLR 2024
4.4 META-LEARNED INITIALIZATION
Figure 5: Selected reconstructions from the CIFAR-10 dataset by taking 3 gradient updates from the meta-learned initialization of (Top) ASMR (Bottom) Instant-NGP.
One distinct advantage that sets ASMR apart from explicit or hybrid INRs, like Instant-NGP (Müller et al. 2022), is its ability to handle tasks that require a global implicit representation, such as gen- erative tasks and tasks involving the use of meta-learning. Here, we follow the setting described in Dupont et al. (2022b) to meta-learn INR for entire dataset and learn an instance-specific global latent vector of size 128 to encode each image. Each latent vector is then injected into the INRs as modulation. Our experiment is conducted the CIFAR-10 dataset. Fig. 5 displays the selected reconstructions after taking 3 gradient steps from the learned meta-initialization. It is evident that additional instance-specifie modulations can be used in conjunction with our hierarchical modula- tion. Furthermore, it also demonstrates that the multi-resolution hierarchical structure of ASMR can be generalized to effectively encode shared structures across images. In contrast, Instant-NGP fails to learn the shared structure. We argue that this is primarily due to the distortion caused by the highly nonlinear coordinate transformation to an unbounded input space resulting from the hashed encoding in Instant-NGP. For implementation details, please refer to Appendix [
5 LIMITATIONS AND DISCUSSION
Incorporating hierarchical modulations into SIREN introduces beneficial inductive bias, leading to higher fidelity reconstructions of rasterized data. However, this also causes AMSR to struggle with smooth signals, such SDFs. The grid-like symmetry bias results in noisy artifacts, especially noticeable when visualizing smooth 3D objects. Furthermore, it also disrupts the clean analytical gradients typically possessed by SIREN. Exploring methods to smoothly extend ASMR to continu- ous data would be an interesting direction for future research. It is also worth noting that grid-based maintain fixed MIP's width and depth while enhancing the model's
INRs like Instant-NGP expressivity by enlarging the hash grid embedding. Although ASMR's 's expressivity is not tied to the MLP's depth, still depends on the width of the network. Therefore, exploring methods that enable purely implicit INRs to also separate expressivity from the network's width could be valuable.
6 CONCLUSION
A novel Activation-Sharing Multi-Resolution (ASMR) coordinate network has been proposed which operates very low inference costs while maintaining all benefits of an implicit representation. This model combines multi-resolution coordinate decomposition with hierarchical modulations, to allow for the sharing of activations across data grids and the decoupling of inference cost from network depth and reconstruction capability. As result, ASMR achieves near O(1) inference complexity regardless of the number of layers. We demonstrate that ASMR is the only model, among all baselines, that can work in the ultra-low MAC regime with less than 2K MACs, while achieving a reconstruction result of >30dB PSNR on low-resolution RGB images. Furthermore, ASMR outperforms its SIREN counterpart large-scale megapixel image fitting tasks with the same parameter count, while significantly reducing MACs by 500x.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by the Theme-based Research Scheme (TRS) project T45-701/22-R, and in part by ACCESS- Al Chip Center for Emerging Smart Systems, sponsored by InnoHK funding, Hong Kong SAR.
REFERENCES
Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.03313, 2023.
Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5799-5809, 2021.
Yishun Dou, Zhong Zheng, Qiaoqiao Jin, and Bingbing Ni. Multiplicative fourier level of detail. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1808 1817, 2023.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. arXiv preprint arXiv:2201. 12204, 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. Coin++: Neural compression across modalities. Transactions Machine Learning Research, 2022b.
Eastman Kodak Company. Kodak lossless true color image suite. http::/rkk.us/graphhie kodak/] 1999. [Accessed 14-08-2023).
Magda Ehlers. The full facial features of pet cat, 2019.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2020.
Sara Fridovich-Keil. Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo
Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5501-5510, 2022.
Zekun Hao, Arun Mallya, Serge Belongie, and Ming- Liu. Implicit neural representations with levels-of-experts. Advances in Neural Information Processing Systems, 35:2564-2576, 2022.
Jason Chun Lok Li, Chang Liu, Bínxiao Huang, and Ngai Wong. Learning spatially collaged fourier bases for implicit neural representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 13492-13499 2024.
David B Lindell, Dave Van Veen, Jeong Joon Park, and Gordon Wetzstein. Bacon: Band-limited coordinate networks for multiscale scene representation. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 16252-16262, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller. tiny-cuda-nn, 4 2021.
Einy-cuda-nn
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Transactions Graphics (ToG), 41 2022.
NASA.
True colors of http://///lallsss...........ooos brrerc//DDoss../no////t/tttrrnnr
2018.
10

---

Published as a conference paper at ICLR 2024
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an corpus based on public domain audio books. In 2015 JEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210, IEEE, 2015.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024 8024-8035. Curran Associates, Inc., 2019. URL http: / http://pprss...ppp......eeeeeaap
Library.
pdf
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 14335-14345, 2021.
Alex Rogozhnikov. Einops: Clear and reliable tensor manipulations with einstein-like notation. In International Conference on Learning Representations, 2022. URL [RLLLtttp:://pereeviee
Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard G Baraniuk, and Ashok Veer- araghavan. Miner: Multiscale implicit neural representation. In European Conference on Com- puter Vision, pp. 318-333. Springer, 2022.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507-18516, 2023.
Shayan Shekarforoush, David Lindell, David J Fleet, and Marcus A Brubaker. Residual multiplica- tive filter networks for multiscale reconstruction. Advances in Neural Information Processing Systems, 35:8550-8556 2022.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10753-10764, 2021.
Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler. Neural geometric level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu. scikit-image: image processing in python. PeerJ, 2:e453, 2014.
11

---

Published as a conference paper at ICLR 2024
Gizem Yüce, Guillermo Ortiz-Jiménez, Beril Besbinar, and Pascal Frossard. A structured dictionary perspective on implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19228-192222 2022.
12

---

Published as a conference paper at ICLR 2024
A MODEL CONFIGURATIONS IN FIG.
The idea of Fig. 2 is to show that our ASMR method allows ultra-low MAC inference and near- constant MAC when increasing the number of layers and hence parameter count. This desirable property of an INR as increasing parameter count is direct indicator of its reconstruction quality.
We compare with two other SOTA INR frameworks that reduce inference cost, namely the Kilo- NeRF family and the LoE family. For all 4 models, we plot their MAC-parameter curve by fixing hidden dimension and increasing the number layers progressively. For a fair comparison, we tune the hidden dimension for each model such that the number of parameters is roughly in the same range.
The configurations of all 4 models are summarized in the following table. The configuration of the SIREN model is identical to that of the ASMR model. Note that the "Partition bases (B,)" for LoE is equivalent weight tile dimensions while that for KiloNeRF is equivalent to grid dimension. Other hyperparameters such as learning rates and position encodings are the same as those presented in the next section.
Model
hidden_dim
L
Partition bases (B,)
Param (K)
MACs (K)
256
7
N/A
329
323.0
256
6
N/A
263
264.2
SIREN
256
5
N/A
197
198.4
256
N/A
132
132.6
256
3
N/A
66
66.8
256
7
[2,2,2,,,222.88]
333
1.6
256
6
[4,2,2,2,2.8]
267
1.6
ASMR
256
5
[8,2,2,288]
200
1.6
256
4
[4,4,4,8]
134
1.3
256
3
[8,8,8]
68
1.3
50
10
[2.2...222.2,2222.22]
87
21.8
50
9
[4,2,2.2,22,,2222]
98
19.3
LoE
50
8
[4.4,2,2.222.22]
118
16.8
50
[4,4,4,2,2,2]
138
14.3
50
6
[4,4,4,4,2]
158
11.8
32
6
[16]
361
5.5
32
5
[16]
293
4.4
KiloNeRF
32
4
[16]
225
3.4
32
3
[16]
158
2.4
Table 7: Model configurations of Figure
B IMPLEMENTATION DETAILS OF ULTRA-LOW MAC EXPERIMENT
All models are trained for 10K iterations and configured optimally to have inference costs as low as possible while maintaining low parameter count and having least I hidden layer.
13

---

Published as a conference paper at ICLR 2024
Learning Rate For KiloNeRF, we follow the learning rate stated in Reiser al. (2021) and use a starting learning rate of 5e-4. For both SIREN and LoE, we did a simple grid search over the set {le-2, 1e-3, le-4) and realized learning rate of le-4 for SIREN and le-2 for LoE the best. For all models, we use the cosine annealing scheduler with minimum learning rate of 1e-6.
Position Encoding Both KiloNeRF and LoE use position encoding as the input to their first layer. We follow the original setting in their papers and set the number of frequencies to 10 for KiloNeRF and for LoE.
The optimal set of hyperparameters obtained in this experiment is also used for the other experiments in this paper.
The following table presents the "boundary" configurations we tested for KiloNeRF and LoE. We show that reducing KiloNeRFFs parameter count number closer our SIREN baseline while maintaining low MAC leads to sub-30dB PSNR. This explains why we chose to use the config- urations the second row for comparison T We also show that further reducing the MACs of LoE while not decreasing parameter count (which makes it unfair to compare with our SIREN (4) and ASMR (4) model) will require reducing the number of layers from 4 to However, this implies that with very small number of hidden units (8), the parameter count is still way our SIREN/ASMR baseline.
Model
hidden_dim
Grid Sizes/ Bases
# Param (K)
MACs (K)
PSNR (dB)
KiloNeRF
8
16x 16
108.8
0.425
23.05
KiloNeRF (in Table 16
3
16x 16
250.1
0.977
34.30
LoE
8
[[32,16], [32,16]]
294.9
0.361
39.47
LoE (in Table
24
[[8,8,8], [8,8,8]
126.0
1.992
36.24
Table 8: Boundary configurations and results for KiloNeRF and LoE.
We also include the detailed model configurations of Figure
which is extension of Table
Model
hidden dim
L
Partition bases (B;)
# Param (K)
MACs (K)
256
6
[4,2,2,2,2,8]
267
1.60
256
5
[8,2,2,2,8]
200
1.60
ASMR
256
4
[4,4,4,8]
134
1.30
256
3
[8,8,8]
68
1.30
24
7
[4,4,4,2,2.2]
38.6
3.72
24
6
[4,4,4.4,2]
43.2
3.14
LoF
24
5
[8,4.4,4]
80.0
2.57
24
4
[8,8,]]
126
1.99
16
6
[16]
459
1.79
16
5
[16]
389
1.52
KiloNeRF
4
[16]
320
1.25
16
3
[16]
250
0.98
Table 9: Model configurations for Figure
14

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS OF NATURAL IMAGE FITTING EXPERIMENT
All models are trained for 10K iterations and have around 130K total parameters. FFN is basically ReLU MLP with Gaussian random Fourier features. For MFN,
the FourienNet instantiation as described in Fathony et al. (2020). Below is the complete set of hyperparameters for all models:
FFN: Ir=le-3, embedding size= 128, o 10, 3 layers, 256 units
SIREN: Ir=le-4, 4i 30, layers, 256 units
MFN (FourierNet): 1r=1e-2, input scale=256,
scale=1, 4 layers, 256 units
WIRE: Ir=le-3, wo = 20, So = 20, layers, 212 units
ASMR: 1r=le-4, w0 30, bases=((4.44..8, [4,4.6,8)). layers, 256 units
Latency results are gathered from various hardware platforms. These include server-level AMD 7413 2.65GHz 24-Core CPU, a desktop 3.1 GHz Dual-Cor Intel Core i5 CPU, and a Quad-core Cortex-A72 (ARM v8) 1.8GHz CPU of Raspberry Pi 4 model B.
D IMPLEMENTATION DETAILS OF AUDIO EXPERIMENT
The first 100 samples with a minimum duration of 2 seconds are selected from the test-clean split of the LibriSpeech dataset. All models undergo training for 10K iterations at learning rate le-4, using only the initial seconds of each sample. The full set of hyperparameters is detailed below:
Instant-NGri levels=7, features per level=2, size of hash map=216, base resolution=125,
finest resolution=8000, per level scale=2, layers, 64 units
KiloNeRF: number of frequencies=12, 12, grid dimensions=[4), 5 layers, 48 units
SIREN: we 30, layers, 128 units
ASMR: wp 30, bases=[10, 10, 16, 20), layers, 128 units
E IMPLEMENTATION DETAILS OF MEGAPIXEL IMAGE FITTING EXPERIMENT
All models trained for 500 epochs. We count epoch each time when all pixels in the Pluto image have been sampled once. For KiloNeRF, we randomly sample 222 4, 194, 304 per training step, while for NGP/SIREN/ASMR, we randomly sample 218 262, 144 pixels per training step. We train these four models on single RTX3090. For SIREN-BIG and ASMR, we sample 22 1, 048, 576 pixels per step and train the models on two RTX A6000s.
The full set of hyperparameters is detailed below:
Instant-NGP: levels=16, features per level=2, size of hash map=216, base resolution=16,
finest resolution=4096 layers, 64 units
KiloNeRF: number of frequencies=10, grid dimensions=[16, 16), 6 layers, 32 units
SIREN: Wa = 30, layers, 512 units
ASMR: wo 30,
layers, 512 units
SIREN-BIGE wi 30, 6 layers, 1024 units
ASMR-BIG: we = 30, base==[[44444422,6666444444
6 layers, 1024 units
In Figure 6. we visualize the reconstructed image at 10 training epochs for all models and realize that ASMR, compared to its SIREN counterpart, can learn the high-fidelity features faster in terms of training steps. This is similar to that of Instant-NGP. On the other hand, KiloNeRF exhibits very blurry tiling effects at such early stage. Combining the findings from Takikawa et al. (2021) which also uses a set of multi-resolution embeddings, we believe that this is a benefit contributed by the multi-resolution coordinate system.
We also attempted to train a LoE model but the lack of original code made the training for megapixel images prohibitive (days NVIDIA RTX3090). As random sampling is required for training
15

---

Published as a conference paper at ICLR 2024
KiloNeRF
Instant-NGP
SIREN
ASMR
100 epochs
500 epochs
Figure 6: Comparison of reconstruction quality of the Pluto Image 100 and 500 training epochs.
megapixel image (as well as other large-scale data formats such as video), carefully written CUDA routines become necessary for the parallel training of weights. In order to match the low MACs of our ASMR model while maintaining a similar parameter count, the only possible configuration of LoE is either have two hidden layers of 16 hidden units and weight tile dimensions of 32 32, 16 x 16, and 16 x 16, or a single hidden layer and only 4 hidden units.
F IMPLEMENTATION DETAILS OF VIDEO FITTING EXPERIMENT
All models are trained for 200 epochs. We count epoch each time when all pixels in the Cat video have been sampled once. For all models, we randomly sample 153600 pixels per training. We measure the PSNR and SSIM scores per frame and take the average across all 300 frames.
The full of hyperparameters detailed below:
KiloNeRF: number of frequencies=10, grid dimensions=(166.6)], 6 layers, 32 units
SIREN: 30. layers. 512 units
ASMR-1: W/ = 30, bases=[[55.31122.1122555 [4.2,2.2,2,2.44, [4,222,,22.244]]] |44,22222224142222222221111 7 layers, 512 units
ASMR-1: wh 30, bases= [4,422,222222 [444,2,,2,,2]],, 7 layers, 512 units
The rest of the configurations of each model and their video-fitting results are presented in the following table.
Model
Param (M)
MACs (K)
PSNR (dB) + std
SSIM + std
SIREN 1.314
1313.79
34.641 + 1.159
0.9079 = 0.01062
KiloNeRF
1.223
6.21
29.560 + 1.208
0.8289 + 0.0228
ASMR-1
1.326
5.36
31.506 + 1.401
0.8536 + 0.0340
ASMR-2
1.326
44.99
31.596 + 0.751
0.8189 + 0.0122
Table 10: Video fitting results on the Cat video.
G ADDITIONAL VIDEO FITTING RESULTS
To further validate ASMR's performance on video data, ASMR is evaluated on the UVG dataset, common video benchmark consisting of 7 HD videos captured at 120fps. Due to hardware con- straints, we downsampled the videos by factor to 150 frames with a resolution of 270 x 480. This allows model to achieve good reconstruction quality while fitting into single NVIDIA 3090 GPU. For the "shakendry" video, however, there are only 75 frames. During each training iteration, sample 194,400 pixels.
16

---

Published as a conference paper at ICLR 2024
The full set of hyperparameters is detailed below:
Instant-NGP: lcvels=17, features per level=2, size of hash map=216, base resolution=[5, 9,
12], finest resolution=[75, 135, 240], 3 layers, 64 units
KiloNeRF: number of frequencies=10, grid dimensions=(3, 5, 5]. 8 layers, 64 units
SIREN: wo 30, 10 layers, 512 units
ASMR: wy 30, 10 layers, 512 units
1,1.1.1111111111
123222221111
Method
#Params (M)
MACs (M)
PSNR(dB)1
SSIMT
Instant-NGi
1.990
0.006
29.8 3.4
0.83 0.07
KiloNeRF
2.194
0.025
30.3
4.3
0.83 + 0.09
SIREN
2.105
2.105
33.6 + 3.6
0.91 0.04
ASMR
2.119
0.120
32.3 1: 4.2
0.86 + 0.07
Table 11: Video fitting results on the UVG dataset.
H IMPLEMENTATION DETAILS OF 3D SHAPES EXPERIMENT
We modified the NGLOD Takikkawa et al. 2021) repository for our occupancy grid experiments. Both SIREN and ASMR are trained for 200 epochs, where each epoch samples 250k points and each training step has batch size of 4096. Among the sampled points, 60% are sampled randomly, 20% are sampled on the surface, and 20% are sampled near the surface.
The full set of hyperparameters is detailed below:
SIREN: wo 30, 10 layers, 512 units
ASMR: wp 30, base==(12222222222222222. (2.2.222222222222222222 [2.2222.2222222222222.... 2.2.2222222222.222222222222 10 layers, 512 units
I IMPLEMENTATION DETAILS OF META-LEARNING EXPERIMENT
We use the same setting as in Dupont et al. 2022bb) where SGD is used for the inner loop with a learning rate of 1e-2, and the Adam optimizer is used for the outer loop with learning rate of 3e-6. We set the size of the latent vector to be 128 and allow gradient updates to generate the reconstruction. The latent vector first mapped to the instance-specifie modulation with size equal to the hidden size multiplied by the number of layers minus 1.
To train ASMR under COIN+ (Dupont et al. 2022b) framework, one can rewrite Equation I as
2, (Wi2; + b, M, dc-)) i i= 1= -
For Instant-NGP, 21-2] 2] is the instance-specific modulation vector to encode each image.
where Φ = 100, ,i,...,
the hashed encoding implementation provided by tiny-ouda-nn (uüller, 2021). The meta-learning code is adapted from the official code released by the author (Dupont al. 2022b). We use batch size of 64 during training. The set of hyperparameters used by each model is given:
directly
Instant-NGP. levels=4, features per level=2, size of hash map=216, base resolution=2, per level scale=2, 5 layers, 512 units
ASMR: wp 50, bases=(2.2,2,222)), 5 layers, 512 units
17

---

Published as a conference paper at ICLR 2024
J INDUCTIVE BIAS OF ASMR
The strong performance of ASMR can be attributed to its assumption of a hierarchical, multi- resolution structure in the underlying data. This assumption provides a powerful inductive bias, allowing ASMR to effectively represent the high-frequency components of the data. Fig, com- pares the reconstructed outputs of ASMR and SIREN during the early training stage on the Cam- eraman image. It shows that ASMR starts with predefined hierarchical repeating pattern on the underlying image, while SIREN begins with relatively low-frequency reconstruction. This induc- tive bias allows ASMR to converge more quickly in the early iterations well achieving better reconstruction quality at the end.
22.995dB
24.540 dB
26.147
27.580 dB
SIREN
26.092 dB
28.994 dB
30.893 dB
32,062 dB
ASMR
200 steps
400 steps
600 steps
800 steps
Figure 7: A comparison of inductive bias between ASMR and SIREN, wherein reconstructed im- ages at the early training stage (200, 400, 600, 800 steps) are shown. By imposing a multi-resolution hierarchical structure the target image, ASMR achieve faster convergence and accurately represent fine details from the start. In contrast, SIREN initially produces a relatively blurry recon- struction.
Level-0: 8x8
Level-1: 64x64
Level-2: 512x512
Figure 8: Feature maps obtained from a selected hidden unit of 3-layer ASMR. It is worth noting that the feature maps shown here not scale, and specify their dimension at the bottom of each feature map.
To provide better understanding of the underlying intivatton-sharnne mechanism of ASMR, we visualize the hidden activations of ASMR. The ASMR employ consists of 3 layers with 256 hidden units. the case of ASMR, we use uniform bases of [8,8.8] for axis partitioning along both the horizontal and vertical dimensions. For brevity and better visualization, display the activation
18

---

Published as a conference paper at ICLR 2024
of only selected channel for illustrative purposes. Figure X depicts the concept of activation- sharing inference ASMR. this approach, modulation vector is shared among coordinates within the same partitioned grid. Subsequently, sequence of upsampling operations is performed, leading to progressively larger feature map in ASMR. Each layer contributes more details, and finer details repeated more frequently due the hierarchical structure of decomposed coordinates.
K PERMUTATION OF BASES
To provide good heuristic for choosing the appropriate partition pattern, conducted set of experiments on the Cameraman image. In particular, we permute the bases of an ASMR model with 4 layers and 512 units, totaling 134K parameters. Table 12 shows that when more activations shared at later stage of the ASMR model using a larger base B3 or equivalently smaller grid size G3. it reduces the number of MACs without compromising the performance in terms of PSNR and SSIM. This entails that ASMR is insensitive to the permutation of bases, avoiding an exponential increase in hyperparameters tune during training.
Partition
MACs (K)
PSNR (dB) 1
SSIM
[o B1, B2, B3]
[4 4, 4,8
1.34
37.755
0.930
4, 4, 8,4
4.42
37.442
0.925
4,8, 4,4
4.61
37.776
0.928
[8, 4, 4,
4.61
37.574
0.927
Table 12: The effect of the bases permutation.
19

---

Published as a conference paper at ICLR 2024
L QUALITATIVE RESULTS ON KODAK IMAGE FITTING TASK
In this section, we present some qualitative results on selected images from the natural image fitting task in Section 4.2
FFN
MFN
ASMR
SIREN
WIRE
FFN
MFN
ASMR
SIREN
WIRE
FFN
MFN
ASMR
SIREN
WIRE
Figure 9: Qualitative results of natural image fitting tasks shown. The selected images for comparison, from top to bottom, are Kodak03, 15dak15, and Kodak23.
20

---

Published as a conference paper at ICLR 2024
M QUALITATIVE RESULTS ON SDF FITTING TASK
In this section, we showcase qualitative results from selected objects in the SDF fitting task. The strong inductive bias of ASMR towards rasterized data hinders its ability encode continuous signals like SDFs. This deficiency is evident in Figure I0 and Figure I1 where ASMR struggles to capture sharp edges and generates noisy samples.
(a) GT
(b) STREN
(c) ASMR
Figure 10: SDF of a piano object.
(a) SIREN
(b) ASMR
Figure 11: Cross section of a simple pyramid object.
21


Reviews:
[{"review_id": "Nob28aht7Q", "reviewer": "Reviewer_zbDF", "summary": "This paper tackles the inefficient inference of INRs. As INR requires computing the individual coordinate's value on all layers, the increase in depth largely affects the inference efficiency, which is a serious problem for high-resolution signals. To reduce the computation, this paper suggests sharing the activation by incorporating multi-resolution coordinates and hierarchical modulation. To be specific, these multi-resolution coordinates are shares the activation and use up-sampling to save the activation storage. The authors demonstrate the efficiency of the proposed method on image and high-resolution signals (e.g., extreme high-resolution image and video).", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The overall writing is clear and the paper is well-organized.\n\nTackles an important problem, i.e., heavy inference time (cost) of existing INRs.\n\nThe overall idea is sounds and reasonable to reduce the inference cost of INRs.", "weaknesses": "**Missing important baseline/related work.**\n- The overall concept is highly similar (although the detail is different) to [1].\n- Using multi-scale INR is explored in [1] (also, [2] uses the same idea in the video domain). Use upsampling (bilinear, nearest, etc) to reduce the number of forward passes by sharing the activations.\n- [1] also uses modulation as well.\n- Due to the existence of [1], I think the paper should change the overall claim, e.g., the first INR to decouple MAC from its depth or using shared activations.\n- Also, the comparison with Multi-scale INR [1] is definitely needed.\n\n**Missing related works: INR modulation.**\n- There exist more recent modulation techniques in the field compared to bias modulation [3,4]. It would also be interesting if ASMR uses more advanced modulations, e.g., low-rank modulation for SIREN [3].\n- missing a reference [9], which is the same method as COIN++ but used for other downstream tasks (e.g., generation, classification).\n\n**The method is somewhat non-trivial to use for 3D scene rendering**\n- Note that 3D is one of the most famous applications of INR.\n- For 3D INRs, there is some paper that tackles a similar issue [5].\n\n**The experiment section should be improved**\\\n(i) Recently, there have been several papers that tackle modality-specific INRs. The authors should claim the benefit of using ASMR compared to modality-specific ones.\n- First, the authors should show multiple modalities rather than images. (Note that video only shows one example). Considering Audio or Manifold [8]\n- Second, the authors need to discuss the benefit of using ASMR over modality-specific INRs. For instance, there are several video INR papers [6,7] showing efficiency.\n- Finally, I think ASMR is hard to use for 3D scenes, so it has less benefit compared to SIREN or FFN.\n\n(ii) Only used one video sample for evaluation. \n- Rather than reporting one sample, it is better to consider a dataset.\n\n(iii) It is hard to understand the intention of Section 5.2 (as it is somewhat trivial). \n- I think the authors are trying to point out that. \n- (a) ‘ASMR uses modulation, but there is no specific modulation technique for multi-grid based INRs’\n- (b) ‘meta-learning modulation is important when using INRs for downstream tasks due to the dimensional efficiency.’ [8,9]\n- I think this claim is not new to the community, and the modulation technique that ASMR is using is from [8,9].\n\n(iv) Comparison with grid-based baselines are missing, e.g., Instant-NGP [10]. \n- Instant-NGP shows image, high-resolution image, and video experiments. \n- It is very worth comparing the efficiency as these methods are proposed for efficiency.\n\n**Summary**\\\nOverall, I quite like the method, but there are several things to be improved, e.g., the major claim should be changed due to the missing important baseline, and comparing with grid-based INRs like instant-NGP [10] and Multi-scale INR [1]. I kindly request the authors to rewrite certain parts and compare ASMR with the baseline in detail throughout the rebuttal.\n\n**Reference**\\\n[1] Adversarial Generation of Continuous Images, CVPR 2021\\\n[2] Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks, ICLR 2022\\\n[3] Modality-Agnostic Variational Compression of Implicit Neural Representations, ICML 2023\\\n[4] Versatile Neural Processes for Learning Implicit Neural Representations, ICLR 2023\\\n[5] Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields, ICCV 2021\\\n[6] NeRV: Neural Representations for Videos, NeurIPS 2021\\\n[7] Scalable Neural Video Representations with Learnable Positional Features, NeurIPS 2022\\\n[8] COIN++: Neural Compression Across Modalities, TMLR 2022\\\n[9] From data to functa: Your data point is a function and you can treat it like one, ICML 2022\\\n[10] Instant Neural Graphics Primitives with a Multiresolution Hash Encoding, SIGGRAPH 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "495nsLzZjT", "reviewer": "Reviewer_24bM", "summary": "This work (ASMR), studies a way of reducing the inference cost of coordinate networks. ASMR combines multi-resolution coordinate decomposition with hierarchical modulations, by sharing activations across grids of the data which decouples the inference cost from the depth. Comparisons against the popular SIREN model are given which shows ASMR outperforms SIREN in both computing cost and reconstruction performance.", "questions": "1. How were the MACs counted (in terms of software implementation)? \n2. My understanding from Figure 1. c about the modulator M is it adds the same pattern details to coarser coordinated activations (upsampled). It is counterintuitive why the sample pattern beats SIREN. Could the authors provide some explanations or intuitions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The pipeline is elegant and seems not difficult to implement. The preconstruction metrics are at least on par with SIREN while the theoretical computation cost is vastly reduced.", "weaknesses": "W1. The number of MACs is certainly a good indicator of inference speed. It would be much more convincing to also have actual inference wall clock timing on several hardware platforms.\n\nW2. It is okay to test only on image/video fitting tasks, but a study on other data types such as audio or 3D shape could be much more convincing to show the generality of the multi-resolution activation-sharing schema.\n\nW3. The word ‘inference bandwidth’ is kinda misused. Maybe consider ‘throughput’ or ‘latency’.\n\nTypo:\n\nW4. Section 3.3 “Equation 1 could be rewritten as”. The original equation 1 was not labeled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q71uYM8gDJ", "reviewer": "Reviewer_rQpt", "summary": "This paper proposes ASMR, a new architecture for implicit neural network (INR) representation. The main selling point is the very low inference cost. ASMR encodes a signal by creating a hierarchical partitioning of the input domain and associating to each partition level (except the first) one transformation layer and one modulation layer.  More precisely, each input dimension of size $N$ is partitioned in $L$ levels such that each level $x_i$ is of a given size $B_i$ and the product of all level sizes correspond to the size of the original dimension.  Every level $i$ of the model takes as input coordinate of that level ($x_i$), transforms it through a neural layer (the \"modulator\"), upscales both a linear transformation of the output of the previous layer and the output of the modulator, sums them and applies a non-linear function. This leads to reduced computation at inference: at a given layer $i\\in\\{1,...,L\\}$ of the model, the number of computed values is multiplied by $B_i$ (for each dimension). In other words, the number of computed values increases from $B_0$ to $N$ with the depth of the model and values computed in the earliest layers are reused gracefully for many coordinates.\n\nThe paper then provides a proof that this architecture leads to an inference cost that is upper bounded by a value that does not depend on the depth of the network. Finally, experiments measures the inference cost and the reconstruction quality on images (Pluto and Kodak) and one video. Meta-learning and the impact of various strategy for the hierarchical decomposition are also evaluated and discussion in two distinct experiments. Between 2 and 5 baselines are considered in most experiments.", "questions": "I would be grateful to the authors for commenting on the weaknesses listed above.\n\nDetails/suggestions:\n- Upon my first reading of the paper, I had some trouble understanding Figure 1. I think it would be helpful, but not necessary, to mention that all bases are of size 2.\n- There is a problem with equation 1: Section 3.3: Equation 1 can be rewritten as [mathematical expression] (1).\n\nTypos:\n- Figure 1, b: hierarcical", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "originality:\n- As far as I know, the proposed approach is novel.\n\nquality:\n- The numerical experiments support the claim that ASMR inference cost does not depend on the number of layers.\n- Inference cost is measured in MACs and not in running time.\n- Despite the lower inference cost, representation quality is similar to or better than the baselines used.\n- Comparison to the most relevant approaches kilo-NERF and LoE.\n\nclarity:\n- The paper is reasonably well written and clear.\n- Experiments are well explained.\n\nsignificance:\n- Inference cost for INR is imho an important and understudied problem.\n- The results seem very good.", "weaknesses": "quality:\n- While several baselines are used in all experiments, instant-NGP is not included in the experiments measuring inference cost. As Instant-NGP is known for its fast inference, I think it is not possible to know whether ASMR is really better than grid-based approaches.\n- In Figure 2, left, kiloNERF always has a higher MAC count than ASMR. In Figure 2, right, kiloNERF often has a lower MAC than ASMR. From the text, I assumed these were the same trained models. This looks inconsistent to me.\n- In Section 3.3, the paper states that the per-sample inference cost depends only on the width of the model and is independent of the depth. I am no sure I agree with that statement, as the depth depends on the choice of the basis and thus on the width of the layers. For example, as far as I understand, doubling the size of the basis also means halving the number of layers.\n\nclarity:\n- The paper did not look polished in some places to me. For example, I am under the impression that multiple notations are used to refer to the same or very similar things, which was confusing to me. Here are a few examples:\n   - a base is denoted by $|x_i$ in Figure 1 and $B_i$ in Section 3.1 (where Figure 1 is referenced)\n   - the cumulative product of bases is denoted by $|x_{i...0}|$ in Figure 1 and by $C_i$ in Section 3.1.\n\nsignificance:\n- For the video experiment, a single video is used. I find that a bit underwhelming as recent papers have typically used video benchmarks for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)