Deep Learning with Learnable Product-Structured Activations
Authors: Saanjali Maharaj
Prasanth B. Nair
Venue: ICLR 2026 Poster (2026)
Keywords: deep learning architecture
implicit neural representation
low-rank tensor decomposition
partial differential equations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: a new deep learning architecture with learnable product-structured activations

Abstract: Modern neural architectures are fundamentally constrained by their reliance on fixed activation functions, limiting their ability to adapt representations to task-specific structure and efficiently capture high-order interactions. We introduce deep low-rank separated neural networks (LRNNs), a novel architecture generalizing MLPs that achieves enhanced expressivity by learning adaptive, factorized activation functions.  LRNNs generalize the core principles underpinning continuous low-rank function decomposition to the setting of deep learning, constructing complex, high-dimensional neuron activations through a multiplicative composition of simpler, learnable univariate transformations. This product structure inherently captures multiplicative interactions and allows each LRNN neuron to learn highly flexible, data-dependent activation functions. We provide a detailed theoretical analysis that establishes the universal approximation property of LRNNs and reveals why they are capable of excellent empirical performance. Specifically, we show that LRNNs can mitigate the curse of dimensionality for functions with low-rank structure. Moreover, the learnable product-structured activations enable LRNNs to adaptively control their spectral bias, crucial for signal representation tasks. These theoretical insights are validated through extensive experiments where LRNNs achieve state-of-the-art performance across diverse domains including image and audio representation, numerical solution of PDEs, sparse-view CT reconstruction, and supervised learning tasks. Our results demonstrate that LRNNs provide a powerful and versatile building block with a distinct inductive bias for learning compact yet expressive representations.

Published as a conference paper at ICLR 2026
DEEP LEARNING WITH LEARNABLE PRODUCT-STRUCTURED ACTIVATIONS
Saanjali Maharaj
Prasanth B. Nair
University of Toronto
University of Toronto
saanjali. maaaaa utorento.ca
prasanth..nir www.snttorrttttocommmmm
ABSTRACT
Modern neural architectures are fundamentally constrained by their reliance on fixed activation functions, limiting their ability to adapt representations to task- specific structure and efficiently capture high-order interactions. We introduce deep low-rank separated neural networks (LRNNs), a novel architecture general- izing MLPs that achieves enhanced expressivity by learning adaptive, factorized activation functions. LRNNS generalize the core principles underpinning continu- ous low-rank function decomposition to the setting of deep learning, constructing complex, high-dimensional neuron activations through multiplicative composi- tion of simpler, learnable univariate transformations. This product structure inher- ently captures multiplicative interactions and allows each LRNN neuron to learn highly flexible, data-dependent activation functions. We provide detailed the- oretical analysis that establishes the universal approximation property of LRNNS and their ability to mitigate the curse of dimensionality for functions with low-rank activations enable LRNNs
structure. Moreover, the learnable product-structte adaptively control their spectral bias, which is crucial for signal representa- tion tasks. These theoretical insights are validated through extensive experiments where LRNNS achieve state-of-the-aar performance across diverse domains in- cluding image and audio representation, numerical solution of PDEs, sparse-view CT reconstruction, and supervised learning tasks. Our results demonstrate that LRNNs provide powerful and versatile building block with distinct inductive bias for learning compact yet expressive representations.
INTRODUCTION
Neural networks equipped with simple activation functions like ReLU, Tanh, and Sigmoid have achieved remarkable success across multitude of domains. While their simplicity is strength, with expressivity achieved through a deep composition layers, these standard activations possess inherent limitations. For instance, it is well known that the spectral bias of activations such ReLU can hinder the representation of high high-frequency details in signals (Rahaman et al.] 2019). This has spurred long line of research into alternative activation functions with enhanced expressivity and improved optimization properties such as convergence and gradient propagation. Early efforts led to activations such as Maxout (Goodfellow al. 2013), which learns piecewise linear function, while others such as Leaky ReLU (Maas et al. (2013). 3), PReLU (He et al., (2015). Swish/SiLU (Ramachan-
dran et al. 2018). E-Swish (Alcaide 2018 and GELU (Hendrycks Gimpel 2023) often Tocus on
improved gradient flow or adaptive non- lineariis.....s. Kunc & Kléma (2024) provide comprehensive
survey of activation functions in deep learning.
A significant leap in expressivity, particularly for representing complex, continuous signals, emerged with the advent of implicit neural representations (INRs). The effectiveness of INRs stems from spe- cialized activation functions designed to capture fine details and high frequencies. often providing better results than standard activations combined with positional encodings (PE) (Mildenhall et al.) 2020). Pioneering work by Sitzmann al. (2020) introduced neural networks with sinusoidal acti- vations (SIREN), which achieves remarkable fidelity in representing images, 3D shapes, and numer- ical solutions of partial differential equations (PDEs). Subsequent developments include Gaussian functions (Ramasinghe & Lucey] 2002). wavelet representations (WIRE) (Saragadam et al. 2023), semi-periodic damped activations (SPDER) (Shah & Sitawarin 2024) hyperbolic oscillation func-

---

Published as a conference paper at ICLR 2026
tions (HOSC) (Serrano al. 2024), sine activations (Saratchandran et al. 2024), and FINER (Lu et al. 2 ; see Essakine et al. (2025) for a recent review. This body of work has demonstrated that the choice and design of the non- non-linearity
crucial for high-fidelitg signal representation.
While tailored activation functions are powerful, they are often designed to capture specific signal properties such periodicity and multi-scale behavior. This motivates the search for architectures that can learn highly expressive and adaptive non-limearities, while maintaining computational ef- ficiency and optimization stability. A recent development in this direction is Kolmogorov-Arool Networks (KANs) proposed by Liu et al. (205), which incorporate learnable activation functions on edges rather than using fixed activations nodes. While KANs offer increased expressivity, they re- quire significantly longer training times and can suffer from optimization instability with larger grid sizes. The ActNet architecture (Ferreira Guilhoto & Perdikaris) 2025) which leverages Laezovich's variant of Kolmogorov's superposition theorem has shown promise in addressing these challenges.
In the present work, we introduce deep low-rank separated neural networks (LRNNs), a novel archi- tecture whose expressivity stems from a different principle: a multiplicative composition of learn- able, univariate functions. LRNNS are inspired by work on low-rank separated representations. originally proposed for approximating multivariate functions sums of products of univariate basis functions (Beylkin et al. 2009; Audouze Nair 2019). LRNNS generalize this idea to create a new class of deep neural network architectures where the neurons equipped with learnable product- structured activations. This structure inherently captures multiplicative interactions and allows each neuron independently learn highly flexible activation function, adapting its non-linearity the learning task, while maintaining computational efficiency.
The notion of low-rank function decomposition that our work builds upon has deep roots in tensor algebra (Kolda & Bader 2009). Low-rank decompositions of model weights have been successfully applied to model compression (Novikov et al. 2015 Lebedev et al., 2015) and fine-tuning large language models (LLMs) (Hu et (022). However, our focus with LRNNS is distinct: rather than using low-rank decompositions for compression, we leverage the multiplicative structure of low- rank function decompositions to enhance expressivity. More specifically, LRNNs utilize adaptive product-structure activations to efficiently capture high-order interactions in contrast to standard neurons that compose features additively. Our main contributions are:
We introduce the LRNN architecture, a generalization of MLPs where each neuron's activation is a product of learnable univariate functions applied to projected inputs, enabling highly adaptive and expressive non- non-lincarities beyond fixed scalar activations.
We provide detailed theoretical analysis establishing universal approximation, the ability of LRNNs overcome the curse of dimensionality for functions with decaying functional ANOVA structure, and insights into how LRNNS can adaptively control their spectral bias.
We demonstrate that the unique theoretical advantages of LRNNs translate into practical impact across diverse domains:
Image representation:
LRNNs achieve 100% success high-fidelity 40 dB target across 1,000 ImageNet images, regime where the SPDER and SIREN baselines frequently fail.
Audio representation: 3-1 1x lower MSE on audio tasks with superior spectral fidelity. Numerical solution of PDEs: LRNNS achieve 8x parameter reduction compared to SIREN and 100- 1000x lower error than KANs on a PDE benchmark.
Sparse-view CT reconstruction: On this benchmark, LRNNS provide artifact- free reconstruc- tion and superior performance with a small number projections (50-100), a critical factor for reducing patient radiation exposure.
LOW-RANK DECOMPOSITIONS IN LEARNING
Low-rank tensor decompositions provide a powerful framework for mitigating the curse of dimen- sionality by representing high-dimensional tensors through interactions among their dimensions, often as products of low-dimensional tensors. The origins of this topic can be traced to work by Hitchcock ( (9277) decomposing tensor into sum of rank-one tensors, which was later refined by (attell[1194)) with parallel proportional and multi-axi axis analysis. The most widely used approach arguably the canonical polyadic (CP) decomposition (Carroll & Chang 1970 Harshman 1970
2

---

Published as a conference paper at ICLR 2026
general family of decompositions that includes the CP decomposition as a special case. The tensor train (TT) decomposition (Oseledets) 2011) is well-studied approach that combines the advantages of CP and Tucker decompositions. is worth noting that since the Eckart- Young-Mirsky theorem only holds for matrices, there is no unique approach for generalizing the notion of singular value decomposition to higher-order ten- sors (Kolda & Bader 2009).
The Tucker decomposition (Tucker 1966) provides
Low-rank tensor decompositions now ubiquitous in many fields (Kolda & Bader 2009), , and applications include scientific computing (Dolgov et al. 2021), dimensionality reduction (Papalex-
akis et 2015; Shashua Levin 2001] Acar et al. 2006). compression of deep learning mod- els (Novikov et al. 2015), reducing memory footprint when fine-tuning LLMs (Hu et al. 2022). and
decomposition L.LM gradients to reduce training memory (Zhao al. 2024).
The present work is motivated by the observation that low-rank tensor decomposition can be inter- preted as a discretization of a continuous low-rank decomposition of multivariate function. This observation underpins the separated rank decomposition (SRD) model (Beylkin et al. 22009), which continuous generalization of the CP decomposition. For a d-dimensionall function, the SRD model takes the form: 5(x Σi= 6 IT (Tz) where is the separation rank, 8, are nor- malization coefficients, and gi.j are approximated using a linear combination of univariate basis functions. The basis functions are typically polynomials
radial basis functions and alternating least squares is used for training (Beylkin et al. 2009) Chevreuil et al.] 2015), which can result in slow convergence and ill-conditioned subproblems when the support region of the basis is dis- joint from the data points. [Audouze & Nair (2019) proposed a sparse SRD approach that uses (-rrgularrizaion and a coordinate descent optimization algorithm to address these challenges.
Other models in the literature that use univariate component functions include projection pursuit regression (Friedman & Stuetzle, 1981) and neural additive models (NAMs) (Agarwal al. 2021)
Tree tensor networks (TTNs) (Shi et al. 2006; Cheng et al. 2019 Bachmayr et al., 2023 Ali & Nouy 2023) are another class of models that use compositions of low-dimensional functions that are not restricted to be univariate. TTNs have a structure similar to an MLP equipped with a sparsity mask. Despite the growing body of theoretical work on this topic, applications have so far been restricted to simple test problems. This can be attributed to the fact that learning the optimal tree structure from data is challenging combinatorial problem.
3 LRNN ARCHITECTURE
In this section, we introduce the LRNN architecture in the supervised learning setting. Let 'D denote
a dataset of N observations, D = ((x(1) ) where x( (s ,..., xtd (2) E Rd is an input
vector with feature dimensions and y()) y denotes noisy observation containing either K regression targets (i.e., y C R²x) or K class labels (i.e., y C z²). Our goal is to construct predictive model for the regression targets or class labels using the dataset D.
3.1 SHALLOW LRNNS
At the core of our approach the LRNN neuron's ability to capture multiplicative interactions through product- structurt activations. We begin with the shallow LRNN architecture, in which a single layer transforms inputs through learnable univariate functions, before extending to deeper compositions. For dataset with K outputs, the shallow LRNN takes the form
d
Stran (x)
Σ St
(1+ of == W 5=1
where r E N is the separation rank, 8, RK are weight vectors, of R - R denotes a univariate component function, 7 d-1/2 is a scaling factor, z' Rd, W E gddd and be gd
can seen from that LRNNs project the d-dimensional input to r latent vectors Rd and produce an output using sum sum-prode uon-dutduct operation. We introduce the term (1 (z)) to ensure tomatic relevance determination (ARD) and make initialization more convenient; see Appendix(A.] for details. The scaling factor 7 d-1/2 plays crucial role analogous to Xavier/He initialization
3

---

Published as a conference paper at ICLR 2026
in standard networks (Glorot & Bengio 2010: He et al. 2015) and the scaling used in LoRA (Hu et al. 2022). We formalize this by establishing the following result (see Appendix AA.3 for proof):
Lemma 1 (Variance-conntoolled initialization). Under mild assumptions on the component func- tions at initialization (zero mean, finite variance), the product-structuret LRNN activation
p(() = [T j=1 (z})) satisfies the following bounds:
d
(i) Var[s(z)]
S e - and (ii)
Var(Dp(z)/Ozz)) S oz c
(2)
1=
where o and oz. denote the variance of the component functions functionn and their first-order derivatives, respectively, at initialization.
follows from this result that the variance of the LRNN activation and the sum of the variances of its gradients are bounded independently of the projection width d. This reveals an intrinsic mechanism for ARD: as projection width d increases, each coordinate's gradient contribution Var Var[00p00r 0zz] = ((1dd) diminishes, while their collective impact remains constant. This ensures stable gradient flow through arbitrarily wide product structures, enabling LRNNS to learn high-dimensional yet well-conditioned representations.
The univariate LRNN component functions gf R + R can be flexibly parametrized, with each of the rd functions typically being small MIP, enabling them to adapt to complex patterns in the data. The parameters of the component functions are learned along with the weight vectors Sy E IR K during training. The hyperparameters of shallow LRNN are the separation rank, r, which controls the model's s expressivity, and the dimensionality of the linear projection layer, d.
Connection SRD: For the special case of scalar targets (i.e., K 1), if we the projection layer to identity (i.c., z = x) and replace (1 + (2())) with of (2f), we recover the SRD model of Beylkkn al. (2009). The LRNN model can hence be viewed as generalization of CP-based function decomposition, which we will later generalize further to deeper architectures.
Generalization of MLPs: LRNNs generalize the familiar MLP architecture. we set d d= 1 and replace of with a standard activation function, LRNNs reduce to a standard shallow MLP. To see this generalization clearly, consider a shallow MLP with r neurons in the hidden layer: (x) = En ((ze), where = w + by is scalar projection of the input with We E R. E IR K and be E denoting the weights and biases, respectively, and
- is a standard MLP activation function. The shallow LRNN in (1 can be rewritten in the same form: Virnn (x) the LRNN product-structture activation
Σi 8/ (z'"), where (xx(2))= [1 - function with z' = W'x b The key distinctions are: (i) each LRNN neuron learns its own distinct learnable activation function pr Rd + IR whereas all MLP neurons share the same fixed activation a R + R operating on scalar projections (ii) LRNN activations achieve this vector- to tc-scalar mapping through multiplicative compositions, enabling efficient representation of higher- order interactions that additive architectures struggle to capture (see Section 3.3).
(05(=())
3.2
DEEP LRNNS
We now extend LRNNs to deeper architectures, enabling them to learn hicrarchical representations through composed transformations. Deep LRNNs stack multiple layers, creating sequence of maps from input to output space, i.e., x() - x!)) +... - (L) + ý for a model with L layers; see Figure 10 for a graphical illustration. This hierarchy progressively transforms inputs into latent representations amenable to efficient low-rank approximation, combining deep learning's 's composi- tional power with the expressivity of low-rank function decomposition.
A deep LRNN architecture with L. layers and r& neurons in the kth hidden layer for mapping a d-dimensional input K-dimensional output can be represented
f(x) Sout (L ((--1) ..... (())
(3)
Maxout networks (Goodfellow al. 2013) are notable exception, also using vector-to-scalar mappings but through max operations rather than products.
4

---

Published as a conference paper at ICLR 2026
where S E xxy. and ((h) 2-1 + R with ro d and Φ x. The output of the k-th
(k) hidden layer can be written as ()(k) (4g) (zl1(0)), (k)) (z2 (22((()))222)) ).... fri (z) where
du
(00)
(.()) (zg (,(())), L t= = 1,2, ...,.. ,k, k = 1,2,.... L. = II (1 + 7 99
(4)
j=1
In the preceding equation 2 2-((k)
( ba(()) RR, and W E Rdxxr 1, and
bl,()) dd denote the weight matrix and bias vector for the f-th neuron, respectively, 9j R I are learnable univariate component functions, and sp
(.(k)
(R) BRd R.
Each LRNN neuron layer applies a product-structured activation /h to de-dimensional projection of the previous layer's output. The matrix Sout maps gb(2 to the target space. Similar to the shallow LRNN, the deep generalization also introduces distinct product-structurg activation functions within and across layers that operate distinct dz- de-dimensiionl projections ]
Parameter sharing can be used to reduce the parameter complexity of deep LRNNs, e.g., sharing the activation function across LRNN neurons in each layer, the term 4i
(K) can be rewritten
(Z.-(8)
II (1 799 (2) (2), where zf((k Wh.0k) +bs(k) j=1
(5)
The use of shared activations for the neurons in each hidden layer reduces the number of learnable univariate component functions from r&dg to d, (i.e., 9j shared across all neurons in the k-th layer). Note that each hidden layer is equipped with a distinct learnable product-structured activa- tion function. We evaluate the trade-off's of this approach in Appendix C (Figure TT comparing this shared activation variant against the standard flexible LRNN architecture. Our results indicate that while parameter sharing improves efficiency lower parameter counts, distinct activations necessary to maximize fidelity for complex high-frequency signals. Another possibility is to share the projection layer across neurons in each layer; however, we found that this approach leads to significant loss in expressivity.
f.(R) Implementation aspects: In implementation, each univariate component function g, within the LRNN's product-structture activation parametrized by a small shallow MI.. These component networks employ standard scalar activations: for implicit neural representation tasks, we use either SPDER activations (e.g., sin(x) / [zr]. sin(x) arctan(x)) SIREN's sinusoidal activa- tion (sin(r)). To ensure stable learning dynamics in deep architectures, we apply LayerNorm to the output vector σilk) after each LRNN layer's 's product-structured computation; see Appendix C.2 for details. This normalization strategy proves crucial for consistent convergence in deeper networks. Implementation details and ablation studies are provided in Appendices) and C respectively.
3.3 THEORETICAL ASPECTS
We establish fundamental theoretical properties of LRNNs that provide insight into their empirical success: universal approximation, mitigation of the curse of dimensionality for structured functions, and adaptive spectral bias control.
Theorem 1 (Universal approximation). If [0. 1] + IR is continuous function, then for every 6 > O, there exists an LRNN with suitably chosen separation rank r such that maxxe|o.11" [(x) -imm(x)) <
This result, establishing universality analogous to that standard MIPs, follows from the Stone- Weierstrass theorem and the fact that LRNNs represent arbitrary polynomial expansions (see Appendix A.4 for proof). Just as the width of an MLP may grow with 1/s, the separation rank r of an LRNN grow arbitrarily large to capture complex functions. Thus, "universal" here does not guarantee a small unless the target function has low-rank or near-sepa structure.
Beyond universal approximation, the stability of training is important in practice. It follows from Lemma 1 (Section 3.1) that variance-controlled initialization ensures variance-controlled learning for LRNNs:
In practice, we set de be the same across layers, i.c., de d vk.
5

---

Published as a conference paper at ICLR 2026
The scaling factor r d- ensures that both forward and backward propagation remain sta- ble regardless of projection width d, with activation variance and the gradient variance sum both bounded from above by constants independent of d.
This property enables automatic relevance determination and stable optimization even for wide prod- uct structures (detailed analysis in Appendix (A.3).
Theorem 2 (Curse of dimensionality mitigationn. For functions whose ANOVA decomposition is dominated by terms involving most & d variables, LRNNs achieve approximation ervor with parameter complexity O (poly(d)/s) rather than exponential in d.
Theorem 2 shows that LRNNS circumvent the curse of dimensionality for a class of structured functions since the parameter complexity grows only polynomially with d rather than the exponen- tial scaling typical of generic approximators. Ap- pendix A.5] provides more precise statement of this result with the technical assumptions and proof. The key insight is that LRNNS naturally encode sum-of-products structures matching ANOVA de- compositions. Functions arising from physical sys- tems often exhibit such decay in interaction order. making LRNNS particularly suitable for scientific computing applications. Figure [ illlustaate this in practice for a synthetic test function with product- structure; see Appendix D for details.
1.
Test RAASE
UW leve
Teral Peramisers
Figure 1: Test RMSE vs. parameter count for LRNN and ReLU MLP.
Lemma 2 (Adaptive spectral bias control). When equipped with periodic activations (e.g., SIREN, SPDER), LRNNs with > generate rich frequenc spectra through combinatorial frequency syn- thesis. single LRNN neuron with d components generates not only the d fundamental frequeccies but also all 2d - 1 possible sum and difference combinations.
This multiplicative frequency synthesis contrasts with MLPs' additive synthesis, where each neuron contributes a single frequency pair. Consequently, LRNNs can represent complex spectra with fewer parameters, particularly for signals with harmonic relationships or intermodulation products. This explains their superior performance on audio and image representation tasks where the ability to capture high- gighffreyeecc details important (see Appendix A.7 for proof and detailed discussion).
4 NUMERICAL EXPERIMENTS
We evaluate LRNNs across diverse domains, including image and audio representation, numeri- cal solution of PDEs, and sparse-view CT reconstruction. All models were implemented in Py- Torch (Paszke et al. 2019) and trained using the Adam optimizer (Kingma & Ba 2013) on a sin- gle NVIDIA 4090 GPU. Our implementation is publicly available https: / gi thub..... com/
Prior presenting task-specific results, we summarize key architectural insights gained through extensive ablation studies (detailed in Appendices H (i) Stability: The multiplicative structure of LRNNs alters activation statistics compared additive networks, making LayerNorm essential for convergence (Appendix[C.2 (ii) Component selection: Using periodic activations (e.g.. SIREN, SPDER) within the univariate components is crucial for minimizing spectral bias in high- tigh-frequency tasks (Appendix (C.3); and (iii) Robustness: LRNNs provide excellent performance in the sparse- data regime, maintaining high reconstruction fidelity (Appendix H.
Image representation: We conducted experiments to study how the performance, measured by peak signal-to-noise ratio (PSNR) scales with parameter count for LRNN, SPDER (Shah & Sitawarin 2024). and MLPs. We refer to our LRNN implementation for this case as LRNN-SPDER, since we use the SPDER activation function sin( ()) / in the MLP parametrizations of the LRNN univariate component functions; see Appendix B for details.

---

Published as a conference paper at ICLR 2026
(a) Cameraman
(b) Rotina
Solepitt)
SPOOR (5)
(3)
(OP) UNS- Meen
Beill:
Potwih (%)
Moan
Number of Parameters (x101)
Number of Perameters (x107)
Figure 2: Scaling laws for image representation task.
As shown in Figure on both the cameraman image (grayscale 256 x 256) and the retina image (RGB 256 x 256), 2-layer LRNN-SPDER consistently outperforms deeper 3- and 5- layer SPDER models and 5-layer MLP models with both ReLU and tanh activations across all tested parameter counts. For cameraman, the performance gap widens between LRNN-SPDER and SPDER up ~300k parameters, suggesting that LRNN-SPDER achieves superior param- eter efficiency compared to its baseline. Similarly, for retina, 2-layer LRNN-SPDER shows increasing performance gains model size grows compared to both SPDER models. These scaling plots highlight LRNN-SPDER's expressivity and learning capacity, surpassing the cor- responding benchmark which the LRNN activations based, even at shallower depth.
SPDER
SREN
WARE
RELU
PSWR (48)
40
400
Figure 3: Cameraman image results: ground truth and reconstructed images from LRNN-SPDER,
SPDER, SIREN, WIRE and ReLU (left); and PSNR convergence history over iterations (right).
We compare the performance of LRNN-SPDER to MLP with ReLU activations, SIREN [Sitzmann al. 2020 WIRE (Saragadam et al. 2023) and SPDER (Shah & Sitawarin (024) on the cameraman image. All models chosen to have 197k parameters. LRNN-SPDER outperformed all the baselines, achieving a PSNR of 107.9 dB-a margin of 58.9 dB over the next best. SPDER (49.0 dB). While such extreme fidelity exceeds visual dis- tinguishability, confirms that LRNNS avoid the spectral saturation limiting standard architectures. Figure shows the qualitative image reconstructions and PSNR conver- gence history for all models. The PSNR convergence plot shows LRNN-SPDER achieving the highest PSNR early in training and continuing to climb even while the other models show signs of convergence.
SPDER
100%
100%
100%
4.55s
4.92s
7.01s
Madel
97%
84%
26%
SPREN
51%
26%
2%
3.87s
4.17s
4.27s
35
Target PSNR (dB)
Figure 4: Average success rate and time for models to reach PSNR targets on 1000 ImageNet images.
We conducted a large-scale robustness study 1,000 im- ages from the ImageNet dataset, each postprocessed to 256 x 256. LRNN-SPDER, SIREN, and SPDER models with 200k parameters per image, totaling 3,000 runs per model. Figure reports the success rate and average wall-clock time required to reach PSNR targets of 33 dB, 35 dB, and 40 dB. We observe that LRNNN consis-
trained for 1,000 epochs using three random seeds

---

Published as a conference paper at ICLR 2026
tently outperform the baselines. For the challenging 40 dB target. LRNNS achieved 100% success rate, whereas SIREN and SPDER failed to reach this target in 98. and 73.6% of cases, respec- tively. Even at lower targets, LRNNS surpass SPDER in both success rate and time-to-solution. While SIREN has shorter runtimes, its success rate significantly drops at higher fidelity targets. See Appendix T for further analysis. Figure compares representative reconstructions epoch 250. LRNN reconstructions are virtually indistinguishable from the ground truth, whereas SPDER exhibits mild degradation and SIREN produces blurrier outputs.
Grond tath
LOWB SPORn
SPEEN
Grourd Tasth
LPVI SPEEN
SPEER
SINN
Figure 5: Qualitative comparison of ImageNet reconstructions after 250 epochs. Selected examples demonstrate that LRNNS capture fine details significantly earlier training than baseline models.
We demonstrate the capabilities of LRNN-SPDER images from the DIV2K dataset. We down- sample the original image with a scaling factor of 4, train the model on the downsampled image, then reconstruct the image by upsampling the model output back to the full resolution. The fre- quency factor wo within the MLP component functions of our LRNN-SPDER model was chosen following de Avila Belbute-Perrs. & Kolter (2023). The upsampled reconstruction is compared to the ground truth in Figure 6 Additional studies on color images are presented in Appendix
Faoestactio
Figure 6: LRNN-SPDER upsampled image reconstruction demonstrated on DIV2K image.
We tested LRNNs on audio signal representation using four diverse clips: instrumental classical music (bach) and male human speech (counting) as used Sitzmann et al. (220);; reggae music with singing (reggae) from the GTZAN dataset (Tzanetakis and female human speech (reading) from the LibriSpeech dataset (Panayotov et al.J 2015). We sin(x) arctan(x) (Shah & Sitawarin, 2024h activations for the MI.P-based component func- tions of LRNN-SPDER and compare against the baseline models, SIREN and SPDER. Table presents MSE loss and frequency similarity (PAG) mean values (std) over 10 runs, while Figure shows time and frequency domain absolute errors and loss convergence for the bach audio. Ad- ditional results and implementation details are presented Appendix The results consistently demonstrate LRNNs' BeneriortyyLLNNNSSPP ing its SPDER baseline, achieving 3x-1ix lower final MSE and higher PAG across all the datasets.
Audio Representation:
al. 2001); ));
significantly outperforms all other models, includ-
8

---

Published as a conference paper at ICLR 2026
SPEN
SPOD1
Abcokto Eror
0.35
URVN SPDER SIREN
0.30-
SPOER
0.00
MSE Lose
10
Time 653
Time 15)
Time (s]
230
120
150
200-
300
100-
Absolute
50-
0
200
400
000
2000
Preouency Oshz)
Frequency (KHz))
Frequency (oHH)
Stop
Figure 7: Absolute error in time and frequency domain and convergence of training MSE loss (mean =lo) for bach audio representation tasks for comparably sized models.
LRNN-SPDER also exhibits faster convergence compared to the baselines. The error distributions in the frequency domain show that LRNN- SPDER preserves spectral integrity essential to human perception (e.g.. timbre, pitch, harmonics). demonstrates superior gener- alization across both temporal and spectral representations compared to baselines.
Table Comparison of MSE loss and PaG on Audio datasets. Values show mean(std) over 10 runs.
MSE Loss (x10-4)
Method
bach
counting
reggae
reading
SIREN SPDER
1.21(0.28)
2.77(0.56)
21.5(6.3)
9.98(1.57)
1.12(0.05)
2.29(0.55)
24.8(7.7)
8,88(2.45)
LRNN-SPDER
0.10(0.01)
0.72(0.03)
7.93(0.11)
1.86(0.30)
This
PAG (sid x10-*)
SIREN 0.9986(5) 0.9906(15) 0.9769(11) 0.9193(94)
SPDER 0,9988(3) 0.9937(6)
0.9729(10) 0,9324(104) LRNN-SPDER 0.9999(0) 0.9967(2) 0.9860(2) 0.9862(31)
PDE benchmark: We eval-
uate LRNNs (with sin(2) activations for the MLP component functions) on the high- freefrryy Poisson PDE benchmark (Liu et al., 2025), comparing against SIREN, MLPs, and KANs. Figure 8 compares the mean squared error (MSE) obtained using LRNN, MLP, and SIREN with different model parameter counts when the frequency parameter is set to = 2 and = 4. Results for KAN1 ([100) = 10) and KAN2 ([100) = 20) are from Liu et al. (025) and are displayed as horizontal lines since the parameter counts are unknown. It can be seen from the results that LRNNs exhibit exceptional performance, particularly in parameter efficiency. For instance, a 16k- (6k-parameter 2-layer LRNN achieves significantly lower error for frequency relative to a parameerrr SIREN with three hidden layers (an 8x parameter reduction).
2 and comparable error at
Furthermore, 37kk-arameete LRNN reduces the error by nearly an order of magnitude compared to the 132k-parameter SIREN when n = 4. Compared to KANs, LRNNS achieve orders of magnitude lower error across all tested frequencies. These results underscore LRNNs' superior expressivity for complex PDE solutions with compact models; see Appendix G for additional details and results.
Sparse-view Computed Tomography (CT) is vital for reducing patient radia- tion exposure, and INRs can reconstruct high-fidelity images from such limited data. We compared LRNN against WIRE, SIREN, Gaussian-activated networks (Gauss), and ReLU with positional en- coding on a 256 x 256 chest CT image task (Saragadam et al. 2023). using 180k parameters for all models.
CT Reconstruction:
Table ssoows that LRNN achieves the highest PSNR (29.13 dB) and SSIM (0. (0.7455). Qualitatively, Figure demonstrates LRNNS' superior reconstruction fidelity: its output is sharper and closer to the ground truth than the blurry results from SIREN and ReL.U+P.. Notably, while the training loss of the LRNN converges similarly to WIRE (the next best in PSNR), the LRNN reconstruc- tion is free from the high-frequency artifacts present in WIRE's output. This suggests that LRNNs find solutions that better correspond to perceptually accurate image features. Appendix H further details ablation studies, confirming the suitability of LRNNs for pearse-view CT. This artifact-free reconstruction from limited projections has direct clinical implications for reducing patient radiation exposure while maintaining diagnostic quality.
9

---

Published as a conference paper at ICLR 2026
10
(a)
(b)
10
10
NSE
MSE
10'
10-3
KANL
10"
10-
LANN
SIREN
40
100
120
60
80
120
Number of parameters (x107)
Number parameters (x10*)
Figure 8: Results for the PDE benchmark.
Ground tnuth
LANN
WIRE
Table CT model performance
comparison.
Model
PSNR
SSIM
LRNN
29.13
0.7455
Gauss
SIRNN
Relli positional cncoding
WIRE
28.83
0.6413
Gauss
27.84
0.6855
SIREN
27.46
0.6877
ReLU
26.89
0.6341
Figure 9: CT ground truth and reconstructed images.
CONCLUDING REMARKS
We introduced deep low-rank separated neural networks (LRNNs), novel architecture that gener- alizes MLPs, achieving enhanced expressivity through learnable product-structured activation func- tions. This design allows LRNN neurons effectively capture complex high-order interactions with compact parameterization. Our theoretical analysis established LRNNs' universal approximation capabilities, their potential to overcome the curse of dimensionality for functions with low-rank structure, and their ability to adaptively control spectral crusia for signal representation tasks.
Our extensive experiments demonstrate that LRNNs hold significant potential across several do- mains. They set new benchmarks on challenging PDE test-case, achieving orders of magnitude lower error with significantly fewer parameters than SIREN, MLPs, and KANs. In INR tasks, LRNNs delivered wfte--ffthhear image reconstruction quality, outperforming SPDER and SIREN even when using their respective component activations. They also produced superior audio fidelity with faster convergence and yielded higher-quality. antifact-free CT reconstructions.
Exciting avenues for future research include extending LRNNs to domains such as video modeling and unsteady PDEs. We view 3D scene reconstruction (NeRFs) tion; we hypothesize that the multiplicative structure of LRNNS is naturally suited for capturing the figh-rrequenc, view-dependent effects. Additionally, experiments in Appendix on classification benchmarks suggest that LRNNs' applicability extends beyond continuous signal representation.
particularly promising direc-
While remarkably effective, LRNNs present opportunities for further refinement. For instance, while our use of forward-mode AD proved highly efficient for Laplacian computations in PDE tasks, the general backward pass currently incurs higher memory footprint than standard MLPs due to intermediate product storage. However, as detailed in Appendix [.2) strategies such as kernel fusion and mixed-precision training offer clear paths to mitigate this. In summary, LRNNs provide a versatile and powerful building block for learning compact and expressive representations across a broad spectrum of machine learning challenges.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is supported by Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant.
REPRODUCIBILITY STATEMENT
We provide complete architectural specifications, hyperparameters, and training procedures in Sec- tions 3 and and Appendices B-I. Our implementation is publicly available athttps phttps://github: com/ dann/dacela Lab/ Irnn All experimental configurations, including learning rates, and model archi- tectures for each benchmark, are detailed in the respective appendices.
ETHICS STATEMENT
Our work introduces a new deep learning architecture with demonstrated advantages in medical imaging (CT reconstruction) and scientific computing (numerical solution of PDEs). While these applications have clear benefits-reducinn patient radiation exposure and advancing computational science- we acknowledge that powerful representation learning tools can be misused. The improved efficiency of LRNNs could potentially lower barriers to applications requiring careful ethical con- sideration. We encourage responsible deployment with appropriate domain expertise and ethical oversight, particularly in medical applications.
During the preparation of this work, the authors used LLMs for proofreading the text and for debug- ging code.

---

Published as a conference paper at ICLR 2026
REFERENCES
Evrim Acar, Seyit Ahmet Çamtepe, and Bülent Yener. Collective sampling and analysis of high order tensors for chatroom communications, In Sharad Mchrotra, Daniel Dajun Zeng, Hsinchun Chen, Bhavani Thuraisingham, and Fei- Wang (eds.), Intelligence and Security Informatics, IEEE International Conference on Intelligence and Security Informaties, ISI 2006, San Diego, CA, USA, May 23-24, 2006, Proceedings, volume 3975 of Lecture Notes in Computer Science, pp. 213-224. Springer, 2006.
Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Benjamin J. Lengerich, Rich Caruana, and Geoffrey E. Hinton. Neural additive models: Interpretable machine learning with neural nets. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: An- mual Conference on Neural Information Processing Systems, pp. 4699-4711, 4711, 2021.
Eric Alcaide. E-swish: Adjusting activations to different network depths. arXiv: 1801,07155 2018.
Mazen Ali and Anthony Nouy. Approximation theory of tree tensor networks: Tensorized univariate functions. Constructive Approximation, 58(2):463-544, 2023. doi: 10.1007/
5003655-033-09620-v.
Christophe Audouze and Prasanth B. Nair. Sparse low-rank separated representation models for learning from data. Proceedings of the Royal Society Mathematical, Physical and Engineering Sciences, 475 80490, 2019. doi: (0098//spa.21888 20..009.
Markus Bachmayr, Anthony Nouy, and Reinhold Schneider. Approximation by tree tensor networks in high dimensions: Sobolev and compositional functions. Pure and Applied Functional Analysis, 8(2):405-44) 428, 2023.
Gregory Beylkin, Jochen Garcke, and Martin J. Mohlenkamp. Multivariate regression and machine learning with sums of separable functions. SIAM Sci. Comput.. 31(3):1840-188 2009.
Russel E. Caflisch, William Morokoff, and Art B. Owen. Valuation mortgage backed securities using Brownian bridges reduce effective dimension. Journal of Computational Finance, I(1): 27-46, 1997.
J.D. Carroll and J. Chang. Analysis of individual differences in multidimennsioal scaling via an n-way generalization of "Eckart-Young decomposition. Psychometrika, 35:283-319, 1970.
R. Cattell. Parallel proportional profiles and other principles for determining the choice of factors by rotation. Psychomelrika, 9:267-283, 1944.
Song Cheng, Lei Wang, Tao Xiang, and Pan Zhang, Tree tensor networks for generative modeling. Physical Review B, (9(1),, April 2019. ISSN 2469-99899 doi: 10.1103/physreev.99.5555 155111.
M. Chevreuil, R. Lebrun, A. Nouy, and Rai. A least-squares method for sparse low rank ap- proximation of multivariate functions. SIAM/ASA Journal Uncertainty Quantification, 3(1): 897-921, 2015.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Con- trol, Signals, and Systems, 2:303-314, 1989. doi: 10.1007/BF02551274,
Filipe de Avila Pabbute-Peres and J. Zico Kolter. Simple initialization and parametrization of sinu- soidal networks via their kernel bandwidth. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, 2023.
Sergey Dolgov, Dante Kalise, and Karl Kunisch. Tensor decomposition methods for high- dimensional Maciitonn-acobb-Belllle equations. SIAM Journal on Scientific Computing, 43 A11,,,, 2021.
Amer Essakine, Yanqi Cheng, Chun- Chun-WWn Cheng. Lipei Zhang, Zhongying Deng, Lei Zhu, Carola- Bibiane Schönlieb, and Angelica I Aviles-Rivees Where do stand with implicit neural rep- resentations? a technical and performance survey. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
12

---

Published as a conference paper at ICLR 2026
Leonardo Ferreira Guilhoto and Paris Perdikaris. Deep learning alternatives of the Kolmogorov superposition theorem. In Y. Yue, A. Garg. N. Peng, F. Sha, and R. Yu (eds.), International Conference on Learning Representations, volume 2025, pp. 78241-78269, 2025.
Jerome H. Friedman and Werner Stuetzle. Projection pursuit regression. Journal of the American Statistical Association, 76(376):817-233, 1981. doi: 10. 1101000016214591181..00000
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neu- ral networks. In Yee Whye Teh and D. Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS, volume 9 of JMLR Proceedings, pp. 249-256. JMLR.org, 2010.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua Bengio. Maxout networks. In Proceedings of the 30th International Conference on Machine Learning, ICML, volume 28 of JMLR Workshop and Conference Proceedings, pp. 1319-1327. JMLR.org, 2013.
Samuel Greydanus and Dmitry Kobak. Scaling down deep learning with MNIST-ID. In Forty-first International Conference Machine Learning, ICMI., 2024.
R.A. Harshman. Foundations of the PARAFAC procedure: Models and conditions for an explana- tory multimodal factor analysis. UCLA Working Papers Phonetics, 16:1-84, 1970.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, ICCV, pp. 1026-1034. IEEE Computer Society, 2015.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). urXiv:1606.08411 2023.
Frank L. Hitchcock. The expression of tensor a polyadic sum of products. Journal of Mathematics and Physics, 6( 1444):164-88,, 1927.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, (2):251-257, 1991.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation large language models. In The Tenth Interna- lional Conference on Learning Representations, ICLR, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations ICLR, 2015.
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Rev., 51(3): 455-500, 2009.
Vladimír Kunc and Jirí Kléma. Three decades of activations: A comprehensive survey of 400 activation functions for neural networks. arXiv:2402.09092 2024.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned CP-decomposstiion In Yoshua Ben- gio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR, 2015.
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. FINER: flexible spectral-bias tuning in implicit neural representation by variable periodic activa- tion functions. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2713-2722. IEEE, 2024.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-Arnole networks. The Thirteenth International Conference Learning Representations, ICLR, 2025.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net- work acoustic models. In Proceedings of the 30th International Conference on Machine Learning ICML, volume 28, 2013.
13

---

Published as a conference paper at ICLR 2026
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), 16th European Conference on Computer Vision ECCV, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pp. 405-421. Springer,, 2020.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry P. Vetrov. Tensorizing neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems, pp. 442-450, 2015.
Ivan V. Oseledets. Tensor-train decomposition. SIAM J. Sci. Comput., 33(5):2295-2311
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur: LibriSpeech: an ASR cor- pus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on, pp. 5206-5210. IEEE, 2015.
Evangelos E. Papalexakis, Konstantinos Pelechrinis, and Christos Faloutsos. Location based social network analysis using tensors and signal processing tools. In 6th IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, CAMSAP 2015, Cancun, Mex- ico, December 13 13-16, 2015, pp. 93-96. IEEE, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8: 143-195, 1999. doi: 10. 10.101750092249990009999
Nasim Rahaman, Aristide Baratin. Devansh Arpit, Felix Draxler, Min Lin, Fred A. Hamprecht, Yoshua Bengio, and Aaron C. Courville. On the spectral bias of neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML, volume 97 of Proceedings of Machine Learning Research, pp. 5301- 5310. PMLR. 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. In 6th In- ternational Conference on Learning Representations, ICLR, Workshop Truck Proceedings, 2018.
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards a unifying framework for In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Gio-
activations in coordinate-MI.PP vanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIII, volume 13693 of Lecture Notes in Computer Science, pp. 142-158. Springer, 2022.
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G. Baraniuk. WIRE: wavelet implicit neural representations. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 18507-18516. IEEE, 2023.
Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey. A sampling theory perspective on activations for implicit neural representations. In Forty- first International Conference on Machine Learning, ICMI., 2024.
Danzel Serrano, Jakub Szymkowiak, and Przemyslaw Musialski. HOSC: A periodic activation function for preserving sharp features in implicit neural representations. arXiv:2401.10967 2024.
Kathan Shah and Chawin Sitawarin. SPDER:
semiperiodic damping-enabled object representation. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
A. Shashua and A. Levin. Linear image coding for regression and classification using the tensor-rank principle. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pp. I-I, 2001.
14

---

Published as a conference paper at ICLR 2026
Y.-Y. Shi, L.-M. Duan, and G. Vidal. Classical simulation of quantum many-body systems with tree tensor network. Phys. Rev. A, 74:022320, Aug 2006.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, 2020.
L. Tucker. Some mathematical notes three-mode factor analysis. Psychometrika, 31:279-311, 1966.
George Tzanetakis, Georg Essl, and Perry Cook. Automatic musical genre classification of audio signals. The International Society for Music Information Retrieval, 2001.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
15

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS OF LRNNS
A.1 LRNN MODEL STRUCTURE
Consider an LRNN with one layer and separation-rank i... f(x)=
- - st (1+ where x E [0, 1] and each of [0. 1] + R is a univariate function. The factors (1 + 7 of (I,)) in our LRNN model explicitly separate constant offset of I from the learnable component of (zz). This is a specific parameterization of the more general multiplicative factors in classical separated- rank decompositions. While this choice does not change the fundamental expressive power, it yields tangible advantages in model initialization, learning dynamics, and interpretation, as we detail next. For simplicity, we will drop the y scaling in the following discussion will study the role of in Section A.3
1. Built-in additive terms. Expanding [1 + (1²))) automatically includes (i) constant off- set 1. (ii) purely additive terms Σ, (22), (iii) all possible higher-order products. Although one can mimic this in purely multiplicative model by adding constant or bias term inside each factor, the (1 + gj) notation makes these offsets more explicit.
2. Initialization and automatic relevance determination. Setting each (-) to zero at initialization yields initial product of 1, natural baseline. The network can then gradually learn interac- tions by adapting specific (-) functions. If a feature I is unimportant, the network can keep ()()) R O, which in experience often leads more stable training while providing in-built automatic relevance determination mechanism.
Connection to functional ANOVA decompositions. The expanded form
II (t,,)) =1 Σ (e (o(o)))(( jek
of,))
reminiscent of functional ANOVA decomposition. In principle, single rank-1 factor can capture constant, additive, pairwise, and all higher-order interactions. Rank-r LRNN is equiv- alent to functional ANOVA decomposition whose component functions are weighted linear combinations of products of univariate functions.
We discuss below some additional reasons why the (1 +) factorization can be advantageous in practice, even though is not a fundamentally different decomposition than classic CP.
Modeling sparse interactions. Suppose we want f to capture a product of only few relevant coordinates (e.g.. Ll,es 25 for some small subset S). In a purely multiplicative model 11 - (zy), one typically sets h (1r;) Ij for E S and (2y))
for j S. With LRNNS, we can do the same by setting 0) (xzz) = x - for j E S and (و,م))
for j & S. Hence, the factor IT T= (1+9) becomes I T.. while all factors corresponding to irrelevant coordinates default to 1. This offset- based parameterization can be more natural to train or initialize.
Compact representation of polynomial features. One can also view (1 + (x;)) as generating function in each coordinate. When of is a complete polynomial, the product term when expanded yields constant, linear, and higher-order powers of zj. A classical fact (see, e.g., Kolda & Bader (2009) for tensor viewpoint) is that polynomials with fully factorable coefficient structure can be captured single rank- I product. Although this observation is not novel (it dates back standard inperaating-uncction ideas and CP decompositions), it illustrates how (1 + 95) can unify constant, additive, and multiplicative terms one factor.
In summary, the (1 + Y (-)) construction is mostly for notational and practical convenience, rather than a departure from classical multiplicative low-rank expansions. From a theoretical perspective, an LRNN is equivalent in representational power to standard rank-r CP model Σ I1 h (2y) that permits each h² to have constant offset. However, specifying these offsets explicitly by (1+9 often simplifies initialization (starting from a constant baseline), captures additive terms by default, and can improve interpretability regarding how interactions are learned during training. Later in this
16

---

Published as a conference paper at ICLR 2026
section, we will provide some theoretical analysis to show how the normalization constant y ensures that the variance of the LRNN output is controlled, which can be beneficial for training.
The deep LRNN architecture is illustrated in Figure 10
Fdd
LRNN Layer
LRNN Neuron
LRNN Neuron
LRNN Neuron
(2)
42 (2
11
2
( (z ()))))
((1)
LRNN Layer
LRNN Neuran
LRNN Neuron
Neuron
(2))
(())
P (22)))
82
(d(2)
y
Figure 10: Deep LRNN architecture.
A.2 INTERPRETABILITY AND INTERACTION ANALYSIS
significant challenge in deep learning is the opaque nature of standard architectures. In a standard MLP, feature interactions are entangled within dense matrix multiplications, making it difficult to isolate the contribution of specific variables. The LRNN architecture, by virtue its separable product structure, offers more structured view of how features are combined, similar in spirit to generalized additive models (GAMs) such as neural additive models (NAMs) proposed by Agarwal al. (2021).
LRNNS gener- alize this by allowing for multiplicative interactions. Each LRNN neuron & computes product of univariate transformations on projected features: (z) = I (1 + (2,)) Since the compo- nent functions of I + R are univariate, they can be visualized directly. By plotting the learned curves (2), practitioners can inspect the nonlinear transformation applied to each projected fea- ture dimension (e.g., whether the model has learned an approximately linear trend, a threshold-like response, or periodic modulation) before these features are combined multiplicatively.
GAMs approximate function as a sum of univariate functions: f(x) =
Beyond visualizing individual components, the LRNN structure enables an explicit, architecture- level notion of which features participate in given interaction. the product above, each factor (1 + (1,f((z))) acts modulator. If, over the data distribution, a component function leams nearly constant mapping of (z) then the corresponding factor close to 1 and effectively does not influence the product. Conversely, (2) varies substantially. it actively modulates the neuron output.
17

---

Published as a conference paper at ICLR 2026
This observation suggests simple diagnostic: by computing the empirical variance of each com- ponent function, Var (2)]] over a validation set, one can construct "interaction heatmaps" that indicate which coordinates are active for each neuron. For example, if specific neuron é exhibits high variance predominantly for components = and j = 4, this provides evidence that this neuron is primarily sensitive to joint variation in feature dimensions and 4 and relatively invariant to the others.
Standard MLPs with conventional activation functions can, in principle, be analyzed using post-hoc attribution or sensitivity methods, but their parameterization does not make such interaction structure explicit. In contrast, LRNNS build this structure into the architecture: the univariate components of and their variances provide direct handle for probing which dimensions and interactions given neuron is using.
A.3 VARIANCE-CONTROLLED CONTROLLED INITIALIZATION; PROOF LEMMA
In this section, we provide proof for Lemma in Section 3.1 and discuss its implications. The proof broken down into a lemma (see Lemma 3) and a corollary (see Corollary [T]. An LRNN neuron acting the d-dimensional projection z (21,.... 2q) is defined by the product-structured activation (zz) = I1 + (z;)), where = V Let 2j w/ + bj. with weights initialized such that for fixed imput x, [B[y]] = and Var[z,] = of for each j. In addition, let each Øj I + I be wicce-differeniable learnable univariate function. We theoretically analyze the role of the scaling parameter 7 under the following standard assumptions at initialization:
Assumption 1 (Independence across dimensions). For any distinct indices j, k {1.....d) d) the random variables 2j and 2 are independent. In other words, (z())) is independent of gx(2x), and (zg) is independent of (zx))) [or j # k; as 9j are deterministic functions of 2j initialization.
Assumption 2 (Properties of component functions). For every j E {l....,d}, d}, (2g)(1,)] = O, Var (2,)] o Elg'( = O, and (z)) = = where og and og are finite constants.
Assumption ensures that the expected partial derivatives El0p/0zij are zero, simplifying variance calculations. We note that while the main conclusion regarding the sum of gradient variances being ((1) holds more broadly (see discussion after Lemma 3). these specific assumptions lead to the following variance bounds.
Lemma 3 (Gradient-variant variance stabilization). Under Assumptions andd 2 the partial derivatives of the LRNN product-struuturred activation cz(z) = 11 (1+ +gg((z))) where 7 Va
1 satisfy
E
0+ = and
g' e El dd.
02k
d
and consequently the sum of the variances of the first-order partial derivatives satisfies the following bound that is independent of d, i.e., Ed Varl
S o² e
Dec
Proof. From the definition of we have 0./02z = T ( II, 1 (1+ (1,7))())) Using Assump- tion to separate expectations and Assumption 2 for the properties of gj and g/. we have
0pp
E
((gA(=#))
II J#k
El1 +r9())))>)))1 = 0. II (1+7 0) = 0.
azk
For the second moment, we have
0p)
9 = -2 [(g((2))))) II
jtk
22 (Varlgh + (E(g((>+)))))
II
2yE(o, "
jzk
1
(oz + 02) [I + 0 (Varlg (Bl,, = 00² (1+ 22 أ-1
k
18

---

Published as a conference paper at ICLR 2026
Using the inequality (1 ) ) S E" for a, m > we have 0+
2 Dz
< (dz
Since
B[0e/Ozz)] = 0, follows that War[O</ = E| )2x)a]. Summing this variance bound over k ,,.., .,. d yields the following upper bound independent of the projection width d.
Varl0p/0zz)
c = =o e
d
k=1
Remark (Relaxing Assumption 2). If Elg Hyg # O, then E[ae/0zz]] 7pg The sum
of variances Ex (ar(0p/0zz) would then be bounded by [[o + (gg)e (g which is still ((1) constant independent of d. Thus, the primary conclusion of wiitth-nndependee total gradient variance still holds, though Assumption simplifies the constant and ensures zero-mean gradients.
We now prove a corollary showing that the variance of the output of the LRNN neuron remains ((1) as d increases.
Corollary 1 (Activation variance). Under Assumptions the variance of the LRNN activation satisfies the inequality Var[p((z) I e 1.
Proof. Using Assumptions and 2 we have [[]] For the second moment, it follows from Assumption
j== (1
that (1g og. which gives
[l
II
B[( + +
[I (1 2yE "B
j=1
j=1
II [1 (1+ ))
j=1
Therefore, Var[p(z)] [l (R(s(z)))) (1 2 3)
Using the inequality (1 + a/m)m S e that holds for a, m 0, yields the stated result.
Remark 2 (Implications). Lemma 3 demonstrates that while the variance of each individual coordinate gooddnnat-gradient 0p/0zk decays like 1/d, their cumulative variance sum remains constant. This suggests an intrinsic mechanism for automatic relevance determination: as projection width d grows, the influence any single projected coordinate 2k the output's gradient variance dimin- ishes. Together with Corollary which ensures O(1) activation variance, this analysis establishes that the scaling factor 1/vd d plays a crucial role analogous to initialization approuches for additive NNs (Glorot & Bengio 2010 He et al. (0155) or the 1/vT scaling in LoRA adapters (Hu al. 2022), ensuring stable propagation in both forward and backward passes for LRNN neurons, regardless of product width.
A.4 UNIVERSAL APPROXIMATION THEOREM: PROOF OF THEOREM
Theorem (Universal approximation theorem). If [0, 1j + R is continuous function, then
for every > 0, there exists an LRNN model finnn()) = Σi= st IT= (1 + (x(z)) with suitably
chosen separation rank < R(s) and univariate component funnttion gf. [0, 1] + E such that
[f(x) [x)n(x)) E.
max [0.1
Proof. We provide a proof based on classical tensor- product expansions and polynomial approxima- tions; see, for example, Hornik 199Tb; p. Pinkus (1999; Cybenko (1989) for analogous MI.P proofs.
TThis follows from the fact that (1 + _ m
(14a/m)"
5" aam
144/m
19

---

Published as a conference paper at ICLR 2026
It follows from the Siine--reesssrsss theorem that any continuous function the compact domain [0, 1]dd can be uniformly approximated by multivariate polynomial. Thus, there exist N E N, real coefficients Oa. and univariate polynomials (0z) [0, + B such that f(x) =
E ae I - (z,e(1z), with s1 f(x) - f(x) & e/2
Noting that Σ as I[ )),)(z,) is a sum of product terms representation, we will set r := N, == (gg, and define the univariate functions (5(z)) ,5,) - 1 in the LRNN representation such that
Σ 1=1
I
1 9/(5) 1+9(())))))
Σ at II 0))((1z). 1=1 j=1
follows from standard universal approximation results (see, for example Hornik (1991)) that each component function of can be approximated by a univariate neural network gj (T,) such that max 19]=()))))))))
(x²)) I
8. Using telescoping product approach yields the inequality
Σ st
II 1=1
[1 gf(,,)) - Σ sz (
(1,))),)))
E=1 j=1
where M := max,, (0 |1+ ( is bounded by construction and Smax max |scl- Choos- ing < (2 M M
completes the proof.
A.5 STRUCTURED FUNCTIONS WITH DECAYING ANOVA INTERACTIONS: PROOF OF THEOREM 2
We show that LRNNS can mitigate the curse of dimensionality for class of high-dimensionaal functions whose functional ANOVA decomposition exhibits a decay the importance of higher- order interaction terms, and whose significant low-order terms themselves approximable by sums of factorized components. The analysis presented here focuses on the case when the LRNŇ component functions univariate MLPs. To prove main result, first establish lemma on the approximation of functions that are already sums of products of univariate functions, where each product involves limited number of variables.
Lemma 4 (LRNN approximation of separable low-rank functions). Let f 10, R denote continuous function that admits the representation fxx) = Σé=1 ae ILEss Fz,((z,), where St S 21. df with |St] S m for some m S d, 2z [0, 1] + RR functions such that lave S c and || I 1, where c 2 0 is a constant. Then for any > 0, there exists an LRNN approximation, firm with O(rm² /Plp//P(c,rr r, r,,,,)) parameters such that supxeto.1j f(x) Fienn (x) S where P(c,T, m,,, In(1 + ).)
St are continuous univariate
Proof. We begin by noting that the target function admits exact representation by an LRNN by appropriately defining the scale and univariate component functions, i.e., f(x) firm (x)
E/= se
1 (1 + +z,)), with s; = a and (2,) = 2),((zz) - 1 Vj E S and / Vjj & St.
Each non-trivial of is approximated by univariate MIP of IR - R with sufficient capacity such 1ox 8. It follows from standard universal approximation results (Homik 1991 Pinkus 1999 that the number parameters needed for each 9 is ((1)6).
that llg
Since
= | < 1, we have |1 [1+5)(=;)) < 1, and |1 = (2) [x,)) 1 + 8 Vj 2t, [0, 1], while [1 + of (22y) and [1 + + 14j Se. 2, [0. (since
of 0 for & St).
Let hi |Sp| S m be the number of interacting variables the ε-th term. To bound the error
between f and fronn we first consider the error for the f-th term in the sum, i.c., Es := ==11less (1 + of(2))) 1 (1 + (1)))) Let Pp..p2.....1 Pk.} be an ordered enumeration of the indices in
20

---

Published as a conference paper at ICLR 2026
St. Using telescoping product argument, we have
22 Er 2
A-1
Spa (xpx) - Dp (Frw))
II
1 + Sp. (m)))
II
1 + Sp. (5pt))
i=k+1
k
8 (1 + 6) =6 Σ
(1++))) 1.
j=0
Since ke S m, we have Es S (1 + 6) - 1. The total approximation error can be bounded as:
sup [(() - x (0,1,
(ix))_mm((x) [x)( S Σ t=1
c((1+)) 1) = rc ((1 + 5) - 1). Jasl Es S E=1
We need to choose ō such that the total approximation error to be less than or equal to E, i.e. re((1 + 5) 1) I E. Using the inequality (1+ + y)* I e/ for y 2 0, 2 with y 8 and k in the preceding equation, we have re (ems 11) S e. Taking the logarithm of both sides and rearranging gives 8 (1/m) In (1 + ..). Noting that we have at most non-trivial univariate functions, with each requiring ((1/5) parameters, we obtain the stated parameter complexity. O
Remark 3 (Simplified parameter complexity). If E & re the parameter count simplifies to O(r"mèc/E) c/E). The parameter complexity grows as m² (m S d) for fixed (r, c). thereby circum- venting the curse of dimensionality.
Remark 4 (Assumption [[L I 1). The assumption [| S 1 is a common approach to normalize components in approximation theory. practical machine learning scenarios, this is of- ten justified since the target function f(x) is typically normalized. The constants r and c. (governing laee) characterize a specific sum-of cun-of-products decomposition assumed to exist for this normalized target function, where its constituent univariate functions 2 have norms bounded by 1. If such representation exists for the normalized target, this condition is met.
We now prove main theorem for a general continuous function f : [0, 1]d + R whose functional ANOVA decomposition takes the form:
f(x)
Σ fs(ss), SC((]
(6)
where each fs depends only on variables xs with S S {1,.... d) and xs:= (z;))es). We assume that standard ANOVA orthogonality conditions hold, e.g., So (xs)dz, = 0 for any j E S and fa = J fx) Our theoretical analysis uses the following assumptions:
Assumption 3 (Decay of ANOVA components). The norms of ANOVA components decay suffi- ciently fast. Specifically, for given E1 > 0, let Itrane = (S S [d] 1Ifssl 2 TS} } be a collection of index sets such that the truncated sum firune (x) Esetrrune fs(xs) satisfies |lf frrune oo S E Let N. = IIrrunel be the number of significant ANOVA terms, and let m. max ||| be their maximum interaction order. In addition, let ma < d, and let Na grow most polynomially with d.
Assumption 4 (Factorizability of ANOVA components). Each ANOVA component can be ap- proximated by a sum-product representation. Specifically, for each ANOVA component fs(xs) AS E Itrunc- there exists E2 > O such that the sum-product representation fs(xs) Eke 0sg,8 Iiees (135, satisfies the error bound [|5 - Tslloo S 62, ki S kkmax for some kmaz 2 1, |ass S c and |10,,s,5lloe 1.
Remark 5 (The assumption |ملا < 1). Each significant ANOVA term fs(x5) is itself a function of at most ma variables. If the target function pre-normalized, its ANOVA components is (which are defined through integrals of f) also inherit scaling properties. It is then plausible that these (scaled) fs terms can be well-approximated by a sum-of-products representation where the individual univariate components $j.3.8 are also normalized. The practical strength of this assumption rests on whether such a 'normalized- comppiieeddcmmmeennnn each relevant fs exists with simultaneously controlled number of terms and bounded coefficients. For ANOVA terms fs that are smooth and depend on a small number of variables (small m.). this is often considered reasonable modeling assumption.
-froduccs approximation for
21

---

Published as a conference paper at ICLR 2026
Theorem 2. If f [0, R satisfies Assumption and Assumption 4 then for any tar- get accuracy E > there exists an LRNN approximation, firmm with parameter complexity O (Nakmax) )² m such that 1uPxe|o.1) [(x) - Suna [x)] S..
Proof. It follows from Assumption 3 that truncating the ANOVA expansion of f gives f(x) s furune (x) = EsEIrmne fs(xs) such that |l fcrueelle S E1. From Assumption 4 we can approx-
imate fs as fx(ss) = kg 08 ILES (I,5, such that the error is |lfs fslloo S E2.
Let h(x) = EsEIimuo fs(xs). Then [| nl1so S ESEIinm IIIs - Islloo S Na62. We now approximate h using an LRNN, Bimn: such that |1h - Jimnllo S Etran. The total error is ||ʃ - Jo I E + Na52 + Stmmn We set ει = e/3, Na82 = E/3 (i.e., 52 = e/(3Na)). and Elrna = e/3 to ensure that the total error is at most E.
To bound the parameter complexity of fienns we note that
ky 0ss ())s,(();))
Mxx)
E
SELirane 9=1
JES
is sum of products involving EsEterune ks Nakmax terms. Each product HiES (1.5, involves |S] S ma univariate functions with coefficients ag where las,æl univariate functions $5.8.8 satisfy the bound || Σ 1. Thus, h matches the form required ma- It therefore follows that h can be approximated by an LRNN firmn to an accuracy Elrn = =E/3, with the number of parameters (using the simplified parameter count) given by OX (Nakma))) m² 9). Absorbing the constant 1/3 into the O t/3
Moreover, the
by Lemma with parameters:
R S Nakmax and
notation yields the stated complexity.
Remark 6 (Special cases). If ma and knax small constants (or grow very slowly with d), and C is small, the complexity depends primarily on N2/e. If the ANOVA decay is such that
Na ~ Ene This complexity is polynomial in d if ma is constant, demonstrating mitigation of the exponential growth in parameter complexity.
k= (d) s O(d".), then the parameter complexity becomes O((dtma kanx ma c)/s).
A.6 REMARKS ON THEOREM 2
Theorem establishes that if a high-dimensional function admits a functional ANOVA decomposi- tion dominated by low-complexity terms, LRNNS can approximate it with number of parameters that scales polynomially with the effective dimension d. To provide further insight, we examine the intuition behind Assumptions and the specific function classes they encompass.
Assumption 3 requires that the number of significant ANOVA terms, Na. grows at most polyno- mially with dimension d. This implicitly constrains the maximum interaction order ma- Since full ANOVA decomposition contains 2d terms, for the truncated sum to scale polynomially (c.g.. Na & (dd*)), the interaction order my must generally be small relative (i.c., my (1)).
This characterizes functions with low effective superposition dimension Caflisch et al. (1997). )
property common in physical systems dominated by main effects and low-order (pairwise or triplet) interactions.
Assumption posits that significant high-order interaction terms can be approximated by a tensor product with limited rank kmaz- This can be viewed as conservation of complexity principle; the curse of dimensionality constrained rather than eliminated. Specifically, if ANOVA terms involve few variables (m,, < d), Assumption 4 is easily satisfied as low- cow-dimensional functions typically admit efficient low-rank approximations. Conversely, a term involves all variables (m, ss d), Assumption 4 requires it to be rank-deficient (small kmaz).
Thus, LRNNs efficiently represent functions that are either interaction-sparee (small m.) or interaction-denss but rank-sparse (small kmaz). Theorem 2 implies that if function has both global interactions (m, z d) and high separation rank, kmaz would necessarily scale exponentially with d, reintroducing the curse of dimensionality.
22

---

Published as a conference paper at ICLR 2026
Pairwise potentials (illustrates Assumption 3 Functions such as Coulomb gravitational potentials V(x)= = Ei Φ[|=;; - myy)) are dominated by pairwise interactions (m, = 2). Even for large d, the number of significant ANOVA terms No grows quadratically in d, satisfying the polynomial-gae -growhh requirement in Assumption
Separable functions (illustrates Assumption 4 A product state f(x) = II sin(2;) has maximum interaction order (m, = d), so it does not have low interaction order. However, it is exactly rank-1 as product of univariate factors (kmax 1), satisfying Assumption - with a single significant ANOVA term (N₂ 1). In this case the parameter bound in
Theorem 2 still scales polynomially in d because both No and kmax are constants.
Sign-parity function (violates Assumptions 3 and + The parity function f(x)
II: sgn(#;) possesses both full interaction order (m, = d) and high separation rank. It cannot be approximated by small sum of smooth product terms and therefore falls outside the scope of Theorem 2.
In summary, when function satisfies these structural assumptions (as in the first two examples), the LRNN architecture naturally aligns with its decomposition. As established in Lemma 4 this alignment allows LRNNs to learn the representation efficiently, achieving parameter complexity that scales polynomially rather than exponentially with dimension. This theoretical result provides an insight into the empirical success of LRNNs on tasks where underlying low-order interaction structures are likely present.
A.7 SPECTRAL REPRESENTATION ANALYSIS OF LRNNS: PROOF OF LEMMA 2
Here, we study the spectral representation capabilities of LRNNs, particularly when equipped with periodic activation functions commonly found in the INR literature, such SIREN (a<() = sin(2))
or SPDER (e.g,, (2) sin(2) / (z)). We use is to denote a scalar frequency parameter, which can be specific to each univariate function (wx) or shared.
For simplicity, we do not consider the normalization term y in our analysis. We begin by recalling the observation in Section 3.1 that LRNNs with d = 1 recover standard MLPs. Writing the LRNN
activation function as (p(x) -T - + (2x)). where 23 v + C. and setting d 1, we have -0²1) = 1 + 9i (21). Let و (21) = (w)=1) be a periodic activation function, where =1 = vf'x + cj. The full LRNN model is a sum of r such rank-1 terms, each with its own scaling factor sg: firan (x) = Ei sxpp(xx) = se(1 (()(,)). Substituting g1,e(21,e) ol..,eVV
c1,e)) leads to
funm (x)
Σ + Σ st s cc.).
The second term in the preceding equation is an MLP with r neurons in the hidden layer, where V1,2 the saput-to-hideer weights, C1,e the hidden biases, is the activation function (scaled by Wi,e), and 8t are the hidden- -otputt weights. The first term, Σr=1 $g, is a constant, acting as an overall output bias for the MLP. Thus, MLP with given periodic activation function can be viewed as a special case of an LRNN with d= 1 and the same base activation. This observation forms the basis for comparing their spectral properties. The key distinction of LRNNs with d > lies in their product structure, which leads to a richer spectral synthesis than the purely additive nature of MLPs which we establish in the following lemma.
Lemma 2 (Combinatorial frequency generation by LRNNs). Consider the LRNN product-
structured activation function sp(x) = 11 + gu (x)), where gı(x) onwwx VI x + cx)) is a univariate function with characteristic frequencies +fy the Fourier domain. Then, the Fourier transform of .(x) contains spectral components at not only the fundamental frequencies tf, but also at all possible sum and difference combinations, i.e.. frequenesss of the form Ekes Syfa where
C {L....,d) and sy E [1].
Proof. The Fourier transform of h4 (x) 1 (x( can be written
F(hx)(E) (() + Flge (E) 5(E) + Gx(E),
23

---

Published as a conference paper at ICLR 2026
where ((E) is the Dirac delta representing the DC component (zero frequency), and Gx(E) is the Fourier transform of ga (x), which by assumption has significant energy at ff..
Since the LRNN activation is product of these (x) terms, i.e., (p(x) I hy hx(x), its Fourier transform takes the form
F())(E) F(h) F(12)) * F(ha) (M/=I (E),
where denotes convolution. Using equation we have
F(P)(E) = ( A-1 (8()) + +G(+)) (E).
Now, expanding the product in the LRNN activation yields
(xx)= II [1 ([ = Σ 9x((x) Σ (x(x))((x)) + icj
II 9k (x).
The Fourier transform of the first term is F(1} O(E) (the DC component). The second term, F(2; g: (x(x) E, Gi(E), contributes the fundamental frequencies =f; from each The third term, F(E Dies (x))gx((x)
Li<r(G; G5)(E) leads pairwise combination frequen- cies, e.g., +f, + fy- This pattern continues for higher-order terms since product of functions 9hi (x) ... 9ky (x) will have a Fourier transform of the form (Gk, * ..... * G2,, (E), generating fre- quencies corresponding to all combinations =f,, - .... ... Thus, a single LRNN activation with d components generates up 2d distinct frequency combinations from just d base frequencies.
Lemma 2 provides some useful insights into differences between how LRNNs and MLPs represent functions in the frequency domain. Standard MLPs (equivalent to LRNNs with d 1) synthe- size functions additively in the spectral domain. Each neuron, with activation 8t0 (wr contributes primarily to a specific frequency pair +f, = EWiV (for SIREN) or a narrow band around these (for SPDER-liks activations). The overall spectrum of an MLP is the linear super- position these individual contributions: F[fMLP} = E, (EG) To represent a complex spectrum with many frequencies, In contrast, LRNNS with > 1 employ a multiplicative synthesis within each rank-1 term. As shown in Lemma 2 single rank-1 1 term generate combinatorial set of frequencies from just d base projected features gk. The full LRNN then additively combines these rank-1 components, i.e., F(Smm >, 8t (E)
br)).
MLP typically requires corresponding number of neurons.
This combinatorial frequency synthesis mechanism provides LRNNs with adaptive spectral bias by learning the base frequencies fa and their corresponding amplitudes, the network im- plicitly controls a rich set of derived frequencies, enabling efficient representation of complex spec- tra with fewer parameters than additive approaches.
B IMPLEMENTATION ASPECTS OF LRNNS
B. PARAMETRIZATION OF LRNN COMPONENT FUNCTIONS
In our architecture, the univariate component functions gj f.1k) R - are parameterized by small, independent MLPs. Each such MLP, denoted MLP(j, E, k). takes the scalar __(k)) as its input, where (,(k) is the j-th component of the vector zf (())) = (1) (k-1)) b)((%) resulting from the linear projection within the LRNN layer.
2j
key design choice for these component MLPs, MLP(j. (j E, is that their first effective operation on [.(k) is activation function. Since 2, 25.(k) is already the output of a linear transformation, the input 2;
applying another linear layer the immediate first step within MLP(j, E, k)) would be redundant. Instead, the structure is:
e.(k) 2.00)) 9j (z/
(8)
24

---

Published as a conference paper at ICLR 2026
(.(k) is structured such that its computation begins with initial activation o, where the MLP 9j
which is chosen to a standard activation or an INR style activation. The output of first stage of MLP(j, E, k) is given by 0(2) (,))))).. This activated value is then processed by the subsequent hidden layers and the final output layer of MIP(i, & k) which is a linear layer without bias term.
All learnable parameters, including those in W b,*(k, and within all parts of 9, ff((k) are opti- mized jointly. The set of all univariate ML.P is implemented efficiently for parallel computation using block-diagonal weight matrices as a custom PyTorch layer. This design choice allows for ef- ficient computation of the LRNN forward pass, as all the univariate functions in an LRNN layer can be computed in parallel.
B.2 MEMORY COMPLEXITY ANALYSIS AND OPTIMIZATION STRATEGIES
While LRNNs demonstrate superior parameter efficiency and expressivity compared to standard architectures, the product-strn product-structured activation introduces distinct memory footprint profile. In this section, we analyze the memory requirements for training LRNNs, focusing on the scaling behavior with respect to the projection width d, and discuss strategies for optimized implementation.
To isolate the memory requirements when training LRNNs, consider a single LRNN layer with separation rank (analogous to neurons in a standard MLP) and projection width d. We assume the univariate component functions (-) are parameterized by a shared, shallow MLP with one hidden layer of width h, setup consistent with our experimental configuration. Let B denote the batch size. In a standard backpropagation framework (c.g., PyTorch Autograd), intermediate activations must be stored during the forward pass compute gradients during the backward pass. We analyze the storage requirements for the activations (excluding model parameters, which are negligible in comparison).
For a standard MLP layer with r neurons, the output takes the form y o (Wx + b). The primary memory cost is storing the pre-activation post-activation vectors, leading to memory require- ment that scales as MMLP Ri ((Br).
An LRNN neuron computes & = I (1 + (24)). The forward pass of this layer involves three distinct stages of activations that must be stored for backpropagation:
1. Input projection: The linear projection creates intermediate vectors E Rdd for each of the rank components leading to a memory cost of: Mproj ((rd).
2. Component function computations: The component MLP g()) applies a non-linearity to each of the r d projections. If g has hidden layer of width h, standard Autograd must store the hidden states of these sub-networks to differentiate through them. This results in a memory cost of: Mg O(Brhh.
3. Product inputs: The outputs of the component functions uj = 1 (g(2)) must be stored compute the gradient of the product operation during backpropagation. This incurs memory cost of: Mpmd (Brd).
The total memory requirement for naive implementation of LRNNS therefore scales as: MARNY & O O(Brdh). Compared to a standard MLP, the memory footprint is scaled by factor of roughly d- h. The primary bottleneck arises during the backward pass of the product term. For single product De
é = == Uj, the partial derivative with respect to the k-th component is Ang = 1 Uj. To
compute this without re-evaluation, the expanded tensor of size B x d will be kept memory in standard Autograd implementation.
The effective memory cost can be significantly reduced by leveraging hardware-aware optimizations that trade computation for memory as outlined below.
Gradient checkpointing. The dependence on the component MLP hidden dimension h can be re- moved via gradient checkpointing. Instead of storing the intermediate states of g(-) (the term Mg), we store only the projected inputs z. During the backward pass, the forward pass of the component MLPs is recomputed on-the-fly fly. Since g()) is typically shallow, the computational overhead is mini- mal, but the memory complexity reduces to: Maar
O(B r d). This removes the multiplier h,
25

---

Published as a conference paper at ICLR 2026
which is significant when using more expressive component functions. This is the strategy employed in our experiments when training LRNNs with large projection widths.
Kernel fusion. Our standard implementation calculates projections, component functions, and products as separate kernel calls, necessitating repeated reads and writes of the large intermediate tensor of size B x r d to global memory. This can be alleviated through kernel fusion, where multiple operations are combined into a single GPU kernel. By loading the projected inputs z, computing g(z), and reducing the product entirely within GPU registers, the intermediate tensors need not be materialized in GPU memory. For the backward pass, the kernel can re-load z, re- compute the values, and generate gradients directly. This effectively shifts the bottleneck from memory capacity to arithmetic intensity.
Mixed precision and stability. Finally, we note that the suriance-controle initialization result in Lemma ensures that the product terms do not suffer from numerical instability (exploding or vanishing values). This stability makes LRNNs well-suited for training in half-precision (FP16 or BF16), thereby reducing the activation memory footprint by 50% compared to FP32 training.
C ABLATION STUDIES
C.I ABLATION STUDIES ON PARAMETER- LRNN MODEL VARIANTS
To investigate the effect of the parameter-sharing LRNN model variants, we perform an abla- tion study. We compare the performance of the flexible LRNN-SPDER (Flex), with its reduced- complexity variant with shared activation functions (SA) on the INR image representation task for two images. For cameraman (Figure 11 left), we observe that for medium-sized models, the SA variant has better parameter efficiency, achieving higher PSNR at the same complexity as the Flex model. However, at higher parameter counts, the PSNR difference becomes negligible, suggesting tradeoff between expressivity and efficiency. For retina (Figure 1 right), the SA model has small benefits parameter efficiency at higher parameter counts. This study indicates while both flexible and reduced-complexity LRNN models perform well, the variant selection is dependent on the task and the computational budget available.
Note that in terms of timing, typical run of LRNN-SPDER variants with comparable hyperparam- eters ran approximately 50 iterations per second for Flex and 37 iterations per second for SA.
Figure 1I: Model complexity neural scaling laws comparing flexible (Flex) and shared activation (SA) variants of LRNN-SPDER on INR image representation tasks for cameraman (left) and retina (right).
C.2 ABLATION STUDIES ON THE USE OF LAYERNORM
LayerNorm was applied only to LRNN in the numerical experiments and not to the baseline mod- els. The reason for this highlights key difference in architectural design. The stability of standard SIREN/SPDER models comes from carefully derived and principled weight initialization scheme that preserves the distribution of activations through the network so that the final output at initializa- tion does not depend on the number of layers. Without this, the accuracy and convergence of deep SIREN/SPDER networks can be very poor.
26

---

Published as a conference paper at ICLR 2026
Our LRNN architecture, however, introduces a fundamentally different activation structure: prod- uet of univariate functions. While we use SIREN style initialization for each individual component function g5. the statistical properties of their product are distinct and more complex than those of single sin function. The SIREN initialization scheme, on its own, is no longer sufficient to guaran- tee that the final output of this product activation will he well-behaved, especially in deep networks. Therefore, we employ LayerNorm as necessary additional step. It acts as dynamic normalization layer that explicitly re-centers and re-scales the output of our product-structured activation after has been computed. This enforces stability by ensuring the inputs to the next layer are consistently well-distributed, regardless of the complex interactions within the product.
To quantify the necessity of this approach, we ran an ab- lation study on the audio representation task. The result in Table shows that removing LayerNorm significantly degrades performance, resulting in a final error that is orders of magnitude higher. This confirms that normal- ization is crucial for stabilizing the training dynamics of deep LRNNs by controlling the scale of the activations passed between layers.
Table 3: LayenNorm ablation study on bach audio for LRNN-SPDER.
Final MSE after 1000 steps
LayerNorm
Yes
3.58e 5
No
2.41e 2
C.3 ABLATION STUDIES ON COMPONENT ACTIVATIONS
To provide quantitative evidence on the role of component activations, we ran an ablation study on the Cameraman and Retina tasks. We compared our standard LRNN-SPDER to an LRNN using
ReLU and Tanh activations in its component functions. The results in Tables 4a 4b
demonstrate
Table 4: Activation ablation study on Cameraman and Retina images.
(a) Cameraman image.
(b) Retina image.
LRNN Component Activation
Final PSNR after 1000 steps (dB)
LRNN Component
Final PSNR after 1000 steps (dB)
Activation
SPDER
107.94
SPDER
47.02 22.55
ReLU
14.40
ReLU
Tanh
14.42
Tanh
16.77
that using bounded, periodic activations is key for high performance on INR tasks. This highlights clear design principle for applying LRNNs in practice. Lemma 2 provides further insight. It shows that by learning the parameters of these component functions (e.g., their frequencies), LRNNS can dynamically control their spectral bias, allowing them to represent a much richer set of frequencies com-
than fixed-activation models like SIREN. This is key reason for their superior performance plex signals. The most important takeaway is that the LRNN structure itself provides a significant performance boost, regardless of the component activation. Our experiments show LRNN-SIREN outperforms SIREN and LRNN-SPDER outperforms SPDER. This demonstrates the general power of our learnable, factorized activation framework.
D SETUP FOR PRODUCT-STRUCTURE TEST FUNCTION SCALING STUDIES
To evaluate the scaling properties of LRNNs functions well-suited to their architecture, we de- fined test function with sum-of-products structure: y(x) Ei=
1 88 I[ of (xrz)- Here, define the component function as (x,,) = sin(nzff=,)) = 1,2..... d. For fixed (n, i, d), we randomly sample 8; and & each from a standard normal distribution. We used Sobol sampling to generate Nerain points as the training dataset for both LRNN and MLP architectures, and an inde- pendent set of Ntest points for evaluating model performance. To generate the scaling laws, Ntrain was fixed at ~ x 10t and we trained LRNNS and MLPs with varying number of learnable parame- ters. As shown in Figure 2 in the main paper, the shallow LRNN (with hidden layer) architecture achieves better generalization performance at lower model complexity than shallow MLP (with one hidden layer). A decrease in test error with increased model size was observed for the shallow
27

---

Published as a conference paper at ICLR 2026
MLP, which implies that this model struggled learn this particular function effectively. The pected trend of decreasing test error with increased model complexity was observed for MLPs when using two hidden layers, with further improvement low LRNN achieved lower test error using fewer parameters than even the three-layer MLP. This indicates that LRNNS exhibit strong scaling capabilities for this class of functions, requiring lower model complexity than standard MLPs to effectively learn data with an inherent sum-of-products structure.
with three hidden layers. However, the shal-
IMAGE REPRESENTATION
The implementation details to reproduce the results for the image representation studies on the cameraman and retina images are presented in Table 5 Note that the nuthor-recommendee settings were used for benchmark models, SIREN (Sitzmann et al. 2020) and SPDER (Shah & Sitawarin, 2024).
Table 5: Implementation details for image representation experiments.
Setting
Details
Iteration count Optimizer Learning rate
All models were trained for 1000 steps.
Adam (Kingma & Ba 2015) was used for all models.
Ir
10 for SIREN and SPDER, as recommended
in (Sitzmann et al. 2020: Shah & Sitawarin 2024).
Ir = 10 for LRNN.
Scheduler
No scheduler for SIREN and SPDER, consistent with (Sitzmann
ct al. 2020 Shah Sitawarin 2024).
LRNN used StepLR from the standard torch. package with step size 100 and decay factor y = 0.8 for Cameraman, 7 = 0.9
for color images.
Benchmark
Three layers of 256 neurons with wi = 30 for both SIREN and SPDER, per (Sitzmann al.. 2020 Shah & Sitawarin 2024).
hyperparameters
LRNN hyperparameters
Two LRNN layers, each of rank 106, win = 30, d 16.
Component functions were single-layer neural networks with hidden neuron and the SPDER activation function
(sin(x) / (x) with w = 30, yielding a parameter count comparable to the other models.
Following the encouraging results on the grayscale image, we conducted a similar scaling study on color image. The aforementioned models were evaluated on the 256 x 256 retina image. Figure 12 shows the reconstructed images of all models compared to the ground truth image and the PSNR convergence plot. observe that LRNN still outperforms the benchmark models, although margin of improvement is smaller than in the grayscale case.
PShall (48)
Motel (depth)
LRNN-SPOER (2)
SPDER: (3)
SIREN (5)
WintE (3)
RELu (2)
300
403
600
000
1000
Taining Steass
Figure 12: Retina image: ground truth (left), reconstructed images using LRNN, SPDER, SIREN
and WIRE models (middle), PSNR convergence history (right).
28

---

Published as a conference paper at ICLR 2026
To evaluate computational efficiency, measured time-to-solution for specific PSNR targets and presented the results in Tables The results for the retina image show that while baseline models may be faster reach low-qualty targets, LRNN achieves the fastest wall-clock time for higher- quality results (over 35dB). This trend is even more pronounced on the cameraman image. While the per-iteration cost of LRNNS can be higher, the time required for high-quality results is significantly better, demonstrating a clear advantage for challenging tasks where quality is paramount.
Table 6: Wall-clock time analysis on cameraman image (seconds).
Max PSNR
Time to reach
Model
30dB
35dB
40dB
SIREN
35.27
8.0
N/A
N/A
SPDER
48.97
2.0
8.3
N/A
LRNN-SPDER
107.94
5.2
6.2
8.7
Table Wall-clock time analysis on retina image (seconds).
Max PSNR
Time reach
Model
30dB
35dB
40dB
SIREN
36.04
3.5
14.1
N/A
SPDER
43.99
2.9
4.8
12.8
LRNN-SPDER 47.02
5.1
5.9
9.3
We present the iteration time for each model for both the cameraman and retina images in Table 8 is observed that while LRNN achieves the highest performance in terms of PSNR, it does require more compute time per iteration compared to some benchmarks like SIREN and WIRE (for Retina).
Table 8: Time taken per iteration when training comparably-sized models for INR image represen- tation.
Image
LRNN
WIRE
SIREN
SPDER
Cameraman
0.0453
0.0324
0.0174
0.0332
Retina
0.0504
0.0220
0.0072
0.0434
Studies were performed on higher-resolution images, kodak and parrot, using the code-base from (Saragadam et al. 2023) for the methods we benchmark against. For these experiments, we ran the benchmarks using models with three hidden layers containing 256 neurons. For SIREN and SPDER, we set wi = 30 and for WIRE, we set wh = 20, on = 30, following the respective author's recommendations. The GAUSS and ReLU+PE models from (Saragadam et al.) 2023) also used architectures of three hidden layers with 256 neurons each. To match this parameter count, chose two-layer LRNN model with each layer being rank 106, wo 30, d 16, and the component functions being single-layer neural networks with one hidden neuron. As in the previous studies, we used the SPDER activation function sin(2) /|2l for the MLPs in the LRNNs' product- structured activation function. Each model was trained for 1000 epochs using the Adam optimizer with batch size of 16384. The LRNN models were trained with initial learning rate of 10 10-3 and learning rate scheduler (StepLR) from torch. opth. with step size of 100 and a decay rate 7 = 0.9. All other models used the LambdaLR scheduler from the torch package to reduce the learning rate to 0. xx the initial learning rate the final epoch, as used in the WIRE experiments (Saragadam al. 2023).
For the higher- kodak image, we compared the LRNN model's 's reconstructed image and the corresponding gradients iterations 10 and 500 in Figure 13 We observe a fair reconstruction and gradients at step 10, and by step 500, the reconstructed image is virtually indistinguishable. Comparing the training loss and PSNR of LRNNN and the 5 benchmarks (SIREN, WIRE, GAUSS,
29

---

Published as a conference paper at ICLR 2026
ReLU (with positional encoding), and SPDER) in Figure 14 training loss and higher PSNR than the benchmark models throughout training.
observe that LRNN achieves lower
Model Outuu of 10
Oriund Tuth Irage
Cradkents at Sies
Hodel Ouip at Step 300
GreuRe Tuth Image
Gradients Step S26
Figure 13: Model output, ground truth and gradient of kodak image at steps 10 and 500.
Taining Los for fammnt Modilll for cddit image
PSUR for Moreet Nodall Kodak iage
1
Training loss and PSNR for kodak image over 1000 iterations. The performance of LRNN is compared against SIREN, WIRE, GAUSS, ReLU (with positional encoding), and SPDER.
Figure 14:
For the parrot image, we again compare the LRNN model output and gradients steps 10 and 500 in Figure 15 The model captures the parrot in the foreground well, and significantly detailed by step 500. observe that the L.NN-rreonstrructed image at step 500 virtually indistinguishable from the ground truth. Comparing the training loss and PSNR of LRNN and the 5 benchmarks in Figure 16 we observe that LRNN outperforms the benchmark models throughout training.
E. IMAGENET STUDY
We provide additional analysis of the ImageNet representation study presented in the main paper, where all models were configured with comparable parameter complexity of 200k. Figure I7 il- lustrates the distribution of PSNR values at multiple training checkpoints. We observe that LRNNs exhibit significantly lower performance variance compared to the baselines. The distributions for SIREN and SPDER show a prevalence low-PSNR outliers, indicating that these models fre- quently fail to reach high fidelity. In contrast, LRNNs consistently attain high PSNR values across seeds and images. This indicates that LRNNs offer superior representational capacity and robust- ness this parameter budget, consistently satisfying high-precision requirements where baseline architectures often fall short.
Table 9 summarizes the success rates and average wall-clock time required for each model reach specific PSNR thresholds (33 dB, 35 dB, and 40 dB) across the 1,000 test images. The results highlight that while baseline models like SIREN may have lower per-iteration costs, they struggle to
30

---

Published as a conference paper at ICLR 2026
Modal Oupput Step
fruth Image
Gradiants
Figure 15: Model output, ground truth and gradient of parrot image steps and 500.
$0400
Figure 16: Training loss and PSNR for parrot image over 1000 iterations. The performance of LRNN
compared against SIREN, WIRE, GAUSS, ReLU (with positional encoding), and SPDER.
31

---

Published as a conference paper at ICLR 2026
scale to high precision; for the 40 dB target, LRNN achieves 100% success rate, whereas SPDER and SIREN succeed in only 26.4% and 1.8% of cases, respectively.
Epoch 1000
45
43
PSTR
40
PSUR
40
PSWR
PSAR
35-
35-
10
10
30
33
25-
25
25
25
20
20
20
UKW SPDEn
SPDER
SPIEN
URM SPOOR
LANN-SPOCR
SIZEN
LANN
SP324
5RCW
Figure 17: PSNR distribution after different epoch numbers for different models run on 1000 Ima- geNet images for three seeds each.
Table 9: Time reach target PSNR for different models on ImageNet (average 1000 images and 3 seeds). Values show average time + standard deviation in seconds, with success rate in parentheses.
Target PSNR (dB)
Model LRNN-SPDER SPDER
Avg Time + Std (Success %)
Failures
4.55 2.87 (100.0%)
33
4.58 + 2.29 (96.7%)
99
SIREN
3.87 = 1.23 (50.7%)
1480
LRNN-SPDER
4 (100.0%)
35
SPDER
5.48 - 2.47 (83.8%)
487
SIREN
4.17 + 1.28 (26.5%)
2204
ARNN-SPDER
7.01 = 3.27 (100.0%)
0
40
SPDER
6.90 2.11 (26.4%)
2207
SIREN
4.27 + 2.03 (1.8%)
2947
F AUDIO REPRESENTATION
The implementation details for the audio representation study presented in the main paper are out- lined here. The code used to produce these results is based the script provided with (Shah) & Sitawarin 2024) which provides results for baseline models, SIREN (Sitzmann et al.] (022) and SPDER (Shah Sitawarin 2024). We create LRNN-SIREN and LRNN-SPDER by using (2) (Sitzmann et al. 2020) and sin( arctan(z) (Shah & Sitawarin, 2024) activations, respec- tively, within their MLP-based univariate component functions, comparing them against their base- line counterparts. The benchmark models, SIREN and SPDER have five layers of 256 neurons each. To match this complexity, the LRNN models have three layers, each of rank 118, with d 10 and component functions being single-layer neural networks with four hidden neurons. For all models, we set wi
(with inputs normalized to [-100, +- 100) following (Shah & Sitawarin 2024) and trained them for 1000 iterations of the Adam (Kingma & Ba 2015) optimizer with initial learning rate set to x 10-4, While SIREN and SPDER had no scheduler on the learning rate to match the original authors" implementation, LRNN with a step size of 100 and 7 = 0.8.
chose StepLR scheduler from the torch.optim package for
We present the absolute error between the prediction and the ground truth in the time domain and the frequency domain. We observe that for both audio clips (bach Figure 7 and counting in Figure 18), the LRNN model predictions match the ground truth closely than the respective
32

---

Published as a conference paper at ICLR 2026
baseline model and hence their error magnitudes smaller than that of the baselines in both time and frequency domains. This is consistent with the statistical results presented in Table in the main paper. Note that the y-axis for these absolute error frequency domain plots is clipped at 150 for of presentation, which excludes few outliers of higher errors in SIREN and SPDER predictions; despite this, LRNN variants exhibit visibly lower profiles.
SIREN
SPOER
LANN-SPDER
Pooolute Eroor
0.3
0.2
0:2-
0.2
Timel [s)
Time (k)
Time [s]
198
150
100
100
Abootute
Frequency (ellg)
Frequency (kH12)
Frequency Odfici
Shop
Absolute error time and frequency domain and convergence of training MSE loss (mean =lo) for counting audio representation tasks for comparably sized models.
Figure 18:
Table 10: Comparison of MSE loss and PAG (mean + stddev) across architectures and audio clips different training steps averaged over 10 runs.
Metric
Audio
Step
LRNN SPDER
SIREN
SPDER
50
1.07 0.12
4.98 10-3
2.43 0.27 10-3
100
2.36 0.55 x 10-4
2.00
10-3
1.51+0.7 10-3
bach
500
1.70 x 10
4.30
10
3.58 1.58:..24
10
1000
1.01 x 10-5
1.21 10-4
1.12 0.05
10-4
1.87
10
7.24 10-3
4.10 10
100
5.45 x 10
4.27 10-3
143+0.22 10-3
counting
500
9.92 0.82 x 10
5.72 10"4
3.92 # 0.75
10
Loss (L)
1000
7.15 x 10
10 2.77
2.29 1 0.55
10
1.03
10
1.57 10-2
1.33 0.19 10
100
5.69 + 0.67 x 10-3
1.37 10-2
1.14 0.38 10-2
reggae
500
1.21 0.14 x 10
5.02 10-3
4.29 1 0.51 10-3
1000
0.01 10
2.13 10-3
2.48 4 0.77 10
2.09
3.72 10
3.09
10
100
1.98 x 10-3
3.05 10-3
2.58 + 0.09 10-3
reading
500
5.63 + 0.08 x 10
1.67 - + 10
1.41 + 0.14
10
1000
1.86 1 0.03
10
9.98 - 0.16
10
8.88 4 0.25
10
0.9860 0.0017
0.9078 0.0094
0.9614 0.0052
100
0.9964 = 0.0010
0.9658 + 0.0028
0.9727 0.0147
bach
500
0.9998 - 0.0000
0.9930 + 0.0007
0.9943 0.0041
1000
0.9999 0. 0.0000
0.9986 + 0.0005
0.9988 0.0003
50
0.9281 = O. 0.0099
0.5020 + 0.0205
0.7989 + 0.0254
100
0.9765 = 0.0014
0.7789 + 0.0276
0.9425 = 0.0115
counting
500
0.9959 = O. 0002
0.9768 + 0.0021
0.9861 = 0.0014
PAG (1)
1000
0.9967 = 0.0002
0.9906 + 0.0015
0.9937 + 0.0006
0.7786 0.0114
0.6107 + 0.0055
0.6896 0.0055
100
0.8974 = 0.0072
0.6816 0.0039
0.7475 - 0.0090
reggae
500
0.9809 = 0. 0.0005
0.9330 - 0.0062
0.9449 = 0.0035
1000
0.9860 = 0.0002
0.9769 1 0.0011
0.9729 1 0.0010
50
0.6315 = 0.0171
0.3100 1 0.0075
0.5246 + 0.0081
100
0.7430 = 0.0133
0.3883 + 0.0070
0.6301 + 0.0210
reading
500
0.9508 = 0.0082
0.8069 + 0.0070
0.8476 + 0.0088
1000
0.9862 = 0.0031
0.9193 0.0094
0.9324 + 0.0104
33

---

Published as a conference paper at ICLR 2026
G NUMERICAL SOLUTION OF PDES
For the task of PDE solution approximation with INR models, we use the Poisson equation Uxx Muy f on the square domain f2 [-1,1]² subject to zero Dirichlet boundary conditions from (Liu et al. 2025). The source term is defined f(z,
> + 4y²) sin( (ann(r))sinnnrlg) sin(TTy") + 2TT (TT( cos my and the exact solution is given by u(x, y) = (in() (T)) sin (iy")) The train- ing objective is composed of two parts: residual loss the PDE interior and boundary loss, formulated as LYDE = aLiim + Lodry, where
Litt
(2x( + (zy f(z))1 and Ladry
Σ (2,). Tbb
Here, {2, (z } are collocation points sampled uniformly within the domain for the interior loss, and on the boundary for the boundary loss. Following (Liu et al. 2023). we set = 0.01.
For the LRNN model, we used two-layer architecture with d 12 and MLP with one hidden layer with one neuron for each component function together with the SIREN (sin(x)) activation function. SPDER was considered a benchmark for this problem as we were unable achieve competitive performance with the / |ax] damping factor. The rank the LRNN layers were chosen from the set {16, 32, 48, 64] to study the scaling properties of LRNNs. For both the LRNN and SIREN models, we chose w = 6.0 as the frequency parameter for all layers. The MLP used the SiLU activation function following the setting in (Liu et al. 2025). We used spatial grid of 41 41 collocation points to compute the loss function. We used learning rate of 10 for LRNNS and MLPs and learning rate of 10 for SIREN. All models were trained for 1000 epochs using the Adam optimizer.
Table 11: Comparison of different methods for the two-dimensional Poisson equation (mean squared error). Results for KAN are taken from Liu et al. (2025).
Method
Rparams (x10²)
71
n1 2
71 =
MSE 4
MSE 4
MSE 1
MLP 12,118
34
6.4 x 10
1.9 x 10
4.6 x 10
MLP [2,256,256,256,1]
132
3.6 X 10-6
1.7 X 10-5
2.9 X 10
SIREN
34
5.0 x 10-7
1.2 x 10-4
1.9 x 10-3
SIREN [2.256.256,256.1]
132
1.3 x 10-7
8.9 X 10-6
5.5 x 10
KAN [2,10,1] G 10
3.6 X 10
1.8 X 10-2
0.531
KAN [2,10.1] G = 20
4.9 X 10
6.7 X 10
8.7 x 10
KAN [2,100,1] 10
1.0 x 10-6
3.6 10-5
9.8 10-3
KAN [2,100,1] 20
0.106
1.8 X 10-2
8.1 X 10-3
KAN [2.10,10,10,1] G 10
1.4 x 10-4
1.4 x 10-2
0.332
KAN 12, G 20
0.990
0.986
0.964
LRNN [2,16,16,1]
8.9 X 10-7
x x 10-5
3.7 X 10-3
LRNN [2,32,32.1]
16
1.5 x 10-7
2.7 10
5.5 x 10
LRNN [2.48,48.1]
34
1 10
8.3 x 10
1.5 x 10
LRNN [2,64,64.1]
57
x 10
6.6 x 10
6.6 x 10
The excellent performance of LRNNs on this particular problem (see Table 11 and Figure consistent with their architectural design, which inherently captures multiplicative int interactions. The exact solution possesses a product-separable structure, making it particularly well-suited for approx- imation by LRNNs. We provide graphical comparison of results obtained using LRNN, MLP and SIREN models Figure 19 For LRNNs we used forward mode AD to efficiently compute the Laplacian this was crucial prevent memory issues arising from the effective number of inter- mediate activations associated with its product- structurd activation function when the model size is increased.
g
34

---

Published as a conference paper at ICLR 2026
High Frequency Poisson (n-4) URNN vs Exact (Epochs: 1000)
toard Solution
UTAN Approoamation
Aboolate trte (uog Scale!
300
1.00
320
400
-255
-035
Doct Solidion
MLP Aippnsmttatioo
Aboolinn Empritag
353
2.50
000
053
8.5
425
025
-030
1550
+058
00.00
-635
-5.5
-073
High Frequency (n=4]
(Epochs: 1000)
1P1
375
4.7
035
353
5.00
035
622
-225
22
-25
C50
030
439
Figure 19: Comparison of approximations obtained using LRNN model with 64k parameters and the MLP and SIREN models with 132k parameters for the Poisson PDE (n = 4). The first row shows the LRNN approximation, the second row shows the MI.P approximation, and the third row shows the SIREN approximation. As shown in Table in the main paper, the small LRNN model significantly outperforms the MLP and SIREN models in terms of the MSE.
35

---

Published as a conference paper at ICLR 2026
Note that the results for KAN in Table 11 of the main paper were taken directly from (Liu al. ).23). as we encountered difficulties in reproducing their reported results using the publicly avail- able code.
H CT RECONSTRUCTION ABLATION STUDIES
These experiments were performed by adapting the script for CT reconstruction provided by Sara- gadam et al. (2023). who introduced this benchmark problem. All models were run for 5000 itera- tions. The learning rates for the benchmarks were consistent with those recommended by |Saragadam et al. )023): 5 x 10-3 for WIRE,
10-1 for ReLU with positional encoding, and x 10 for SIREN and Gauss. We also chose Iy = 1 x 10 for LRNN. For the study presented in the main paper and the ablation study presented below for number of projections, the model sizes are as fol- lows: 3 layers of 256 neurons each for all benchmarks, and for LRNN, two-layer model with each layer being rank 82, d = 10, and the component functions being single-layer neural networks with four hidden neurons match the complexity of the benchmarks. For only the LRNN model, a CeduueeLRonPlatte patience of 25. All other models used the LambdaLR scheduler from the torch.optim package to reduce the learning rate to 0.1x the initial learning rate in the final epoch, as used in the WIRE paper experiments. For the study in the main paper and the ablation study on model complexity, all models were provided with 100 CT projections. The results of the two ablation studies are discussed below.
scheduler from the torch.optim package was used with a factor of 0.5 and a
Number of CT Projections. The CT problem was solved by all five models of similar complexity (1.8 x 10a parameters) for different numbers of CT measurements. We compare both the PSNR and SSIM in Table 12 We also consider the reconstructed images in Figure 20 compared to the ground truth in Figure 7 in the main paper. None of the models have clear reconstruction with only 20 projections, however LRNN and SIREN perform decently well with only 50 projections. As demonstrated in Figure LRNN is the top performer in PSNR at both 50 and 100 projections and in SSIM at 100 and 150 projections. When more projections (150, 200, 300) are available, WIRE achieves the highest PSNR and LRNN second-best by a small margin of at most 1.53dB. This indicates that while LRNN has some limitations. it is a good choice for this problem, particularly in sparse-view cases with less CT measurements to limit exposure to patients.
Table 12: Comparison of PSNR and SSIM across models for different numbers of CT measurements.
Number of CT Meas.
LRNN
SIREN
WIRE
Gauss
ReLUU+P
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
25.60
0.5449
25.59
0.5646
0.2199
0.3351
25.75 35.75
0.5926
50
28.26
0.6766
27.44
0.6817
24.63
0.4006
26.54
0.5680
26. 18.23
0.6180
100
29.13
0.7455
27.16
0.6877
28.83
0.6113
27.84
0.6855
26.89
0.63/11
150
29.43
0.7557
27.45
0.6854
30.42
0.7470
27.79
0.6945
26.97
0.6327
200
29.71
0.7578
27.31
0.6871
31.
0.78.19
27.80
0.6926
26. 26.94
0.6221
29.77
0.7713
27.53
0.6970
0.7739
27.99
0.7044
27.05
0.6094
Model Complexity. An ablation study also performed to determine the impact of model com- plexity, as defined by total number of model parameters, on performance on the CT reconstruction task. All models were provided 100 CT measurements. The range of model size tested varied from 5.9 x 10a to 19.9 x 10ah Based on Table 13 and Figure 22 we observe that all models with the exception of WIRE do not exhibit any sigmificant difference PSNR over the range of model com- plexities-ther they stay within the same 1dB range. For this reason, the reconstructed images are not presented since they would be difficult to differentiate with the human eye. The models tend to follow trend of marginal increase in PSNR with significant increase in parameter count. There are some outliers to this trend, most notably the largest WIRE model. This may be explained by the fact that the smaller models had depth of two, while the largest benchmark models comprised three layers, which may not be well-suited to the WIRE framework. We observe that for 100 projections, LRNN achieves the highest PSNR at all tested model sizes. Overall, this ablation study suggests that model complexity does not have significant impact on performance once a certain minimum complexity is achieved. This is useful to note in problems limited by computational budget.
36

---

Published as a conference paper at ICLR 2026
projections
projections
100 projections
150 projections
projections
300 projections
LRNN
SIREN
Figure 20: Comparison of CT reconstructed image by similarly complex models for different num- bers of projections.
WIRE
29.0
Gauss
ReWU+PE
PSNR (dB)
28.5
27.5
26.5
100
150
175
200
225
Number of Parameters (x103)
Figure 21: Comparison of PSNR for similarly complex models given different numbers of CT mea- surements.
Table 13: Comparison of PSNR and SSIM across models for different number of parameters (100 projections).
Parameters (x10*)
LRNN
SIREN
WIRE
Gauss
ReLU-PE
SSIM
PSNR
SSIM
PSNR
SSIM
5.90
0.7167
26.91
0.6530
27.88
0.6041
27.67
0.6670
26.51
0.6142
9.00
29.00
0.7297
27.09
0.6655
28.30
0.6358
27.64
0.6727
26.48
0.6111
12.85
29.63
0.7392
27.21
0.6705
28.84
0.6179
27.73
0.6702
26.63
0.6250
15.50
29.33
0.7547
27.53
0.6007
28.03
0.6425
27.81
0.6860
26.66
0.6269
18.
29.13
0.7455
27. A6
0.6877
28. 83
0.6113
27.84
0.6855
26.89
0.63
0.6966
26.58
0.5005
37

---

Published as a conference paper at ICLR 2026
LRNN
29.5
29.0
Gauss
RELU-PPE
PSNR (dB)
28.0
27.5
26.5
75
100
125
150
175
200
225
Number of Parameters (x103)
Figure 22: Comparison of PSNR for models of varying complexity given the same numbers of CT measurements.
I STUDIES ON IMAGE CLASSIFICATION DATASETS
We present controlled experiments on classification benchmarks to isolate and evaluate the contri- bution of the LRNN layer building block. Our goal is not to achieve state-of-the-ard large- scale datasets perform rigorous ablation studies comparing LRNN layers directly against their MLP, CNN, and KAN counterparts under identical conditions. We selected MNIST and MNIST-ID datasets for these experiments since they enable controlled comparisons with well-understood baselines and the computational efficiency allows extensive ablation studies. Furthermore, MNIST- ID's procedural generation and shuffle variant specifically test architectural biases.
would require sophisticated architectures and training protocols- -but rather
We consider LRNN models which take the vectorized image input as well as ConvLRN, model similar standard CNN which involves convolutional and pooling layers followed by an LRNN layer (instead of an MLP block) 4 Our numerical experiments demonstrate that:
1. MNIST-1D benchmark: On the MNIST-ID benchmark, both LRNN models consistently outperform their direct MLP and CNN counterparts; see Table 14 and Figure 23
2. Shuffled pixels test-case: the MNIST-ID benehmark with shuffled pixels, both LRNN models maintain higher accuracy than their counterparts, suggesting they learn different feature representations (see Table 14p.
3. General-Purpose Layer (LRNN vs. MLP vs. KAN): When treating MNIST as a vector task, our LRNN achieves 98. 1% accuracy, which improves upon the standard MLP's 97.0% accuracy and matches the accuracy much larger KAN model while being z=50x faster train (Table 15).
4. As Component in CNN (ConvLRNN vs. a baseline CNN): matches the 99,1 19.199 % accuracy of the baseline CNN on MNIST, but with only one-third of the parameters (77k vs. 225k, Table 15).
The ConvLRNN model
We now present the detailed experimental setup and results. We first evaluated LRNN and ConvL- RNN on MNIST-ID (Greydanus & Kobak (024) against CNN and MLP baselines. All models were run with 5 different random seeds and the average results are presented in Table 14] We observe that ConvLRNN outperforms the CNN in terms of test accuracy on the unshuffled data as well as train and test accuracy on the shuffled data. The CNN has high accuracy on unshuffled data, but struggles
Note that it is also possible to modify the LRNN neuron entirely replace the convolutional layers and incorporate positional encoding instead of typical CNN's fixed encoding. We leave this for future work.
38

---

Published as a conference paper at ICLR 2026
to perform shuffled data. Although ConvLRNN's performance also degrades shuffled data, maintains 98, 48.48 train accuracy (vs CNN's 94.46%) and achieves 64.04% test accuracy a percentage point improvement over CNN's 57.16%.
Table 14: Classification accuracies on MNIST-1D.
Model
71 Aparrma
Unshuffled
Shuffled
Train Ace (%)
Test Acc (%)
Train Ace (%)
Test Acc (%)
CNN
5.21 x 103
100.00 + 0.00
93.06 + 0.80
94.46 + 1.48
57.16 +:2 1.29
MLP
1.52 x 10a
100.00 + 0.00
65.00 + 1.30
100.00 0.00
65.20 + 1.42
COnYLRNN
1.79 x 10
100.00 + 0.00
94.12 + 0.47
98.48 + 1.44
64.04 + 2.49
LRNN
4.63 x 10
100.00 0.00
67.18 + 0.78
100.00 0.00
67.34 + 0.85
MLPs on the other hand perform consistently whether the data is shuffled or unshuffled, although the unshuffled test accuracy is much lower than that of the convolutional models. LRNN outper- forms MLP by 2-3% on both the shuffled and unshuffled data. Overall, ConvLRNN has the highest performance on the unshuffled data while LRNN does the best of these benchmarks on the shuffled data. To graphically illustrate this, we present plots of the train and test accuracy for typical run in Figure 23 We use this example to demonstrate the versatility LRNNs and their applicability as general-purpose layer illustrated by the broad range of tasks it performs well on.
Training accuracy
Test accuracy
100
100
90
85
Accuracy
80
75
CNN
70
MLP
ConVLRNN
After shuffling
2500
5000
7500
2500
5000
7500
step
Training step
Figure 23: Train and test accuracies MNIST-ID ID.
We evaluated five models standard MNIST: three using the flattened 784-dimensional vector representation (MLP. KAN, LRNN) and two using the 2D image structure (CNN, ConvLRNN). Results are presented in Table 15
For the vector-based models, LRNN achieves 98.1% 1% test accuracy, outperforming MLP (97.0%) while matching KAN's accuracy. Critically, LRNN trains in 19.38 seconds compdrabl MLP's 16.65 seconds and 55x faster than KAN's s 1064 seconds. Despite having fewer parameters than KAN (1.04M vs 1. 1IM, LRNN delivers equivalent accuracy with dramatically better computational efficiency.
For the convolutional models, ConvI matches CNN's 99.1 accuracy while using only 77k parameters- 3x reduction from CNN's 225k parameters. This demonstrates that LRNN layers can serve as efficient drop-in replacements in standard architectures, maintaining performance while significantly reducing model complexity.
These controlled experiments demonstrate that the LRNN layer provides tangible bene- fits poss-prretter efficiency and computational speed-whhte used as a drop-in replacement for stan- dard layers. While we do not claim state-of-the-arr on classification (our focus remains on con- tinuous signal representation). these results validate IRRNN versatile building blocks that could
39

---

Published as a conference paper at ICLR 2026
Table 15: Model performance on MNIST.
Train Accuracy (%)
Test Accuracy (%)
Model
Uparams
Train Loss
Wall Time(s)
MLP
7.95 10
9.72
10
98.6
97.0
16.65
KAN
1.11
106
9.24 x 10 3
100.0
98.1
1063.99
CNN
2.25
10a
7.05
10
99.8
99.1
382.51
LRNN
1.04
109
1.19
10-2
100.0
98.1
19.38
ConvLRNN
7.68
10*
8.30
10-3
99.8
99.1
422.31
benefit future architectural designs. Exploring LRNN integration in modern architectures for com- plex classification tasks remains interesting direction for future work.
40


Reviews:
[{"review_id": "OOpBGx8yo3", "reviewer": "Reviewer_XhCn", "summary": "The paper presents a method for learning more expressive activation functions as products of small neural networks themselves.  Theoretical results are presented on the universal expressivity of this parameterization and its ability to efficiently represent functions which have low rank structure to exploit.  Principled initialization schemes are proposed and theoretically justified.  Experiments on function reconstruction tasks show a significant improvement compared to state of the art architectures, while experiments with standard classification tasks show a marginal improvement to test accuracy.", "questions": "1. What are the architecture size/shape parameters (depth/width) of the small MLPs used for the g's throughout?  \n\n2. How sensitive is the performance to these architecture size choices for the g's?  As I understand the scaling law experiments increase the parameter count with the width of the small MLPs, is there sensitivity to the number of layers or rank as well?\n\n3. Is there a typo in eq (2) and where this appears in the appendix as well?  An extra comma on the right-hand side.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written with a clear exposition and convincing experimental results.  The theoretical results were also presented in a clear and complete manner.  I particularly appreciated the commentary and theoretical results on parameter complexity, this is often omitted during discussions of universal approximation.  This work gives a significant improvement to architectures for signal reconstruction specifically and should be accepted.", "weaknesses": "Some minor comments for improving the development:\n\nThe commentary on parameter sharing at the end of Section 3.2 was of immediate interest during a first read but is minimal in the main text and pushed to the appendix.  An extra comment here and at least a reference to where this appears in the appendix would make for a smoother read.\n\nThere is a single comment on memory considerations in the concluding remarks, but having more commentary on this would be very useful for practitioners interested in using this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6JaNQqoYOH", "reviewer": "Reviewer_VXpv", "summary": "The paper introduces Deep Low-Rank Separated Neural Networks (LRNNs) — a new neural network architecture that generalizes multilayer perceptrons (MLPs) by endowing each neuron with a learnable, product-structured activation function. Instead of relying on fixed non-linearities (e.g., ReLU, Tanh, Sine), LRNNs construct activations as multiplicative compositions of learnable univariate functions. This design captures multiplicative and higher-order interactions efficiently, inspired by low-rank separated representations and tensor decomposition theory. The authors provide theoretical guarantees, including universal approximation, variance-controlled initialization, and curse of dimensionality mitigation for low-rank structured functions. Empirical results are demonstrated on images, audio, PDE solving, sparse-view CT reconstruction. Overall, LRNNs offer a unified, theoretically grounded architecture that merges ideas from tensor decompositions, implicit neural representations (INRs), and adaptive activations.", "questions": "1. Can author(s) discuss how the effective separation rank evolve during training? Does over-parameterizing  r lead to redundancy or implicit sparsity (similar to low-rank dropout)?\n2. The variance-controlled initialization ensures bounded gradients theoretically. In practice, do we observe gradient explosion/vanishing for deeper networks?\n3. Can there be a discussion around the interpretability of LRNNs?\n4. Could the product-structured activation principle be incorporated into attention or convolutional blocks to yield hybrid LRNN-transformer models?\n5. Given the increased per-layer product cost, can the author(s) comment on the potential use of mixed-precision or kernel-fusion optimizations to reduce compute overhead?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well-written and motivated.\n2. Related work section is well rounded.\n3. The paper is easy to follow.\n4. The idea of learnable product-structured activations is novel unifying classical low-rank function decomposition with modern deep learning architectures.\n5. The theoretical results are are well motivated and rigorously derived.\n6. Experiments span multiple domains such as image and audio INRs, PDE solving, and CT reconstruction — demonstrating the versatility of LRNNs.\n7. LayerNorm regularization, parameter sharing across layers, and use of small MLPs for component functions are well justified.\n8. I believe this can open up new architectural choices for many domain including scientific machine learning.", "weaknesses": "1. Explicit computation requirement comparison is missing for the numerical experiments, these would make the proposed architecture much more favourable.\n2. The paper acknowledges that LRNNs require higher memory due to product-term storage. More quantitative profiling (time per iteration, memory vs MLPs) would help assess practicality for large-scale settings.\n3. The plot(s) can be made more clearly legible.\n4. Some ablations regarding the architecture choices should be discussed in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzCNDltiJM", "reviewer": "Reviewer_XNN7", "summary": "This paper presents a novel approach to expressing a neural representation, primarily utilizing MLPs, by introducing a low-rank formulation and incorporating activation functions as learnable univariate functions. The paper presents preliminary results on the tasks of signal representation and inverse problems, including sparse-CT and super-resolution.", "questions": "1. For tasks such as 3D scene reconstruction (NERFs), where signal representations are most widely used, have the authors tried/observed any significant performance improvements (faster convergence?) \n\n2. L78: The Authors claim that a low-rank assumption would enhance expressivity. However, that relies on a few key assumptions: for the data to have a low rank of “r”, it would mean that it can be marginalized into r independent variables/distributions. Only in such a case could the LRNN’s design “enhance expressivity.” Is this true in the case of natural images or audio’s frequency spectra? \n\n3. Compared to PCA with “r” components, what benefit does a 2-layer LRNN yield, besides it being nonlinear?\n\n4. The results for the cameraman image appear to be inconsistent with those in Table 7 and Figure 3 (PSNR plot). In Table 7, it is reported that SPDR reaches 30 dB in 2 seconds, before LRNNs. However, the plot shown in Figure 3 doesn’t concur with this finding. It seems that both the figure and the table correspond to the same experiment. Please explain the discrepancy.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The premise of decomposing each layer into a low-rank layer is interesting and the preliminary results indicate its promising ability for signal representation and inverse problems. \n\n- The paper presents a careful explanation of how LRNNs are implemented, along with added derivations to ensure convergence and stability.  \n\n- The paper presents promising initial results, highlighting the impressive ability of LRNNs for signal representation and PDEs.", "weaknesses": "Lacks rigorous evaluation: The paper doesn’t show results on image representation or audio representation across a wide variety of samples. The ability of LRNNs to perform across a wide variety of data, not just a few select images, is crucial to understanding the value of this contribution.  Please display results on a large number of examples (e.g., 1000) such as Celeb-A, FFHQ, or ImageNet. Previous papers on signal representations (Functa[1,2], TransINR[3], STRAINER[4]) have shown a rigorous assessment of INRs on the task of image fitting. Furthermore, for tasks such as super-resolution and CT reconstruction, only one image/signal is used, which does not accurately convey the general performance of LRNNs.\n\nInsufficient analysis of LRNN: The paper does not address why a deep low-rank network is able to correctly capture high-quality image or audio-level details. Typically, low-rank approximations work when a distribution can be easily marginalized into “r” independent vectors. Given that LRNNs use a small “r”,  this seems rather counterintuitive as to why it would work. Any qualitative or formal insights would greatly benefit the paper.\n\n[1]From data to functa: Your data point is a function and you can treat it like one, Dupont et.al\n[2]Spatial Functa: Scaling Functa to ImageNet Classification and Generation, Bauer et.al.\n[3]Transformers as Meta-Learners for Implicit Neural Representations, Chen et.al\n[4]Learning Transferable Features for Implicit Neural Representations, Vyas et.al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9A8wGinjbY", "reviewer": "Reviewer_yobX", "summary": "This paper presents a novel architecture, low-rank separated neural networks (LRNNs), that can be understood as MLPs with learnable activations taking a particular product-factorized form. The authors introduce their architecture, compare it to an MLP, and prove a universal approximation theorem as well as theoretical results studying their architecture's parameter efficiency and inductive bias. They then carry out a thorough empirical study of their architecture, showing that it outperforms prior work in terms of parameter efficiency, representation quality, and convergence rate when used as an implicit neural representation, exhibits good parameter efficiency for solving low-dimensional Poisson equations, and achieves better reconstruction quality on audio representation and CT scan reconstruction.", "questions": "How should we interpret the assumptions for Theorem 2? In particular: \n\n-What are some examples of functions \"whose ANOVA decomposition is dominated by terms involving at most $m ≪ d$ variables\" and functions that violate this assumption? \n- Does the $m_a ≪ d$ condition in Assumption 3 of the proof (lines 1026-1031) correspond to a particular big-O or little-o bound on the interaction order $m_a$?\n- How restrictive is Assumption 4? It seems that it must be fairly restrictive, because if $m_a = d$, then one of the terms in the functional ANOVA decomposition involves all $d$ variables and the restrictions on $f$ come entirely from Assumption 4, but the complexity of the LRNN approximation still grows only polynomially with $d$ according to Theorem 2. So Assumption 4 should be pretty strong to mitigate the curse of dimensionality in this way.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, and I was able to follow the main ideas without much trouble. The authors generally do a good job of explaining the significance of their theoretical results. However, I have some questions about the assumptions for Theorem 2, which I detail below.\n- I appreciate that the authors have included theoretical analysis for their architecture's parameter efficiency in addition to the standard universal approximation results. I have some questions for Theorem 2, which I detail below.\n- The empirical results are strong across the board, though as I note below, LRNN's particularly strong performance in image representation might be overkill for real-world tasks.", "weaknesses": "- I'd appreciate the authors providing some more intuition for the assumptions of Theorem 2, since this is a key theoretical result motivating their new architecture. I include some specific questions below.\n- The time-to-solution results in Table 1 are a valuable addition to the other data on LRNN-SPIDER's image representation performance. One could argue that the very high PSNRs (>100 dB) reported for LRNN are overkill for real-world tasks, and that the baseline methods reach a threshold of sufficient quality faster than LRNNs. For instance, I cannot discern any visual difference between the SPIDER reconstruction (35.3 dB) and the LRNN reconstruction (107.9 dB) in Figure 3, nor can I discern a visual difference between any of the non-ReLU baselines for the retina image in Figure 10.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)