Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey
Authors: Amer Essakine
Yanqi Cheng
Chun-Wun Cheng
Lipei Zhang
Zhongying Deng
Lei Zhu
Carola-Bibiane Schönlieb
Angelica I Aviles-Rivero
Venue: Accepted by TMLR ()

Abstract: Implicit Neural Representations (INRs) have emerged as a paradigm in knowledge representation, offering exceptional flexibility and performance across a diverse range of applications. INRs leverage multilayer perceptrons (MLPs) to model data as continuous implicit functions, providing critical advantages such as resolution independence, memory efficiency, and generalisation beyond discretised data structures. Their ability to solve complex inverse problems makes them particularly effective for tasks including audio reconstruction, image representation, 3D object reconstruction, and high-dimensional data synthesis. This survey provides a comprehensive review of state-of-the-art INR methods, introducing a clear taxonomy that categorises them into four key areas: activation functions, position encoding, combined strategies, and network structure optimisation. We rigorously analyse their critical properties—such as full differentiability, smoothness, compactness, and adaptability to varying resolutions—while also examining their strengths and limitations in addressing locality biases and capturing fine details. Our experimental comparison offers new insights into the trade-offs between different approaches, showcasing the capabilities and challenges of the latest INR techniques across various tasks. In addition to identifying areas where current methods excel, we highlight key limitations and potential avenues for improvement, such as developing more expressive activation functions, enhancing positional encoding mechanisms, and improving scalability for complex, high-dimensional data. This survey serves as a roadmap for researchers, offering practical guidance for future exploration in the field of INRs. We aim to foster new methodologies by outlining promising research directions for INRs and applications.

Published in Transactions on Machine Learning Research (02/2025)
Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey
Amer Essaking Yanqi Cheng", Chun-Wun Cheng', Lipei Zhang', Zhongying Deng', Lei Zhu", Carola-Bibiane Schönlieb', Angelica I Aviles-Rivero
Department of Applied Mothemalics and Theorelical Physics, Universily of Cambridge ENS Paris-Saclay
ROAS & DSA, The Hong Kong University of Science and Technology (Guangzhou) Yau Mathemalical Sciences Center, Tsinghua University
Reviewed on OpenReview: http::///gpparrrreeeee////ooooep
Abstract
Implicit Neural Representations (LNRs) have emerged as a paradigm in knowledge represen- lation, offering exceptional flexibility and performance across a diverse range of applications. INRs leverage multilayer perceptrons (MLPs) model data as continuous implicit func- tions, providing critical advantages such resolution independence, memory efficiency, and generalisation beyond discretised data structures. Their ability to solve complex inverse problems makes them particularly effective for tasks including audio reconstruction, image representation, 3D object reconstruction, and dige-dimensional data synthesis. This sur- vey provides a comprehensive review of ttate-of-the-art INR methods, introducing a clear laxonomy that calegorises them inlo four key areas: activation functions, position encod- ing, combined strategies, and network structure optimisation. We rigorously analyse their critical properties- as full differentiability, smoothness, compactness, and adaptability to varying resolutions-wile also examining their strengths and limitations in addressing locality biases and capturing fine details. Our experimental comparison offers new insights into the trade-offs between different approaches, showcasing the capabilities and challenges of the latest INR techniques across various tasks. In addition to identifying arcas where cur- rent methods excel, we highlight key limitations and potential avenues for improvement, such as developing more expressive activation functions, enhancing positional encoding mecha- nisms, and improving scalability for complex, high-dimensional data. This survey serves as roadmap for researchers, offering practical guidance for future exploration in the field of INRs. We aim to foster new methodologies by outlining promising research directions for INRs and applications.
1 Introduction
Knowledge representation (Brachman, 2004) is a foundation in computational fields, playing a critical role in enabling systems to efficiently model, interpret, and manipulate information across varions domains. Deep neural networks have demonstrated a powerful capacity for learning robust knowledge representation from data, and have become the predominant tools for addressing complex tasks in areas like computer vision (LeCun al., 2015). The significance of effective knowledge representation extends beyond traditional methods, as it directly influences the performance and scalability of systems when handling diverse types of information such images, video, and audio, whether in 1D, 2D, 3D formats. Conventional approaches to encoding input signals typically rely on explicit discretisation, where the input space is segmented into distinct
"Equal contribution.

---

Published in Transactions on Machine Learning Research (02/2025)
(a) Activation Function - (xx)
(b) Position Encoding r(x)
a (x)
x
y
SIREN / WIRE / GAUSS / HOSC/ / SINC/ / FINER
FOURIER FEATURES
(c)
Activation Function + Position Encoding
(d) Network
a (x)
y
/F
In. Fil Fr.
x
:
In
Incode Block
Fil.
TRIDENT
INCODE
MFN FR
Fourier Basis
Figure 1:
The four categories of seate-of-thh-and (SOTA) implicit neural representation (INR) methods. The
yellow blocks highlight the specific components each method enhances. Specifically: (a) focuses improving activation functions, (b) enhances position encoding, (c) integrates both (a) and (b) to simultaneously improve activation functions and position encoding, and (d) advances the overall network structure.
elements, such as point clouds (Achlioptas et al., 2018; Fan et al., 2017), voxel grids (Gadelha et al., 2017: Liao et al., 2018; Stutz Geiger, 2018; Jimenez Rezende et al., 2016), and meshes (Kanazawa et al., 2018; Ranjan et al., 2018; Wang et al., 2018). While these methods can achieve adequate results, they present significant challenges when dealing with nigh-dimensional data (Mescheder et al., 2019). The computational cost of discretisation rises sharply with increasing dimensionality, making it inefficient, particularly for complex irregular spaces. Moreover, traditional discretisation methods tend to require substantial memory, posing limitations for large-scale applications. Recent works have shown efficient representation of complex data, for instance latent representation, as seen in autoencoders and variational autoencoders (VAEs) (Tischannen et al., 2018), which encode signals into low-dimensional latent spaces through the use of neural networks, and structured dictionary learning (Chen al., 2018; Tasissa al., 2023), which seeks approximate signals as sparse linear combinations of learned basis elements. Implicit Neural Representations (LNRs) offer a promising alternative by using continuous functions to represent data, addressing many of the limitations of explicit methods, such as memory inefficieccy and the high computational cost associated with discretisation.
The rise and continued development of INR have recently emerged as new way to learn representation efficiently. Implicit representations differ from explicit (or discrete) representations by encoding informa- tion as a continuous generator function, which maps input coordinates to corresponding values within the defined input space, rather than directly storing feature or signal values. Consequently, there has been significant interest in utilising these networks as implicit functions, with notable success (Mildenhall et al., 2021; Mescheder et al., 2019; Xie et al., 2022). Specifically, Multi-Layer Perceptrons (MLPs) are trained to parameterise signals by taking coordinates inputs, applying mapping technique that projects this input into a nigher-dimensional space, and predicting the associated data values at each coordinate. In this frame- work, the MLP functions as an Implicit Neural Representation encoding the signal's information within its weights. For example, when applied to image data, pixel coordinates are fed into the MLP, which gener- ates the corresponding RGB values, effectively learning continuous, high-resolution representation of the image. However, the classic use of the ReLU activation function often resulted in suboptimal performance across many applications. To address this issue, reparametrised learning techniques (Rahaman et. al., 2019)
2

---

Published in Transactions on Machine Learning Research (02/2025)
have been first used to adjust the weights and mitigate bias, further enhancing the network's performance. (Fathony et al., 2020) presented new architecture where the output of each layer is multiplied by a Gabor wavelet. Further researchers have introduced various activation functions. These include periodic sinu- soidal functions (Sitzmann et al., 2020), fimeueeryy localised Gabor wavelets (Saragadam et al., 2023), Gaussian functions (Ramasiinhe & Lucey, 2022), and the FINER network (Liu et al., 2024). Additionally, TRIDENT (Shen et al., 2023) is a network that integrates both positional encoding and carefully chosen activation function.
Implicit neural functions have been further adjusted in various tasks, including image generation (Reddy et al., 2021), super-rosolttio (Wu et al., 2021; Chen et al., 2021), 3D object reconstruction (Chabra et al., 2020; Mescheder et al., 2019; Mildenhall et al., 2021), and modelling of complex signals (Xu et al., 2022). Its application on modeling 3D scenes has been developed as Neural Radiance Field scene representations (NeRFs) (Mildenhall et al., 2021) allowing for photorealistic image generation from multiple viewpoints. The use of multi-layer perceptrons (MLPs) for image and shape parameterisation provides distinct advantages. First, MLPs are resoluiioninnlependent as they operate within a continuous domain, enabling them to generate values for coordinates beyond pixel- or roxel-based grids. Thus, it improves performance in vision tasks. Second, their memory requirements are not constrained by signal resolution, allowing more memory- efficient representations compared to traditional grid or voxel methods (Huang et al., 2021; Park et al., 2019). The memory demand scales according to the complexity of the signal rather than the resolution. Additionally, MLPs address the limitations of locality biases often found in convolutional neural networks (CNNs), which can hinder generalisation (Chen& & Zhang, 2019). Finally, MLP-based models are fully differentiable, offering adaptability across various applications (Zhang et al., 2024; Liu et al., 2020). Their weights can be optimised using gradient-based techniques, providing the flcxibility needed for diverse tasks (Zhang et al., 2023; Xie et al., 2022; Tancik et al., 2020; Cheng et al., 2023). Another body of work is that of (Shenouda et al., 2024). The authors presented insights that may complement the broader discussions in this survey, particularly in exploring the potential of ReL.Ubaaed architectures for INR tasks.
Implicit Neural Representations (INRs) are increasingly intersecting with broader trends in Artificial Intel- ligence (AI). For instance, the adoption of INR in transformer architectures enables more expressive and scalable mechanisms for capturing complex data relationships (Chen & Wang, 2022). Another promising direction involves diffusion models that leveraging INR as guide (Hui et al., 2024) expanding their poten- tial applications. Within this expanding landscape, INRs hold a unique position as compact, continuous representations of data, complementing other Al paradigms and offering a flexible tool for solving diverse problems in machine learning and beyond.
Despite the signíficant advancements in Implicit Neural Representations (INRs), there remains a notable gap in the literature that this work aims to address. The objective of this survey is to provide a comprehensive examination of diverse INR methods, offering both an in-depth analysis of their foundational principles and thorough exploration of their wide-ranging applications. While progress has been made, existing reviews, such as the one by (Molaci et al., 2023) that focuses on medical applications, do not include experimental comparisons across different approaches, nor do they cover the full spectrum of use cases. Moreover, these works primarily address task-specific applications, rather than delving into the underlying technical principles that define INRs. As many of the tasks are built on the same core INR techniques, there is a need for a broader comparison that highlights differences in performance across various methodologics. This survey aims to bridge that gap by delivering an extensive review of INR methodologies and applications, while also offering a performance comparison to reveal their strengths and limitations. By doing so, provide more complete understanding of the field and its potential for future advancements.
To the best of our knowledge, this is the first survey paper to comprehensively explore both the fundamental and advanced functions of INRs through practically experimental comparisons across various applications, where we considered the SOTA methods listed in Figure 2. Our work serves as a systematic guide and roadmap for researchers, offering new perspectives on the capabilities of INR. models. Additionally, we aim to inspire the broader research community to further investigate the potential of INRs across various domains. We believe this paper will help future exploration and innovation, encouraging deeper engagement with INR methodologies.
3

---

Published in Transactions on Machine Learning Research (02/2025)
Name
Category
Name
Category
SIREN
(a)
GAUSS Beyond Periodicily
(a)
Reference
Reference
CVPR 2020 (Sitamann et al., 2020)
ECCV 2020 (Ramasinghe & Lucey, 2022)
Spatial Compactness
Prequency Compactness
Number of Hyperperamnserra
Spatial Compactness
Frequency
Number
Implemennaatio
Adaptivity
Implementaation
Compactness
Adaptivity
Hyperparamettrs
X
X
X
Name
Category (a)
Name
Category
WIRE
FINER
(a)
Reference
Reference
CVPR 2023 (Saragadam et al., 2023)
CVPR 2024 (Liu al,, 2024)
Frequency
Spatial Compactnese
Frequency Compactness
Number
Implementation
Compactnes
Compartness
Adaptivity
Hyperparameterr
Implementstion
Adaptivity
Hyperparameter
V
Name
Category (b)
Name
Category
FOURIER FEATURES
MFN Multiplicative Filter Networks (d)
Reference
Reference
NeurIPS 2020 (Mescheder et al., 2019)
ICLR 2021 (Fathony et al., 2020)
Spatial Compactuoss
Frequency
Number of Hyperparametern
Spatial Compactuces
Frequecy
Number
Implementation
Compactness
Adaptivity
Implementation
Compactaess
Adaptivity
Hyperparameters
X
Name
Category
Name
Category
INCODE
(d)
FR
Fourier Reparameterized Training
(d)
Reference
Reference
WACV 2024 (Kazerouni al., 2024)
CVPR (Shi al., 2024)
Spatial Compactness
Frequency Compactuess
Number of Hyperparametera
Spatial Compactuess
Fregumey
Number Hyperparametero
Implementation
Adaptivity
Implementation
Compactness
Adaptivity
X
X
Name
Category
Name
Category
Name
Categury
HOsc
(a)
SINC
Sampling Theory Perspoctive
TRIDENT
(c)
Reference
Reference
Reference
Arxiv 2024 (Serrano al., 2024)
PMLR 2024 (Saratchandran al., 2024)
Arxiv 2024 (Shen et al., 2023)
Implementation
Hyperparamelorss
Iusplementation
Number Hyperpatamnctrss
Implementtation Not Public
Hyperparameetens
Not Publie
Not Public
Spatial Compactness
Proquency
Spatial Compactuens
Frequency Compactuesa
Spadial
Fregumeyy Compactness
Compactanos
A daptivitt
Adaptivity
Compactness
Adaptivity
X
X
X
X
Figure 2: A comprehensive comparison of the INR methods, each represented by method card. The cards outline key properties, including frequency compactness, spatial compactness, adaptability of the methods, and implementation details with the number of hyperparameters. The categorics mentioned correspond to those in Figure 1.
Contributions. This survey introduces clear tazonomy of existing state-of-the-art (SOTA) INR tech- miques, organising them into four key categories that represent critical advancements in the field (see Fig- ure 1). First, methods in the activation function category (a) enhance the expressiveness and adaptability of INRs by improving activation functions, resulting more flexible and capable representations. Notable examples include SIREN (Sitzmann al., 2020), WIRE (Saragadam et al., 2023), GAUSS (Ramasinghe & Lucey, 2022), Hosc (Serrano et al., 2024), SINC (Saratchandran al., 2024), and FINER (Liu al., 2024), each offering distinct benefits in signal modelling through specialised activations. Second, the position encoding category (b), represented by techniques such as FOURIER FEATURES (Mescheder et al,, 2019), focuses on refining how positional information encoded into the model, enhancing the ability to capture fine-grained details in complex signals. Third, methods that combine activation functions and position encoding (c), like TRIDENT (Shen al., 2023), address both aspects simultancously, providing more robust and flexible approach representation lcarning. Finally, the network structure category (d), featur- ing techniques such INCODE (Kazerouni et al., 2024), MEN (Fathony et al., 2020), and FR (Shi et al.,
4

---

Published in Transactions on Machine Learning Research (02/2025)
SIREN
GAUSS
WIRE
HOSC
SINC
FINER
WIVV
Figure 3: Visualisation of the activation functions used in the method, categorised as (a) in Figure 1.
2024), focuses on optimising the overall network architecture, incorporating additional components like in- coding blocks and filters to enhance learning and generalisation. These four categories collectively define the landscape of current INR research. Our work not only introduces this taxonomy but also further explores these foundational approaches through experimental analysis. Moreover, our survey offers valuable insights into why current methods are effective and highlights the key factors influencing performance trade-offs. In contrast to existing studies, we provide comprehensive performance comparison across various inverse problem tasks.
2 Where Do We Stand with Implicit Neural Representations?
What Are Implicit Neural Representations? Implicit neural representations consider the problem of finding a continuous representation of data. Given an imput z, we are interested in learning a function that maps x a quantity of interest axx), while verifying an implicit equation depending on a(x): : Rour and possibly its derivatives:
v.a()),..)
Our function of interest could be simply be the image, mapping the coordinates to their pixel values. To do that, we parametrise a(x) as an MLP network with N layers, where the output of each layer is given by:
y: (Wig)
(2)
where a is the activation function, W; E R are the weights, and b, E R. the biases of the i-th layer. 2o is the input coordinate (i.c., pixel coordinate for an image), and the final output reads: yas WNYN-1 by-
ReLU has been extensively utilised for function approximations in MLPs (Hanin & Rolnick, 2019). While they have demonstrated excellent performance across various applications, they often struggle to faithfully represent complex signals and capture fine details. This limitation ariscs because these networks tend to prioritise learning fow-frequency components. Consequently, numerous approaches have been proposed in the literature to mitigate this bias.
General Propertics of INRs. Implicit Neural Representations (INRs) have several defining mathe- matical properties that distinguish them from traditional data representations. One key advantage is their continuous nature, where data is encoded not as discrete samples but as continuous functions. This allows INRs to represent high-resolution and fine-grained details without the need for large, memory-intensive stor- age, making them particularly efficient for high-dimensional data. The ability to interpolate between points seamlessly, without. being coistr by a predefined resolution, is a core feature of this framework. The following list outlines some of the mathematical general properties that underpin INRs.
Continuity: INRs model data as continuus functions. For an input æ R the output a(x) o is given by continuous function f ti > 1a where: f(x) is continuous over the domain of a.
Differentiability INRs are fully differentiable, allowing optimisation via gradient descent. Given the func- tion f(x) representing the data, the derivatives of f with respect the input exist and are continuous: v (x) and v (x) exist and are continuous.
5

---

Published in Transactions on Machine Learning Research (02/2025)
Resolution Independence: The output of INRs is not tied to the resolution of the data. For any imput coordinate v, INRs can generate corresponding values without a predefined grid structure, Lee, they operate on a continuous domain: f(z) can produce outputs for any I E IR" independent of any fixed resolution.
The representation of data through INRs is compact in memory usage, as the function f()) is encoded within the weights of neural network, rather than as large array of discrete samples. If the model uses N parameters to define f(z), memory usage scales with N instead of the data resolution: Memory usage scales as O(N), where N the number of parameters the network.
Compactness:
Frequency Adaptivity: Through techniques such as positional encoding or adaptive activation functions, INRs can capture a wide range of frequencies in the data. For input z, INR models can handle both low-frequency and highffreuuecyy components: f(z) can represent both low-frequency and high-frequeng signals, depending on model design.
Smoothness of Representations: INRs tend produce smooth representations by virtue of their construc- tion. The activation functions, such sinusoidal or wavelet-based functions, ensure that the output is smooth and continuous across the imput domain: f(x) is typically smooth over its input domain.
Scalability: INRs scale well to complex, high-dimensional tasks. The memory and computational require- ments grow with the complexity of the function f(x); not with the dimensionality of the output space: Computational complexity scales with the complexity of f()). not directly with the data dimensionality.
2.1 Taxonomy INR: (a) Activation Function
To begin, present first category in our taxonomy and review the ex- isting literature. One of the critical challenges in Implicit Neural Representa- tions (INRs) is overcoming the bias towards learning low-frequency components, commonly observed in neural networks that use traditional activation functions such as ReLU (Hanin Rolnick, 2019). The ReLU function, while effective in many settings, has a discontinuous nature that limits ability to capture higher-frequency signals, making suboptimal for tasks that require detailed high-resolution data representation. However, recent work Shenouda et al. (2024) has demonstrated that by imposing specific constraints inspired by B- spline wavelets, ReLU-based networks can overcome this limitation, enabling them to approximate their computational efficiency. Furthermore, alternative activation functions such as sigmoid or tanh, while continuous, are not expressive enough to ade- quately capture the full range of frequency components necessary for complex tasks.
Activation Function x(x)
(x)
components effectively while maintaining
To address these limitations, several novel activation functions have been pro- posed in the literature, specifically designed to overcome the low-frequency bias.
y
These functions are engineered to better capture wider range of frequencies, enabling more detailed and accurate representations in INR models. We next explore some of the key activation functions that have been introduced, high- lighting their contributions to advancing the performance of INRs.
SIREN WIRE / GAUSS! HOSC/ / SINC FINER
SIREN (Sitzmann et al., 2020) The authors in (Sitzmann et al., 2020) proposed SIREN using the sine function as an activation function:
(x) = (inn
where is scaling hyperparameter chosen tailored for cach task to modulate the low-frequency bias. This activation function stands out due to its periodic nature, enabling it to model broad spectrum of frequencies effectively. The periodic properties of SIREN allow the network to represent high- figh-frequency variations with case, making it particularly suitable for tasks requiring intricate details, such as image and audio reconstruction, and solving complex differential equations. One key benefit of the sine- based activation function is its hnest, which not only facilitates stable gradient flow during training but also allows for

---

Published in Transactions on Machine Learning Research (02/2025)
explicit calculation of derivatives. This capability makes SIREN highly effective for applications involving inverse problems, where precise derivative computation is critical. Additionally, the smooth, continuous representation that SIREN offers enables it to generalise well across varying resolutions and achieve high- quality, resolution-indee innrriddenndeeeeenntt outputs. The authors proposed a specialised weight initialisation scheme to maintain the distribution of activations throughout the network layers, which ensures efficient. convergence and avoids issues like vanishing or exploding gradients. This initialisation technique further enhances SIREN's capacity to capture fine-grained details while remaining memory efficient.
However, despite these advantages, the SIREN activation function also has certain limitations. Due to its reliance on a fixed periodic function, it can struggle to faithfully represent complex high-froquency details, which may result in artifacts or reduced accuracy in applications with highly detailed or non-smooth features. Furthermore, SIREN's reliance on a single frequency scaling parameter can limit its flexibility in adapting to diverse signal characteristics, making it less versatile for some complex tasks.
GAUSS FUNCTION (Ramasinghe & Lucey, 2022) When exploring beyond periodicity, the network employs the Gaussian function the activation function, given by:
(x) exp((-sx)²),
(4)
where s controls the width of the frequency. The Gaussian activation function offers several benefits, particu- larly its smoothness, which makes it well-suited for applications requiring continuous and localised represen- lations. Because it is localised in both the spatial and frequency domains, it is advantageous in lasks where smooth transitions and localised details are important, such in denoising or the reconstruction of smooth objects. However, notable drawback is its lack of periodicity, which limits its effectiveness in represent- ing high-frequency signals. This makes the Gaussian activation function less suitable for tasks that involve intricate or oscillatory data, such as audio reconstruction or high-detail image synthesis, where capturing fine details at higher frequencies is crucial. The balance between its smoothness and the inability to handle figh-freynency information defines its strengths and weaknesses, depending on the application at hand.
WIRE (Saragadam et al., 2023) The authors proposed using the Gabor wavelet as an activation function, defined as:
a (x) =xxp(-(sx)² iwer)
where the imaginary part is represented by the i-term. The Gabor wavelet is effective in minimising the product of its standard deviations in both the time and frequency domains, allowing it to represent features across both space and frequency. This dual localisation enables the function to capture fine-grained details and frequency components more effectively. As result, the Gabor wavelet combines the advantages of both the SIREN network, which excels in higg-fryyyency ropresentations, and the Gaussian network, known for its smooth, spatial localisation.
While the Gabor wavelet offers strong representational capabilities, a potential drawback is its relatively higher computational cost compared to simpler activation functions. The neod to process both spatial and frequency components simultaneously can lead to increased complexity in network training and inference, making it less efficient in scenarios where computational resources or speed are critical. In addition, the Gabor wavelet's ability to capture fine-grained details across both space and frequency, while advantageous, may also increase the risk of overfitting, particularly when dealing with noisy or sparse data. The model may become overly sensitive to minor variations in the input, making it less generalisable to unseen data. While most of the INR activation functions only require one hyperparameter, WIRE requires two hyperparameters (s and w), which in lurn results in more computational effort to choose the appropriate hyperparameters.
Hosc (Serrano et al., 2024) Similar to the SIREN network, the authors in (Serrano et al., 2024) proposed the Hosc function, defined as:
(x) = (nnn
(6)
where 3 is a sharpness factor that controls the steepness and behavior of the hyperbolic function.
The Hosc function inherits several properties from SIREN, particularly in terms of computational efficiency and low memory usage. Additionally, higher values of 33 enable the capture of fine details, especially due to
7

---

Published in Transactions on Machine Learning Research (02/2025)
abrupt amplitude changes at z na, while lower values of 3 emphasize lower-frequency components, similar to SIREN's s behavior. The authors further proposed network architecture where / increases across layers, enabling the model to effectively represent both high and low-frequency signals. They also introduced Ada- Hosc, an alternative version where 3 is treated as a learnable parameter, which further leverages the simple derivative form of the Hosc function for improved adaptability. Whilst Hosc function offers flexibility, it has several potential disadvantages. The sharpness factor 3 must be carcfully tuned to balance the capture of high- and low-frequency components, and improper tuning could lead to unstable training or convergence issues. Additionally, as & increases across layers, the computational complexity and training time may also increase, potentially slowing down the overall process.
SINC FUNCTION (Saratchandran et al., 2024) Motivated by the classic Shannon sampling theorem and drawing similarities between INR and sampling, the authors in (Saratchandran et al., 2024) proposed the sinus cardinal as an activation function, which form given by:
(x)) sinc(wx))
The SINC is localised in the space domain, but not in the frequency domain. It also forms generating system, meaning the set of its translates {x - sine(x- - k)}k Z} can be used for sampling, enabling the network to approximate signals and represent high-order features effectively. The SINc activation function, motivated by sampling theory, offers several distinct advantages. One of its primary strengths is its localisation in the space domain, which makes it particularly effective at representing spatial information. Additionally, the SINC function can generate set of translates that approximate signals, making it well-suited for tasks that involve high-order feature representation. Its connection to sampling theory provides theoretical foundation for reconstructing signals, which enables accurate approximation of continuous data.
However, a key limitation of the SINC is its lack of localisation in the frequency domain. This means it might struggle to represent or capture high-frequency variations as effectively as other functions designed with periodicity or high-frequency adaptivity. As a result, while the SING excels in certain applications where spatial representation is paramount, it may not be the best choice for tasks that require precise handling of complex, high-frequency features.
This trade-off between space localisation and frequency adaplability highlights ils strengths in certain con- texts, while also pointing to its limitations in handling tasks involving detailed frequency components.
FINER (Liu et al., 2024) The SIREN model struggles to represent a broad spectrum of frequencies due to its reliance on a fixed scaling value. This limitation restricts the range of reconstructed frequencies, which may be insufficient, particularly when the frequency distribution of the signal is unknown or varies significantly. To overcome this issuc, the authors of the work (Liu et al., 2024) proposed an activation function given by:
(x) sin(w[[x] + 1)x))
(8)
This activation function is both smooth and periodic, inheriting all the properties of the sine function. In addition, the scaling parameter w varies dynamically across the network based on the bias of different nodes. Higher bias values correspond to rapidly varying functions that can capture high frequencies, while lower bias values correspond to slower variations, effectively representing lower frequencies. This dynamic scaling provides the FINER approach with greater flexibility to represent broader range of frequencies. Specifically, nodes with higher bias can capture high-frequency details, while nodes with lower bias are better suited for representing lower-frequency information. This adaptability offers FINER a major advantage over fixed- scaling models like SIREN, as it is able to represent a much broader range of frequencies, particularly when the frequency distribution is variable or unknown. Accordingly, the authors proposed to initialise the bias coefficients an uniform distribution:
b~ U)-
(9)
Choosing a sufficiently large value of l ensures that the set of frequencies captured by the model is not constrained by the initialisation.
However, there are also potential disadvantages to consider. The reliance on dynamic bias scaling, while powerful, can introduce complexity in tuning and initialisation. The effectiveness of FINER depends on
8

---

Published in Transactions on Machine Learning Research (02/2025)
choosing appropriate range for the bias coefficients, which may require careful experimentation. Moreover, if the initial bias range is not selected carefully, the model might not be able to capture the full spectrum of frequencies required for certain tasks, potentially limiting its generalisation abilities in some cases.
2.2 Taxonomy INR: (b) Positional Encoding
FOURIER FEATURE (Tancik ct al., 2020) Another approach to learn high-frequency features in- volves mapping the input coordinates to higher-dimensionall space using a Fourier feature mapping 7. This enables the network to effectively represent these features, addressing limitations in capturing fine details. Several encoders have been proposed in the literature (Tancik et al., 2020), including:
This simple encoding projects the input into two oscillatory components, providing a basic yet effec- tive way to introduce periodicity and high-freyeency information into the network.
Basic: 7(x) =
Position Encoding r(x)
x
y()
Positional Encodings: 7(x) = [x....,, con....,cs(((2))) (2mw))/m x), sin)ssinnnnuuue /2m =))""
where w represents the frequency hyperparameter and m the embed- ding size. This type of encoding enhances the network's capacity to caplure both fine and coarse features, making highly effective for tasks involving spatial data including computer vision applications.
Random Fourier Features: 7(x) [cos(2mBB), sin sin(2xBx)) with B being random Gaussian matrix sampled from N w0, In this approach, the transformation matrix B is sampled from a Gaussian distribution (0, Random Fourier features offer a stochastic way to approximate complex functions, and they are particularly useful in applications that benefit from randomness, such as kernel approxima- tion uncertainty modelling.
y
After applying the Fourier feature mapping to the input, the resultant features are fod into a Multi-Layer Perceptron (MLP) using RcLU activation functions.
FOURIER FEATURES
2.3 Taxonomy INR: (c) Activation Function + Position Encoding
Activation Function (x)
+
Position Encoding (x)
TRIDENT (Shen et al., 2023) Trident is a mix between the two approaches of
using an activation function, and applying a Fourier mapping. The first layer of the net work expressed as:
((x)
y1 = (1)(x) + bi
(10)
where is the Fourier features mapping, which is formulated (2)
=
cos(2 x), sin(2no =])""
The subsequent layers output are computed as:
D
(11)
where alx) = exp(-s=²) is the activation function. This activation function serves three essential purposes. Firstly, the network can effectively represent high-order features. This property ensured by expressing the exponential function as an infinite series of cosine functions:
y
inf
exp(-cos(z)3)-
Σ 1=1
A, cos((2r)
(12)
TRIDENT
9

---

Published in Transactions on Machine Learning Research (02/2025)
Secondly, the activation function also enables Trident to represent broad spectrum of frequencies. This capability is demonstrated by expressing the cosine function as series of shifted and scaled components:
(
2
Σ n2 n1) k=0
cos(x)"=
cos((n- - -2k)=))
(13)
2 2
2 (k)
This formulation ensures that the activation function can span a wide frequency range, enhancing the model's capacity to learn complex patterns. Finally, TRIDENT ensures compact spatial localisation by selecting the coefficients An according to a Gaussian window. This configuration helps the network focus on specific regions in the input space, improving its ability to capture fine-grained spatial features.
2.4 Taxonomy INR: (d) Network
INCODE (Kazerouni et al., 2024) Another approach that enhances the SIREN architecture involves using an activation function of the form:
(x) = aasnnnnnuus (bux c) + d,
(14)
where a controls the amplitude, determining the overall strength of the activa- tion function; b manages frequency scaling, influencing the range of frequencies the network can capture, introduces a phase shift, shifting the waveform hor- izontally; and d sets the vertical shift, adjusting the baseline or "brightness" of the signal. In this formulation, w serves as a fixed hyperparameter that de- fines the base frequency of the sinusoid, while b acts as a learnable parameter, allowing the network to lune the frequency dynamically.
Network
In. Fil. Fr.
A key highlight of the INCODE approach is that these parameters a, 6, c, and d-are not fixed but are predicted dynamically by separate module known as the harmoniser network. This enables the model to adjust the characteristics of the activation function in real time, resulting in more adaptive and expressive representations. With this flexibility, the network is better equipped to capture a wider range of patterns, both fee-ffreuencyy and low-fre wwwfrequency, making it suitable for tasks that involve complex and multi-scale data.
In. Fil Fr.
In. Fil. Fr.
The intuition behind this design lies in the ability to modulate the signal dy- namically, mimicking the behavior of real-world signals that exhibit varying frequencies and amplitudes. By learning to adjust these parameters during training, the network can efficiently model data with varying characteristics, such as audio signals with fluctuating pitch or images with diverse levels of detail. This approach offers several advantages. The dynamic modulation of froquency and amplitude provides the network with enhanced expressiveness, allowing it to generalise better across tasks that involve varying input pat- terns. Moreover, the flexibility in phase and baseline shifts ensures that the network can align and normalize data effectively, further improving its ability to learn complex relation- ships. However, this increased fiexibility also introduces additional complexity. The harmoniser network adds computational overhead, and the need to carefully tune multiple parameters may make training more challenging. Additionally, while the dynamic nature of the activation function enhances expressiveness, it may increase the risk of overfitting, especially when applied to small or noisy dalasets.
In. Incode Block
Fil. Filter
Fr.
Fourier Basis
INCODE / MFN / FR.
MFN Multiplicative Filter Networks (Fathony et al., 2020) The Multiplicative Filter Networks (MFN) architecture offers an alternative to traditional Multi-Layer Perceptrons (MLPs) by replacing them with a sequence of Hadamard products and nonlinearities. In this approach, the output at each layer is computed by applying a nonlinear activation function to the input, followed by a Hadamard (element-wise, product between the transformed input and the weight matrix. This design introduces a new way to model complex interactions within the data, leveraging multiplicative relationships rather than the additive ones typically found in ML.P architectures.
10

---

Published in Transactions on Machine Learning Research (02/2025)
INR
Baseline Complexity
Extra Complexity
The sinusoidal operation is computed in constant time, with additional complexity beyond O(1).
SIREN
The Gaussian function involves squaring the input, multiplying, and applying the exponential function, all of which are O(1).
GAUSS
Combines Gaussian 0(1) operations with cosine O(1),
WIRE
0(1)
resulting in no additional complexity beyond O(1).
Combines sine O(1) with hyperbolic tangent O(T).
Hosc
The sharpness factor 8 introduces additional complexity.
Involves sine and division operations.
SINC
The division introduces negligible overhead, remaining O(1).
The sinusoidal operation with dynamic scaling based on |ar] introduces no additional asymptotic complexity, as absolute walue and addition are 0(1).
FINER
Table 1: Computational Complexity of Activation Functions in INR Methods.
The MEN framework supports two specific types of activation functions. The first is the sinus nonlinearity, defined as:
ox;; (' sin(w"x o'),
(15)
where in the ith layer, the parameter 6e includes the frequency wf and phase shift oi allow the model to capture oscillatory patterns effectively. The second is the Gabor wavelet, given by:
of = -xp(-"""'= sin(w'n (wwwwwx+)) o'),
(16)
which introduces both spatial and frequency localisation with an additional parameter, the scale term yf. The combination of these functions enables MEN to model a wide range of data patterns by approximating signals as a mixture of sinusoidal and wavelet components. One of the key advantages of MFN is their ability to represent complex, high-order details that are often challenging for standard architectures like SIREN or Fourier Feature Networks (FOURIER FEATURES). By using multiplicative interactions and flexible nonlincarities, MFN can capture intricate dependencies within the input data. This makes them particu- larly effective in applications requiring fine-grained representation and high-frequency details, such as audio synthesis, image reconstruction, and other tasks involving implicit neural representations.
However, the MPN architecture also comes with certain challenges. The use of Hadamard products and complex activations can increase the computational cost compared to simpler activation functions like SIREN. Training MEN can require more careful hyperparameler tuning, the interplay between frequency, phase, and spatial parameters can be sensitive to initialisation. Moreover, while the multiplicative design enhances the model's expressiveness, it may also increase the risk of overfitting.
Fourier Reparameterised Training (Shi et al., 2024) Another approach to bypass the low- frequency bias is to use an appropriate weight reparametrisation. The idea is to reparametrise each row of the weight matrix of the i-th layer W; as a sum of Fourier bases. The output of cach layer is given by:
V: = (A, b,)
(17)
Where A; is learnable weight matrix and B; E M.: is the Fourier basis matrix given by
B = coscooss (w Φx)
(18)
for i= I,..,, M and j I,.., Nout- We denote B.dd the entries of B, (z) which is the sampling position soquence, and M as the number of Fourier basis considered.
The authors proposed to choose P phases as an array of phase shifts, which are evenly distributed over the interval from [0, 2T.. and to take 2F frequencies, consisting of a low-frequence basis { 2 2,.., 1) and a high frequency basis (1,2,..., F), which gives total of M 2FP bases. F and are hyper parameters and should be chosen for each task independently.
As for the sampling sequence, the number of sampling points is the number of features, while the points were
11

---

Published in Transactions on Machine Learning Research (02/2025)
chosen to be sampled uniformly from the interval [-*F, FFF. FFFF Finally, it was proposed to draw the weights of the learnable matrix A₃ following the distribution:
Altl u
(1V
6
6
(19)
a
where a= = M E/
While Fourier Reparameterised Training (FR) offers powerful approach to mitigating low-frequency bias, it also presents cortain challenges and disadvantages. A key limitation lies in the increased complexity of the model. Reparameterising the weight matrices using Fourier bases introduces additional hyperparamelers, such as the number of frequency components F, the number of phase shifts P, and the sampling sequences. These hyperparameters must be carefully selected for each specific task, which can require extensive experi- mentation and tuning, increasing the burden on practitioners.
Another disadvantage involves the computational overhead. Since each layer's weight matrix reparame- terised as a sum of Fourier bases, the model becomes more computationally intensive compared to simpler architectures. This increased complexity can lead to longer training times and higher memory usage, which may limit the applicability of FR models in resource-oonstraneed environments real-time applications.
2.5 On the Complexity of INRs
The Table 1 provides overview of the computational complexity associated with various activation fune- tions used in implicit neural representations (INRs). All the functions listed share a baseline computational complexity of O(),, which means that evaluating these functions for a single input requires constant time. This characteristic is crucial for INR methods, as they often deal with high-dimensional data where com- putational efficiency is paramount. Each activation function in the table demonstrates O(1) complexity, but they vary in the operations they involve and their ability to handle specific tasks. For instance, the SIREN function relies solely on a sine operation, making straightforward and efficient with no additional computational overhead. This simplicity makes it well-suited for representing periodic and oscillatory pat- terns in data. It can be proven from the results that it is the second fastest, 482 seconds, among all tested methods for the dosing task reported in Table 3. The Gaussian (GAUSS) activation function, while also O(1), requires squaring the input, multiplying it by a scalar, and applying an exponential operation. These operations remain computationally lightweight, with empirically reported 709 seconds for denoising task in Table 3, yet they provide the function with the ability to represent smooth and spatially localised features, which is especially useful for tasks involving smooth transitions. WIRE takes this further by combining the Gaussian operation with cosine function. Although this combination adds slight computational overhead compared to standalone functions like SIREN or GAUSS (as verified in Table 3 for the denoising task, 1485 seconds, which is double the computational time of GAUSS and triple that of SIREN), it still retains the O(1) complexity. The dual spatial and frequency localisation capabilities provided by WIRE justify this marginal increase in computational effort, making it versatile for representing more complex patterns. Hosc builds on a sine function by applying a hyperbolic tangent transformation. While this adds an extra step, both the sine and hyperbolic tangent functions are computed in constant time, maintaining 0(1) complexity. The flexibility introduced by the sharpness factor 3 allows HOSC to adapt between high- and low-frequency signal representations without compromising computational efficiency. The SINC, inspired by the sinus cardinal, involves both sine computation and a division operation. Although the division introduces a minor com- putational overhead, it is negligible and does not affect the overall O(1) complexity. SINC's strong spatial localisation capabilities make it particularly valuable for signal reconstruction tasks, though its frequency adaptability may be more limited than other functions.
3 Experimental Results
In this section, we present a comprchensive comparison of the statc-of-tho-art implicit neural represcntation (INR) methods that have publicly available implementations. These methods were evaluated across a diverse set of tasks, ranging from 1D audio reconstruction to various image reconstruction tasks, including denoising,
12

---

Published in Transactions on Machine Learning Research (02/2025)
super-resolution and CT reconstruction, well as a 3D column occupancy task. This broad evaluation aims to highlight the strengths and limitations of each method in different application domains, providing holistic view of their capabilities and performance across multiple dimensions.
For a fair comparison hyperparameters, per tasks, recommended in the respective papers. All models were trained and tested on RTX 4070 GPU with 8 GB of RAM.
implemented INR. methods, we implemented each approach using the default
3.1 1D Applications: Audio Reconstruction
The task of audio reconstruction involves approximating the function R: RR R, which represents the audio signal. We aim to represent 6 seconds musical piece by Bach. For all approaches, we use a 5-layer neural network and apply a scaling factor of 30 to the first layer to better capture the high-frequency components typically present audio signals. The experiments are run for 2000 iterations.
Ground Truth
INCODE
FR
Loss: 0.00000736
Loss: 0.00000415
FINER
WIRE
FOURIER FEATURES
Loss: O. 0.0000121 GAUSS
Loss: 0.0986
Loss: 0.0141 SIREN
MFN
Loss: 0.0183
Loss: 0.0241
Loss: 0.0000394
Figure 4: The visualisation with La loss metric comparison on audio reconstruction task across the implicit neural representation methods.
Figure 4 indicates the overall performance among all the methods of audio reconstruction. This comparison highlights the performance of various implicit neural representation (INR) methods in reconstructing an audio signal, with FR (Fourier Reparameterised Training) and INCODE showing the best results. FR achieves the lowest loss due to its effective use of Fourier reparameterisation. which captures both low- and sigh-frequency components, overcoming the low-frequency bias typical of many models. INCODE also performs exceptionally well, thanks to its dynamic scaling of frequencies, which allows to adapt to the complexities of the signal. SIREN and FINER follow closely, leveraging periodic activation functions and frequency scaling to handle oscillatory patterns effectively.
In contrast, methods like WIRE and MFN struggle to capture the intricacies of the waveform, resulting in noisier oulputs. WIRE'S Gabor wavelet, while useful for localised frequency represenlations fails to generalise across the entire frequency spectrum of the audio, while MEN's multiplicative structure is more challenging tune for this task. FOURIER FEATURES and GAUSS offer moderate performance, with the former limited by fixed feature sets and the latter by the absence of periodicity, making them less capable of capturing high-frequency details.
13

---

Published in Transactions on Machine Learning Research (02/2025)
Table 2: Comparison of the implicit neural representation methods on PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Mcasure) for CT image roconstruction across varying numbers of projections (20, 50, 100, 150, 200, and 300).
20 projections
50 projections
100 projections
Method
PSNRt
SSIM
PSNR1
SSIM
PSNR1
SSIM
INCODE (Kazerouni et al., 2024)
24.38
0.651
27.33
0.762
31.48
0.890
FR (Shi et al., 2024)
25.58
0.738
29.14
0.852
31.18
0.917
FINER (Liu et al., 2024)
25.53
0.731
28.20
0.841
30.92
0.888
WIRE (Saragadam et ul., 2023)
20.73
0.417
25.01
0.651
28.83
0.826
FOURIER FEATURES (Tancik et al., 2020)
25.64
0.780
26.44
0.803
26.74
0.802
GAUSS (Ramasinghe Lucey, 2022)
22.21
0.548
26.44
0.752
27.80
0.764
MFN (Fathony et al, 2020)
21.10
0.402
22.70
0.487
25.30
0,643
SIREN (Sitzmann al., 2020)
20.85
0.421
24.42
0.607
29.61
0.842
150 projections
200 projections
300 projections
Method
PSNR
SSIM
PSNR
SSIM
PSNR1
SSIM
INCODE (Kazerouni et al., 2024)
33.94
0.939
34.29
0.946
34.76
0.953
FR (Shi et al., 2024)
30.25
0.907
30.49
0.910
30.40
0.910
FINER (Liu et al., 2021)
31.84
0.914
32.13
0.922
32.24
0.927
WIRE (Saragadam et al., 2023)
30.54
0.891
31.88
0.916
30.95
0.902
FOURIER FEATURES (Tancik
al., 2020)
26.98
0.804
26.94
0.803
26.94
0.794
GAUSS (Ramasinghe & Lucey, 2022)
27.85
0.766
27.89
0.773
27.90
0.848
MEN (Fathony et al., 2020)
28.29
0.793
30.50
0.868
33.63
0.935
SIREN (Sitzmann et al., 2020)
31.58
0.907
32.02
0.918
32.71
0.928
3.2
2D Inverse Problems: Image Reconstruction
We conducted series of experiments various inverse problems in 2D, including CT reconstruction, image denoising, and single-image super-resoluton, to evaluate the performance and robustness of the proposed methods.
3.2.1 CT Reconstruction
CT reconstruction is a process used in medical imaging create detailed cross-sectional images of the body from multiple X-ray projections taken at different angles. Implicit neural representations (INR) can reconstruct an image using given number of measurements by minimising the following loss function:
L= = |sinogrammouttuut) sinogram(truth)]
(20)
where the sinogram represents the transformation of the image into a set of projections. In all methods considered, we focus on the problem of recovering an image from varying numbers of measurements. Specif- ically, we conduct experiments with 20, 50, 100, 150, 200 and 300 measurements. We employ a two-layer neural network with 300 hidden features and train it for 5000 iterations.
Both Figure and Table indicate that the reconstruction quality improves for all methods as the number of projections increases, which aligns with the general expectation that more projections provide richer informa- tion, leading to more accurate and detailed reconstructions. Observing from Table 2, INCODE demonstrates superior performance, achieving the highest PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Sim- ilarity Index Mcasure) values in all scenarios, particularly excelling at higher projection counts with scores of 34.76 dB for PSNR and 0.953 for SSIM at 300 projections. At lower projection counts, FR stands out, delivering the best results with a PSNR of 29.14 and SSIM of 0.852 at 50 projections, and further improving 31.18 PSNR and 0.917 SSIM at 100 projections, making highly effective with limited data. FINER per- forms competitively as the number of projections increases, while WIRE and MEN consistently lag behind, exhibiting the lowest. scores across all scenarios. Moderate performers like FOURIER FRATURES and GAUSS
14

---

Published in Transactions on Machine Learning Research (02/2025)
20 Projections 50 Projections
100 Projections
200 Projections
300 Projections
INCODE
FR
FINER
WIRB
FOURIER
FEATURES
GAUSS
MFN
SIREN
Figure The visual comparison of CT reconstruction results using the implicit neural representation methods across 20, 50, 100, 200, and 300 projections.
15

---

Published in Transactions on Machine Learning Research (02/2025)
show adequate results in low to medium projection settings but are outperformed by INCODE and FR at higher counts. These findings underscore the strengths and limitations of each method, highlighting INCOdE as the most robust approach for high-quality CT reconstruction, especially when more projection data is available.
In Figure 5, INCODE and FR. consistently produce the clearest reconstructions across all projection settings, with INCODE showing superior noise suppression and finer structural details, even at lower projections such as 20 and 50. FINER, while performing well at higher projection counts, exhibits noticeable artifacts and lower fidelity at lower projection numbers, particularly at 20 projections. WIRE and GAUSS struggle significantly in low-projection scenarios, with severe blurring and structural distortions that persist even at 100 projections, highlighting their limited ability to capture fine details. FOURIER FEATURES achieves moderate performance, performing better than WIRE and GAUSS but failing to reach the level of clarity achieved by INCODE and FR. At high projections (200 and above), all methods show considerable improvement, but differences remain evident; for instance, MFN and SIREN display sharper boundarics and fewer artifacts at 300 projections, yet they still lag behind INCODE in overall image quality. This visual comparison underscores the robustness of INCODE and FR across different projection settings, while revealing the limitations of other methods, particularly in scenarios with sparse data.
When considering the properties across all methods, INCODE stands out due to its balanced combination of spatial and frequency compactness, which is reflected in its superior performance across all CT reconstruc- tion tasks, particularly at higher projection counts. Methods such as FR and FOURIER FEATURES excel in CT reconstruction due to their strength in frequency-basee bbssed representations, allowing them to effectively capture fine details and structural information, especially when the number of projections is moderate to high. On the other hand, methods like SIREN and GAUSS, which lack adaptivity and frequency compactness, struggle to achieve comparable performance, particularly at lower projection counts where these properties are crucial for maintaining image quality. WIRE and FINER, despite having adaptivity and more flexibility duc to their higher number of hyperparameters, show limitations in handling high-frequency details, Icading to suboptimal reconstructios in sparse projection settings. Thus, the results indicate that while adaplivity and spatial compactness are beneficial, the inclusion of frequency compactness and a balanced set of hyper- parameters are critical factors for achieving high-quality CT reconstructions, as exemplified by the robust performance of INCODE, FR, and FOURIER FEATURES across different projection scenarios.
3.2.2 Image Denoising
Image denoising involves recovering the original, noise-free image from a noisy observation. We evaluate the different approaches by introducing two types of noise: Poisson noise, which typically arises in low-light conditions and photon-limited imaging, and Gaussian noise, which is common electronic sensor noise and general environmental interference. We employ a two-layer neural network with 256 hidden features and train it for 2000 iterations.
We report the results in Figure 6 and Table 3.
The results of the denoising task highlight the strong performance of INCODE, FR, and FINER, which achieve the best balance between denoising quality and fine detail preservation. INCODE has the highest PSNR (29.63) and provides a visually sharp reconstruction, closely matching the ground truth. However, it comes at the cost of significant computational time (2370 seconds), making it less practical for real-time applications. In contrast, FR and FINER offer similarly high-quality results with PSNR values of 29.47 and 29.05, respectively, but with much lower computation times (752 and 643 seconds), making them more efficient for practical use.
SIREN also performs well, with a reasonable trade-off between accuracy (PSNR 28.60) and computational speed (482 seconds). On the other hand, FOURIER FEATURES, while the fastest (403 seconds), sacrifices reconstruction quality, evident in its lower PSNR (27.16) and noisier output. Methods like WIRE, GAUSS, and MFN fall behind in both visual quality and PSNR, with their outputs appearing either over-smoothed retaining significant noise, making them less competitive for high-quality denoising tasks.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Ground Truth
Noisy Image
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 6: The visual comparison across the 8 implicit neural representation methods the denoising task.
Table 3: The metric comparison of the PSNR performance, and the time that the 8 implicit neural repre- sentation methods used for the same number iterations.
METHOD
INCODE
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
PSNR f
29.63
29.47
29.05
28.74
27.16
28.09
28.22
28.
Time(s) $
2370
752
643
1485
403
709
1280
482
That is, FR and FINER emerge as the most balanced methods, providing high-quality reconstructions with manageable computation times, while INCODE is ideal when quality is prioritised over speed. FOURIER FEATURES, though fast, is less suitable for tasks requiring fine detail preservation.
3.2.3 Single Image Super Resolution
In the super resolution task, aimed to enhance the quality of images by reconstructing high-resolution visuals from low-resolution inputs. We conducted experiments at four distinct resolution scales: 2x, 4x, 8x, and 16x.
In Table 4, we observe that the performance methods varies significantly across different upscaling factors. For the 2x and resolutions, INCODE,and FR. show superior performance terms of all Peak Signal-to- Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) scores, indicating their capability to capture fine details and maintain structural integrity these moderate upscaling factors. As the resolution factor increases to 8x and 16x, FR and FINER emerges as the top performer, showcasing its robustness to high upscaling, which is the more challenging cases. Conversely, GAUSS and MFN struggle al higher upscaling factors, as evidenced by their relatively low SSIM and LPIPS values, indicating that these methods are less effective at reconstructing high-frequency details. The performance gap becomes more significant as the resolution incrcases, highlighting their limited capacity for handling higher upscaling tasks. On the other hand, while SIREN and FOURIER FEATURES do not perform well the other methods across all four scales, they exhibit less variation in performance the resolution
17

---

Published in Transactions on Machine Learning Research (02/2025)
Table 4: The metric evaluations across PSNR, SSIM and LPIPS of image super resolution task with 2x, 4x 8x, and 16x resolutions.
2x
Method
PSNR
SSIMf
LPIPS
PSNR
SSIM1
LPIPS
IXCODE (Kazerouni et al., 2024)
29.56
0.896
0.176
27.43
0.816
0.422
FR (Shi et al., 2024)
29.10
0.879
0.243
27.49
0.822
0.371
FINER (Liu et al., 2024)
29.50
0.892
0.191
27.44
0.818
0.395
WIRE (Saragadam et al., 2023)
28.91
0.874
0.252
25.93
0.754
0.447
FOURIER FEATURES (Tancik al., 2020)
26.31
0.767
0.428
25.73
0.733
0.473
GAUSS (Ramasinghe Lucey, 2022)
28.08
0.851
0.324
24.10
0.681
0.619
MEN (Fathony et al., 2020)
29.28
0.890
0.203
24.99
0.716
0.610
SIREN (Sitzmann al., 2020)
29.00
0.877
0.241
27.27
0.811
0.409
8x
16x
Method
PSNR
SSIMf
LPIPS
PSNRI
SSIM1
LPIPS
INCODE (Kazerouni et al., 2024)
25.43
0.731
0.597
22.91
0.638
0.715
FR (Shi et al., 2024)
23.75
0.662
0.643
23.40
0.682
0.637
FINER (Liu et al., 2024)
25.69
0.743
0.544
23.39
0.667
0.665
WIRE (Saragadam et al., 2023)
21.72
0.558
0.703
18.06
0.422
0.773
FOURIER FEATURES (Tancik al., 2020)
22.31
0.549
0.473
20.65
0.518
0.702
GAUSS (Ramasinghe & Lucey, 2022)
20.06
0.464
0.872
17.14
0.349
0.868
MFN (Fathouy et al., 2020)
19.49
0.411
0.750
16.61
0.290
0.934
SIREN (Sitzmann al., 2020)
24.66
0.703
0.583
19.09
0.509
0.780
increases, suggesting a consistent, though lower, reconstruction capability that is less sensitive to the increase in upscaling factors.
While observing from the visualisation results, in Figure 7, 8, 9, and 10, the conclusion coincide with the table result with more detailed features. From the zoom in view of Figure 8 for the lower resolution lask, observe INCODE and FINER successfully reconstructed the detailed fur at the animal's ear and the mustache. However, as the scaling increases, INCODE starts to show color artifacts in the reconstruction result, while FINER oversmooths the details. This oversmoothing is likely a result of FINER'S lack of spatial compactness, which prevents from preserving fine features at higher resolutions. In fact several other methods has shown artifacts with different features when the super resolution scale increases. For example, the Fourier based methods that lack of compact frequency and spatial representations shows undesirable repetitive patterns at finer scales, which FR exhibits twisting pattern blobs, and FOURIER FEATURES shows the grid like artifacts. Similarly, noise artifacts become more alleviated in the reconstruction results of WIRE, MFN, GAUSS, and SIREN all of which lack the task-specific paramcters necessary for tuning to different levels of reper-rro scales. This adaptability proves to be beneficial, as both the visualisations and the metrics show that INCODE, FINER, and FR which has this task-specific parameters perform better across different scales.
3.3 3D applications: Occupancy Reconstruction
The 3D occupancy task involves representing a 3D shape by determining whether points in space are inside or outside the object. For the dragon mesh, a complex model with 2,748,318 points and 5,500,000 triangles, we aim capture its detailed geometry by classifying points as either inside (1) or outside (0) the shape. The final reconstructed model has 566,098 vertices and 1,132,830 triangles, and the task evaluates how well the occupancy representation matches the true shape.
We can observe in Table 5 that FINER stands out in the Intersection over Union (IoU) metric, with a value of 0.00064 higher than the second-performing method, INCODE. Given the mesh size, this difference is significantly notable.
The visualisation comparison can be viewed in Figure 11, where all methods successfully reconstruct the global structure of the occupancy. However, the zoomed-in view indicates variations in texture among the
18

---

Published in Transactions on Machine Learning Research (02/2025)
Ground Truth
Low Resolution
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 7: The visualisation comparison of the INR methods on super resolution task with 2x
Ground Truth
Low Resolution
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 8: The visualisation comparison of the INR methods on super resolution task with 4x.
reconstruction results. In particular, MFN reconstructed a non-smooth surface with significant deformation in the local structure. Tt is worth noting that MFN, along with FR and FOURIER FEATURES, which the
19

---

Published in Transactions on Machine Learning Research (02/2025)
Ground Truth
Low Resolution
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 9: The visualisation comparison of the 8 INR methods super resolution task with 8x,
Ground Truth
Low Resolution
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 10: The visualisation comparison of the 8 INR methods super resolution task with 16x.
only 3 methods lacking frequency compactness, performed the worst among all compared methods. This
20

---

Published in Transactions on Machine Learning Research (02/2025)
Table 5; The metric comparison of 3D occupancy task across the INR methods.
METHOD
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
IoU 1
0.99564
0.99136
0.99628
0.99454
0.99424
0.99510
0.97540
0.99552
highlights the importance of frequency compactness when reconstructing finely detailed surfaces in the 3D occupancy task.
Ground Truth
INCODE
FR
FINER
WIRE
FOURIER FEATURES
GAUSS
MFN
SIREN
Figure 11: The visualisation comparison the 8 INR methods on 3D shape representation task.
4 Discussion & Future Work
The results from the experiments highlight that different. INR methods vary performance based on the tasks they are applied to, such as image denoising, super-resolution, and audio reconstruction. INCODE consistently stands out a top performer across several tasks, with high PSNR scores and superior ability to capture fine details, especially in challenging tasks like CT image reconstruction and super-resolution. However, INCODE comes with a drawback of significantly high computational cost, making it less feasible for real-time applications where efficiency is crucial. This is a clear trade-off between quality and time efficiency that practitioners need to consider.
FR (Fourier Reparameterised Training) also demonstrates excellent performance, particularly due its ability handle broad spectrum of frequencies through Fourier reparameterisation, addressing the low- frequency bias inherent. in many models. It. strikes good balance between accuracy and computational
21

---

Published in Transactions on Machine Learning Research (02/2025)
efficiency, offering viable alternative to INCODE for many tasks, especially when real-time performance is concern. FINER is another high-performing model, with its dynamic scaling feature allowing it to adapt to varying frequency distributions. This makes it highly effective across multiple applications, though like FR, it requires careful tuning of hyperparameters. SIREN, while effective in capturing high-frequency oscillatory patterns, falls slightly behind in lasks that require broader frequency adaplability, though its relatively fast. training time makes suitable for practical implementations. Lower-performing methods like WIRE, GAUSS, and MFN struggle with both accuracy and generalisation. These methods show limitations in handling complex high-frequence information or fine detail preservation, particularly in tasks like super-resolution denoising where these features are critical.
For applications where computational efficiency is less critical and the highest reconstruction quality is needed, INCODE would be the preferred choice. However, for real-time applications or cases where a balance between quality and speed is crucial, FR and FINER offer strong alternatives with their adaptable frequency handling and more manageable computation times. SIREN can be a good option when tasks require periodic signal reconstruction but with less variability in frequency content. One area where INR methods still face challenges is scalability, particularly in handling extremely high-resolution highly detailed tasks. Future work could explore more efficient frequency encoding mechanisms that reduce the computational overhead without sacrificing quality. Additionally, optimising activation functions to be more adaptive to task-specific requirements could improve generalisation across diverse datasets and applications. Enhancing the dynamic adaptability of models like FINER could allow for more robust performance across tasks that involve varying levels of detail, potentially improving their usability in real-world scenarios.
Finally, focusing on methods that efficiently integrate both spatial and frequency compactness while reducing computational demands should be priority for future research. Such advancements would make INR models more accessible for real-time applications while maintaining high fidelity in complex tasks like image and audio synthesis. While the surveyed methods provide a comprehensive and well-structured taxonomy of implicit neural representations (INRs), there are several areas present opportunitics for deeper exploration to enhance our understanding of this domain. For instance, while the interplay between activation functions and position encoding is acknowledged, further investigation into how these components synergise could reveal additional avenues for optimisation. Similarly, potential approaches beyond the proposed taxonomy, such hybrid models that integrate INRs with explicit representations, offer promising direction for enhancing performance in tasks requiring both adaptability and precision, such as 3D reconstruction. Furthermore, the influence of hyperparameters, including encoding frequencies, network architectures, and activation scaling, remains a pivotal factor in fino-tuning method effectiveness across varions data modalities. By building on the insights provided this survey, future work can explore these areas to refine and expand the utility of INR techniques in a broader range of applications. The development of implicit neural representations (INRs) has the potential to transform various domains positively. However, it crucial to consider ethical implications and ensure their responsible use to avoid unintended societal consequences.
References
Panos Achlioptas, Olga Diamanti, loannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3d point clouds. In International conference on machine learning, pp. 40 49. PMLR, 2018. 2
Ronald J Brachman. Knowledge Representation and Reasoning. Morgan Kaufman/Fllsevier 2004.
Rohan Chabra, Jan E Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Compuler SerionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 608-625. Springer, 2020. 3
Yinbo Chen and Xiaolong Wang. Transformers as meta-learners for implicit neural representations. In European Conference on Computer Vision, pp. 170-187. Springer, 2022.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVE conference computer vision and paltern recognition. pp. 8628 8638, 2021. 3
Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018. 2
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 5939 5948, 2019. 3
Yanqi Cheng, Lipei Zhang, Zhenda Shen, Shujun Wang, Lequan Yu, Raymond Chan, Carola-Bibiano Schöulieb, and Angelica I Aviles-Rivero. Single-shot plug-aud-play methods for inverse problems. arXiv preprint XrXi:22311. 13682, 2023.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference on Learning Representations, 2020. 3, 4, 10, 14, 18
Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shape induction from 2d views of multiple objects. In 2017 international conference on 3d vision (3DV), pp. 402-411. IEEE, 2017. 2
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advonces in neural information processing systems, 32, 2019. 6
Zhichun Huang, Shaojie Bai, and Zico Kolter. Implicit Implicit layers for implicit representations.
Advances in Neural Information Processing Systems, 2021. 3
Mude Hui, Zihao Wei, Hongru Zhu, Fei Xia, and Yuyin Zhou. Mierodiflusion: Implicit representation-guide diffusion for 3d reconstruction from limited 2d microscopy projections. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 11460-11469, 2024.
Danilo Jimenez Rezende, SM Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. Advances in neural information processing systems, 29, 2016.
Angjoo Kanazawa, Shubham Tulsiani, Alexci A Efros, and Jitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 371-33113 386, 2018. 2
Amirhossein Kazerouni, Reza Azad, Alireza Hosseini, Dorit Merhof, and Ulas Bagci. Incode: Implicit neural conditioning with prior knowledge embeddings. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compuler Vision, pp. 1298 1307, 2024. 4, 10, 14, 18
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521((753)::366 444, 2015.
Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representa- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2916 2925, 2018. 2
Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2019 2028, 2020. 3
Zhen Liu, Hao Zhu, Qi Zhang, Jingde Fu, Weibing Deng, Zhan Ma, Yanwen Guo, and Xun Cao. Finer: Flexible spectral-bias tuning in implicit neural representation by pariable-poriodie activation functions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 2713-2722,
2024. 3, 4, 8, 14, 18
23

---

Published in Transactions on Machine Learning Research (02/2025)
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 4460 4470, 2019. 2, 3,
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing seenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 106, 2021. 3
Amirali Molaei, Amirhossein Aminimehr, Armin Tavakoli, Amirhossein Kazerouni, Bobby Azad, Reza Azad, and Dorit Merhof. Implicit neural representation in medical imaging: A comparative survey. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023. 3
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IREE/CVP conference on computer vision and pattern recognition, pp. 165 174, 2019. 3
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International conference on machine learning, pp. 5301 5310. PMLR, 2019. 2
Sameera Ramasinghe and Simon Lucey. Beyond periodicity: Towards unifying framework for activations in coordinate-mlps. In European Conference Compuler Vision, pp. 142-158. Springer, 2022. 3, 4, 7, 14, 18
Amurag Ranjan, Timo Bolkart, Soubhik Sanyal, and Michael J Black. Generating 3d faces using convolutional mesh autoencoders. In Proceedings of the European conference on computer vision (ECCV), pp. 704-720,
2018. 2
Pradyumna Reddy, Zhifei Zhang, Zhaowen Wang, Matthew Fisher, Hailin Jin, and Niloy Mitra. A multi- implicit neural representation for fonts. Advances in Newral Information Processing Systems, 34:12637- 12647, 2021. 3
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 18507 185166, 2023. 3, 4, 7, 14, 18
Hemanth Saratchandran, Sameera Ramasinghe, Violetta Shevchenko, Alexander Long, and Simon Lucey, A sampling theory perspective on activations for implicit neural representations. arXio preprint 2XXi:2242.054222 2024. 8
Danzel Serrano, Jakub Szymkowiak, and Przemyslaw Musialski. Hose: A periodic activation function for preserving sharp features in implicit neural representations. arXiv preprint arXiv:2401.10967 2024. 4, 7
Zhenda Shen, Yanqi Cheng, Raymond H Chan, Pietro Liò, Carola-Bibiane Schönlicb, and Angelica Aviles-Rivens Trident: The nonlinear trilogy for implicit neural representations. arXiv preprint wxXiv:2311.13611 2023. 9,
Joseph Shenouda, Yamin Zhou, and Robert D Nowak. Relus are sufficient for learning implicit neural representations. arXiv preprint arXiv:2406.0255 2024. 3, 6
Kexuan Shi, Xingyu Zhou, and Shuhang Gu. Improved implicit neural representation with fourier repa- rameterized training. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 25985 25994, 2024. 4, 11, 14, 18
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representat ions with periodic activation functions. Advances in neural information processing systems, 33: 7462-743 -473, 2020. 3, 4, 6, 14, 18
David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. Tn Proceedings of the IEEE conference computer vision and patlern recognition, pp. 1955 1964, 2018. 2
24

---

Published in Transactions on Machine Learning Research (02/2025)
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7544 2020. 3, 9, 14, 18
Abiy Tasissa, Pranay Tankala, James M Murphy, and Demba Ba. K-deep simplex: Manifold learning via local dictionaries. IEEE Transaclions on Signal Processing, 2023. 2
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autocncoder- based representation learning. ar preprint 1rxiw:18122 812.05069, 2018. 2
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pp. 62- 2018. 2
Qing Wu, Yuwei Li, Lan Xu, Ruiming Feng, Hongjiang Wei, Qing Yang, Boliang Yu, Xiaozhao Liu, Jingyi Yu, and Yuyao Zhang. Irem: High-resolution magnetic resonance image reconstruction via implicit neural representation. Medical Image Computing and Computer Assisted Citrvention-MMICCA. 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VT 24.
pp. 65 74. Springer, 2021.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. Tn Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022. 2,
Dejia Xu, Peihao Wang, Yifan Jiang, Zhiwen Fan, and Zhangyang Wang. Signal processing for implicit neural representations. Advances in Neural Information Processing Systems, 35:13404 15:11404-13418, 2022. 3
Lipei Zhang, Yanqi Cheng, Lihao Liu, Carola-Bibiane Schönlieb, and Angelica I Aviles-Rivero. Biophysics informed pathological regularisation for brain tumour segmentation. arXiv preprint arXiv:2408.09136, 2024. 3
You Zhang, Hua-Chich Shao, Tinsu Pan, and Ticlige Mengke. Dynamic cone-beam et reconstruction using spatial and temporal implicit neural representation learning (stinr). Physics in Medicine & Biology, 68 (4):045005, 2023. 3
25


Reviews:
[{"review_id": "P9TUVyx6TU", "reviewer": "Reviewer_64LX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nI think this is a good paper that attempts to unify a wide range of techniques for INRs. It brings a lot of new insight into how to compare different INR architectures and techniques and how to choose the appropriate methodology for a particular INR task. The paper does an excellent job of providing a taxonomy of the different methods and providing an empirical comparison of them.\n\n### Weaknesses\n- As a survey paper, I am curious about how INRs hold up against traditional image processing techniques. What are the advantages of an INR approach for CT reconstruction compared to standard TV regularized reconstruction?\n- There has been some recent work showing that, despite conventional belief, ReLU neurons can be used in INRs without positional encodings [1]. That paper makes a clear distinction between optimizing the INR to fit an image and the expressive power of the INR. Since this work is claiming that it provides an \"extensive review of INR methodologies and applications\" it would be good to at least mention this work if not include it in the experiments. \n- In light of the work mentioned above I feel the following statement, at the beginning of Section 2.1, should be refined. \n    - \"The ReLU function, while effective in many settings, has a discontinuous nature that limits its ability to capture higher-frequency signals, making it suboptimal for tasks that require detailed or high-resolution data representation.\"\n    - The discontinuities are not the issue. Rather its the monotonic nature of the ReLU that creates an ill-conditioned loss landscape making it difficult to optimize the network via gradient descent. \n- Regarding WIRE, the authors should also mention that another drawback is that it requires two hyper-parameters ($s$ and $\\omega$) whereas other activation functions only require one. This in turn results in more computational effort to choose the appropriate hyperparameters.\n- Many of these INR methodologies depend, crucially, on how the hyperparameters are chosen. However, while the experimental section uses the default hyperparameters from each method it is difficult to conclude whether some methods are truly better or just better tuned. In addition it is my understanding that there are different \"default\" hyperparameters depending on the task. For instance in WIRE the hyperparameters that work well for audio reconstruction are not necessarily expected to work for CT reconstruction or super-resolution. So were the same hyperparameters used across tasks or did the experiments use the default hyperparameters for each task?\n- The paper does not consider or mention any work that tries to provide a theoretical understanding of why these methods work so well. For instance [2][3]. This might be worth mentioning these to make the survey more self-contained.\n- (minor) There are a number of small writing issues that should be revised.\n\n[1] Shenouda, Joseph, Yamin Zhou, and Robert D. Nowak. \"ReLUs Are Sufficient for Learning Implicit Neural Representations.\" Forty-first International Conference on Machine Learning.\n\n[2] Yüce, Gizem, et al. \"A structured dictionary perspective on implicit neural representations.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Najaf, Mahrokh, and Gregory Ongie. \"Towards a Sampling Theory for Implicit Neural Representations.\" arXiv preprint arXiv:2405.18410 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a survey of the current state-of-the-art INR models. It develops a taxonomy for categorizing different INR approaches. These include four key areas: activation functions, positional encoding, combinations of these and network structure optimization. They analyze the trade-offs between current state-of-the-art INR techniques based on the properties of each as well as their own experiments.  The conclusions from the experiments are that INCODE and FR perform the best across nearly all INR tasks but require extensive computation. Finally, the paper highlights new avenues for improvement in all four of these key areas.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Refine the statements on ReLU networks in light of [1].\n- Include some discussion/experiments comparing INR methods to traditional image processing techniques.\n- Writing issues brought up by the other reviewers. \n- For the experiments, the authors should clarify whether they used the same default hyperparameters across all tasks or a different default for each task.\n- One experimental setting where the hyperparameters are optimized for each INR method to provide a fair comparison.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OzKVBOt5HF", "reviewer": "Reviewer_Nozp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-written, and easy to follow. \n- The numerical section of the paper is strong, showing comparison of different INR methods across diverse tasks.\n- I also appreciate the taxonomy proposed in the paper. Within this structure, it is easy to understand the different INR methods, for what data/signal they are suited for, and advantages/disadvantages. \n\n**Weaknesses**\n- I think the paper would benefit from defining more precisely computational complexity. In different places, computational complexity of different methods is referred as high or low. It is understandable that it might be hard to get exact computational complexity. I think the paper would benefit, at the very least, providing rough dependence of complexity on network and problem parameters. This will allow to at least appreciate why some methods are more expensive than others.\n- There are few minor typesetting/writing issues that need to be corrected\n- There are related methods which share similar themes. One example is latent representations (Autoencoders), where signals are encoded into low-dimensional latent spaces using neural networks (e.g., as autoencoders or variational autoencoders (VAEs)). Another set of methods is structured dictionary learning, where signals are approximated as sparse combinations of basis elements learned from data. Few references on the latter method are noted below. I think comparing these approaches from INR in the introduction section would be useful. \n\n>References\n- Chen, Yubei, Dylan Paiton, and Bruno Olshausen. \"The sparse manifold transform.\" Advances in neural information processing systems 31 (2018).\n- Tasissa, Abiy, et al. \"K-Deep Simplex: Manifold Learning via Local Dictionaries.\" IEEE Transactions on Signal Processing (2023).\n- Tschannen, Michael, Olivier Bachem, and Mario Lucic. \"Recent Advances in Autoencoder-Based Representation Learning.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a review paper on Implicit Neural Representations (INRs), which are neural networks used to model signals (e.g., images, 3D shapes and audio) as continuous functions. The paper starts by highlighting the advantages of IRNs, compared to existing methods. It then introduces a taxonomy that enables to compare the different INR methods based on their general structure. The taxonomies are: Activation function, positional encoding, activation function + positional encoding, and network structure.  Within these taxonomies, the paper summarizes key INR methods highlighting their advantages and disadvantages. In addition, the numerical experiment of the paper compares the different INR methods on different applied tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The review recommends addressing some of the weaknesses listed above.\n\n> Minor: Suggestions to improve writing/typesetting\n\n- Page 2: The second sentence reads disjoint from the rest. Maybe combine the two sentences.\n\nSpecifically, Multi-Layer Perceptrons (MLPs) are trained to parameterise signals by taking input coordinates by a mapping technique that projects the network input into a higher-dimensional space. It predicts the associated data values.\n- Full stop after equation 1\n- Add full stop after \"the final output reads: equation\".\n- Add comma after equations 3-9\n- Be consistent: low-frequency bias or low frequency bias\n- For equation (3), use math mode for sin\n- Define i in (5) for the general reader\n- Page 7 before equation 7:  which form is give by -->  which form is given by:\n- Define Z, which appears in the set of translates, after equation 7\n- In discussing Sinc function, be consistent with choice: Sinc or sinus cardinal \n- Suggested rephrase before equation 8: To overcome this issue, the authors of the model\n- Suggested rephrase before equation 9: Accordingly, the authors initialise the bias coefficients as:\n- Define U in (9)\n- After equation (11), exp should be typeset in math mode. Similar comment to cos and exp in (12) and (13)\n- Also, add full stops after equations (12) and (13)\n- Comma after equation (15)\n- cos should be in math mode in equation 18\n- Reviewer suggests rewriting the two sentences after equation (18) i.e., The sentence that starts with \"Where\" \n- Add comma after equation (20)", "broader_impact_concerns": "I have no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nEwjIrcbHf", "reviewer": "Reviewer_7Zht", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper is well-written, technically sound, and informative, providing a valuable resource for researchers interested in INRs, both new and experienced. \n\nThe experiments cover a diverse set of tasks, demonstrating the capabilities and challenges of different INR methods in practice. The results and discussion can both help practitioners select the appropriate approach for their task as well as help guide new research in the field. \n\nThe included literature is up-to-date, focusing on works from the last ~4 years. \n\nWeakness and suggested changes to improve them are included in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a survey of Implicit Neural Representations, including a taxonomy of INR technique improvements in literature as well as experimental comparisons.  The taxonomy identifies 4 categories of advancements: activation functions, position encoding, combined strategies, and network structure optimization. For each category, approaches from relevant works are summarized and compared on the basis of their mathematical properties, and then empirically evaluated across multiple applications for 1D, 2D, and 3D signals. The work concludes with a discussion including recommendations for which approach to use, given task characteristics and computational efficiency constraints.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend this work for acceptance upon making the following changes.\n\nAs all of the activation functions presented in Section 2 are univariate, providing a plot for each (as a single figure) would help your audience visualize and compare them. This would be particularly useful for the less well-known functions like sinc as well as uncommon combinations of functions such as the HOSC activation function. \n\nFor each experiment and each method, you used the default hyperparameter settings presented in their respective paper. While I agree this is a simple fair setting, it would be worth explicitly adding this caveat to your discussion of the results of each experiment, such as MFN not capturing higher frequencies in section 3.1.\n\nAlthough the final section is labeled “Discussion and Future Work”, only the final paragraph (just over 3 lines of text) discusses future work in INRs. What holes can you identify between the surveyed methods? Are there (potential) approaches outside of the taxonomy? How do hyperparameters (or lack thereof) affect the utility of a given approach, and how could this be amended in further development? What is the interplay between INRs and other trends like transformers, NeRFs, etc.? For example, can the weights of an INR network be generated directly by another model rather than trained from scratch? These later questions could also be supplemented with discussion in Section 1 to frame where INRs already fit within the context of AI.\n\nSpecific typo/error corrections for clarity: \n\nPage 2: “by taking input coordinates by a mapping technique that projects the network input into a higher-dimensional space. It predicts the associated data values.” -> “by taking coordinates as inputs, applying a mapping technique that projects this input into a higher-dimensional space, and predicting the associated data values at each coordinate.”\n\nPage 4: “...with N layer, …” -> “...with N layers, …”\n\nPage 5: “The ReLU function […] has a discontinuous nature…” -> “The ReLU function […] has a non-smooth nature…”\n\nPage 6: “The authors in (Sitzmann et al., 2020) proposed SIREN…” and “The authors propose using the Gabor wavelet”: please use consistent tense (either always past or always present). Searching through the document for “author” shows a lot of inconsistent tense usage.\n\nPage 6: “When exploring beyond periodicity, the network employs…” sounds like the network is exploring beyond periodicity. You could rather say something like “In non-periodic activation functions, the authors propose using the Gaussian function, defined as: …”. Later in your evaluation of this technique, you say “it is advantageous in tasks where … localised details are important” but then say it is “less suitable for tasks that involve intricate [...] data”, but these seem contradictory.\n\nPage 8: “...the authors to initialise…” -> “...the authors proposed to initialise…”\n\nPage 10: Please define $i, \\theta_i, \\gamma_i,$ and $\\mu_i$. Also, Equation 16 has an extraneous left parenthesis.\n\nPage 10: “...appropriate weight reprametisation of weights.” -> “...appropriate weight reparameterisation.\"\n\nPage 11: “...that has publicly…” -> “...that have publicly…”", "broader_impact_concerns": "There is no included broader impact statement. One could be included (either separately or as a paragraph in the finally section) to strengthen the work, discussing topics on ethical issues surrounding INRs such as deepfakes.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the detailed reviews and discussion, I recommend accepting this paper for publication in TMLR with Survey Certification. The paper presents a comprehensive survey of Implicit Neural Representations (INRs) that is thoroughly supported by evidence. All three expert reviewers recommended acceptance and praised its value for both novice and experienced researchers in the field. The authors were responsive to reviewer requests, strengthening the paper in their revision.

The paper's strong technical quality and broad appeal to TMLR's audience are evident in its comprehensive taxonomy, thorough experimental comparisons, and clear practical guidance. The reviewers highlighted different strengths: unification of INR techniques, strong numerical experiments, and thorough literature coverage respectively.
Decision: Accept (poster)