Title: Understanding Bias Terms in Neural Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Implicit Neural Representation

INR Classification

Neural Field

Coordinate-based Network
Abstract: In this paper, we examine the impact and significance of bias terms in Implicit Neural Representations (INRs). While bias terms are known to enhance nonlinear capacity by shifting activations in typical neural networks, we discover their functionality differs markedly in neural representation networks.
Our analysis reveals that INR performance neither scales with increased number of bias terms nor shows substantial improvement through bias term gradient propagation. We demonstrate that bias terms in INRs primarily serve to eliminate \textit{spatial aliasing} caused by symmetry from both coordinates and activation functions, with input-layer bias terms yielding the most significant benefits.
These findings challenge the conventional practice of implementing full-bias INR architecture. 
We propose using freezing bias terms exclusively in input layers, which consistently outperforms fully biased networks in signal fitting tasks.
Furthermore, we introduce Feature-Biased INRs~(Feat-Bias), which initialize input-layer bias with high-level features extracted from pre-trained models. This feature-biasing approach effectively addresses the limited performance in INR post-processing tasks due to neural parameter uninterpretability, achieving superior accuracy while reducing parameter count and improving reconstruction quality.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the effect of bias term in commonly used INRs which utilize periodic activation functions. In contrast to piece-wise linear activation functions like ReLU, the bias term does not have the same influence on the network behavior and prediction performance as for commonly used ReLU-networks. The authors find that while the bias term has less influence on overall performance, completely omitting it leads to \"spatial aliasing\" an infolding of spatial content. As a solution, they propose using a frozen bias term in the first layer, which is distilled from a larger ViT. This approach aims to shift the INR's symmetry, thereby eliminating the infolding spatial content.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Novelty: The study of the bias term in INRs utilizing periodic activation functions, as opposed to piece-wise linear ones, is both novel and compelling.\n- Efficiency: Employing a fixed bias term in the first layer and eliminating additional bias terms in subsequent layers can lead to savings in computational resources and memory consumption.\n- Clarity: The paper is well-written, with a clear and easy-to-follow line of argumentation. \n\nWeaknesses:\n1. While the intuition regarding spatial aliasing is compelling, the paper lacks a theoretical analysis to substantiate these insights. \n2. Although the authors provide good empirical evidence for spatial aliasing when a bias term is omitted, they do not offer qualitative empirical evidence demonstrating that their proposed approach (using a fixed bias term only in the first layer) effectively eliminates spatial aliasing. Thus, besides the empirical performance evaluation, qualitive results should be provided. \n3. The usage of a compressed feature vector from a larger ViT, while conceptually interesting, raises questions about its necessity and whether the feature vector genuinely remains useful given the potential for inherent aliasing introduced by the compression pipeline of this feature vector. \n4. The authors claim that the bias term in INRs with periodic sine activation results in a phase shift (line 111). However, this potentially interesting property is not further discussed, despite its possible role in the observed spatial aliasing.\n\nQUESTIONS:\n1. Could the authors add a comparison between Feat-Bias and a randomly sampled bias vector which follows different distributions (e.g., uniform-like, Gaussian-like)? Is the use of a feature vector from a ViT necessary?\n2. While the authors provide several examples for \"spatial aliasing\" for bias free IRNs, it would be interesting to visualize the reconstruction results yielded by networks with frozen biases like done with Feat-Bias. For a fair comparison it would be even more interesting to qualitatively compare standards IRN with periodic activation function with learned biases, fixed biases and Feat-Bias.\n3. The visualization of the spatial aliasing suggested that the diagonal line is the center line, however, it seems as the artifacts are negatives of the original one flipped upside down, is there a reason why the authors used a diagonal line?\n4. In the context of INRs, [a,b] used INRs, or more precisely, Hypernetworks, to learn CNN kernel weights. Would the authors think that this kind of inherent symmetry might even be beneficial for kernel learning with INRs? \n\n[a] David W. Romero, Robert-Jan Bruintjes, Jakub Mikolaj Tomczak, Erik J Bekkers, Mark Hoogendoorn, and Jan van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum? id=3jooF27-0Wy.\n\n[b] Grabinski, Julia, Janis Keuper, and Margret Keuper. \"As large as it gets-studying infinitely large convolutions via neural implicit frequency filters.\" Transactions on Machine Learning Research 2024 (2024): 1-42.\n\nLIMITATIONS:\nAs the authors state, a theoretical explanation and proof are missing. Therefore, the evidence relies solely on the empirical study and the intuition regarding the inherent symmetry induced by coordinates and the periodic activation function. However, the qualitative empirical analysis and evidence are incomplete and should be added to compensate for the missing theoretical evaluation.\n\nSince the authors don't present qualitative results for their approach, it's unclear whether the spatial aliasing problem is fully resolved or merely shifted. This is particularly relevant for the Feat-Bias configuration (a frozen bias term only in the first layer), which could potentially lead to shifted artifacts because the bias term isn't adapted during learning.\n\nIf the authors can thoroughly evaluate their configuration and demonstrate through qualitative empirical evaluation (going beyond the current performance evaluation) that adapting the bias term to a fixed Feat-Bias leads to fewer artifacts, and thus should become the new standard in INRs, I would be happy to raise my score.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the role of bias terms in Implicit Neural Representations (INRs) and challenges the common assumption that biases primarily enhance the network’s nonlinearity. Through extensive analysis, the authors demonstrate that biases mainly serve to eliminate spatial aliasing artifacts rather than increase model expressiveness. Based on this insight, the paper proposes a novel bias modulation technique called Feat-Bias, which strategically leverages bias terms to reduce aliasing effects in INR reconstructions.\n\nUnfortunately, the paper is poorly written and difficult to follow. Although the authors emphasize reconstruction quality, the experimental setup for this task is weak. Conversely, the experiments primarily focus on the INR classification task. However, the section “3 Application: Featuring INRs with Bias Terms” is unclear and poorly organized. Furthermore, the key component, Feat-Bias, is insufficiently explained. In addition, Figure 4 is not referenced anywhere in the manuscript.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The main topic is very interesting.\n\n2. The hypothesis in the paper is well defined.\n\nWeaknesses:\n\n1. The paper lacks theoretical justification for the proposed hypothesis.\n\n2. The notation of (Observation #1), (Observation #2), and (Observation #3) is unclear. The authors should precisely formulate their theses and provide theoretical support.\n\n3. Reconstruction quality should be evaluated on different types of datasets, such as videos, 3D objects, and sounds—not only images.\n\n4. The baselines used for reconstruction tasks are weak. The authors should include comparisons with:\n\n- FreSh: Frequency Shifting for Accelerated Neural Representation Learning (ICLR 2025)\n- SPDER: Semiperiodic Damping-Enabled Object Representation (ICLR 2024)\n\n5. The authors should consider more architectures, such as Fourier Feature Networks and Positional Encoding.\n\n6. More activation functions should be tested.\n\n7. The Feat-Bias method is unclear and poorly described in the paper. Additionally, Figure 4 is not cited anywhere in the main text.\n\n8. Training curves in Figure 2 are incomplete. It is unclear what happens when the model is trained for 10,000 iterations.\n\n9. Figure 1 is not convincing since the differences in PSNR are small, and I do not observe any visible difference in the images.\n\n10. Perhaps dividing the reconstruction and original image for comparison could help illustrate the differences more clearly.\n\nQUESTIONS:\n1. The structure of the paper should be improved. The authors need to clearly specify which task is central to the paper—reconstruction or classification—and precisely describe the key architectural components.\n\n2. A theoretical justification for the proposed approach should be included.\n\n3. Additional experiments on diverse datasets and with different architectures are necessary.\n\n4. The Feat-Bias method is unclear and insufficiently described.\n\n5. Figures and visualizations should better emphasize the main advantages of the paper.\n\nLIMITATIONS:\nThe authors have not adequately addressed their work's limitations and potential negative societal impact. To improve, they should explicitly discuss the scope and boundaries of their method, such as the types of data and tasks for which their approach may be less effective.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper investigates the infulence of the bias terms in INRs through both theoretical analysis and empirical validation. The authors demonstrate that bias terms in INRs primarily serve to eliminate spatial aliasing caused by symmetry from both coordinates and activation functions. To further address this issue, the paper introduces Feat-Bias, which initialize input-layer bias with high-level features extracted from pre-trained models.

Extensive experiments are performed on a representation task to validate the effectiveness of bias terms, and on a classification task using the CIFAR-10 dataset to demonstrate the performance of Feat-Bias.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. Analyzing the role of bias terms in INRs from the perspective of spatial aliasing caused by symmetry is both novel and well-motivated.

2. Extensive experiments are carefully designed to validate the effect of bias terms.

**Weaknesses**

1. While the theoretical analysis is insightful, some of the claims would benefit from more formal proofs or clearer mathematical exposition.

2. The classification results are promising, but currently limited to CIFAR-10. Additional datasets would help assess the generality and robustness of Feat-bias in broader settings.

QUESTIONS:
1. The classification experiment is conducted only on CIFAR-10, which may not sufficiently demonstrate the generalization ability of Feat-bias.

2. Could the authors offer a more formal mathematical characterization of how bias terms influence coordinate symmetries and contribute to the suppression of spatial aliasing?

3. Ablation experiment is limited.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

comments: SUMMARY:
- The authors propose Feat-Bias which initializes input-layer bias terms of an INR network with features extracted from pretrained models and keeps them frozen during training.
- Feat-Bias results in improved INR classification performance.
- The authors provide an explanation for why the bias terms can be replaced and how that can result in improved performance

STRENGTHS AND WEAKNESSES:
Strengths:
- The authors provide a clear explanation of how standard bias term training does not improve performance, that the role of bias terms is to prevent spatial aliasing, and that using bias terms in only the input layer is sufficient.
- This is a simple, easy to implement idea that has a performance benefit.
Weaknesses:
- It is not clear to me that knowledge is really being distilled from the pretrained network into the INR bias weights rather than that the pretrained weight initialization has some kind of regularization effect. The fact that significant dimensionality reduction is needed to squeeze the pretrained weights into the INR weights suggests to me that it might be mostly regularization. I would like to have seen a baseline where a distribution was fit to the pretrained weights and then the INR bias weights were randomly sampled from that distribution.

QUESTIONS:
- Why do the authors think that the transferred bias terms result in improved performance? Could it be regularization?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03