Title: Normalize Filters! Classical Wisdom for Deep Vision
Venue: NeurIPS 2025
Accepted: yes
Keywords: convolutional neural networks

deep learning

filter normalization

codomain symmetry
Abstract: Classical image filters, such as those for averaging or differencing, are carefully normalized to ensure consistency, interpretability, and to avoid artifacts like intensity shifts, halos, or ringing. In contrast, convolutional filters learned end-to-end in deep networks lack such constraints. Although they may resemble wavelets and blob/edge detectors, they are not normalized in the same or any way. Consequently, when images undergo atmospheric transfer, their responses become distorted, leading to incorrect outcomes. We address this limitation by proposing filter normalization, followed by learnable scaling and shifting, akin to batch normalization. This simple yet effective modification ensures that the filters are atmosphere-equivariant, enabling co-domain symmetry. By integrating classical filtering principles into deep learning (applicable to both convolutional neural networks and convolution-dependent vision transformers), our method achieves significant improvements on artificial and natural intensity variation benchmarks. Our ResNet34 could even outperform CLIP by a large margin.  Our analysis reveals that unnormalized filters degrade performance, whereas filter normalization regularizes learning, promotes diversity, and improves robustness and generalization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uncovers that filter kernels in convolutional neural networks (CNNs) are usually not normalized (meaning that the sum of all positive respectively all negative components is 0 or 1 respectively 0 or -1). Classic results from image processing show that such filters are not equivariant to gain and bias changes to their input, which could hinder generalization to gains and biases which were not seen during training. A simple formulation to normalize the filters in the abovementioned way is proposed, and the subsequent equivariance to gain and bias (and even invariance to bias in the usual case of filters with both positive and negative components) is proven. Very extensive evaluation demonstrates that these properties actually improve image classification accuracy under unseen gains and biases in many cases, and various analyses show that this method makes networks substantially more well-behaved.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written, introduces the problems with non-normalized filters very clearly, relates them nicely to classic results from image processing, and formally proves its claims.\n2. The proposed change to CNNs is very non-invasive, elegant, and simple to implement. It could thus enjoy widespread adoption.\n3. The evaluation is very extensive and doesn't hide the fact that in some cases, this method doesn't actually improve performance, but at least retains it. In addition, various behavioral analyses of the networks provide interesting insights, e.g., on hidden feature distribution or saliency.\n\nWeaknesses:\n1. I think section 3 could be streamlined a bit more to make it even easier to digest and follow. For some ideas, see my questions below.\n2. The mix of evaluation plots and tables, also including the supplementary material, seems a bit messy.\n\nQUESTIONS:\n1. Looking at Eq. 1, I wonder whether and how these results also apply to fully connected layers?\n2. Line 174f: To a cursory reader, it might be non-obvious that after the normalization, is is guaranteed that w.l.o.g., $||w^+|| = 1$ and $||w^-|| =$ either 1 or 0, and hence $r(w)$ is either 0 or 1, thus the filter is either strictly averaging or strictly differencing, and Eq. 11 can be used to prove the equivariance and invariance guarantees. I suggest clearly describing this reasoning and really \"hammering it in\", as I believe it is a key step to understanding your contribution.\n3. If I'm not mistaken, the theoretical equivariance and invariance results only cover spatially uniform gain and bias. As you're also evaluating on spatially varying gain and bias, is it possible to elaborate on that? Do you just assume local spatial uniformity since filter kernels are usually small? If yes, I think this should be mentioned in the paper.\n4. What is meant by \"co-domain symmetry\", mentioned in various places in the paper?\n5. Line 212f: Is there a reason why you introduce sort pooling layers? In general, I would appreciate a more complete description of all tested architectures, including network width, depth, activation functions, pooling layers, residual connections, and the use of batch normalization in the interest of reproducibility.\n6. Lines 214 & 243: Is there a specific reason why you chose SGD over Adam?\n7. Lines 326f: \"Artifacts reduction\" and \"a more targeted correction\" are rather handwavy justifications for the superiority of filter normlization over instance normalization. In addition, no intuition to what this actually means is provided. I suggest you either elaborate on this or remove these claims.\n\nFurthermore, here are some minor questions and potential improvements:\n- Lines 191 & 202: I don't understand what the $\\mathbf{1}()$ function in the definitions of $\\tilde{x}$ is doing.\n- Lines 194 & 198: Shouldn't it be the Hadamard product $\\circ$ in the definitions of $\\tilde{x}$ here?\n- Fig. 5: Please add small labels to the vertical axes of the plots so one can grasp how large the accuracy improvement really is.\n- Fig. 6: Could you specify in the paper what \"moderate\" and \"strong\" corruption are concretely?\n- Fig. 9: Similarly, could you specify which corruption you used here?\n\nLIMITATIONS:\nIn many cases, it appears that image classification accuracy actually doesn't improve that much when introducing filter normalization. Big improvements are mostly seen when the bias is very high (evaluation set $D_S$), only few images are used for training (Fig. 7), or the ExDark dataset is used.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new filter normalization technique to enhance the capability of image processing of deep convolutional networks. Classical / human-crafted image filters are usually normalized to maintain consistency, interpretability, and to avoid visual artifacts. Well the data-driven image filters usually do not have this feature. However, the lack of normalization would lead to distorted filter responses, especially under atmospheric / lighting change environment. The proposed method discovers that all the convolutional filters can be reformulated as a combination of averaging filter and differencing filter, and hence proposes to normalize any given / learnt convolutional filter to ensure the consistency and interpretability. Extensive experiments on a wide range of application show the effectiveness of the proposed filter normalization method.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n1. The proposed method comes with a clear and elegant formulation and solid derivation and proof. \n2. The observation of the unnormalized convolutional filter in deep learning is critical and useful. This fact could potentially discover some intrinsic features of convolutional deep neural networks. \n3. The paper is strongly motivated and clearly structured. The proposed method could be applied to many real application problems such as photo editing, intensity correction, astronomic image processing, etc. \n4. The extensive experiments show that the proposed method has achieved the beset performance across different application scenarios. \n\n\n### Weakness\n\n1. The header of section 3.1 \"Any filter is both averaging and differencing\" is not guaranteed. It should be better rephrased as \"any linear / convolutional filter ... \"\n2.  The experiment on data augmentation on CIFAR-10 with only atmospheric augmentations may not be very representative.  First, the data size of CIFAR and the image size are both small. Recent advances in image processing are all conducted on large scale high resolution data. It is suggested to make more thorough experiments on those dataset to make augmentation conclusion. Second, the experiment results do not show a clear trend that data augmentation techniques make model suffer, even though the performance on D_s alone is not as good as expected.\n\nQUESTIONS:\nThis paper is technically sound and well formulated. The authors are suggested to address the weakness in the weakness session during rebuttal period.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper proposes to separate and normalize filters in convolutional layers into positive and negative filters to help mitigating against gain and bias. They show experiments with a neural networks modified this way and compare with other computer vision models .

STRENGTHS AND WEAKNESSES:
**Strengths**
* Simple modification to convolutional layers with apparent gains in robustness and interpretation

**Weaknesses**
* experiment could include other corruptions beyond these customized datasets.
* comparison of ResNet fine-tuned with CLIP on zero-shot is somewhat misleading, and perhaps not the best choice. We know that CLIP does not do well on robustness to covariate shift, I am not sure what the experiment shows for the actual topic of the paper.
* not widely accepted use of the word "atmosphere"  effect, The paper even shows some experiments on astronomy data taken from space.

QUESTIONS:
1. Would it be possible to also add a fine-tuned CLIP for comparison
2. Would it be possible to add a non-customized dataset to actually see how the modification to the filters behave when corruptions are not just gain and bias?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the issue of deep learning models' reduced robustness to intensity variations in images, which stems from the unnormalized nature of learned convolutional filters. It proposes a novel Filter Normalization method that decomposes convolutional filters into their averaging filter and differencing filter components and normalizes them individually. This modification is designed to ensure that filters exhibit atmosphere-equivariance, leading to consistent and interpretable responses under varying illumination conditions. Experiments on both artificially corrupted and natural datasets demonstrate that this approach significantly enhances model robustness and generalization across various CNN and Vision Transformer architectures. The authors claim that this method, by integrating classical filtering principles, enables leaner models to outperform larger, unnormalized counterparts, including CLIP, on these specific benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n### 1. Addresses a Critical Problem in Deep Learning\nThe paper tackles the significant and long-standing issue of robustness in neural networks, particularly concerning image intensity variations, which is crucial for real-world applications like autonomous driving and medical imaging.\n\n### 2. Novelty and Effectiveness of Filter Normalization\nThe core idea of normalizing convolutional filters by separating positive and negative parts and applying learnable scaling and shifting is novel. This approach is well-motivated by classical filter theory. Its effectiveness is demonstrated by consistent performance gains across CNNs and Vision Transformers on both artificial and natural benchmarks.\n\n### 3. Simplicity and Broad Applicability\nThe proposed method is simple to implement (\"simple yet effective modification\") and can be readily integrated into various convolution-based architectures, suggesting high potential for wide adoption and future development in different domains\n\n# Weaknesses\n\n### 1. Ambiguous Definition and Proof of Co-domain Symmetry and Atmosphere-Equivariance\nThe paper claims that its method ensures \"atmosphere-equivariant, enabling co-domain symmetry\", but it lacks clear mathematical definitions for these terms. While Equation (11) demonstrates linearity for individual filter responses, the logical connection between this and the general concept of \"atmosphere-equivariance\" or \"co-domain symmetry\" for the entire network remains insufficiently elaborated.\n\n### 2. Insufficient Consideration of Non-linearities\nThe theoretical derivations for consistent responses (e.g., Equation 11) apply primarily to linear convolutional layers. The paper does not adequately discuss how the proposed filter normalization impacts or interacts with non-linear activation functions (like ReLU) within deep networks, which could dilute the \"interpretable responses\" and overall equivariance in a multi-layer network.\n\n### 3. Unfair Comparison with CLIP\nThe paper strongly highlights outperforming CLIP on corrupted datasets. However, this comparison is unfair as CLIP (zero-shot) models are pre-trained on significantly larger datasets (e.g., LAION-5B) compared to the ImageNet-1k training of the proposed R34N. While the R34N demonstrates efficiency, presenting this as a direct outperformance of a model trained on orders of magnitude more data is misleading and should be heavily qualified. I'm not saying the necessity of a fair comparison, but the authors should not claim like \"better than CLIP trained on much larger datasets\".\n\n### 4. Limited Scope and Public Credibility of Experimental Results\nThe majority of reported performance gains are on custom-designed, artificially corrupted ImageNet and CIFAR-10 datasets. These simulated atmospheric effects might be overly simplistic and not fully representative of complex real-world conditions. While some results on natural datasets like ExDark and LEGUS are provided, they are limited (presented inline, not in comprehensive tables, and not compared against all baselines). Moreover, the absence of evaluations on well-established public benchmarks for robustness, such as relevant subsets of ImageNet-C (e.g., brightness, contrast, haze), diminishes the credibility and generalizability of the reported performance.\n\nQUESTIONS:\n- __Clarify Theoretical Concepts:__ Provide clear mathematical definitions for \"co-domain symmetry\" and \"atmosphere-equivariance.\" Elaborate on how Equation (11) directly translates to the broader concept of atmosphere-equivariance for deep networks.\n\n- __Address Non-linearity Impacts:__ Discuss the implications of non-linear activation functions (e.g., ReLU) on the proposed filter normalization's effects and the concept of \"interpretable responses\" throughout the entire network.\n\n- __Refine CLIP Comparison:__ Rephrase claims regarding CLIP's \"outperformance\" to explicitly acknowledge and emphasize the vast disparity in training data scale (ImageNet-1k vs. LAION-5B).\n\n- __Strengthen Experimental Validation:__ Evaluate the proposed method on well-established public robustness benchmarks, particularly relevant subsets of ImageNet-C (e.g., brightness, contrast, haze). Expand the natural data experiments beyond just inline text, presenting them in comprehensive tables that compare all baselines where applicable.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}