Title: The Quest for Universal Master Key Filters in DS-CNNs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Convolutional Neural Networks

Depthwise Separable Convolutions

Gaussian Derivatives
Abstract: A recent study has proposed the ``Master Key Filters Hypothesis" for convolutional neural network filters. This paper extends this hypothesis by radically constraining its scope to a single set of just 8 universal filters that depthwise separable convolutional networks inherently converge to. While conventional DS-CNNs employ thousands of distinct trained filters, our analysis reveals these filters are predominantly linear shifts (ax+b) of our discovered universal set. Through systematic unsupervised search, we extracted these fundamental patterns across different architectures and datasets. Remarkably, networks initialized with these 8 unique frozen filters achieve over 80\% ImageNet accuracy, and even outperform models with thousands of trainable parameters when applied to smaller datasets. The identified master key filters closely match Difference of Gaussians (DoGs), Gaussians, and their derivatives, structures that are not only fundamental to classical image processing but also strikingly similar to receptive fields in mammalian visual systems. Our findings provide compelling evidence that depthwise convolutional layers naturally gravitate toward this fundamental set of spatial operators regardless of task or architecture. This work offers new insights for understanding generalization and transfer learning through the universal language of these master key filters.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the Master Key Filters Hypothesis for Depthwise Separable Convolutional Neural Networks (DS-CNNs). It improves on previous works by showing that the trained filters can be replaced by affine transformations of a small set of base filters, with little harm to the model accuracies. These filters are found by training an autoencoder on a collection of existing trained models' filters and then drawing samples with it. This work also draws theoretical insights by comparing these base filters with Gaussian filters and shows they coincide. Extensive numerical experiments are done on ImageNet, CIFAR-10, and a few others using different sizes of ConvNeXt models. Through these experiments, this work freezes the filters to be the few base filters it has chosen and arrives at a comparable accuracy after training for certain epochs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The results presented are inspiring and practically beneficial.\n- The experiments are done extensively on a number of datasets.\n- The paper is well written.\n- The accuracy loss incurred by filter pruning (in Table 1) is less when the model is larger, which is good since large models are the motivation for pruning. In particular, this trend seems to have a connection with overparametrization and the NTK results e.g. [1,2], in which case large MLPs find good parameters close to their initializations. The trend in Table 1 may imply the larger the model is, the less it needs to diverge from the base filters. This may be an interesting connection to make, if I am not mistaken here.\n\nWeaknesses:\n- No statistical confidence is given. The authors argue that running experiments on ImageNets is costly. But is it possible to run and give a confidence on the smaller datasets used in Section 4.2?\n- Apart from ImageNet, the other datasets are relatively easy tasks. It remains a question whether such a nice concentration on the filters still holds under more complicated visual tasks.\n\nTypos:\n- Line 173, \"a small set\".\n- Line 226, \"Figure 4\".\n- Line 236, \"This filter likely contributes\".\n- Line 244-245, missing $\\sigma$ as subscript in defining gaussians (for $G_1$, $G_2$ to make sense in DoG).\n- Table 3, \"normal training\"\n\nReferences:\n\n[1] Jacot, Arthur, Franck Gabriel, and Clément Hongler. \"Neural tangent kernel: Convergence and generalization in neural networks.\" Advances in neural information processing systems 31 (2018).\n\n[2] Allen-Zhu, Zeyuan, Yuanzhi Li, and Zhao Song. \"A convergence theory for deep learning via over-parameterization.\" International conference on machine learning. PMLR, 2019.\n\nQUESTIONS:\n1. What is $n$ in Eq (1)?\n2. The authors find the affine transform of a single base filter for each trained filter. What is different if one finds a linear combination of all base filters for each trained filter? Since the authors use only a small set of base filters, I think the number of coefficients is still acceptable.\n3. From a perspective different from saving memory, can one facilitate training by first freezing the 8 base filters for a few epochs and then allowing them to be trained for a few epochs? It seems from Table 3 that frozen filters give faster convergence in 300 epochs, compared to fully trainable filters.\n4. Is there a connection to overparametrizations? See Strengths.\n5. Is there also a connection to the recent research on the implicit low-rank bias of deep CNNs [1,2]? This line of research essentially shows, when trained with regularization, deep CNNs tend to have most of their filters aligned with the pooling (up to linear transformation) so the information kept by the trainable filters is also kept by pooling. In this case, most filters are linear transformations of each other. In particular, it raises the question if the base filters found in this work happen to allow most information to pass under any (or the popular) poolings.\n\n*I would appreciate any answers the authors have to these questions.\n\nReferences:\n\n[1] Súkeník, Peter, Marco Mondelli, and Christoph Lampert. \"Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?.\" arXiv preprint arXiv:2405.14468 (2024).\n\n[2] Wen, Yuxiao, and Arthur Jacot. \"Which Frequencies do CNNs Need? Emergent Bottleneck Structure in Feature Learning.\" International Conference on Machine Learning. PMLR, 2024.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describe a method to identify master key filters from separable CNNs, and claims that these filters mimics the receptive field of in mammalian visual systems. Furthermore, the authors show that the separable CNNs with frozen master key filters (with learnable bias) are able to achieve comparable performance as the fully trainable counterpart.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea that the filters in separable CNNs mimics receptive fields in mammalian visual systems is very intriguing, which also challenges the assumptions that the filters for later layers extract are more complicated than earlier ones.\n2. The flow of the paper is smooth --- from hypothesis to investigation; from results to analyzing; from observations to experiments.\n\nWeakness:\n1. The writing overall is not very clear, especially when technical precision is required (see Questions section for details). But I would like to point out an example that annoys me most here --- the paper heavily uses the term 'linear shift' without a precise description. When I first read it, I thought it means geometric shift instead of amplitude shift. Also, such mapping should be called 'affine' instead of 'linear'.\n2. The methods used in the paper (both for analysis and experiments) are not the most solid ones in my opinion. For example, a variational encoder is more suitable for the scenario, but the authors choose to use a vanilla encoder-decoder.\n\nQUESTIONS:\n**Writing**: I would strongly recommend the authors to enhance the technical precision of the writing.\n  - The paper only studies depth-wise CNN, but keep writing depth-wise separable CNNs (DS-CNNs). For a 7x7 filter, a depth-wise filter has 49 parameters, but a depth-wise separate one has only 14 parameters.\n  - From Line 114-117, I don't understand why centering is an issue as the filter is flattened into a vector anyway. Also, the authors write 'scaling their length to 1', but the readers would naturally think it is the filter length (7).\n - The network has 4 encoder/decoders layers, but Figure 2 only has one.\n - In Line 140, the matrix F is never defined before and never used later.\n - In Line 142, the authors write 'a linear approximation with respect to the decoded filters' --- the readers would think it is the linear combination of decoded filters.\n - Equation (1)-(3) are very standard linear regression --- there is no need to distract the audience with details.\n - The greedy algorithm is only mentioned by words, and not in maths (or pseudo code).\n - The whole Section 3.1 is three page long without subsection or delimiters.\n\n**Method**:\n  - The auto encoder-decoder model is not the most up-to-date method for unsupervised clustering.\n  - I wonder if there is any specific reason why a standard VAE (or a more advanced categorical VAE) is not used in this setting.\n  - Also, the proposed method naturally applies to standard CNN instead of depth-wise CNN --- does it mean the phenomenon can only be observed in depth-wise but not a standard one?\n\n**Experiments**:\n  - From my understanding, the majority of the parameters remain learnable in the experiment setup. For example, if the number of channels if 1024, the number of parameters in the depth-wise filter is only 49*1024~=50K, while the number of parameters in the fully-connected part is 1024x1024=1M --- so the parameters in the depth-wise filter is negligible.\n  - I hypothesize that if you initialize the depth-wise filters randomly and remain unchanged during learning, the model is able to achieve similar results as the fully-learnable one.\n  - Also, I would recommend adding experiments compressing a standard CNN in a similar manner - if the compressed CNN achieve similar result, that will make the hypothesis in the paper more convincing.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an investigation on the diversity of learned layers in convolutional NNs with depthwise separable convolutions (DS-CNNs). The results of this investigation show that all the learned spatial convolutional kernels are the linear transformations of 8 basic layers. Two main experimental verifications are as follows: 1) All spatial kernels in a trained DS-CNN can be exchanged for the linear transformation of a closest kernel from this set with small accuracy drop; 2) All spatial kernels can be initialized with kernels from this set, leaving only the bias trainable, with small accuracy drop.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. The paper presents and experimentally verifies a novel and fascinating idea. The experimental part is convincing.\n2. The process of searching for these filters is explained in detail.\n\n**Weaknesses**:\n\nThe paper is very insightful and captivating. My comments are mostly about the lack of detailed experimental description and suggestions for new experiments:\n1. The paper glosses over two important experimental details: 1) How large is the model bank which the pretrained kernels were taken from? 2) How to choose the correct kernel from these 8 for a specific layer in the model during initialization? **Action item**: please answer these questions in the paper.\n2. Also, after establishing the 8 classes of filters, a question about the importance and role of each filter naturally arises. It would be nice to see some statistics: which class is the most popular in the pretrained networks? How are they situated with respect to the layer depth? **Action item**: please compute these statistics.\n3. Moreover, it would be very interesting to compare the training loss dynamic with and without initialization, which was done in [1]. **Action item**: please report the graphs with training/validation loss dynamic.\n4. The authors consider only kernels of size 7 by 7. Arguably, the 3 by 3 kernels are also widely used – do they have the same properties? **Action item**: please confirm experimentally what would happen if 3 x 3 convolutions would be replaced by a closest from a similar set of 8 3 x 3 convolutions.\n\nAlso, some small problems with the manuscript:\n1. In my opinion, the “Related work” section is unnecessarily long. What is the connection between the current paper and scale-space theory? Moreover, I would assume that there is no need in writing formulae for depthwise convolutions, as it is a widely known concept. **Action item**: please reduce this section by removing irrelevant discussion.\n2. In lines 149-150 the authors propose a well-known solution to a linear regression problem, but there is no link or citation. **Action item**: please cite a book or another article with this formula.\n3. In line 197 the authors refer to the last row of Table 1, while they probably meant the row before last.\n\n[1] Xu, Chunyu, and Hong Wang. \"Research on a convolution kernel initialization method for speeding up the convergence of CNN.\" Applied Sciences 12.2 (2022): 633.\n\nQUESTIONS:\nPlease see **action items** in “Weaknesses”. I will be willing to accept the paper if the first three are done. Moreover, I have some smaller questions:\n1. Did you think about using the discovered fact for purposes of quantization? How effective would such quantization be?\n2. Are you planning to release the code? It would be nice for everyone to be able to initialize/exchange existing layers in their models with the discovered 8 classes.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the shape of depth-wise convolution filters. Specifically, based on \"Master Key Filters Hypothesis\" from previous paper, the paper finds a few universal filters that can represent and replace depth-wise convolution filters in various networks. An auto-encoder in the filter domain is trained on a pre-trained filter set and utilized to find the initial 50 filters. The depth-wise convolution filters in ConvNeXt v2 models are quantized with 50 filters with two parameterizations, and reduced to 8 filters at last. The paper claims that it extends \"Master Key Filters Hypothesis\" to only 8 filters with meaningful performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Writing is clear and easy to understand\n- It is interesting to replace ConvNeXt models's filters with only eight filters\n\nWeaknesses\n- The contribution overlaps with previous papers\n  - \"Master Key\" is interesting, but has already been discussed in previous papers. It significantly limits the novelty and contribution of the paper.\n  - I acknowledge that implementing the master keys as eight filters is novel and makes a contribution. But, it is a bit insufficient for me.\n    - The Master Key Filters Hypothesis: Deep Filters Are General in DS-CNNs, AAAI 2025\n    - Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels, ICLR 2024\n- Limited practical contribution\n  - I don't think \"Master Key\" has practical value. It doesn't reduce FLOPs or improve network throughput, and the reduction in the number of parameters is negligible. But it drops the performance.\n  - The experiments like Table 3 can make a practical contribution. But Table 3's settings are outdated, and improvements are not significant.\n- Additional parameters `A, B, shift` for \"master keys\"\n  - The \"Master Key\" introduces additional parameters to represent the filters. It is reasonable to use a few parameters to represent all filters, but I feel it conflicts with \"Master Key Filters Hypothesis\"\n  - Actually, I think `A, B` are necessary to make performance without re-training, even when \"Master Key Filters Hypothesis\" is true. But I wonder how the shift works. If the shift is a two-dimensional float vector that moves \"Master Keys\" with bilinear interpolation, it should be mentioned and clarified in the paper.\n- Limited applicable architectures\n  - \"Master Key\" is only validated on limited architectures that use depth-wise convolution with pre-Layer Norm blocks: ConvNeXt and HorNet. I doubt the finding can be generalized to CNN without pre-LayerNorm or depth-wise convolution.\n  - The paper's findings are only valid on depth-wise convolution-based architectures. Thus, it can't cover a wide range of architectures, such as ViT variants and CNN without depth-wise conv, which reduces the contribution.\n  - The paper's experiments only cover pre-LayerNorm architectures. It has to be verified in depth-wise conv without pre-LayerNorm: EfficientNet V2, and MobileNetV3.\n- Finding \"master keys\" requires evaluation on the test set\n  - The paper reduces the filters based on test accuracy, limiting insights into the filter shape required for performance. It would be better to provide an alternative way to reduce filters without an actual performance test.\n\nQUESTIONS:\n- \"Master Key\" topic is already addressed in previous papers. Considering these two papers, what is the unique novelty and contribution of the paper?\n    - The Master Key Filters Hypothesis: Deep Filters Are General in DS-CNNs, AAAI 2025\n    - Unveiling the Unseen: Identifiable Clusters in Trained Depthwise Convolutional Kernels, ICLR 2024\n- Finding the minimum \"Master Key\" looks interesting. But, what are the practical benefits of this technique? Can this reduce computation or improve generalization performance? \n- I can't understand how \"shift\" works. Does \"shift\" use integers or floats? Is this done by learnable parameters like a and b?\n- I think the \"Master Key\" still has a few degree of freedom: a, b, and shifts. Does it perform better than 3x3 depth-wise convolution? Because it also reduces the # of params, I think it is worth comparing with \"Master Key\".\nThe proposed method should not be a weighted sum of \"Master Keys\". However, I think some points are confusing. Would you please add a formula for selecting \"Master Key\" $f'_c$ to clarify this?\n- What is the definition of $n$ in Eq. (1) and (2)?\n- ConvNeXt and Hornet are pre-LayerNorm based residual architectures. Does \"Master Keys\" work on architectures without pre-LayerNorm? How about non-residual architectures like MobileNetV1?\n\nLIMITATIONS:\nIt would be better to explain the architecture limitation if it is not working on non pre-LayerNorm architecture.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-07"}