Title: Structured Initialization for Vision Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transformers

Weight Initialization

Attention Mechanism

Convolutional Neural Networks

Inductive Bias
Abstract: Convolutional Neural Networks (CNNs) inherently encode strong inductive biases, enabling effective generalization on small-scale datasets. In this paper, we propose integrating this inductive bias into ViTs, not through an architectural intervention but solely through initialization. The motivation here is to have a ViT that can enjoy strong CNN-like performance when data assets are small, but can still scale to ViT-like performance as the data expands. Our approach is motivated by our empirical results that random impulse filters can achieve commensurate performance to learned filters within a CNN. We improve upon current ViT initialization strategies, which typically rely on empirical heuristics such as using attention weights from pretrained models or focusing on the distribution of attention weights without enforcing structures. Empirical results demonstrate that our method significantly outperforms standard ViT initialization across numerous small and medium-scale benchmarks, including Food-101, CIFAR-10, CIFAR-100, STL-10, Flowers, and Pets, while maintaining comparative performance on large-scale datasets such as ImageNet-1K. Moreover, our initialization strategy can be easily integrated into various transformer-based architectures such as Swin Transformer and MLP-Mixer with consistent improvements in performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nVision Transformers enable global attention across image patches. However, in the vision domain, local patches tend to exhibit strong correlations, and ViTs struggle to capture such local dependencies effectively. As a result, ViTs often require more training time and larger training datasets, which poses challenges for training on small datasets. This paper introduces a structured initialization method that mimics the behavior of CNNs, helping ViTs better capture local correlations from the start. The proposed initialization leads to improved accuracy compared to standard random initialization and is particularly valuable when training ViTs on small datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is clearly written and well-organized.\n- The proposed initialization is simple to implement and effective, without requiring architectural changes or pretrained weights.\n- The method demonstrates consistent performance improvements across a wide range of datasets, particularly on small and medium-scale datasets.\n\nWeaknesses:\n- In Table 2, the reported accuracy for DeiT-Base is 76.25%, which is significantly lower than the 81.8% reported in the `timm` library. This 5–6% discrepancy suggests a potential issue with the experimental setup. The authors mention in the appendix that the gap is due to differences in learning rate scaling and data augmentation settings, and they report an improved accuracy of 81.24% for the default initialization under corrected settings. However, it is surprising that this corrected result is not included in Table 2 of the main paper. This omission is problematic, as the presented results show both the mimetic baseline and the proposed method underperform compared to random initialization from `timm`, which contradicts the paper’s stated goal of outperforming standard initialization. Moreover, the corrected accuracy (81.24%) still shows a ~0.5% gap from the 81.8% reference, indicating that the issue has not been fully resolved. I recommend the authors refer to the official DeiT repository for verified training hyperparameters: https://github.com/facebookresearch/deit/tree/main. Given that this paper targets a top-tier conference like NeurIPS, such discrepancie on such a well known baselines are not acceptable.\n- While the paper claims broad applicability to Swin and MLP-Mixer models, the experimental evaluation for these architectures is very limited. For example, MLP-Mixer is evaluated only on CIFAR-10. If the authors wish to claim general applicability in the abstract and contributions, a more comprehensive evaluation is warranted.\n- Since the use of random impulse filters may bias the Vision Transformer toward more local attention at the expense of global attention, the paper would benefit from including results on ImageNet variants such as ImageNet-V2, ImageNet-R, or ImageNet-Sketch to demonstrate robustness to distribution shifts.\n- Since the proposed method mimics CNN-like locality, which typically leads to faster convergence, an ablation study on the number of training epochs would be insightful. It would be interesting to see whether the structured initialization leads to faster convergence compared to the baselines. For example, instead of training for 300 epochs, train for 100 epochs and compare the results... I would expect to get better results.\n- The impulse filter generation is stochastic, so final performance may depend on the random seed. An evaluation of variance across different seeds would help assess the robustness of the method.\n- The paper claims (line 36) that prior initialization methods are tied to specific architectures, but it does not specify which methods these are or explain how the limitation applies. This point needs further elaboration.\n- The models are referred to as “ViT” even though they are trained using DeiT training recipes. To avoid confusion with other works (e.g., “How to Train Your ViT”), it would be more accurate to refer to these models as “DeiT.”\n\nQUESTIONS:\n- There is a lack of explanation regarding the visualization of the attention maps. The paper states that the proposed method is superior because it does not exhibit purely diagonal behavior, unlike the baselines. However, this point is not well elaborated. Could the authors clarify why non-diagonal attention patterns are beneficial?\n- In Appendix B, the authors state that the proposed impulse initialization does not prevent the model from attending globally. However, the attention map visualizations suggest that the self-attention behavior in the proposed method is strongly influenced by the initialized impulse patterns. This appears to present a potential contradiction. Could the authors clarify how the model transitions from the localized, impulse-driven patterns at initialization to globally-aware attention during training?\n- In the training pipeline, you mention that for the small datasets, “We follow the training recipe from [Initializing models with larger ones]”. However, that paper appears to rely on pretrained models for training. Could you clarify what specific aspects of their approach you adopted for the small datasets? Did you use pretrained models, or were there other elements of their training procedure that you followed?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel initialization strategy for ViTs that leverages the inductive biases of CNNs to improve the training of ViTs on small-scale datasets. The core idea is to initialize the attention maps based on random impulse convolution kernels, using random impulse convolution filters to initialize the query (Q) and key (K) matrices, thereby incorporating the architectural bias of CNNs into the initialization process without altering the Transformer architecture. Both the theoretical and empirical justification demonstrate the technical soundness of the method, and extensive results on various datasets reflect the effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Paper strength**:\n1. Combine CNN and ViT Advantages: The proposed method combines the inductive biases of CNNs with the flexibility of ViTs, unlike methods that rely on pre-trained models, this structured initialization does not require large-scale pre-training, making it accessible for tasks with limited data.\n2. Easy-to-implement and Maintains Transformer Flexibility: The proposed structured initialization dose not change the architecture, which is somewhat easy to implement and preserving ViTs’ ability to learn dynamic, long-range global features.\n3. Theoretical analysis and extensive evaluation: the theoretical elaboration provide fundamental insight and extensive results reflect the effectiveness of the proposed method.\n\n\n**Paper weakness**:\n\n1. About the weights of value (V) and projection layers: The current method focuses on the initialization of the Q and K matrices, neglecting the initialization of value weights (V) and the projection matrix. A more comprehensive initialization strategy that includes more components could potentially enhance the overall quality of ViTs, as well as provides more in-depth analysis.\n2. Why only focusing on limited-data scenarios? I wonder the practical meaning of this method to enable ViTs to perform comparable with CNNs under limited data. In other words, Transformers are better for scaling and CNNs are more suitable in relatively limited data applications, since CNNs perform well under such setting, why this paper focuses such setting instead of exploring the scaling capability of ViTs under large-scale settings?\n3. Integration with architectural design method: Since the proposed method requires no architectural modification, it is encouraged to involve methods with dedicated model designs to further testify the flexibility and compatibility of the method.\n4. Multi-scale structural initialization : considering the importance of multi-scale and hierarchical nature of images, it is suggested to explore multi-scale structural initialization and evaluate if there are consistent performance gains.\n\n**Justification**:\n1. In my opion, CNNs are relatively more suitable when limited data is available and Transformers have better scalibility, this paper's motivattion to enable Transformers to be as effective as CNNs under such scenarios is not suffecient. The essentiality and timeliness of this research is somewhat limited.\n2. The empirical evaluation could not comprehensively reflect the generalization and efficacy of the proposed method.\n3. Could the proposed method be scaled for larger model parameters with more training data, this is doubt to me.\nTherefore, I tend to vote for rejection.\n\nQUESTIONS:\nPlease refer to the weaknesses section.\n\nMinor-suggestions: \nLine 63: missing a blank after [21, 15]\nLimitations: although the authors discuss the limitations, it is better to include potential plans to address them.\nImplementation details: more details about the experimental settings and hyper-parameters should be provided.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes structured initialization for Vision Transformers, in which the authors replace the usual random-weight or mimetic initializations of Q and K with carefully designed random impulse convolution filters that embed the locality bias of CNNs directly into the initial attention maps. They justify this choice theoretically by showing that, owing to the rank-deficiency of patch embeddings, a set of impulse kernels spans the same functional space as trained spatial filters, so learning can be delegated to the channel-mixing layers while the spatial bias is “baked in.” Empirically, this single-line initialization change boosts ViT-Tiny accuracy by 2~8 points on six small/medium benchmarks (Food-101, CIFAR-10/100, STL-10, Flowers, Pets), matches or betters baseline and mimetic schemes on ImageNet-1K, and transfers cleanly to Swin Transformer and MLP-Mixer variants.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n1. The idea of injecting inductive bias purely through initialization is highly innovative. It elegantly sidesteps the trade-offs of hybrid architectures, preserving the model's full expressivity while providing a much-needed structural prior for training.\n2. The experiments are well-designed, covering a wide range of dataset sizes, multiple architectures. The results are consistently and significantly positive.\n\nCons:\n1. The authors commendably acknowledge that their ImageNet baseline performance is slightly lower than in the original papers - DeiT. While their focus on relative improvement is valid, it slightly tempers the impact of the absolute numbers reported.\n2. In the large-scale dataset experiments shown in Table 2, as the model size increases, the performance improvement with impulse becomes more significant. Is there any detailed analysis on this? If the gain is solid, it would be worth a deeper investigation. Based on this observation, it may be valuable to try running ViT-base on the small/medium-scale datasets in Table 1 to see if even greater improvements can be achieved.\n3. The work focuses exclusively on impulse filters. While justified by Corollary 2 and the feature selection argument, it is unclear if other simple structures (e.g., Gaussian kernels, DoG filters) could offer different and potentially useful biases.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new initialisation method for non-CNN architectures, based on CNN like filters. They use impulse filters for this and provide theoretical justification. The initialization strategy differs from prior work by initializing different attention heads with different filters. On small-scale datasets, this initialization improves performance significantly, similar to architectural inductive biases. However, unlike architectural inductive biases, they don't limit scalability on larger scale datasets (up to IN-1K).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is clearly written.\n- Inducing inductive bias through initialization rather than the architecture is an interesting and promising direction, as it benefits from the best of both worlds: good on small-scale without limiting large-scale.\n- The paper finds theoretical justification for findings in recent work that show for ConvMixer like architecture, the spatial mixing doesn't have to be trained to achieve good performance.\n- The paper shows their initialization even outperforms prior weight selection methods which use pre-trained models.\n- The pretraining results are promising, highlighting that the benefit of the initialization remains even after pre-training.\n\nWeaknesses:\n- Though the paper shows they can improve ViT on small-scale datasets, they don't provide a comparison with ConvMixer (or similar). Thus it's unclear whether their method makes ViT actually useful on a small-scale, as one otherwise might still better use a CNN.\n- DeiT-B 224 without distillation shows 81.4 accuracy on ImageNet-1K. In your reproduction it's 76.25. What explains this difference? It's a bit concerning to me, as there might be a problem in the training pipeline, which might change the conclusion, though unlikely.\n- The paper argues this initialization is useful in domain-specific use-cases, as there limited data is available. However, this is never verified. Though experiments are done on small scale generic datasets, no pre-training is used there (except for one table with IN1K pre-training, which is arguable too limited). Therefore, it seems to me important, that it is verified that current off-the-shelf large-scale pre-trainings are indeed less effective on domain-specific data (which is dissimilar from generic pre-training datasets), and structured initialisation is more effective. I do understand this might be challenging and don't think this is crucial for this paper to be accepted.\n\nQUESTIONS:\n- Are there downsides to the structured initialisation?\n- Would any benefit of initialisation for fine-tuning a pre-trained model remain when pre-training is scaled up?\n- The paper claims better robustness to hyperparameter selection, but doesn't show it in the main paper. Maybe it's better to include it in the paper as it's part of the main message (including the conclusion)?\n- Can we be sure that the initialization doesn't limit performance compared to random initialization when training on even larger scale datasets than IN1K? Or different types of training, such as self-supervised objectives. I think it's likely, but it's not sure.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}