Relying heavily on atrous convolution in vision transformers may have potential drawbacks or limitations. One limitation is the risk of overfitting to specific patterns present in training data due to increased receptive fields with high dilation rates. This could lead to reduced generalization capabilities when applied to unseen data. Moreover, excessive use of atrous convolution may introduce computational inefficiencies, especially in scenarios where a balance between computational cost and model performance is crucial.