the whole image but swin transformers limit the self-attention window so if the authors increase the resnet's window size then they should reach the same compromise this results in an increase in performance of nothing but it's clear that small kernel sizes hurt the performance and finally the sixth and final stage comes with micro design choices such as replacing the rectified linear unit activation function with the gaussian error linear unit used in bird or gptNUMBER_TOKEN or vit they are using the activation functions less often as transformers also do in the mlp block only and with the same motivation to do everything more transformer style the authors also use fewer normalization layers because transformers use them less often than the resnet as well they are replacing batch normalization with layer normalization short reminder in batch norm a mini batch of data that has multiple examples and features associated to it the normalization happens across the batch or sample dimension the batch normalization effect is that the magnitude of the features is similar across data samples in the batch layer norm normalizes across the feature dimension such that per data sample the features have around the same magnitude historically layer norm was something that worked better for recurrent neural networks whereas batch norm was mostly used in convenience in surprise all these micro modifications have a tremendous impact on performance and by using separate down sampling layers this enhanced resnet achieves an accuracy of NUMBER_TOKEN imagenet top NUMBER_TOKEN accuracy exceeding the swin transformer and these results are comparable because the swin transformer plays in the same league in terms of computation and size if you're curious to know how this fares in the big model league you just have to look at the gray bars in the figure where they start from a resnet NUMBER_TOKEN and add all these steps with similar gains okay cool we have seen steady performance increase and really impressive numbers and results but what are the lessons here with respect to our confident versus transformer boxing match so the first lesson here is that not only transformers but modernized convnets scale too in terms of size and data and we have not discussed the data aspect i know check out the additional experiments with more data in the paper the second lesson is related to the tasks convnets can compete in object detection and semantic segmentation with the swin transformer which we remember was a hierarchical version of vit maybe you are wondering what exactly the novelty of this paper is well we cite it is also worth noting that none of the design options discussed thus far is novel they have all been researched separately but not collectively over the last decade and while the authors here are genuine convolution advocates by reading this paper we must ask ourselves what if the bigger lesson of this paper applies to other architectures too what if while everyone in nlp suddenly jumped onto the next stage dominated by transformers forgot to stepwise improve rnn's or their tree rnns or everything that is not fashionable anymore so far we thought that self-attention and the way that transformers process information in parallel highways is the recipe for success but this paper suggests that it is rather the many seemingly tiny architecture hyper parameters and not the architecture itself that can tweak the way to state of the art another thing we wonder and this is maybe something only the authors can answer is whether the experiments were really conducted in the order presented in the paper and in this figure here it is more plausible that these design choices were added in all possible orders and configurations and that the authors picked and presented everything where improvement over improvement can be seen don't get us wrong we
NO_SEGMENT_TOKEN