Scale-Aware Modulation Meet Transformer
Authors: name: Weifeng Lin; name: Ziheng Wu; name: Jiayu Chen; name: Jun Huang; name: Lianwen Jin
Published: 2023-07-17 15:47:48

Abstract: This paper presents a new vision Transformer, Scale-Aware Modulation
Transformer (SMT), that can handle various downstream tasks efficiently by
combining the convolutional network and vision Transformer. The proposed
Scale-Aware Modulation (SAM) in the SMT includes two primary novel designs.
Firstly, we introduce the Multi-Head Mixed Convolution (MHMC) module, which can
capture multi-scale features and expand the receptive field. Secondly, we
propose the Scale-Aware Aggregation (SAA) module, which is lightweight but
effective, enabling information fusion across different heads. By leveraging
these two modules, convolutional modulation is further enhanced. Furthermore,
in contrast to prior works that utilized modulations throughout all stages to
build an attention-free network, we propose an Evolutionary Hybrid Network
(EHN), which can effectively simulate the shift from capturing local to global
dependencies as the network becomes deeper, resulting in superior performance.
Extensive experiments demonstrate that SMT significantly outperforms existing
state-of-the-art models across a wide range of visual tasks. Specifically, SMT
with 11.5M / 2.4GFLOPs and 32M / 7.7GFLOPs can achieve 82.2% and 84.3% top-1
accuracy on ImageNet-1K, respectively. After pretrained on ImageNet-22K in
224^2 resolution, it attains 87.1% and 88.1% top-1 accuracy when finetuned with
resolution 224^2 and 384^2, respectively. For object detection with Mask R-CNN,
the SMT base trained with 1x and 3x schedule outperforms the Swin Transformer
counterpart by 4.2 and 1.3 mAP on COCO, respectively. For semantic segmentation
with UPerNet, the SMT base test at single- and multi-scale surpasses Swin by
2.0 and 1.1 mIoU respectively on the ADE20K.