iFormer: Integrating ConvNet and Transformer for Mobile Application
Authors: name: Chuanyang Zheng

We present a new family of mobile hybrid vision networks, called iFormer,
with a focus on optimizing latency and accuracy on mobile applications. iFormer
effectively integrates the fast local representation capacity of convolution
with the efficient global modeling ability of self-attention. The local
interactions are derived from transforming a standard convolutional network,
i.e., ConvNeXt, to design a more lightweight mobile network. Our newly
introduced mobile modulation attention removes memory-intensive operations in
MHA and employs an efficient modulation mechanism to boost dynamic global
representational capacity. We conduct comprehensive experiments demonstrating
that iFormer outperforms existing lightweight networks across various tasks.
Notably, iFormer achieves an impressive Top-1 accuracy of 80.4\% on ImageNet-1k
with a latency of only 1.10 ms on an iPhone 13, surpassing the recently
proposed MobileNetV4 under similar latency constraints. Additionally, our
method shows significant improvements in downstream tasks, including COCO
object detection, instance segmentation, and ADE20k semantic segmentation,
while still maintaining low latency on mobile devices for high-resolution
inputs in these scenarios.