Efficient Modulation for Vision Networks
Authors: Xu Ma
Xiyang Dai
Jianwei Yang
Bin Xiao
Yinpeng Chen
Yun Fu
Lu Yuan
Venue: ICLR 2024 poster (2024)
Keywords: EfficientMod
Efficient Networks
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: In this study, we introduce Efficient Modulation (EfficientMod), a novel design for efficient networks that oupforms other methods in terms of both results and latency.

Abstract: In this work, we present efficient modulation, a novel design for efficient vision networks. We revisit the modulation mechanism, which operates input through convolutional context modeling and feature projection layers, and fuses features via element-wise multiplication and an MLP block. We demonstrate that the abstracted modulation mechanism is particularly well suited for efficient networks and further tailor the modulation design by proposing the efficient modulation (EfficientMod) block, which is considered the essential building block for our networks. Bene- fiting from the prominent representational ability of modulation mechanism and the efficiency of efficient modulation design, our network can accomplish better accuracy-efficiency trade-offs and set new state-of-the-art performance for efficient networks. When integrating EfficientMod block with the vanilla self-attention block, we obtain the hybrid architecture and further improve the performance without sacrificing the efficiency. We carry out comprehensive experiments to verify EfficientMod’s performance. With fewer parameters, our EfficientMod-s performs 0.6 top-1 accuracy better than the prior state-of-the-art approach EfficientFormerV2-s2 without any training tricks and is 25% faster on GPU. Additionally, our method presents a notable improvement in downstream tasks, outperforming EfficientFormerV2-s by 3.6 mIoU on the ADE20K benchmark. Code and checkpoints are available at https://github.com/ma-xu/EfficientMod.

Published as a conference paper at ICLR 2024
EFFICIENT MODULATION FOR VISION NETWORKS
Xu Ma', Xiyang D Jianwei Yang", Bin Xiao?, Yinpeng Chen', Yun Fu', Lu Yuan? Northeestter University MMicrosoft
ABSTRACT
In this work, we present efficient modulation, novel design for efficient vision networks. We revisit the modulation mechanism, which operates input through convolutional context modeling and feature projection layers, and fuses features via element-wise multiplication and an MLP block. We demonstrate that the mod- ulation mechanism is particularly well suited for efficient networks and further tailor the modulation design by proposing the efficient modulation (EfficientMod) block, which is considered the essential building block for our networks. Benefit- ing from the prominent representational ability of modulation mechanism and the proposed efficient design, our network can accomplish better trade-offs between accuracy and efficiency and set new tfateoof-the-ar performance in the zoo of efficient networks. When integrating EfficientMod with the vanilla self-attention block, we obtain the hybrid architecture which further improves the performance without loss of efficiency. We carry comprehensive experiments to verify Effi- cientMod's performance. With fewer parameters, our EfficientMod-s performs 0.6 top-1 accuracy better than EfficientFormerv22 and 2.9 better than MobileViTv2-1100 at the same GPU latency. Additionally, our method presents a notable improvement in downstream tasks, outperform- ing EfficiendFormer22- by 3.6 mIoU on the ADE20K benchmark. Code and checkpoints are available at
and is 25% faster on GPU,
INTRODUCTION
Vision Transformers (ViTs) (Dosovitskiy et al., 2021; Liu et al., 2021; Vaswani et al., 2017) have shown impressive accomplishments on wide range of vision tasks and contributed innovative ideas for vision network design. Credited to the self-attention mechanism, ViTs are distinguished from conventional convolutional networks by their dynamic properties and capability for long-range context modeling. However, due to the quadratic complexity over the number of visual tokens, self-attention is neither parameter- nor computation-effiee efffciient... This inhibits ViTs from being deployed on edge mobile devices and other real-time application scenarios. To this end, some attempts have been made to employ self-attention within local regions (Liu et al., 2021; Chen et al., 2022a) or to selectively compute informative tokens (Rao et al., 2021; Yin al., 2022) to reduce computations. Meanwhile, some efforts (Mehta & Rastegari, 2022; Chen et al., 2022b; Graham et al., 2021) attempt to combine convolution and self-attention to achieve desirable effectiiveness-efffiienn trade-offs.
Most recently, some works (Liu et al., 2022b; Yu et al., 2022a; Trockman & Kolter, 2022) suggest that a pure convolutional network can also attain satisfying results compared with self-attention. Among these, FocalNet (Yang et al., 2022) and VAN (Guo et al., 2023), which are computationally efficient and impleementationn-riennlll show cutting-edge performance and significantly outperform ViT counterparts. Generally, both approaches consider context modeling using a large-kernel convo- lutional block and modulate the projected input feature using element-wise multiplication (followed by an MLP block), as shown in Fig. lb. Without the loss of generality, we refer to this design as Modulation Mechanism, which exhibits promising performance and benefits from the effectiveness of convolution and the dynamics of self-attention. Although the modulation mechanism provides satisfactory performance and theoretically efficient (in terms of parameters and FLOPs), it suffers unsatisfying inference speed when the computational resource is limited. The reasons are two-fold: i) redundant and isofunctional operations, such as successive depth-wise convolutions and redun- dant linear projections take up large portion of operating time; ti) fragmentary operations in the context modeling branch considerably raise the latency and are in contravention of guidance G3 in ShuffleNetv2 (Ma et al., 2018).

---

Published as a conference paper at ICLR 2024
MLP block
MUP block
Unified block
Interact
FC
FC
FC
GELU
repeat
FC
FC
Layer Norm
Layer Norm
CTX
Atten. Block
Interact
Conv. block
Interact
FC
FC
FC
GELU
softmar
Conv
CTX
K
V
Conv
V
FC
FC
FC
FC
Block
FC
Layer Norm
Layer Norm
Layer Norm
(a) Transformer block
(b) Modulation design
(c) Our EfficientMod block
Figure 1: Comparison of Transformer, abstracted modulation design, and our EfficientMod block. @ is element-wise multiplication and @ means matrix multiplication. Compared to Transformer and abstracted modulation, our unified block efficiently modulates the projected values (V) via simple context modeling design (CTX). Dimension number is indicated in (c) to aid comprehension.
In this work, we propose Efficient Modulation, a simple yet effective design that can serve as the essential building block for efficient models (see Fig. 1c). In comparison modulation blocks behind FocalNet (Yang et al., 2022) and VAN (Guo et al., 2023), the efficient modulation block is more simple and inherits all benefits (see Fig. 1b and Fig. 1c). In contrast to the Transformer block, our EfficiintMod's computational complexity is linearly associated with image size, and we emphasize large but local interactions, while Transformer is cubically correlated with the token number and directly computes the global interactions. As opposed to the inverted residual (MBConv) block (Sandler et al., 2018), which is still the de facto fundamental building block for many effective networks, our solution uses fewer channels for depth-wise convolution and incorporates dynamics (see Table for comparison). By analyzing the connections and differences between our and these designs, we offer deep insight into where our effectiveness and efficiency come from. By examining the similarities and distinctions, we gain valuable insights into the efficiency of our approach.
With our Efficient Modulation block, we introduce a new architecture for efficient networks called EfficientMod Network. EfficientMod is a pure convolutional based network and exhibits promising performance. Mcanwhile, our proposed block is orthogonal to the traditional self-attention block and has excellent compatibility with other designs. By integrating attention blocks with our EfficientMod, get hybrid architecture, which can yield even better results. Without the of neural network searching (NAS), our EfficientMod offers encouraging performance across a range of tasks. Compared with the previous state-of-the-as method EfficientFormerr22 (Li al., 2023b), EfficientMod-s outperforms EfficientFormerV2..2 by 0.3 top-1 accuracy and is 25% % faster on GPU. Furthermore, our method substantially surpasses EfficientFormerV2 on downstream tasks, outperforming by 3.6 mloU on the ADE20K semantic segmentation benchmark with comparable model complexity. Results from extensive experiments indicated that the proposed EfficientMod is effective and efficient.
2 RELATED WORK
Efficient ConvNets Designs. One of the most profound efficient networks MobileNet (Howard et al., 2017), which decouples conventional convolution into point-wise and depth-wise con- volution. By doing so, the parameter number and FLOPSs are radically reduced, and the inference speed is substantially boosted. Subsequently, MobileNetV2 (Sandler et al., 2018) 8) further pushed the field by introducing the inverted bottleneck block (also known as the MBCony block), which is now the de facto fundamental building block for most efficient networks (Tan & Le, 2019; Guo et al., 2022; Li et al., 2021; Peng et al., 2021; Tu et al., 2022). In addition, some other contributions also noteworthy. Network architecture search (NAS) MobileNetv3 (Howard et al., 2019), EfficientNet (Tan & Le, 2019; 2021), 1), and FBNet (Wu et al., 2019; Wan et al., 2020; Dai et al., 2021), etc. ShuffleNet (Zhang al., 2018) leverages group operation and
provide better network designs like
2

---

Published as a conference paper at ICLR 2024
channel shuffling to save computations. ShuffleNetv2 (Ma et al., 2018), FasterNet (Chen et al., 2023), and GhostNet (Han et al., 2020a) emphasize the effectiveness of feature re-use. Regarding efficient design, model is similar to the MBConv block, but the inner operations and mechanisms behind it are different. Regarding design philosophy, our model is similar to FocalNet Yang et al. (2022) and VAN Guo al. (2023), but is considerably more efficient and elegant. We discuss the connections and differences in Sec. 3.5 and Table 6.
Transformers in Efficient Networks. Transformer has garnered considerable interest from the vision community, which undoubtedly includes effective networks. Some methods, such as Mobile- Former (Chen et al., 2022b), contemplate adding self-attention to ConvNets capture local and global interactions concurrently. Self-attention, however, endures high computational costs due to the quadratic complexity of the number of visual tokens. To eschew prohibitive computations. EfficientFormer and EdgeNeXt (Maaz et al., 2023) consider MBConv blocks in the early stages and employ self-attention the later stages when the token number (or feature resolution) is small. In contrast to earlier efforts, which combined self-attention and MBCоπw block to achieve a trade-off off between efficiency and effectiveness, we distilled the inherent properties of self-attention, dynamics, and large receptive field and introduced these properties to our EfficientMod. We also explore the hybrid architecture that integrates self-attention and EfficientMod for better performance.
Discussion on Efficient Networks. Although parameter number and FLOPs are widely employed metrics to assess the theoretical complexity of model, they do not reflect the network's 's real-time cost, endorsed in ShuffleNet2 (Ma et al., 2018). Practical guidelines for efficient network design are critical, like fewer network fragments (Ma et al., 2018) and consistent feature dimension (Li et al., 2022), etc. FasterNet (Chen al., 2023) also demonstrates that low FLOPs do not necessarily lead to low latency due to inefficient low floating-poit operations per second. In this work, we present EfficientMod and incorporate prior observations into design to achieve practical effectiveness.
3 METHOD
3.1 REVISIT MODULATION DESIGN
We first derive the general concept of modulation mechanism from VAN and FocalNet.
Visual Attention Networks. VAN (Guo et al., 2023) considers a convolutional attention design, which is simple yet effective. Specifically, given input feature x eehxx first project x to new feature space using fully-connected (FC) layer (with activation function) f (.) and then feed it into two branches. The first branch ct (-) extracts the context information, and the second branch is an identical mapping. We use element-wise multiplication to fuse the feature from both branches, and a new linear projection p (-) is added subsequently. In detail, VAN block can be written as:
Output = p (ctx (f (z)) o (x)),
(I)
ctx (t) = g (DWConV7,3 (DWConvs,1 (2C)))),,(2))).
(2)
where @ is element-wise multiplication, DWConv means depth-wise convolution with kernel size and dilation d, and g (-) is another FC layer in the context branch. Following the design philosophy of MetaFormer (Yu et al., 2022a), the VAN block is employed as a token-mixer, and a two-layer MLP block (with a depth- wise convolution) is adjacently connected as a channel-mixer.
FocalNets. FocalNets (Yang et al., 2022) introduced the Focal Modulation that replaces self- attention but enjoys the dynamies and large receptive fields. FocalNet also considers parallel two branches design, where one context modeling branch ctx (-) adaptively aggregates different levels of contexts and one linear project branch u (-) project x to new space. Similarly, the two branches are fused by element- weement-wise multiplication, and an FC layer p (-) is employed. Formally, the hierarchical modulation design in FocalNet can be given by (ignoring the global average pooling level for clarity):
ctx (z) = { act (DWConva, (f (f(z))
where etx includes L levels of context information that are hierarchically extracted by depth-wise convolutional layer with a kernel size of k,, 2 (.) project c-channel feature to a gating value. act (-) is GELU activation function after each convolutional layer.
3

---

Published as a conference paper at ICLR 2024
Abstracted Modulation Mechanism. Both VAN and FocalNet demonstrated promising represen- tational ability and exhibited satisfying performance. By revisiting as aforementioned, we reveal that both methods share some indispensable designs, which greatly contribute their advancements. Firstly, the two parallel branches are operated individually, extracting features from different feature spaces like self-attention mechanism (as shown in Fig. la). Secondly, for the context modeling, both considered large receptive fields. VAN stacked two large kernel convolutions with dilation while FocalNet introduced hierarchical context aggregation as well as global average pooling achieve a global interaction. Thirdly, both methods fuse the features from two branches via element multiplication, which is computationally efficient. Lastly, a linear projection is employed after feature fusion. We argue that the gratifying performance of the two models can be credited to the above key components. Meanwhile, there are also distinct designs, like the particular implementations of context modeling and the design of feature projection branches (shared or individual projection). Consolidating the aforementioned similarities and overlooking specific differences, we abstract the modulation mechanism as depicted in Fig. lb and formally define the formulation as:
(4) The abstracted modulation mechanism inherits desirable properties from both convolution and self- attention but operates in a convolutional fashion with satisfying efficiency in theory. Specifically, Eq. 4 enjoys dynamics like self-attention due to the element-wise multiplication. The context branch also introduces local feature modeling, but large receptive field is also achieved via large kernel size (which is not a bottleneck for efficiency). Following VAN and FocalNet, a two-layer MLP block is constantly introduced after the modulation design, as shown in Fig. Ic. Besides aforementioned strengths that make modulation mechanism suitable for efficient networks, we also tentatively introduce a novel perspective in Appendix Sec. K that modulation has the unique potential to project the input feature to very high dimensional space.
Output = p (ctx (x) o u (z)).
3.2 EFFICIENT MODULATION
Despite being more efficient than self-attention, the abstracted modulation mechanism still fails to meet the efficiency requirements of mobile networks in terms of theoretical complexity and inference latency. Here, we introduce Efficient Modulation, which tailored for efficient networks but retains all the desirable properties of the modulation mechanism.
Sliming Modulation Design. A general modulation block has many fragmented operations, as illustrated in Fig. lb. Four FC layers are introduced without considering the details of the context modeling implementation. As stated in guideline G3 in ShuffleNetv2 (Ma et al., 2018), too many fragmented operations will significantly reduce speed, even if the computational complexity may be low by tweaking the channel number. To this end, we fuse the FC layers from the MLP and modulation blocks as shown in Fig.lc. We consider v (-) to expand the channel dimension by an expansion factor of r and leverage p (-) to squeeze the channel number. That is, the MLP block is fused into our modulation design with flexible expansion factor, resulting in a unified block similar to the MBConv block (we will discuss the differences and show our superiority in Table. 6).
Simplifying Context Modeling. We next tailor our context modeling branch for efficiency. given the input x, we first project x to a new feature space by a linear projection f (z). Then, depth-wise convolution with GELU activation is employed to model local spatial information. We set the kernel size to 7 to balance the trade-off between efficiency and large receptive field. Lastly, a linear projection g (a) is employed for channel communication. Notice that the channel number kept the same throughout the context modeling branch. In short, our context modeling branch can be given by:
ctx (x) = (act (DWConv7,1 (f (z)))).
(5)
This design much simpler than the context modeling in VAN and FocalNet. We discard isofunctional depth-wise wise convolutions by one aurge-kernel depth-wise convolution. We acknowledge that this may slightly degrade the performance as compromise to efficiency. Ablation studies demonstrate that each operation in context branch is indispensable.
3.3 NETWORK ARCHITECTURE
With the modifications as mentioned above, we arrive at our Efficient Modulation block depicted in Fig. Ic Next, we instantiate our efficient networks, Please see Appendix Sec. B for more details.
4

---

Published as a conference paper at ICLR 2024
First, we introduce a pure convolutional network solely based on the EfficientMod block. Following common practice (Li et al., 2023b; Yu et al., 2022a), we adopt a hierarchical architecture of 4 stages; each stage consists of series of our EfficientMod blocks with residual connection. For simplicity, we used overlapped patch embedding (implemented with a convolutional layer) to down-size the features by factor of 4, 2, 2, and 2, respectively. For each block, we normalize the input feature using Layer Normalization (Ba et al., 2016) and feed the normalized feature to our EfficientMod block. We employ Stochastic Depth (Huang et al., 2016) and Layer Scale (Touvron al., 202 to improve the robustness of our model. Notice that our EfficientMod block is orthogonal to the self-attention mechanism. Following recent advances that combine convolution and attention for better performance (Li et al., 2023b; Mehta & Rastegari, 2022; Chen et al., 2022b; Pan et al., 2022), we next combine our EfficientMod with attention block to get a new hybrid design. We consider the vanilla attention block as in ViT (Dosovitskiy et al., 2021) without any modifications. The attention blocks are only introduced in the last two stages, where the feature size is relatively small. We vary the width and depth to match the parameters in the pure ianoouuionallbasee EfficientMod counterpart for a fair comparison. We introduce three scales ranging from 4M to 13M parameters, resulting in EfficientMod-xxs, EfficientMod-xss and EfficientMod-s.
3.4 COMPUTATIONAL COMPLEXITY ANALYSIS
We also examine our design's 's theoretical computational complexity and practical guidelines.
Given input feature x RC H x W, the total parameters number of one EfficientMod block is 2 (r + 1) + I2C, and the computational complexity is O (2(r+))HWCC H)HWC2 HWA*C), where k is kernel size and is the expansion ratio in v (-). We ignore the activation function and bias in learnable layers for simplicity. Compared with Attention, our complexity is linear to the input resolution. Compared with MBConv, we reduce the complexity of depth-wise convolution by factor of rr which is crucial for effectiveness as validated in Table 6.
Besides the theoretical computational complexity, also pro- vide some practical guidelines for our design. 1) We reduce the FLOPs by moving more parameters later stages where the feature resolution is small. The reason behind is that our Effi-
One Thread Latency (ms)
Repeat Reshape
76.40
21.8%
5,83
cientMod's FLOPs are basically equal to the input resolution x the number of parameters. Following this guideline, we can add more blocks or substantially increase the width in later stages.
5.1%
62.72
5.53
Note that this guideline is not unique to our EfficientMod and can be applied to all FC and Convolutional layers. II) We only introduce attention blocks to the last two stages, as a common practice in many works (Li et al., 2023b; Mehta & Rastegari, 2022; Yu al., 2022b; Mehta & Rastegari, 2023) considering self-attention's 's computational complexity. III) We use Repeat operation match channel number to save CPU time with a light overhead on GPU. EfficientFormer observed that the Reshape often bottleneck for many models. Here, we introduce more details. Reshape is considerably sluggish on the CPU but is GPU-friendly. Meanwhile, Repeat operation is swift on CPU but come-consuming on GPU. As shown in Fig. 2, two solutions (Repeat and Reshape) can be used for interact in EfficientMod, we select Repeat to get the optimal GPU-CPU latency trade-off.
GPU
CPU
Figure 2: From Repeat to Reshape, EfficientMod-s GPU latency decreases 5. 1% and CPU latency increases 21.8%.
3.5 RELATION TO OTHER MODELS
Lastly, we discuss the connections and differences between our EfficientMod block and other notable designs to emphasize the unique properties of our approach.
MobileNetV2 ushered in new era in the field of efficient networks by introducing mobile inverted bottleneck (MBConv in short) block. Compared to the MBConv block that sequentially arranges the FC layer, our EfficientMod block separates the depth-wise convolutional layer and inserts it from the side into the middle of the two-layer FC network via element- wise multiplication. We will show that our design is a more efficient operation (due to the channel number reduction of depth-wise convolution) and achieve better performance (due modulation operation) in Table 6.
5

---

Published as a conference paper at ICLR 2024
Table ImageNet-1K classification performance. We compare EfficientMod with SOTA methods and report inference latency, model parameters, and FLOPs. The latency is measured on one P100 GPU and Intel(R) Xeon(R) CPU E5-2680 CPU with four threads. We use tiny gray color to indicate results trained with strong training strategies like re-parameterization in MobileOne and distillation EfficientFormerV2. Benchmmak results on more GPUs can found Appendix Sec. H.
Latency (ms)
Model
(9))
Params (M)
FLOPs (G)
Size.
GPU
CPU
MobileNerV2x1.0 (2018)
71.8
2.1
3.8
3.5
0.3
224
FasterNet-110 (2023)
71.9
2.5
6.8
3.9
0.3
224
EdgeViT-XXS (2022)
74.4
8.8
15.7
4.1
0.6
2242
MobileOne-SI (2023)
74.6 (75.9)
1.5
6.9
4.8
0.8
224
MobileViT-XSS (2022)
74.8
4.1
21.0
2.3
1.1
2562
EfficienFFormer22-550 (2023b)
73.7 (75.7)
3.3
10.7
3.6
0.4
224
EfficientMod--xxs
76.0
3.0
10.2
4.7
0.6
224
MobileNerV2x1.4 (2018)
74.7
2.8
6.0
6.1
0.6
224
DeïT-T (2021a)
74.5
2.7
16.5
5.9
1.2
224
FasterNet-TI (2023)
76.2
3.3
12.9
7.6
0.9
224
EfficientNet-BO (2019)
77.1
3,4
10.9
5,3
0.4
224
MobileOne-S2 (2023)
(77.4)
2.0
10.0
7.8
1.3
224
Edge (2022)
77.5
11.8
21.4
6.8
1.1
224
1obileViTVv-1100 (2023)
78.1
5.4
30.9
4.9
1.8
256
EfficieenfFormer22-511 (2023b)
77.9 (79.0)
4.5
15.4
6.2
0.7
224
EfficientMod-xs
78.3
3.6
13.4
6.6
0.8
224
PoolFormer-s12 (2022a)
77.2
5.0
22.3
11.9
1.8
224
FasterNet-T2 (2023)
78.9
4.4
18.4
15.0
1.9
224
EfficientFormer-LII (2022)
79.2
3.7
19.7
12.3
1.3
224
MobileFormer-508M (2022b)
79.3
13.4
142.5
14.8
0.6
224
MobileOne-SAe S (2023)
(79.4)
4.8
26.6
14.8
3.0
224
MobileViTv2-L.5 (2023)
80.4
7.2
59.0
10.6
4.1
256
VigeV (2022)
81.0
20.5
34.7
13.1
1.9
2242
EfficientFormerV2-550 (2023b)
80.4 (81.6)
7.3
26.5
12.7
1.3
224
EfficientMod-s
81.0
5.5
23.5
12.9
1.4
224
SENet introduces dynamics to ConvNets by proposing channel-attention mechanism (Hu et al., 2018).
An SE block can be given by y x sig (W₂ (aet(Wzz)) Many recent works (Tan & Le, 2019;
Zhang et al., 2022; Liu et al., 2022a) incorporate it to achieve better accuracy while maintaining low complexity in theory. However, due to the fragmentary operations in SE block, would significantly reduce the inference latency on GPUs. On the contrary, our EfficientMod block inherently involves channel attention via y = ctx (x) (zr), where q (r) adaptively adjust the channel weights of ctx (2:).
EXPERIMENTS
In this section, validate EfficientMod four tasks: image classification on ImageNet- IK (Deng et al., 2009), object detection and instance segmentation on MS COCO (Lin et al., 2014), and semantic segmentation on ADE20K (Zhou al., 2017). We implement all networks PyTorch and convert to ONNX models on two different hardware:
GPU: We chose the P100 GPU for latency evaluation since it can imitate the computing power of the majority of devices in recent years. Other GPUs may produce different benchmark results, but observed that the tendency is similar.
CPU: Some models may operate with unpredictable latency on different types of hardware (mostly caused by memory accesses and fragmented operations). We also provide all models' measured latency on the Intel(R) Xeon(R) CPU E5-2680 CPU for a full comparison.
For the latency benchmark, we set the batch size to for both GPU and CPU to simulate real-world applications. To counteract the variance, inference time. We use four threads following the common practice. For details on more devices (e.g., different GPUs, iPhone, etc.), please check supplemental material.
repeat 4000 runs for each model and report the mean

---

Published as a conference paper at ICLR 2024
1.8 fpuer
Arch.
Model
Params
FLOPs
Acc.
Epoch
RSB-ResNet-18 (2021)
12.0M
1.8G
70.6
300
79
RepVGG-A1 (2021)
12.8M
2.4G
74.5
120
Top-1 accuracy
PaolFormer-s12 (2022))
11.9M
1.8G
77.2
300
GhostNetw2x1.66 (2022)
12.3M
0.4G
77.8
450
27
RegNetX-3.2GGF (2020)
15.3M
3.2G
78.3
100
KfkiianFormerv22
Conv
FasterNet-T2 (2023)
15.0M
1.9G
78.9
300
FasterNet
ConvMLP-M (2023a)
17.4M
3.9G
79.0
300
75
MobdleVIT
GhostNet-A (2020b)
11.9M
0.6G
79.4
450
Mobile VITv2
MobileOne-Sa (2023)
14.8M
3.0G
79.4
300
EfiiiatMMod
73
EfficientMod-s
12.9M
1.5G
80.5
300
2.5
3.5
4.5
65
+ Atten.
EllccietMMods
12.9M
14G
81.0
300
ONNX GPU Lutency (ms)
Figure 3:
The trade-off between
Table 2: We compare our convolution- based model with
ONNX GPU lateney and accuracy.
others and show improvements in the hybrid version.
4.1 IMAGE CLASSIFICATION ON IMAGENET-IK
We evaluate the classification performance of EfficientMod networks on ImageNet-IK: Our training recipe follows the standard practice in DeiT (Touvron et al., 2021a), details can be found in Appendix Sec. 5. Strong training tricks (e.g., re-parameterization and distillation) were not used to conduct a fair comparison and guarantee that all performance was derived from our EfficientMod design.
We compare our EfficientMod with other efficient designs and present the results in Table 1. Distinctly, our method exhibits admirable performance in terms of both classification accuracy and inference latency on different hardware. For instance, our EfficientMod-s performs the same as Edg but runs 15 milliseconds (about 73%) faster on the GPU and 11 milliseconds (about 32%) faster on the CPU. Moreover, our model requires fewer parameters and more minor computational complexity. EfficientMod-s also outperforms FffficientFormer V2-S2 by 0.6 improvements and runs 1.8ms (about 25%) faster on GPU. Our method performs excellently for different scales. Be aware that some efficient designs (like MobileNetV2 and FasterNet) prioritize low latency while other models prioritize performance (like MobileViTvv and EdgeViT). In contrast, our EfficientMod provides state-of-the-arr performance while running consistently fast on both GPU and CPU.
To better grasp the enhancements of method, use EfficientMod-s as example and outline the specific improvements of each modification. The results of our EfficientMod, from the pure convolutional-baase We note that even the pure convolutional-basee version of EfficientMod already produces impressive results at 80.5%, significantly surpassing related convolutional- based networks. By adapting to hybrid architecture, we further enhance the performance to 81.0%.
version to the hybrid model, are presented in Table 2.
EFormerv2
s0 (3.mm))
sl (4.5m
s2 (23m0)
w/ Distill.
73.7
77.9
80.4
w/ Distill.
75.7 (22)
79.0
81.6
EmicientMod
xXS (mm))
(360))
w/o Distill.
76.0
78.3
81.0
w/ Distill.
77.1
79.4
81.9.00
Meanwhile, some methods are trained with strong train- ing strategies, like re- parametterization (Ding et al., 2021) in MobileOne and distillation (Hinton et al., 2015) in FfficientFormerV2. When trained with distillation (following the setting in (Li et al., 2023b)), we improve EfficientMod-s from 81.0 to 81.9%, as shown in Table 3. All following results are without distillation unless stated otherwise.
Table 3: Results w/o and w/ distillation.
4.2 ABLATION STUDIES
Compare other Modula- tion models. We compare our EfficientModxxxs with FocalNet and VAN-BO, that has a similar number of parameters. For a fair comparison, we customize Focal- Net_Ting Jrf by reducing the channel number or the blocks. We tested three variants, selected the best one, and termed it FocalNet @ 4M. Since Conv2Former (Hou et al., 2022) code has not been fully released, we didn't 't consider it in our comparison. From Table 4, we see that EfficientMod outperforms other modulation methods for both accuracy and latency.
Model
Top-1(56) GPU (ms), CPU (ms))
Param.
FLOPs
VAN-BO
75.4
4.5
(15)
16.3 (163)
4.1M
0.9G
@MM
74.5
4.2 (1 1.)
16.8 (16))
4.6M
0.7G
MfficientMod-x
76.0
3.0
10.2
4.7M
0.6G
Table Compare EfficientMod with other modulation models.
7

---

Published as a conference paper at ICLR 2024
Figure 4: We directly visualize the forward context modeling results as shown in Eq. 5. The visualization results suggest that our context modeling can emphasize the conspicuous context. backward gradient is required as in Class Activation Map (Zhou et al., 2016).
Latency (ms)
f()
Conv g()
Acc.
Arch.
Model
Params
F1.OPs
Acc.
GPU
CPU
72.7 -7.8
MBCony
6.4M
1.6G
72.9
4.4
122.2
78.6 .09
EfficientMod
6.4M
1.6G
72.9
2.9
19.3
iso.
72.3 -8.2
MBConv
12.4M
3.1G
77.0
6.5
196.5
79.8 -0.7
EfficientMod
12.5M
3.1G
77.6
4.2
39.1
79.6 0.9
MBConv
6.4M
0.7G
76.9
5.4
18.7
EfficientMod
6.4M
0.7G
77.4
3.8
12.4
80.5
hier.
MBConv
12.9M
1.6G
79.8
59.6
mul.-
sum
79.5 -1.0
EfficientMod
12.9M
1.5G
80.5
5.8
25.0
Table 5: Ablation studies based on EfficientMod-s-Coon w/o attention.
Table 6: Comparison between MBConv and EfficientMod with isotropic (iso.) and hierarchical (hier.) architecture.
Ablation of each component. We start by examining the contributions provided by each component of our design. Experiments are conducted on the convolutional EfficientMod-s without introducing attention and knowledge distillation. Table 5 shows the results of eliminating each component in the context modeling branch. Clearly, all these components are critical to our final results. Introducing all, we arrive at 80.5% top-1 accuracy. Meanwhile, also conducted experiment validate the effectiveness of element-wise multiplication. We substitute it with summation (same computations and same latency) to fuse features from two branches and present the results in the last of the table. As expected, the performance drops by 1% top-1 accuracy. The considerable performance drop reveals the effectiveness of our modulation operation, especially in efficient networks.
Connection to MBConv blocks. To verify the superiority of EfficientMod block, compare design and the essential MBConv with isotropic and hierarchical architectures, respectively. Please check Appendix Sec. B for detailed settings. With almost the same number of parameters and FLOPs, results in Table 6 indicate that our EfficientMod consistently runs faster than MBCony counterparts by a significant margin on both GPU and CPU. One most probable explanation is that our depth-wise convolution is substantially lighter than MBConv's (channel numbers are and respectively, where set to 6). Besides the faster inference, our design consistently provides superior empirical results than MBConv block. Please check Appendix Sec. G for more studies on scalability.
Context Visualization. Inherited from modulation mechanism, our EfficientMod block distin- guish informative context. Following FocalNet, we visualize the forward output of the context layer (computing the mean value along channel dimension) in EfficientMod-Conee Clearly, our model consistently captures the informative objects, and the background is restrained, suggesting the effectiveness of the modulation design in efficient networks.
shown in Fig. 4.
4.3 OBJECT DETECTION AND INSTANCE SEGMENTATION ON MS COCO
To validate the performance of EfficientMod on downstream tasks, we conduct experiments on MS COCO dataset for object detection and instance segmentation. We validate our EfficientMod-s top of the common-usee used detector Mask RCNN (He et al., 2017). We follow the implementation of previous work (Yu et al., 2022a; Wang et al., 2021; 2022; Tan & Le, 2021), and train the model using ix scheduler, i.e., 12 epochs. We compare our convolutional and hybrid EfficientMod-s with other methods and report the results in Table. 7. Results suggest that EfficientMod consistently outper-
8

---

Published as a conference paper at ICLR 2024
Table 7: Performance in downstream tasks. We equip all backbones with Mask-RCNN and train the model with (1x) scheduler for detection and instance segmentation on MS COCO. We consider Semantic FPN for semantic segmentation on ADE20K. Our pre-trained weights are from Table 3.
MS COCO
ADE20K
Arch.
Backhone
Params
AP"
AP.
AP
AP
AP
AP
75
Params
FL.Pss
mloU
Conv:
ResNet-18
31.2M
34.0
54.0
36.7
31.2
51.0
32.7
15.5M
32.2G
32.9
Pool
PoolF--S12
31.6M
37.3
59.0
40.1
34.6
55.8
36.9
15.7M
31.0G
37.2
Cone
EfficientMod-s
32.6
42.1
63.6
45.9
38.5
60.8
41.2
16.7M
29.0G
43.5
Atten.
PVT-Tiny
32.9M
36.7
59.2
39.3
35.1
56.7
37.3
17.0M
33.2G
35.7
Hybrid
EfficientF-LI
31.5M
37.9
60.3
41.0
35.4
57.3
37.3
15.6M
28.2G
38.9
Iyybrid
PVTv2-B1
33.7M
41.8
64.3
45.9
38.8
61.2
41.6
17.8M
34.2G
42.5
Hybrid
FfficientFv2-22
32.2M
43.4
65.4
47.5
39.5
62.4
42.2
16.3M
27.7G
42.4
Hybrid
MfficientMod-s
32.6M
43.6
66.1
47.8
40.3
63.0
43.5
16.7M
28.1G
46.0
forms other methods with similar parameters. Without self-attention, our EfficientMod surpasses PoolFormer by 4.2 mAP for detection and 3.6 mAP on instance segmentation task. When introducing attention and compared with hybrid models, our method still outperforms others on both tasks.
4.4 SEMANTIC SEGMENTATION ON ADE20K
We next conduct experiments on the ADE20K (Zhou et al., 2017) dataset for the semantic segmen- tation task. We consider Semantic FPN (Kirillov et al., 2019) the segmentation head due to its simple and efficient design. Following previous work (Yu et al., 2022a; Li et al., 2023b; 2022; Wang al., 2021), we train our model for 40k iterations with total batch size of 32 on 8 A100 GPUs. We train our model using Adam W (Loshchilov & Hutter, 2019) optimizer. The Cosine Annealing scheduler (Loshchilov & Hutter, 2017) is used to decay the learning rate from initialized value 2e-4.
Results in Table demonstrate that EfficientMod outperforms other methods by substantial margin. Without the aid of attention, our convolutional EfficientMMdds already outperforms PoolFormer by 6.3 mloU. Furthermore, the pure convolutional EfficientMod even achieves better results than the attention-e euuipped methods. better than the prior SOTA efficient method EfficientFormett (42.4 43.5). The design of our EfficientMod block is the sole source of these pleasing improvements. When introducing Transformer blocks to get the hybrid design, further push the performance to 46.0 mIoU, using the same number of parameters and even fewer FLOPs. Hybrid EfficientMod-s performs noticeably better than other hybrid networks, outperforming PvTv2 and EfficientFormer by 3.5 and 3.6 mIoU, respectively. Two conclusions are offered: 1) EfficientMod design makes significant advancements, demonstrating the value and effectiveness of approach; Large receptive fields especially helpful for high-resolution input tasks like segmentation, and the vanilla attention block (which achieves global range) can be an off-the-shelf module for efficient networks. Please check Appendix Sec. F for the analysis of the improvement gap between MS COCO and ADE20K.
this regard, our convolutional EfficientMod-- performs 1.1 mloU
CONCLUSION
We present Efficient Modulation (EfficientMod), unified convolutional-basee building block that incorporates favorable properties from both convolution and attention mechanisms. EfficientMod simultaneously extracts the spatial context and projects input features, and then fuses them using a simple element-win wise multiplication. EfficientMod's elegant design gratifies efficiency, while the inherent design philosophy guarantees great representational ability. With EfficientMod, we built a series of efficient models. Extensive experiments examined the efficiency and effectiveness of our method, EfficientMod outperforms previous SOTA methods in terms of both empirical results and practical latency. When applied to dense prediction tasks, EfficientMod delivered impressive results. Comprchensive studies indicate that our method has great promise for efficient applications.
Limitations and Broader Impacts. The scalability of efficient designs is one intriguing but under- studied topic, like the huge latency gap in Table 6. Also, employing large kernel sizes or introducing attention blocks might not be the most efficient way to enlarge the receptive field. We have not yet observed any negative societal impacts from EfficientMod. Instead, we encourage study into reducing computations and simplifying real-world applications with limited computational resources.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: 06450, 2016.
Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-loca attention for vision transformers. ICIR, 2022a.
Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S-H Gary Chan. Run, don't walk: Chasing higher flops for faster neural networks. CVPR, 2023.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Nobile-former: Bridging mobilenet and transformer. In CVPR, 2022b.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using predictor pretraining. In CVPR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In CVPR, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In ICCV, 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu, Cmt: Convolutional neural networks meet vision transformers. In CVPR, 2022.
Meng-Hao Guo, Cheng-Ze Lu, Zheng- Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. Computational Visual Media, 9(4):733-752, 2023.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In CVPR, 2020a.
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing Xu, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. NeurlPS, 2020b.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NIPS 2014 Deep Learning Workshop, 2015.
Qibin Hou, Cheng-ZZ Ze Lu, Ming-Minn Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer- style convnet for visual recognition. arXiv preprint arXiv:2211. 19943, 2022.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 0777.044466. 2017.
Jie Hu, Li Shen, and Gang Sun. squeeze-and-excitaee networks. In CVPR, 2018.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
10

---

Published as a conference paper at ICLR 2024
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In CVPR, 2019.
Jiachen Li, Ali Hassani, Steven Walton, and Humphrey Shi. Convmlp: Hierarchical convolutional mlps for vision. In CVPR, 2023a.
Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang. and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. NeurIPS, 2022.
Yanyu Li, Ju Hu, Yang Wen, Georgios Evangelidis, Kamyar Salahi, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Rethinking vision transformers for mobilenet size and speed. ICCV, 2023b.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Lue Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2/04.05707, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Jihao Liu, Xin Huang, Guanglu Song, Hongsheng Li, and Yu Liu. Uninet: Unified architecture search with convolution, transformer, and mlp. In ECCV, 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2019.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In ECCV, 2018.
Muhammad Maaz, Abdelrahman Shaker, Hisham Cholakkal, Salman Khan, Syed Waqas Zamir, Rao Muhammaa Anwer, and Fahad Shahbaz Khan. Edgenext: efficiently amalgamated enn- transformer architecture for mobile vision applications. In ECCV, 2023.
general-purpose, Sachin Mehta and Mohammad Rastegari. Mobilevit: Light- general- purpose, and mobile- friendly vision transformer. ICLR, 2022.
Sachin Mehta and Mohammad Rastegari. Separable self-attention for mobile vision transformers. TMLR, 2023.
Junting Pan, Adrian Bulat, Fuwen Tan, Xiatian Zhu, Lukasz Dudziak, Hongsheng Li, Georgios Tzimiropoulos, and Brais Martinez. Edgevits: Competing light-weight enns on mobile devices with vision transformers. In ECCV, 2022.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. ICCV, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NewrIPS, 2021.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Mingxing Tan and Quoe Le. Efficientnet; Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In ICMI, 2021.
11

---

Published as a conference paper at ICLR 2024
Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Chao Xu, and Yunhe Wang. Ghostnetv2: Enhance cheap operation with long-range attention. NeurlPS, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021b.
Asher Trockman and J Zico Kolter. Patches are all you need? arXiv preprint arXiv:2201.099 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In ECCV, 2022.
Pavan Kumar Anasosalu Vasu, James Gabriel, Jeff Zhu, Oncel Tuzel, and Anurag Ranjan. Mobileone: An improved one millisecond mobile backbone. CVPR, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In CVPR, 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Wenhai Wang, Enze Xie, Xiang Li. Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved basclines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. NeurlPS 2021 Workshop on ImageNet: Past, Present, and Future, 2021.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019.
Jianwei Yang, Chunyuan Li, Xiyang Dai, and Jianfeng Gao. Focal modulation networks. NeurIPS, 2022.
Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In CVPR, 2022.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In CVPR, 2022a.
Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer basclines for vision. arXiv preprint 2Xx:22210.13552 2022b.
Haokui Zhang, Wenze Hu, and Xiaoyu Wang. Parc-net: Position aware circular convolution with merits from convnets and transformer. In ECCV, 2022.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In CVPR, 2018
Bolci Zhou, Aditya Khosla, Agata Lapedriza. Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2024
A CODES AND MODELS
The codes can be found in the supplemental material. We provide amonymous links to the pre-trained checkpoints and logs. The ReadME.nt md file contains thorough instructions conduct experiments. After the submission, we will make our codes and pre-trained checkpoints available.
B DETAILED CONFIGURATIONS
Stage
size
EfficientMod-xxs
EfficiientMddxs
EfficientMod-s
EfficiientModdS(Comm)
Stem
W
Conv(kerne stride=4)
Dim=32
Dim=32
Dim=32
Dim=40
Stage 1
H x W
Blocks = [2,0]
Blocks = [3,0]
Blocks = [4.
Blocks = [4,0] 0]
Down
x 8
W
Conv stride=2)
Dim=64 Blocks = [2,0]
Dim=64
Dim=64
Dim=80
Stage 2
w
8
Blocks = 3.0
Blocks = [4.0]
Blocks = [4.0]
Down
H x T6
W
Conv(kernel stride=2)
Stage 3
16
x
W
16
Blocks
Dim=128
Blocks = [4.3]
Dim=144
Blocks [8,4]
Dim=144
Blocks [12,0]
Dim=160
Down
Conv stride=2)
32
32
Dim=256 Blocks = =[2,2]
Dim=288
Dim=312
Dim=344
Stage 4
H
W
32
Blocks = [2.3]
Blocks = [8, 4]
Blocks = [8. 0]
Head
Global Average Pooling & MLP
Parameters (M)
4.7
6.6
12.9
12.9
Table 8: Detailed configuration of our EfficientMod architecture. Dim denotes the input channel number for each stage. Blocks [b,, by] indicates we use by EfficientMod blocks and by vanilla attention blocks, respectively. For EfficientMod block, alternately expand the dimension by factor of 1 and 6 (1 and 4 for EfficientMod-xs)) For the vanilla attention block, we consider heads by default.
Detailed Framework Configurations We pro- vide a detailed configuration of our Efficient- Mod in Table 8. Our EfficientMod is hicrar- chical architecture that progressively downsizes the input resolution by 4, 2, 2, 2 using the tradi- tional convolutional layer. For the stages that include both EfficientMod and attention blocks, we employ EfficientMod blocks first, then uti- lize the attention blocks. By varying channel and block numbers, we introduce EfficientMod- xxs, EfficientMMd-xs, EfficientMod-s, and pure convolutional version of EfficientMod-s.
Hyper parameters
EfficientMod
Batch size
256x8 = 2048
Optimizer
AdamW
Weight decay
0.05
Clip-grad
None
I.R scheduler
Cosine
Learning rate Epochs
4e-3
300
Warmup epochs Hflip
5
0.5
Vilip
0.
Color-jitter
0.4
AutoAugment
rand-m---mstdo.5-inn
Aug-repeats
0
Detailed Ablation Configurations For the isotropic designs in Table 6, we patchify the in- put image using 14x 14 patch size, bringing a resolution of 16 x 16. We adjust the depth and width to deliberately match the number of pa- rameters as EfficientMod-xx and EfficientMod-s We also vary the expansion ratio to match the number of parameters and FLOPs for MBConv and EfficientMod counterparts. We chose 256 and 196 for the channel number and 13 and 11 for the depth, respectively. Similar to EfficientMod-s and EfficientMod- the generated models will have 12.5M and 6.4M parameters, respectively. By doing so, guarantee that any performance differences result purely from the design of the MBCony and EfficientMod blocks. For hierarchical networks, we replace the EfficientMod block with the MBConv block and vary the expansion ratio the match parameter number.
Random erasing prob
0,25
Mixup
0.8
Cutmix
1.0
Label smoothing
0.1
Layer Scale
1c-4
Drop path
(0., 0., 0.02) O.
Drop block
Figure 5: Training hyper-parameters.
13

---

Published as a conference paper at ICLR 2024
(expand by factor of 6)
x
ctx
out
ctx (corresponding above v)
(expand by a factor of
ctx
out
ctx (corresponding above v)
Figure 6: Visualization of each expanded and associated modulation result ctx in more detail. We provide the final output and the context modeling result (ctr) for reference.
1.8 ms (25%) Gaster
& top-1 better
Top-1 accuracy (%6)
[MMicienFFommer22
[-ppj]
EfficienFoormer2
FasterNet
FasterNet
75
MobIleVIT
Mobile VIT
ViT22
Mobile vITv2
EffcientMod
EfficientMod
73
73
2.5
3.5
4.5
5.5
6.5
7.5
15
35 ONNX CPU Latency (ms)
55
ONNX GPU Latency (ms)
Figure 7: Comparison of the Lateney- trade-off between EfficientMod and other methods on GPU and CPU devices. EfficientMod consistently performs far better than other methods.
Training details The detailed training hyper parameters are presented Table 5.
C VISUALIZATION OF MODULATION
We provide complete visualization of modulation design, seen in Fig.6. We showcase the project input v, the context modeling output cla, and the corresponding modulation outcomes ctr u.ehe block's output presented last. The visualization implementation is the same as the settings in Section 4.2. We divide the feature into chunks along the channel dimension when the input feature expanded by factor (for example, in Fig. 6), and we next visualize each chunk and its accompanying modulation result. Interestingly, as the channel count increases, different thunks show generally similar results v. The difference substantially accentuated after being modulated by the same context, indicating the success of the modulation mechanism.
D LATENCY COMPARISON
We also demonstrate the accuracy versus GPU and CPU latency for various methods, as seen in Fig. 7. We eliminate specifie models that run significantly slower (such Edge VIT GPU) or yield
14

---

Published as a conference paper at ICLR 2024
much lower accuracy (such as MobileNettV)) for better visualization. Our approach consistently surpasses related work, especially on GPU, by a clear margin. Our EfficientMod outperforms MobileViTvv2 by 2.8 top-1 accuracy ImageNet with the same GPU latency. We are 25% faster than the previous state-of-the-art approach FffcientFormery2 to obtain results that are similar or slightly better (0.3%). ). On the CPU, also achieve promising latency-accuracy trade-off, demonstrating that our EfficientMod can be utilized as a general method on different devices.
E DISTILLATION IMPROVEMENTS
We provide detailed experiments for distillation on each model. Our teacher model is the widely used RegNetY-160 (Radosavovic et al., 2020), the same as the teacher in Ef- ficientFormer22 for fair comparison. Though other models may offer su- perior improvements, a better teacher model is not the primary objective of this study. The right table shows that knowledge distillation a potent way to improve our method's 's top-1 accuracy about % without introducing any computational overhead during inference.
Model
Param
FLOPs
Distill.
Top-1
76.0
EfficientMod-a xXs
4.7M
0.6G
77.1 (11.1)
78.3
EfficientMod-x -s
6.6M
0.8G
79.4 (1)
81.0
EfficientMod-s
12.9M
1.4G
81.9 (10.9)
EfficientMod-s (Conv)
80.5
12.9M
1.5G
81.5 (t
Table 9: Detailed improvements from Distillation.
F ANALYSIS ON IMPROVEMENT GAP BETWEEN OBJECT DETECTION AND SEMANTIC SEGMENTATION
Why does ModelMod improve significuntly on ADE20K but only modestly on MS COCO? Besides the differences in datasets and evaluation metries. we attribute this discrepancy the number of parameters in detection or segmentation head. Notice that EfficientMod only introduces 12M parameters. When equipped with Mask RCNN, the additional parameters are over 20M ( (ver 60% in total), dominating the final detection network. Hence, the impact of the backbone is largely inhibited. On the contrary, Semantic FPN only introduces 4-5M parameters for semantic segmentation on ADE20K (about 25% in total). Hence, EfficientMod'ss capabilities are fully utilized.
G SCALABILITY OF EFFICIENTMOD
Latency against input resolution. We first validate our scalability for the input resolution. vary the input resolution from 224 to 512 by step size of 32. We compare our convolutional and hybrid vari- ants of EfficientMod with some strong baselines, including FobileFormer (Chen et al., 2022b), Mo- ViTe (Mehta & Rastegari, 2023), and Efficient- FormerV2 (Li et al., 2023b). For a fair comparison, consider model size in the range of 10-15M parameters for all models. From Fig, 8, we can ob- serve that our method and EfficientFormerV2 show promising scalability to the input resolution when
(am) gggg caaair mn XNNO
Mobild Fommer
MobiieViTV2
PfficerMMal-Coo
EfficierMMal
Inpui Resolution
compared with MobileFormer and MobileViTv2. Figure 8: Impact of input size GPU latency. Compared to EfficientFormer22 convolutional and hybrid) also exhibits even lower latency when the resolution small.
method (both
Compare with MBConv. We also investigate the scalability of the width and kernel size for our Effi- cientMod. We compare EfficientMod and the commonly used MBCony from MobileNetV2 (Sandler al., 2018) using the same settings described in Sec. B. To match the computational complexity and parameter number, the expansion ratio is set to and 7 for EfficientMod and MBCony, respectively. As shown in Fig. 9, our EfficientMod consistently runs faster than MBConv block regardless of width
15

---

Published as a conference paper at ICLR 2024
12
ElTicientMod
Expon. (EfficiemMod)
Expon. (MBComv)
EfficiemModd
MBCony
ONNX GPU Lateney (ms)
ONNX GPU Lateney (199)
Expon. (EfficienMMd)
(MBCony)
64
96
128
260
192
224
256
268
320
352
354
Channel Number
Kernel Size
Figure 9: Scalability comparison between our EfficientMod and MBConv blocks by varying the width and kernel size. We use dotted line show the latency tendency. Model architecture is inherited from Table isotropic design.
Table 10: Latency benchmark results across multiple GPU instances. We conducted three tests for each GPU on different nodes and averaged the results from 500 runs for each test. We report the sean.se latency in the table. The setting is the same in Table 1.
Model
Top- ((9)
P100
T4
V100-SXM2
A100
Param.
FLOPs
MobileNetV2x1.0
71.8
2.2.0.00
1.400.000
1.100.00
1.3,0,00
3.5M
0.3G
PasterNet-T00
71.9
2.5.00000
1.8.00m
1.7,001s
2.0.0061
3.9M
0.3G
EdgeViT-XXSS
74.4
8,8.0.000
4.7 4.7.0.23
2.4x0000
2.7.0.000
4.1M
0.6G
MobileViT-XS
74.8
4.2.0.0
3.520m0
2.3
2,4
2.3M
1.1G
Ellicien(FormerV2-500
73.7
3.3.0.000
2. 140.000
2.0.00000
2.4.0.000
3.6M
0.4G
EfficientMMo-xxs
76.0
3,0.0.000
2.0.0.00
1.9.0m0
2.2.0.000
4.7M
0.6G
MobileNetV2x1.4
74.7
2.8.0.000
2.0.0.0
1.2.0000
1.4.0,000
6.1M
0.6G
DeiT-T
74.5
2.7.0.00
20000
1.8
2.2.0.000
5.9M
1.2G
FasterNet-Ti
76.2
2.0.
2.340013
1.9.00m
2.140.100
7.6M
0,9G
EfficientNet-BO
77.1
3.4.0.000
2. 2.6.00003
2.0.0000
2.3.0.000
5.3M
0.4G
MobileOne-S2
(77.4)
2.0.0.000
1.720mm
1.1 20000
1.6.0:000
7.8M
1.3G
EdgeViT-XS
77.5
12.1.0.0
5.9.0.43
2.3.00000
2.6.0,000
6.8M
1.IG
MobileViTvv-11.0
78.1
5.4
4.5.0m
2.9.00e
3.0.0003
4.9M
1.8G
EllicienfForneeV2-5S1
77.9
4.5 .0,000
2.8.0.000
2.5.0000
3.0.0.000
6.2M
EfficientMod xs
78.3
3.6.0.000
2.5.000
2.3
2.6,0000
6.6M
0.8G
PoolFormer-s12
77.2
4.9 1..0000
3,9.0.10
2.4g000
2.2.0000
11.9M
1.8G
FasterNet-T2
78.9
4.3.0.04
3.5.0.000
2.5.0010
2.940013
15.0M
1.9G
EfticienntFomer-
79.2
3.7.0.000
2.8.0.000
1.7.000
1.9.0,003
12.3M
1.3G
MobilcFoomee-508M
79.3
13.6.0000
11.5
7.6
7.8.0005
14.8M
0.6G
MobileOne-S4
(79.4)
4.7.0.003
3.6.0.000
2.3.0.000
2.7.0.000
14.8M
3.0G
Mobile
80.4
7.2
7.0.00s3
3.8
3.3
10.6M
4.1G
ViT-S
81.0
20.7 20.70
10.0.0043
3.800
4.2,0.000
13.1M
1.9G
EffcienfFormer22-5
80.4
20.00000
4.6.0m0
3.8
4.5.0.00
12.7M
1.3G
EfficientMod-s
81.0
5.5 5.5.0.000
3.9.0.000
3.3.0000
3.8.0.010
12.9M
1.4G
or kernel size. When increasing the width kernel size, the latency tendency our EfficientMod is much smoother than MBConv's, suggesting EfficientMod has great potential to be generalized to larger models.
H BENCHMARK RESULTS ON MORE GPUs
In addition to the results the P100 GPU presented in Table 1, also conducted latency benchmarks on several other GPU instances, including the T4, V100 -XMM, and 11000SXM44400E We observed that there might be some variances even when the GPU types are the same. Therefore, we randomly allocated GPUs from our server and conducted three separate benchmark tests. Table 10 reports the mean and standard deviation values.
As shown in the table, our EfficientMod consistently performs fast on different GPU de- vices. An interesting finding is that the results A100 have higher latency than V100.
16

---

Published as a conference paper at ICLR 2024
Table 1I: Latency benchmark for object detection, Instance segmentation, and semantic segmentation.
MS
COCO
ADE20K
Arch.
Backbone
Params
Aphlx
A.
Latencyansenu
Latencyozsionn
Params
mIoU
Latencyssn-sin
Cony.
ResNet-18
31.2M
34.0
31.2
15.8ms
19.2ms
15.5M
32.9
8.5ms
Pool
P12
31.6M
37.3
34.6
30.7mu
66.9ms
15.7M
37.2
15.7mms
Conv.
32.6M
42.1
38.5
28.3ms
33.6ms
16.7M
43.5
17.3ms
Arten.
PVI-Timy
32.9
36.7
35.
19.6ms
37.0ms
17.0M
35.7
II 8ms
Hybrid
Efficient--LI
31.5M
37.9
35.4
20.6ms
29.9m 99.9ms
15.6M
38.9
12.3ms
Hybrid
PVTv2-BI
33.7M
41.8
38.8
24.4ms
41,7ms
17.8M
42.5
Hybrid
EfficientF.v2o6
32.2M
43.4
39.5
47.9ms
53.0ms
16.3M
42.4
26.5ms
Hybrid
EfacieniMod-s
32.6M
43.6
40.3
29.7ms
48.7ms
16.7M
46.0
17.9ms
Firstly, we show that this is common phe- nomenon for almost all models, as we can see in the table. Secondly, we observed consistently low GPU utilization for A100, consistently below 40%, indicating that 1100's strong performance is not being fully hamnessed. Thirdly, we evaluated batch size to simulate real-world scenarios. When scaling up the batch size, GPU utilization increased, resulting lower latency for A A100, as depicted above (we toke EfficientMod-S as an example). Lastly, we highlight that latency could be influenced by intricate factors that are challenging to debug, including GPU architectures, GPU core numbers, CUDA versions, Operating Systems, etc.
Batch Size
2
4
16
32
64
V100-SXM2
3.3
3.9
5.3
8.0
13.7
25.4
48.2
A100-SXM4
3.7
3,9
4.2
5.7
8.9
14.6
27.2
I LATENCY BENCHMARK ON DOWNSTREAM TASKS
Besides the study on the scalability of EfficientMod in Sec. G, also explore the latency real- world downstream tasks. We directly benchmark methods in Table 7 on one A100 GPU (without converting to ONNX format) and report the latency in Table 11.
Clearly, our EfficientMod also exhibits promising efficiency on these tasks. An intriguing observation that PoolFormer-S12 exhibits the highest latency. This is particularly interesting, considering that the core operation within the network is the pooling operation. We consider two factors could be contributing this phenomenon: 1) the PoolFormer network architecture might be optimized for efficiency. 2) pooling operations might not be as highly optimized in CUDA as convolutions (a phenomenon we' also noticed our backbone design). Additionally, we have observed that as the input resolution increases, the latency gap between Hybrid EffficiettMod-s and Conv EfficientMMdds widens. This attributed to the computational complexity introduced by the Attention Mechanism in hybrid version. One potential remedy is to reduce computations by downsizing the resolution for the attention block, similar to the approach employed in EfficientFormerV2. However, our Hybrid EfficientMod-s maintains competitive and promising latency results compared to methods like EfficieettFomeer22-52 and other alternatives.
J OPTIMIZATION FOR MOBILE DEVICE
As presented in previous results, our EfficientMod mainly focuses on GPU and CPU devices. Next, we explore the optimization for mobile devices. We convert our PyTorch model to Core ML model using coremltools! We then make use of the iOS application from MobileOne (Chen et al., 2022b) and benchmark latency on an iPhone 13 (iOS version 16.6.1).
We take a pure convolution based EfficientMod--x xxx (without attention module, which achieves 75.3% top-1 accuracy) and compare it with other networks in Table 12. Based on our observation that permute operation is exceptionally time-consuming in CoreMI. models, we replace the permute linear layer with a convolutional layer, which is mathematically equal. Our model is able to achieve 75.3% top-1 accuracy at 1.2ms latency on iPhone 13. Inspired by the analysis of normalization layers in EfficientFormer (Li et al., 2022), further remove all Layer Normalization and add Batch Normalization layer after each convolutional layer (which can be automatically fused during inference) and re-train the model. By doing so, we reduce the latency to 0.9 ms and achieve a 74.7% 7% top-1 accuracy, which already achieves a promising result. We further slightly adjust the block and
17

---

Published as a conference paper at ICLR 2024
Model
Top-1
iPhone Latency
Params
FLOPs
MobileNetV2x1..0
71.8
0.9ms
3.5M
0.3G
FasterNet-TO
71.9
0.7ms
3.9M
0.3G
ViT-XXXS
74.4
1.8ms
4.1M
0.6G
MobilcOne
74.6
0.8ms
4.8M
0.8G
MobileViT-XS
74.8
25. 25.8ms
2.3M
1.1G
EfficienFFormer22--00
73.7
0.9ms
3.6M
0.4G
MobileNetV2x1.44
74.7
1.Ims
6.1M
0.6G
DeiT-Tiny
74.5
1.7ms
5.9M
1.2G
EfficieetMMod-xxs(conn))
75.3
1.2ms
4.4M
0.7G
EfficientMood-xxs(coon))
74.7
0.9ms
4.4M
0.7G
EfficientMModxxxx(conny
75.2
1.0ms
4.8M
0.6G
Table 12:
We benchmark latency on iPhone 13 explore the optimization on mobile devices. Note: results with strong training tricks (e.g., re- p-raameterization and distillation) are ignored for fair comparison. "*" means we remove LN and add a BN after each convolutional layer. indicates that we slightly adjust the channel and block number for better accuracy.
channel numbers and get 75.2% accuracy at 1.0ms. The strong performance indicates that our proposed building block also performs gratifyingly mobile devices.
K TENTATIVE EXPLANATION TOWARDS THE SUPERIORITY OF MODULATION MECHANISM FOR EFFICIENT NETWORKS
It has been demonstrated that modulation mechanism can enhance performance with almost no additional overhead in works such as Yang et al. (2022); Guo et al. (2023) and in Table 5. However, the reason hidden behind is not fully explored. Here, tentatively explain the superiority of modulation mechanism, and show that modulation mechanism is especially well suited for efficient networks.
Recall the abstracted formula of modulation mechanism in Eq. that y p (ctx (z) (z)). it can be simply rewritten as y f(x²), where f(x²) ctx (a:) @ (ar) and ignore p (-) since it is
learnable linear projection. Hence, we can recursively give the output of 1-th layer modulation block with residual by:
31 3o f1(1)),
(6)
42 = x1 f2 {2(x)),
(7)
= 2n + {1 + f2(20) 2f2 x * (xx2)) + (fi
(8)
क = 0igi (x%) a (2)) agga(za) + (2
(9)
where / indexes the layer, aj the weight for each item, ai indicates the combined function for 1-th item, and we do not place emphasis on the details of gi- With only a few blocks, we can easily project the input space. For instance, with only 10 modulation blocks, we will get dimeenimall feature space. Hence, we can conclude that i) modulation mechanism is able reduce the requirement of channel number since it naturally project input feature to very high dimension in distinct way; ii) modulation mechanism does not require a very deep network since several blocks are able to achieve high dimensional space. However, in the case of large models, the substantial width and depth of these models largely offset the benefits of modulation. Hence, we emphasize that the abstracted modulation mechanism is particularly suitable for the design of efficient networks.
very high dimensional feature space, even infinite-dimensional
Notice that the tentative explanation presented above does not amount to a highly formalized proof. Our future effort will center comprehensive and in-depth investigation.
18

---

Published as a conference paper at ICLR 2024
L DETAILED ANALYSIS OF EACH DESIGN
Efficiency of Slimming Modulation Design We have integrated an additional MLP layer into the EfficientMod block validate the efficiency of slimming modulation design. This modification was aimed at assessing the impact of slimming on both performance and computational efficiency, Remarkably, this resulted in a notable reduction in both GPU and CPU latency, with negligible impact on accuracy. This underlines the effectiveness of our slimming approach in enhancing model efficiency without compromising accuracy.
Method
Param.
FLOPss
Top-1
GPU Latency
CPU Latency
Efficienttodd--Comy (sperate MLP)
12.9M
1.5G
80.6
6.2
26.2
EfficientModss-Conv
12.9M
1.5G
80.5
5.8 ms
25.0 ms
To further validate the efficiency of our approach in simplifying context modeling, compared our single kernel size (7x7) implementation against mul- tiple convolutional layers with varying kernel sizes, as the implementation of FocalNet. Experiments conducted based EfficcientModssCCom variant. Our findings reinforce the superiority of using single, optimized kernel size. This strategy not only simplifies the model but also achieves better accuracy-latency trade-off, demonstrating the practicality and effectiveness of design choice.
Efficiency of simplifying Context Modeling
Kernel Sizes
Param.
FLOPs
Top-1
GPU Latency
CPU Latency
[3, 3]
12.7M
1.4G
79.7
5.8 ms
28.5 ms
[3, 5]
12.8M
1.5G
80.1
6.1 ms
29.0 ms
[3, 7]
12.9M
1.5G
80.2
6.4 ms
29.7 ms
5]
12.9M
1.5G
80.2
6.3 ms
29.2 ms
[]
13.0M
1.5G
80.3
6.6 ms
29.8 ms
[3,5,]] 13.1M
1.5G
80.5
7.2 ms
32.4 ms
[7]
12.9M
1.5G
80.5
5,8 ms
25.0 ms
Integrating Attention in EfficientMod The introduction of vanilla attention in the last two stages of EfficientMod aimed to improve global representation. We adjusted the block and channel numbers to ensure the parameter count remained comparable between EfficientMod-s-COnn and EfficientMod- s. The results highlight that EfficientMod-s not only shows improved performance but also reduced latency, thereby validating approach in integrating attention for enhanced efficiency.
Method
Param.
FLOPs
Top-1
GPU Latency
CPU Latency
EfficientMod-ssConv
12.9M
1.5G
80.5
5.8 ms
25.0 ms
EfficientMod-s
12.9M
1.4G
81.0
5.5 ms
23.5 ms
As shown above, the additional experiments and analyses affirm the distinct contributions and efficacy of each design element in our model, suggesting our EfficientMod can achieve a promising latency-accurage trade-off.
19


Reviews:
[{"review_id": "4zya1XQK6o", "reviewer": "Reviewer_xsDm", "summary": "This paper introduced a new model structure based on previous modulation designs to further improve the efficiency (especially inference latency) and performance. The paper revisited previous modulation designs and improved the efficiency by reducing fragmented operations and simplifying the structure. The proposed method shows better performance than previous efficient networks on ImageNet with lower latency. The improvements also transfer to detection and segmentation.", "questions": "Please see the weakness part", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThe paper had a clear introduction to previous works and how is the proposed method motivated from these works. This makes it easier to follow the work and understand how it works.\n2.\tThere are extensive experiments on multiple tasks. And the proposed method achieves better performance and latency than previous efficient models.", "weaknesses": "1.\tThere are limited technical contributions in the work. This paper focuses on improving the latency of previous works. The improvements/changes from previous works are mainly engineering designs, for example, fuse multiple FC layers together, fuse multiple DWConv into a larger one, replace reshape operation with repeat. The guidance is mainly from previous works such as ShuffleNet v2, which is to reduce fragmented operations for improved latency. There are limited new insights.\n2.\tIt is not clear how much efficiency improvement does each design contribute. I suggest the author to conduct a thorough ablation study to show the impact of each structure change, and explain why it could achieve improvement.\n3.\tFig 1 is good to illustrate the difference between the proposed method and previous works. But it could be better to expand Fig 1 (c) in details when explaining the method. This makes it easier to understand the proposed structure and details.\n4.\tIn Table 1, why VAN and FocalNet results are not included? They seem to be the most relevant works.\n5.\tIn Table 2, why adding Attention even reduced the FLOPs?\n6.\tIn Table 1, are the GPU and CPU latency of different models measured on the same device?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okwV8G9sDS", "reviewer": "Reviewer_fvW3", "summary": "This paper proposes an efficient modulation block to build efficient vision networks. The proposed EfficientMod block mainly consists of a simple context modeling design (CTX) with FC and Conv layers. Extensive experiments on image classification, object detection, and semantic segmentation demonstrate that the proposed method achieves strong performance compared with prior methods.", "questions": "See the weakness part", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed model is simple yet effective.\n2.\tThe proposed model shows strong performance on several benchmarks, including ImageNet, COCO, and ADE20K.", "weaknesses": "1.\tIn Table 2, there are no latency reported for state-of-the-art efficient models.\n2.\tThe proposed method seems simple and more analysis and motivations for the design are needed to understand the principal of the design choice.\n3.\tImportant baselines such as ConvNeXt and Swin Transformer are not included in the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Io9VBb5CpQ", "reviewer": "Reviewer_Q55t", "summary": "The authors propose a unified convolutional-based building block, EfficientMod, which incorporates favorable properties from both convolution and attention mechanisms. Comparing the prior work shown in Figure 1(b), EfficientMod firstly fuses the FC layers from the MLP and the modulation block to achieve better efficiency, resulting in a unified block. Secondly, EfficientMod includes simplified context modeling, which employs one large-kernel depth-wise convolution layer between two linear projection layers. Extensive experiments and comparisons demonstrate the effectiveness of the proposed method across a range of different tasks (classification, detection, and segmentation).", "questions": "My main concern lies with points 1 and 2 in the weaknesses. I look forward to the authors’ response.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed method, EfficientMod, is remarkably simple, which could exert a significant influence when deploying a deep model on a resource-limited device.\n- The experimental results clearly showcase the effectiveness of EfficientMod in outperforming existing state-of-the-art methods across various tasks (classification, detection, and segmentation).", "weaknesses": "- Examining Figure 1, and comparing (b) and (c), the proposed EfficientMod block fuses the MLP on the top and the modulation block as one unified block to improve efficiency. It is conceivable that this might limit performance when using the same number of parameters. The authors should elucidate the principles behind this design, not only from the perspective of efficiency but also in terms of representational ability.\n-  Building on the first point, it is imperative to present a comparison between (b) and (c) with the same number of parameters, both in terms of performance and efficiency, and under the same training settings. For example, a comparison between (b), only with fused MLP, and (c).\n- Could the authors discuss whether the transformer block can also benefit from the proposed method for efficiency (at least it is feasible to fuse the MLP into one unified block)?\n- In Figure 8 and Figure 9, one of the notations in the legend should be 'EfficientMod'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2mZjNUj7w1", "reviewer": "Reviewer_6eCe", "summary": "UPDATE: the rebuttal has answered many of the issues and I have reflected this in the score.\n\n\nThis paper describes an optimized deep learning architecture for vision tasks. It is related to a line of work that utilizes mixtures of transformers and CNNs or adds modulation to CNNs in order to come up with an architecture with high accuracy and low computational complexity and low latency.", "questions": "Q: Is Figure 1a) missing the softmax part of the transformer architecture? I think this is one of the key differences to the modulation designs.\n\nQ: In section 3.1, the FC layer seems the same as a 1x1 pointwise convolutional layer. If this is correct, it might be beneficial to mention this.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S: One strength of the paper is the relative simplicity of the architecture compared to some related works.\n\nS: Another strength seems to be good empirical results on selected vision tasks.\n\nS: Even though on the surface the work seems to be incremental improvement over VAN and FocalNets, this paper generalizes them nicely and provides a simpler alternative, which also seems to perform better. Also, any improvement in efficient models for vision tasks is naturally important.", "weaknesses": "W: The paper is mainly constructive and experimental in nature. In the appendix there is a tentative explanation that describes that the modulation might lose effectiveness in larger networks. Expanding on this and at the same time showing this in larger networks would make the contribution stronger.\n\nW: One of the most curious things about the paper is the ablation results in Table 5. From there it seems that replacing the modulation with just a regular residual connection (sum) has quite modest performance drop (abs perf drop 1%). Without the multiplication, unless I am mistaken, the architecture reduces to a ResNet with specific hyperparameters and two-path construction. Could the authors discuss this. I think both contributions are valuable, but I am wondering whether it is correct to attribute the performance of the architecture to the modulation since other aspects seem to play even larger role than abs. perf. of 1%. \n\nW: Continuation of the above (without mult). Would the architecture without mult be second best of the architectures compared in Table 2. If so, please add it there.\n\nW: Without mult cont: What would be the performance of the ResNet (no mult) version with the VIT-style attention layers on top?\n\nW: Since this work is most closely related to VAN and FocalNets those works should be at least briefly mentioned in the related work section as well. \n\nW: In Fig 1 and corresponding text, would it make sense to mention where nonlinearities are applied?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)