Focal Modulation Networks
Authors: Jianwei Yang
Chunyuan Li
Xiyang Dai
Jianfeng Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Focal Modulation
Self-Attention
Convolution
Visual Backbone
Image Classification
Object Detection
Image Segmentation

Abstract: We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by a focal modulation module for modeling token interactions in vision. Focal modulation comprises three components: $(i)$ hierarchical contextualization, implemented using a stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, $(ii)$ gated aggregation to selectively gather contexts for each query token based on its content, and $(iii)$ element-wise modulation or affine transformation to fuse the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with similar computational cost on the tasks of image classification, object detection, and semantic segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-1K. After pretrained on ImageNet-22K, it attains 86.5% and 87.3% top-1 accuracy when finetuned with resolution 224$^2$ and 384$^2$, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R-CNN, FocalNet base trained with 1$\times$ outperforms the Swin counterpart by 2.1 points and already surpasses Swin trained with 3$\times$ schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. These results render focal modulation a favorable alternative to SA for effective and efficient visual modeling. Code is available at: https://github.com/microsoft/FocalNet.

Focal Modulation Networks
Jianwei Yang, Chunyuan Li, Xiyang Dai, Jianfeng Gao
(jianwyan, cchunyllxide (fgga)mmiiccr www.orooottttomommmmmmmmmmmoo..e com
Abstract
We propose focal modulation networks (FocalNets in short), where self-attention (SA) is completely replaced by focal modulation module for modeling token interactions in vision. Focal modulation comprises three components: (i) hierar- chical contextualization, implemented using stack of depth-wise convolutional layers, to encode visual contexts from short to long ranges, (di) gated aggrega- tion to selectively gather contexts for each query token based its content, and (tix) element-wise modulation or affine transformation to fuse the aggregated context into the query. Extensive experiments show FocalNets outperform the state-of-the-art SA counterparts (e.g., Swin and Focal Transformers) with simi- lar computational cost on the tasks of image classification, object detection, and semantic segmentation. Specifically, FocalNets with tiny and base size achieve 82.3% and 83.9% top-1 accuracy on ImageNet-IK.. After pretrained on ImageNet- 22K, it attains 86.5% and 87.3% top-1 1 accuracy when finetuned with resolution 224 and 3842, respectively. When transferred to downstream tasks, FocalNets exhibit clear superiority. For object detection with Mask R- R-CNN, FocalNet base trained with 1 outperforms the Swin counterpart by 2.1 points and already sur- passes Swin trained with 3 schedule (49.0 v.s. 48.5). For semantic segmentation with UPerNet, FocalNet base at single-scale outperforms Swin by 2.4, and beats Swin at multi-scale (50.5 v.s. 49.7). Using large FocalNet and mask2former, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation. These results render focal modulation favorable alter- native to SA for effective and efficient visual modeling. Code is available at: bittp://githue htt.eh/w...co.o.ccoooo////aaalaa
Introduction
Transformers 66
originally proposed for natural language processing (NLP), have become
prevalent architecture in computer vision since the seminal work of Vision Transformer (VT)) [9], Its
promise has been demonstrated various vision tasks including image classification 63
69 74 46
87 65], object detection [3, 97 93 [5] segmentation (67 72 [3 and beyond [38 91
68 36
In Transformers, the self-attention (SA) is arguably the key to its success which enables input- dependent global interactions, in contrast to convolution operation which constrains interactions in local region with a shared kernel. Despite this advantages, the efficiency of SA has been concern due to its quadratic complexity over the number of visual tokens, especially for high-resolution inputs. To address this, many works have proposed SA variants through token coarsening [69], window
attention [46] 65, 87]. dynamic token selection [5T 8L 50), or the hybrid [79] [4]. Meanwhile,
number of models have been proposed by augmenting SA with (depth-wise) convolutions to capture
long-range dependencies with good awareness of local structures [74] 22
78
20
18
35 [7).
In this work, we aim at answering the fundamental question: Is there better way than (hybrid) SA to model depe-deee long-range interactions? We start with an analysis on the current advanced designs for SA. In Fig. [a), we show window-wist attention between the red query token and With simple window-shift
the surrounding orange tokens proposed in Swin Transformer 46
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

[a] Window wise SA
(b) Focal Attention
(c) Focal Modulation
Figure 1: Illustrative comparison among (a) Window-wise Self-Attention (SA) [66 (b) Focal Attention (FA) 77 and (c) the proposed Focal Modulation. Given the query token window-wise SA captures spatial context from its surrounding tokens FA additionally uses far-away summarized tokens and Focal Modulation first encodes spatial context at different levels of granularity into summarized tokens (
which are
then adaptively fused into the query token depending the query content. Green and purple arrows represent the attention interactions and query-dependent aggregations, respectively (we do not draw all arrows clarity). Both window-win wise self-attention and focal attention involve heavy interaction and aggregation operations, while our focal modulation turn both of them light-weight. Figures better viewed in color.
strategy, Swin attains superior performance ResNets across various vision tasks. To enlarge the receptive field, focal attention [79] is proposed to additionally aggregate summarized visual tokens far away capture coarse coarse-grained, long-range visual dependencies, as shown in Fig. [b). To produce the outputs, both methods involve heavy interactions (green arrows) followed by equally heavy aggregations (purple arrows) between the query and a large number of spatially distributed tokens (context features), which are extracted via either window partition or unfolding. In this work, we take an alternative way by first aggregating contexts around each query and then modulating the query with the aggregated context. This alteration still enables input-dependent token interaction, but significantly eases the process by decoupling the aggregation from individual queries, hence making the interactions light-weight upon couple of features. As shown in Fig. [c), we can simply apply query-agnostic aggregations (e.g.. depth-wise hrpth-wise convolution) to generate summarized tokens at different levels of granularity. Afterwards, these summarized contexts are selectively aggregated depending on the query content, and finally fused into the query vector. We call this new token interaction mechanism focal modulation, with which we replace SA in Transformers to build a simpler and attention-free architecture, called Focal Modulation Network, or FocalNer in short.
Finally, extensive experiments on image classification, object detection and segmentation, show that our FocalNets consistently and significantly outperform the SoTA SA counterparts with comparable costs. Notably, our FocalNet achieves 82.3% and 83.9% top-1 accuracy using tiny and base model size, but with comparable and doubled throughput than Swin and Focal Transformer. respectively. When pretrained on ImageNet-22K., our FocalNets achieve 86.5% and 87.3% in 224² and 384² resolution, respectively, which are comparable or better than Swin at similar cost. The advantage is particularly significant when transferred to dense prediction tasks. For object detection on COCO 42) our FocalNets with tiny and base model size achieve 46.1 and 49.0 box mAp on Mask R-CNN 1x, surpassing Swin with 3x schedule (46.0 and 48.5 box mAP) For semantic segmentation on ADE20k (95), our FocalNet with base model size achieves 50.5 mIoU at single-scale evaluation, outperforming Swin at multi-scale evaluation (49.7 mloU). Using the pretrained large FocalNet, we achieve 58.5 mIoU for ADE20K semantic segmentation, and 57.9 PQ for COCO Panoptic Segmentation based on Mask2former [12]. Furthermore, we apply our focal modulation to monolithic ViT and clearly demonstrate superior performance across different model sizes.
2 Related Work
Self-attentions. Transformer [66] is first introduced to vision in Vision Transformer (ViT) [19 by
splitting an image into a sequence of visual tokens. The self-attention (SA) strategy in ViTs has
demonstrated superior performance to modern convolutional neural networks (ConvNets) such as
ResNet (27] when trained with optimized recipes [19 63). Afterwards, multi-scale architectures [5] 69 78), light- convolution layers (74 22 39, local self-attention mechanisms 46 87
[4] 79
and learnable attention weights (84) have been proposed to boost the performance and support high-resolution input. More comprehensive surveys are covered in R4
24 34 Our focal modulation
significantly differs from SA by first aggregating the contexts from different levels of granularity and then modulating individual query tokens, rendering attention-fre -rre mechanism for token

---

interactions. For context aggregation, our method is inspired by focal attention proposed in [79]. However, the context aggregation for focal modulation performed each query location instead of target location, followed by modulation rather than an attention. These differences in mechanism lead to significant improvement of efficiency and performance as well. Another closely related work is Poolformer [83] which uses a pooling to summarize the local context and a simple subtraction to adjust the individual inputs. Though achieving decent efficiency. Poolformer lags behind popular vision transformers like Swin performance. As we will show later, capturing local structures at different levels essential for performance.
MLP architectures. Visual MLPs can be categorized into two groups: (i) Global-mixing MLPs, such as MLP-Mixer [60] and ResMLP [62]. perform global communication among visual tokens through spatial-wist wise projections augmented by various techniques, such as gating, routing, and Fourier transforms
59 (is) Local-mixing MLPs sample nearby tokens for interactions, using spatial shifting, permutation, and pseudo-kernel mixing [82] 22,
41 g 23 Recently, Mix- Shift-MLPP [92] exploits both local and global interactions with MIPs, in a similar spirit of focal attention 799 Both MLP architectures and our focal modulation network are attention-free. However, focal modulation with multi-level context aggregation naturally captures the structures in both short- and long-range, and thus achieves much better accuracy accuracy-efficincy trade-off.
Convolutions. ConvNets have been the primary driver of the renaissance of deep neural networks in computer vision. The field has evolved rapidly since the emerge of VGG [52]. InceptionNet [56]
and ResNet [27 Representative works that focus on the efficiency of ConvNets are MobileNet 30).
ShuffleNet 90) and EfficientNet [7)] Another line of works aimed at integrating global context to
compensate Cenv such as SE-Net 2 Non-local Network (71), GCNet (2), LR-Net [31 and
C3Net 80 etc. Introducing dynamic operation is another way to augment ConvNets as demonstrated
in Involution (37 and DyConv [0]. Recently, ConvNets strike back from two aspects: (i) convolution layers are integrated to SA and bring significant gains [74 22 20 the vice versa [64]; (ii) ResNets have closed the gap to ViTs using similar data augmentation and regularization strategies 73, and replacing SA with (dynamic) depth-wise convolution
25 can also slightly surpass Swin.
Our focal modulation network also exploits depth-wit wise convolution as the micro- aacro--rchitecture but goes beyond by introducing multi-level context aggregation and input-dependee modulation. We will show this new module significantly outperforms raw convolution networks.
3 Focal Modulation Network
3.1 From Self-Attention to Focal Modulation
Given visual feature map X E HX as input, generic encoding process generates for each visual token (query) 2; E RRC, a feature representation y; R via the interaction with its surroundings X (e.g., neighboring tokens) and aggregation M over the contexts.
Self-attention. The self-attention modules use a late aggregation procedure formulated as
y: (T X).
where the aggregation Mi over the contexts X is performed after the attention scores between query and target are computed via interaction Ti.
Focal modulation. In contrast, focal modulation generates refined representation y, using early aggregation procedure formulated as
# (M,
where the context features are first aggregated using M2 at each location i, then the query interacts with the aggregated feature based on T2 to form y;-
Comparing Eq. (1) and Eq. (2). we see that (i) the context aggregation of focal modulation M2 amortizes the computation of contexts via shared operator (e.g., depth-wise convolution), while Mi in SA is more computationally expensive as it requires summing over non-shareable attention scores for different queries; (ii) the interaction Tz is a lightweight operator between a token and its context, while T₁ involves computing token-to-token attention scores, which has quadratic complexity.
Based on Eq. (2), we instantiate our focal modulation to
y, g(xx) C m(i, X),
(3)
3

---

Output
Output
Hararchcal
Z ouc
Apgregation
Interact
Altentics
t=
tate
glte
Appregate
gite
Looar
Uneer
Viase
Ixput
Input (b) Tocal-MMoulation
Input [c] Context Aggregation
(a) Self-Attention
Figure 2: Left: Comparing SA (a) and focal modulation (b) side side. Right: Detailed illustration of context aggregation in focal modulation (c).
where q()) is query projection function and @ is the element-wise multiplication. m(-) is context aggregation function, whose output is called modulator. Fig. [[a) and (b) compare self-attention and focal modulation. The proposed focal modulation has the following favorable properties:
Translation invariance. Since g(-) and ml-) are always centered at query token and no positional embedding is used, the modulation is invariant to translation of input feature map X.
Explicit input-dependeney. The modulator is computed via mn-) by aggregating the local features around target location i, hence our focal modulation is explicitly input-dependent.
Spatial- and channel-specifie. The target location i pointer for ml-) enables spatial-specifts modulation. The element-wise multiplication enables channel-specifie modulation.
Decoupled feature granularity. q(-) preserve the finest information for individual tokens, while m(-) extracts the coarser context. They are decoupled but combined through modulation.
In what follows, describe in detail the implementation of m(-) in Eq. 3
3.2 Context Aggregation via m(-)
It has been proved that both short- and long-range contexts important for visual modeling [79 18 47]. However, a single aggregation with larger receptive field is not only computationally expensive in time and memory, but also undermines the local fine-grainee structures which particularly useful for dense prediction tasks. Inspired by (7),, we propose a multi-scale hierarchical context aggregation. As depicted in Fig. 2 (c), the aggregation procedure consists of two steps: hierarchical contextualization to extract contexts from local to global ranges at different levels of granularity and gated aggregation to condense all context features at different granularity levels into the modulator.
Step 1: Hierarchical Contextualization. Given imput feature map X, we first project into new feature space with linear layer z fx(X) E x Then, hierarchical presentation of contexts is obtained using a stack of L depth-wise convolutions. At focal level E {1, L}. the output Z is derived by:
z' 5- GeLU(DW-CCnw(('1)) " MxWxC
(4)
where f is the contextualization function at the f-th level, implemented via depth-wise convolution DW-Conv with kernel size 3e' followed by GeLU activation function 28 The use of depth- wist convolution for hierarchical contextualization Compared to pooling [83] 32), depth- wise convolution is learnable and structure-aware. In contrast to regular convolution, it channel-wist and thus computationally much cheaper.
Eq. (4) is motivated by its desirable properties.
Hierarchical contextualization of Eq. generates I. levels feature maps. At level f, the effective receptive field is of = 1 + Σi= E 1). which is much larger than the kernel size k. To capture global context of the whole input, which could be high-resolution, we apply a global average pooling on the L-th level feature map - Avg-Pool(Z") Thus, we obtain in total (L 1) feature maps
{z L+1 1= which collectively capture short- and long-range contexts at different levels of granularity.

---

Figure Visualization gating values G in Eq. (5) at last layer of our FocalNet (T 3) pretrained on ImageNet-1K The columns from left to right are input images, gating maps focal level 1,2,3 and global level.
Figure 4: Visualization modulator values (cor- responding to the right side of @ in Eq. (6) at the last layer in FocalNet. The original modulator map is upsampled for display.
Step 2:
Gated Aggregation.
In this step, the (L + 1) feature maps obtained via hierarchical contextualization are condensed into modulator. In an image, the relation between visual token (query) and its surrounding contexts often depends on the content itself. For example, the model might rely on local fine- mne-grained features for encoding the queries of salient visual objects, but mainly global coarse-graan grarse- features for the queries of background scenes. Based on this intuition, we use a gating mechanism to control how much aggregate from different levels for each query. Specifically,
linear layer to obtain spatial- and level-aware gating weights G = (X) (LL+)) Then, we perform weighted sum through an element-wise multiplication to obtain single feature map Zout which has the same size the input X,
L+I
Z out Σ
O''
(5)
where G E Bxxxx1 is slice of G for the level f. When visualizing these gating maps in Fig. we surprisingly find our FocalNet indeed learns gathering the context from different focal levels adaptively
expect. As fine-grained local structure at low focal level, while a token in a uniform background needs to be aware of much larger contexts from higher levels. Until now, all the aggregation is spatial. To enable the communication across different channels, we use another linear layer h(.) to obtain the modulator map M h(Zout) # xC, In Fig. 4 we visualize the magnitude of modulator M at the last layer of our FocalNet. Interestingly, the modulators automatically pay more attention to the objects inducing the category, which implies simple way of interpreting FocalNets.
sec, for token
small object, it focuses
the
Focal Modulation. Given the implementation of ml(-) as described above, focal modulation Eq.(3) can be rewritten at the token level as
= ((x.) O
g x
(6)
where of and z are the gating value and visual feature at location i of G and Z', respectively. We summarize the proposed focal modulation in Pytorch-style pseudo code in Algorithm 1 As we can see, it can easily implemented with a few convolution and linear layers.
3.3 Complexity
In focal modulation as Eq. (6). there mainly three linear projections g(-), ft(-,, and fz(-) for Z Besides, requires a lightweight linear function (-) for gating and L depth- wise convolution fa (.) for hierarchical contextualization. Therefore, the overall number of learnable parameters 3C + CCL + +) (k+)2 Since T. and (1:²)2 are typically much smaller than C, the model size is mainly determined by the first term besides the linear projections and the depth-wist wise convolution layers, the element-wise multiplications introduce O(C(L + 2)) for each visual token. Hence, the total complexity for feature map is O(HW (3C + C(2L + 3) + (k+)))) For comparison, a window- wise attention in Swin Transformer with window size is O(HW x (3C² + 2Cw²)), where is the window size.
will show in Sec. [ Regarding the time complexity,
5

---

Algorithm 1: Pscudo code for Focal Modulation.
Input/output shape: (8, H, C) Batchaize B; Feature nap height H, width w, dim c Pocal levels: L; Conv kernel size at level é: k''
def init( ):
(L+1)), Conv2d(C, 1) be. - [Bequential (ConvadCC C, x", groups-C), GeLU() for range(L)] (LinearCC C), Dropout())
pj_in, pi_cxt Linear(C,
pj_out
def forvard(x, m=)):
x --
3, 2)
z, gate split(x, (c, C, L+1), for E. in (L):
he_ (2)
Eq.
hierarcbical contextaalization
10
+ gete (:, (:6+1]
# Eq B
gated aggregation
11
=-n GoLUUz (MLUUz..man(diin(22...
gatal;
12
==q- pl_cxx((n)
Eq. W tocal modulation
13
return pj_out( x.permute(O, 2, 3, 1)
3.4 Network Architectures
We use the same stage layouts and hidden dimensions as in Swin [46] and Focal Transformers 79 but replace the SA modules with the focal modulation modules. We thus construct series of Focal Modulation Network (FocalNet) variants. In FocalNets, we only need to specify the number of focal levels (L) and the kernel size (k") each level. For simplicity, we gradually increase the kernel size by 2 from lower focal levels to higher ones, i.e., k = b--1 - 2. To match the complexities of Swin and Focal Transformers, we design small receptive field (SRF) and a large receptive field (LRF) version for each of the four layouts by using 2 and 3 focal levels, respectively. We use non-overlapping convolution layers for patch embedding at the beginning (kernel size=4 4, stride=4) and between two stages (kernel size=2 x 2, stride=2), respectively.
4
Experiment
4.1
Image Classification
Following the recipes in 63 46 79), we train FocalNet-T, FocalNet-S and FocalNet-B with ImageNet- IK training set and report Top-1 accuracy (%) on the validation set. Training details are described the appendix.
We compare different methods on ImageNet- IK classification 16
To verify the effectiveness of FocalNet, we compare it with three groups of methods based on ConvNets, Transformers and MLPs. The results are reported in Table [ We see that FocalNets outperform the conventional CNNs (e.g., ResNet
(27] and the augmented version (73). MLP architectures such MLP-Mixer [61] and gMLP 43 and Transformer architectures DeiT (63) and PVT [69]. In particular, we compare FocalNets against Swin and Focal Transformers which use the same architecture verify FocalNet's ''s stand-alone effectiveness the bottom part. We see that FocalNets with small receptive fields (SRF) achieve consistently better performance than Swin Transformer but with similar model size, FLOPS and throughput. For example, the tiny FocalNet improves Top-1 accuracy by 0.9% over Swin-Tiny. To compare with Focal Transformers (FocalAtt), we change to large receptive fields (LRF) though is still much smaller than the one used in AtcalAtt. Focal modulation outperforms the strong and sophisticatedly designed focal attention across all model sizes. More importantly, its run-time speed is much higher than FocalAtt by getting rid of many time-consuming operations like rolling and unfolding.
Model augmentation. We investigate whether some commonly used techniques for vision transform- ers can also improve our FocalNets. First, we study the effect of using overlapped patch embedding for downsampling (22] Following (74), we change the kernel size and stride from (4, 4) to (7, 4) for patch embedding at the beginning, and (2, 2) to (3, 2) for later stages. The comparisons are reported in Table [ Overlapped patch embedding improves the performance for models of all sizes, with slightly increased computational complexity and time cost. Second, we make our FocalNets deeper but thinner as in [8 96. In Table 3, we change the depth layout of our FocalNet-T from 2-2-6-2 to 3-3-16-3, and Net from 2-2-18-2 to 4-4-28-4. Meanwhile, the hidden dimension at first stage is reduced from 96, 128 to 64, 96, respectively. These changes lead to smaller model sizes and fewer FLOPs, but higher time cost due to the increased number of sequential blocks. It turns out that

---

Model
#Params.
(M)
FLOPs Throughput Top-1 (imgsis)
(%)
Model
Overapped ParchEimeed
APaarams. FLOP: Thrughout Top-1
(M)
(G)
(imgehs)
(5)
FocalNet- (SRF)
28.4
4,4
743
82.1
ResNet-50
25.0
76.2
FocalNet-T (SRF)
30.4
730
ResNet-101 27
45.0
7.9
745
77.4
FuaiNNa-S (SRF)
49.9
8.6
434
83.4
ResNet- 152
27
60.0
11.0
522
78.3
FocaiNet-S (SRF)
51.8
8.6
424
83.4
ResNet-50-SB 733
25.0
79.8
FocalNet-B (SRF)
ResNet-100 101-SB SB
45.0
7.9
745
81.3
FocalNet-B (SRF)
91.6
15.3
278
84.0
ResNet-
[A]
60.0
11.6
522
81.8
Table 2: Effect of overlapped patch embedding.
DW-Net-T 25
24.2
3.8
1030
81.2
DW-Net-B 25
74.3
12.9
370
83.2
Model
Depth
Dim. 4Params.
FLOPs Throughout Top-1
Mixer B/16 61
59.9
12.7
455
76.4
FocalNes-T (SRF)
2:2.6.2
28.4
gMIPP-S
19.5
79.6
FocalNerT (SRF)
3-3-16-3
25.1
4.0
82.7
gMLP-B 1
73.4
15,8
301
81.6
FocalNet--S (SRF) 2-2-18-2
96
499
8.6
434
83.4
ResMLPP-224
62
30.0
6.0
871
79.4
S (SRF)
4-4-28-4
38.2
64
440
83.5
ResMLP-B24
129.1
81.0
FocalNet-B
2-2-18-2
FocalNer-B (SRF) 4-4-28-4
85.1
14.3
247
$4.1
DeiT-Small/16
-3
22.1
79.9
Table 3: Effect of deeper and thinner networks.
DeiT-Base/16 66
86.6
17.5
291
81.8
PVT-Small 69
24.5
3.8
794
79.8
PVT-Medium 69
44.2
517
81.2
Model
Img. Size #Params
FLOPs Throughput Top-1
PVT-Large 69
61.4
9.8
352
81.7
ResNat-101x3
384'
388.0
204.6
84.4
PoolFormer-m33
56.2
82.1
ResNet-152x4
480²
937.0
840.5
85,4
PoolFormer-m448
X3
73.5
11.6
347
82.5
VIT-B/16 19
384"
86.0
55.4
99
$1.0
Swin-Tiny 46
28.3
4.5
760
81.2
ViT-L/16
384
190.7
30
FocalNes- (SRF)
28.4
4.4
743
82.1
Swin Base 46
224"/224'
88.0
15.4
291
85.2
Swin
49.6
FocalNet-B
2249/224F
88.1
280
Swin Wase-Base [46]
FocalNet-S (SRF)
FocalNet- (SRF)
49.9
88.1
87.8
15.4
15.3
8.6
280
291
434
83.7
83,4
83.5
Swin-Large
Swin Base 46
FocalNet-L
FocalNer-B
56
2249/224
224'/224'
22491384
384'/384'
197.1
196.5
88.1
88.0
44.8
47.1
34.2
34.5
155
144
94
91
86.5
86.5
86.3
86.4
FocalAtt-Tiny 79
28.9
4.9
319
82.2
Swin-1 Lare 20
3847/384"
196.5
104.0
49
87.3
TocalNet-T
28.6
4.5
696
82.3
FocalNet-L.
22451384
197.1
100.6
50
87.3
FocalAtt-Small
51.1
192
83.5
SocalNet (LRF)
50.3
8.7
406
83.5
Table 4: ImageNet-1K finetuning results with mod-
FocalAtt-Base [79
89.8
83.8
els pretrained on ImageNet-22K. Numbers before and
FocalNet B (LRF)
88.7
15.4
269
83,9
after "/" resolutions used for pretraining and fine- tuning. respectively.
Table 1: ImageNet-IK classification comparison.
going deeper improves the performance of FocalNets significantly. These results demonstrate that the commonly used model augmentation techniques developed for vision transformers can be easily adopted improve the performance of FocalNets.
ImageNet-222K pretraining. We investigate the effectiveness of FocalNets when pretrained on ImageNet-22K which contains 14.2M images and 21K categories. Training details are described in the appendix. We report the results in Table 4 Though FocalNet-B/L are both pretrained with 224 x 224 resolution and directly transferred target domain with 384 x 384 image size, we can see that they consistently outperform Swin Transformers.
4.2 Detection and Segmentation
Object detection and instance segmentation. We make comparisons object detection with COCO 2017 [42. We choose Mask R-CNN [26) as the detection method and use Toca/Net-T/S/B pretrained on ImageNet- as the backbones. All models are trained the 11 118k training images and evaluated on 5K validation images. We use two standard training recipes, 1x schedule with 12 epochs and 3 schedule with 36 epochs. Following [46], we use the same multi-scale training strategy by randomly resizing the shorter side of an image to [480, 800]. Similar to [79], we increase the kernel size je by for context aggregation at all focal levels to adapt to higher input resolutions. Instead of up-sampling the relative position biases as in (79]. Focal uses simple zero- proo--pading for the extra kernel parameters. This expanding introduces negligible overhead but helps extract longer range contexts. For training, we use AdamW [48] the optimizer with initial learning rate 10-4 and weight decay 0.05, All models are trained with batch size 16. We set the stochastic drop rates to 0, 0.2, 0.3 in 1x and 0.3, 0.5, 0.5 in 3x training schedule for FocalNet-T/S/B, respectively.
The results are shown in Table We measure both box and mask mAP, and report the results for both small and large receptive field models. Comparing with Swin Transformer, FocalNets improve the box mAP (AP*) by 2.2, 1.5 and 1.9 in 1x schedule for tiny, small and base models, respectively. In 3x schedule, the improvements are still consistent and significant. Remarkably, the 1x performance of FoB (45.9/ 9/48.8) rivals Swin-T/B (46.0/48.5) trained with 3x schedule. When comparing with FocalAtt 79]] FocalNets with large receptive fields consistently outperform under all settings
7

---

Backbone
iParums FLOPs
Musk R CNN Is
Mask R CNN 3x
(M)
(G)
A1a
AI%
AP%
Apm
APSO
AP
AP
AP%
AP%
APm
AP
AP%
ResNet50
38.0
34,4
55.1
36.7
41.0
44.9
37.1
PVT:Smal 69
44.1
245
40.4
62.9
43.8
37.8
60.1
40.3
43.0
65.3
46.9
39.9
62.5
42.8
SVT.
41.4
40.3
63.2
46.8
69.2
51.2
42.6
66.3
Swin-TTiy 44
273
263
4377
66.8
47.7
59.3
633
42.7
46.0
"..T
50,3
-1.6
63.1
44.3
FocalNet-T (SRF)
48.6
267
45.9
023)
68.3
50.1
41.3
65.0
44.3
47.6
0LB
69.5
$2.0
42.6
66.5
45.6
FocalArt- Tiny
48.8
291
44.8
67.7
49.2
41.0
64.7
44.2
47.2
69.4
51.9
42.7
66.5
45.9
FocalNett- (LRF)
48.9
268
46.1
(1.3)
68.2
50.6
41.5
65.1
44.5
48.0
(HL.))
69.7
53.0
42.9
66.5
46.1
ResNet101 22
63.2
336
40.4
61.1
44.2
36.4
57.7
38.8
42.8
63.2
47.1
38.5
60.1
41.3
ResNeX1101-322
77
62.8
340
41.9
62.5
45.9
37.5
59.4
40.2
44.0
64.4
48.0
39.2
61.4
41.9
PVT-Mcdium 60
302
42.0
45.6
39.0
61.6
42.1
44.2
66.0
48.2
40.5
63.1
43.5
TVTn 14
76.3
340
45.2
67.6
49.3
41.5
64.5
44.8
48.0
69.5
52.7
43.0
66.8
46.6
Smial
690
354
463
51.3
42.1
65.8
452
48.
70.2
5335
43.3
67.3
FocalNet-S (SRF)
70.8
356
48.0 (+)
69.9
52.7
42.7
66.7
45.7
48.9 ()
70.1
53.7
43.6
67.1
47.1
SocalAAt-Salll [9]
47.4
51.9
66.6
46.1
48.8
70.5
53.6
43.8
67.7
47.2
RocalNet-S (L.FF)
72.3
365
48.3 (
70.5
53.1
43.1
67.4
46.2
49.3
70.7
54.2
43.8
67.9
47.4
ResNeXt101 64v4d 221
102.0
493
42.8
63.8
47.3
38.4
60.6
41.3
44.4
64.9
48.8
39.7
61.9
42.6
PVT-Large:00) [0]
81.0
364
42.9
46.6
39.5
61.9
42,5
44.5
66.0
48.3
40.7
63.4
43.7
Twins SVT-L [4
119.7
474
45.9
41.6
Swin-Base
107.1
197
46.
69.2
51.6
42.3
66.0
45.5
4K.
69.8
53.5
43.7
66.8
46.3
RocalNet-B (SRF)
109.4
496
48.8
01.
70.7
53.5
43.3
67.5
46.5
49.6
oLD
70.6
54.1
44.1
68.0
47.2
FocalAtt- 77
110.0
67.3
49.0
53.6
43.7
RocalNectB (LRF)
111.4
507
49.0
70.9
53.9
43.5
67.9
46.7
49.8
00.5
70,9
54.6
44.1
68.2
47.2
Table S: COCO object detection and instance segmentation results with Mask R-CNN
[26].
Backbome
Crop
FLOPss mloUU
Method
Backhone
*Param.
FLOPss
AP
APb
API
ResNet101 101
27
512
86
1029
-4.9
DW-Net-T 25
R:50
27
82.0
82.0
730
739
49.9
46.3
64.3
68.6
54.3
50.5
DW-Na-B
DW-NaiT
Twins-SVTL
25
14
512
512
133
132
924
45.5
48.8 50,2
48.3
C. Mask R-CNN 1
Swin-T [4
85.6
742
50.5
69.3
54.9
FocalNer-T (SRF)
51.5
70.1
Swin-T 16
512
60
941
44.5
45.8
FocalAn-T 79
86.7
770
51.5
70.6
55.9
DocilNa-T((RHE
46.5
47.2
FocalNet-T (LRF)
51.5
70.3
FocalAu-T
512
998
45.8
47.0
FocalNa-T (LRF)
512
61
949
46.8
47.8
R-50 27
106.1
166
44.5
63.4
48.2
Swin 16
512
1038
47.6
49.5
Swin:T [46
47.9
SocalNet-S (RFF)
49.3
50.1
Spurse R-CNN 33
FocalNNet-T (SRF)
110.5
172
49.6
69.1
54.2
FacalAnsS (7]
512
1130
48.0
50.0
FocalAtt-
49.0
69.1
FocalNat (LRF)
512
84
1044
49.1
50.1
FocalNet-T
111.2
178
49.9
69.6
$4.4
Swin.B 4X
512
1188
48.1
49.7
FocalNet-B (SRF)
512
174
1180
50.2
51.1
R-50 27]
32.1
43.5
61.9
FocalAm-8 [7)
512
126
1354
49.0
50.5
Swin T [66
35.7
212
47.2
66.5
51.3
FocalNat-B (LRF)
512
126
1192
50.5
5144
ATSS SS
-ocalNec-T (SRF)
215
49.2
68.1
Table
Semantic segmentation on
FocalAnt-T 79
36.8
239
49.5
68.8
53.9
FocalNer: (LRF)
37,2
220
49.6
68.7
$4.5
ADE20K 995
All models are trained
Table 6: A comparison of models with different object detection
with UperNet evaluation.
77
MS means multi-scale
methods, trained using the 3x schedule.
and cost much less FLOPS. For instance segmentation, we observe the similar trend that of object detection for FocalNets. To further verify the generality FocalNets, we train three detection models, Cascade Mask R-CNN (.. Sparse RCNN (55) and ATSS [88] with FocalNet-T the backbone. We train all models with 3x schedule, and report the box mAPs in Table
6 As we can see, FocalNets bring clear gains to all three detection methods over the previous SoTA methods.
Semantic segmentation. We benchmark FocalNets on semantic segmentation, a dense prediction task that requires fine- -raiind understanding and long-range interactions. We use ADE20K [95] for our experiments and follow [46] to use UperNet [75] as the segmentation method. With Fecal T/S/B trained on ImageNet- 1K as the backbones, we train UperNet for 160k iterations with input resolution 512 x 512 and batch size 16. For comparisons, we report both single- and multi-scale (MS) Table 7 shows the results with different backbones. FocalNet outperforms Swin and Focal Transformer significantly under all settings. Even for the base models, FocalNet (SRF) exceeds Swin Transformer by 2.1 and 1.4 at single- and multi-scale, respectively. Compared with Focal Transformer, FocalNets outperform Focal Transformer, with a larger gain than that Swin Transformer, and consume much less FLOPs. These results demonstrate the superiority of FocalNets on the pixel-level dense prediction tasks, in addition to the instance-level object detection task.
Given the superior results for FocalNets on segmentation tasks shown in Table we further investigate its effectiveness while scaling up. Particularly, to fairly compare with Swin-L pretrained on ImageNet- 22K with 384 x384, we also pretrain our FocalNet-L. on ImageNet-22K with 384x 384 with 3 focal levels and kernel sizes [3. 5, 7]. We use Mask2former
[2 for semantic segmentation on ADE20K and
panoptic segmentation on COCO. As shown in Table g FocalNet-L achieves superior performance to
8

---

Rackhone
Methed
#Param
miet)
AMS
Backhone
Method
Parrmm.
PQ
AP
mioU
HRNet-048
OCRNNe
K3
71M
45.7
200
M6
88M
48.4
ReaNet-50 50
DETR 3
43.4
ResNot-00
27
K-Net
Sain-B 46
Twins- SVEL
UperNet
UperNet
25
133M
121M
48.8
48.1
50.2
49.7
RoaNet-50 27
SegFormer
Panoplic
20
47M
50.0
MrT-85
SegFormer
85M
51.0
51.8
ResNer-50
27
Mask2Fomer
12
44M
$1.9
41.7
62.4
VTT-1166
1+
SETR [1]
308M
50.3
WTT-L/16
Swin-1. A6
Segmenter
UperNet Hd
334M
234M
52.1
53.6
53.5
PVTv2-B5
20
SegForeer
Panoptic
5
101M
54.1
Swin-L.
46
K-Net
by
54.3
Swin-T
MaskFormrr
42M
47.7
33.6
60.4
Swin:L]
16
PuichDiverse
21
234M
53.1
54.4
Swin-8
MaskFormer
102M
51.1
37.8
62.6
VOLO-D5
UperNet
35
54.3
Swin-T
70
Mask2Formmr
I2
47M
53.2
43.3
63.2
Focal-L.
UperNe:
240M
54.0
35.4
Swin-B B
Mask2Fornncr
12
107M
55.1
45.2
65.1
CSwat-L!
UperNer
:5
205M
54.0
55.7
Swin-Li 86
MaskFormar
D3
212M
52.7
40.1
64.8
Swinv2-G
BEITA
UperNe:
UperNe:
MM
>3.06
441M
56.7
59.1
57.0
Swin-Li
46
SegFormer
Panoptic
[
55.8
Adapter
Mask2Fommer
568M
58.3
59.0
Swin-Li
Masm2Fomer
(200 querkes)
216M
57.8
48.6
67.4
Swin:L!
Mask2Fommer
H
216M
56.4
57.7
Mask2Foomer
L
Swin-L-FaPN!
Farmerummer
P
56.1
57.3
Focal-L.f (Ours)
(200) querkes)
226M
57.9
48.4
67.3
ScMaak'
33
58.2
Table 9: Panoptic segmentation on COCO 42 f means pretraining with ImageNet-22K. All models evaluated on minival with single-scale. PQ, AP and mloU are threc metrics for measuring the panoptic segmentation, instance segmentation and semantic segmentation, respectively.
RocalNer-L. (Curs)
Fask2Fommer
t
218M
57.3
58.5
Table 8: Systematic comparisons of semantic seg-
mentation ADE20K validation set. indicates pretraining with ImageNet-2255 and means using extra data additionally.
#Param.
Throughput Top-1
FocalNet-T (LRF)
9. v(x) AEE sf
28.6
Depth-width ConvNet
V. ()
28.6
4.47
738
81.6
Pooling Aggregator
9. ((z.) MS MD of Avg-Pooliz
28.3
4.37
676
80.5
Global Pooling Aggregator
9. v(z..) @ h(g Avg (oo)))
28.3
4.36
75.7 66
Multi-scale Self-Attention (QKV first)
9, MH. #i a ( f. A Identstyf.)
28.6
Multi-scale Self-Attention (QKV later)
U. MISSA
f. Identity)))
28.6
7.26
448
80.8
Sliding-wlndow Self-Attention
9, M(a.))
7
28.3
4.49
Table 10: Performance for different FocalNet model variants.
Swin-L with similar model size and same pretraining data. We note that the methods in gray font like Swinv2-G and ViT-Adapter-L achieve better performance but much more parameters and training data. We leave the further scaling-up of our FocalNets as future work. In Table 9 we compare different models for panoptic segmentation outperforms Swin-L. on PQ, rendering a new state-of-the-art for panoptic segmentation. These results clearly demonstrate the effectiveness of our FocalNets for various segmentation tasks.
COCO with 133 categories. Our FocalNet-L slightly
4.3 Network Inspection
Model Variants. We compare in Table 10 different model variants derived from FocalNet.
Depth-wise ConvNet. It feeds the feature vectors at the top level L to a two-layer MLP. The resultant model is close to DW-Net [25). Although it can achieve 81.6%, surpassing Swin (81.3%), it underperforms FocalNet by 0.7%. FocalNet uses depth-wise convolutions as component but differently for aggregating contexts, which is then used to modulate each individual tokens.
Pooling Aggregator. It replaces the wese convolution module with average pooling, and is similar to MetaFormer (83) in terms of token aggregation. Average pooling has slightly lower complexity but leads to a significant drop of accuracy by 1.8%. Compared with depth-wise and thus incapable of capturing visual structures.
convolution, pooling is vermutation-invariine
Global Pooling Aggregator. It removes local aggregations all levels and only keeps the global one (1L+1). This variant resembles SENet (32.. It turns out that global context alone is insufficient for visual modeling, leading significant 6.7% drop.
Multi-scale Self-Attention. Given the summarized tokens different levels, a straightforward way to combine them is performing a SA among all of them. We have developed two SA methods: computing q, k, before and after aggregation, respectively. Both methods result in visible performance drop and increase the run time latency, compared to FocalNet.
Sliding-window Self-Attention. Finally, we apply hiding-windoo SA for each visual token within a window. Since it involves dense interactions for each -ee-ggaaied tokens, the time and memory cost explodes, and the performance is worse than FocalNet.

---

Model
FLOPs
Throughput Top-1 AP"
AP"
Level (Kernels)
Receptive Field
#Param. FLOPSs
Throughuut Top-1
TocalNet-T (LRFF
4.48
82.3
46.2
(3-55
28.4
4.41
743
82.1
(3-5-7)
28.6
696
Additive
4.49
670
81.5 com: 45.6
41.1 (45)
No global pool
4,48
683
82.0 (03) 45.8 (0.) 41.2 (04)
0
28.3
4.35
883
75,7
Top-only
No gating
4.48
4.49
698
707
81.9 (0.9) 45.7 (0.)) 41.2 (0.)
81.9 (0.) 45.6 co.)) 41.1 (45)
43-5-7-9
(3)
21
29.0
28.3
4.59
592
81.5
82.2
Table 1I: Component analysis for focal modulation.
(13)
28.8
661
81.9
Table 12: Model performance with number of fo- cal levels L. "Receptive Field" refers to effective receptive field at the top level regardless of the global average pooling.
Four separate changes made to the original FocalNet. Throughput reported on image classification. All variants have almost the same size (28.6M) as the default model.
Component Analysis. Here we ablate FocalNet study the relative contribution of each component. The result is reported in T1 where we investigate the impact of the following model architecture changes on model performance:
we change the element-wise multiplication to addition in Eq. (6. which converts the modulator into a bias term. This leads to 0.7% accuracy drop, which indicates that element-wise multiplication is more powerful way of modulation than addition.
Replacing Multiplication with Addition:
No Global Aggregation: we remove the top global average pooling in focal modulation. It hurts the performance by 0.3%. Even though the hicrarchical aggregation already covers relatively large receptive field, global information (Z (Z still useful for capturing global context.
Top-only Aggregation: Instead of aggregating the feature maps from all focal levels, we only use the top level map. In this case, the features at lower levels that are more "local" and "fine- grained" are completely discarded. This change leads to 0.4% performance drop, which verifies hypothesis that features at different levels and spatial scopes compensate each other.
None-gating Aggregation: We remove the gating mechanism when aggregating the multiple levels of feature maps. This causes 0.4% drop. As we discussed earlier, the dependencies between visual token (query) and its surroundings differ based on the query content. The proposed gating mechanism helps the model to adaptively learn where and how much to gather.
In Table T2, we study the effect of varying the focal level (i.e. the number of depth-wise convolution layers L). In our experiments reported above, the results show that large receptive field in general achieves better performance (LRF v.s. SRF). Here, we investigate by further altering L. In additional to setting L = 2 and 3, we also try L 0, L 1, and L 4. Accordingly, increasing L brings slight improvement and finally reaches a plateau. Surprisingly, a single level with kernel size 3 can already obtain decent performance. When we increase the single-level kernel size from to 13, there a slight 0.1% drop, and a 0.4% gap to the one with three levels but same size of receptive field (second row). This indicates that simply increasing the receptive field does not necessarily improve the performance, and a hierarchical aggregation for both fine- and coarse-grained contexts is crucial.
At last, we study whether our focal modulation can fit the monolithic architectures like ViTs. We replace all SA modules in VITs with focal modulation to construct monolithic FocalNet-T/S/B. We use patch size 16 and three focal levels with kernel sizes 3.5 and 7, so that the effective receptive field is close to the global SA in ViT. As shown in Table [33 FocalNets consistently outperform ViTs, with comparable FLOPs and inference speed.
Model
Dim 4Param. FLOPPs Th. (imguh)
Top-1
VI-T//1
2834
72.2
RacalNe-T/166
2334
74.1 (+1.9)
VIT-8/16
384
22.1
4.6
1060
19.9
S/11
384
22.4
4.3
920
80.9 (11.))
VIE-R/16
17.6
81.8
FacalNet-B/16
16.9
82.4 (606)
Table 13: Comparisons between FocalNet and ViT both with monolithic architectures.
5 Conclusion
We have proposed modulation, a new mechanism that enables input-dependent token interactions for visual modeling. consists of hierarchical contextualization to gather for each query token its contexts from short- to long-ranges query content, followed by a simple modulation. With focal modulation, we built a series of simple attention-free Focal Modulation Networks (FocalNets). Extensive experiments show that FocalNets significantly outperform the SoTA SA counterparts (e.g., Swin and Focal Transformer) with similar time-/memory the tasks of image classification, object detection and semantic segmentation.
gated aggregation to adaptively gather contexts based on the
10

---

References
[1] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 516-6162, 2018.
[2] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu, Genet: Non-local networks meet squeeze- excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[4] Shuning Chang. Pichao Wang, Fan Wang, Hao Li, and Jiashi Feng. Augmented transformer with adaptive graph for temporal action proposal generation. arXiv preprint arXiv:2103. 16024, 2021.
[5] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification, 2021.
[6] Liang-Chiee Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 501-818, 2018
[7] Qiang Chen, Qiman Wu, Jian Wang, Qinghao Hu, Tao Hu, Errui Ding, Jian Cheng, and Jingdong Wang. Mixformer: Mixing features across windows and dimensions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5249-5259, 2022.
[8] Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. CycleMLP: A mip-like architecture for dense prediction. arXiv preprint arXiv:2107. 10224, 2021.
[9] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. arXiv preprint arXiv:2 15436, 2021.
[10] Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kemels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 11030 11039, 2020.
[11] Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08.8 085344 2022.
[12] Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. arXiv preprint arXiv:2112.0152. 2021.
[13] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[14] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers. arXiv preprint arXiv: 104.22004,13440, 2021.
[15] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.0990 1.09094, 2020.
[16] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[17] Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. arXiv preprint arXiv:2204.08 03645, 2022.
[18] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprint arXiv:2107.00652, 2021.
[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11 11929, 2020.
[20] Peng Gao, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation network. arXiv preprint (rXiv:2106.01400 2021.

---

[21] Chengyue Gong. Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification, 2021.
[22] Jianyuan Guo, Kai Han, Han Wu, Chang Xu. Yehui Tang, Chunjing Xu, and Yunhe Wang, Cmt: Convolu- tional neural networks meet vision transformers. arXiv preprint arXiv:2107.0626. 2021.
[23] Jianyuan Guo, Yehui Tang, Kai Han, Xinghao Chen, Han Wu, Chao Xu, Chang Xu, and Yunhe Wang. Hire-mlp: Vision mip via hierarchical rearrangement. arXiv preprint arXiv: 13341, 2021.
[24] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, ct al. A survey on visual transformer. arXiv preprint arXiv:20/2./2555 2020.
[25] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang. Demystifying local vision transformer: Sparse connectivity, weight sharing, and dynamic weight. arXiv preprint arXiv:2106.00 04263, 2021.
[26] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961 29692999922200 2017.
[27] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[28] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.0841 2016.
[29] Qibin Hou, Zihang Jiang, Li Yuan, Ming-Minn Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mip-like architecture for visual recognition. arXiv preprint arXiv:2 12368, 2021.
[30] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:
Mixiv11704: 1004.0044400000 04861, 2017.
[31] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3464-3473, 2019.
[32] Jie Hu, Li Shen, and Gang Sun. Squeere-and--xcitation networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[33] Jitesh Jain, Anukriti Singh. Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. arXiv preprint arXiv:2112.12782 2021.
[34] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: survey, arXiv preprint arXiv:2101.01 169, 2021.
[35] Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7287-7296, 2022.
[36] Bing Li, Cheng Zheng, Silvio Giancola, and Bernard Ghanem. Sctn: Sparse convolution-rransformer network for scene flow estimation. arXiv preprint arXiv:2105.04447, 2021.
[37] Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Involution: Inverting the inherence of convolution for visual recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12321-12330, 2021.
[38] Xiangyu Li, Yonghong Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li. Trear: Transformer-basee rgb-d egocentric action recognition. arXiv preprint
2021.
[39] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. Localvit: Bringing locality vision transformers. arXiv preprint (XXiv:204400707, 2021.
[40] Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, Tong Lu, and Ping
Luo. Panoptic segformer: Delving deeper into panoptic segmentation with transformers, 2021.
[41] Dongze Lian. Zehao Yu. Xing Sun, and Shenghua Gao. As-mlp: An axial shifted mlp architecture for vision. arXiv preprint arXiv:2107.0839) 2021.
[42] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
12

---

[43] Hanxiao Liu, Zihang Dai, David So, and Quoe Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34, 2021.
[44] Hanxiao Liu, Zihang Dai, David R So, and Quoc V Le. Pay attention to MLPs. arXiv preprint arXiv:2105.0080 08050. 2021.
[45] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xic, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111.090 2021.
[46] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arX1v:2103.14330 2021.
[47] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.0354. 2022.
[48] Iyy Loshchilov and Frank Hutter. arXiv:1711.05001.22 2017.
Decoupled
weight decay regularization. arXiv preprint
[49] Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Sparse-mips fully-mlp architecture with conditional computation. arXiv preprint arXiv:2109.020m 2021.
[50] Lingehen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan,, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2309-112318 2022.
[51] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34: 34:13937-13999 2021.
[52] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. urXiv preprint 1etp::4499555555 2014.
[53] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint urXiv:2105.05633 2021.
[54] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-rcsolution representation learning for human pose estimation. In CVPR, 2019.
[55] Peize Sun, Rufeng Zhang. Yi Jiang. Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. arXiv preprint arXiv:2011.12453 124500 2020.
[56] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[57] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114 6114. PMLR, 2019.
[58] Chuanxin Tang, Yucheng Zhao, Guangting Wang. Chong Luo, Wenxuan Xic, and Wenjun Zeng. Sparse mlp for image recognition: Is self-attention really necessary? arXiv preprint arXiv:2109.05422, 2021.
[59] Ychui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is wave: Phase-aware vision mlp. arXiv preprint arXiv:2111. 2021.
[60] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. MLP-mixer: An all-mip architecture for vision. arXiv preprint arXiv:2105.00c 0000000 2021.
[61] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers. Jakob Uszkoreit, al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34, 2021.
[62] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.0344 2021.
13

---

[63] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 1rX7v:2012.1287.. 2020.
[64] Hugo Touvron, Matthieu Cord. Alaacldin El-Nouby, Piotr Bojanowski, Armand Joulin, Gabriel Synnaeve, and Hervé Jégou, Augmenting convolutional networks with attention- baasd aggregation. arXiv preprint arXiv:21 12. 1,362,, 2021.
[65] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12894 12904, 2021.
[66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
[67] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. arXiv preprint arXiv:20/2.00759, 2020.
[68] Ning Wang, Wengang Zhou, Jie Wang, and Houqaing Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking, arXiv preprint arXiv:2103. /161,, 2021.
[69] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.1 12/22, 2021.
[70] Wenhai Wang, Enze Xie, Xiang Li. Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. v2: Improved baselines with pyramid vision transformer. Compulational Visual Media, 8(3):415-424 424, 2022.
[71] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7794-7803, 2018
[72] Yuqing Wang. Zhaoliang Xu, Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011. 2020.
[73] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint IeX::ii:1100004400 2021.
[74] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103. 15808, 2021.
[75] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference Computer Vision (ECCV), pages 418 434, 2018.
[76] Enze Xic, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.
[77] Saining Xie. Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[78] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9981-9990, 2021.
[79] Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for docal-global interactions in vision transformers. arXiv preprint arXiv:2107.00644 2021.
[80] Jianwei Yang. Zhile Ren, Chuang Gan, Hongyuan Zhu, and Devi Parikh. Cross- choss-channel communication networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 1297-1306, 2019.
[81] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Paylo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10809-10818 2022.
[82] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun. and Ping Li. ML-MLP22 Improved spatial-shir mlp architecture for vision. arXiv preprint 21X0v2200..0107, 2021.
14

---

[83] Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer actually what you need for vision. arXiv preprint arXiv22111.1418, 2021.
[84] Li Yuan, Qibin Hou, Zihang Jiang. Jiashi Feng. and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint arXiv:2106. 13112, 2021.
[85] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. arXiv preprint arXiv:1909. 11055, 2019.
[86] Hang Zhang, Chongruo Wu, Zhongyue Zhang. Yi 7hu, Haibin Lin, 7hi Zhang, Yue Sun, Tong Hc, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004, 08955, 2020.
[87] Pengchuan Zhang, Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.15355 2021.
[88] Shifeng Zhang, Cheng Chi, Yonggiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9759-9768, 2020.
[89] Wenwei Zhang. Jiangmiao Pang. Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation, 2021. 2021.
[90] Xiangyu Zhang. Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848 6856, 2018.
[91] Jiaojiao Zhao, Xinyu Li, Chunhui Liu, Shuai Bing, Hao Chen, Cees GM Snoek, and Joseph Tighe. Tuber: Tube-transformer for action detection. arXiv preprint arXiv:2104.00000 2021.
[92] Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Mixing and shifting: Exploiting global and local dependencies in vision mlps. arXiv preprint arXiv:2202.065100 2022.
[93] Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:201 1.09315, 2020.
[94] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfong Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from sequeecc-to-sequencc perspective with transformers. arXiv preprint arXiv:2012. 158400 2020.
[95] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 033-641,270 2017
[96] Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transformers. arXiv preprint arXi::1106.03714 2021.
197] Xizhou Zhu Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
15


Reviews:
[{"review_id": "pTmP323UBFH", "reviewer": "Reviewer_1dzg", "summary": "This paper proposes a focal modulation module to replace the attention module in Transformers. Specifically, this module contains Hierarchical Contextualization (several layers of depthwise convolutions), Gated Aggregation, and element-wise modulation to fuse information from the token itself and the context. Actually, the element-wise modulation can also be regarded as a gate mechanism. Thus, this module sounds like a convolution + gate mechanism. This paper does extensive experiments on various vision tasks to show the advantages of the proposed modules. ", "questions": "See weaknesses. ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The writing of this paper is clear and integrated. \n\n2) The experiments in this paper are extensive, including several vision tasks and ablation studies. It is convinced that Focal Net is better than Swin and Focal Transformers. \n\nNeutrality: Focal Modulation can be regarded as convolutions plus gate mechanisms. The idea is OK, not incremental but not good enough. Thus, I write the idea aspect into neutrality.\n\nWeaknesses:\n\nIn ablation table 9, about the fusion between token itself and context, it only shows the experiment that replaces Multiplication with Addition. What about totally removing the query branch and moving the parameters and computation to other components?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nDQIPPFHeav", "reviewer": "Reviewer_EqAk", "summary": "In this paper, the authors introduce a novel Transformer (or Network) architecture, termed as Focal Modulation Network. FocalNet deals with the problem of efficient long-range feature modeling. Different from window-wise self-attention (Swin) and focal attention, FocalNet adaptively aggregates surrounding tokens from different levels of granularity. This is novel and interesting. Extensive experiments on MS COCO, ImageNet, and ADE20k demonstrate the effectiveness of the proposed method. In summary, the idea is interesting, and the performance is promising; I would like to recommend acceptance.", "questions": "See above weaknesses.\n\nAlso, a question is why the authors consider the contextualizations hierarchically rather than parallelly? I thought the contextualizations would be independent when I read the introduction for the first time. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. I enjoy reading this paper, the writing, and the presentation. \n2. The idea is interesting and novel. Considering different levels of granularity makes sense in image processing.\n3. The experimental results are promising, demonstrating state-of-the-art performance.\n\n**Weaknesses:**\n1. From L124-L132, it is easy to understand the Translation invariance and Decoupled feature granularity, but why input-dependency is considered as an advantage? Also,  Spatial- and channel-specific is achieved by depth-wise operations and cannot be considered as an advantage.\n2. I like the idea of different levels of granularity, but this can simply be considered as a multi-scale depth-wise convolution in the implementation (Fig. 2c), which limits the novelty. Multi-scale/ multi-branch always works and introduces no novelty.\n3. Some important papers are missing (not compared), like ConvNeXt, that only uses depth-wise convolution as well, and MixFormer, which can be considered as a strong baseline. Both are published in CVPR'22 and released much earlier than the submission dealine. The authors should cite and compare these baselines.\n[1] Liu, Zhuang, et al. \"A convnet for the 2020s.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Chen, Qiang, et al. \"MixFormer: Mixing Features across Windows and Dimensions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "X06OAAwL_Pa", "reviewer": "Reviewer_RTBL", "summary": "This paper proposes Focal Modulation, which uses context at multiple spatial scales from a stack of convolutions combined with gated aggregation to produce a modulation.  This modulates each query point through elementwise multiplication.  Focal Modulation is evaluate as a drop-in replacement for Self-Attention.  The method is tested in several experiments against strong baselines, and has a thorough ablation analysis.\n\n## Update after author response\n\nThank you for the responses and further experiments.  To be clear, I was mostly concerned with the kernel size experiment as a way to glean the variance of other results.  Since you have provided direct experiments to address that concern, I am more confident that this is a paper worthy of acceptance.", "questions": "I would like to see some notion of noise in the results based on random seed.  It is tempting to draw conclusions from the results as presented, but some of the effect sizes are small.  ", "limitations": "The limitations and societal impact are well addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a novel method for improving computer vision architectures based on visual transformers.  Focal Modulation is a clever way to include expanded context at each layer while simultaneously removing expensive pairwise interaction terms present in self-attention.  The method is described clearly, the diagrams are illustrative, and the experimental evaluations and ablations are thorough.  This paper is a solid contribution to an important field of computer vision. However, I would like to see some better estimate of variance of results.  For example, in Figure 5 in the appendix, the mAP results vary considerably depending on the setting of the kernel size.  There seems to be little relationship between the kernel size and the mAP, but the mAP varies considerably (41.2 to 41.6).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PdLcJbthm72", "reviewer": "Reviewer_pZvC", "summary": "The paper propose a focal modulation module that is more effective and efficient for modeling\ntoken interactions. Their main contributions are proposing an efficient way to input-dependent long-range contextual interactions. The authors conduct experiments on tasks of image classification, detection, and segmentation. The experiments' results show the SoTA performance.", "questions": "### questions\n(1) About training consuming. Will Context Aggregation cause low latency?\n(2) About Table 10. It seems that low receptive field impacts the performance little. What if bottom-only with small kernels? Is it means the context is not important? It's confusing. \n(3) About Network Inspection. A Global Pooling Aggregator will lead to a significant drop. May because of low parameters? If we replace local aggregations with several simple downsample convolutions, does it will work?\n", "limitations": "yes", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The performance achieves SOTA on almost tasks.\n+ The paper is well-written.\n+ The authors conduct abundant ablation studies to validate the effectiveness of each design in the proposed method.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept