Window Attention is Bugged: How not to Interpolate Position Embeddings
Authors: Daniel Bolya
Chaitanya Ryali
Judy Hoffman
Christoph Feichtenhofer
Venue: ICLR 2024 poster (2024)
Keywords: Bug fix
window attention
position embeddings
high resolution finetuning
image classification
video classification
object detection
instance segmentation
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: Interpolating absolute position embeddings for models with window attention (e.g., Hiera, ViTDet) is wrong. We fix it, obtaining significant gains in accuracy / efficiency.

Abstract: Window attention, position embeddings, and high resolution finetuning are core concepts in the modern transformer era of computer vision. However, we find that naively combining these near ubiquitous components can have a detrimental effect on performance. The issue is simple: interpolating position embeddings while using window attention is wrong. We study two state-of-the-art methods that have these three components, namely Hiera and ViTDet, and find that both do indeed suffer from this bug. To fix it, we introduce a simple absolute window position embedding strategy, which solves the bug outright in Hiera and allows us to increase both speed and performance of the model in ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k pretraining. This all stems from what is essentially a 3 line bug fix, which we name "absolute win".

Published as a conference paper at ICLR 2024
WINDOW ATTENTION IS BUGGED: HOW NOT TO INTERPOLATE POSITION EMBEDDINGS
Daniel Bolyal
Chaitanya Ryali
Judy Hoffman'
Christoph Feichtenhofer
Georgia Tech
2 FAIR, Meta
ddblya, judyyjyyyyyyy Jegatcch edu, feicheehoerettttthhreoeeeeeeee Gemeta.eeeemmm..cmmmmm com
ABSTRACT
Window attention, position embeddings, and high resolution finetuning are core concepts in the modern transformer of computer vision. However, find that naively combining these near ubiquitous components can have detrimental effect on performance. The issue is simple: interpolating position embeddings while using window attention is wrong. We study two state-of-the-art methods that have these three components, namely Hiera and ViTDet, and find that both do indeed suffer from this bug. To fix it, we introduce a simple absolute window position embedding strategy, which solves the bug outright in Hiera and allows us to increase both speed and performance of the model in ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box mAP on COCO, making it wfte-of-theear for models that only use ImageNet- Ik pretraining. This all stems from what is essentially a line bug fix, which we name "absolute win".
INTRODUCTION
Transformer- based architectures dominate many tasks throughout computer vision (Zhai et al., 2021; Li al., 2022a; Kirillov al., 2023). But despite their ubiquity. these architectures are relatively new, and thus many best practices have yet to be set in stone. In this paper, we focus on a relatively unassuming operation in modern vision transformers: interpolating position embeddings.
Absolute position embeddings added at the beginning of a transformer allow the model to distinguish between tokens based on location, an important detail for most vision tasks. While natural language processing (NLP) can sometimes get away with omitting position embedding (Haviv et al., 2022), those in computer vision often find themselves adding additional position embeddings to deal with more difficult tasks (e.g., detection in Li et al. (2022a)). Several recent works even design custom position embeddings to inject directly into attention (Liu et al., 2021b; b;; Graham et al., 2021; Li et al., 2022b), that location information is right where it is needed.
However, these additional position embeddings can be costly: techniques like attention bias (Liu et al., 202 bb; Graham et al., 2021) or relative position embeddings (Li et al., 2022b;a) are added directly into the attention matrix. Not only are these operations slow, but they also cannot benefit from recent innovations such as Flash Attention (Dao al., 2022; Dao, 2023) that speed up transformers by not constructing the attention matrix. Ideally, we would like to avoid relative position embeddings and just use simple and fast absolute position embeddings like the original ViT (Dosovitskiy et al., 2020).
So, why do most architectures not use absolute positioning embeddings? One potential reason becomes apparent when we study Hiera (Ryali et al., 2023), modern hierarchical vision transformer that only uses absolute position embeddings. Hiera is as powerful and more efficient than other state-of-the-ard vision architectures, while being composed entirely of simple ViT blocks. Instead of adding more position bias architecturally (e.g.. with relative position embeddings), it learns good spatial biases through a strong pretext task (i.e., MAE (He et al., 2022)). This makes it the perfect case study for modern architectural design paired with simple absolute position embeddings.
And immediately, an issue presents itself: Hiera does not interpolate well. When finetuning Hiera on images that are even slightly larger than what it was trained on, the accuracy of the resulting model plummets. This includes mediocre results on detection, where ViT outperforms Hiera (Ryali et al., 2023). The culprit, we discover, is the interaction between window attention and absolute position embeddings. That is, having window attention and absolute position embeddings in the same model
Work done during an intersship Meta. Code and models http://gtthbbbomm/ccccooccerrete

---

Published as a conference paper at ICLR 2024
Scenario 1 (e.g,, Hiera)
(b) High Res Finetuning w/ Win Attn
(a) Pretrain w/ Window Attn
Interpolate
(e) The Fix
Learned Pas Embed is windowed.
X Pos Embed misaligned with windows.
Scenario 2 (e.g., VITDet)
(d) High Res Finetuning w/ Win Attn
(c) Pretrain Global Attn
Intorpoot - To
x
X
X
Use window-awwn
absolute embedding.
global.
Embed windowed.
Figure I: Window Attention is Bugged when used with absolute position embeddings and finetuned high resolution. We study two methods that suffer from this bug: Hiera (Ryali et al., 2023) and ViTDet (Li et al., 2022a). For each, we show channel of their position embeddings. (a) Hiera's embedding learns repeated patterns aligned with window attention. (b) Interpolating breaks alignment. (c) ViTDet pretrains with global attention, adding windows for finetuning. (d) After interpolating, each window only has a piece of the full position embedding, despite each using the full attention operation from pretraining. (c) We introduce absolute window embeddings (Fig. 3) to fix both.
induces a bug when interpolating to larger images. To solve this, we introduce simple absolute window-awe awarre position embedding (namely, "absolute win") that can be interpolated to any image size without issue. This change alone is enough completely alleviate any problems with image size in Hiera, resulting in strong performance on both image and video benchmarks at high resolution.
However, the problem goes deeper: this is issue in any model that uses both absolute position embeddings and window attention at once, e.g., ViTDet (Li et al., 2022a), a state-of-the-a method for detection without extra detection data. In fact, Li et al. find that adding relative position embeddings is necessary for good performance, and we believe this bug is the reason. By using our absolute win method for detection, we remove almost all the relative position embeddings in the model. This allows for an up to 43% speed-up up over the original model while increasing mAP. Furthermore, fixing the bug unlocks other techniques that allow us to increase Hiera's 's performance on detection by up to 1.5 mAP on COCO (Lin et al., 2014), significantly outperforming ViT and establishing new state-of-the-aa for detection and instance segmentation with only ImageNet- 1 pretraining.
Note that we do not claim to introduce any exceedingly novel techniques here. Instead, we identify and analyze a bug present in the current state-of-the-an introduce simple strategy to fix it, and establish best practices for interpolating position embeddings. We then show how this alone is enough to improve the state-of-the-arr in image and video classification, detection, and instance segmentation.
BACKGROUND AND RELATED WORK
In this paper, explore bug caused by the combined use of three popular components in computer vision: window attention, absolute position embeddings, and high resolution finetuning.
Window Attention. transformers, global attention (Vaswani al., 2017) compares each token to all other tokens. This is the default for ViT' (Dosovitskiy et al., 2020), but it is often costly and potentially wasteful for larger image sizes. Several alternative attention approaches have been proposed for vision, such as pooling attention (Wang et al., 2021; 2022; Fan et al., 2021; Li et al., 2022b) or linear attention (Choromanski et al., 2020; Wang et al., 2020; Bolya et al., 2022). However, particularly popular is window attention (Liu al., 202 lb; Chu et al., 2021; Dong al., 2022; Li et al., 2022a; Ryali et al., 2023), which compares each token only to those in the same window as itself. This makes compute function of window size, rather than image size, making it much faster.
2

---

Published as a conference paper at ICLR 2024
in-1k acc @ ft res
model
pos embed
224px
256px
HHcra-LLAA
absolute
85.6
85.2 (-04)
absolute win
85.7
86.0 (40.3)
Hiera-Bsww
absolute
81.8
81.4 c 0.4)
absolute win
82.4
82.7 (+0.3)
Table 1: Finetuning at higher res than the de- fault (224px) fails when using a Hiera model with its default absolute pos emb. Absolute win fixes this for MAE trained models and increases performance for fully supervised (Sup) models.
Pretrained with MAE
Fully Supervised
Figure 2: Abs Pos Embed for Window Attn. A channel from the learne absolute position em- beddings in Hiera (Ryali et al., 2023), simple hierarchical VIT that uses window attention.
Absolute Position Embeddings. By default, attention has no spatial bias, i..., each token is treated the same regardless of its position. The original transformer (Vaswani et al., 2017) as well as ViT (Dosovitskiy et al., 2020) introduce spatial bias using an absolute position embedding (abs pos embed) different for each spatial location and added directly to each token after the embedding or patchification step. Subsequent strategies have been proposed that add this bias directly to the attention layers, such as relative position embeddings (Shaw et al., 2018; Huang al., 2020; Liu et al., 2021a; Wu et al., 2021), attention bias (Liu et al., 2021b; Graham et al., 2021), 1), or rotary embeddings (Su et al., 2021). While these effective, they often much slower than simply adding absolute embedding at the start of the network. Furthermore, works such as Hiera (Ryali et al., 2023) show that relative position embeddings (relpos) not necessary long the model is trained effectively.
High Resolution Finetuning, Finetuning models at higher resolutions (high res) has been a common practice in computer vision the past (Huang al., 2019; Tan Le, 2019), so it's not surprise that the same is true for modern vision transformers (Liu et al. (2021b); Li et al. (2022b), etc.). In fact, the only component that depends on image size in most ViTs is the position embedding. The standard strategy to resize, e.g., absolute position embedding is to simply upsample them with bicubic interpolation as Dosovitskiy et al. (2020). However, this paper we show that doing this in an architecture that has both window attention and absolute position embeddings lead to errors.
DISCOVERING A BUG
We begin our exploration of absolute position embeddings with Hiera (Ryali et al., 2023), a modern hierarchical (i.c., multi-scale) vision transformer that forgoes architectural bells-and aillssle-whistless in favor of learning spatial bias through strong pretext task (MAE, He et al. (2022)). This includes using absolute position embeddings like VIT (Dosovitskiy et al., 2020) instead of more complex relative position embeddings (Li et al., 2022a;b) or attention bias (Liu et al., 202 Graham al., 2021).
While Hiera works well in its default configuration, we obtain puzzling result when finetuning the model on even slightly larger image size. If we take original Hiera-L model pretrained on 224px images and finetune it on 256px images for ImageNet- 1k (Deng et al., 2009), the top-1 accuracy drops by 0.4% (see "absolute" in Tab. 1). Needless to say, this is the opposite of what's supposed to happen: prior work significantly benefits from finetuning on larger images.
The problem. For efficiency, Hiera uses "mask unit" attention within the first two stages of the network. This is essentially window attention, but each window always corresponds to the same 8 x tokens in the input (i.c., the windows get smaller as the tokens are pooled). This results in some interesting emergent properties when used alongside learned absolute position embeddings.
In Fig. 2, we show two channels from Hiera's learned position embeddings, one from the public Hiera-B model pretrained with MAE, and the other from training Hiera-B fully supervised from scratch. In both cases, the learned position embeddings are tiled. That is, there is a repeating 8 x 8 pattern in the embedding. And unsurprisingly, this has the same layout Hiera's 's window attention.
3

---

Published as a conference paper at ICLR 2024
Absolute Win
Window Embedding
Interpolate
Global Embedding
Window Embed Similarity
Global Embed Similarity
Figure 5: Absolute Win Similarity. Cosine similarity between tokens in the learned window and global embeddings (Fig. 3). E.g., the top left patch for window embed compares the top left token in the window embedding with all other tokens in the window embedding.
Figure 3: Absolute Win embraces the behavior in Fig. 2 by explicitly learning position embed- dings as the sum of a tiled window embedding and an interpolated global embedding. See Ap- pendix D for more learned embedding examples.
position embedding
in- acc @ ft res
method global size
224px
256px
Window Sinileity
absolute win
14 x 14
85.7
85.9 (103)
3.4
7 7
85.7
86.0 (20.3)
Pretext Task
3 x 3
85.5
85.9 (40.4
MAE Images
MAE Video
global only
7 7
85.5
85.8 (+0.3)
window only
none
84.8
85.1 (+0.)
Training Elapsed
Table 2: Varying Global Size. We default to a global embedding of 7 7 for the global embed- ding in Tab. 1. This turns out to be optimal for both 224px and 256px acc (Hiera-LMAR).
Figure 4: Window Repetition increases over training for the baseline Hiera models. Here we measure the average pairwise cosine similarity between the position embedding of each window.
Normally, this would not be an issue. However, when finetuning on larger image sizes, it is standard practice in vision interpolate the position embeddings (i.e., via bicubic interpolation). But in this case, naively interpolating misaligns the learned embeddings for each window with the windows themselves (see Fig. lab). In most cases, this drastically changes the position embeddings for each window, which explains the low accuracy after finetuning in Tab. 1.
Why happens. Position embeddings add spatial bias to the model, which allows attention to learn operations that depend on token's position. Thus, learned position embeddings are closely tied to the behavior of attention. Since each window shares the same weight matrix (for Q. K, and V), any update made to attention in one window would affect all other windows as well. This would cause the behavior of attention to average out for each window during training. And if position embeddings closely tied to this behavior, then it is not surprising that they would become similar well.
However, that is not the full picture: the tiling effect is significantly more pronounced in an MAE trained model than fully supervised one (Fig. 2). MAE deletes most windows during pretraining, which seems to encourage the model to learn redundant operations for each window. That is, each window should be treated the same matter the position, not all windows are guaranteed exist in the forward pass. This produces a starkly different embedding, which suggest that MAE pretrained and fully supervised models are learning completely different attention operations.
To motivate this hypothesis, in Fig. 4 we plot the average pairwise cosine similarity of the windows in Hiera-L's position embedding throughout training. For both images and video, pretraining with MAE causes the window similarity to increase drastically over the course of training (though not to 1.0, because Hiera still has global attention as well). Video reaches lower similarity, perhaps because video compression makes discerning the precise location of token not as beneficial. In contrast, fully supervised image model does increase in similarity, but not as much. The trend suggests each model relies spatial information differently, with supervised models potentially not enough.
The fix. While not necessarily a problem itself, Hiera's 's tiled position embedding behavior prevents it from interpolating properly. But instead of treating it as a problem, fix by embracing the
4

---

Published as a conference paper at ICLR 2024
tiling behavior and baking it in to the position embeddings. Since Hiera has both window and global attention (window in stages and 2, global in 3 and 4), we define a separate window embedding and global embedding. The former is the size of one window (8 8 tokens), while the latter is the size of the feature map without windows (7 x 7). Both are randomly initialized and learned during training.
To construct Hiera's full 56 x 56 position embedding, tile the window embedding and interpolate the global embedding, then add the results (see Fig. 3). This is similar to the learned behavior of absolute embeddings (Fig. 2), but we can now interpolate properly (Fig. le): tile the window embedding and interpolate the global embedding to the desired size (cropping as necessary). This is a drop in replacement to the original absolute position embeddings, but with slightly fewer parameters.
To test our strategy, we pretrain a Hiera-L model with 400 epochs of MAE on ImageNet- Ik using 224px images and then finetune on either 224px or 256px images in Tab. 1. When finetuning on 256px images, the original absolute embeddings drop accuracy by 0.4%. Our absolute window embeddings, on the other hand, both increase accuracy at 224px (slightly, likely due to faster convergence) and obtain the expected accuracy after finetuning at 256px. Moreover, they significantly increase performance for a fully supervised Hiera-B." Thus, we entitle this method "absolute win".
Analysis. We can study the effect position embeddings have by measuring their similarity. If the position embedding for two tokens are similar, then they are likely to be paired in attention (see Appendix E). We present the similarity of individual tokens within the learned 8 x window and 7 x 7 global embeddings in Fig. 5 for our Hiera-I, model trained with absolute win. The global embeddings act as expected: tokens close by (or in a close row or column) are considered similar. However, the window embeddings act very differently. In fact, they seem to perform dilated convolution.
Furthermore, we study the importance of each component in Tab. 2. Our default 7 x 7 size for global embeddings seems to be optimal. Moreover, both window and global embeddings alone fix the bug, since the window embed will always align with window attn and the global embed is at the resolution where windows are pooled to 1 x 1. Neverthless, together they result in the most performant model.
4 EXPANDING TO DETECTION
In Sec. 3, we explored how an interaction between window attention and absolute position embeddings caused undesirable behavior in learned position embeddings using Hiera (Ryali et al., 2023) as our test platform. However, Hiera is not the only model that uses both window attention and absolute position embeddings. In fact, common strategy when applying vanilla vision transformers (VIT, Dosovitskiy et al. (2020)) to high resolution downstream tasks is to add window attention.
4.1 VITDET
Of particular note is ViTDet (Li et al., 2022a), popular method to use ViT's for object detection. ViTDet takes a standard 224px ViT model (i.e., with 14 x 14 tokens) pretrained with MAE (He et al., 2022), and then finetunes resolutions of 1024px or more. But doing this with global attention is too expensive, so instead the authors convert all but 4 of the attention layers in the model to window attention, with each window being 14 x 14 tokens (i.e., the same as the pretrained model). Naturally, ViT uses absolute position embeddings, so to allow the model to input larger images, ViTDet naïvely interpolates them. Finally, ViTDet adds relative position embeddings (Li al., 2022b) to the attention module in each layer, which the authors find important for good performance.
Does ViTDet also suffer from the bug? ViTDet has all the necessary components for the titular bug to occur: window attention, absolute position embeddings, and naïve interpolation; but this time, in different order. Hiera trains a model with both absolute position embeddings and window attention, and then tries to interpolate the result. On the other hand, ViTDet takes interpolatable position embedding (learned or sinusoidal), interpolates it, and then adds window attention during finetuning.
While not as egregious, this is still wrong. As described in Sec. 3, position embeddings are how attention implements spatial operations. Drastically changing the position embeddings when finetun- ing, then, would change the operation performed from what the model learned during pretraining.
This suggests pure transformers (ViT, Hiera) learn suboptimal pos embeds when fully supervised on IN-1k. With careful design, may be possible obtain MAF-like performance w/ supervised training (see Appendix B).
5

---

Published as a conference paper at ICLR 2024
COCO mAP
COCO mAP
relpos
abs win
box
mask
ms/im
strategy relative
location
box
mask
ms/im
all
X
55.6
49.2
97
all attn
55.8
49.3
97
V
55.8 (+0.))
49.3 (40.))
97
win atin
55.0
49.0
75
none
X
54.6
48,5
66
global attn
55.8
49.6
68
V 55,0 (+0.4)
49.1 (+0.6)
66
absolute
patchify
55.1
49.1
66
ga only
X
55.1
48.9
69
global block
55.1
49.0
66
55.8 (40.7)
49.6 (+0.7)
68
global qkv
55.0
49.0
67
Table 3: Fixing ViTDet-L. by tiling the original position embeddings. Absolute win reduces the need for relative position embeddings (relpos) to only the global attention (ga) layers.
Table 4: Global Embed for ViTDet. Learning absolute position embeddings from scratch dur- ing finetuning doesn't work, so we opt to use relpos added to just global attention instead.
Interpolating the position embedding and adding window attention certainly "drastically changes the position embeddings" for each window by spliting one embedding into multiple parts (see Fig. led).
Applying Absolute Win. To fix this, we can apply the same "absolute win" technique, but in a different way. The original ViT model operates at a resolution of 14 x 14 tokens. Based on that, ViTDet sets the size of each window to 14 x 14 in the interpolated model. Thus, if we want attention to perform the same operation it did during pretraining, we simply need to ensure that each 14 x 14 window has the same position embedding- -ic.. by tiling the position embeddings instead of naively interpolating them. Essentially, we construct absolute win embedding with the original pretrained position embeddings acting as window embeddings (Fig. 6). Note that ViTDet has 4 global attention layers, so still need global embedding. We will discuss how to achieve that in the next section.
If we apply this technique to a ViTDet model on COCO (Lin et al., 2014) using the same training setup Li et al. (2022a), we only get small gains for both box and mask (see Tab. 3). Does this mean that ViTDet is not actually affected by the bug? Actually, the secret is in the relative position embeddings (relpos): not adding them during finetuning drops performance by up 1 mAP box and 0.7 mAP mask for the original model. If we then apply absolute win without relpos, we gain +0, mAP box and +0.6 mAP mask. This means that a portion of the gain for box mAP and most of the the gain from mask mAp for using relpos in the first place is due to the bug.
Removing Relative Position Embeddings. Does this matter? If relpos can make up the difference, why would we want to remove it? Simple: relpos is slow. In Tab. 3, we present the inference runtime for each model 100 fp16). Using relpos in every attention layer can slow the model down by 47%.
We now return to the component missing from our implementation of absolute win for detection: the global embedding. Using the pretrained model's embeddings as the finctuned model's window embeddings is correct for the window attention layers. However, the model still has 4 global attention layers that now use the wrong position embeddings. We addressed this in Sec. 3 with a global embedding, but the pretrained model does not have one here. We could try train own during finetuning, but this would adversely affect the windows from pretraining (Tab. 4: absolute, patchify).
Instead, we want to inject this position information directly into the layers with global attention, avoiding those with window attention. However, if we add a zero-initialized absolute position embedding to either the input to the block or the outputs of qkv (as is done DETR (Carion et al., 2020)) for blocks with global attention, we also see no gain (Tab. 4). Not being able to learn good absolute embeddings during supervised finetuning is not so surprising, as we found supervised Hiera-B B in Sec. 3 to also not be able to learn good absolute position embeddings from scratch. But we do have an embedding strategy that does not require good pretext task to learn: namely, relpos.
Thus, our final method for absolute win applied to ViTDet is to tile the pretrained model's position embeddings and learn new relative position embeddings for only the global attention layers. The result in Tab. 3 is model that is stronger than the original ViTDet-I., while being significantly faster.
4.2 HIERADET
If absolute win can improve VIT for detection, then what about Hiera? In Ryali et al. (2023), Hiera outperforms VIT in almost all cases when not finetuned at higher res. However, Hiera-L performs
6

---

Published as a conference paper at ICLR 2024
COCO mAP
model
abs win
box
mask
ms/im
VIT-L
X
55.6
49.2
97
ViT-L
55.8
49.6
68
Hiera-L
X
55.0
48.6
Hiera-I.
x 55.6
49.3
92
Hiera-L#
<
56.2
50.0
67
Table 5: Fixing HieraDet-L. by applying abso-
Absolute Win for ViTDet
Absoluta Win for HieraDet
lute win recursively. Our implementation ($) already outperforms the original (Ryali et al., 2023), but absolute win embeddings allows Hi- eraDet to considerably outperform even ViT
Figure 6: Absolute Win for Detection. For detection, we apply absolute win recursively by tiling the original model's pos embed.
significantly worse than VIT-L. for detection (55.0 v.s. 55.6 box mAp on COCO). Of course, this result is bugged. Hiera uses the same set up as ViTDet, i.e. funetune at 1024px images. Thus, we should expect to see a drop in accuracy as in Tab. 1. Naturally, we can fix this with absolute win.
Building a Model. We build on the methodology of Ryali et al. (2023) to construct our HeeraDet. We replace the VIT backbone with Hiera and use an FPN (Lin et al., 2017) to aggregate multi-scale features. We then use the same Mask R-CNN (He et al., 2017) heads as ViTDet. Similar to ViTDet, we replace the global attention in Hiera with window attention of the same scale (i.e., 14 x 14 windows for stage 3 and x 7 windows for stage 4) and leave the existing windows in stages I and 2 unchanged. Then allow mixing between windows like ViTDet, we choose 3 equally dispersed attention layers in stage 3 to remain global (which we find to be optimal, see Appendix C).
Applying Absolute Win. But what do we do for position embeddings? We now have two "scales" of window attention: the 8 x 8 windows originally within Hiera, and the 56 x 56 windows (i.e., 14 14 at stage and 7 X at stage 4 after downsampling) added following ViTDet. Simple we have multiple scales of window attention, so apply absolute win recursively. At the lowest level, we have our pretrained Hiera model with 8 x 8 window and 7 x 7 global embeddings. Then we construct a 56 56 embedding from these as in Fig. 3, and then use that the window embedding for the entire HieraDet model (see Fig. 6). We also add relpos to just the global attention layers to serve as the new global embedding. Finally, with the position embeddings corrected, we can now use layer- Ir decay (Clark et al., 2020; Bao et al., 2021) which boosts performance for larger models."
We show the results of these changes in Tab. 5 for a Hiera-L on COCO. Without absolute win embeddings, our HicraDet is already +0.6 box mAP and +0.7 mask mAP over the original Hiera for detection results in Ryali et al. (2023), matching the performance of ViTDet-L. However, applying absolute win unlocks Hiera's 's performance even further, adding another +0.6 box MAP and +0.7 mask mAP on top. This makes our Hier with absolute win +1.2 box mAP and +1.4 mask mAP over the original, while being faster (due to having only 3 relpos).
5 RESULTS
With the bug fixed, both Hiera and ViT perform better on several tasks. Here we evaluate absolute win v.s. the original models and other sate--of-the-ars See Appendix A for implementation details.
Image Recognition. In Tab, we evaluate the effectiveness of absolute win for Hiera on ImageNet- Ik (Deng et al., 2009) by pretraining for 1600 epochs on 224px images* using the hyperparameters from Ryali et al. (2023) and then finetuning on either 224px or 384px images. We then compare the resulting models to other existing work, i.e. MViTv2 (Li et al., 2022b) and Swin (Liu et al., 2021b), lb), that have both 224px and 384px results as well as the original Hiera models (Ryali et al., 2023) which finetune at 384px. In particular, we care about the performance increase from finetuning at larger image size (i.c., not the raw acc, as that would be unfair for supervised models). Compared to models of the same compute, Hiera with absolute win benefits more from increased resolution.
DeiTDet uses this, hut the Hicra detection experiments in Ryali et al. (2023) do not. It turns out that this hecause the bugged interpolated embeddings need to he retrained in order to get good accuracy. See Appendix for discussion about pretraining with higher resolutions.
7

---

Published as a conference paper at ICLR 2024
224px
384px
in 1k acc @ res
model
acc
flops
acc
Bops
acc
model
224px
256px
320px
448px
MViTv2-B
84.4
10.2G
85.2
3677
+0.8
Hiera-B
84.5
84.2 (-0.3)
84.6 (+0.1)
84.8 (+03)
Hiera-B
84.5
9.4G
85.0
30.3G
+0.5
w/ abs win
84,5
84.9 (+0.4)
85.4 (+0.9)
85.7 (-1.2)
Hleera-Beewa
84.5
9.4G
85.8
30.3G
+1.3
Hiera-B+
85.2
84.3 (.0.9)
85.1 (-0.1)
84.9 (-0.3)
Swin-B
83.5
15.4G
84.5
47.0G
+1.0
w/ abs win
85.1
85.3 (+0.2)
86.0 (+0.9)
86.2 (+L.)
Hiera-B+ B+
85.2
12.7G
85.2
40.3G
+0.0
Hiera-I.
86.1
85.7 (-0.4) 86.2 (40.1)
86.6 (-0.5)
Hiera-B+ase win
85.1
12.7G
86.2
40.3G
+1.1
w/ abs win
86.1
86.5 (10.4)
86.7 (40.6)
87.0 (10.9)
MVVTV2-L
85.3
42.1G
86.0
140G
+0.7
Hiera-H
86.9
86.5 (- (.04)
86.9 ( 0.0.0
86.5 (.0.)
Hicra-L
86.1
40.3G
86.4
128G
+0.3
w/ abs win
86.8
87.0 (+0.2)
87.2 (+0.4)
87.3 (-0.5)
Hiera-Lats win
86.1
40.3G
86.9
128G
+0.8
Hiera-H1
86.9
125G
86.7
383G
-0.2
Table 8: Image Size Comparison. Evaluating
Hiera-Habc win
86.8
125G
87.3
3830
+0.5
the performance of absolute win on additional finetune resolutions. Without absolute win, per- formance significantly degrades. Note that we use the same hyperparameters in each experi-
Table 6: ImageNet-ll Results. Comparing to other models that train at both 224px and 384px.
Hiera with our absolute win gains more from high res than others (Li et al., 2022b; Liu et al., 2021b; Ryali et al., 2023). Fully supervised.
ment here, which is not necessarily optimal.
model
frames
size
mAP
flops
VIT-L
16
224
37.0
5970
16x224px
32x 320px
A
MViTv2-L
40
312²
38.5
28286
model
acc
Bops
acc
Boos
acc
Hiera-L
16
224
39.8
413G
ViT-L
85.2
597G
86.1
3958G
+0,9
Hiera-Labe win
32
320
42.4
2962G
Hiera-L
87.3
413G
87.4
2962G
+0.1
VIT-H
16
224
39.5
633G
Hiera-Labs
87.1
413G
88.0
2962G
+0.9
Hicra-H
16
224
42.5
672G
VIT-H
86.6
1192G
87.4
7397G
+0.8
Hiera- Hals
32
320
43.8
7003G
Hiera-11
87.8
1159G
88.1
7003G
+0.3
Hiera-Hass win
87.6
11590
88.4
7083G
+0.8
Table 9: AVA v2.2 Results. Video action local-
ization compared to models pretrained on K400
Table 7: Kinetics-400 Results. Frames X img
with masked image modeling (Tong et al., 2022; Ryali et al., 2023; Wei et al., 2022).
size w/ 3 spatial and 5 temporal crops. ViT uses 4 temporal crops at high res (Tong et al., 2022).
while retaining 224px acc. We further test this in Tab. 8, where we present more finetune resolutions. While the original model can lose accuracy at some resolutions, absolute win solves the issue.
Video Recognition. We perform a similar experiment in Tab. 7 on Kinetics-400 (Kay et al., 2017) action classification for video using 3200 ep of pretraining and the same hyperparameters as Ryali et al. (2023). We then finetune using 32 frames (each of 320px) to compare with Video MAE (Tong et al., 2022). We interpolate the temporal embedding linearly and use absolute win on the spatial embedding. While our 224px models are not strong as Hiera's, s, our 320px results clearly show the benefits of absolute win, resulting in a Hicra-L model that outperforms Video MAE by 1-2 % at high res. In Tab. 9, we test this further by finetuning the resulting model on AVAv2.2 (Gu et al., 2018), an action localization dataset. This allows us to outperform the prior state-of-the-an for K400 pretraining by +2.6 AP for and +1.3 AP for H. Thus, absolute win is as effective on video as images.
Object Detection and Segmentation. We evaluate our absolute win applied to ViTDet and HieraDet for object detection and instance segmentation on COCO (Lin et al., 2014) using Detectron2 (Wu et al., 2019) training on train2017 and testing on va12817. VIT baselines are from ViTDet (Li et al., 2022a) and Hiera baselines are from the original paper (Ryali et al., 2023). For VIT, we keep all hyperparameters the same, but for Hiera we use our own implementation described in Sec. 4.2.
In Tab. 10, we compare ViTDet and HieraDet with absolute win the results from their original papers by finetuning Mask R-CNN (He et al., 2017) or Cascade Mask R-CNN (Cai & Vasconcelos, 2019). In every case, absolute win improves performance. Notably, the performance increase is slight for ViTDet (as its extra relpos hides the improvement). but version is much faster (see Fig. 7). Furthermore, absolute win both speeds up HieraDet and results in huge performance improvements: up to 1.5 Abhox and 1.6 Ammakk for the base model. The resulting Hiera models outperform ViTDet.
In Fig. 7, we benchmark our models against ViTDet as well as MV1Tv2 (Li et al., 2022b) and Swin (Liu et al., 2021b) using ImageNet-211 pretraining from Li et al. (2022a), and the more recent Swin
8

---

Published as a conference paper at ICLR 2024
Mask R CNN Ap
Cascade Mask R-CNN
model
Ap
Ap
Apilask
ViT-B
51.6
45.9
54.0
46.7
wi Aboolute Win Hlla
VIT-Babs win
52.2
46.2
54.1
47.0
Box AP
Hiera-B
52.2
46.3
Hiera-B- abs win
53.7
47.9
55.8
48.4
statich
Haxy pouk
Hicra-B+
53.5
47.3
Stute au
Hiera-B+* abs win
54.3
48.2
56.8
49.1
01.5
VIT-L
55.6
49.2
57.6
49.8
VIT-L
55.8
49.6
57.8
50.0
Hiera-L
55.0
48.6
Mask AP
wi
Hiera-L abs
56.2
50.0
58.4
50.7
WT
VIT-H
56.7
50.1
58.7
50.9
VIT-H win
56.9
50.3
58.8
51.0
vithaL
Masy puot
Hiera-H
57.0
50.3
59.0
51.1
abs win
MVTT2
Table 10: COCO w/ Absolute Win. Compari- son between our ViTDet and HieraDet with ab- solute win and without." Our implementation.
Swaln
Inference Time (ms im)
Figure 7: COCO Runtime. Benchmarked on the same A100 using Mask R-CNN (fp16, bs=8).
single scale test Ap
multi multi-scll scale test Aph
model
A Apmisk
Ap Apillock
single-scale test
Swin-L. HTC-+, 21K,
57.1
49.5
58.0
50.4
model
Aprrak
Apmask
Aphox
MViTv2-H Caa, 21K, sup
57.1
48.8
58.4
50.1
Detic Zhou et (2022)
41.7
41.7
CBNetV2 HEC, 21K,
59.1
51.0
59.6
51.8
CBNetV2 Fu (2001)
2021 Comp Winner
49.2
45.4
ViTDet-I Cas,
60.4
52.0
61.3
53.1
ViTDet-L ei (2022a)
46.0
34.3
51.2
w/ absolute win
ViTDet-H 1i (2022))
48.1
36.9
53.4
Hiera-B Cas, IK, MAE
57.7
49.6
59.3
51.2
w/ absolute win
Hiera-B+ Cas, IK, MAF
58.5
50.4
60.0
52.0
Hiera-B abs win
42.9
32.0
47.7
Hiera-I. Cas, MAF
60.1
51.6
61.5
53.0
Hiera-B+ abs
43.9
30.7
48.9
Hiera-H Cas, IK, MAR
60.7
52.0
61.7
53.3
Hiera-L abs win
47.1
37.4
52.5
Table 11: COCO Results on minival for meth- ods pretrained on ImageNet (Liu et al., 2021b; Li ct al., 2022b; Liang et al., 2022; Li et al., 2022a).
Hiera-H abs win
48,9
38.0
54.6
Table 12: LVIS v1 Results. System-level com- parison using the same setup as Li et al. (2022a).
using SimMIM (Xie et al., 2022) pretraining. Our method significantly speeds up both ViTDet and HieraDet, with the resulting HieraDet being faster and more accurate than everything else.
Finally, like in Li et al. (2022a), we present additional results using cascade (cas), 1280px images, and softnms (Bodla et al., 2017) with and without test-time augmentation. In Tab. 11, we compare the resulting model to other ImageNet pretrained work. HicraDet with absolute win is extremely strong: with our bug fix, Hiera-B outperforms methods using heavier backbones and pipelines like HTC (Chen et al., 2019). Moreover, multi-scale testing requires interpolating the position embedding. Thus, our models with absolute win benefit more, with Hiera-L outperforming ViT-H on box mAP.
We conduct the same experiment on LVIS (Gupta al., 2019) using the setup from ViTDet in Tab. 12. Our version of HieraDet also significantly outperforms ViTDet. It comes close to the results from the LVIS-optimized 2021 challenge winner (Fu et al., 2021), despite using only baseline LVIS strategies.
CONCLUSION
We find a bug, fix it with absolute win, and show extensively how doing so improves the state-of-the art. Our findings lead to a few practical suggestions. From Hiera: embracing the emergent behavior of model can lead to significant gains. From ViTDet: minimizing the disparity between pretraining and finetuning can lead to optimal performance. But more than that, we discover a potential issue for any method with window attention and absolute position embeddings. While we only show two examples here, several others use these components. Moreover, other methods could be partially affected- e.g.. Swin has window attention but uses relative embeddings, which if replaced with absolute embeddings could lead to a faster model. We hope that future work can use these techniques to obtain more powerful and more efficient state-of-the-art, which would see as absolute win.

---

Published as a conference paper at ICLR 2024
REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv:2106.086 08254 (vs.CV), 2021.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-mms-improving object detection with one line of code. In ICCV, 2017.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, and Judy Hoffman. Hydra attention: Efficient attention with many heads. arXiv:2209.07488 [cs.CV], 2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance
segmentation. TPAMI, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In CVPR, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv:2009. 14794 Lcs.LG), 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 2021.
Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. ICLRR 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. RandAugment: Practical automated data augmentation with reduced search space. In CVPR, 2020.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv 2307.08691 [cs. 2023.
Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. arXiv:2205.14135 [cs.LG], 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In CVPR, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
WeiFu Fu, CongChong Nie, Ting Sun, Jun Liu, TianLiang Zhang, and Yong Liu. Lvis challenge track technical report 1st place solution: distribution balanced and boundary refinement for large vocabulary instance segmentation. arXiv:2111 1.06668 (cs. CV), 2021.
Benjamin Graham, Alaaeldin El-Nouby. Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's ''s clothing for faster inference. In ICCV, 2021.
10

---

Published as a conference paper at ICLR 2024
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaanarasimhann George Toderici, Susanna Ricco, Rahul Sukthankar, et al. AVA: A video dataset of apatio-temporally localized atomic visual actions. In CVPR, 2018.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: dataset for large vocabulary instance segmentation. In CVPR, 2019.
Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. Transformer language models without positional encodings still learn positional information. In EMNLP, 2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. NeurIPS, 2019.
Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. arXiv:2009. 0ttt20009.33333 13658 (cs.CL). 2020.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv:1705.06955 [cs. CVJ, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. ICCV, 2023.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. ICLR, 2017.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. arXiv:2203. 16527 [cx. CVJ, 2022a.
Yanghao Li, Chao Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In CVPR, 2022b.
Tingting Liang, Xiaojie Chu, Yudong Liu, Yongtao Wang, Zhi Tang, Wei Chu, Jingdong Chen, and Haibin Ling. Cbnet: A composite backbone network architecture for object detection. TIP, 2022.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitiik.. Microsoft COCO: common objects in context. ECCV, 2014.
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Yahui Liu, Enver Sangineto, Wei Bi, Nicu Sebe, Bruno Lepri, and Marco Nadai. Efficient training of visual transformers with small datasets. Advances in Neural Information Processing Systems, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po- Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, and Christoph Feichtenhofer. Hiera: A hierarchical vision transformer without the bells-and- whistles. ICML, 2023.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv: 1803.08 0255 [es. CI.), 2018.
11

---

Published as a conference paper at ICLR 2024
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. JMLR, 2014.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.090 09864 [cs. CLJ, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv:2203. 2tt.:2203..2600 /2602 [cs. CV). 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 2017.
Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. FxXx:22006.04768 [cs.LG], 2020.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. CVM, 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yual Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training, In CVPR, 2022.
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In ICCV, 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://gi thwb. cttp hwtpf/fbooorseerrc/eeeeeeeeeeeee deterrroonneeeennnn 2019.
Zhenda Xie, Zheng Zhang, Yue Cao,, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.
Simmim: A simple framework for masked image modeling. In CVPR, 2022.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. ECCV, 2022.
12

---

Published as a conference paper at ICLR 2024
A
IMPLEMENTATION DETAILS
Here we list the details and configurations for each set experiments.
A.1 IMAGE RECOGNITION
For our image recognition experiments, train and evaluate ImageNet-1K (Deng et al., 2009) using MAF pretraining for 1600 epochs following the same hyperparameters in Ryali et al. (2023). For finetuning, our hyperparameters follow Ryali al. (2023) for the most part. We us these same hyperparameters for all image sizes in Tab. and Tab. 8. Note that different hyperparameters might be optimal for higher resolution, but that of scope for our experiments here. We list the settings we use for finetuning in Tab. 13 below.
config optimizer
value Adam W 81 =090., 0.999 0.05
optimizer momentum
weight decay
learning rate
2e-3 (B); le-3 (B+, I,, H) cosine decay
learning rate schedule
warmup epochs
epochs
100 (B, B+); (L, H)
augmentation batch size
RandAug (9, 0.5) 1024
(Cubuk et 2020)
mixup (Zhing
0.8
cutmix (Yin et al., 2019)
1.0
label smoothing (Seegedy et al., 2016)
0.1
drop path (Lansoo et al., 2017)
0.1 (B, B+); 0.2 (L3; (II)
dropout (Snvistava al.,
0.7 (B, B+); 0.9 (L): 0.85 (II)
Table 13: Finetuning settings for ImageNet-1K. We use the same settings for each image size.
Note that finetune the large image models directly from the pretrained model. Also, for larger image sizes for H models, we had to use both activation checkpointing and torch 2.0 scaled dot product attention to allow the same batch size fit in GPU memory. The latter could affect accuracy.
A.2 VIDRO RECOGNITION
For video recognition, we conduct experiments on Kinetics-400 (Kay et al., 2017) action classification and AVA v2.2 (Gu et al., 2018) action localization. Like for ImageNet-Ik Ik, for Kinetics-400, we use the same 3200 epoch pretraining settings as Ryali et al. (2023), as well as the same finetuning settings for the original 16x 224px resolution. For 32x 320px, we finetune with the settings in Tab. 14.
config
value Adam W S1. B2 0.9. 0.999 1e-8
optimizer
optimizer momentum
weight decay
learning rate learning rate schedule
1.6e-4 cosine decay 30
epochs
repeated sampling augmentation batch size
2
RandAug (7. 0.5) 2020)
128
gradient clipping
5.0
label smoothing (Soegedy 2016)
0.1
drop path (Lamson 2017)
0.5 (Lazx s20p))....... 0.6 (Hazx32p))
dropout (Srivastava al., 2014)
0.5
layer-wise decay (Clark al., 2020)
0.9
Table 14: Finetuning settings for Kinetics-400. Finetuning 32 frames and 320px resolution.
We finetune the high res models on top of the 16x224px finetuned model, not from the pretrained one (to save time). Activation checkpointing was used for both models, and mixup disabled because
13

---

Published as a conference paper at ICLR 2024
each gpu had batch size of 1. To interpolate the position embeddings temporally, linearly interpolate the temporal embeddings from Hiera, and use absolute win for the spatial embeddings. Like with the low resolution models, sample frames with temporal stride of 4.
We take the 32x320px finetuned Kinetics-400 model and finetune it further on AVA v2.2 using the following settings in Tab. 15.
config optimizer
values
SGD
weight decay
1e-8
learning
3.6 (Lx2xx20ppp 3.2 (Hzzx320p))
learning rate schedule
cosine decay
warmup epochs
5 3202x ) 8 (Hszx
epochs
30
batch size
128
drop path (Lanson of al,, 2017)
dropout (Srivastava et al., 2014)
0.5
layer-wise decay (Clark al., 2020)
0.9
Table 15: Finetuning settings for AVA. Finetuning at 32 frames and 320px resolution.
The model required us to use a batch size of per GPU even with activation checkpointing, which we found to be unstable. Thus, we use lower learning rate and more warmup epochs for the H model to compensate. is possible that a higher learning rate would result better performance.
A.3 DETECTION AND INSTANCE SEGMENTATION
For detection with ViTDet, we use all the same parameters as Li et al. (2022a), Our settings for HieraDet, however, are very different Ryali al. (2023). We list those settings in Tab. 16 here:
config
values Adam
optimizer
optimizer momentum
81. 0.9. 0.999
0.9.9
weight decay
0.1
learning rate learning schedule
1.4c-4 (B); 7 (B+): 1.2c-4 (L): 6c-5 (H)
step-wit dep-wise decay
epochs
100 B+, L); 75 (H)
augmentation
L.S. [0.1, 2.0]
batch size drop path (Lanson 2017)
64
0.2 (B): 0.3 (B+); 0.45 (L); 0.55 (H) 0.85 (B, B+): 0.9 (L): 0.925 (H)
layer-wise decay 2020)
window attn size global attn layers relpos
8, 4. 14. 12-16-20 (B, B+); 23-33-43 (I., H)
global attn layers only
Table 16: Settings for COCO. HieraDet finetuning settings for both Mask and Cascade R-CNN.
We use these settings for every COCO experiment, except for our 1280px experiment where we increase the droppath for H to 0.6. When method.
softnms (Bodla et al., 2017), use the linear
Our settings for LVIS the same in Tab. 16, except loss as in Li et al. (2022a). To compensate we also adjust the learning rates: 1.4e-4 (B); 1.4e-4 (B+); 2.4e-4 (L.); 6e-5 (H). Finally, all models use 100 epochs for LVIS like in Li et al. (2022a).
repeat factor sampling and federated
A.4 SPEED BENCHMARKING
We use a single NVIDIA A100 40GB GPU to benchmark speed for baselines and our approach. All detection results were benchmarked with batch size of 8 using fp16 in detectron2, taking multiple tests and throwing out the first 25%. For attention layers (in ViT and Hiera) nor using relpos, accelerate attention using torch 2.0's scaled dot product attention function. We used Li et al. 's Detectron2 code for each approach. We approximate Hiera without absolute win as our version of HeeraDet with all layers having relpos. The full benchmark results follows in Tab. 17.
14

---

Published as a conference paper at ICLR 2024
ms/ im
method
B+
L
MVITv2
47
104
210
Swin
33
44
VIT
97
133
VIT abs win
68
101
Hicra
47
57
92
Hiera abs win
36
42
67
107
Table 17: Detection Benchmarking. The numbers for Fig. 7. Using 1024px images on COCO with
Mask R-CNN on a single A100 with batch size of and half precision.
Model Superdeed
model
pos embed
acc
A
Hiera-Lsap
absolute
76.1
absolute win
79.8
+3.7
Window
Hicra-Bsup
absolute
81.8
absolute win
82.4
+0.6
2006
B0%
tools
Table 18: Fully Supervised Training. Absolute win can significantly improve the performance of Hiera model trained from scratch. However, Hicra-L here benefits significantly more than Hiera-B. Fig. 8 has potential answer: Hiera- L. doesn't learn the repeated window pattern.
Training Elspoed
Figure 8: Window Repetition for fully super- vised baseline Hiera models ImageNet- Ik. Like Fig. 4, but here showing both Hiera-B and Hiera-L models. Notably, the L model does not reach the same level of window similarity as B.
B ABSOLUTE WIN FOR FULLY SUPERVISED MODELS
It's common knowledge that training pure transformers fully supervised with only ImageNet- I data is suboptimal (Dosovitskiy et al., 2020; Zhai et al., 2021; He et al., 2022). This is true for Hiera as well (Ryali et al. (2023) Appendix). However, the specific mechanism that causes this deficiency or the details behind why pretraining with, e.g., MAE helps much is still not fully understood.
In Sec. 3, make two interesting observations: Hiera's window repetition behavior is less pronounced when training fully supervised from scratch than when training with MAE, and absolute win can improve performance for fully supervised models. order to shed some light on the differences between what supervised and MAE-pretrained models learn, we explore these two observations.
In Fig. 8, we show the window similarity over fully supervised training of Hiera model from scratch. While the Hiera-B model learns some kind of repeated window embedding, the Hiera-L model specifically does not. And we find that this is correlated with accuracy. In Tab. 18, we show the final accuracies of these two models, and the Hiera-L model performs significantly worse than the Hiera-B model. This is not surprising, since it is likely for these larger models to overfit on ImageNet-Ik. However, interestingly using absolute win increases the accuracy of the model by 3.7%.
We would like to emphasize: simply rearranging few parameters at the start of the network drastically improves performance for a supervised Hiera-L model. It does not seem reasonable. then, that the accuracy drop is a wholly attributable to overfitting. There are, after all, 48 layers of parameters between the position embeddings and the outputs of Hiera-L model. If the model can overfit with one position embedding, then it can overfit with a slightly different position embedding.
Instead of overfitting, seems that these models do not learn good prior for attention. While MAE requires spatial reasoning to perform well, fully supervised classification does not. Attention does not have explicit signal from supervised classification, making more likely for a fully supervised model get stuck in bad local optimum for attention. Simply rearranging the position embedding can allow the model to find a new optimum that makes better use of spatial information.
But, we note that supervised training is not necessarily the problem. To test this, perform one more experiment through the lens of repeated window position embeddings. This time, we take a baseline Hiera-L model pretrained with MAE and reset the position embeddings, then do 50 epochs of supervised training. In Fig. 9, we compare this to training that same model from seratch (for 300 supervised epochs). And interestingly enough, the model is able relearn its position embeddings
15

---

Published as a conference paper at ICLR 2024
Window Similarity
00.4
Weights
MAE Reset PasEma
0.2
From Scratch
20% 40%
80%
100%
Training Elapsed
Figure 9: Resetting the Pos Embed for MAE pretrained Hiera-L model before finetuning. Unlike training completely from scratch, the MAE-pretrained model quickly relearns position embedding with high window similarity. However, doesn't ''t fully reach the original similarity.
under supervised training. Thus, the task itself is not directly responsible for the repeated position embeddings. Instead, the learned attention operation is, meaning that MAE enables the model to settle into good local optimum for attention. Though, note that the model with reset position embeddings does not relearn exactly the same position embedding (falling just short on window similarity). obtains 84.1 accuracy v.s. the original model's 85.6%.
A couple of take-aways from this line of inquiry: (1) Overfitting may not be the full story of poor performance of large transformers trained from scratch ImageNet-Ik- learning suboptimal attention operations may play a role; (2) the improved results from MAE pretraining can potentially be replicated when training with only supervised classification, provided the model finds the right local optimum; (3) absolute win, some other way of subtly nudging the model in the right direction, can significantly improve performance for fully supervised models. We leave further exploration of this subject to future work.
C HIERADET ABLATIONS
Here ablate the global attention placement for HieraDet mentioned in Sec. 4.2. As described in Tab. 19 below, we find that placing global attn layers too early or too late in the network hurts performance. Then, we find that we only need 3 global attention layers get optimal performance. Finally, we ablate the stride for placing these layers and find 10 to be optimal. So, our final placement for Hiera-L is the last layer of stage 3 (43) and then two more layers placed before it at 33 and 23.
ga layers
Aphox
Apmus
ga layers
Aph
Apmms
none
55.2
49,2
none
55.2
49.2
10,3
54.0
48.2
42,3
55.4
49.4
10.3. 26.3
55.2
49,2
42,a. 47,4
55.3
49.2
26.3. 42,3
55.8
49.6
(a) Early Layers. Placing a ga layer early (e.g.. layer 10) actually hurts performance.
(b) Late Layers. Making the last layer in stage 4 layer also hurts performance slightly.
ga layers
Apbox
Apmask
layers
Aphox
Apilask
31.3. 43G3
55.9
49.6
19.3. 31 43
56.0
49.9
19.3. 31,a, 43
56.0
49.9
23.3, 33 43,g
56.1
49.9
19.3. 27.3. 35.s, 43ws
56.0
49.7
27.3. 35.3, 43,a
55.8
49.6
31 37.3, 43
55.9
49.8
41.3. 42.3, 43.8
55.8
49.5
(c) Number of Layers. We do not find any benefit from using more than ga layers.
(d) Placement Stride. We find stride 10 going hack from last layer stage to the best.
Table 19:
HieraDet Global Attention Layers. Ablating the placement of global attention layers in Hiera, here for Hiera-L. model. The stage the layer belongs to is denoted by s We find the optimal strategy to be placing 3 global attention layers in stage 3 of the model. Note that we use absolute win in these experiments, but different hyperparameters for training than the main paper.
16

---

Published as a conference paper at ICLR 2024
D MORE EMBEDDING VISUALIZATIONS
Here present visualizations of more channels of the position embeddings described in the main paper. To show the generality of this issue, we provide visualization of the publicly released "bugged" Hiera models different scales: tiny (Fig. 10), base (Fig. 11), and huge (Fig. 12). In each case, we present a random sample of position the position embeddings. Fig. 13 shows more examples of absolute win (as in Fig. 3) for Hiera-I. model trained with the fix.
Figure 10: Hiera-T MAE Image Original Position Embedding. Channels of the position embedding for the public Hiera-T model pretrained using MAE on images.
Figure 11: Hiera-B MAE Image Original Position Embedding, Channels of the position embedding for the public Hiera-B model pretrained using MAE on images.
17

---

Published as a conference paper at ICLR 2024
Figure 12: Hiera-H MAE Image Original Position Embedding, Channels of the position embed- ding for public Hiera-H model pretrained using MAE on images.
Global Embedding
Global
Embedding
Window
Window
Embedding
Embedding
Global
Global
Embedding
Embedding
Window Embedding
Window Embedding
Figure 13: Hiera MAE Image Absolute Win Position Embedding. Channels of the position embedding for our Hiera-L model pretrained using MAE on images with absolute win.
D.I WINDOW SIMILARITY
While the "windowness" of the original Hiera model's position embeddings are apparent from visualization, also measure this issue quantitatively by looking the average "similarity" between windows at the end training as in Fig. 4. We' ve computed this statistic for all the publicly available Hiera ImageNet- 1k trained models in Tab. 20:
18

---

Published as a conference paper at ICLR 2024
Model
B
B+
H
Window Similarity
0.83
0.81
0.84
0.77
0.74
0.66
Table 20: Window Similarity for the publicly released original Hiera MAE models, measured by
taking the average pairwise cosine similarity between all pairs of windows in the position embedding.
While the larger models have less similarity. the similarity is still high for all models. Note that the values in this table are larger than the final number reached in Fig. because we trained those models using 400 MAE epochs, whereas the original Hiera models train for 1600 MAE epochs.
E THEORETICAL JUSTIFICATION FOR ABSOLUTE WIN
We've showcased the existence of a bug between interpolating position embeddings and window attention and how absolute win fixes it empirically, In this section, we present additional theoretical justification for why the bug happens and how to fix it.
E LINKING POSITION EMBEDDINGS ATTENTION
In the main paper, we treat the position embedding and the action of attention as intrinsically linked. Here, we show mathematically why this is the case.
Absolute position embeddings (e.g., in Hiera or ViT) are added directly after patchification. For each patchified token at position i with n features, D; IR, the result of adding the corresponding position embedding, Pi, is
x = x; +
(1)
This o/ immediately passed into the first block of the network and thus the first attention layer. For the attention matrix (ignoring heads without loss of generality), the model constructs queries and keys k as q: z g and w. & for weight matrices Wg. W3 Rax=. The (i. j)th element of the attention matrix pre-softmax is then
akk = w W
For brevity, let Wss W W.. Then, if we substitute back wi, get:
k = (z, p.) Wiy (zy +p))"
When we expand the above expression, we get 4 terms term without posemb, 2 cross terms, and the following explicit position embedding term:
Rh = ps W
Importantly, the same We is being used for each position, the only spatially varying parameters in this term come from the position embeddings. Thus, any repeated pattern we see in the actual values visualize in Fig. 2, with similarity plotted in Fig. 5) explicitly affects the action
of P4 and Py (what of attention in this first layer of the network. Then, because of skip connections, it also does so in subsequent layers (though with less effect in deeper layers). This is consistent with the commonly held belief that position embeddings have more effect at the start of the network than at the end.
E.2 How REPEATED POSITION EMBEDDINGS CAUSE THE BUG
Following Sec. E.I, we can emulate what happens to cause the bug by explicitly defining p. to be periodic. For simplicity, will treat and p as 1D in space, but this can be extended any number of dimensions.
Let denote the length of this ID image used during pretraining. Then, let us further assume that the model learns a periodic pattern for p that repeats every tokens. Then, during finetuning, we upsample the image size to l.. If we upsample the position embeddings by interpolation, then notably,
19

---

Published as a conference paper at ICLR 2024
the periodicity of the position embeddings will change, i... t' + t. However, the proportion each period takes up in the image will remain the same, i.c. t'/e- = 1/6.
Now, let us turn our attention to attention. Consider model that uses window attention with window size of w, which doesn't 't change during finetuning. Assuming the window attention is what caused the periodicity in the position embeddings, then w t (which is true empirically). Then by Eq. 4, the position bias for the window starting at position would be
p; W Pit1 W gt
DeW WaPPitt Pit1 W 3kPp++
P: W Pit1 W skPi+w-1
(5)
:
Pitw-1 Worp'
Pi-w W WakPi
For brevity, let's consider just one row of this matrix, specifically p. Wig ct reason about this vector, we will make one additional assumption about what the position embeddings learn. And that is, adjacent position vectors away positions learn dissimilar values (p; 7 Pi+w-1). Note that the exact specifics of what is exactly similar to what not explicitly a requirement for the derivation, just this general fact that things inside the window are treated differently to those across the window. But we can see similar trend empirically in Fig. 5 window similarity. The tokens at the edges are not similar to tokens of the opposite edge, and spatially close tokens more similar.
[pt
Pi+w-1] 1²" To
window learn similar values (p: ~ Pi+1) and far
Then, observe that in this case our row p, Wgg [pr along the first couple of elements, and then lower values for the remaining elements. Now, let's focus on what happens in the higher resolution image, where w remains the same, but 1' = (f/e) L.
Pi+--1] 1 will have high magnitude values
Presume t' = t + 1. Now let's again consider our vector: p; W qb [p: ... pi+m=1] 11" When i O, the situation doesn't 't change much: Po and P1 are similar, and Pw-1 is still dissimilar to Po- However, the situation changes when we look at the next window. is now periodic with a period of i', which is t
w + Thus, Pw is not prs but rather Pe-1- So in the next window, Wyk gle [Pwr ... p2 1"" the two spatially adjacent Pw and Pu+1 are actually pr and Pv, which, modulo the period of t' is the same as comparing Po and pr- Thus, these two adjacent tokens in the window now have position bias in attention that says they are not specially adjacent, and in fact on the other side of the window from each other. Moreover, if we take the two farthest tokens in that window, with position embeddings pw and P2w-1, modulo e' this becomes w and Pr-3 which (if t > 3) are considered similar.
This continues with different patters for all other windows, meaning that whatever position bias the model learned to rely on during pretaining is now completely lost for finetuning. It is not surprising then, that the accuracy of the model greatly suffers when this happens. Thus, this mismatch between and w is the bug.
One interesting property of this result is that, if t' is multiple of w, the bug has less of effectt, as the attention bias matrices will then repeat over the windows with a period of t' //u, rather than shift in every window. Conversely, the effect of the bug is the greatest when t' is slightly different than w, as shown in the above example, We can see this empirically in Tab. 8, where 256px images (pretrained on 224px) show the largest drop in performance for the bugged model.
E.3 How TO FIX THE ISSUE
The bug is caused by interpolating the position embeddings, which results in t' * t. The simplest way to alleviate the issue, then, is just to ensure that L' t. We do this in two ways for absolute win: first, window position embeddings explicitly set t' = f, as we manually define it to be periodic with a period of = w, and this doesn't 't change when the image size changes. Second, our global position embedding can explicitly forbid the embeddings from learning periodic pattern in the first place by being too small to be affected by the tokens inside of a window. In this case, the model does not have the stipulation that p: A pi+tw-1 and instead changes gradually such that the shifting of windows do not invert what was learned during training. As these are orthogonal solutions, we add them to create absolute win.
20

---

Published as a conference paper at ICLR 2024
We choose these solutions because they are simple and they closely resemble what the original model learns, meaning it can be used as a drop-in replacement. However, there are other potential solutions we don't I consider in the paper. Specifically, from the derivation in Sec. E.2 hints at another solution that can still be periodic. Instead of having the model learn p: & w-+ (i.e., periodic with harsh boundaries), we could impose some sort of regularization ensure that the periodicity is continuous. That is, p.pl+ 2 piP(+1- If we enforce the model to learn a smooth transition, the attention bias will change less from pretraining to finetuning. However, this type of approach would likely require choosing good loss function and tuning hyperparameters to get the same accuracy as the original model. Thus, we leave this direction for future work to explore.
Alternatively, we could simply increase the window size in the larger resolution model, i.c. set w/ = (e) / w: This would also work, but creates a more expensive model (as attention now over more tokens), and is not actually always possible. Since w has to be an integer (we can't reasonably attend over half a token), this only works for specific resolutions.
F PRETRAINING AT HIGHER RESOLUTIONS
In the main paper, we explore increasing the input resolution of Hiera models by pretraining with MAE using the standard 224px image resolution and finetuning at different downstream resolutions. Would it be beneficial to pretrain at the same resolution we finetune at? In this section, we perform some preliminary experiments to answer that question, both for finetuning on ImageNet-I Ik and for COCO detection.
We first begin by pretraining and finetuning at the same resolution on ImageNet-Ik In each case, we use the same window size (8) and adjust the background position embedding size accordingly (i.c., 14 for 224, 16 for 256, and 24 for 384). We present these results in Tab. 21.
pretrain
finetune res
in- lk
model
mask%
res
acc
Hiera-L.
60
224
256
86.5
60
256
256
86.4
Hiera-L
60
224
384
86.9
60
384
384
86.8
70
384
384
87.0
80
384
384
86.9
384
384
86.4
Table 21: High-Res Pretraining on ImageNet-1k 1k using Heira-L with absolute win. We also vary the pretrain masking ratio in order to increase task difficulty.
However, the result is that pretraining a higher resolution actually hurts performance, especially if you don't change the task difficulty of the MAE pretraining step. We can get some minor gains for 384px images by pretraining and finetuning with 384 and using 70% masking ratio, but this is nearly worth it for the extra pretraining time.
This result isn't 't too surprising. Prior work (Xie et al., 2022) has found that pretraining a slightly lower resolution than finetuning performs better on ImageNet- Ik using other architectures. But while pretraining at a higher resolution may not be beneficial for ImageNet- it could have much greater effect on downstream tasks like detection, where the resolutions are much higher than anything used in image classification.
Thus, we also finetune these higher resolution models on COCO and report results in Tab. 22. But note that there's an additional axis of consideration here: when applying absolute win for detection, we tile the original position embedding and replace the global attention with window attention to match that tiling. So what resolution do we construct the original position embedding with? Using a bigger resolution means bigger tiles and more pairwise computation in each window.
And here, the trend reverses: the models pretrained with higher resolutions that performed worse on ImageNet- 1k perform noticable amount better COCO, even with the same tiling resolution. This important because despite the higher cost of pretraining, models with the same tiling resolution cost the same amount to train and evaluate detection. So in the case of pretraining with 256
21

---

Published as a conference paper at ICLR 2024
pretrain
finetune tile res
model
mask%
res
Aphox
Apaask
Hiera-L
60
224
224
56.2
50.0
224
56.6
384
224
56.4
50.0
Hiera-I.
60
224
256
56.3
49.8
256
256
56.5
49.9
Hicra-I.
60
384
56.2
384
384
56.5
50.0
Table 22: High-Res Pretraining for COCO using the Heira-L pretrained on ImageNet- 1 with absolute win in Tab. 21. Tile res denotes the resolution the position embedding is tiled at as well as the window size for window attention (which are equivalent when using absolute win). Note that higher tile resolution means slower model, good performance low tiling resolutions is ideal.
and finetuning on detection with a tiling res of 224, we essentially got +0.4 box and +0.1 mask AP for free at finetune inference time!
It's likely possible to use this technique to outperform our results in Tab. 11. And moreover, this approach would likely work for other backbones, c.g. with ViTDet, well. Simply pretrain at 256, downsample the position embedding to 224, and then finetune detection with a position embedding and window tiling of 224. We leave this for future work explore.
22


Reviews:
[{"review_id": "n3vfW43QZk", "reviewer": "Reviewer_hUZp", "summary": "The paper identifies an issue that arises when using absolute position embeddings with windowed attention in vision transformers. The key points are:\n\n- Many vision transformers use both absolute position embeddings and windowed attention for efficiency. However, naively interpolating the position embeddings when finetuning on larger image sizes misaligns them with the window partitions. \n\n- This causes significant performance degradation when finetuning on larger images, which the paper demonstrates on Hiera and ViTDet models.\n\n- To fix this, the authors propose \"absolute win\" embeddings, which separately learn an interpolated global position embedding and a tiled window position embedding. This aligns the embeddings with window attention properly.\n\n- Applying absolute win embeddings improves performance substantially in Hiera and ViTDet across tasks like image classification, object detection, video action recognition.\n\n- With the fix, Hiera and ViTDet establish new state-of-the-art results on COCO and other datasets using only ImageNet-1K pretraining, while also being faster due to reduced need for relative position biases.\n\n- The paper provides practical guidelines for using absolute position embeddings with windowed attention when finetuning vision transformers at higher resolutions.\n\nThe key contribution is identifying and fixing a bug that improves SOTA vision transformers. The fix is simple but impactful - just changing the position embeddings.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors identify an important yet overlooked interaction between commonly used components in vision transformers - window attention and absolute position embeddings. When naively combined and interpolated for high-resolution finetuning, they show this leads to degraded performance.\n\nThe key novelty is in formally characterizing and providing strong empirical evidence for this bug. While simple in retrospect, it is a creative insight to recognize that the position embeddings effectively \"tile\" based on the window size due to the weight sharing in attention. The visualizations and analysis of the learned embeddings clearly demonstrate this phenomenon.\n\nFixing the bug is also quite elegant - simply embrace the tiling behavior with \"absolute win\" position embeddings. This simple change allows models to properly interpolate and achieves significantly improved performance across various vision tasks.\n\nThe paper is very clearly written and easy to follow. The introduction motivates the problem well, and the methodology is laid out in a structured manner. Experiments comprehensively ablate the impact of the bug fix on state-of-the-art methods.\n\nOverall, this is a significant finding - identifying and correcting an important issue affecting a large class of vision transformer models. The clarity and thoroughness of the empirical validation make a compelling case for the prevalence and implications of this bug.", "weaknesses": "* The proposed fix of \"absolute win\" embeddings is intuitive, but the paper does not provide much analysis into why tiling the window embeddings specifically works better than other potential solutions. Some ablation studies on the design choices could add more clarity.\n* The simplicity of the proposed fix somewhat limits the depth of technical novelty and contribution. Providing more design motivation and analysis for \"absolute win\" would strengthen this aspect.\n* Tiling the window position embeddings makes sense based on the observed tiling behavior, but other arrangements could also resolve the misalignment issue when interpolating. The design space is not explored in depth.\n* While shown to work empirically, the approach lacks theoretical analysis into why tiling embeddings avoids the misalignment issue during interpolation. A more formal understanding could improve the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HNqJCqePB4", "reviewer": "Reviewer_QscK", "summary": "The authors discuss the challenges faced when combining window attention, position embeddings, and high-resolution finetuning in the modern transformer era of computer vision. They identify a bug that arises when interpolating position embeddings while using window attention. Two state-of-the-art methods, Hiera and ViTDet, are found to be affected by this bug. To address this, the authors introduce an \"absolute window position embedding strategy\" which rectifies the issue in Hiera and enhances the speed and performance of ViTDet.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "In-depth Analysis: The paper provides a comprehensive analysis of the problem, identifying the root cause of the issue with window attention and position embeddings.\n\nSolution-Oriented: The authors not only identify the problem but also propose a solution in the form of an \"absolute window position embedding strategy\".\n\nEmpirical Evidence: The paper presents empirical results, showcasing the effectiveness of their proposed solution, especially with the HieraDet model's performance on COCO.\n\nRelevance: The topic is highly relevant given the prominence of transformer architectures in computer vision.", "weaknesses": "Limited Scope: The study primarily focuses on two methods, Hiera and ViTDet. Expanding the scope to include more methods might provide a broader understanding of the issue.\n\nAssumption-based: Some conclusions, especially regarding the behavior of position embeddings, seem to be based on observations and assumptions. More rigorous testing might solidify these claims.\n\nFurthermore, the paper's contribution appears to be limited. It primarily addresses a minor bug in Hiera and ViTDET. The resulting improvements, while noteworthy, are not substantial, suggesting that the issue wasn't fundamental. Nonetheless, I appreciate the effort and contribution of this work", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z1Lwj8CQHB", "reviewer": "Reviewer_7g6r", "summary": "This paper identifies and fixes a bug that occurs when using both window attention and absolute position embeddings in vision transformers. The authors show that naively interpolating absolute position embeddings when finetuning on larger image sizes misaligns the embeddings with the window attention, hurting performance.\n\nThe bug is analyzed in detail using Hiera and ViTDet as case studies. Hiera learns repetitive absolute position embeddings aligned with its window attention during pretraining. Interpolating these embeddings when finetuning on larger images misaligns them, causing a drop in accuracy.\n\nViTDet suffers from a similar issue - it uses interpolated absolute position embeddings with added window attention, which provides each window only a sliced portion of the full embedding.\n\nTo fix the bug, the authors propose \"absolute win\" - embedding strategies that align position embeddings with window attention after interpolation. This involves learning separate window and global position embeddings.\n\nOn Hiera, absolute win embeddings fix the bug and achieve strong image classification accuracy when finetuning on larger resolutions. The embeddings also enable removing most relative position embeddings from ViTDet while increasing performance.\n\nThe authors then apply absolute win to create HieraDet for detection, outperforming ViTDet by 1.5+ box AP on COCO using only ImageNet pretraining.", "questions": "Does relative position encoding exhibit similar phenomena as absolute position encoding?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Identifies and clearly analyzes an important bug in modern vision transformers\n- Proposes a simple and effective solution - \"absolute win\" position embeddings\n- Achieves SOTA detection performance with HieraDet using only ImageNet-1k pretraining\n- Improves performance across multiple architectures (Hiera, ViTDet) and tasks", "weaknesses": "- The gains from fixing this bug seem somewhat task/architecture dependent (smaller gains on ViTDet).\n- Unclear if the bug manifests in other architectures with window attention and absolute embeddings.\n- Does not explore more powerful global position embedding strategies for ViTDet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ukHQ4Yyr3p", "reviewer": "Reviewer_4xm4", "summary": "This paper studies a new problem that there could be a misalignment in the learned position embedding when fine-tuning a model at a higher resolution, the model is composed of both window-attention and absolute position embedding. The solution is to applied separate embeddings for window and global position. The proposed separate position embedding can improve the suffered models, Hiera and ViTDet for classification and object detection respectively.", "questions": "There exist weaknesses in the current draft, which should be discussed further before publication.\n\n1. The motivation: \nThe whole problem(bug) is found and discussed based on the \"learned\" feature embedding, which is shown in Figs. 1 and 2 in the draft. And this example plays a foundation role in this study, why it is a bug and what the fix is. However, I am trying to figure out why the visualization can be considered position information? This study uses Hiera as an example to showcase the bug, and Hiera is a modified version of MViTv2. I know some of the learned position embedding also goes through or fused with semantic features, so how can one confirm the defined position embedding is really position embedding? Could the learned block also contain semantic feature and could the semantic feature be the reason behind?(Please correct me if I am wrong on Hiera, MViTv2 uses decomposed position). Assuming if it is position embedding, how one confirm the bug is from window-attention + absolute position? Could the learned embedding is the main reason, in another word, what if I use pre-defined absolute position information with window-attention, could the bug still be there?(I guess the performance could drop, but the claim was on exposing the bug.) \n\n2. The fix:\nThis study proposes to use separate position embedding for window and global for the bug. This solution could be trivial or the demonstration of the question can be further studied. Hiera is a simplified version of the model, MViTv2, and Hiera replaced relative position with absolute position on purpose because they did not observe differences in their ablation table. Empirically, if mask unit(window-att)+ absolute position could lead to lower performance on higher resolution images, one can simply use relative position as used in original MViTv2. This proposed method can be formulated as combining both window and global position for a slightly higher performance, as shown in Tab 2 in the draft. A further question is that if the method is proposed based on the \"bug\" not fixing it, is it still a \"bug\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Position embedding is widely used in modern Transformer-based vision architectures. This paper aims at studying important questions on position embedding, and attention-mechanism. The draft is well-written and organized, the proposed method is clearly demonstrated.", "weaknesses": "The paper is well-presented, but some issues could be further discussed or studied to avoid misleading. A list of questions is shown below, a short summary for the weaknesses is:  1. The whole idea is based on the visualization of the position embedding, but if the visualization can represent position information is a question, I hope the draft can further explain this question to make the draft clear. 2. The proposed method utilizes the \"bug\" of the learned window position embedding, which raises a question of if the learned embedding is a bug.\nThis draft could be summarized as HieraV2 which uses both global and window position embedding based on the feature in Hiera.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)