SimPLR: A Simple and Plain Transformer for Efficient Object Detection and Segmentation
Authors: Duy Kien Nguyen
Martin R. Oswald
Cees G. M. Snoek
Venue: Accepted by TMLR ()

Abstract: The ability to detect objects in images at varying scales has played a pivotal role in the design of modern object detectors. Despite considerable progress in removing hand-crafted components and simplifying the architecture with transformers, multi-scale feature maps and pyramid designs remain a key factor for their empirical success. In this paper, we show that shifting the multiscale inductive bias into the attention mechanism can work well, resulting in a plain detector ‘SimPLR’ whose backbone and detection head are both non-hierarchical and operate on single-scale features. We find through our experiments that SimPLR with scale-aware attention is plain and simple architecture, yet competitive with multi-scale vision transformer alternatives. Compared to the multi-scale and single-scale state-of-the-art, our model scales better with bigger capacity (self-supervised) models and more pre-training data, allowing us to report a consistently better accuracy and faster runtime for object detection, instance segmentation, as well as panoptic segmentation. Code is released at \url{https://github.com/kienduynguyen/SimPLR}.

Published in Transactions on Machine Learning Research (02/2025)
SimPLR: A Simple and Plain Transformer for Efficient Object Detection and Segmentation
Duy-Kien Nguyen University of Amsterdam
d. d. ngaaeen@uoo..l
Martin R. Oswald University of Amsterdam
m.r.oswalddrwv..t
Cees G. M. Snoek University of Amsterdam
cgmsnock@uva.nt
Reviewed on OpenReview: attps:// opennevieew forum? id= 6L01y8ZE0F
Abstract
The ability to detect objects in images at varying scales has played a pivotal role in the design of modern object detectors. Despite considerable progress in removing hand-crafted components and simplifying the architecture with transformers, multi-scale feature maps and pyramid designs remain key factor for their empirical success. In this paper, we show that shifting the multiscale inductive bias into the attention mechanism can work well, resulting in a plain detector 'SimPLR' whose backbone and detection head are both non-hierarchical and operate on single-scale features. We find through our experiments that SimPLR with scale-aware attention plain and simple architecture, yet competitive with multi-scale vision transformer alternatives. Compared the multi-scale and single-scale state-of-the-art, our model scales better with bigger capacity (self-supervised) models and more pre-training data, allowing us to report consistently better accuracy and faster runtime for object detection, instance segmentation, well as panoptic segmentation. Code is released at https://gi thub::/b/gtttt....///gg com///ieendunggyyanS/SnR.RR
1 Introduction
After its astonishing achievements natural language processing, the transformer (Vaswani el al., 2017) has quickly become the neural network architecture of choice in computer vision, as evidenced by recent success in image classification (Liu et al., 2021; Dosovitskiy et al., 2021; Dehghaní et al., 2023), object detection (Carion et al., 2020; Zhu et al., 2021; Nguyen et al., 2022; Lin et al., 2023) and segmentation (Wang al., 2021a; Cheng et al., 2022; Li et al., 2023). Unlike natural language processing, where the same pre-trained network can be deployed for a wide range of downstream tasks with only minor modifications (Brown et al., 2020; Devlin et al., 2019), computer vision tasks such as object dotection and segmentation require a different set of domain-specifie knowledge to be incorporated into the network. Consequently, is commonly accepted that modern object detector contains two main components: a pro-trained backbone as the general feature extractor, and lask-specific head that conducts detection and segmentation tasks using domain knowledge. For transformer- based vision architectures, the question remains whether to add more inductive biases to learn them from data.
The spatial nature of image data lies at the core of computer vision. Besides learning long range fealure dependencies, the ability of capturing local structure of neighboring pixels is critical for representing and understanding the image content. Building upon the successes of convolutional neural networks, a line of research biases the transformer architecture to be multi-scale and hierarchical when dealing with the image input, i.e. Swin Transformer (Liu et al., 2021) and others (Fan et al., 2021; Wang et al., 2021b; Heo et al.,

---

Published in Transactions on Machine Learning Research (02/2025)
BaiinBBacboone Detection
End-to-End Detection
This paper: SimPLR
Convolution Head
Transformer
Transformer Head
Head
single-scale
plain
feature pyramids
hierarchical
feature pyramids
plain
Figure Object detection architectures. Left: The plain-backhone detector from (Li et al., 2022) whose input (denoted in the dashed region) are multi-scale features. Middle: State-of-the-art end-to-end detectors (Nguyen et al,, 2022; Li et al., 2023; Hu al., 2023) utilize hierarchical backhone (f.e., Swin (Liu et al., 2021)) create multi-scale imputs. Right: Our simple single-scale detector following the end-to-end framework. Where existing detectors require feature pyramids to be effective, we propose plain detector, SimPLR, whose backbone and detection head are non-hierarchical and operate single-scale feature map. The plain detector, SimPLR, performs par even better compared hierarchical and/or multi-scale counterparts, while also being more scaling-efficient.
2021). The hicrarchical design makes it casy to create multi-scale features for dense vision tasks and allows pre-trained transformers to be seamlessly integrated into convolution-based delection head with feature pyramid network (Lin et al., 2017), yielding impressive results in object detection and segmentation. However, the inductive biases in the architectural design make it benefit less from self-supervised learning and the scaling of model size (Li et al., 2022).
An alternative direction pursues the idea of a simple transformer with "less inductive biases" and emphasizes learning vision-specific knowledge directly from image data. Specifically, the Vision Transformer (ViT) (Dosovitskiy et al., 2021) stands out as a plain architecture with a constant feature resolution, and acts as the feature extractor in plain-backbone detection. This is motivated by the success of ViTs scaling behaviours in visual recognition (He ct al., 2022; Bao et al,, 2022; Dehghani et al., 2023). In addition, the end-to-end detection franework proposed by Carion el al. (2020) with transformer-basee detection head further removes many dendnee components, like non non-maxinum suppression or corrrectiiooove-uniiiii computation, that encodes the prior knowledge for object dotection.
The plain design of ViTs, however, casts doubts about its ability capture information of objects across multiple scales. While recent studies (Dosovitskiy et al., 2021; Li et al., 2022) suggest that ViIs with global self-attention could potentially learn translation-cquiiariant and scalo-cquivariant propertics during training, leading object detectors still require multi-scale feature maps and/ /or hierarchical backbone. This observation holds true for both convolutional (Ren et al., 2015; He et al., 2017; Li et al., 2022) and Transormer-based detectors (Nguyen et al., 2022; Cheng et al., 2022; Lin et al., 2023; Li et al., 2023). Unlike hicrarchical backbones, the creation of feature pyramids conflicts with the original design philosophy of ViTs, Therefore, our goal is pursue plain detector whose backbone and detection head are both single-scule and non-hterarchical. This is accomplished by directly incorporating multi-scale information into the attention compulation.
In this paper, introduce SimPLR, plain detector for both end-to-end detection and segmentation frameworks. In particular, our detector extracts the singlo-scale fcature map from a ViT backbone, which is then fed into the transformer encoder-decoder via simple projection to make the prediction (see Fig. 1). To deal with objects of various sizes, we propose to incorporate scale information into the attention mechanism, resulting in an adaptive-scale attention. The proposed attention mechanism learns capture adaptive scale distribution from training data. This eliminates the need for multi-scale feature maps from the ViT backbone, yielding simple and efficient detector.
2

---

Published in Transactions on Machine Learning Research (02/2025)
The proposed detector, SimPLR, turns out to be effective solution for plain detection and segmentation. We find that multi-scale feature maps are not necessary and the scale-aware attention mechanism, that learns multi-scale information in the attention computation, adequately captures objects at various sizes from output features of a ViT backbone. Despite the plain architocture, our detector shows competitive performance compared to the strong hierarchical-backoonee or multi-scale detectors, while being consistently faster. Morcover, the effectiveness of our detector is observed not only in object detection but also instance segmentation and panoptic segmentation. Finally, SimPLR shows to be a scaling-efficient detector when moving to bigger models, indicating plain detectors to be promising direction for dense prediction tasks.
2 Related Work
Backbones for object detection. Inspired by R-CNN (Girshick et al., 2014), modern object detection methods utilize a task-specific head on top of a pre-trained backbone. Initially, object detectors (Simonyan & Zisserman, 2015; Xie et al., 2017; He et al., 2016; Huang et al., 2017) were dominated by convolutional neural network (CNN) backbones (LeCun & Bengio, 1995) pre-trained on ImageNet (Deng el al., 2009). With the success of the transformer in learning from large-scale text data (Brown et al., 2020; Devlin et al., 2019), many studics have explored the transformer for computer vision (Chen et al., 2020a; Dosovitskiy et al., 2021; Liu et al., 2021). The Vision Transformer (ViT) (Dosovitskiy et al., 2021) with a simple design demonstrated the capability of learning meaningful representation for visual recognition. By removing the need for labels, methods with self-supervised learning have emerged as an even more powerful solution for pre-training general vision representations (Chen et al., 2020b; He et al., 2022). We show through experiments that SimPLR takes advantages of the significant progress in representation learning at scale with ViTs for object detection and segmentation.
End-to-end detection and segmentation. The end-to-end framework for object detection proposed in DETR (Carion et al., 2020) removes the need for many hand-crafted modules. This was made possible by adopting the Transformer as the detection head to directly give the prediction. Follow-up works (Dong et al., 2021; Wang et al., 2021a; Nguyen et al., 2022) extended the transformer- based head in end-to-end frameworks for instance segmentation, and panoptic segmentation. This inspired MaskFormer (Cheng et al., 2021) and K-Net (Zhang et al., 2021) to unify segmentation tasks with a class-agnostic mask prediction. Pointing out that MaskFormer and K-Net lag behind specialized architectures, Cheng el al. (Cheng el al., 2022) introduce Mask2Former, reaching strong performance on segmentation tasks. Yu et al. (Yu et al., 2022) replace self-attention with k-means clustcring, further boosting the effectiveness of the network. Another direction is to improve the object query in the decoder via a denoising process (Zhang et al., 2023; Li et al., 2023) or a matching process (Jia et al., 2023; Zong et al., 2023). While simplifying the detection and segmentation framework, these architectures still require an hierarchical backbone along with feature pyramids. The use of feature pyramids increases the sequence length of the input to the transformer- detection head making the detector less efficient. In this work, we enable a plain detector by removing hierarchical and multi-scale constraints.
Plain detectors. Following the goal of less inductive biases in the architecture, recent studies focus on the non-hierarchical and single-scale detector. Motivated by the success of ViT, a line of research considers plain-backbone detectors that replace the hierarchical backbone with a VIT. Initially, Chen et al. (Chen et al., 2022) present UVIT as a plain detector that contains a ViT backbone and a single-scale convolutional detection head. Since the backbone architecture is modified during pre-training to adopt the progressive window attention, UVIT unable to take the advantages of existing pre-training approaches with ViTs. ViTDet (Li et al., 2022) tackles this problem with simple adaptations of the ViT backbone during fine-lming. These simple modifications allow ViTDet to benefit directly from recent supervid learning with ViTs (L.., MAE (He et al., 2022)), resulting in strong results when scaling to larger models. Despite enabling plain-backbone detectors, feature pyramids are still an important factor in ViTDet to detect objects at various scales.
Most recently, Lin et al. (Lin ct al., 2023) introduce the transformer-based detector, PlainDETR, which also removes the multi-scale input. However, it still relies on multi-scale features to generate the object proposals for its decoder. In the decoder, PERRnDETR replaces the standard Hungarian matching (Kuhn, 1955) with
3

---

Published in Transactions on Machine Learning Research (02/2025)
hybrid matching (Jia et al., 2023) to strengthen its prediction, while decoder preserves simple design in Zhu et al. (2021); Nguyen et al. (2022). Moreover, PlainDETR can only perform object detection while SimPLR facilitates detection, instance segmentation and panoptic segmentation. We believe to be the first to remove the hicrarchical and multi-scale constraints which appear in the backbone and the input of the transformer encoder for bolh detection and segmentation lasks. Our proposed scale-aware attention can further plug into current end-to-end frameworks without significant architectural changes. Because of the simple and plain design, SimPLR is more efficient and effective when scaling to larger models.
3 SimPLR: Simple and Plain Detector
Multi-scale feature maps in a hierarchical backbone can be easily extracted from the pyramid structure (Liu et al., 2016; Lin et al., 2017; Zhu et al., 2021). When moving to a VIT backbone with a constant feature resolution, the creation of multi-scale feature maps requires complex backbone adaptations. Morcover, the benefits of multi-scale features in object detection frameworks using ViTs remain unclear. Studies on plain- bain-- detection (Li et al., 2022; Chen et al., 2022) conjecture the high-dimensional VIT with self-attention and positional embeddings (Vaswani et al., 2017) is able to preserve important information for localizing objects. From this conjecture, we hypothesize that a proper design of the transformer- baased head will enable a plain detector.
Our proposed detector, SimPLR, is conceptually simple: a pre-trained VIT backbone to extract plain features from an image, which are then fed into a single-scale encoder-decoder to make the final prediction. Thus, SimPLR eliminates the non-trivial creation of feature pyramids from the ViT backbone. But the single-scale encoder-decoder requires effective design to deal with objects at different scales. First, we review box-atlention in Nguyen et al. (2022) as our strong baseline in end-to-end detection and segientation using feature pyramids. Then, we introduce the key elements of our plain detector, SimPLR, including its scale-aware attention that is the main factor for learning of adaptive object scales.
3.1 Background
Our goal is to further simplify the detection and segmentation pipeline from Zhu et al. (2021); Li et al. (2022); Nguyen et al. (2022), and to prove the effectiveness of the plain detector in both object detection and segmentation tasks. We utilize the sparse box-attention mechanism from Nguyen et al. (2022) as strong baseline due to its effectiveness in learning discriminative object representations while being lightweight in computation.
We first revisit box-attention proposed by Nguyen et al. Nguyen et al. (2022). Given the input feature map from the backbone, the encoder layers with box-attention will output contextual representations. The contextual representations are utilized to predict object proposals and to initialize object queries for the decoder. We denote the input feature map of encoder layer as E IR HeH.xx and the query vector q E Idd with He, We d denoting height, width, and dimension of the input features respectively. Each query vector E IRd in the input feature map is assigned a reference window r=(ær, w., h1. where x,, indicate the query coordinate and w, h are the size of the reference window both being normalized by the image size, The box-attention refines the reference window into a region of interest, r', as:
r' = Fscale |erranslate (r, q), q) Fscale (r, q) = [x,y, y, w + Aw. h + An] Feranslace (r. [x + Ar, y + Ag. w. h]
(1)
(2)
(3)
where Fscale and Firanslate are the scaling and translation transformations, Ar, Ay: Au and A are the offsets regarding to the reference window r. A linear projection (Rd + R²4) is applied on q to predict offset parameters
(i.e., Az. Ay A and AR) w.r.t. the window size.
Similar to self-attention Vaswani et al. (2017), box-attention aggregates multi-head features from regions of interest:
AtteeiionAtttnntinn Attention
head,) Wo
(4)
4

---

Published in Transactions on Machine Learning Research (02/2025)
During the i-th attention head computation, 2x2 feature grid is sampled from the corresponding region of interest ry resulting in set of value features E R2x2xds The 2x2 attention scores are efficiently generated by computing a dot-product between E Rd and relative position embeddings (A3 E [[xxxxd followed by a softmax function. The attended feature head, E Rd is a weighted average of the 2x2 value features in i with the corresponding atiention weights:
a = softmax(q (g]);)
(5)
2x
head; = BoxAttention(g. ka 03) = Σ ayti
(6)
2=0
where q E IRd, K; R2x2xd, U, _xx2xd, are query, key and value vectors of box-attention, aj the j-th attention weight, and Vi is the j-th feature vector in the feature grid ve. To better capture objects at different scales, the box-attention in Nguyen et al. (2022) takes multi-scale feature maps, {e²) 4), as its inputs. In the i-th attention, feature grids are sampled from each of multi-scale feature maps in order to produce head,
Sparse attention mechanisms like box-attention lie at the core recent end-to-end detection and segmentation models due to their ability of capturing object information with lower complexity. The effectiveness and efficiency of these attention mechanisms bring up the question: Is multi-scale object information learnable within delector that is non-hierrrrchial and single-scale?
3.2 Scale-aware attention
The output features of the encoder should capture objects at different. scales. Therefore, unlike the feature pyramids where each set of features encodes specific scale, predicting objects from plain feature map requires its feature vectors to reason about dynamic scale information based on the image content. This can be addressed elfectively by a multi-head allention mechanism that caplure different scale information in each of its attention heads. In that case, global self-attention is a potential candidate because of its large receptive field and powerful representation. However, its computational complexity is quadratic w.r.t. the sequence length of the input, making the operation computationally expensive when dealing with high-resolution images. The self-attention also leads to worse performance and slow convergence in end-to-end detection (Zhu et al., 2021). This motivates to propose multi-head scale-aware attention mechanism for single-scale input.
Scale-aware attention. In sparse multi-scale attention mechanisms, such as deformable attention (Zhu et al., 2021) or box-attention (Nguyen et al., 2022), each feature vector is assigned to a scale in the feature pyramid. As a result, feature vectors learn to adapt to that specific scale. While this behaviour may not impact the multi-scale deformable attention or multi-scale box-attention which utilizes feature pyramids for detecting objects it poses a big challenge in learning scale-equivariant features from a single-scale input.
To address this limitation, propose two variants of multi-head scale-aware attention (i.e., fired-scole and adaptive-scale) that integrate different scales into each attention head, allowing query vectors to choose the suitable scale information during training. Our proposed attention mechanism is simple: we assign anchors of m different scales lo attention heads of each query. We use m anchors with size w=h E {s. 2 y where 8 is the size of the smallest anchor, and m is the number of scales. Surprisingly, our experiments show that the results are not sensitive to the size of the anchor, as long as a sufficient number of scales is usod.
i) Fized-Seale Allention. Given anchors of m scales, we distribute them to atlention heads in a round-robin manner. Thus, in multi-head fixed-scale attention, 1 attention heads are allocated for each of the anchor scales. This uniform distribution of different scales enables fixed-scale attention to learn diverse information from local to more global context. The aggregation of n heads results in scale-aware features, that is suitable for predicting objects of different sizes.
ii) Adoptive-Scale Attention. Instead of uniformly assigning m scales to n attention heads, the adaptive-scale attention learns to allocate a scale distribution based on the context of the query vector. This comes from the motivation that the query vector belonging to small object should use more attention heads for capturing fine-grained details rather than global context, and vice versa.
5

---

Published in Transactions on Machine Learning Research (02/2025)
linear
linear
Anchor offsets per scale
softmax
4,44,
4
A Ay Aey
Antentin weights acales
0.3
0,2
query
aggregate
scale 1
head
scale
scale 3
Figure 2: Illustration of the proposed adaptive-scale attention. In the i-th attention head, given a query vector and anchors of three scales, adaptive-scale attention learns attend region of interest w.r.t. each scale. It then generates atlention weights these scales adaptively based on the query vector to produce head, As this mechanism allows each vector in our plain feature map learn suitable scale information during the training process, no longer need hierarchical backbone along with feature pyramids.
In each attention head, given the quory vector & E Rd in the input feature map and m anchors different scales, {r the adaptive-scale attention predicts offsets all anchors, (Az,; An,s Aw, An, 1m-1 Besides, we apply scale temperature to each set of offsets before the transformations:
2
2
Fscale
F. D. + Aw
A
(7)
X
2
2
Firanslate (T5, =
x Ar
1 Ay
W. h
(8)
X
where X the scale temperature corresponding to ry. The scale temperature allows the transformation functions capture regions of interest, ry, corresponding to the scale of anchors. The adaptive-scale attention produces feature vector per scale from the corresponding region of interest. The attention weights scales are then generated by applying lincar projection on query vector followed by softmax normalization. The attended feature, head, is the weighted sum of feature vectors with the corresponding attention scores of m scales (see Fig. 2). This allows the attention mechanism to learn suitable scale distribution based on the content of query vector. The adaptive-scale attention provides efficiency due to sparse sampling and strong flexibility to control scale distribution via its attention computation.

---

Published in Transactions on Machine Learning Research (02/2025)
Figure 3: Masked Instance-Attention Left: Box-attention (Nguyen et al., 2022) samples 2 x 2 grid features in the region of interest. Right: Proposed masked Instance-attention for dense grid sampling. The x attention scores are denoted in four colours and the masked attention scores are in white. The masked instance-attention preserves the efficiency by sparse sampling while better capturing objects of different shapes.
3.3 Object representation for panoptic segmentation
Panoptic segmentation proposed by Kirillov et al. (2019) requires the network to segment both "things" and "stuff" To enable the plain detector on panoptic segmentation, we make an adaptation the mask prediction of SimPI.R. Following Cheng et al. (2022), we predict segmentation masks of both types by computing the dot-product between object queries and high-resolution feature map (f.e., feature scale). As the ViT and SimPLR encoder features of lower resolution, simply interpolate the last encoder layer to scale and stack K=1 scale-aware allention layer top. This simple modification produces a high resolution feature map that is beneficial for learning fine-grained details.
Masked instance-attention In order to learn better object representation in the decoder, we propose masked instance-attention that is also a sparse attention mechanism for efficiency. The masked instance- attention follows the grid sampling strategy of the box-attention Nguyen et al. (2022), but improves the attention computation to better capture objects of different shapes.
To be specific, the region of interest r' is divided into 4 bins of x 2 grid, each of which contains x grid features sampled using bilinear interpolation. Instead of assigning
attention weight to each feature vector, linear projection (Rd - R2x2) is adopted to generate the 2 x 2 attention scores for 4 bins. The 2 x m feature vectors within the same bin share the same attention weight. Inspired by Cheng et al. (2022), we utilize the mask prediction the previous decoder layer as the attention mask to the attention scores (sco
?
Fig. 3).
A = repeat(a)
(9)
head, = softmax + MMV,
(10)
where on is the attention weight corresponding to k-th bin with & E {1,..., 4), A E Rm is the attention scores expanded from a, M {0, inf) indicates whether to mask out the value, and V e mm is the value features sampled from r'. By utilizing the mask prediction from previous decoder layer, masked instanco-attention effectively capture objects of different shapes.
3.4 Network architecture
SimPLR follows the common end-to-end detection and segmentation framework with a two-stage design (Carion et al., 2020; Zhu et al., 2021; Nguyen et al., 2022). Specifically, we use plain ViT as the backbone with 14 x 14 windowed allention and four equally-distributed global altention blocks as in Li et al. (2022). In the detection head, the SimPLR encoder receives input features via a projection of the last feature map from the ViT backbone. The object proposals are then generated using single-scale features from the encoder and top-scored features are initialized as object queries for the SimPLR decoder to predict bounding boxes and masks.

---

Published in Transactions on Machine Learning Research (02/2025)
Formally, apply projection f to the last feature map of the pre-trained VIT backbone, resulting in the input feature map e E BRH. xd where He, We are the size of the feature map, and d is the hidden dimension of the detection head. In SimPLR, the projection f is simply a single convolution projection, that provides us a great ficxibility to control the resolution and dimension of the input features e to the encoder. The projection allows SimPLR to decouple the feature scale and dimension between its backbone and detection head to further improve the efficiency. This practice is different from the creation of SimpleFPN in Li et al. (2022) where different stack of multiple convolution layers is used for each feature scale. We show by experiments that this formulation is key for plain detection and segmentation while keeping network efficient.
Plain backbone. SimPLR deploys ViT as its plain backbone for feature extraction. We show that SimPLR can take advantages of recent progress in self-supervised learning and scaling ViTs, To be specific, SimPLR generalizes to ViT backbones initialized by MAE (He et al., 2022) and BEiTv2 (Peng et al., 2022). The efficient design of SimPLR allows us to effectively scale to larger ViT backbones which recently show to be even more powerful in learning representations (He et al., 2022; Zhai et al., 2022; Dehghani et al., 2023).
4 Implementation Details
The creation of input features. In Fig. 4, we compare the creation of input features to detection head between SimpleFPN and our method. Tn Li et al. (2022), the multi-scale feature maps are created by different. sets of convolution layers. Instead, SimPLR simply applies deconvolution layer following by GroupNorm layer (Wu & He, 2018).
SimpleFPN (PlainBBackbone ViTDet)
Single-scale (Ours)
1/32
1/16
1/B
1/4
conv
cDnV
conv
conv
conv
conv
conv
conv
pool
transposed
transposed
transposed
cOnV
conv
conv
transposed
Neck
conv
Neck
ViT
VIT
Figure 4: The creation of input features. Left: The creation of feature pyramids from the last feature of the plain backbone, VIT, in SimpleFPN (Li al., 2022) where different stacks of convolutional layers are used to create features at different scales. Right: The design of our single-scale feature map with only one layer.
Losses in training of SimPLR. We use focal loss (Lin et al., 2017) and dice loss (Milletari et al., 2016) for the mask loss: Luasek Afocal Crocal + Ldice with Afocal Adice 5.0. The box loss is the combination of & loss and GloU loss (Rezatofighi et al., 2019), Lhox de, Li, + >giion with de, 5.0 and Agion = 2.0. The focal loss is also used for our classification loss, Leb. Our final loss is formulated as: L= = Luask + Lbox + Cets (Aels = 2.0 for object delection and instance segmentation, dets- = 4.0 for panoptic segmentation).
Hyper-paramctors of SimPLR. SimPLR contains 6 encoder and decoder layers. The adaptivo-scale attention in SimPLR encoder samples 2 x 2 grid features per region of interest. In the decoder, we compute masked instance-attention on a grid of 14 x 14 features within regions of interest. The dimension ratio of fora-forward sub-layers is 4. The number of object queries 300 in the docoder as suggested in Nguyen et al. (2022). The size of the input image is 1024 1024 in both training and inference. Note that we also use this setting for the baseline (i.e., BoxeR with VIT backbone).
8

---

Published in Transactions on Machine Learning Research (02/2025)
backbone
detection head
model size
dim.
# heads
feature scale
encoder dim.
decoder dim.
# heads
feature scale
Base
768
12
384
256
Large
1024
16
1
768
384
16
T6
Huge
1280
16
960
384
T6
Table 1: Hyper-parameters of backbone and detection head for different sizes of SimPLR (base large Note that these settings are the same for all three tasks.
huge models).
Method
FPS
A FPS [%]]
APb
Apm
Feature pyramids
DETR (Carion et al., 2020)
15
125%
46.5
n/a
DeformableDETR (Zhu et al., 2021)
100%
54.6
n/a
BoxeR (Nguyen al., 2022)
100%
55.4
47.7
Plain detector
PlainDETR (Lin et al., 2023)
12
100%
53.8
n/a
SimPLR
140%
55.7
Table 2: SimPLR is effective end-to-end detector. Our comparison is between SimPLR with adaptive-scale attention and other end-to-end detectors (e.g., BoxeR with box-attention, DeformableDETRR with deformable attention, and PlainDETR with self-attention). For fair comparison, we report results of all methods using the VYT-B pre-trained with MAE as backbone. For methods that take feature pyramids as input, we employ SimpleFPN with ViT from (Li al., 2022). Despite being a plain detector, SimPLR shows competitive performance compared to multi-scale alternatives, while being 40% faster during inference.
In the experiment, we show that the decouple between feature scale and dimension the VIT backbone and the detection head helps to boost the performance our plain detector while maintaining the efficiency. This comes from the fact that the complexity of global self-attention the ViT backbone increases quadraticaly w.r.t. the feature scale and the detection head enjoys the high-resolution input for object prediction. Note that with ViT-H as the backbone, follow Li al. (2022) to interpolate the kernel of patch projection from the pre-trained 14 x 14 into 16 x 16. The hyper-parameters for each SimPLR size (Base, Large, and Huge) are in Tab. 1.
5 Experiments
Experimental setup. We evaluate our method on COCO (Lin al., 2014), a commonly usod datasct for object detection, instance segmentation, and panoptic segmentation tasks. In addition, we also report the performance of our method on panoptic segmentaion using the CityScapes dataset (Cordts et al., 2016). By default, we use plain features with adaptive-scale attention described in Sec. 3 due to its strong performance and ability to perform both detection and segmentation; and initialize the VIT backbone from MAE (He et al., 2022) pre-trained on ImageNet-1KK without any labels. In both fixed-scale and adaptive-soale attention, are the
we set the number of scale m=4 and the anchor size s-32. Unless specified, our same as in (Nguyen et al., 2022). The frames per second (FPS) of all methods are measured on an A100 GPU. For all experiments, our optimizer is AdamW (Loshchilov & Hutter, 2019) with a learning rate of 0.0001. The learning rate is linearly warmed up for the first 250 iterations and decayed 0.9 and 0.95 fractions of the total number of training steps by factor 10. ViT-B (Dosovitskiy et al., 2021) is set as the backbone. The input image size is 1024 x 1024 with large-scale jitier (Ghiasi et al., 2021) between scale range of [0.1, 2.0). Due to the limit of our academic computational resources, we report the ablation study using the standard 5x schodule setting with a batch size of 16 as in Nguyon et al. (2022). In the main experiments, we follow the finetuning recipe from Li et al. (2022).
SimPLR is an effective end-to-end detector. In Tab. 2, we first show the comparison between SimPLR and other end-to-end object detectors, with all of them using the plain ViT backbone. Note that SimPLR with scale-aware attention removes the need for multi-scale adaptation of the VIT.
9

---

Published in Transactions on Machine Learning Research (02/2025)
We observe that SimPLR is better than multi-scale end-to-end detectors in both detection and segmentation. Specifically, SimPLR reaches 55.7 in APb and 17 Taaee-er-secone using only single-scale input. In terms of accuracy, we outperform DETR and are slightly better than BoxeR (55.4). Notably, our plain detector ~40% faster runtime. We also compare with PlainDETR (Lin et al., 2023) which a recent plain object delection method. As discussed Sec. 2, PlainDETR and our work approach the plain detector in different ways. PlainDETR designs a strong decoder that compensates for single-scale input, while our goal is learn scale equivariant features in the backbone and the encoder. As shown in Tab. 2, SimPLR improves PlainDETR by ~ 2 AP point in object detection. Moreover, SimPLR also supports instance segmentation, where PlainDETR does not. Despite the different approaches, both PlainDETH and our work indicate that plain detection holds a great potential.
Huge
Feature Pyramids
Swin
MVT
51
Large
MTDet
Mask2Former
50
Plain Features
SimPLR
Apm
49
Base
48
SimPLR
MVIT
Mask2Former
47
VTDett
Swin
46
40 ms
80 ms
160 ms
320 ms
# Run time (log-scale)
Figure 5: Scaling comparison. We compare plain detector, SimPLR, with recent multi-scale detectors including both end-to-end detector like Mask2Former and plain-backbone detector like ViTDet. Larger circles indicate bigger models (Base, Large, Huge). Because of its plain and simple architecture, SimPLR performance scales with size. Our biggest model achieves stronger performance and faster runtime than all its multi-scale counterparts.
Scaling comparison. Fig. 5 illustrates the tradeoffs between accuracy and runtime for various recent. detectors, plotted log-scale graph. The y-axis represents Average Precision (AP), while the x-axis shows runtime in milliseconds. Different models are indicated using colored markers, with larger circles representing bigger model sizes (Base, Large, Huge). The detectors compared include feature pyramid- based models such as Swin, MVIT, ViTDet, and Mask2Former, as well as our plain-feature detector, SimPLR. The graph demonstrates that SimPLR achieves considerable improvements both accuracy and runtime efficiency as model size increases.
Specifically, the huge model of SimPLR reaches accuracy of 51.9 AP, maintaining similar runtime to the large model of ViTDet, which achieves 49.9 AP. Moreover, SimPLR's huge model is faster than Mask2Former, which achieves an accuracy of 50.1 AP in instance segmentation. These results highlight SimPLR's scaling efficiency and effectiveness as a detector. The graph and corresponding data support the conclusion that SimPLR is a scale-officient detector, offering onhanced performance and faster runtimes compared to its multi-scale counterparts.
State-of-thh-art comparison. We show Tab. that SimPLR obtains strong performance on object detection and instance segmentation. When moving huge model, our method outperforms multi-scale counterparts like the FaskeFormer segmentation model (Cheng al., 2022) by 1.8 AP point. Despite
10

---

Published in Transactions on Machine Learning Research (02/2025)
Object Detection AP
Instance Segmentation Apm
methad
backhone pre-train
FPS
APh
AP AP,
AP
AP
AP
Feature pyramids
Swin (Láu et al,, 2021)
Swin-T,
sup-21K
55.0
38.3
59.4
71.6
47.2
28.7
50.5
66.0
10
Mask2Former (Cheng et al., 2022)
Swin-I
sup-21K
n/a
50.1
29.9
53.9
72.1
MVIT (Pan al., 2021)
MViT-L
sup-21K
55.7
40.3
59.6
71.4
48.3
31.1
51.2
66.3
ViTDet (Li et al,, 2022)
ViT-L
MAE
57.6
40.5
61.6
72.6
49.9
30.5
53.3
68.0
MVTI (Li et al,, 2022)
MVTI-H
sup-21K
55.9
40.8
59.8
70.8
48.3
30.1
51.1
66.6
ViTDet (Id al,, 2022)
VIT-H
MAR
58.7
63.0
73.9
32.0
54.3
68.9
Plain features
SimPLR
VIT-L
MAE
58.5
42.2
62.5
73.4
50.6
32.
54.2
69.8
SimPLR
VIT-L BETIV2
58.7
40.4
63.2
74.8
50.9
30.4
55.1
70.9
SimPLR
VIT-H
MAE
59.8
42.2
63.8
74.9
51.9
32.2
55.7
71.0
Table State-of-the-art comparison and scaling behavior for object detection and instance segmentation. We compare methods using feature pyramids us. plain features on COCO val (n/a: entry is not available). Backbones with MAE pre-trained on ImageNet- 1K while others pre-trained on ImageNet-21K. With only single-scale features, SimPLR shows strong performance compared to multi-scale detectors including transformer based detectors like Mask2Former and plain-backbone detector like ViTDet.
COCO PQth
CityScapes PQ
method
backbone
pre-train
FPS
PQ
PQ"
Feature pyramids
Panoptic FON (Li et al., 2021)
Swin-L
sup-21K
Panoptic DeepLab (Cheng et al., 2020)
SWideRNet
sup-1K
66.4
MaskFormer (Cheng et al., 2021)
Swin-B
sup-1K
51.1
56.3
43.2
MaskFormer (Cheng et al., 2021)
Swin-B
sup-21K
51.8
44.1
Musk2Former (Cheng al., 2022)
Swin-B
sup-1K
55.1
61.0
46.1
7
MaskePormer (Cheng et al., 2022)
Swin-B
sup-21K
56.4
62.4
47.3
66.1
7
FaskeFormer (Cheng et al., 2022)
Swin-T.
sup-21K
57.8
64.2
48.1
66.6
4
Plain features
SimPLR
VIT-B
sup-1K
55.5
61.4
46.2
13
SimPLR
VIT-B
BETTV2
56.7
62.8
47.4
66.7
13
SimPLR
VIT-L
BETIV2
58.8
65.3
48.8
67.4
8
Table 4: State-of-the-art comparison and scaling behavior for panoptic segmentation. We compare between methods using feature pyramids (top row) ws. single-scale (bottom row) on COCO val and CityScapes val. FPS is reported on COCO. SimPLR with single-scale input shows better results when scaling to larger backbones, while being ~ 2x faster compared to Mask2Former.
involving more advanced attention blocks designs, i.e., shifted window attention in Swin (Liu et al., 2021) and pooling attention in MVTT (Fan et al., 2021), detectors with hierarchical backbones benefit less from larger backbones. SimPLR is better than the plain-backbone detector, ViTDet, across all backbones in terms of both accuracy (by ~ AP point) and inference speed. It also worth noting that SimPLR provides good predictions on small objects for both detection and segmentation, despite its reliance on single-scale input only.
Tab. 4 shows the comparison for the panoptic segmentation COCO and CityScapes datasets. We observe the similar trend that SimPIR indicates good scaling behavior. Specifically, in the large model, SimPLR outperforms Mask2Former by 1 PQ point while running 2x faster. We also explore the influence of supervised pre-training on the plain ViT backbone. To keep it simple, we compared Mask2Former with Swin-B and SimPLR with ViT-B both using supervised pre-training on ImageN We find that SimPLR still indicates stronger results when using only single-scale input.
Ablation on SimPLR pre-training data and strategies. Tab. 5 compares the ViT backbone of SimPLR when pre-trained using different rategies with varying sizes of pre-training data. As expected, SimPLR with
11

---

Published in Transactions on Machine Learning Research (02/202))
Object
Detection
Instance Segmentation
data
strategy
APh
AP
AP
APp
Apm
AP
AP
AP
Supervised pre-training
IN-IK
DEIT
53.6
33.7
58.1
71.5
46.1
24.5
50.4
67.2
IN-IK
DEVTv3
54.0
34.3
58.8
70.5
46.4
24.8
51.1
66.7
IN-21K DEMTv3
54.8
35.4
59.0
72.4
47.1
25.8
51.2
68.5
Self-supervised pre-training
IN-1K
MAE
55.4
36.1
59.1
70.9
47.6
26.8
51.4
67.1
IN-21K
BEITV2
55.7
36.5
72.4
48.1
26.7
68.9
Table 5: Ablation SimPLR pre-training data and strategies with the plain VIT-B backbone evaluated COCO object detection and instance segmentation. We compare the plain backbone pre-trained using supervised methods (lop row) self-supervised methods (boftom row) with different sizes of pre-training dataset (ImageNet-IIK us. ImageNet-21K). It can be seen that SimPLR with the plain VIT backbone benefits more pre-training data (e.g.. ImageNet-1KK Im22eN))-21K) and better pre-training approaches (e.g., supervised learning vs. sell-supervised learning). Here, we use the 5x schedule.
attention
APb
Apm
Apb
Apm
Apb
AP"
base
53.6
46.1
base
53.6
46.1
base
53.6
46.1
fixed-scale
55.0
47.2
16
55.1
47.4
54.6
47.0
adaptive-scale
55.4
47.6
32
55.4
47.6
4
55.4
47.6
55.1
47.4
55.2
47.6
(a) Scale-aware attention.
(b)
Anchor size.
(c) Number of anchor scales.
Adeaptive-ccale atteation (with d-scale)
64
120 128
236
meibum umat
Size of detected objects
scale
APb
Apm
base:
53.6
46.1
1/4
55.4
47.7
1/8
55.4
47.6
Visualization of scale distribution (e)
1/16
54.3
46.7
learnt in multi-head adaptive-scale attention of object proposals. Objects are classified into small, medium, and large based on their area.
(d) Scales of input features.
Table Ablation of scale-aware attention SimPLR using a plain VIT backbone on COCO val. Tables (a-d): Compared to the baseline, which employs BoxeR and box-attention (Nguyen et al., 2022) with single-scale features, SimPLR, with scale-aware attention improves performance consistently for settings (default setting highlighted). Figure (e): our adaplive-scale attention captures different scale distribution in its attention heads based on the context of query vectors. Specifically, queries of small objects tends to focus on anchors of small scales (t.e., mainly 32 32), while query vectors of medium and large objects distribute more attention computation into larger anchors. All experiments are with 5x schedule.
the ViT backbone benefits from better supervised pre-training methods. Among them, DEÎTv3 (Touvron et al., 2022) shows better results than DEiT (Touvron et al., 2021), and pro-training on more data (i.e., ImageNel)21K) further improves DEiTv3 performance. SimPLR with the ViT backbone benefits even more from self-super methods like MAE (He et al., 2022), which already provide strong pre-trained backbones when only pro-trained on ImageNet-1K: The best performance is reported with self-supervised method, BEiTv2 (Peng al., 2022) with more pro-training data of ImageNet-21K. This further confirms that SimPLR, profits from the progress of self-supervised learning and scaling ViTs with more data.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Ablation of scale-aware attention. Next, we ablate the scale-aware attention mechanism. As bascline we compare to the standard box-attention from Nguyen et al. (2022) with single-scale feature input directly taken from the last feature of the VIT backbone (denoted as "base").
From Tab. 6a, we first conclude that both scale-aware attention strategies are substantially better than the baseline, increasing AP by up to 1.8 points. We note that while fixed-scale attention distributes 25% of its attention heads into each of the anchor scales, adaptive-scale attention decides the scale distribution based the query content. By choosing feature grids from different scales adaptively, the adaptive-scale attention is able to learn suitable scale distribution through training data, yielding better performance compared to fixed-scale attention. This is also verified in Fig. 6e where queries corresponding to small objects tend to pick anchors of small sizes for its attention heads. Interestingly, queries corresponding to medium and large objects pick not only anchors of their sizes, but also ones of smaller sizes. A reason may be that performing instance segmentation of larger objects still requires the network to faithfully preserve the per-pixel spatial details.
In Tab. 6b, we compare the performance of SimPLR across several sizes (s) of the anchor. They all improve over the baseline, while the choice of specific base size makes only marginal differences. Our ablation reveals that the number of scales rather than the anchor size plays an important role to make our network more scale-aware. Indeed, in Tab. 6c, the use of 4 or more anchor scales shows improvement up to 0.8 AP over 2 anchor scales; and clearly outperforms the naive baseline. Last, we show in Tab. 6d that the decouple between feature scale and dimension of the VIT backbone and the detection head features helps to boost the performance of our plain detector by ~1 AP point, while keeping its efficiency. This practice makes scaling of SimPLR to larger VIT backbones more practical.
We provide qualitative results for object detection, instance segmentation, and panoptic segmentation on the COCO dalaset in Fig. 6 and panoptic segmentaion on the CityScapes dataset in Fig. 7.
Limitations. Our final goal is to simplify the detection pipeline and to achieve competitive results at the same time. We find that the adaptive-scale attention mechanism that adaptively learns scale-aware information in its computation plays a key role for a plain detector. However, our adaptive-scale attention still encodes the knowledge of different scales. In the future, we hope that with the large-scale training data, a simpler design of the attention mechanism could also learn the scale cquivariant property. Furthermore, SimPLR faces difficulties in detecting and segmenting large objects in the image. To overcome this limitation, we think that design of attention computation which effectively combines both global and local information is necessary.
6 Conclusion
We presented SimPLR, a simple and plain object detector that eliminates the requirement for handerafting multi-scale feature maps. Through our experiments, we demonstrated that a transformer-based detector, equipped with a scale-aware attention mechanism, data. The elficient design of SimPLR allows it to take advantages of significant progress in scaling ViTss, reaching highly competitive performance on three tasks on COCO: object detection, instance segmentation, and panoptic segmentation. This finding suggests that many handcrafted designs for convolution neural network in computer vision could be removed when moving to transformer-basee architecture. We hope this study could encourage future exploration in simplifying neural network architectures especially for dense vision tasks.
effectively learn scale-oquivariant features through
References
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert pre-training of image transformers. In ICLR, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Noclakantan, Pranav Shyam, Girish Sastry, Sandhini Agarwal Amanda Askell, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark,
13

---

Published in Transactions on Machine Learning Research (02/2025)
Instance
Panoptic
Instance
Panoptic
Figure 6: Qualitative results for SimPLR instance detection and segmentation (left) and panoptic segmentation (right) on the COCO val set. PLR gives good predictions on small objects (e.g., the first three rows). However, it is still challenging for the model to detect individual objects in the crowded scenes (e.g., the last row).
Figure 7: Qualitative results of panoptic segmentation generated by SimPLR on Cityscapes val. It can be seen that SimPLR correctly predicts small objects like pedestrians various scenes.
Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoftrey Hinton. simple framework for contrastive learning of visual representations. In ICMT, 2020b.
Wuyang Chen, Xianzhi Du, Fan Yang, Lucas Beyer, Xiaohua Zhai, Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, and Denny Zhou. A simple single-scale vision transformer for object localization and instance segmentation. In ECCV, 2022.
Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Glang-Chich Chen. Panoptic-deeplab: simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, 2020.
Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. In NeurIPS, 2021.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Mask2former: Masked-attention mask transformer for universal image segmentation. In CVPR, 2022.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, and al. Scaling vision transformers to 22 billion parameters. In ICML, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. CVPR, 2009.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In ACL, 2019.
Bin Dong, Fangao Zeng, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. SOLQ: Segmenting objects by learning queries. In NewrIPS, 2021.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worthh 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR, 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017.
Kaiming He, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autocncoders are scalable vision learners. In CVPR, 2022.
Bycongho Hco, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. In ICCV, 2021.
Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. DAC-DETR: Divide the attention layers and conquer. In NeurIPS, 2023.
15

---

Published in Transactions on Machine Learning Research (02/2025)
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, 2023.
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollar. Panoptic segmentation. CVPR, 2019.
Harold W. Kuhn. The Hungarian method for the assignment problem. Naval Res. Logist. Quart, 1955.
Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech and Time Series, pp. 255-258 MIT Press, 1995.
Feng Li, Hao Zhang, Huaizhe xu, Shilong Liu, Tei Zhang, Lionel M. Ni, and Heung- Young Shum. Mask dino: Towards a unified transformer-basod framework for object detection and segmentation. In CVPR, 2023.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang. Zeming Li, Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, 2021.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. In ECCV, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. CVPR, 2017.
Tsung-Y? Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Yutong Lin, Yuhui Yuan, Zheng Zhang, Chen Li, Nanning Zheng, and Han Hu. DETR does not need multi-scale or locality design. In ICCV, 2023.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexan- der C. Berg. SSD: Single shot multibox detector. In ECCV, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
Duy-Kien Nguyen, Jihong Ju, Olaf Booij, Martin R. Oswald, and Cees G. M. Snoek. Boxer: Box-attention for 2d and 3d transformers. In CVPR, 2022.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint Xiv::2200. 06366, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NewrIPS, 2015.
Seyed Hamid Rezatofighi, Nathan Tsoi, Jun Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese. Generalized intersection over union: A metric and A loss for bounding box regression. In CVPR, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for largo-scale image recognition. Tn JCLR, 2015.
16

---

Published in Transactions on Machine Learning Research (02/2025)
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient. image transformers and distillation through attention. In International Conference on Machine Learning, 2021.
Hugo Touvron, Matthieu Cord, and Herve Jegou. Deit iil: Revenge of the vit. Xi preprint arXi::2204007118 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Blang-Chiel Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In CVPR, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021b.
Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017.
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang- Chich Chen. k-means mask transformer. In ECCV, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Houng- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. In NewrIPS, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In ICCV, 2023.
17


Reviews:
[{"review_id": "wYkzJK3i7S", "reviewer": "Reviewer_cMeH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- A. I believe the \"engineering contributions\" discussed above are of possible interest to the community and shows that a sparse scale-aware attention can be relatively fast and have good performance vs commonly adopted architecture choices (ViTDet, Deformable DETR)\n\n- B. Presentation is clear \n\nWeaknesses:\n\n - A. A central claim in the paper is that multiscale feature maps and pyramid designs are _unnecessary_, but there is a lack of experimental evidence for this scientific claim. For e.g. results in Table 2 are all using a ViT-B, a plain, non-hierarhichal backbone and so there isn't a true FPN. If there isn't a comparison to a true hierarchichal backbone (e.g. Swin, Hiera trained w/ MAE) in a controlled experiment setup (as in Table 2), I'm not sure how it can be claimed that multiscale feature maps and pyramid designs are unnecessary - an alternative hypothesis could simple be that SimpleFPN isn't very effective.\n\nRelatedly, another important baseline here is a plain backbone using (single scale) features from multiple layers - after all, an FPN is not just about multiscale features but that high-resolution features come from earlier in the network.\n\n- B. A core motivation of the work is to question the role of and remove hand-crafted inductive biases (such as multiscale feature maps, pyramids) and see how far we can get with the ViT-style approach of learning from data instead of baking such biases in. And yet, as a reader, respectfully, it feels a bit like a bait-and-switch. Instead of having multiscale feature maps, a multiscale bias is instead pushed into the attention mechanism. The distinction that the detector still uses single scale feature maps strikes me as artificial (scientifically speaking).\n\n- C. It worries me that all results for Table 2 use the same hyperparameters and could result in bias. For e.g. ViTDet swept over hyperparams for other methods (e.g. Swin) to mitigate.\n\n- D. I worry how COCO specific (e.g. small number of classes) the detection/instance seg results are. While the alternatives of ViTDet, Swin are known to also work well on LVIS. Having some LVIS number is an important baseline in current literature in detection/instance seg.\n\n- E. Were all the FPS number in Figure 5 run by you? Maybe I missed it what detection framework each of the baselines in the figure are using? \n\n- F. Were all relevant FPS numbers run with optimized attention kernels (e.g. flashattention v2)? e.g. DETR baseline in Table 2. Can the authors compare the speed of their sparse detection head with baselines using flashattention? what's the precision used?\n\nThis figure has been used to claim that SimPLR is \"more scaling efficient\". To support such a claim, I would have expected relevant baselines (e.g. from table 2) scaled as well. As the figure currently stands, I'd read it instead as SimPLR's performance scales with size, rather than SimPLR is more scaling efficient than alternatives.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes \"a plain detector\" for object detection and segmentation, where \"plain\" means that the backbone is not hierarchichal (e.g. ViTDet) and multiscale feature maps (e.g. Deformable DETR) are not used in the detection/segmentation modules\n\nFrom the perspective of \"engineering contributions\":\n\n- Gains in speed and accuracy have been reported, e.g. 1.4x faster than ViT-B based Deformable DETR (+1.1 box AP) and BoxeR (+0.3 box AP) baselines\n\n- SimPLR performs reasonably at larger scale as well, e.g. +1.1 vs VitDet-H on COCO detection and has reasonable performance on panoptic segmentation", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Is it possible to add experiments to back some of the scientific claims? e.g. see weakness A. Alternatively, would the authors be ok with relaxing the claims to something more along the lines of \"shifting the multiscale inductive bias into the attention mechanism can work well, resulting in speed and accuracy gains\"?\n\n - Would the authors be ok with relaxing the claims of \"removing simplifying and removing multiscale inductive biases\" (see weakness B)?\n\n- Can FPS numbers wherever relevant, be reported with optimized attention kernels and please add additional details regarding baselines in Figure 5? Can you report the speed of the sprase, scale-aware attention layer vs a regular self-attention layer (using optimized kernels)? \n\n- Can the authors run appriopriate baseline for scaling for Figure 5 to support \"more scaling efficient\" or alternatively relax the claim to something along the lines of \"SimPLR performance scales with size\"?\n\n- [Not critical] Can you sweep over hyperparamters for the baselines? (see weakness C)\n\n- Can you report performance on LVIS? (see weakness D)", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IGxaecPGz0", "reviewer": "Reviewer_TLhE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main contribution of this work is a simplification of previous multiscale DETR-style pipelines.  The results are strong relative to the state of the art and computationally efficient.  This is a valuable contribution to the community and merits acceptance.\n\nThe main weaknesses of this work are writing quality and missing relevant baselines.\n\nOn writing:\n* The explanation of the proposed method is not well self-contained and requires familiarity with earlier papers (DETR, Deformable DETR, Boxer, etc).  While the first two are fairly well known in the community, I do not think Boxer is as well known and could use more details (i.e. pretend the reader has never read Boxer)\n* Fixed scale attention and adaptive scale attention — both of these seem very similar, I’m not sure if I understand why the adaptive approach would intuitively be that much better and they even seem similar in ablations.  \n* One confusing notational issue is that m here denotes the number of scales while in the original Boxer paper it is used to denotes the resolution of the grid used to sample within a box.  Consider using another letter?\n* Masked instance attention is also confusing as the effectiveness of the design is not ablated and it is not motivated.  The 2 × 2 grid seems to come from nowhere... it is never mentioned here and I was only able to find it after looking in the appendix of the Boxer paper.  Perhaps it would be better to write this more generally for kxk grids and offer a motivation for this module?  I would also suggest a more clear picture of this architecture.\n* Is there a clearer way to explain what makes the SimPLR approach faster than other approaches?  \n\nFor relevant baselines (in the state-of-the-art comparisons), \n* In what sense is this table a fair comparison across approaches?  Are, e.g. input resolutions fixed across models?\n* The CoDETR paper for example reports reaching 65 box AP points...  I don’t think this detracts from whether this paper should be accepted, but should be cited.\n* The VitDet paper also achieves better numbers than what is reported in this submission (see table 6 in the VitDet paper)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an approach for object detection and instance/panoptic segmentation following in the general approach of DETR style detectors where a pretrained feature extractor is followed by processing using a transformer encoder/decoder.  Following a series of recent works, this paper replaces the attention in the transformer part of the model by a more targeted form (box attention) where the attention is restricted to grid points sampled from a bounding box.  The novel contribution in this paper is that the bounding boxes are predicted relative to anchors with multiple scales.  \n\nThis approach multi-scale-in-anchors (but not multi-scale-in-image-features) approach allows them to get strong results without using multiple input resolutions from a feature pyramid and simplifies the architecture somewhat.  And I would note that it is conceptually similar to earlier single stage anchor-based detectors like SSD (Liu et al). Results on standard academic benchmarks are quite strong.  Within DETR style detectors it has competitive (even SOTA) box mAP and mask mAP and is competitive with the SOTA in general and is fast and arguably a simpler approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, addressing the issues raised in the above would strengthen the work in my opinion but are not critical to securing my recommendation for acceptance (I do recommend acceptance).\n\nSome other thoughts/suggestions to consider:\n* One question to investigate is whether SimPLR trains faster than prior work since the FPN presumably would require extra time to train after initialization from the base network.\n* If I were playing devil’s advocate I would say that there is still more headroom for simplification and the multiscale anchor design is not that simple conceptually — there are details here for example that could easily lead to errors in implementation.  I would also recommend a discussion of what specific further simplifications researchers might want to explore", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uz9lPUePOD", "reviewer": "Reviewer_ajHh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "> Strengths \n\n- The motivation behind this work is compelling, as it explores a promising direction focused on developing simple and straightforward architectural designs for fundamental visual recognition tasks, including object detection and panoptic segmentation. \n- The writing is clear, and the proposed scale-aware attention method is simple and easy to follow.  \n  \n> Weaknesses\n  \n- The key design of this work lacks deep technical contributions, and similar ideas have been thoroughly analyzed in previous works such as Deformable DETR, Mask2Former, and Plain-DETR. \n\n- The effectiveness of the proposed method is not convincingly demonstrated. The authors should elucidate why such a simplification of multi-scale deformable attention can outperform previous, more advanced designs. The following major concerns should be addressed:  \n  \n1. **Efficiency Comparison (Table 2)**: Why is SimPLR more efficient than Plain-DETR? Are the comparisons fair? How does the choice of backbone and the number of parameters affect the results? It is acknowledged that the plain Transformer decoder design is less efficient than Deformable attention; if this is the primary reason, the advantage is due to the sparsity introduced by Deformable attention.  \n  \n2. **Performance Comparison (Tables 3 and 4)**: Why does the proposed SimPLR outperform previous methods like ViTDet and Mask2Former? How would the combination of Deformable attention with ViTDet perform? Additionally, the authors should include comparisons with Plain-DETR (AP=60 or 63.9 with or without pre-training on Object 365).  \n  \n3. **Mathematical Formulation**: The mathematical formulation is confusing. For instance, in the section on adaptive-scale attention, what does $[r_j]_{j=0}^{m-1}$ represent? It is assumed to denote the region proposals associated with different scales. The authors should clarify this notation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this paper lies in the introduction of a scale-attention scheme (multi-head scale-aware) aimed at enhancing the multi-scale capability of a given single-scale feature map. According to the description in Section 3.2 and illustrated in Figure 2, the proposed adaptive-scale attention can be seen as a simplification of the more powerful multi-scale deformable attention proposed in the previous work, Deformable DETR [1]. Deformable DETR predicts adaptive offsets not only across different attention heads but also across various feature map scales.\nRegarding the masked instance-attention, it is fundamentally the same as the masked attention mechanism introduced in Mask2Former [2]. Additionally, the proposed use of a single-scale feature map based on a plain ViT has been thoroughly investigated in Section 5.4 of the prior work, PlainDETR [3].\nThe effectiveness of the proposed method is not convincingly demonstrated. Based on my experience, the observed gains over previous methods such as Deformable DETR, Mask2Former, and PlainDETR seem to stem more from the selection of superior pre-trained backbones or the deformable attention design, like BEiTv2 and DeformableDETR, rather than the proposed scale-attention scheme.\n\n\n[1] Deformable DETR: Deformable Transformers for End-to-End Object Detection, ICLR’2021\n\n[2] Masked-attention Mask Transformer for Universal Image Segmentation, CVPR’2022\n\n[3] DETR Does Not Need Multi-Scale or Locality Design, ICCV’2023", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should enhance this work by designing a novel and effective module that genuinely outperforms previous approaches or by offering new insights to the community. The current version fails to meet these criteria. Additionally, the authors should address the aforementioned weaknesses.", "broader_impact_concerns": "There are no concerns regarding the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommend acceptance (although one only leans toward acceptance) while one reviewer leans toward rejection. After examination of the paper, reviews, and responses the action editor sides with acceptance. All reviewers agree that the work is in some dimensions simpler than other approaches, although there is disagreement about the degree and novelty of the proposed architecture and sparse multi-scale attention. Furthermore all reviewers accept the empirical results on accuracy and speed, even if further results are requested, and the provided experiments are interpreted and justified in the discussion. The reviewer ajHh has made a case against the technical significance and novelty of the submission, with detailed discussion, but the action editor overrides this criticism in light of the TMLR criteria. The reviewer cMeH has pointed out concerns about the scope and magnitude of the claims, but these have been discussed in the rebuttal, and can be addressed by minor edits in the revision.

On the balance, the contribution of this work in the investigation of sparse attention for detection merits acceptance. The experiments show the feasibility and efficiency of this choice compared to multi-scale representations and detection architectures and demonstrate generality by addressing object detection, instance segmentation, and panoptic segmentation.

The action editor recommends acceptance with minor revisions in order to incorporate the feedback from review. In particular, the following suggestions deserve to be highlighted:

- Please incorporate the response https://openreview.net/forum?id=6LO1y8ZE0F&noteId=hxjDMmn5S8 to explain the fairness of the comparisons and additional clarifications.
- Please incorporate the response https://openreview.net/forum?id=6LO1y8ZE0F&noteId=r09UbE17jE to address the justified questions by reviewer ajHh. It is important that the controlled comparisons and the analysis of dependence on the backbone and model size are there, but these might need further signposting for the reader to easily find these results.
- Please incorporate the nuanced explanation of their contribution in this final response https://openreview.net/forum?id=6LO1y8ZE0F&noteId=yszK6UrbBy  in the introduction or discussion of the paper. The action editor agrees that the empirical results are non-trivial and the success of the proposed sparse attention across a single feature scale is informative.
- Please make minor edits to revise the claims, as discussed by the authors with cMeH, to propose SIMPLR as a viable alternative rather than proof that multi-scale design is unnecessary.

The action editor thanks the authors and the reviewers for engaging in the TMLR process, and congratulates the authors on their paper!
Decision: Accept (poster)